Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters

🎯 Pitch

An 11B active parameter MoE model matches GPT-5.2-level agentic performance while using an order of magnitude fewer active parameters. The paper solves the training instability that previously made efficient inference-class models collapse on complex reasoning by introducing MIS-PO, an RL method using discrete binary masking that slashes gradient variance. This proves that compute-efficient frontier intelligence is not a capacity problem but a training stability problem.

1. Executive Summary

This paper introduces Step 3.5 Flash, a sparse Mixture-of-Experts (MoE) language model with 196B total parameters and only 11B active per token, designed to deliver frontier-level agentic intelligence while minimizing inference latency and cost for real-world industrial deployment. The model is evaluated against leading closed and open-source systems across reasoning benchmarks (AIME 2025, IMO-AnswerBench, LiveCodeBench-v6), code agent benchmarks (SWE-Bench Verified, Terminal-Bench 2.0), and general agent benchmarks (𝜏2-Bench, BrowseComp, GAIA), achieving scores such as 85.4% on IMO-AnswerBench and 88.2% on 𝜏2-Bench—competitive with models like GPT-5.2 xHigh and Gemini 3.0 Pro despite activating an order of magnitude fewer parameters. The architecture co-designs three efficiency mechanisms: a 3:1 interleaved sliding-window-to-full-attention layout with head-wise gating (reducing long-context prefill and decode FLOPs versus uniform full attention), expert-parallel load-balanced MoE routing (preventing straggler-induced throughput collapse in distributed deployment), and lightweight multi-token prediction heads for speculative decoding (adding only 0.41% parameter overhead). For stable post-training at scale, the paper proposes MIS-Filtered Policy Optimization (MIS-PO), a reinforcement learning method that replaces continuous importance weighting with discrete, dual-level binary masking at both token and trajectory granularities, substantially reducing gradient variance and enabling reliable off-policy RL on long-horizon reasoning tasks for MoE architectures. The framework establishes that a model with 11B active parameters can match ~100B+ active-parameter frontier systems on both reasoning and agentic workloads only when training stability challenges—including Muon numerical sensitivity, localized activation blow-ups in deep MoE layers, and expert collapse beyond routing collapse—are systematically identified and mitigated through a comprehensive micro-batch-level observability stack.

2. Context and Motivation

The Core Problem: Efficiency Is the Missing Frontier for Agentic Intelligence

The paper addresses a specific, increasingly acute tension in the deployment of large language models: as models grow powerful enough to serve as autonomous agents, their computational cost becomes the primary barrier to practical adoption. The problem is not that models lack intelligence — the paper explicitly acknowledges that "open-source large language models have rapidly narrowed the performance gap with closed-source frontier systems across verifiable tasks" (Section 1). Rather, the bottleneck has shifted from capability to deployability. Models that can reason through complex multi-step problems, interact with tools, write and execute code, and conduct multi-turn research sessions also consume enormous amounts of compute per interaction. For many real-world use cases — on-device deployment, edge computing, resource-constrained industrial settings — running a model with hundreds of billions of active parameters per token is simply infeasible regardless of how intelligent it is.

The authors frame this as a three-way tension between intelligence, cost, and latency (Section 2.1). Prior work has largely optimized for the first two: building models that are as capable as possible within a training budget. But for agents — systems that interact with environments over many turns, often with long context windows — a third constraint dominates: wall-clock time. In interactive agentic workflows, they argue, "minimized latency translates directly to reduced wall-clock time for task completion, or conversely, allows for increased intelligence within a fixed time budget via test-time scaling" (Section 2.1). A model that takes 30 seconds to reason through a single step may be less useful in practice than a slightly less capable model that responds in 3 seconds, because the faster model can be allowed more reasoning steps within the same time budget.

This efficiency imperative is particularly acute for long-context agentic workloads, which exhibit a distinctive computational profile: "extensive context prefilling followed by prolonged, multi-turn interactive decoding" (Section 2.1). Prefilling a 128K-token context incurs quadratic attention costs. Multi-turn decoding generates thousands of tokens across dozens or hundreds of interaction rounds. Together, these costs can make agentic deployment economically unviable at scale, even when the underlying model is capable enough to solve the task.

Why This Problem Matters Now

The paper identifies a shift in the AI landscape that makes this problem urgent. In 2024–2025, frontier models became capable enough to serve as autonomous agents — writing code in real repositories, navigating terminal environments, conducting multi-step web research, and managing customer service interactions across complex state spaces. Benchmarks like SWE-Bench Verified, Terminal-Bench 2.0, BrowseComp, and 𝜏2-Bench emerged to measure these capabilities, and frontier models began achieving non-trivial scores on them. But the models achieving these scores — GPT-5.2 xHigh, Claude Opus 4.5, Gemini 3.0 Pro, DeepSeek V3.2 — all activate 32B or more parameters per token (Table 5), making them expensive to serve at the throughput required for production agentic workloads.

The authors argue that open-source models face a particularly acute version of this problem:

"open-source models still trail closed-source frontiers in complex reasoning. Furthermore, critical efficiency bottlenecks hinder their application in long-context agentic tasks, let alone deployment in edge or resource-constrained settings." (Section 1)

This is a compound gap: open-source models lag in both raw capability and inference efficiency. Bridging either gap alone would be insufficient — a model that matches frontier intelligence but costs 10× more to serve is not deployable, and a model that is cheap to serve but cannot handle complex agentic tasks is not useful. The paper positions Step 3.5 Flash as an attempt to close both gaps simultaneously: achieving frontier-level performance on reasoning and agentic benchmarks while activating only 11B parameters per token, enabling deployment on hardware configurations (8-GPU nodes, high-end workstations with 128GB memory) that are practical for industrial use.

The Efficiency Landscape Before Step 3.5 Flash

To understand what Step 3.5 Flash contributes, it's necessary to survey the prior approaches to model efficiency and where they fall short. The paper engages with four broad strategies:

1. Sparse Mixture-of-Experts (MoE). The dominant approach to decoupling total knowledge capacity from per-token inference cost is sparse MoE, as pioneered in Switch Transformers, GShard, GLaM, ST-MoE, and DeepSeekMoE (Section 2.2, references [22–26]). In MoE architectures, each transformer block contains multiple "expert" feed-forward networks, and a learned router selects a small subset (typically top-𝑘) to activate for each token. This allows the model to possess far more total parameters than are used for any individual token, theoretically providing the knowledge capacity of a large dense model at the inference cost of a small one. Prior works — notably DeepSeek-V3, MiMo, GLM-4.5, LongCat-Flash, Kimi-K2, and MiniMax-M2.1 — had demonstrated that sparse MoE models in the 15–37B active parameter range could approach the performance of much larger dense models (references [1–6]).

However, sparse MoE introduces severe new training and deployment challenges that prior work had not fully resolved:

Training instability: MoE models are notoriously difficult to train at scale due to expert collapse (some experts receive negligible gradient signal and "die"), routing imbalance (token assignment concentrates on a few experts, creating stragglers), and localized activation pathologies (Section 4.1). Prior work had developed auxiliary load-balancing losses and loss-free balancing to address routing collapse at the dispatch level, but the paper observes that expert collapse can manifest "even when router dispatch remains stable" (Section 4.1.2) — a subtler failure mode that prior diagnostics would miss.
Deployment stragglers: Under expert parallelism (EP), where different experts are hosted on different GPUs, token assignment skew causes some GPUs to process many more tokens than others. At synchronization points (e.g., all-reduce after the MoE layer), all GPUs must wait for the slowest, throttling end-to-end throughput. The paper notes that "end-to-end latency can be dominated by stragglers induced by routing imbalance" (Section 2.1), and standard load-balancing losses that operate at the global batch level do not guarantee balance at the micro-batch level where deployment latency is determined.
Off-policy RL instability: MoE models amplify a problem already present in RL for language models. When a policy is updated, the router's expert assignments shift, causing the distribution of which experts are activated to diverge between the inference engine (where rollouts are generated) and the training engine (where gradients are computed). This training-inference mismatch produces high-variance gradients that destabilize optimization, particularly on long-horizon reasoning tasks where small per-token discrepancies compound across trajectories. The paper cites recent work bridging RL stability with MoE routing consistency (references [36–38]) and builds on this direction.

2. Sparse and Efficient Attention. The quadratic complexity of full self-attention with respect to sequence length has motivated a proliferation of alternatives: sliding-window attention (SWA), linear attention, sparse attention patterns, and hybrid layouts combining local and global attention. The paper references LongFormer, Gemma 3, Command A, and DeepSeek-V3.2-Exp as prior works using hybrid or sparse attention (references [1, 27, 48, 49, 58]). These approaches reduce prefill cost but introduce their own challenges:

Linear attention complicates speculative decoding: While linear attention (e.g., Transformers are RNNs, Fast Weight Programmers) achieves linear complexity, its recurrent state-update mechanism "complicates efficient draft tree generation and parallel tree verification needed for speculative decoding" (Section 2.1). Since speculative decoding is the paper's chosen method for reducing autoregressive latency, attention mechanisms that interfere with it are non-starters.
Sliding-window attention degrades long-range modeling: SWA limits each token's attention to a local window (e.g., 512 tokens), which is computationally efficient but loses the ability to attend to distant context. Prior works attempted to compensate with sink tokens (learnable, data-independent vectors that absorb unused attention mass) or by interleaving occasional full-attention layers. The paper's own experiments (Table 1, Table 10) show that a naive 3:1 SWA-to-full layout (denoted 𝑆3𝐹1) "consistently underperforms a dense attention baseline" by several points across benchmarks, and that sink tokens alone are insufficient to close the gap.
Existing hybrid layouts have an unfavorable cost–quality tradeoff: The paper's ablation (Table 1) on a 30B-A3B proxy model shows that a straightforward alternating SWA/full layout (𝑆1𝐹1) achieves the best downstream quality, including the strongest long-context score (29.6), but requires "substantially higher attention-side prefill/decode FLOPs (~1.58/1.65×)" compared to the 3:1 layout — about a 60% cost increase. This means prior hybrid attention designs either sacrificed too much quality (𝑆3𝐹1) or too much efficiency (𝑆1𝐹1).

3. Multi-Token Prediction and Speculative Decoding. To reduce autoregressive generation latency, prior work introduced multi-token prediction (MTP), where the model predicts multiple future tokens simultaneously, and speculative decoding, where a lightweight draft model generates candidate tokens that the main model verifies in parallel. DeepSeek-V3 and MiMo-V2-Flash had previously integrated MTP heads (references [3, 29, 30]), demonstrating that lightweight prediction modules could accelerate generation with modest parameter overhead. However, prior MTP implementations typically used the same attention mechanism as the main backbone, adding substantial computational cost. The paper's innovation here is minimal: replace full attention in MTP heads with SWA and dense FFNs, reducing the per-head parameter cost to 0.81B (0.41% of total) and making speculation lightweight enough to run without competing for bandwidth with the main model's decoding.

4. Reinforcement Learning for Post-Training. By early 2025, RL with verifiable rewards (RLVR) had become standard for improving reasoning capabilities in LLMs. Methods like DeepSeek-R1, DAPO, Open-Reasoner-Zero, and DeepSeekMath had demonstrated that RL could substantially boost math and code performance when a reliable reward signal is available (references [90, 94, 106, 129]). However, these methods face severe stability challenges when applied to MoE models at scale:

Importance sampling variance: RL for LLMs typically uses off-policy data: the inference engine generates rollouts under an older policy, and the training engine updates the current policy using importance sampling to correct for the distribution shift. At the token level, importance weights are the ratio of current to rollout policy probabilities. Over long trajectories, "minor token-level probability shifts compound into noisy gradients that impede convergence" (Section 5.2). This is the classic high-variance problem in off-policy RL, exacerbated by the enormous action space (128,896 tokens) and long horizons (thousands of tokens) of reasoning tasks.
Infrastructure divergence: Modern RL training pipelines separate the inference engine (for generating rollouts) from the training engine (for computing updates). These systems use different hardware configurations, different optimization settings, and sometimes different numerical precision, causing systematic discrepancies in the policy probabilities they compute. The paper notes this as a "primary source" of gradient variance (Section 5.2).
MoE-specific distribution shift: When the policy updates, the MoE router's expert assignments shift. This means the same token might be routed to different experts under the training and inference policies, creating a type of distributional shift that dense models do not experience. Recent work (GSPO, reference [36]) had proposed Group Sequence Policy Optimization to address this by aligning router behavior between training and inference, but the paper finds GSPO's efficiency and stability to be insufficient for large-scale MoE training (Figure 7).

Prior RL methods attempted to address these issues through importance weight clipping (PPO-style), conservative policy updates (trust-region methods), or router replay (freezing the router during RL). Each approach involves a tradeoff: clipping reduces variance but introduces bias; conservative updates slow learning; router replay prevents the model from adapting its expert usage to new capabilities acquired during RL.

Where Prior Approaches Collectively Fall Short

The paper's central claim is not that any individual prior technique is wrong, but that the combination of challenges — efficient attention for long contexts, stable MoE training, low-latency speculative decoding, and variance-controlled RL — had not been solved in an integrated system. The evidence for this claim is the performance landscape in Table 5: existing open-source models with comparable active parameter counts (MiMo-V2 Flash at 15B, GLM-4.7 at 32B, DeepSeek V3.2 at 37B) consistently trail frontier closed-source models on agentic benchmarks, and models that match frontier performance on reasoning (Kimi K2.5 at 32B active, DeepSeek V3.2) do so at substantially higher inference cost than 11B active parameters would allow.

The paper identifies four specific failure modes in prior work:

No prior model achieved frontier-level agentic performance with only 11B active parameters. The models closest in activation count — MiniMax M2.1 (10B active) and MiMo-V2 Flash (15B active) — achieve 60.4% and 80.9% on IMO-AnswerBench respectively (Table 5), versus Step 3.5 Flash's 85.4% and GPT-5.2 xHigh's 86.3%. On SWE-Bench Verified, they achieve 74.0% and 73.4% versus Step 3.5 Flash's 74.4%. The paper's claim is not that Step 3.5 Flash dominates these models on every metric, but that it is the first to achieve competitive or superior performance across both reasoning and agentic benchmarks while activating fewer parameters than any frontier-class alternative, narrowing the "compound gap" in both intelligence and efficiency simultaneously.
MoE training stability diagnostics were insufficient. Prior work had developed tools for detecting routing collapse (dead experts receiving negligible dispatch) and global load imbalance. The paper introduces finer-grained monitoring — per-expert activation norms, parameter norms, max-to-median ratios — that reveals failure modes invisible to standard diagnostics: "Router dispatch statistics are typically stable and are not sensitive indicators of expert collapse" (Section 4.1.2). This is an infrastructure contribution as much as a modeling one: the paper's lightweight asynchronous metrics server (Section 3.3) enables continuous logging at micro-batch granularity across 4,096 GPUs with only ~100ms overhead per iteration, making stability monitoring actionable at scale where prior systems would miss these signals or incur prohibitive telemetry costs.
RL methods for MoE models either sacrificed efficiency or stability. PPO with importance weighting suffers from high variance. GSPO reduces variance but "plateaus around iteration 200" on MoE models and "exhibits an increasing training-inference discrepancy during training" (Figure 7(b)). The paper proposes MIS-PO as a method that simultaneously achieves lower gradient variance (Figure 5, middle panel), higher sample efficiency (Figure 5, left panel), and controlled training-inference mismatch (Figure 7(b), right panel) by replacing continuous importance weights with discrete binary filtering at both token and trajectory levels.
Attention efficiency and model quality remained in tension. The paper's architecture ablations (Tables 1 and 2) systematically quantify this tradeoff and propose a solution: increasing SWA query heads from 64 to 96 (nearly a "free lunch" since "in long-text scenarios, the overhead of naive SWA is very small") plus head-wise gated attention as a learnable, data-dependent alternative to fixed sink tokens, together closing most of the quality gap to full attention while maintaining the 3:1 SWA ratio's efficiency advantage.

How This Paper Positions Itself

The paper does not claim to invent any of its individual components — sparse MoE, sliding-window attention, multi-token prediction, and reinforcement learning for reasoning are all established techniques. Its contribution is integration and optimization under a unified design philosophy (Section 2.1): that for agentic workloads, the architecture must be co-designed with the deployment infrastructure, the training stability infrastructure, and the post-training RL framework as a single system. This systems-level perspective is what distinguishes the paper from component-level innovations.

The paper explicitly positions itself within the trajectory of DeepSeek-V3, MiMo, GLM-4.5, and Kimi-K2 — the leading open-source MoE models — but argues that these prior works each optimized for a subset of the relevant constraints. DeepSeek-V3 focused on training efficiency and general capability but used uniform attention. MiMo-V2-Flash achieved good reasoning performance but trailed on agentic benchmarks. Kimi-K2 demonstrated strong agentic coding (SWE-Bench Verified 76.8%, Terminal-Bench 50.8%) but at 32B active parameters and 1T total — 3× and 5× Step 3.5 Flash's counts respectively. GLM-4.5 combined agentic, reasoning, and coding capabilities but also at substantially higher activation (32B).

A key positioning claim is that Step 3.5 Flash redefines the efficiency frontier — the Pareto boundary of intelligence versus inference cost — rather than simply pushing one end of the tradeoff. The paper's Figure 1 and Table 5 are designed to make this case visually: Step 3.5 Flash's bar is shown alongside models with much larger active parameter counts, achieving comparable heights. The intended message is that the model has shifted the curve upward for the 11B-active-parameter class, making it competitive with the 32–37B class.

Finally, the paper positions its RL contribution — MIS-PO — as addressing a specific gap in the MoE RL literature. The key reference points are PPO (high variance, unstable for MoE), GSPO (reduces variance but insufficiently stable for large-scale training), and the broader off-policy RL literature's trust-region methods. MIS-PO is presented as a simpler, more stable alternative that treats the inference policy as a Metropolis-Hastings proposal distribution and the training policy as the target, filtering rather than reweighting off-distribution samples. The paper argues this is particularly well-suited to MoE architectures where training-inference mismatch manifests at both the token level (probability ratios) and the expert level (routing consistency), requiring filtering at both granularities.

3. Technical Approach

3.1 Reader Orientation

Step 3.5 Flash is a complete language model system — not just an architecture, but an integrated stack spanning model design, distributed training infrastructure, stability diagnostics, pre-training curriculum, supervised fine-tuning, and reinforcement learning — all co-designed to produce a model that activates only 11B parameters per token yet performs competitively with models activating 32–37B+ parameters on reasoning and agentic benchmarks. The "shape" of the solution is a density-first design philosophy: rather than maximizing capability at any cost and then optimizing efficiency post-hoc, every architectural decision is evaluated against a three-way tradeoff between intelligence, inference latency, and deployment cost, with the goal of moving the entire Pareto frontier upward for the 11B-active-parameter class.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major interacting components, which are developed and optimized as an integrated whole rather than independently:

Sparse MoE Backbone with Hybrid Attention — The 45-layer Transformer that performs the actual language modeling. It contains 196B total parameters but activates only 11B per token through fine-grained mixture-of-experts routing (288 experts per MoE layer, top-8 activated) combined with a 3:1 ratio of sliding-window to full attention layers. This component determines the model's fundamental capability-to-cost ratio.
Training Infrastructure (Steptron Framework) — The distributed system that enables training the 196B-parameter model across 4,096 H800 GPUs. It includes the optimizer (Muon with Newton-Schulz orthogonalization), hybrid parallelization (8-way pipeline + 8-way expert + ZeRO-1 data parallelism), communication optimizations, and the lightweight asynchronous metrics server for stability monitoring. This component determines whether the architecture can be trained at all.
Pre-Training and Mid-Training Pipeline — The curriculum that takes the model from random initialization to a strong general-purpose foundation. It spans 17.6T tokens of pre-training (broad open-domain coverage → annealed high-quality specialization → 32K context initialization) followed by 750B tokens of mid-training (specialization at 32K → long-context extension to 128K). This component determines the quality of the initialization for downstream post-training.
Post-Training Pipeline (SFT + RL) — The refinement process that converts the base model into an agentic system. It includes two-stage SFT (large-scale multi-domain → reasoning-dense OOD injection), domain-specific RL expert construction, self-distillation into a unified student, and large-scale RL using MIS-PO for variance-controlled off-policy optimization. This component determines how much the model improves beyond its pre-trained capabilities.
Agent Infrastructure — The deployment scaffold that enables the model to interact with tools (code interpreters, web browsers, terminals, file systems) during evaluation and real-world use. It includes template design (XML-based tool calls, selective reasoning retention), the Session-Router for scalable container orchestration, and cross-framework adaptation to diverse agentic interaction protocols.

Information flows through these components sequentially: pre-training produces the base model → mid-training extends context and adds agentic priors → SFT teaches instruction following and domain-specific behaviors → RL (using MIS-PO) refines reasoning and tool-use through verifiable rewards → the agent infrastructure wraps the model for interactive tool-augmented deployment.

3.3 Roadmap for the Deep Dive

First, the sparse MoE backbone and hybrid attention architecture (Section 2.2), because the model's efficiency-capability tradeoff is fundamentally determined by which parameters are activated per token and how attention scales with context length. Understanding the 3:1 SWA-to-full layout, the head-wise gating mechanism, and the expert-parallel load balancing strategy is prerequisite to understanding everything else.
Second, the training infrastructure and stability diagnostics (Sections 3, 4.1), because large-scale MoE training fails in specific, diagnosable ways (Muon numerical sensitivity, expert collapse beyond routing collapse, localized activation blow-ups) that must be understood to appreciate why the architecture was viable. The observability stack is as much a contribution as the model design.
Third, the pre-training and mid-training curriculum (Section 4.2), covering the data mixture, schedule, and hyperparameters that produce the base model, since this establishes the foundation on which post-training builds.
Fourth, the post-training pipeline (Sections 5.1–5.3), including the two-stage SFT, expert model construction and self-distillation, the MIS-PO reinforcement learning algorithm (the paper's primary algorithmic contribution), and the data synthesis and curation process, since these are what convert a strong base model into a frontier-level agentic system.
Fifth, the agent infrastructure (Section 5.4) that enables tool-use during deployment, covering template design, context management for multi-turn interactions, and the scalable session management architecture.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems engineering paper whose core thesis is that frontier-level agentic intelligence can be achieved with 11B active parameters only if architecture, training stability, pre-training curriculum, post-training RL, and agent infrastructure are co-designed as a single integrated system rather than optimized independently. The paper's primary algorithmic contribution is MIS-PO (a variance-reduction method for off-policy RL on MoE models), but the bulk of the technical content addresses the engineering challenges that make the integrated system viable.

3.4.1 Sparse MoE Backbone with Hybrid Attention

The architecture is a 45-layer Transformer with a specific layout: the first 3 layers use dense (non-MoE) feed-forward networks, and the remaining 42 layers use sparse MoE feed-forward networks. The total parameter count is 196B, but per token, only the embedding layer (shared across all tokens), the attention parameters, the dense FFNs in the first 3 layers, and the top-8 selected experts out of 288 in each MoE layer are activated, yielding 11B active parameters per token. Table 6 summarizes all key dimensions.

MoE Expert Configuration. Each MoE layer contains 288 "routed" experts plus one "shared" expert. The shared expert is always activated for every token (it does not participate in routing), providing a stable baseline representation. The router selects the top-𝑘 = 8 routed experts per token based on learned affinity scores. Each routed expert is a standard SwiGLU feed-forward network with hidden size 1,280 (versus 11,264 for the dense FFN layers). This fine-grained expert design (many small experts rather than few large ones) is chosen to increase specialization capacity while keeping per-expert computation minimal.

The total parameters are computed as: 196B for the backbone, with an additional 0.81B per MTP head (3 heads total = 2.43B, bringing the full model to ~198B). Per token, the activated parameters are 11B for the backbone and 13B including MTP-3.

Hybrid Attention Layer Layout. The attention layers follow a repeating 4-layer motif: three sliding-window attention (SWA) layers followed by one full attention layer, denoted 𝑆3𝐹1. This motif is repeated 11 times (44 layers), with the first layer being an additional full attention layer (bringing the total to 45 layers as shown in Figure 2). The SWA window size 𝑊 = 512 means each token attends only to the previous 512 tokens in SWA layers, reducing the quadratic attention cost to linear.

The choice of 3:1 ratio over alternatives is motivated by the ablation in Table 1. A uniform full-attention layout (𝐹𝐹𝐹𝐹) achieves strong quality but incurs approximately 2.68× the decode FLOPs and 2.90× the prefill FLOPs of the 𝑆3𝐹1 baseline at 64K-256K context lengths (Table 8). An alternating 1:1 layout (𝑆1𝐹1) achieves the best downstream quality (LongCtx score 29.6 vs. 28.8 for 𝐹𝐹𝐹𝐹) but at ~1.58×/1.65× the decode/prefill FLOPs — about a 60% cost increase. The 𝑆3𝐹1 layout gives the best efficiency but "consistently underperforms a dense attention baseline across various benchmarks" (Table 10: BBH 61.7 vs. 66.0, MMLU-Pro 33.7 vs. 35.7). The paper therefore introduces two mechanisms to bridge this quality gap at minimal overhead.

Augmented Query Heads in SWA. The base configuration uses 64 query heads for both full attention and SWA layers. The paper increases the SWA query head count to 96 while keeping full attention at 64. Since the key-value head count remains 8 (GQA-8), this changes the query-to-KV ratio from 8:1 to 12:1 in SWA layers. Table 1 shows this change recovers most of the quality loss: BBH goes from 61.7 (𝑆3𝐹1 with 32 heads) to 65.3 (𝑆3𝐹1+Head with 48 heads), and MMLU-Pro from 33.7 to 37.4. The overhead is minimal because "in long-text scenarios, the overhead of naive SWA is very small" — the additional query heads increase FLOPs by only ~1–2% in decode and ~4–8% in prefill (Table 7), which the paper characterizes as "nearly a 'free lunch'."

The reason this works is that SWA layers process less information than full attention layers (each token sees only 512 tokens rather than the full sequence), so giving them more query heads provides additional representational capacity to extract useful patterns from the limited window, compensating for the loss of long-range information.

Head-wise Gated Attention. Rather than using fixed, data-independent sink tokens (which previous work introduced to absorb unused attention mass and stabilize SWA), the paper adopts a learnable, input-dependent gating mechanism. Each attention head computes its standard output $\mathbf{y}_i \in \mathbb{R}^d$ (a weighted sum of value vectors), and then a scalar gate $g_i \in [0,1]$ is computed from the input representation $\mathbf{x}_i$ and applied multiplicatively:

$g_i = \sigma(\mathbf{w}_{\text{gate}}^\top \mathbf{x}_i)$

where $\sigma(\cdot)$ is the sigmoid function, $\mathbf{w}_{\text{gate}} \in \mathbb{R}^{d_{\text{model}}}$ is a learnable vector (shared across positions but unique per head), and $\mathbf{x}_i \in \mathbb{R}^{d_{\text{model}}}$ is the input representation at position $i$ .

What it computes: a scalar between 0 and 1 that gates how much of the attention head's output to pass through. The gated output is $\mathbf{o}_i^{\text{gate}} = g_i \cdot \mathbf{y}_i$ , i.e., the standard attention output scaled by the input-dependent gate.

Why this form: the paper shows (Equation 6 in Appendix A.1) that substituting the sigmoid gate into the attention computation yields an equivalent formulation where $\exp(-g_i) Z_i$ acts as an input-dependent sink mass in the softmax normalizer, where $Z_i = \sum_{j'} \exp(s_{i, j'})$ is the partition function. Unlike a fixed sink token (which adds a constant mass to the denominator regardless of input), the learned gate allows each head to dynamically adjust how much attention mass is "absorbed" based on whether the current input contains useful information in the local window. When the window contains no useful context, the gate can close (near 0), effectively discarding the uninformative attention output rather than forcing it to influence the representation. When the window contains relevant information, the gate opens (near 1), allowing standard attention to proceed.

The choice of head-wise gating over sink tokens is validated in Table 2, where a 100B-A10B model trained with head-wise gating achieves an average score of 64.4 across six benchmarks (BBH, MMLU, GPQA, MBPP, C-EVAL, CMMLU) versus 62.5 for fixed sink tokens — a +1.9 point improvement. The paper notes that head-wise gating adds negligible computational overhead (Table 7 shows essentially zero additional FLOPs or latency), making it a strict improvement over sink tokens for this architecture.

MoE Expert-Parallel Load Balancing. Standard load-balancing losses (e.g., auxiliary loss or loss-free bias adjustment) operate at the global batch level, encouraging uniform token distribution across experts averaged over many tokens. However, under expert parallelism (EP), where different experts are hosted on different GPUs, end-to-end latency is determined by the slowest GPU at each synchronization point. If token assignment is skewed at the micro-batch level — even if balanced globally — the GPU hosting the overloaded expert becomes a straggler, throttling throughput.

The paper introduces an EP-group balance loss that operates at the granularity of EP ranks. The loss partitions the 288 experts into $G$ disjoint groups $\{E_g\}_{g=1}^G$ corresponding to the EP rank assignment (experts on the same GPU share a group). For each token $t$ , let $S_t$ be the set of top- $K$ selected experts (with binary mask $s_{t,e} = \mathbf{1}[e \in S_t]$ ) and $p_{t,e}$ be the router's probability for expert $e$ . The loss computes:

$p_e = \frac{1}{T} \sum_{t=1}^T p_{t,e}, \quad f_e = \frac{1}{TK} \sum_{t=1}^T s_{t,e}, \quad p_g = \sum_{e \in E_g} p_e, \quad f_g = \sum_{e \in E_g} f_e$

where $T$ is the number of tokens in the micro-batch, $p_e$ is the average routing probability for expert $e$ , $f_e$ is the average fraction of tokens dispatched to expert $e$ , $p_g$ is the total routing probability mass for EP group $g$ , and $f_g$ is the total fraction of tokens dispatched to group $g$ .

The EP loss is then:

$\mathcal{L}_{\text{EP}} = G \sum_{g=1}^G f_g \cdot p_g$

What it computes: a scalar that is minimized when $f_g$ and $p_g$ are uniform across groups (each group receives an equal share of tokens). The product $f_g \cdot p_g$ penalizes groups that receive a disproportionate fraction of tokens, multiplied by $G$ to scale the loss appropriately for the number of groups.

Why this form: standard auxiliary losses encourage per-expert balance but do not constrain per-group (per-GPU) balance. If experts 1–12 are all on GPU A and experts 13–24 on GPU B, global balance could mean GPU A processes 60% of tokens (spread across its 12 experts) while GPU B processes 40%. The EP loss explicitly penalizes this imbalance by operating at the group level, ensuring that each GPU processes roughly equal numbers of tokens at each micro-batch, preventing stragglers. The product form $f_g \cdot p_g$ (rather than e.g., an L2 penalty) is chosen because it is differentiable with respect to router probabilities and directly measures the joint distribution of dispatch and routing mass.

This loss is applied with coefficient 0.001 throughout pre-training and disabled during mid-training and post-training (where the router is frozen).

Multi-Token Prediction (MTP). To accelerate autoregressive decoding through speculative decoding, the model includes three multi-token prediction heads. Each MTP head consists of one SWA layer and one dense FFN, adding only 0.81B parameters per head (~0.41% of total). The MTP- $h$ head (for $h \in \{1, 2, 3\}$ ) predicts the token at position $t + 1 + h$ conditioned on the backbone's hidden state at position $t$ . During speculative decoding, a lightweight draft model (or the MTP heads themselves) proposes candidate future tokens, and the main model verifies them in parallel, reducing the number of sequential forward passes needed.

The training of MTP heads follows a staged approach to control overhead:

Main training: Only MTP-1 is trained (backbone + MTP-1 loss). The MTP loss weight is 0.3 during Pre-training Stage 1 and 0.1 during Pre-training Stage 2 and mid-training.
Lightweight final phase: Once the backbone is well-trained, MTP-2 and MTP-3 are initialized by cloning MTP-1's weights and jointly fine-tuned. This avoids the computational cost of training three MTP heads throughout the entire pre-training run.

The paper adopts a position-dependent loss reweighting from Fast-MTP: predictions for more distant tokens (MTP-3 predicts $x_{t+4}$ ) are weighted lower than nearer tokens (MTP-1 predicts $x_{t+2}$ ), preventing the model from over-optimizing for distant-token prediction at the expense of near-token accuracy.

The MTP heads use SWA rather than full attention to keep them lightweight. Since they only need local context to predict the next few tokens (long-range dependencies are already captured in the backbone's hidden state), SWA provides sufficient information with substantially lower computational cost.

GQA-8 for Deployment Alignment. The model uses Grouped-Query Attention with 8 key-value heads (GQA-8). This choice is explicitly motivated by deployment constraints: standard 8-GPU server nodes are ubiquitous, and GQA-8 aligns KV-cache sharding with 8-way tensor parallelism, improving memory access patterns. The paper notes that while GQA-8 makes attention more memory-bandwidth bound (since fewer KV heads mean each head processes more queries), this "also creates computational slack that can absorb speculative drafting and verification overhead, enabling aggressive multi-token speculation without a proportional latency penalty." In other words, the GPU compute units that would be idle waiting for memory under GQA-8 can instead be used to verify draft tokens, effectively hiding the speculative decoding overhead.

RoPE Configuration. The model uses Rotary Position Embeddings with base frequency $\theta = 10,000$ for both full attention and SWA during 4K training. For context extension, only the full attention layers' RoPE is scaled: $\theta_{\text{Full}} = 1,000,000$ at 32K (end of pre-training), increased to $\theta_{\text{Full}} = 5,000,000$ at 128K (mid-training), while SWA layers maintain $\theta_{\text{SWA}} = 10,000$ throughout. This selective scaling reflects the fact that SWA layers only ever see a 512-token window, so they never need to represent positions beyond the base frequency's effective range, while full attention layers must distinguish positions across the entire 128K context.

3.4.2 Training Infrastructure and Stability Diagnostics

Compute Cluster and Parallelization. The model is trained on 4,096 NVIDIA H800 GPUs, with each node containing 8 GPUs interconnected via NVLink and NVSwitch for intra-node communication and 8×200 Gbps RoCE links for inter-node connectivity.

The Steptron framework (built on PyTorch and Megatron-LM) implements a hybrid parallelization strategy:

8-way pipeline parallelism (PP) with virtual pipeline stages (VPP) to partition the 45-layer model across devices along the depth dimension, reducing per-device memory.
8-way expert parallelism (EP) to distribute the 288 experts across devices, so each device hosts only a subset of experts and processes only the tokens routed to its experts.
ZeRO-1 data parallelism (DP) to shard optimizer states across devices, reducing per-device memory for the non-expert parameters.

The paper adopts a decoupled parallelization scheme from Megatron-Core: the attention modules and MoE modules use independent parallel groups, with separate gradient reduction and scaling within each module's data-parallel group. This avoids the communication overhead of synchronizing gradients across the entire model for every parameter.

Communication Optimizations. Two optimizations jointly reduce iteration time by up to 5%:

Fabric-aware communication scheduling: DP traffic is partitioned into intra-node (NVLink) and inter-node (RoCE) phases, and these phases are pipelined to utilize both fabrics simultaneously rather than sequentially. This prevents RoCE link saturation from becoming a bottleneck while NVLink is underutilized.
Communication-aware rank placement: Using job-level communication profiles (historical data on which ranks communicate most), ranks are placed across switches to minimize hop counts and steer heavy traffic away from inter-switch hotspots, reducing congestion.

Muon ZeRO-1 Resharding. The Muon optimizer requires full (unsharded) per-parameter gradients for its Newton-Schulz orthogonalization step, which conflicts with ZeRO-1's reduce-scatter operation that shards gradients across DP ranks. The naive solution (all-reducing FP32 gradients to reconstruct full gradients before Muon) nearly doubles communication. The paper instead assigns whole parameters to DP ranks and repacks the gradient buffer into a rank-major layout, so a single reduce-scatter delivers each parameter's complete gradient to its owner (the rank responsible for that parameter's optimizer state). Since expert parameters are smaller and more numerous, this hybrid strategy applies repacking only to expert parameters (using DP all-reduce for non-expert parameters), reducing end-to-end iteration time by approximately 5% with less than 4 GB additional memory.

GPU Kernel Optimizations. At the kernel level, the paper fuses QK normalization with RoPE in attention (reducing kernel launch overhead), fuses multiple small operators in MoE layers, and implements a fused MoE gather/scatter with grouped GEMM similar to SonicMoE, reducing memory traffic and kernel launch overhead.

Fine-grained Selective Checkpointing. To reduce peak activation memory, the training framework supports per-layer, submodule-level toggles for activation recomputation (attention, FFN, normalization, SiLU, MoE permutation). Only the most memory-intensive components are recomputed during the backward pass, minimizing the recomputation overhead while keeping peak memory within GPU limits.

Lightweight Asynchronous Metrics Server. This is a critical piece of infrastructure for training stability. A 4,096-GPU workload generates nearly 6 million metric messages per iteration (one per micro-batch per GPU, covering per-expert activation norms, gradient norms, routing distributions, etc.). Performing synchronous global reductions on this telemetry within the training loop would add several seconds per iteration — effectively doubling the iteration time.

The solution is a decoupled architecture: each rank uses StepRPC (an in-house asynchronous communication framework) to offload local metrics to a remote Metrics Server, adding only ~100ms overhead per iteration. The server buffers incoming metrics until all ranks signal end-of-iteration, then triggers reduction and database persistence asynchronously. The server itself is a multi-process system with two decoupled modules:

Message Receiver: optimized for high-throughput ingestion of millions of messages with minimal latency.
Reduction Processor: responsible for aggregation and database persistence, using multi-core parallelism to keep pace with the telemetry stream.

This infrastructure is what enables the fine-grained stability diagnostics described in Section 4.1 — without it, the per-expert, per-micro-batch signals that reveal localized activation blow-ups and expert collapse would be either invisible (aggregated away in global statistics) or prohibitively expensive to collect.

3.4.3 Training Stability Mechanisms

The paper identifies three dominant failure modes that the observability stack helps diagnose and mitigate.

Muon Numerical Sensitivity. The Muon optimizer orthogonalizes gradients using a Newton-Schulz (NS) iteration to produce a semi-orthogonal update direction. The paper adopts the Polar Express variant, which converges faster, and runs a fixed $T = 6$ steps to balance optimization quality and throughput.

However, the paper observes "sharp, unrecoverable loss spikes" despite using the recommended safety scaling. The spikes are non-deterministic (often avoided by resuming from a nearby checkpoint), suggesting a numerical pathology. Simulations reveal that bfloat16 Polar Express can "rarely yield extreme intermediate outliers under certain update statistics due to cumulative error in addition." The solution is to cast only the Polar Express iteration (state and intermediates) to float16 while keeping the rest of training in mixed-precision. After this change, spikes do not recur.

The choice of Muon over Adam is not extensively justified in the main text, but the paper notes "modest, consistent loss reduction when using a faster-converging orthogonalization approximation" in early experiments, and the optimizer is used throughout pre-training, mid-training, and post-training.

Expert Collapse Beyond Routing Collapse. Standard MoE diagnostics monitor routing statistics: if an expert receives negligible dispatch (near-zero fraction of tokens), it is considered "dead." The paper's prior work (Step-3) had identified a subtler failure mode: experts can die even when router dispatch appears healthy. The signals are:

Vanishing expert activations: the norm of the expert's FFN intermediate activations drifts toward near-zero, even though tokens continue to be routed to it.
Stagnant or decaying expert parameter norms: the Frobenius norm of the expert's weight matrices stops growing or decreases, indicating the expert is receiving negligible effective gradient signal despite being "active" in the dispatch sense.

Two factors contribute to this pathology:

Routed-expert aggregation requires explicit scaling. When a shared expert is used alongside routed experts, the relative contribution must be calibrated. The paper states: "While smaller models may implicitly learn such a balance, larger models are less reliable at self-calibration. A mismatch can suppress the effective contribution of routed experts even if routing frequencies appear healthy."
Micro-batch balancing can be overly restrictive under fine-grained sparsity. For models with many small experts (288 per layer), micro-batch-level load-balancing constraints (as in Switch-style routing) force every expert to receive tokens in every micro-batch, which can prevent specialization. The paper recommends broader-scope balancing (global-batch statistics) or loss-free bias adjustment.

The key diagnostic recommendation is to monitor per-expert activation norms (RMS/mean norm at the MoE FFN intermediate) and parameter norms (Frobenius norms of expert projection matrices). When a subset of experts drifts toward near-zero activations/updates while the median remains stable (indicated by decreasing min-to-median ratios), it provides early warning of expert death long before routing statistics would detect it.

Localized Activation Blow-up in MoE Layers. As expert specialization matures during training, a specific pathology emerges in the deeper MoE layers (particularly in the final layers, e.g., Layer 45). The activation norm of a small subset of experts (often just one or two per layer) grows exponentially, while the majority of experts in the same layer remain well-behaved. This is visualized in Figure 4, Panel (c), which shows the maximum expert output norm in Layer 45 exploding to >10,000 while the median remains stable at ~5–10.

Critically, this instability is entirely masked by the training loss (Figure 4, Panel (a)), which shows negligible variation across methods (no clipping, weight clipping, activation clipping). This means that loss-based monitoring — the standard approach in most training pipelines — would completely miss a catastrophic internal instability.

The paper traces this pathology to a combination of factors:

High-frequency bi-grams trigger expert specialization: tokens that frequently co-occur (e.g., common punctuation patterns, special token sequences) cause a specific expert to specialize in predicting the second token given the first. Since the bi-gram is very common, this expert receives many tokens and its specialization is reinforced.
Pre-norm architecture enables unbounded amplification: in a pre-norm Transformer, the final output is $\text{RMSNorm}(\sum_l \text{attn}_l + \sum_{l,e} \text{expert}_{l,e})$ . If one expert's output dominates, the RMSNorm normalizes the direction but does not constrain the magnitude of that expert's contribution relative to others. A single expert can amplify its output boundlessly, making the final prediction near-deterministic (always predicting the second token of the bi-gram).
SwiGLU produces sparse, large activations under alignment: the SwiGLU activation is $\mathbf{W}_{\text{down}} (\text{SiLU}(\mathbf{W}_{\text{gate}} \mathbf{x}) \odot \mathbf{W}_{\text{up}} \mathbf{x})$ . When $\text{SiLU}(\mathbf{W}_{\text{gate}} \mathbf{x})$ and $\mathbf{W}_{\text{up}} \mathbf{x}$ are strongly aligned and concentrate on a few dimensions, their element-wise product produces activations with extreme magnitudes. The paper finds that for outlier experts, $\|\text{SiLU}(\mathbf{W}_{\text{gate}} \mathbf{x})\| \cdot \|\mathbf{W}_{\text{up}} \mathbf{x}\| \approx \|\text{SiLU}(\mathbf{W}_{\text{gate}} \mathbf{x}) \odot \mathbf{W}_{\text{up}} \mathbf{x}\|$ , indicating near-perfect alignment rather than the near-orthogonal relationship expected for well-behaved features.
Muon amplifies persistent low-rank updates: the gradient with respect to the outlier expert's parameters is "not only abnormally low rank, but also consistently points in a direction that emphasizes the magnitude without rotation." Muon's orthogonalization eliminates gradient magnitude information, so the optimizer consistently applies full-magnitude orthogonal updates in the same low-rank direction, rapidly increasing the singular values of the weight matrices.

The paper evaluates two mitigation strategies (Figure 4):

Weight clipping: after each checkpoint, if an expert projection matrix $W$ 's maximum activation norm $\max_{\mathbf{x}} \|W\mathbf{x}\|$ exceeds a threshold $\tau$ , rescale via $W \leftarrow W \cdot \tau / \max_{\mathbf{x}} \|W\mathbf{x}\|$ . This is similar to MuonClip in attention. However, Figure 4(c) shows that weight clipping "merely delays this explosion" — the max norm eventually rises again.
Activation clipping: apply element-wise clipping directly to the MoE FFN intermediate activations before the output projection. Figure 4(c) shows that activation clipping "effectively bounds maximum norms, ensuring stability across all layers."

The paper argues that activation clipping is preferable because: (a) it directly addresses the root cause (the sparse, large-magnitude intermediate activations produced by SwiGLU alignment), (b) it has "negligible negative effects, as well-behaved activations rarely exceed the threshold," and (c) the max-to-median ratio of per-expert activation norms provides a robust, necessary metric for monitoring — even when loss is indistinguishable, this ratio "reliably unmasks underlying instability."

3.4.4 Pre-Training and Mid-Training Curriculum

Data Mixture. The pre-training corpus combines general open-domain data with agentic-oriented data, structured across several sources:

General Knowledge Data is sourced through StepCrawl, an in-house crawling and curation system processing ~1B pages daily. StepCrawl uses a WebOrganizer-style model fine-tuned for the pipeline to filter SEO-driven and low-utility pages, and to guide crawl-budget allocation across site categories to preserve diversity. All crawling respects robots.txt. Collected content passes through multi-stage quality filtering, deduplication, and sanitization.

The quality refinement pipeline applies:

Quality stratification: documents are labeled by an ensemble of six lightweight scorers/classifiers into quality tiers (High/Medium-High/Medium/Medium-Low/Low). The final recipe keeps High/Medium-High/Medium and discards Medium-Low/Low. For book and paper corpora, only High/Medium-High tiers are retained during annealing.
Embedding-based cluster rebalancing: large-scale Chinese/English web data is embedded, clustered via k-means (100K+ clusters), and clusters with disproportionate mass are down-sampled. This improves a broad set of benchmarks during the cooldown stage.
Knowledge-intensive mining: a curated inventory of high-value entities, concepts, and relations is used to retrieve knowledge-dense documents from the full corpus in embedding space, ranked by a knowledge-density model. Some retrieved content undergoes controlled rephrasing and QA synthesis before being mixed back into training.

Code Data is curated using a modified OpenCoder pipeline. The key modification is a calibrated relaxation: standard OpenCoder filtering generates "hits" for each document (each hit is a heuristic violation indicating potential noise). The paper finds that accepting documents with up to 6 hits (hit0–6) yields the best overall benchmark performance, "retaining a wider variety of high-signal code compared to the original strict constraints." Code-centric data is upsampled during annealing and mid-training.

PR/Issue/Commit Data is constructed from GitHub repositories with 10+ stars, yielding a 5-million-sample foundation. Four training subsets are derived:

Base PR/Issue/Commit Data: Full commit histories extracted via GHArchive and GitHub API. Changes are extracted and a portion validated against git diff ground truth. Filtered to 20+ mainstream languages and strictly deduplicated against SWE-Bench Verified and SWE-Bench Multilingual.
Concatenated PR-Dialogue Data (90B tokens): Two Agentless-inspired templates are applied: (a) File localization — given problem description and repository structure, identify target file paths; (b) Code repair — given problem description and file content, generate precise modifications via SEARCH/REPLACE blocks. During annealing, only template scaffolding is masked; during mid-training, the data is converted to chat dialogs with user prompts masked.
Rewritten Reasoning-Oriented Data (12B tokens): From the Python subset, bug-fix samples are derived via LLM change-type annotation. Two rewriting strategies produce: (a) reasoning reconstruction — an LLM reconstructs the PR author's problem-solving process (analysis, root cause, solution design, implementation), injected into PR-Dialogue format; (b) Active Reading notebooks — PR/issue/commit data is converted into structured learning outlines, then synthesized into coherent technical notes. Hallucinated/inconsistent traces are filtered.
Environment-based Seed Data: Executable environments are curated from raw PR/issue/commit records using the environment building pipeline. Candidate samples are filtered for test-patch inclusion and validated for environmental reproducibility, yielding hundreds of thousands of seed samples with problem descriptions, code changes, and test functions, serving as the foundation for agentic coding capabilities.

Tool-Use and Reasoning Data includes synthetic and semi-synthetic samples spanning math, code, science, and general knowledge, plus domain-specific samples targeting search agent, SWE agent, and tool execution. During mid-training, long-context samples (natural long documents and long-form synthetic tasks) are introduced.

Pre-Training Schedule. Pre-training proceeds in two stages totaling ~17.6T tokens:

Pre-training Stage 1: Open-domain pre-training on 14.6T tokens at 4K context, using broad coverage to maximize foundational capability.
Pre-training Stage 2: Annealing + long-context initialization on 3T tokens. The first 2T tokens remain at 4K context but with the data mixture annealed toward code and PR/Issue/Commit-centric sources while increasing the share of higher-quality knowledge and reasoning-dense samples. The final 1T tokens extend the context window to 32K under the same annealed mixture.

Mid-Training Schedule. Mid-training adds ~750B tokens across two stages:

Mid-training Stage 1: Specialization at 32K on 386B tokens. 81B tokens (21%) are replayed from pre-training to mitigate distribution shift and stabilize specialization, while the remainder emphasizes software-engineering and tool-use-centric mixtures.
Mid-training Stage 2: Long-context specialization on 364B tokens at 128K context. 10.5B replay tokens are retained. The mixture includes synthetic long-horizon reasoning, natural long documents (selected from pre-training data with length > 32K), and domain-specific data for code agent, search agent, and tool-use.

Pre-Training Hyperparameters. The Muon optimizer is used with weight decay 0.1 and gradient clip 1.0. The learning rate schedule:

Linear warmup from 0 to $2.5 \times 10^{-4}$ over 2,000 steps.
Cosine decay to $5 \times 10^{-5}$ over Pre-training Stage 1.
Secondary cosine decay from $5 \times 10^{-5}$ to $2 \times 10^{-5}$ over the 4K portion of Pre-training Stage 2 (2T tokens).
Fixed at $2 \times 10^{-5}$ for the 32K portion (1T tokens).

The global batch size increases from 4,096 to 16,384 over the first 400B tokens, then remains at 16,384 for the remainder of pre-training, dropping to 2,048 for the 32K portion of annealing. The MTP loss weight is 0.3 in Pre-training Stage 1 and 0.1 in Pre-training Stage 2. For loss-free load balancing, the bias update rate is 0.001 for the first 14.6T tokens and decays to 0.0 during annealing. The EP-group balance loss has coefficient 0.001 throughout pre-training.

Mid-Training Hyperparameters. Muon continues. MoE router weights are frozen, the EP-group balance loss is disabled, and the MTP loss weight is fixed at 0.1. The learning rate warms from 0 to $2 \times 10^{-5}$ over the first 3% of iterations, is kept constant in Mid-training Stage 1, and decays to $7.3 \times 10^{-6}$ in Mid-training Stage 2.

Meta Token Conditioning. Each training example is associated with a metadata string containing content type (e.g., Code, Book, Paper, Web), language (e.g., EN, ZH), domain, and source. The metadata is prepended to the token sequence as $\mathbf{s} = [\mathbf{M}; \mathbf{x}]$ . For the first ~3.8T tokens, the model is trained to predict all tokens including metadata ( $\mathcal{L}_{\text{full}}(\theta) = -\sum_{t=1}^{|\mathbf{s}|} \log P_\theta(s_t | \mathbf{s}_{<t})$ ). After this phase, metadata positions are kept in context but masked from the loss, so only payload tokens contribute to the training objective ( $\mathcal{L}_{\text{mask}}(\theta) = -\sum_{t=|\mathbf{M}|+1}^{|\mathbf{s}|} \log P_\theta(s_t | \mathbf{s}_{<t})$ ). The hypothesis is that by ~3.8T tokens, the model has learned to effectively use metadata as a conditioning signal, so further optimization of metadata prediction is unnecessary, and allocating all optimization pressure to payload tokens improves token efficiency.

3.4.5 Post-Training Pipeline: SFT and Self-Distillation

Two-Stage SFT. The post-training process begins with supervised fine-tuning on a curated dataset of 871K samples (7.23B tokens), with the distribution shown in Table 3: Math (68K samples, 11.19% of tokens), Code (86K, 21.10%), STEM (120K, 6.31%), Logic (93K, 13.87%), General (314K, 9.16%), Code Agent (37K, 17.70%), Tool-use (115K, 8.72%), Search Agent (20K, 8.75%), Long Context (16K, 4.00%).

The first stage executes large-scale multi-domain SFT with difficulty-aware filtering and strategic balancing to foster broad agentic behaviors. The second stage explicitly maximizes reasoning density by injecting out-of-distribution (OOD) signals — approximately 30K expert-level chemistry trajectories and synthetic arithmetic tasks — over just three epochs. The paper argues this targeted exposure to distinct reasoning patterns "unlocks latent capabilities" and "equips the model with the sophisticated structural complexity necessary to initialize the subsequent domain-specific RL phase."

The SFT data undergoes a unified three-stage filtration pipeline:

Rule-based filtering: removes degenerate patterns (infinite repetition, harmful content, PII).
Model-based filtering: specialized models detect and remove linguistically inconsistent data with unnatural language mixing.
Decontamination: both exact matching (with digit masking) and $N$ -gram matching against evaluation benchmarks.

Expert Model Construction. After SFT, the unified model undergoes domain-specific reinforcement learning across Math, Code, STEM, Tool-use, Long Context Understanding, Human Preference, and Agentic Reasoning, producing specialized "expert" models for each domain. These experts are trained via RL with domain-specific reward signals (verifiable rewards for math/code/STEM, model-based rewards for tool-use and human preference).

Self-Distillation. Rather than deploying multiple expert models (which would be operationally complex), the paper consolidates the divergent expert capabilities into a single "student" model initialized from the mid-training checkpoint. The expert models generate high-quality trajectories using the same prompt distribution as the first-stage SFT corpus. Rejection sampling eliminates undesirable patterns such as language mixing or overthinking. The student model is then fine-tuned on these expert-generated trajectories, "centralizing expert knowledge into a single student model" and "significantly reducing the optimization burden on subsequent RL stages."

The self-distillation uses Muon with 3% warmup and cosine decay from $1.0 \times 10^{-5}$ to $5.0 \times 10^{-6}$ , with MTP loss weight 0.1, global batch size 32, global sequence length 128K, and $\theta_{\text{Full}} = 5,000,000$ .

3.4.6 MIS-Filtered Policy Optimization (MIS-PO)

This is the paper's primary algorithmic contribution and the core of the scalable RL framework. It addresses the stability challenges of off-policy reinforcement learning for MoE models on long-horizon reasoning tasks.

Problem Setting. In RL for LLMs, the goal is to optimize a policy $\pi_\theta$ (the model's token-by-token generation probabilities) to maximize terminal rewards over trajectories $\tau = (s_0, a_0, \ldots, s_T)$ , where $a_t$ is the token generated at state $s_t$ (the prefix of previously generated tokens). For reasoning tasks, these trajectories can be thousands of tokens long, and the reward $R$ is typically 1 if the final answer is correct and 0 otherwise (for verifiable tasks) or a scalar score from a reward model (for non-verifiable tasks).

The fundamental difficulty is that the inference engine that generates rollouts runs a slightly different policy $\pi_{\theta_{\text{vllm}}}$ than the training engine's current policy $\pi_{\theta_{\text{old}}}$ , due to infrastructure differences (different hardware, different numerical precision, different batching). Standard off-policy RL uses importance sampling to correct for this discrepancy: each action's contribution to the loss is weighted by the ratio $\pi_\theta(a_t|s_t) / \pi_{\theta_{\text{vllm}}}(a_t|s_t)$ . Over long trajectories, the product of these ratios can vary by orders of magnitude, producing extremely high-variance gradient estimates.

For MoE models, an additional source of mismatch exists: the router's expert assignments shift when the policy updates, meaning the same token might be processed by different experts under the inference and training policies, creating a type of distributional shift that dense models don't experience.

MIS-PO Core Idea. Instead of scaling gradients by continuous importance weights, MIS-PO discretely filters samples based on whether they fall within a trust region around the target distribution. It treats the inference policy as a Metropolis Independence Sampling (MIS) proposal distribution and the training policy as the target. Samples that are sufficiently close to the target are treated as effectively on-policy and used for updates; samples that have drifted too far are discarded.

The filtering operates at two distinct granularities:

Token-level filtering: for each token $t$ , compute the probability ratio $x_t = \pi_{\theta_{\text{old}}}(a_t|s_t) / \pi_{\theta_{\text{vllm}}}(a_t|s_t)$ . Apply a binary indicator $\mathbb{I}(x_t) = \mathbf{1}[\rho_{\min} \leq x_t \leq \rho_{\max}]$ that accepts tokens whose ratio falls within a pre-specified range.

Trajectory-level filtering: compute the geometric mean ratio across the trajectory $\bar{\rho}(\tau) = (\prod_t x_t)^{1/T}$ , where $T$ is the trajectory length. Apply the same binary indicator $\mathbb{I}(\bar{\rho}(\tau)) = \mathbf{1}[\rho_{\min} \leq \bar{\rho}(\tau) \leq \rho_{\max}]$ that accepts or rejects the entire trajectory.

The reformulated actor loss replaces continuous importance weights with these dual-level discrete masks:

$\mathcal{L}_{\text{actor}} = -\mathbb{E}_{\tau \sim \pi_{\theta_{\text{vllm}}}} \left[ \mathbb{I}(x_t) \cdot \mathbb{I}(\bar{\rho}(\tau)) \cdot \log \pi_\theta(a_t|s_t) \cdot \hat{A}_t \right]$

where $\mathbb{I}(x_t)$ is the token-level binary filter, $\mathbb{I}(\bar{\rho}(\tau))$ is the trajectory-level binary filter, $\log \pi_\theta(a_t|s_t)$ is the log-probability of the chosen token under the current policy, and $\hat{A}_t$ is the advantage estimate (typically $\hat{A}_t = \hat{R} - V_\phi(s_t)$ for terminal reward $\hat{R}$ with optionally bootstrapped values).

What it computes: for each token in each trajectory, if the token's probability ratio and the trajectory's geometric mean ratio both fall within the trust region, the token contributes to the loss with weight 1; otherwise, it contributes with weight 0 (is masked out). The loss maximizes the log-probability of selected tokens weighted by their advantage — standard policy gradient, but only on filtered, on-distribution data.

Why this form: importance sampling $\pi_\theta / \pi_{\theta_{\text{vllm}}}$ produces weights that can be arbitrarily large or small for individual tokens, creating high-variance gradient estimates. The binary indicator replaces this continuous reweighting with hard filtering: tokens that are too far from the target distribution are simply excluded. This is a stricter but lower-variance alternative — it reduces the effective sample size but ensures that the samples used for updates are reliable. The two-level filtering (token and trajectory) ensures that both local mismatches (a single token with an unusual probability ratio) and global mismatches (an entire trajectory that has drifted) are detected and filtered. The trajectory-level geometric mean is used rather than the arithmetic mean because probability ratios multiply across tokens (by the chain rule of probability), so the geometric mean correctly captures the average per-token ratio.

Comparison to Importance Sampling and GSPO. The paper explicitly positions MIS-PO against two alternatives:

PPO with importance sampling suffers from high variance. Figure 5 (middle panel) shows PPO's actor gradient norm exhibits large spikes, while MIS-PO's is significantly lower and smoother. The left panel shows MIS-PO achieves higher rewards with faster convergence, and the right panel shows MIS-PO maintains higher entropy (better exploration).
GSPO (Group Sequence Policy Optimization) replaces token-level importance ratios with the geometric mean of trajectory-level ratios, effectively using a single importance weight per trajectory. The paper extends GSPO to the actor-critic setting (Equation 16 in Appendix D.2.3) by replacing standard token-level importance sampling with $r_\tau(\theta) = (\prod_t \pi_\theta(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t))^{1/T}$ and applying the same token- and trajectory-level masking as MIS-PO. However, Figure 7 shows that GSPO underperforms MIS-PO on both dense and MoE models: on the MoE model (Figure 7(b)), GSPO "plateaus around iteration 200" while MIS-PO continues improving, and GSPO "exhibits an increasing training-inference discrepancy during training, quantified by the density ratio $\pi_{\theta_{\text{old}}} / \pi_{\theta_{\text{vllm}}}$ ." MIS-PO consistently maintains this discrepancy within a stable range.

Truncation-Aware Value Bootstrapping. Long-horizon reasoning trajectories frequently exceed the maximum generation length and are truncated. Naively assigning zero reward to truncated trajectories conflates two different outcomes: "the model ran out of tokens" versus "the model produced an incorrect answer." This ambiguity penalizes valid reasoning that simply needed more steps.

The paper replaces the zero reward for truncated trajectories with a bootstrapped value estimate:

$\hat{R}_i = \begin{cases} V_\phi(s_T) & \text{if the response is truncated} \\ R_i & \text{otherwise} \end{cases}$

where $V_\phi(s_T)$ is the critic's estimated value of the final (truncated) state $s_T$ , and $R_i$ is the terminal reward for non-truncated trajectories.

What it computes: instead of treating truncation as a failure (reward 0), it treats it as a horizon interruption — the model was in state $s_T$ and would have eventually received some expected cumulative reward, estimated by the value function. This bootstrapped reward replaces the missing terminal reward.

Why this form: the alternative (zero reward) creates a perverse incentive for the model to produce shorter responses to avoid truncation, penalizing thorough reasoning. Value bootstrapping removes this penalty while maintaining the correct learning signal: if the model was on a promising path when truncated, the bootstrapped reward will be positive, encouraging the model to either complete the reasoning faster or the system to increase the generation limit.

The paper reports this technique "stabilizes training even at truncation rates as high as 20%."

Routing Confidence as a Stability Proxy. The paper introduces $\Sigma_k$ , the Routing Confidence, defined as the average probability mass of the activated experts (the sum of the router's top- $k$ probabilities). Low $\Sigma_k$ implies high routing uncertainty — the router assigns similar probabilities to many experts, so small changes in the policy can cause different experts to be selected, amplifying training-inference mismatch.

Through preliminary experiments, the paper identifies a stability phase transition: models with low routing confidence are brittle and require extreme stabilization (e.g., Router Replay, strict on-policy updates). Models with high routing confidence maintain robustness, enabling off-policy training without complex interventions. The paper does not quantify the threshold, but uses this as a diagnostic: when training starts, if $\Sigma_k$ is low, additional stabilization is needed; if it is high, MIS-PO alone suffices.

RL Training Dynamics. Figure 6 shows the RL training trajectory over ~250 steps. The training reward (left panel) rises steadily from ~0.60 to ~0.70, indicating stable learning. Concurrently, downstream evaluation improves: IMO-AnswerBench +3.2% (82.3 → 85.5), CF-Div2-Stepfun-cpp +6.1% (80.3 → 86.4), ARC-AGI-1 +10.6% (46.2 → 56.8), HLEtext +3.4% (19.9 → 23.3). The paper does not provide separate held-out validation curves, so these gains may partially reflect training set overlap.

Extended Training Dynamics on MoE. Figure 8 shows an extended training run of MIS-PO on the MoE model using a challenging dataset. Over ~800 steps, the reward continues to increase (0.35 → 0.60), the actor gradient norm remains stable with no spikes (middle panel shows raw gradient norm without smoothing or downsampling, emphasizing stability), and entropy decays slowly (0.48 → 0.24), maintaining exploration. This empirically validates that MIS-PO scales to extended RL training on MoE architectures without collapse.

Hyperparameters. The rollout configuration varies by task:

Reasoning: 256 unique prompts, 16 responses each (4,096 total rollouts per generation).
Human preference: 512 unique prompts, 8 responses each.
Tool-use: 128 unique prompts, 8 responses each.

Both sampling temperature and top- $p$ are set to 1.0 with maximum sequence length 128K. After rollout, samples are partitioned into mini-batches and trained for a single epoch: 4 mini-batches for the actor, 12 mini-batches for the critic. Muon optimizer with weight decay 0.1. Actor learning rate $2 \times 10^{-6}$ with 20 warmup steps. Critic learning rate $5 \times 10^{-6}$ with 50 warmup steps. Both $\gamma$ (discount) and $\lambda$ (GAE lambda) are set to 1. An unbiased KL loss with coefficient 0.001 is applied in the final stage.

The MIS-PO masking bounds are:

Token-level: $[\rho_{\min}, \rho_{\max}] = [0.5, 2]$ — tokens with probability ratios outside this 4× range are discarded.
Trajectory-level: $[\rho_{\min}, \rho_{\max}] = [0.996, 1.001]$ — trajectories with geometric mean ratios outside this extremely narrow range (0.5% deviation) are discarded.

The trajectory-level bounds are notably tight, indicating that the method discards a substantial fraction of trajectories. The token-level bounds are much looser, suggesting that local probability mismatches are more tolerable than global distribution shift. The paper reports the "all accept ratio" (fraction of trajectories passing both filters) in Figure 7, which typically ranges from 0.08 to 0.20, meaning 80–92% of trajectories are discarded. This is an intentionally conservative design: by using only the most reliable 8–20% of data, MIS-PO achieves much lower gradient variance at the cost of sample efficiency, which is acceptable because the inference engine can generate more rollouts in parallel.

Reward System. The paper decouples RL into RL with verifiable rewards (RLVR) and RL with non-verifiable rewards (e.g., RLHF), each using distinct reward signals:

Verifiable rewards (RLVR): Rule-based checkers for logic, instruction following, and code (sandbox execution against test cases with soft reward proportional to fraction of tests passed). For STEM tasks, model-based verifiers (gpt-oss-120b) are used instead of vanilla math-verify, improving performance by an average of 2.0% over 450 training steps in ablations. The verifier uses a structured Chinese prompt that checks: (1) overall completeness, (2) final-answer match to reference, (3) reasoning process correctness, (4) format requirements, (5) sub-question handling, with the final output being <correct> True </correct> or <correct> False </correct>.
Non-verifiable rewards: A pairwise Generative Reward Model (GenRM) — a reasoning model that outputs a confidence score indicating the likelihood of a response winning against a fixed reference. The confidence score is converted to a Bradley-Terry win rate as the reward signal. Length control is modeled within GenRM as a confidence score penalty, propagated to the win-rate reward to suppress excessive length growth. Zero reward is assigned to responses with fabricated citations, overconfident claims, or language inconsistencies.
Agent rewards: Search tasks use an LLM-based entity-matching score. Report generation uses a rubric-based LLM judge producing ternary judgments (satisfied, partially satisfied, unsatisfied), mapped to asymmetric binary rewards (satisfied → positive, others → negative) because the intermediate category "often misaligns with expert preferences."

The GenRM is trained by fine-tuning the SFT model with RM-specific prompts on curated pairwise preference data using a logsigmoid loss. To improve robustness, MetaRM — an additional verifier that detects spurious reasoning (correct preference derived from flawed logic) — reduces the training reward when such patterns are detected. In ablation, MetaRM-augmented GenRM outperforms vanilla GenRM by 0.5–3% on every benchmark over 200 RL training steps.

3.4.7 Data Synthesis and Curation for Post-Training

The prompt pool for RL is constructed by aggregating open-source data, synthetic generations, and user trajectories. A unified synthesis and curation pipeline applies strict global filtering with domain-specific refinement.

General and Reasoning. Training data aggregates community prompts, expert responses, and synthetic data from diverse open-source sources spanning mathematics (OpenR1-Math-220K, NuminaMath, Big-Math, Orca-Math, Olympiads, Cn-k12, DeepMath-103K, OpenThoughts, s1, AM-Thinking, LIMO), coding (TACO, DeepCoder, CodeContests+), and science/open-ended QA (CAMEL, Llama-Nemotron, MegaScience, WildChat).

Domain-specific refinement includes: in mathematics, specialist-guided rejection sampling and synthetic large-number arithmetic to ensure numerical stability; for programming, prioritization of offline executable algorithmic challenges while strictly purging RAG-related hallucinations (the model falsely claiming access to external search engines); for science, restriction to unambiguous questions with unique, determinable solutions.

Open-source checkers are expanded and samples augmented with real-world constraints. General prompts from open-source, synthetic, and user trajectories form a diverse, difficulty-balanced pool, yielding millions of samples at the billion-token scale.

Generalized Tool Learning. An execution-driven data generation framework produces tool-use trajectories. Instead of random exploration or model-based simulation, the framework decomposes tool-use behavior into atomic intents modeled by a finite state machine (FSM), separating abstract tool-call logic from parameterized execution constraints. Data is generated through a sample–execute–verify loop with rejection sampling: all candidate trajectories are executed in real environments and validated by deterministic feedback, ensuring fidelity and eliminating hallucinated behaviors. By compositionally combining atomic intents, the framework supports scalable generation of complex, controllable tool-use scenarios. Over 100K high-quality trajectories (billions of tokens) are produced, providing supervision for tool-based planning, reasoning, and execution.

Code Agents. A specialized agentic pipeline, evolved from the SWE-factory framework, synthesizes environments and solutions under verifiable reward signals. The pipeline treats environment construction as a first-class capability alongside bug fixing and feature implementation. Key features include:

A cross-task memory pool that retrieves historical build successes as few-shot demonstrations.
A loop-detection mechanism to prevent redundant exploration.
Approximately 40% environment-building success rate, forming a positive feedback loop for model self-evolution through dense supervision from construction trajectories (shell commands, error recovery).
Trajectory normalization: transient failures and redundant execution patterns that don't contribute to final resolution are abstracted and masked.

The pipeline curates 50K verified environments spanning 15K+ GitHub repositories and 20+ programming languages. The paper reports bidirectional transfer: construction expertise accelerates coding performance, and coding within constructed environments improves construction accuracy.

Search and Research Agents. A graph-based and multi-document synthesis pipeline enforces multi-hop reasoning by performing topological expansions on knowledge graphs (e.g., Wikidata5m) and simulating cross-website browsing trajectories. Generated queries are validated against DeepSeek-R1: any query solvable by this strong reasoning model without tool interaction is excluded, guaranteeing that the data requires external retrieval. Trajectories are refined through a structured report generation pipeline enforcing rigorous instruction compliance and structural integrity, with strict adherence to preset research plans (trajectories that deviate are discarded). Valid outputs undergo iterative cleaning via model-based judgers and heuristic rules to resolve fine-grained issues such as informal writing, temporal hallucinations, and mixed-language artifacts.

3.4.8 Agent Infrastructure

Reasoning with Tool-Use Template Design. The paper evaluates three strategies for managing reasoning history in multi-turn tool interactions:

Discard reasoning history at every turn (as in DeepSeek-R1): incentivizes independent generation per turn but "leads to task failure in long-horizon tasks (e.g., coding sessions exceeding 100 turns)" because the model loses context of its previous reasoning.
Retain full reasoning history: provides complete context but "incurs prohibitive context consumption, which rapidly saturates the model's capacity and blocks subsequent tool invocations."
Selective retention (chosen): preserve reasoning traces exclusively for the tool-use trajectory triggered by the most recent user instruction. This provides reasoning coherence for the current task without accumulating irrelevant history across task boundaries.

For tool-use format, the paper compares JSON and XML. JSON's rigid syntax (escape sequences, delimiters) "frequently induces parsing errors in small, under-trained models." XML allows flat string output with lower grammatical overhead, so XML is selected for robustness in complex agentic coding scenarios.

Scalable Code Agent Infrastructure. The deployment architecture centers on a proprietary Session-Router that orchestrates container lifecycles via Kubernetes and ensures interaction consistency through Tmux. This supports thousands of concurrent environments with seamless state persistence, eliminating manual, scaffold-specific Docker configurations.

To prevent overfitting to specific agentic frameworks, the model is trained to adapt to diverse interaction paradigms: academic standards (OpenHands, SWE-agent, Terminus-2) and enterprise-grade protocols (Kilocode, Roocode, ClaudeCode). By exposing the model to varied interaction patterns during training, it remains robust regardless of the underlying execution environment.

3.4.9 Summary of Key Design Choices and Their Justifications

3:1 SWA-to-full attention with 96 SWA query heads over uniform full attention or 1:1 layout: maximizes long-context efficiency while recovering most quality loss through additional query capacity, at negligible overhead (~1–2% FLOPs increase for decode).
Head-wise gated attention over fixed sink tokens: provides input-dependent attention mass modulation, improving average benchmark performance by +1.9 points while adding zero practical FLOPs or latency overhead.
EP-group balance loss over per-expert balance only: directly targets the deployment bottleneck (GPU-level stragglers) rather than just the statistical bottleneck (expert-level imbalance), using a product-form loss that is differentiable and interpretable.
Activation clipping over weight clipping for activation blow-up mitigation: addresses the root cause (sparse, large-magnitude SwiGLU intermediate activations) rather than the symptom (large weight norms), providing bounded stability where weight clipping merely delays explosion.
MIS-PO over importance-sampling PPO or GSPO: replaces continuous reweighting with discrete filtering at both token and trajectory levels, substantially reducing gradient variance at the cost of sample efficiency (80–92% of trajectories discarded), which is acceptable in the large-scale, high-throughput inference setting.
Truncation-aware value bootstrapping over zero-reward truncation: prevents the model from being penalized for thorough reasoning that exceeds generation limits, using the critic's value function to estimate the expected future reward from the truncated state.
Self-distillation over multi-model deployment or direct RL integration: consolidates diverse expert capabilities into a single student model using expert-generated trajectories and rejection sampling, providing a stable initialization that reduces optimization burden on subsequent RL stages.
XML tool-use format over JSON: reduces parsing errors in complex coding scenarios by using a simpler grammar with lower syntactic overhead.
Selective reasoning retention over discarding or retaining all history: balances coherence (necessary for long-horizon tasks) with context efficiency (necessary to avoid saturation), preserving only the reasoning trace for the current user instruction's trajectory.

4. Key Insights and Innovations

Innovation 1: MoE Training Instability Is Diagnosable Through Activation-Side Signals Invisible to Standard Monitoring — and the Loss Function Is the Wrong Place to Look

The most intellectually distinctive contribution of this paper is not the model architecture or the RL algorithm, but a diagnostic reframing of what it means for MoE training to be "stable." The dominant prior assumption — implicit in most large-scale training efforts — is that a smooth, non-diverging loss curve indicates healthy optimization. The paper systematically dismantles this assumption for MoE models, presenting concrete evidence that catastrophic internal pathologies can develop and propagate while the training loss remains indistinguishable from a healthy run.

This is not a small empirical observation. It is a fundamental challenge to the standard monitoring practice in the field, where loss curves, gradient norms, and routing dispatch statistics are the primary (and often only) stability signals. The paper shows that none of these standard signals would detect the three failure modes it documents:

Localized activation blow-up (Section 4.1.3, Figure 4) is the most striking case. In the deeper MoE layers, the activation norm of a small subset of experts (one or two per layer) grows exponentially — exceeding 10,000 while the median expert norm remains at ~5–10. The training loss across methods with and without mitigation is "indistinguishable" (Figure 4(a)). A training engineer monitoring only the loss would see a perfectly healthy run while, internally, a few rogue experts are learning a degenerate shortcut that makes model predictions near-deterministic for specific input patterns. The model is effectively "collapsing" in a way that does not register on the loss because the shortcut is loss-minimizing — it correctly predicts the next token for a high-frequency bi-gram by routing through a single expert and ignoring all other pathways.

Expert collapse beyond routing collapse (Section 4.1.2) breaks the standard assumption that "dead experts" can be detected by tracking dispatch frequency. The paper's prior work (Step-3) already identified that experts can die with vanishing activations and stagnant parameter norms even when router dispatch statistics appear healthy. Prior approaches implicitly assumed that if tokens continue to flow to an expert, the expert is learning. The paper shows this is false: shared-expert scaling mismatches and overly restrictive micro-batch load-balancing can suppress the effective contribution of routed experts without changing dispatch patterns. The diagnostic move here is shifting from "router-side" monitoring (which experts receive tokens?) to "expert-side" monitoring (are the experts actually producing meaningful outputs and receiving meaningful gradients?). This is a conceptual advance in understanding MoE failure modes, not just an engineering fix.

Muon numerical sensitivity (Section 4.1.1) demonstrates that optimizer choice interacts with numerical precision in ways that standard mixed-precision training recipes do not anticipate. The Polar Express iteration inside Muon can produce extreme intermediate outliers in bfloat16 due to cumulative addition error — a pathology that manifests as non-deterministic, unrecoverable loss spikes that standard safety scaling does not prevent. The solution (casting only the NS iteration to float16) is simple, but the diagnostic lesson is broader: optimizers that perform non-trivial matrix operations (Newton-Schulz orthogonalization) introduce numerical failure modes that are invisible to standard gradient-norm monitoring and require optimizer-internal visibility to detect.

Why this reframing matters beyond this paper. The field is rapidly moving toward larger MoE models with increasingly fine-grained expert specialization. If loss-based monitoring is insufficient at 196B total parameters (where these pathologies were observed), it will be even more insufficient at larger scales. The paper's infrastructure contribution — the lightweight asynchronous metrics server that makes per-micro-batch, per-expert activation and parameter norm monitoring feasible at 4,096-GPU scale with only ~100ms overhead — is an enabling contribution for the broader community, not just for this model. It establishes a blueprint for what large-scale MoE monitoring should look like: continuous, fine-grained, activation-side, and decoupled from the training loop.

The max-to-median ratio of per-expert activation norms (Figure 4) is proposed as a specific, robust diagnostic metric that "reliably unmasks underlying instability" even when loss is flat. This is a practical, validated diagnostic that other teams can adopt directly. Prior work on MoE stability (Switch Transformers, ST-MoE, DeepSeekMoE) focused on routing balance and auxiliary losses; this paper argues — with evidence — that stability monitoring needs to operate at the activation level, not just the routing level, and that expert-level granularity (not just layer-level aggregates) is essential.

The significance of this contribution is amplified by a crucial detail: the failure modes are not rare edge cases. The localized activation blow-up is described as occurring "during the main training phase" as "expert specialization matures" — i.e., it is a systematic consequence of successful training, not a bug triggered by specific data or configurations. The combination of fine-grained expert specialization, SwiGLU activation, pre-norm architecture, and Muon orthogonalization creates conditions where this pathology is essentially inevitable at scale. The paper's contribution is both the diagnosis and a practical mitigation (activation clipping, which addresses the root cause — sparse, aligned intermediate activations — rather than the symptom, which is large weight norms that weight clipping only temporarily suppresses).

This innovation is fundamental, not incremental, because it shifts the field's conceptual model of what training stability means for sparse MoE models and provides both the diagnostic tools and the infrastructure to make that new model operational.

Innovation 2: MIS-PO Replaces Continuous Importance Reweighting with Discrete Binary Filtering — and Shows This Is Both Simpler and More Stable for MoE Off-Policy RL

The paper's primary algorithmic contribution is MIS-PO, but the intellectual contribution is not the specific masking formula. It is the rejection of importance sampling as the default mechanism for handling off-policy data in LLM reinforcement learning, and the demonstration that a simpler, more conservative alternative — binary acceptance/rejection based on probability ratios — is both more stable and more sample-efficient in practice for MoE architectures.

Prior assumption. The standard approach to off-policy RL, from PPO through its LLM adaptations, is to correct for distribution shift by multiplying policy gradients by importance weights $w_t = \pi_\theta(a_t|s_t) / \pi_{\text{behavior}}(a_t|s_t)$ . The intuition is elegant: actions that are more likely under the current policy than the behavior policy should be upweighted, and vice versa. This produces an unbiased (or low-bias) gradient estimator under mild assumptions. The entire RLHF and RLVR literature for LLMs — RLHF, DPO, RLAIF, DeepSeekMath, Open-Reasoner-Zero — adopts this mechanism either explicitly (through PPO-style clipping) or implicitly (through reweighting in the loss).

The problem, which the paper argues has been systematically underestimated for MoE models, is that the variance of importance-weighted estimators grows catastrophically with trajectory length. For reasoning tasks where trajectories can be thousands of tokens, the product of per-token probability ratios can span many orders of magnitude. Prior work attempted to mitigate this through clipping (PPO clips individual ratios), through trajectory-level aggregation (GSPO uses geometric mean ratios), or through trust-region constraints (limiting how far the policy can move). But all of these approaches remain within the importance-sampling paradigm: they reweight samples, just with constraints on how extreme the weights can be.

The conceptual shift in MIS-PO. MIS-PO breaks from this paradigm entirely. Instead of asking "how much should I weight this sample?", it asks "should I use this sample at all?" The acceptance criterion is binary: if the probability ratio (at the token level) and the geometric mean ratio (at the trajectory level) fall within pre-specified bounds, the sample is treated as effectively on-policy and given unit weight. Otherwise, it is discarded.

This is conceptually analogous to rejection sampling in Monte Carlo methods: rather than correcting for mismatch between proposal and target distributions through reweighting (which can produce high-variance estimates when the distributions diverge), you simply discard samples that are too far from the target and treat the remaining ones as coming from the target. The tradeoff is obvious: lower variance (because there are no extreme weights) at the cost of lower effective sample size (because many samples are discarded). For LLM RL where the inference engine can generate millions of tokens in parallel, sample efficiency is not the bottleneck — gradient variance is.

Why this is more than just "a different masking scheme." The paper's comparison against GSPO (Figure 7) reveals a deeper point: even when GSPO also uses binary masking (the same token-level and trajectory-level filters), its reliance on importance-weighted updates within the accepted samples produces worse stability. On the MoE model, GSPO exhibits "an increasing training-inference discrepancy during training" (Figure 7(b), right panel), quantified by the density ratio $\pi_{\theta_{\text{old}}} / \pi_{\theta_{\text{vllm}}}$ . This means that even among the "accepted" samples (those passing the binary filters), the distribution shift between training and inference policies continues to grow under GSPO, eventually destabilizing training. MIS-PO, by treating accepted samples as on-policy (weight 1) rather than importance-weighted, constrains this discrepancy within a stable range.

The conceptual implication is that importance weighting actively exacerbates training-inference mismatch in MoE models, not just suffers from it. The paper doesn't fully explain the mechanism, but the evidence suggests that the multiplicative weight updates create a feedback loop: large importance weights produce large gradient updates, which shift the router's expert assignments, which increase the training-inference discrepancy, which produces even larger importance weights in the next iteration. Binary filtering breaks this loop by capping the effective influence of any sample at 1.

The MoE-specific dimension. The paper argues, but does not conclusively prove, that MoE models are particularly vulnerable to importance-sampling instability because of the additional distribution-shift channel through the router. When the policy updates, the expert assignments change, meaning the same token is processed by different experts under the training and inference policies. This creates a type of shift that dense models don't experience, since for dense models the entire network processes every token identically regardless of the policy's output distribution. The paper's Routing Confidence diagnostic ( $\Sigma_k$ , the average probability mass of activated experts) is proposed as a way to quantify this effect: low routing confidence means high uncertainty in expert assignment, which amplifies the impact of policy updates on routing behavior.

Evidence quality. The core evidence for MIS-PO's advantages is in Figures 5, 7, and 8. Figure 5 compares MIS-PO against PPO on the internal model, showing lower gradient norm, higher reward, and slower entropy decay. Figure 7 compares against GSPO on both dense and MoE models, showing MIS-PO maintains stable training where GSPO degrades. Figure 8 shows extended training on the MoE model with sustained stability over 800 steps. The paper reports that MIS-PO discards 80–92% of trajectories (the "all accept ratio" in Figure 7 ranges from 0.08 to 0.20), which underscores how conservative the filtering is — but also that the remaining 8–20% provides sufficient signal for effective learning.

The reported downstream improvements (Figure 6: +3.2% on IMO-AnswerBench, +6.1% on competitive programming, +10.6% on ARC-AGI-1, +3.4% on HLEtext) are significant but are reported on the final model, not as a controlled ablation isolating MIS-PO from other post-training improvements (SFT, self-distillation, reward design). So the causal attribution is unclear: MIS-PO enables stable training, but how much of the final performance is due to MIS-PO versus the quality of the SFT initialization, the reward models, or the data curation is not decomposed.

This innovation is incremental in its mechanism — binary filtering and trust-region methods have long histories in Monte Carlo and optimization — but fundamental in its implication for the LLM RL community: that importance sampling is not the right default for off-policy correction in MoE models, and that a simpler, more conservative approach (discard rather than reweight) can be more effective at scale. The paper makes a convincing case that the field's attachment to importance weighting is a historical accident (carried over from PPO's original design for continuous control) rather than a principled choice for the LLM MoE setting, and that breaking from this default is necessary for stable RL at the frontier.

Innovation 3: Hybrid Attention Quality Loss Can Be Recovered Through Query Head Augmentation and Learned Gating — Without Increasing the Attention Footprint

The paper's architecture contribution is not that SWA is efficient (this was known) or that hybrid attention layouts can work (this was demonstrated in Gemma 3, Command A, DeepSeek-V3.2-Exp), but rather a specific empirical finding about how to close the quality gap between hybrid and full attention at minimal cost, with a principled explanation for why the two mechanisms complement each other.

Prior state. The standard approach to combining local and global attention has been to interleave full-attention layers with efficient local-attention layers at some ratio. The dominant assumption was that the quality loss from replacing some full-attention layers with local (SWA) layers was an inevitable cost of efficiency — you accept some degradation to get the FLOPs savings. Sink tokens were introduced as a way to mitigate SWA's limitation (its inability to absorb unused attention mass when the local window contains no useful information), but they are data-independent and fixed, meaning they provide the same "sink capacity" regardless of whether the current input actually needs it.

The paper's finding. The ablation in Table 1 shows that a naive 3:1 SWA-to-full layout loses quality relative to full attention across multiple benchmarks (BBH: -4.3, MMLU-Pro: -2.0, SFT LongCtx: -1.3). However, increasing the number of SWA query heads and adding head-wise gated attention together recover most of this loss, and in some cases (BBH during pre-training: 65.3 vs. 66.0 for full attention) close the gap entirely, all while maintaining the 3:1 ratio's ~2.7× decode and ~2.9× prefill FLOPs advantage over uniform full attention (Table 8). The overhead of these interventions is characterized as "nearly a 'free lunch'" and "negligible" in both FLOPs and latency (Table 7).

Why these two mechanisms work together — the conceptual argument. The paper provides a deeper explanation than "we tried it and it worked." The head-wise gating mechanism (Equation 6 in Appendix A.1) is shown to be mathematically equivalent to an input-dependent sink token, where $\exp(-g_i) Z_i$ acts as a dynamic sink mass that adapts to the input. In SWA layers, the local window may or may not contain useful context for a given token. A fixed sink token always reserves the same attention capacity for "nothing," regardless of need. Head-wise gating allows each head to independently decide, based on its input, whether to "close the gate" (discard uninformative local attention) or "open the gate" (use the local attention output). Meanwhile, the increased query head count provides additional representational capacity to extract information from the limited 512-token window, partially compensating for the loss of long-range context that SWA imposes. Together, the gating determines when to attend locally, and the augmented heads determine how well to attend when they do. The head-wise gating ablation in Table 2 (on a separate 100B-A10B model, with the $S_3F_1$ layout and 512 window size held fixed) shows a +1.9 point average improvement over fixed sink tokens, confirming this is not simply an artifact of query head augmentation.

Distinctive contribution. Prior work on hybrid attention either: (a) used more expensive ratios (S1F1, ~60% more FLOPs) to preserve quality (e.g., Gemma 3), (b) accepted quality loss as the price of efficiency, or (c) used linear attention to achieve efficiency but sacrificed compatibility with speculative decoding. The paper's contribution is demonstrating that the quality-efficiency Pareto frontier is not as steep as prior results suggested — that with the right combination of architectural tweaks, the 3:1 ratio can achieve near-full-attention quality at a fraction of the cost, without sacrificing speculative decoding compatibility. The head-wise gating result is particularly notable because it is a strict improvement: it adds negligible overhead and never hurts (the paper reports no benchmarks where it underperforms sink tokens or no-gating baselines).

This is an incremental innovation — it refines known techniques (SWA, hybrid attention, sink tokens, GQA) rather than introducing fundamentally new ones — but it is practically significant because it directly enables the 11B-active-parameter deployment target. Without recovering the quality gap, the efficiency gains of SWA would be partly offset by capability loss, undermining the paper's central claim of frontier-level intelligence at low activation cost.

Innovation 4: Test-Time Compute via Tool-Integrated Reasoning and Parallel Coordination Demonstrates That Agentic Capabilities Can Be Amplified by Inference-Time Architecture, Not Just Training-Time Scale

The paper's findings on test-time scaling (Section 5.4, Appendix D.3, Tables 12–13) are not presented as a standalone contribution like MIS-PO, but they constitute a coherent finding with implications beyond this specific model: that tool-augmented reasoning and parallel trajectory synthesis are complementary mechanisms for improving agentic performance at inference time, and that their combination can produce gains that neither achieves alone.

Tool-integrated reasoning (giving the model a Python interpreter during inference) improves performance across reasoning benchmarks: AIME 2025 +2.5% (97.3 → 99.8), HMMT 2025 Nov. +4.0% (94.0 → 98.0), HLEtext +3.4% (23.1 → 26.5). These are not marginal gains — on AIME 2025, the model is already at 97.3%, so a +2.5% improvement represents solving most of the remaining unsolved problems.

PaCoRe (Parallel Coordinated Reasoning) launches parallel reasoning trajectories and synthesizes their insights through multi-round coordination, decoupling reasoning capacity from context limitations. The standard PaCoRe configuration (®K = [4, 4, 4, 4] — four parallel trajectories feeding into another round of four, repeated four times) produces substantial gains: IMO-AnswerBench +3.4% (85.4 → 88.8), LiveCodeBench-v6 +2.5% (86.4 → 88.9), CF-Div2-Stepfun-cpp +7.2% (86.1 → 93.3), Arena-Hard-v2.0 +19.1% (74.0 → 93.1). On Arena-Hard in particular, this vaults Step 3.5 Flash from a competitive score (74.0, behind GPT-5.2 xHigh at 80.6) to a commanding lead (93.1, well ahead of all baselines in Table 5).

Tool-integrated PaCoRe — combining both interventions — is explored as a preliminary result (Table 13): on GPQA-Diamond, tool use alone gives 84.4%, adding PaCoRe brings it to 85.7% (+1.3%); on HLEtext, tool use alone gives 26.5%, adding PaCoRe brings it to 28.2% (+1.7%). These are smaller gains than adding PaCoRe to non-tool reasoning, but the direction is positive and the paper frames it as a "promising avenue for agentic test-time scaling."

What's distinctive. The finding is not that test-time compute helps (this is well-established, e.g., in compute-optimal scaling work and in the reasoning literature), or that tool use helps (similarly well-established). It is that these mechanisms operate on different bottlenecks and combine productively. Tool-integrated reasoning addresses tasks where computation (numerical calculation, programmatic search, simulation) is the limiting factor — the model knows what to do but cannot execute it purely through text generation. PaCoRe addresses tasks where reasoning diversity is the limiting factor — the model can reason, but any single reasoning trajectory may miss important considerations, and synthesizing multiple perspectives produces more robust solutions. The fact that adding PaCoRe to tool-integrated reasoning yields further gains suggests these are indeed complementary dimensions.

The practical significance is amplified by Step 3.5 Flash's extreme inference efficiency. PaCoRe with four parallel trajectories of four rounds (effectively ~16× the inference budget of a single trajectory) is only feasible because the model is fast enough that the wall-clock cost remains tolerable. The paper reports sustaining ~170 tokens/s on Hopper GPUs during the first week on OpenRouter, which makes multi-round parallel reasoning practical in a way it would not be for a slower model. This is a concrete instantiation of the paper's design philosophy (Section 2.1): "minimized latency translates directly to reduced wall-clock time for task completion, or conversely, allows for increased intelligence within a fixed time budget via test-time scaling."

This is an incremental but practically important finding — it validates that the efficiency gains from the architecture are not just about reducing cost but about enabling inference-time strategies (PaCoRe, tool use) that would be too slow to deploy otherwise, creating a virtuous cycle where efficiency enables capability. The paper does not provide a thorough ablation of where the gains come from (e.g., is it parallelism, synthesis quality, or iteration depth that matters most?), so the finding is suggestive rather than conclusive.

5. Experimental Analysis

Evaluation Methodology

Datasets. Pre-training evaluation uses 13 benchmarks spanning general language understanding (BBH, MMLU, MMLU-Redux, MMLU-Pro, HellaSwag, WinoGrande, GPQA, SuperGPQA, SimpleQA), mathematics (GSM8K, MATH), coding (HumanEval, MBPP, HumanEval+, MBPP+, MultiPL-E), and Chinese understanding (C-EVAL, CMMLU, C-SimpleQA). Post-training evaluation uses 22 benchmarks grouped into reasoning (AIME 2025, HMMT 2025 Feb/Nov, IMO-AnswerBench, LiveCodeBench-v6, CF-Div2-Stepfun-cpp, MMLU-Pro, GPQA-Diamond, HLEtext), code agents (SWE-Bench Verified, SWE-Bench Multilingual, Terminal-Bench 2.0), general agents (BrowseComp, BrowseComp with Context Management, BrowseComp-ZH, GAIA, xbench-DeepSearch, RESEARCHRUBRICS, 𝜏2-Bench), general capability (Arena-Hard v2, MultiChallenge, IFBench), and long context (LongBench v2, MRCR-8needle, FRAMES-Oracle, RepoQA). Additional internal benchmarks cover data analysis (50 items) and consulting/recommendations (500 queries). All evaluation details, including prompts and few-shot configurations, are documented in Appendix E.
Base model(s). The model is Step 3.5 Flash, a 196B-parameter sparse MoE with 11B active parameters per token, trained from scratch using the full pipeline described in Sections 3–5. For pre-training comparisons, the baselines include MiMo-V2 Flash Base (15B active, 309B total), GLM-4.5 Base (32B active, 355B total), DeepSeek V3.1 Base (37B active, 671B total), DeepSeek V3.2 Exp Base (37B active, 671B total), and Kimi-K2 Base (32B active, 1043B total). For post-training comparisons, baselines include MiniMax M2.1 (10B active, 230B total), MiMo V2 Flash (15B active, 309B total), GLM 4.7 (32B active, 355B total), DeepSeek V3.2 (37B active, 671B total), Kimi K2.5 (32B active, 1T total), Gemini 3.0 Pro, Claude Opus 4.5, and GPT-5.2 xHigh (parameter counts undisclosed for closed models). The paper justifies PaLM 2-S* as "representative of the capabilities of many contemporary LLMs" and notes it sits in a useful performance regime (~10–19% pass@1) where there is substantial room for test-time compute to improve results.
Metrics. The primary metric is accuracy (% correct), with answer grading following the official protocols of each benchmark. For competitive programming (CF-Div2-Stepfun), pass@8 accuracy and a simplified Codeforces rating (ignoring submission time penalties) are reported. For MRCR-8needle, the Area Under Curve (AUC) at 128K context is computed by the trapezoidal rule, normalizing by context width. For Terminal-Bench 2.0, avg@8 over 8 trials is reported. For BrowseComp and related search benchmarks, avg@3 is reported (except GPT-5.2 xHigh uses avg@1). For SWE-Bench Verified and Multilingual, the standard resolve-rate metric is used with an average of 4 runs. RL training dynamics are tracked via reward curves, actor gradient norms, and entropy.
Baselines. The paper compares against a comprehensive set of both open-source and closed-source models, listed in Table 5 for post-training and Table 4 for pre-training. Specific evaluation protocols vary: when original scores are unavailable or the paper's reproduced results differ, the paper reports its own re-evaluation under the same conditions (marked with * in tables). Scores quoted from non-official sources (technical reports, independent evaluation platforms) are marked with †. For pre-training, DeepSeek scores are quoted from the MiMo-V2-Flash report. For post-training reasoning benchmarks, the paper also reports PaCoRe-augmented results (parallel coordinated reasoning with ®K = [4, 4, 4, 4]) as a separate column to show test-time scaling benefits.
Generation budget / compute accounting. For post-training evaluations, pass@1 accuracy is reported using multiple independent generations per problem: 64 for AIME 2025, HMMT 2025 Feb, and HMMT 2025 Nov; 8 for IMO-AnswerBench, LiveCodeBench, GPQA-Diamond, and MultiChallenge; 1 for HLE; and 4 runs for all other benchmarks. For PaCoRe, the configuration ®K = [4, 4, 4, 4] launches four parallel reasoning trajectories and synthesizes them across four rounds. The model uses decoding temperature and top-p of 1.0, with a maximum sequence length of 256K and YaRN scaling factor of 2.0 for positional embeddings during long-context evaluation. For the agent infrastructure, max interaction turns are limited to 350 for SWE-Bench, 200 for Terminal-Bench 2.0, 400 for search benchmarks, and 30 for RESEARCHRUBRICS, with per-turn output limits and total time limits varying by task.
Cross-validation / statistical protocol. The paper does not employ formal cross-validation for the final evaluation results. For benchmark evaluations, multiple independent runs are averaged (64 for AIME/HMMT, 8 for IMO-AnswerBench/LiveCodeBench, 4 for SWE-Bench, 8 for Terminal-Bench 2.0, 3 for BrowseComp and RESEARCHRUBRICS) to reduce stochastic variance. For internal benchmarks (data analysis, consulting), results are averaged over 3 identical runs. The paper conducts comprehensive benchmark decontamination via exact matching (with digit masking) and N-gram matching to prevent test set leakage. For the CF-Div2-Stepfun benchmark, the evaluator's reliability is validated: it correctly identifies 100% of accepted submissions and 92.45% of failed submissions from real contests. For Terminal-Bench 2.0, each task's checker is verified and refined against its problem statement, improving overall accuracy by approximately 1.5%.

Main Quantitative Results

Pre-Training Performance (Table 4)

Step 3.5 Flash Base achieves competitive performance against substantially larger models across general, mathematics, code, and Chinese understanding benchmarks despite activating only 11B parameters. On BBH, it scores 88.2, within 0.5 points of the best baseline (DeepSeek V3.2 Exp Base at 88.7, which activates 37B parameters — ~3.4× more). On MMLU, it scores 85.8, trailing the best baseline (Kimi-K2 Base, 87.8 with 32B active and 1043B total) by 2.0 points while using ~5.3× fewer total parameters. The most striking efficiency result is on SimpleQA, where Step 3.5 Flash scores 31.6, surpassing DeepSeek-V3.2-Exp Base (27.0) and GLM-4.5 Base (30.0) while using only 196B total parameters versus 671B (DeepSeek) and 355B (GLM). The paper characterizes this as demonstrating "stronger capability density per parameter budget."

On mathematics, Step 3.5 Flash scores 88.2 on GSM8K and 66.8 on MATH. The GSM8K score is competitive but trails MiMo-V2 Flash Base (92.3) and Kimi-K2 Base (92.1), while MATH at 66.8 sits between DeepSeek V3.2 Exp Base (62.5) and MiMo-V2 Flash Base (71.0). On code, HumanEval at 81.1 and MBPP at 79.4 are competitive, with MultiPL-E HumanEval at 67.7 being notably strong — well ahead of all other baselines (DeepSeek V3.2 Exp Base: 45.7; Kimi-K2 Base: 60.5; MiMo-V2 Flash Base: 59.5), potentially reflecting the heavy code emphasis in the pre-training and mid-training data mixtures. On Chinese benchmarks, C-EVAL at 89.6 and CMMLU at 88.9 are competitive with the best baselines.

The paper does not report perplexity or bits-per-byte on held-out validation sets, making it difficult to assess the pre-training base model's generalization beyond benchmark-specific fine-tuning. The pre-training evaluation uses standard few-shot prompting (3-shot to 10-shot depending on benchmark), consistent with the protocols described in Appendix E.1.

Post-Training Reasoning Performance (Table 5, Vanilla Column)

Step 3.5 Flash achieves frontier-level reasoning performance with only 11B active parameters. On AIME 2025, it scores 97.3%, ahead of DeepSeek V3.2 (93.1%), Kimi K2.5 (96.1%), Gemini 3.0 Pro (95.0%), and Claude Opus 4.5 (92.8%), and within 2.7 points of GPT-5.2 xHigh (100.0%). The gap to GPT-5.2 xHigh on this benchmark is 2.7%, representing a small fraction of unsolved problems. On HMMT 2025 Feb., Step 3.5 Flash scores 98.4%, outperforming all baselines except GPT-5.2 xHigh (99.4%) and Claude Opus 4.5 (which scores 92.9% — note the † marker indicating non-official source). On HMMT 2025 Nov., the score of 94.0% places it ahead of DeepSeek V3.2 (90.2%) and Kimi K2.5 (91.1%), and competitive with GPT-5.2 xHigh (97.1%).

On IMO-AnswerBench, which tests robust mathematical reasoning with formatted answers, Step 3.5 Flash scores 85.4%, ahead of all open-source baselines (Kimi K2.5: 78.3%; DeepSeek V3.2: 81.8%; GLM 4.7: 82.0%) and competitive with closed-source models (Gemini 3.0 Pro: 83.3%; Claude Opus 4.5: 84.0%; GPT-5.2 xHigh: 86.3%). The gap between Step 3.5 Flash and GPT-5.2 xHigh is 0.9 percentage points — within the margin of statistical noise for 8-sample pass@1 estimation. On LiveCodeBench-v6, which evaluates code generation across recent competitive programming problems, Step 3.5 Flash scores 86.4%, ahead of DeepSeek V3.2 (83.3%), Kimi K2.5 (85.0%), and Claude Opus 4.5 (84.8%), and within 1.3 points of GPT-5.2 xHigh (87.7%). On CF-Div2-Stepfun-cpp, a custom C++ competitive programming benchmark, Step 3.5 Flash scores 86.1% (avg@8), ahead of all baselines including Gemini 3.0 Pro (83.5%) and GPT-5.2 xHigh (not reported on this benchmark).

On knowledge-intensive reasoning, MMLU-Pro at 84.4% and GPQA-Diamond at 83.5% are competitive but trail the strongest closed-source models: on GPQA-Diamond, Gemini 3.0 Pro achieves 91.9%, GPT-5.2 xHigh achieves 92.4%, reflecting a 7.9-point or 8.9-point gap. On HLEtext (Humanity's Last Exam, text-only subset), Step 3.5 Flash scores 23.1%, competitive with Kimi K2.5 (25.1%) and DeepSeek V3.2 (24.8%) but substantially behind Gemini 3.0 Pro (37.7%) and GPT-5.2 xHigh (35.5%) — a 12–14 point gap on this extremely challenging benchmark.

Post-Training Agentic Performance (Table 5, Vanilla Column)

On code agent benchmarks, Step 3.5 Flash demonstrates strong software engineering capabilities. SWE-Bench Verified at 74.4% is competitive with all baseline models (range: 73.1% for Kimi K2.5 to 80.9% for Claude Opus 4.5, with GPT-5.2 xHigh at 80.0%). SWE-Bench Multilingual at 67.4% is mid-pack: ahead of DeepSeek V3.2 (66.7%) but behind Kimi K2.5 (70.2%) and MiMo V2 Flash (72.5%). Terminal-Bench 2.0 at 51.0% is competitive with the best open-source baselines (Kimi K2.5: 50.8%; DeepSeek V3.2: 46.4%) and within 3 points of GPT-5.2 xHigh (54.0%). Notably, the paper reports that 88.6% of successful trajectories on Terminal-Bench 2.0 completed within 30 interactions (out of 200 allowed), suggesting the model is efficient in its tool-use rather than relying on exhaustive exploration. Ablation studies on Terminal-Bench 2.0 (Table 16) show that limiting the per-turn output to 16K tokens causes the largest performance drop (to 48.03%) because the model's reasoning for complex tasks requires longer outputs.

On general agent benchmarks, BrowseComp (without context management) at 51.6% is competitive with DeepSeek V3.2 (52.0%) but trails Kimi K2.5 (60.6%) significantly. With context management enabled, BrowseComp jumps to 69.0%, surpassing DeepSeek V3.2 (67.5% — note that DeepSeek V3.2 also uses context management for its reported BrowseComp score) and GPT-5.2 xHigh (65.8%). The paper reports that the Discard-all context management strategy achieves 66.0% on a 200-instance BrowseComp subset, while a Multi-Agent orchestration strategy reaches 68.5% (Table 17), suggesting that the reported 69.0% may use an ensemble of strategies.

GAIA at 84.5% is the strongest result among all evaluated models (GPT-5.2 xHigh: 83.5%; Gemini 3.0 Pro: 76.6%; Claude Opus 4.5: 76.1%). This is a significant result because GAIA tests multi-step reasoning with tool use in realistic scenarios, and Step 3.5 Flash's score represents state-of-the-art performance on this benchmark at the time of evaluation. On xbench-DeepSearch-2505, Step 3.5 Flash scores 83.7%, behind GPT-5.2 xHigh (83.0%) but ahead of all other baselines. On the newer xbench-DeepSearch-2510, the score drops to 56.3%, behind GPT-5.2 xHigh (67.0%) and Claude Opus 4.5 (59.3%) but ahead of DeepSeek V3.2 (55.7%) and Kimi K2.5 (40.0%). RESEARCHRUBRICS at 65.3% surpasses all evaluated models, including commercial agent systems (Gemini DeepResearch: 63.69, OpenAI DeepResearch: 60.67; Table 18). 𝜏2-Bench at 88.2% is third behind Claude Opus 4.5 (92.5%) and Gemini 3.0 Pro (90.7%), and 2.7 points ahead of GPT-5.2 xHigh (85.5%).

On general capability benchmarks, Step 3.5 Flash's Arena-Hard v2.0 score of 74.0% is competitive with GLM 4.7 (73.1%) and DeepSeek V3.2 (66.0%) but substantially behind Gemini 3.0 Pro (85.8%) and GPT-5.2 xHigh (80.6%). MultiChallenge at 55.7% trails GPT-5.2 xHigh (71.9%) and Gemini 3.0 Pro (71.8%) by a wide margin. IFBench at 67.4% is mid-pack, behind GPT-5.2 xHigh (75.4%) and Gemini 3.0 Pro (72.8%) but ahead of Claude Opus 4.5 (58.0%).

On long-context benchmarks, Step 3.5 Flash scores LongBench v2 at 57.5%, competitive with the range (53.9% to 61.0%) but behind the best closed-source models (Gemini 3.0 Pro: 70.0%; Claude Opus 4.5: 67.8%). MRCR-8needle AUC at 28.8% is at the lower end of the range, substantially behind GPT-5.2 xHigh (88.2%) and Claude Opus 4.5 (54.0%). FRAMES-Oracle at 76.5% and RepoQA at 88.5% are competitive with the model range, though notably behind GPT-5.2 xHigh (87.3% and 93.8%, respectively) and Claude Opus 4.5 (85.8% and 95.7%).

PaCoRe Test-Time Scaling (Table 5, PaCoRe Column)

Applying PaCoRe with ®K = [4, 4, 4, 4] produces substantial gains across reasoning, general, and long-context benchmarks. On AIME 2025, PaCoRe improves accuracy from 97.3% to 99.9% (+2.6%), surpassing GPT-5.2 xHigh's reported 100.0% (though noting that 99.9% at 64 samples versus 100.0% may reflect statistical noise rather than a meaningful difference). On HMMT 2025 Feb., PaCoRe achieves 100.0% (from 98.4%, +1.6%), matching or exceeding all baselines. On HMMT 2025 Nov., PaCoRe achieves 97.8% (+3.8% from 94.0%). On IMO-AnswerBench, PaCoRe achieves 88.8% (+3.4%), surpassing GPT-5.2 xHigh (86.3%). On LiveCodeBench-v6, PaCoRe achieves 88.9% (+2.5%), surpassing GPT-5.2 xHigh (87.7%). On CF-Div2-Stepfun-cpp, PaCoRe achieves 93.3%, a dramatic +7.2% improvement that puts it well ahead of all baselines (Gemini 3.0 Pro: 83.5%, which is the next best vanilla score reported on this benchmark).

On general capability, Arena-Hard v2.0 jumps from 74.0% to 93.1% under PaCoRe (+19.1%), surpassing all baselines including GPT-5.2 xHigh (80.6%) and Gemini 3.0 Pro (85.8%) by a substantial margin. This is the largest single improvement from PaCoRe, suggesting that multi-round synthesis is particularly effective for the open-ended instruction-following tasks that Arena-Hard evaluates. MultiChallenge improves from 55.7% to 60.8% (+5.1%). HLEtext improves from 23.1% to 27.9% (+4.8%), closing some of the gap to frontier models.

Interestingly, IFBench degrades under PaCoRe, dropping from 67.4% to 56.8% (−10.6%). The paper does not comment on this regression, but it may indicate that PaCoRe's synthesis process introduces formatting inconsistencies that IFBench's strict instruction-following evaluation penalizes. On long-context benchmarks, LongBench v2 improves from 57.5% to 62.0% (+4.5%), but MRCR-8needle degrades from 28.8% to 26.3% (−2.5%). The degradation on MRCR suggests that PaCoRe's parallel trajectories may interfere with the precise needle retrieval task, perhaps due to context window fragmentation across parallel branches.

Tool-Integrated Reasoning (Table 12)

Equipping Step 3.5 Flash with a Python interpreter during inference (tool-integrated reasoning with 100-turn limit) produces consistent improvements: AIME 2025 +2.5% (97.3 → 99.8), HMMT 2025 Feb. +0.3% (98.4 → 98.7, a small gain on an already near-ceiling score), HMMT 2025 Nov. +4.0% (94.0 → 98.0), IMO-AnswerBench +1.3% (85.4 → 86.7), GPQA-Diamond +0.9% (83.5 → 84.4), HLEtext +3.4% (23.1 → 26.5), ARC-AGI-1 +1.7% (54.8 → 56.5). The largest absolute gains are on the benchmarks where computational execution (rather than purely linguistic reasoning) is most needed: HMMT 2025 Nov. and AIME 2025 involve substantial numerical computation.

Tool-Integrated Parallel Reasoning (Table 13)

Combining tool use with PaCoRe yields further improvements: GPQA-Diamond goes from 84.4% (tool only) to 85.7% (+1.3%), and HLEtext goes from 26.5% (tool only) to 28.2% (+1.7%). These are preliminary results, and the paper notes they represent a first exploration of extending PaCoRe to multi-turn interactive environments. The state-aware input serialization protocol (Table 14) is introduced to enable PaCoRe in tool-augmented settings.

Internal Benchmark Results (Tables 19, 20)

On the Data Analysis Benchmark (50 tasks with an average of 26.9 rubric items each), Step 3.5 Flash scores 39.6%, second only to Claude Opus 4.5 (45.0%) and essentially tied with GPT-5.2 (39.3%). The paper notes that Step 3.5 Flash "demonstrates a favorable speed–capability trade-off, maintaining solid analytical quality while delivering faster responses." On the Consulting and Recommendations Benchmark (500 queries across four dimensions: Usefulness, Logic, Instruction Following, Tone), Step 3.5 Flash scores 70.5% average, virtually tied with Gemini 3.0 Pro (70.6%) and ahead of DeepSeek V3.2 (70.3%) and GLM 4.7 (70.3%). Notably, Step 3.5 Flash's Logic score (62.1%) is competitive with the best models (GPT-5.2: 81.9% in Logic is the outlier, but this is the same model used as one of the judges, introducing potential circularity). The paper characterizes this as the model "reducing hallucinations and logical failures" relative to its size class.

Search Agent Tool-Use Gains (Table 11)

The paper evaluates the gain from enabling search tools (Δtool = Score_with_tools − Score_without_tools), decoupling the model's parametric knowledge from its ability to leverage external retrieval. Step 3.5 Flash achieves the highest average Δtool at 52.0 across the five evaluated benchmarks (BrowseComp: Δ50.1; BrowseComp-ZH: Δ41.9; GAIA: Δ67.5; xbench-DeepSearch-2505: Δ57.7; xbench-DeepSearch-2510: Δ42.7), substantially ahead of all baselines. Kimi K2.5 has the next highest average Δ at 40.2, and Gemini 3.0 Pro averages only 29.7. The paper argues this metric is "critical because high absolute scores on benchmarks like BrowseComp can sometimes stem from strong internalized knowledge rather than effective search strategies." This is a methodological contribution to agent evaluation: reporting the tool-use gain isolates the model's retrieval capability from its memorized knowledge, providing a cleaner signal of agentic competence.

Ablation Studies and Robustness Checks

Architecture ablations on 30B-A3B (Table 1, Table 10): The full pipeline (1.4T token pre-training followed by SFT with 32K long-context extension and 64K SFT) evaluates four attention layouts. The key finding is that the naive S3F1 layout (3:1 SWA-to-full) achieves the lowest FLOPs (1.00/1.00 normalized for decode/prefill) but degrades quality: BBH drops from 66.0 (FFFF) to 61.7, and SFT LongCtx drops from 28.8 (FFFF) to 27.5. The S3F1+Head variant (96 SWA query heads) recovers most of this gap: BBH at 65.3 vs. 66.0, MMLU-Pro at 37.4 vs. 35.7, and SFT General at 44.0 vs. 42.7, while increasing FLOPs by only ~1–2% (decode) and ~4–8% (prefill). The S1F1 layout achieves the best overall SFT quality (LongCtx: 29.6) but at ~1.58–1.71× the FLOPs — a 60% cost increase. The paper selects S3F1+Head as the default.

Head-wise gating vs. sink tokens on 100B-A10B (Table 2): In a controlled pretraining comparison (S3F1 layout, W=512, ~250B tokens), head-wise gated attention improves average performance from 62.5 to 64.4 (+1.9). The largest gains are on BBH (70.6 → 73.7, +3.1), MMLU (65.1 → 67.0, +1.9), and CMMLU (74.6 → 77.1, +2.5). This validates that head-wise gating is a strict improvement over fixed sink tokens at scale.

Activation clipping vs. weight clipping (Figure 4): On the full Step 3.5 Flash training run, activation clipping bounds maximum expert output norms (Layer 45 max stays at ~10^2) while weight clipping only delays the explosion (max eventually rises to ~10^5). Training loss is indistinguishable across methods, confirming that loss-based monitoring would miss this pathology. The paper establishes max-to-median ratio of per-expert activation norms as the monitoring metric.

MIS-PO vs. PPO (Figure 5): On the internal model over ~4,000 training steps, MIS-PO achieves higher reward (~0.55 vs. ~0.45), lower actor gradient norm (no spikes vs. large spikes for PPO), and slower entropy decay (0.4 → 0.2 vs. 0.4 → 0.1 for PPO). The paper characterizes this as improved "sample efficiency," "stability," and "exploration persistence."

MIS-PO vs. GSPO on dense and MoE models (Figure 7): On the dense model (Figure 7a), MIS-PO achieves higher reward, higher all-accept ratio, and comparably low gradient norm. On the MoE model (Figure 7b), the differences are starker: GSPO plateaus around iteration 200 while MIS-PO continues improving, and GSPO "exhibits an increasing training-inference discrepancy during training, quantified by the density ratio π_θold / π_θvllm" which MIS-PO maintains in a stable range near 1.0. This is the key evidence that MIS-PO is specifically beneficial for MoE architectures.

Extended MIS-PO training on MoE (Figure 8): Over ~800 steps on a challenging dataset, reward increases from 0.35 to 0.60, gradient norm remains stable with no smoothing, and entropy decays from 0.48 to 0.24. This demonstrates scalability.

Terminal-Bench 2.0 ablations (Table 16): Limiting per-turn output to 16K tokens causes the largest drop (50.98 → 48.03), and disabling context management under 16K limits drops further (45.22). Reducing max rounds from 200 to 100 has minimal impact (50.42), and reducing time limit from 6h to 2h causes a moderate drop (49.72).

Context management strategies on BrowseComp subset (Table 17): On 200 BrowseComp instances, accuracy ranges from 49.5% (no context management, 86 real steps) to 68.5% (Multi-Agent, 721 steps). The Discard-all strategy achieves 66.0% with 302 steps, and the paper positions accuracy as scaling with real steps — "intensive context management effectively converts increased computation into superior performance."

MetaRM ablation (Section 5.2.2): MetaRM-augmented GenRM outperforms vanilla GenRM by 0.5–3% on every benchmark over 200 RL training steps. No table is provided for this ablation; it is only mentioned in text.

Model-based verifier for STEM (Appendix D.2.2): Using gpt-oss-120b as a verifier model for STEM tasks outperforms vanilla math-verify by an average of 2.0% over 450 RL training steps. No table is provided.

ReST^EM revision model (Appendix K, Figure 16): This negative result shows that applying ReST^EM to further optimize the revision model backfires: at 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal sequential-to-parallel ratio. The paper hypothesizes that on-policy data collection amplifies spurious correlations, destabilizing revision training. This is not an ablation of Step 3.5 Flash itself but of an alternative post-training approach tested during development.

Step 3.5 Flash + Step-GUI for mobile agents (Section E.3.3): An edge-cloud collaborative paradigm (Step 3.5 Flash for planning, Step-GUI for on-device execution) achieves 57.0% on AndroidDaily Hard versus 40.0% for edge-only Step-GUI. This validates the model's utility as a reasoning orchestrator in resource-constrained deployment settings.

Multiple agent scaffold evaluations on SWE-Bench Verified (Section E.2.2): Step 3.5 Flash achieves 74.4% with OpenHands CodeAct Agent, 74.2% with SWE-Agent, and 72.0% with Claude Code, demonstrating robustness across diverse agentic frameworks.

Critical Assessment

The paper's central claim is that Step 3.5 Flash achieves "frontier-level intelligence with only 11B active parameters." The evidence in Table 5 broadly supports this claim for the specific benchmarks evaluated: on reasoning benchmarks (AIME 2025, IMO-AnswerBench, LiveCodeBench-v6), Step 3.5 Flash is competitive with or superior to all models evaluated, including those activating 3–4× more parameters. On agentic benchmarks (SWE-Bench, Terminal-Bench 2.0, GAIA, BrowseComp with Context Management), it matches or exceeds open-source alternatives at higher activation counts. However, several important qualifications apply.

The claim holds conditionally across benchmark categories. The model's strongest results are on mathematical reasoning and structured agentic tasks (SWE-Bench, GAIA, 𝜏2-Bench). On knowledge-intensive reasoning (GPQA-Diamond: -8.9 points vs. GPT-5.2 xHigh; HLEtext: -12.4 points vs. Gemini 3.0 Pro) and general instruction-following (Arena-Hard: -6.6 points vs. GPT-5.2 xHigh; MultiChallenge: -16.2 points vs. GPT-5.2 xHigh), the gaps to frontier closed-source models are substantial — 8 to 16 percentage points. These are not marginal differences; they represent qualitatively different levels of capability on difficult, open-ended tasks. The paper's claim of "frontier-level" intelligence is most convincing for structured, verifiable tasks (math, code, SWE) and less convincing for open-ended reasoning and instruction-following.

The evaluation relies heavily on pass@k with large k, which benefits from the model's inference efficiency but makes direct comparison to single-sample evaluations difficult. All baseline models use comparable pass@k budgets (64 for AIME, 8 for IMO-AnswerBench), so the comparison is fair on this dimension. However, the PaCoRe results use substantially more inference compute: ®K = [4, 4, 4, 4] effectively multiplies the inference budget by ~16×. The PaCoRe column in Table 5 is therefore a test-time scaling result, not a model capability result, and the gains (+19.1% on Arena-Hard, +7.2% on CF-Div2-Stepfun) should be attributed to the inference strategy as much as to the model. The paper is transparent about this (PaCoRe is a separate column), but readers may conflate the PaCoRe-augmented scores with vanilla model capability.

The absence of latency measurements in evaluation is a significant gap. The paper's design philosophy emphasizes that "minimized latency translates directly to reduced wall-clock time for task completion" (Section 2.1) and reports ~170 tokens/s on Hopper GPUs during the first week on OpenRouter. However, no latency benchmarks are reported for the evaluation tasks — we don't know the wall-clock time for SWE-Bench, Terminal-Bench, or BrowseComp relative to baseline models. A model that achieves 74.4% on SWE-Bench but takes 3× longer per task than DeepSeek V3.2 (which achieves 73.8%) would have a less favorable effective capability-to-latency ratio than the raw accuracy suggests. The time limits reported (6h for Terminal-Bench, 1200s tool execution timeout for SWE-Bench) are upper bounds, not typical completion times.

Single model, single training run. All results are from Step 3.5 Flash, a single trained model. There are no error bars, confidence intervals, or multiple training seeds to assess variance. For benchmarks with multiple evaluation runs (pass@64, avg@8), the reported numbers are point estimates without uncertainty quantification. This makes it impossible to assess whether, e.g., the 74.4% on SWE-Bench Verified is statistically distinguishable from DeepSeek V3.2's 73.8% or Kimi K2.5's 73.1%.

No controlled ablation isolating MIS-PO's contribution to final performance. The downstream evaluation improvements attributed to RL (Figure 6: +3.2% on IMO-AnswerBench, +6.1% on CF-Div2, +10.6% on ARC-AGI-1, +3.4% on HLEtext) represent the total improvement from the entire RL phase, which includes SFT initialization, expert model construction, self-distillation, reward model design, data curation, and MIS-PO. There is no controlled experiment comparing MIS-PO against PPO or GSPO on the final model's downstream benchmarks. The ablation in Figure 7 compares training dynamics on an internal model, not final evaluation scores. The paper demonstrates that MIS-PO enables stable training, but the causal link between MIS-PO (rather than, say, the quality of the reward model or the data mixture) and the final benchmark scores is untested.

The MetaRM and model-based verifier ablations are mentioned without supporting tables or figures. Two of the post-training improvements (MetaRM over vanilla GenRM by 0.5–3%, model-based verifier over math-verify by +2.0%) are stated in text without any visual evidence, making it impossible to assess the reliability or statistical significance of these claims.

Internal benchmarks use the same model families as judges. The Consulting and Recommendations Benchmark uses GPT-5.2 as the judge for three of four scoring dimensions (Usefulness, Logic, Instruction Following) and Claude Opus 4.5 for the fourth (Tone). GPT-5.2 is also one of the evaluated models, and it achieves the highest Logic score (81.9%). This circularity — using a model to judge itself and its competitors — is a known source of bias in LLM-as-judge evaluations. The paper states that "judge reliability is validated through an alignment study with human experts, yielding a high Pearson correlation," but this validation result is not reported with specific numbers or methodology.

The difficulty estimation cost for the compute-optimal policy is unaccounted for in the PaCoRe results. While PaCoRe is a test-time strategy that does not require explicit difficulty estimation, the paper does not discuss the compute overhead of generating parallel trajectories, synthesizing them, and managing the multi-round coordination. The total inference budget for PaCoRe is approximately 16× a single trajectory, plus the cost of synthesis rounds. The paper's efficiency claims (11B active parameters, ~170 tokens/s) suggest this is affordable, but no end-to-end wall-clock comparison is provided against baseline models running simpler inference strategies.

The gap on long-context retrieval (MRCR-8needle) is large and unexplained. At 28.8% AUC (26.3% with PaCoRe), Step 3.5 Flash is at the bottom of the evaluated models on this benchmark, with GPT-5.2 xHigh at 88.2% and Claude Opus 4.5 at 54.0%. MRCR tests the ability to retrieve multiple needles from long contexts — a capability that the hybrid attention architecture (S3F1 with 512-token SWA windows) might be expected to struggle with, since needles outside the local window are invisible to SWA layers and must be routed through the sparse full-attention layers. The paper does not analyze this result or discuss whether it reflects a fundamental limitation of the architecture or a correctable issue.

The tool-use gain metric (Δtool, Table 11) is a valuable methodological contribution but potentially confounded by base model capability. A model with very low baseline (no-tools) performance has more room to improve with tools than a model that already answers many questions correctly from internal knowledge. Step 3.5 Flash's low baseline scores on these benchmarks (0–26% without tools) make the high Δtool partly a reflection of low starting performance. However, Kimi K2.5 has even higher baselines on some benchmarks (e.g., 40.3 on BrowseComp-ZH vs. Step 3.5 Flash's 25.0) and achieves lower Δtool, suggesting the result is not purely an artifact. The metric is more informative when baseline scores are comparable across models, which they are not in Table 11.

Experiments that would strengthen the paper: (1) A controlled evaluation isolating MIS-PO's contribution by comparing final model checkpoints trained with MIS-PO vs. PPO vs. GSPO under otherwise identical data and initialization. (2) Latency benchmarks on SWE-Bench, Terminal-Bench, and BrowseComp to quantify the wall-clock efficiency advantage. (3) Multiple training seeds or bootstrap confidence intervals to assess statistical reliability of the benchmark comparisons. (4) A breakdown of PaCoRe's contribution: how much gain comes from parallelism, how much from multi-round synthesis, and how much from increased total inference compute. (5) Analysis of the MRCR-8needle failure to determine whether it is architectural (SWA window size limits needle retrieval), training-related (insufficient long-context needle-retrieval data), or correctable through prompting. (6) Human evaluation on a subset of agentic tasks to validate that benchmark scores reflect genuine capability rather than benchmark-specific optimization — the paper's comprehensive decontamination efforts suggest awareness of this concern, but no human validation is reported.

6. Limitations and Trade-offs

The Model Produces Systematically Longer Reasoning Trajectories Than Frontier Competitors

The assumption or constraint. The paper openly acknowledges in Section 7 that "Step 3.5 Flash achieves frontier-level intelligence but currently requires longer generation trajectories than Gemini 3.0 Pro to reach comparable quality." This is not presented with quantitative evidence, but the implication is clear: the model compensates for lower per-token capability (from having only 11B active parameters) by generating more tokens — reasoning longer, exploring more alternatives, producing more verbose outputs.

The consequence. This limitation undermines the paper's central efficiency narrative. The headline claim — frontier-level intelligence at 11B active parameters — is true in terms of accuracy per inference operation (FLOPs per token), but potentially misleading in terms of accuracy per wall-clock task completion. If the model requires 2–3× more tokens to solve a problem than Gemini 3.0 Pro, the effective latency advantage of 11B active parameters is reduced or negated. For latency-sensitive agentic deployments — where the paper explicitly argues that "minimized latency translates directly to reduced wall-clock time for task completion" (Section 2.1) — this is a critical unmeasured quantity. A model that processes ~170 tokens/s but generates 3× more tokens to reach the same answer takes ~1.5× longer total wall-clock time than a model processing ~100 tokens/s with concise outputs. The paper's design philosophy frames inference latency as a first-class constraint, yet the evaluation provides no latency benchmarks for any of the 22 main evaluation tasks. The time limits reported (6h for Terminal-Bench, 1200s tool execution timeout for SWE-Bench) are hard upper bounds, not typical completion times, and no comparison to baseline models' task completion times is provided.

What evidence exists in the paper. The paper provides no quantitative evidence on this limitation. Section 7's acknowledgment is the only mention. No token-count statistics, no completion-time comparisons, and no analysis of output verbosity relative to baselines is presented. The internal Consulting and Recommendations Benchmark (Table 20) notes that Step 3.5 Flash "demonstrates a favorable speed–capability trade-off" in an aside, but no numbers are provided. The PaCoRe results indirectly highlight this issue: PaCoRe multiplies the inference budget by ~16× (four parallel trajectories, four rounds), and the gains (+19.1% on Arena-Hard, +7.2% on competitive programming) demonstrate that additional tokens convert into accuracy — but the paper does not report whether baseline models could achieve similar gains with comparable test-time compute budgets. A practitioner evaluating whether to deploy Step 3.5 Flash versus, say, DeepSeek V3.2 (37B active) needs to know the total tokens generated per task for both models at comparable accuracy levels, and the paper provides no basis for this comparison.

Mitigation status. The paper acknowledges the limitation and states an intention to address it: "Next step we will prune and compress the thinking for better efficiency while maintaining the same competitive performance" (Section 7). No method, timeline, or preliminary results are provided. This is a statement of future direction, not a mitigation.

Difficulty Estimation for Deployment Is Completely Unaccounted for in the Efficiency Claims

The assumption or constraint. The paper does not address prompt difficulty estimation for deployment. This is not a limitation of the model architecture or training but of the evaluation framework: the benchmarks used (AIME, HMMT, SWE-Bench, etc.) have uniform formats and known difficulty ranges. In real-world deployment with heterogeneous user queries, the agentic infrastructure needs to decide how much test-time compute to allocate per query (e.g., whether to run PaCoRe, how many parallel trajectories to launch, whether to enable tool use). The paper provides no mechanism for making these decisions dynamically, no difficulty estimator, and no analysis of the cost of deciding how to allocate inference compute. The PaCoRe configuration (®K = [4, 4, 4, 4]) is applied uniformly to all problems, which is wasteful for easy queries and potentially insufficient for the hardest ones.

The consequence. The efficiency gains from the architecture (11B active parameters, ~170 tokens/s throughput) are partially offset by the absence of adaptive compute allocation. In a deployment with mixed-difficulty queries, the uniform application of expensive test-time strategies (PaCoRe, tool use) wastes inference budget on easy problems that could be solved with a single trajectory. Conversely, the hardest problems — where the paper shows the model trails frontier systems substantially (HLEtext: -12.4 points vs. Gemini 3.0 Pro; MRCR-8needle: -59.4 points vs. GPT-5.2 xHigh) — might benefit from strategies that exceed the fixed PaCoRe budget. Without difficulty estimation, the effective cost-per-useful-answer in deployment could be substantially higher than the per-token throughput numbers suggest. The paper's earlier discussion of compute-optimal test-time scaling (from the prior literature they cite, e.g., Snell et al., 2025, reference [45]) makes this absence notable: the paper is aware of the difficulty-conditional allocation problem but does not address it.

What evidence exists in the paper. None. The paper does not measure or discuss difficulty estimation, adaptive compute allocation, or the overhead of deciding inference strategies per query. The PaCoRe results in Table 5 show uniform gains, but the distribution of gains across difficulty levels is not analyzed. On some benchmarks (AIME 2025, HMMT 2025 Feb.), PaCoRe provides small gains (+2.6%, +1.6%) because the vanilla model is already near ceiling, suggesting the ~16× compute overhead is wasted on these problems. On others (Arena-Hard v2.0: +19.1%), the gains are enormous, suggesting the vanilla model would have been undertested without PaCoRe. A difficulty-conditional policy could selectively apply PaCoRe to problems where it helps most, improving the effective compute-to-accuracy tradeoff, but no such policy is developed.

Mitigation status. Not addressed. This is a deployment engineering gap, not a model limitation, but it directly impacts the practical cost-effectiveness of the system. The paper's agent infrastructure (Section 5.4) focuses on session management, template design, and cross-framework generalization — not on dynamic resource allocation. No future work is suggested on this topic.

The Model's Long-Context Retrieval Capability (MRCR-8needle) Is Near the Bottom of the Evaluated Models, and the Paper Does Not Investigate Why

The assumption or constraint. Step 3.5 Flash achieves an MRCR-8needle AUC of 28.8% (vanilla) and 26.3% (PaCoRe) at 128K context length — the lowest among all models evaluated in Table 5. This compares to 88.2% for GPT-5.2 xHigh, 73.0% for Claude Opus 4.5, and 36.5% for Kimi K2.5 (also a sparse model). MRCR-8needle tests the ability to retrieve multiple pieces of information ("needles") distributed throughout a 128K-token context — a capability directly relevant to the long-context agentic workloads the paper's architecture is designed for. The model's poor performance on this benchmark may reflect a fundamental limitation of the hybrid attention architecture: with 75% of attention layers using 512-token sliding windows, any information outside the local window of the current token is invisible to those layers. Needles must be routed through the sparse full-attention layers (only 25% of layers), which must both process the needle and propagate its information to the checkpoint where it's needed for retrieval — all while competing with all other long-range information in the context.

The consequence. For agentic deployments that involve retrieval from long documents, multi-document synthesis, or long conversational histories, this limitation could manifest as unreliable information access. A search agent built on Step 3.5 Flash might successfully retrieve documents but fail to extract and synthesize specific facts from them if those facts are separated by more than 512 tokens. The strong performance on BrowseComp with Context Management (69.0%) and GAIA (84.5%) suggests the model can compensate for this weakness through external tool use (search, visit, file parsing), effectively offloading retrieval to tools and using the model primarily for synthesis. But for tasks where the relevant information is already in the context window and must be retrieved without tool interaction, the MRCR-8needle result suggests the model is significantly less capable than its frontier competitors. This is particularly concerning because the model's design philosophy explicitly targets "extensive context prefilling" (Section 2.1) as a key agentic workload — if the model cannot reliably retrieve information from the prefill it's optimized for, the architecture's efficiency advantage is applied to a capability that doesn't work well.

What evidence exists in the paper. Table 5 reports the 28.8% MRCR-8needle score with no analysis, ablation, or discussion. The paper provides no investigation of whether the limitation is: (a) architectural (SWA window size insufficient for needle retrieval, and sparse full-attention layers insufficient to compensate), (b) training-related (insufficient long-context needle-retrieval data in mid-training), (c) inference-related (the chosen YaRN scaling factor of 2.0 degrades retrieval capability at 128K), or (d) a combination. The architecture ablations in Section 2.3 and Tables 1–2 provide some indirect evidence: the S3F1 layout reduces LongCtx quality relative to full attention (SFT LongCtx: 27.5 for S3F1 vs. 28.8 for FFFF and 29.6 for S1F1), but the S3F1+Head variant partially recovers it (28.2). This suggests the chosen architecture is fundamentally less capable at long-context retrieval than full-attention or higher-ratio hybrid layouts, and the paper did not find a way to fully close this gap. Notably, the paper evaluates long-context capability on four benchmarks (LongBench v2, MRCR-8needle, FRAMES-Oracle, RepoQA), and Step 3.5 Flash's performance is competitive on three of them — MRCR-8needle is the outlier, suggesting the limitation is specific to precise multi-needle retrieval rather than general long-context understanding.

Mitigation status. The paper does not acknowledge or address this limitation. No ablation studies specifically target the MRCR-8needle result. The Context Management strategies explored on BrowseComp (Table 17) might be applicable — Discard-all and Multi-Agent approaches could potentially improve retrieval by forcing re-reading — but this is not tested. The YaRN scaling factor of 2.0 at 256K evaluation context (applied to full-attention layers only) is mentioned but not ablated for MRCR performance. A practitioner evaluating this model for retrieval-heavy agentic workloads would need to conduct their own evaluation to determine whether the MRCR-8needle result reflects a genuine deployment risk for their specific use case.

The FLOPs-Matched Comparison Against Larger Models Is Fundamentally Absent

The assumption or constraint. The paper makes strong efficiency claims: Step 3.5 Flash achieves performance "on par with frontier models such as GPT-5.2 xHigh and Gemini 3.0 Pro" despite activating only 11B parameters — an order of magnitude fewer than the estimated active parameters of these closed-source models. However, the paper provides no FLOPs-matched or latency-matched comparison against any larger model. The headline Table 5 compares raw accuracy without controlling for inference compute. The PaCoRe column demonstrates that additional inference compute (16× budget) improves Step 3.5 Flash's performance, but the paper does not report what performance baseline models would achieve with comparable test-time compute budgets. If GPT-5.2 xHigh or Gemini 3.0 Pro were allowed best-of-16 or a comparable parallel reasoning strategy, their scores might also improve, potentially maintaining or widening the gap. The paper's efficiency claims (11B active parameters, ~170 tokens/s) are about cost per token, not cost per correct answer, and the evaluation does not provide the latter.

The consequence. The paper's central marketing claim — "redefining the efficiency frontier" (title text in Figure 1) — is supported by parameter-count comparisons but not by total-inference-compute comparisons. This is a significant omission because the prior literature the paper cites (specifically Snell et al., 2025, reference [45]) established the methodology for FLOPs-matched comparisons between test-time compute and model scale, demonstrating that smaller models with compute-optimal test-time strategies can outperform larger models in certain regimes. The paper positions itself within this trajectory but does not conduct the analogous experiment. Without it, the efficiency claims rely on an implicit assumption that per-token cost is the relevant efficiency metric, but Section 2.1 explicitly argues that wall-clock time to task completion is the metric that matters for agents. The absence of FLOPs-matched or latency-matched comparisons means the paper has not demonstrated that the 11B-active model + test-time compute strategy is more efficient than a larger model with a simpler strategy at achieving a target accuracy.

The practical implication for practitioners: when deciding whether to deploy Step 3.5 Flash versus a larger model on the same hardware, they need to compare total cost (tokens × cost-per-token) at equivalent accuracy. Step 3.5 Flash's lower cost-per-token is partially offset by its tendency to generate longer trajectories (Limitation 1) and by the overhead of PaCoRe when deployed. Without FLOPs-matched baselines, the paper provides no basis for this comparison. A larger model running best-of-1 might achieve comparable accuracy to Step 3.5 Flash running PaCoRe at lower total inference cost, but the paper provides no data to evaluate this hypothesis.

What evidence exists in the paper. The paper provides no FLOPs-matched or latency-matched comparison. The only head-to-head efficiency comparison is the parameter-count comparison in Table 5 (11B active vs. 32–37B active for open-source baselines, unknown for closed-source). The PaCoRe results in Table 5 provide a within-model test-time scaling curve, but the baseline models are evaluated at their default inference budgets (typically pass@1 or pass@k with single trajectories), not at comparable total inference compute. The paper's architecture ablations (Tables 1, 8) measure relative FLOPs (decode/prefill FLOPs normalized to the S3F1 configuration) but these are per-token FLOPs, not total-task FLOPs, and do not account for differences in generation length or test-time strategies. The paper does report efficiency metrics for the training infrastructure (communication optimization reduces iteration time by ~5%, Muon ZeRO-1 resharding reduces iteration time by ~5%), but these are training-time efficiencies, not inference-time comparisons against baseline models.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation and does not suggest future work on FLOPs-matched inference comparisons. Given the paper's design philosophy centered on efficiency and the explicit citation of the compute-optimal test-time scaling literature, this omission is notable. A future version of this work should include experiments where Step 3.5 Flash (with and without PaCoRe) is compared to a larger model (e.g., DeepSeek V3.2 or Kimi K2.5) under equalized inference budgets across a range of accuracy targets, following the methodology established by prior work.

The Post-Training RL Improvements Cannot Be Attributed to MIS-PO Specifically — the Causal Chain Is Untraced

The assumption or constraint. The paper presents MIS-PO as the primary algorithmic contribution in post-training and provides extensive training-dynamics comparisons against PPO (Figure 5) and GSPO (Figure 7) showing improved stability and efficiency. However, the downstream benchmark improvements attributed to RL (Figure 6: +3.2% on IMO-AnswerBench, +6.1% on CF-Div2, +10.6% on ARC-AGI-1, +3.4% on HLEtext) represent the total improvement from the full RL phase, which includes: (1) two-stage SFT on 871K curated samples, (2) domain-specific RL expert construction with specialized reward models, (3) self-distillation consolidating expert knowledge into a student model, (4) the GenRM reward model with MetaRM augmentation, (5) the model-based verifier for STEM tasks (reported to improve by +2.0% over vanilla math-verify), (6) truncation-aware value bootstrapping, (7) the curated RL prompt dataset, and (8) MIS-PO itself. The paper provides no controlled ablation isolating MIS-PO's contribution to final downstream accuracy by, for example, training equivalent models with PPO or GSPO under otherwise identical data, initialization, and reward signals.

The consequence. The paper's claim that MIS-PO "enables reliable off-policy RL on long-horizon reasoning tasks for MoE architectures" is a claim about training stability and efficiency, not about final model capability. The training-dynamics evidence (Figures 5, 7, 8) supports the stability claim. But the downstream benchmark improvements are the metric that matters for practitioners, and the paper provides no evidence that these improvements would not have been achieved with another stable RL method (GSPO with better hyperparameters? PPO with router replay?) or that the improvements are primarily driven by the RL algorithm rather than the data, rewards, and initialization quality. The +10.6% gain on ARC-AGI-1 is particularly striking and might reflect data curation (the model seeing similar reasoning patterns during post-training) rather than algorithmic innovation. Without a controlled ablation, a practitioner adopting Step 3.5 Flash's post-training recipe cannot determine which components are essential and which are incidental — do they need MIS-PO, or would a simpler method work if the data and rewards are high-quality?

What evidence exists in the paper. The paper provides three types of evidence for MIS-PO, none of which address the causal attribution problem for final benchmark scores:

Training-dynamics comparisons (Figures 5, 7, 8) — show MIS-PO achieves higher reward, lower gradient norm, and more stable training than PPO and GSPO on an internal model. These are process metrics, not outcome metrics. They demonstrate that MIS-PO stabilizes the training process, but not that the stabilized process produces a better final model than alternative stabilization strategies.
Ablation components mentioned in text — MetaRM improves GenRM by 0.5–3% over 200 steps, model-based verifier improves STEM tasks by +2.0% over 450 steps. These are mentioned without supporting tables or figures, and their combined contribution is not decomposed. If these components contribute +2–3% each, they could account for most of the +3.2–10.6% total RL improvement, leaving little room for MIS-PO-specific gains.
Training reward curves (Figure 6, left) — show steady reward increase from ~0.60 to ~0.70 over ~250 steps. This demonstrates learning occurred, but does not isolate MIS-PO's role.

A controlled experiment — training two identical model checkpoints from the same post-SFT initialization with the same data, reward models, and hyperparameters, but varying only the RL algorithm (MIS-PO vs. GSPO vs. PPO) and comparing final downstream accuracy — is the standard way to establish algorithmic contribution, and the paper does not include it.

Mitigation status. Partially addressed through the training-dynamics ablations in Appendix D.2.3 and the extended MoE training run in Figure 8, which demonstrate that MIS-PO maintains stability where alternatives degrade during training. The paper implicitly argues that stability is the bottleneck — without MIS-PO, the RL training would collapse or plateau, precluding any downstream improvements. This is a plausible argument, and the GSPO comparison on MoE models (Figure 7b, showing GSPO plateaus around iteration 200) provides some evidence. However, it is not a direct test: the paper does not show that GSPO, if allowed to continue training with the plateaued reward, would fail to achieve the same downstream improvements (plateaued reward could still produce a better policy through entropy reduction or other mechanisms). The paper also does not explore whether alternative stabilization strategies — router replay, stricter trust-region constraints, on-policy corrections — could achieve similar stability to MIS-PO without the 80–92% sample rejection rate. A practitioner choosing an RL method for their own MoE model would want to know whether MIS-PO's aggressive filtering is necessary or whether gentler alternatives could work with proper tuning.

Hard Problems Remain Essentially Unsolved — The Model Cannot Create Capability It Lacks

The assumption or constraint. Step 3.5 Flash's strongest results are on benchmarks where the model receives formatted problems with clear correctness criteria and where test-time compute (PaCoRe, tool use) can convert additional inference budget into higher accuracy. On benchmarks testing broad, deep knowledge and open-ended reasoning — HLEtext (23.1%), GPQA-Diamond (83.5% vs. 92.4% for GPT-5.2 xHigh), MultiChallenge (55.7% vs. 71.9% for GPT-5.2 xHigh), Arena-Hard v2.0 (74.0% vs. 80.6% for GPT-5.2 xHigh) — the gaps to frontier closed-source models range from 6.6 to 16.2 percentage points. These are not marginal differences; they represent qualitatively different levels of capability. The paper does not analyze whether these gaps are limited by the 11B-active-parameter architecture (insufficient knowledge capacity), by pre-training data (insufficient coverage of the tested knowledge domains), or by post-training (insufficient RL on these task types). But the pattern is consistent: on problems requiring broad factual knowledge and nuanced judgment, the smaller model cannot match larger ones, even with test-time compute amplification.

The consequence. This limitation defines a clear deployment boundary for Step 3.5 Flash. The model excels at structured reasoning with definite answers (math competitions, competitive programming, software engineering with verifiable tests, web research with fact-checkable outputs) — tasks where correctness can be verified and where test-time compute strategies (tool use, parallel reasoning, iterative refinement) can bridge capability gaps. It is substantially weaker at open-ended reasoning, instruction following, and knowledge-intensive judgment — tasks where there is no verifiable ground truth, where quality is assessed by human preference or multi-dimensional rubrics, and where test-time compute cannot directly verify correctness. For a practitioner building a coding agent, SWE-bench agent, or math tutoring system, Step 3.5 Flash may be an excellent choice. For a practitioner building a general-purpose assistant that must handle arbitrary user queries with nuanced, context-aware responses, the 6.6–16.2 point gaps on general benchmarks may be unacceptable, especially given that closed-source alternatives with higher per-token costs may still be cheaper in total if they require fewer generation attempts to produce a satisfactory answer.

The paper's own pre-training results (Table 4) provide suggestive evidence about the source of this limitation. On knowledge-intensive benchmarks: SimpleQA at 31.6% (vs. 35.3% for Kimi-K2 Base at 1043B total parameters), MMLU at 85.8% (vs. 87.8% for Kimi-K2 Base), GPQA at 41.7% (vs. 43.5% for MiMo-V2 Flash Base at 309B total). These gaps are smaller but present even in the base model, suggesting that some of the post-training performance gap on knowledge-intensive tasks may be architectural — the model simply has less total knowledge capacity than models with 3–5× more total parameters, and post-training cannot create knowledge that wasn't acquired during pre-training. The paper's contrast between the model's 97.3% on AIME 2025 (a reasoning task that can be solved through step-by-step deduction) and 23.1% on HLEtext (which requires broad, deep factual knowledge across many domains) illustrates this boundary clearly. Test-time compute amplifies reasoning; it does not fabricate knowledge.

What evidence exists in the paper. Table 5 provides the evidence, broken out by benchmark category. The pattern is visible across multiple benchmarks:

Structured reasoning: AIME 2025 (97.3%, vs. GPT-5.2 xHigh 100.0%, gap: -2.7), IMO-AnswerBench (85.4%, vs. 86.3%, gap: -0.9), LiveCodeBench-v6 (86.4%, vs. 87.7%, gap: -1.3). Gaps of 0.9–2.7 points on tasks with verifiable answers.
Open-ended reasoning and instruction following: HLEtext (23.1%, vs. Gemini 3.0 Pro 37.7%, gap: -14.6), GPQA-Diamond (83.5%, vs. GPT-5.2 xHigh 92.4%, gap: -8.9), MultiChallenge (55.7%, vs. GPT-5.2 xHigh 71.9%, gap: -16.2), Arena-Hard v2.0 (74.0%, vs. Gemini 3.0 Pro 85.8%, gap: -11.8). Gaps of 8.9–16.2 points on tasks with subjective or multi-dimensional quality criteria.

PaCoRe closes some of these gaps (Arena-Hard: 74.0 → 93.1%, MultiChallenge: 55.7 → 60.8%, HLEtext: 23.1 → 27.9%), but substantial gaps remain on HLEtext (27.9% vs. 37.7% for Gemini) and MultiChallenge (60.8% vs. 71.9% for GPT-5.2 xHigh), and PaCoRe comes at ~16× the inference budget.

Mitigation status. The paper partially acknowledges this limitation in Section 7: "Step 3.5 Flash is tailored for coding and work-centric tasks, but may experience reduced stability during distribution shifts. This typically occurs in highly specialized domains or long-horizon, multi-turn dialogues, where the model may exhibit repetitive reasoning, mixed-language outputs, or inconsistencies in time and identity awareness." However, this acknowledgment frames the limitation as a stability and distribution-shift issue, when the evidence suggests it is also a capability ceiling issue — the model simply scores substantially lower on open-ended knowledge-intensive benchmarks regardless of stability. The paper's stated future direction — "efficient universal mastery" through on-policy distillation — suggests awareness that closing the generalist gap requires further work, but no concrete plan or results are provided.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a stability-first paradigm for large-scale sparse MoE training that shifts the field's diagnostic attention from loss curves and routing statistics to activation-level signals previously invisible in standard monitoring pipelines. Prior work on MoE training (Switch Transformers, ST-MoE, GShard, DeepSeekMoE) treated stability primarily as a routing problem — ensuring balanced token dispatch, preventing expert starvation, and designing auxiliary losses. The paper demonstrates that routing balance is necessary but insufficient. Three distinct failure modes — localized activation blow-up, expert collapse with healthy routing, and Muon-induced numerical instability — can develop silently while the training loss remains smooth and unremarkable. The evidence in Figure 4 is particularly forceful: whether activation clipping, weight clipping, or no mitigation is used, the loss curves are indistinguishable, yet internally the maximum expert output norm in deep layers differs by orders of magnitude (stable at ~10^2 vs. exploding to ~10^5).

This diagnostic reframing changes what it means to "monitor training" for MoE models. The standard practice — watching the loss, gradient norms, and expert dispatch frequencies — is demonstrated to be blind to the most consequential failure modes. The paper's contribution is not just identifying these pathologies but providing the infrastructure (the lightweight asynchronous metrics server, Section 3.3) and the specific diagnostic metric (max-to-median ratio of per-expert activation norms, Section 4.1.3) that make them actionable. This is a concrete, adoptable blueprint: any team training a large sparse MoE should add per-expert activation norm monitoring at micro-batch granularity, and should track the max-to-median ratio specifically. The paper has effectively changed the monitoring baseline for the field.

The paper also resolves a methodological tension in how the LLM RL community handles off-policy data for MoE architectures. Prior to this work, the default approach — carried over from PPO's original continuous-control design — was importance sampling, with modifications to reduce variance (clipping, trajectory-level aggregation, trust regions). The paper's comparison against GSPO (Figure 7) reveals a previously unappreciated pathology: even when both token-level and trajectory-level binary masking are applied, importance-weighted updates within the accepted samples cause the training-inference density ratio π_θold / π_θvllm to drift progressively on MoE models, destabilizing optimization where MIS-PO's unit-weight treatment maintains it within a stable range. This is not merely an implementation detail — it suggests that importance weighting actively exacerbates the very distribution shift it's designed to correct in MoE models, likely because expert routing changes amplify the feedback loop between large weight updates and policy divergence.

This finding should shift the community's default: for MoE RL at scale, binary acceptance/rejection with unit weights (MIS-PO style) is a safer default than importance reweighting with any clipping scheme. The paper's evidence that MIS-PO discards 80–92% of trajectories yet achieves superior sample efficiency (Figure 5) and stability (Figures 7, 8) demonstrates that gradient variance reduction matters more than effective sample size for this setting. The practical implication is that RL infrastructure for MoE models should be designed around high-throughput rejection sampling rather than careful importance weight computation — a simpler engineering target that may generalize better.

The architectural finding on hybrid attention quality recovery resolves a previously pessimistic assumption. Prior work accepted that replacing full-attention layers with sliding-window attention at high ratios necessarily degraded model quality, and mitigated this either by using expensive 1:1 interleaving (Gemma 3, Command A) or by accepting quality loss. The paper's ablation (Table 1) demonstrates that two low-cost interventions — increasing SWA query heads from 64 to 96 and adding head-wise gated attention — together close most of the quality gap at the efficient 3:1 ratio with approximately 1-2% FLOPs overhead. This changes the design space for efficient Transformers: the quality-efficiency Pareto frontier is more favorable than previously believed, and the key ingredients are additional SWA query capacity and data-dependent gating rather than expensive full-attention layers. The mathematical equivalence established in Appendix A.1 — that head-wise gating acts as an input-dependent sink token — provides a principled explanation for why this combination works, which should accelerate adoption.

The paper's tool-use gain metric (Δtool, Table 11) provides a methodological improvement for agent evaluation. By reporting the improvement from enabling search tools rather than just absolute scores, the metric disentangles parametric knowledge from retrieval capability. This matters because high absolute scores on knowledge-intensive benchmarks like BrowseComp can reflect strong memorization rather than effective search strategies, and the Δtool metric makes this distinction explicit. The paper's finding that Step 3.5 Flash achieves the highest average Δtool (52.0) despite mid-range absolute scores on some benchmarks demonstrates that the metric captures something distinct from raw accuracy. This should influence how agent benchmarks are reported and compared.

Follow-Up Research This Work Enables

Controlled decomposition of MIS-PO's contribution to downstream accuracy versus other post-training components. The paper demonstrates that MIS-PO stabilizes training dynamics (Figures 5, 7, 8), but the +3.2% to +10.6% downstream benchmark improvements (Figure 6) represent the combined effect of SFT, self-distillation, reward model design (MetaRM, model-based STEM verifier), truncation-aware value bootstrapping, and RL algorithm choice. A controlled experiment that trains three identical post-SFT checkpoints with identical data, reward models, and hyperparameters, varying only the RL algorithm (MIS-PO vs. GSPO vs. PPO with router replay), and evaluates all three on the full benchmark suite from Table 5, would establish the causal contribution of MIS-PO to final capability. Without this, practitioners cannot determine whether the algorithm, the reward models (+2.0% from model-based verifiers, 0.5-3% from MetaRM), or the data curation is the primary driver of improvements. A strong follow-up would report full benchmark results for all three algorithmic conditions at matched training steps, with multiple seeds per condition.

Diagnosis and architectural mitigation of the MRCR-8needle retrieval gap. Step 3.5 Flash's MRCR-8needle AUC of 28.8% is far below frontier models (GPT-5.2 xHigh: 88.2%) and below other sparse models (Kimi K2.5: 36.5%). The paper provides no analysis of whether this gap is caused by (a) the 512-token SWA window preventing needle access for 75% of layers, (b) insufficient long-context needle-retrieval training data during mid-training, (c) the YaRN scaling factor of 2.0 degrading positional embeddings at 128K, or (d) the GQA-8 key-value head configuration limiting retrieval precision. A systematic ablation would vary the SWA window size (512 vs. 1024 vs. 2048), the SWA-to-full ratio (3:1 vs. 2:1 vs. 1:1) at iso-FLOPs, the YaRN scaling factor, and the GQA head count, measuring MRCR-8needle at each configuration. A complementary experiment would add needle-retrieval-specific training data during mid-training and measure improvement. If the limitation is architectural (hypothesis a), this would indicate that SWA-based hybrid attention has a fundamental retrieval ceiling that cannot be fully trained away, which would be an important negative result for the efficient-attention community. If it's training-related (hypothesis b), a simple data remedy would be available. The paper's existing architecture ablation infrastructure (30B-A3B proxy, 100B-A10B scaled experiments) provides a natural testbed for this investigation.

Difficulty-adaptive inference-time compute allocation for PaCoRe deployments. The PaCoRe results (Table 5) demonstrate that uniform application of parallel coordinated reasoning (4 parallel trajectories × 4 rounds, ~16× inference budget) produces gains ranging from +1.6% (HMMT 2025 Feb., where vanilla is already 98.4%) to +19.1% (Arena-Hard). This variation suggests substantial gains from selectively applying PaCoRe only to problems where it helps. A natural extension would train a lightweight difficulty classifier — using the PRM's score distribution on a small number of initial samples, or fine-tuning a small model on the benchmark's problem text — that predicts whether PaCoRe will improve accuracy by more than a threshold. The classifier could then route easy problems to single-trajectory inference and hard ones to PaCoRe, potentially capturing most of the PaCoRe gain at a fraction of the inference cost. The experiment would report accuracy vs. total inference budget curves for uniform PaCoRe, uniform vanilla, and difficulty-adaptive routing, demonstrating the efficiency frontier achievable with smart allocation. This directly extends the compute-optimal test-time scaling framework that the paper cites (Snell et al., 2025, reference [45]) to the PaCoRe setting.

Verifier over-optimization analysis for MIS-PO's masking thresholds. MIS-PO discards 80-92% of trajectories based on probability ratio thresholds at the token level [0.5, 2] and trajectory level [0.996, 1.001]. The trajectory-level threshold is extraordinarily tight — only trajectories with geometric mean ratios within 0.5% of 1.0 are accepted. This suggests the method operates in a regime where most generated data is discarded, and the effective data distribution is heavily truncated. A systematic sweep of the trajectory-level bounds — e.g., [0.9, 1.1], [0.95, 1.05], [0.99, 1.01], [0.996, 1.001] — measuring both training stability (gradient norm, density ratio drift) and downstream benchmark accuracy after fixed training steps would characterize the stability-efficiency tradeoff. Wider bounds accept more data (better sample efficiency) but risk instability; narrower bounds are safer but may discard too much signal. Understanding where the phase transition occurs — at what bound width does training collapse for this architecture — would provide practical guidance for practitioners applying MIS-PO to their own models and would test whether the extremely tight bounds used in the paper are necessary or conservative.

Extension of activation-based MoE stability monitoring to other architectures and scales. The paper's diagnostic methodology — per-expert activation norm tracking, max-to-median ratio monitoring, micro-batch granularity logging via asynchronous metrics server — was developed and validated on a 196B-parameter, 288-expert MoE. Whether the same pathologies (localized activation blow-up, expert collapse with healthy routing) occur at different scales (e.g., 50B or 500B total parameters), with different expert granularities (e.g., 64 large experts vs. 288 small ones), or with different activation functions (e.g., GELU instead of SwiGLU) is unknown. A study applying the same monitoring infrastructure to diverse MoE architectures during training — particularly the SwiGLU and pre-norm combination that the paper identifies as enabling the blow-up pathology — would determine whether these are universal MoE phenomena or specific to the paper's configuration. The finding that Muon amplifies the blow-up through persistent low-rank updates (Appendix B) suggests that the pathology's severity may depend on optimizer choice, making this study relevant for teams choosing between Muon and Adam for their MoE training.

Combining MIS-PO with revision-based self-improvement for agentic RL. The paper's post-training pipeline uses domain-specific RL experts followed by self-distillation into a unified model, but never combines the iterative revision paradigm (where the model conditions on its own previous incorrect outputs) with MIS-PO's filtered off-policy RL. The complementary strengths — revisions improve the proposal distribution (the model generates better candidates by learning from mistakes), MIS-PO improves the optimization process (stable gradient updates from filtered data) — suggest a combined approach could yield gains beyond either alone. A concrete experiment would use the revision model to generate multi-turn trajectories (initial answer → revision → revision → ...) and apply MIS-PO to filter and learn from these trajectories, with the revision history providing additional conditioning signal for the value function and policy. The experiment would compare: (a) MIS-PO on single-turn trajectories, (b) MIS-PO on revision trajectories with naive filtering, and (c) MIS-PO on revision trajectories with the masking adapted to handle the larger distribution shift from multi-turn generation. The ReST^EM negative result in Appendix K (revision model performance degrading under on-policy optimization) suggests that combining revisions with RL is non-trivial and that MIS-PO's stability properties may be particularly valuable here.

Practical Applications and Downstream Use Cases

On-device coding assistants with cloud-offloaded reasoning. Step 3.5 Flash's configuration — 196B total parameters fitting within a 128GB workstation memory budget, with 11B active for fast local inference — makes it uniquely suited for hybrid edge-cloud coding workflows. The architecture was explicitly designed to fit within the 128GB memory of "high-end workstations" (Section 2.1). In this scenario, the model runs locally on a developer's machine for standard code completion and explanation tasks, achieving ~170 tokens/s without network latency. For complex multi-file refactors or SWE-bench-style bug fixes, it can escalate to a cloud deployment with tool access and PaCoRe enabled. The paper's results support this tiered approach: SWE-Bench Verified at 74.4% demonstrates competitive software engineering capability, and Terminal-Bench 2.0 at 51.0% (with 88.6% of successful trajectories completing within 30 interactions) demonstrates efficient interactive debugging. The key practical benefit is that most coding interactions — estimated at 80-90% of developer time in industry surveys — are simple completions and queries that don't require frontier-level reasoning, and serving these locally eliminates cloud costs and latency entirely. Only the hard problems pay the cloud inference premium. The Step 3.5 Flash + Step-GUI result on AndroidDaily Hard (57.0% vs. 40.0% for edge-only, Appendix E.3.3) validates the broader edge-cloud collaboration paradigm beyond coding.

Automated software engineering pipelines with self-evolving environments. The paper's code agent data pipeline (Section 5.3.3) — which achieves 40% environment-building success rate over 15K GitHub repositories and 20+ programming languages — enables a production deployment where Step 3.5 Flash continuously expands its own testing environments. The bidirectional transfer finding (construction expertise accelerates coding, coding within constructed environments improves construction accuracy) means this is a virtuous cycle: as the model solves more SWE tasks, the environments become richer, and richer environments produce better training data for the next iteration. The 50K verified environments already curated provide a strong cold start. A practical deployment would run Step 3.5 Flash with the code agent infrastructure (Session-Router, Kubernetes orchestration, Tmux session persistence) as a continuous integration bot that: (1) identifies real GitHub issues (using the PR/Issue/Commit dataset construction methodology from Appendix C.2.2), (2) synthesizes an environment if one doesn't exist, (3) attempts a fix with 350-turn interaction budget, (4) validates against test cases, and (5) submits a PR if successful. The environment-building pipeline ensures the system doesn't stall on unfamiliar repositories. The paper's 74.4% SWE-Bench Verified rate suggests approximately 3 out of 4 issues would be resolved successfully, making this economically viable for organizations with large codebases and active issue trackers.

Cost-efficient deep research and report generation for consulting and legal workflows. Step 3.5 Flash's combination of strong search agent performance (GAIA: 84.5%, highest among all evaluated models; RESEARCHRUBRICS: 65.3%, also highest; BrowseComp with Context Management: 69.0%) and competitive consulting scores (70.5% on the internal Consulting and Recommendations Benchmark, tied with Gemini 3.0 Pro) makes it suitable for automated research report generation at scale. The key economic advantage is the 11B-active-parameter inference cost: a consulting firm or legal practice processing thousands of research queries monthly can achieve GPT-5.2-competitive quality (65.3 vs. 57.8 on RESEARCHRUBRICS, 84.5 vs. 83.5 on GAIA) at a fraction of the per-query inference cost, assuming the closed-source models use substantially more active parameters. The RESEARCHRUBRICS score specifically addresses structured report quality against expert-written rubrics, which directly maps to professional deliverables. The tool-use gain metric (Δtool, Table 11) provides additional confidence: the model's outsized improvement when search tools are enabled (average Δ = 52.0 vs. 29.7-40.2 for competitors) indicates genuine retrieval and synthesis capability rather than parametric memorization, which is crucial for research tasks where answers depend on external, up-to-date information rather than training data. A deployment would use the ReAct framework with the full search agent tool suite (batch_web_surfer, file, file_parser, shell, todo, tmux) described in Appendix E.2.3, with the 30-turn interaction limit and per-turn 16K token output limit used for evaluation.

When to Prefer This Method

The paper positions Step 3.5 Flash as a density-first alternative to larger sparse MoE models and closed-source frontier systems, with explicit architectural tradeoffs that define its deployment envelope:

Prefer Step 3.5 Flash when the workload is dominated by structured reasoning tasks with verifiable correctness (competitive programming, math competitions, software engineering with test suites) and inference latency or cost is a binding constraint. Evidence: 97.3% on AIME 2025 vs. 93.1-100.0% range for models activating 3-4× more parameters; 74.4% on SWE-Bench Verified vs. 73.1-80.9% range. The PaCoRe test-time scaling option provides a tunable accuracy-vs-latency knob for high-stakes queries.
Prefer Step 3.5 Flash when deploying on 8-GPU nodes (the GQA-8 design is explicitly aligned to this topology) or high-end workstations with 128GB memory. Evidence: architecture designed to fit within this memory budget (Section 2.1); sustained ~170 tokens/s on Hopper GPUs during first-week OpenRouter deployment. Larger models may not fit in memory or may require more complex model parallelism.
Prefer larger models (DeepSeek V3.2, Kimi K2.5) or closed-source systems (GPT-5.2 xHigh, Gemini 3.0 Pro) when the workload is dominated by open-ended instruction following, nuanced judgment, or broad knowledge recall without verifiable ground truth. Evidence: 8.9-16.2 point gaps on GPQA-Diamond, MultiChallenge, HLEtext, and Arena-Hard v2.0 (vanilla) versus GPT-5.2 xHigh and Gemini 3.0 Pro.
Prefer larger models when long-context needle retrieval is critical and cannot be offloaded to tools. Evidence: 28.8% on MRCR-8needle vs. 88.2% for GPT-5.2 xHigh, 73.0% for Claude Opus 4.5. This gap is large enough that PaCoRe (26.3%) does not close it.
Prefer MIS-PO over PPO or GSPO when conducting off-policy RL on MoE models with long-horizon trajectories, especially when training-inference infrastructure divergence is high (different hardware for rollouts vs. training). Evidence: Figures 5, 7, 8 show MIS-PO maintains stable gradient norms and controlled density ratios where PPO exhibits large spikes and GSPO exhibits progressive density-ratio drift on MoE models. The 80-92% trajectory rejection rate is acceptable when the inference engine can generate rollouts in parallel at high throughput.
Do not prefer MIS-PO when the training and inference policies are nearly identical (low infrastructure divergence), in which case the aggressive filtering discards useful data unnecessarily, or when sample generation is expensive (e.g., rollouts require paid API calls to external tools), in which case the low effective sample rate is economically prohibitive. The paper provides no evidence on these boundary conditions — these are logical extrapolations from the method's design.