Arcee Trinity Large Technical Report

🎯 Pitch

A 400B-parameter sparse MoE model with only 13B active parameters can match dense models 30× its size in benchmarks—and it trains to 17 trillion tokens with zero loss spikes, a feat typically rare at scale. The secret is SMEBU, a momentum-based load balancing trick that prevents expert collapse without the awkward sign flips of conventional routing.

1. Executive Summary

This technical report introduces the Trinity family of open-weight sparse Mixture-of-Experts language models—Trinity Nano (6B total, 1B activated), Trinity Mini (26B total, 3B activated), and Trinity Large (400B total, 13B activated)—trained on up to 17 trillion tokens using the Muon optimizer with zero loss spikes throughout. The architecture integrates interleaved local and global attention with gating, depth-scaled sandwich norm, and sigmoid routing, while Trinity Large additionally introduces Soft-clamped Momentum Expert Bias Updates (SMEBU) —a new MoE load balancing strategy that replaces per-step sign-based bias updates with a tanh-soft-clamped, momentum-smoothed formulation to enable convergence near local minima and prevent expert collapse. The models achieve strong benchmark performance: Trinity Large Base scores 82.58 on MMLU (5-shot) and 65.20 on Minerva MATH500 while maintaining extreme sparsity with roughly 4× fewer activated parameters than comparably performing dense models, establishing that carefully engineered sparse architectures with stable training recipes can match or exceed the capabilities of much larger active-parameter models in the open-weight regime.

2. Context and Motivation

The Core Problem: Scaling Model Capacity Without Scaling Inference Cost

The fundamental tension this paper grapples with is one that has defined the frontier of language model development over the past several years: how do you continue scaling model capability when the computational and economic costs of inference are becoming prohibitive? The paper addresses this from multiple angles simultaneously—architectural efficiency, training stability, data quality, and inference throughput—but the organizing problem is the scaling-versus-efficiency tradeoff.

To understand why this tension matters, consider the trajectory that brought the field here. The dominant paradigm for improving language model performance has been straightforward: increase the number of parameters, increase the amount of training data, and increase the amount of compute, following the predictable scaling relationships characterized by Kaplan et al. (2020) and refined by Hoffmann et al. (2022). This approach has been remarkably successful—the progression from GPT-2 (1.5B parameters) to GPT-3 (175B) to GPT-4 (reportedly well over 1T) produced qualitative leaps in capability that justified the exponentially increasing costs.

However, this scaling paradigm runs into a hard practical constraint: inference cost scales with total parameter count, not activated parameter count. A dense 400B-parameter model requires approximately 800GB of memory just to store the weights in FP16, far exceeding the memory capacity of any single GPU available today, and the per-token FLOPs cost is proportional to the full 400B parameters. This creates a deployment bottleneck that exists independently of training economics. Even if an organization can afford to train a trillion-parameter dense model, serving it at scale—with acceptable latency, throughput, and cost per query—is a fundamentally different challenge.

The paper articulates this deployment reality explicitly in the introduction (Section 1), noting that:

"there is also a growing need for inference-time efficiency as workflows and contexts over which the LLMs are required to operate on grow larger and larger."

This is not merely a cost concern. The paper identifies a second, application-driven pressure toward inference efficiency: the rise of reasoning models and long-context processing. As models like OpenAI's o1 (OpenAI et al., 2024) and DeepSeek-R1 (Guo et al., 2025) have demonstrated, allowing models to "think" via intermediate output for tens or even hundreds of thousands of tokens before returning a final answer substantially improves performance on complex reasoning tasks. The paper notes:

"These long output chains, often coupled with large inputs that the models need to reason over, further underscore the need for models to be fast and efficient at inference time."

This creates a compounding effect: reasoning models are most valuable precisely when they generate very long chains of intermediate computation, but those long chains are also the most expensive to serve. An architecture that can dramatically reduce per-token inference cost—by activating only a small fraction of total parameters—is therefore disproportionately valuable in the reasoning regime.

Why Sparse Mixture-of-Experts and Why Now

The paper positions Sparse Mixture-of-Experts (MoE) as the primary answer to this scaling-versus-efficiency dilemma. The basic idea, introduced by Shazeer et al. (2017), is to replace single dense feed-forward layers with multiple "expert" sub-networks, only a subset of which are activated for any given input token. This decouples total parameter count (which determines the model's capacity and expressiveness) from activated parameter count (which determines inference cost). A model can have 400B total parameters—providing the representational capacity of a very large dense model—while only activating 13B per token, making it as cheap to serve as a much smaller dense model.

The paper explicitly frames MoE adoption in the context of recent industry trends (Section 1):

"Sparse Mixture-of-Experts (MoE) models have emerged as a prominent way for companies to scale up their largest models while being much more efficient and economical to train."

They cite DeepSeek-V3 (DeepSeek-AI et al., 2025a), GLM-4.5 (GLM-4.5 Team et al., 2025), Kimi K2 (Kimi Team et al., 2025a), and MiMo-V2-Flash (Xiaomi LLM-Core Team et al., 2026) as evidence that this architectural choice has moved from research curiosity to industry standard for frontier models. The Trinity family is positioned as part of this wave, but with an emphasis on extreme sparsity—Trinity Large activates only 13B of its 400B parameters per token, yielding roughly a 30:1 ratio of total to active parameters. The paper explicitly contrasts this with other models: in the evaluation comparison (Figure 3), Trinity Large Base achieves competitive scores with GLM 4.5 Base "despite having 4× higher degree of sparsity and a roughly 2.5× lower active parameter count."

Where Existing MoE Approaches Fall Short

The paper identifies several specific limitations in prior MoE work that motivate its architectural and training innovations:

Training instability is the elephant in the room. While MoE models are well-motivated theoretically, their training has historically been plagued by stability issues. The paper is unusually candid about the difficulties encountered during Trinity Large development (Section 6):

"On our first few runs, the loss decreased as expected and expert utilization looked roughly balanced early on. After some progress, routing behavior drifted and expert load became increasingly uneven, eventually resulting in collapsed experts."

This is not a Trinity-specific problem. The MoE routing mechanism creates a self-reinforcing feedback loop: if certain experts start receiving more tokens, they get more gradient updates, become better at the tasks those tokens represent, and attract even more tokens on subsequent steps. Conversely, experts that receive fewer tokens receive less training signal, become worse, and become progressively less likely to be selected. This "rich get richer" dynamic can lead to expert collapse, where a small number of experts dominate and the rest become effectively dead weight, undermining the whole purpose of the MoE design.

Most prior approaches to this problem use auxiliary loss functions—explicit training objectives that penalize imbalance in the expert load distribution. However, as the paper notes in Section 2.3, auxiliary-loss-free methods (introduced by Wang et al., 2024a) are preferable because they avoid introducing gradient interference between the language modeling objective and the load balancing objective. The standard aux-loss-free approach updates expert biases using a per-step sign-based adjustment:

"Under the assumption that the ideal expert bias value is a fixed value, we note that the standard aux-free load balancing cannot precisely converge on that value, as each local update under the sign(·) operator is always ±λ. We hypothesize that this is a cause of instability for MoE training with the standard aux-loss-free objective."

The paper's key insight here is that this sign-based update is fundamentally incapable of settling. As the load imbalance approaches zero, the updates still oscillate with magnitude ±λ, preventing convergence to a stable configuration. The paper argues that this "oscillation near local minima" contributes to training instability, particularly as the number of experts increases (since "the per-layer norm of the bias step also increases"). This is the specific gap that SMEBU (Section 2.3) is designed to fill.

Attention mechanisms haven't been optimized for MoE efficiency. Prior work has explored attention modifications in isolation—grouped-query attention (GQA; Ainslie et al., 2023) to reduce KV-cache size, sliding window attention (Jiang et al., 2023; Gemma Team et al., 2024b) to reduce the quadratic cost of long sequences, and linear attention variants (Kimi Team et al., 2025b; MiniMax et al., 2025) to achieve sub-quadratic scaling. However, the paper argues that these modifications haven't been systematically combined with MoE architectures to maximize both training and inference efficiency. The specific combination in Trinity—3:1 interleaved local/global attention with GQA, QK-normalization, gated attention, and sliding window in local layers—is motivated by the observation (following Yang et al., 2025) that local and global attention layers learn complementary roles, with local layers handling position-sensitive pattern matching and global layers handling long-range information integration.

Training optimizers are leaving efficiency on the table. The vast majority of large language models are trained with AdamW (Loshchilov and Hutter, 2019) or variants thereof. The paper notes (Section 1) that the Muon optimizer (Jordan et al., 2024a) "enables a larger critical batch size and has higher sample efficiency than the widely used AdamW optimizer." This is significant because it means that, for the same amount of training compute, Muon can extract more learning per token—essentially compressing the training budget required to reach a given performance level. While Liu et al. (2025b) had demonstrated Muon's scalability for LLM training, Trinity represents one of the largest-scale deployments of Muon in a production model, validating its viability at the frontier scale.

Training data quality and quantity at scale remains an art. The paper identifies data curation as a critical differentiator. Prior work has demonstrated the importance of data quality (Blakeney et al., 2024; Hu et al., 2024) and the effectiveness of synthetic data generation (Maini et al., 2025), but producing 8 trillion tokens of high-quality synthetic data at scale presents infrastructure and engineering challenges that are rarely documented in the literature. The paper explicitly positions its data effort as one of the largest publicly documented synthetic data generation campaigns for pretraining:

"To our knowledge, this represents one of the largest publicly documented efforts in synthetic data generation for pretraining, demonstrating that carefully curated synthetic data can be produced at the multi-trillion token scale required for frontier model development."

Document packing introduces subtle but consequential batch correlation. The paper identifies a problem that, to its credit, is rarely acknowledged in training infrastructure discussions: on-the-fly document packing can create minibatch-level correlation when long documents span multiple consecutive batches. The argument (Section 3.2) is worth quoting in full because it articulates a non-obvious mechanism:

"We hypothesize that imbalanced minibatches have particular impact in at least two scenarios: (i) in limited-batchsize settings, and (ii) in regimes where the model becomes more data efficient per step. In both settings, informally, if we treat each unbalanced minibatch as an ideally-representative sample from its own target data generator, we can see that we effectively optimize a different target distribution per step."

The paper introduces the Random Sequential Document Buffer (RSDB) as a solution, and quantifies the improvement with the novel Batch Heterogeneity (BatchHet) metric, showing a 4.23× reduction in BatchHet and a 2.4× reduction in step-to-step loss variance when RSDB is enabled (Section 3.2). This is an infrastructure contribution that the paper ties directly to training stability, arguing that the observed loss variance reduction correlates with lower probability of training instability events.

The Open-Weight Imperative

Beyond the technical motivations, the paper articulates an important deployment-side motivation: open-weight models serve a distinct and growing need in enterprise and organizational settings. The paper states (Section 1):

"Deployment settings commonly require organizational and regulatory considerations, motivating models that can be audited, hosted, and adapted within environments completely owned by the organizations. In particular, enterprise deployments frequently require clarity about data provenance, licensing, and jurisdictional controls, and therefore benefit from open-weight foundations that can be owned and operated without reliance on opaque third-party checkpoints."

This is a substantive point, not mere positioning. As AI regulation evolves (the EU AI Act, emerging frameworks in the US and elsewhere), the ability to audit model training data, verify licensing compliance, and host models within controlled infrastructure is becoming a hard requirement for many deployment contexts. The Trinity family, with its open-weight release and public technical documentation, serves this segment of the market directly.

How This Paper Positions Itself

The paper positions Trinity not as a single breakthrough but as a systems-level demonstration: showing that careful integration of multiple architectural innovations—extreme MoE sparsity, interleaved local/global attention with gating, the Muon optimizer, SMEBU load balancing, and scalable synthetic data curation—can produce models that are competitive with much larger active-parameter models while maintaining training stability and inference efficiency.

The contributions are explicitly layered (Section 2 structure):

Architecture contributions: The specific combination and scale of MoE sparsity (30:1 total-to-active ratio), the particular attention configuration (3:1 local/global with gated attention), and the depth-scaled sandwich norm initialization constitute a recipe that others can adopt or adapt.
Stability contributions: SMEBU is presented as a novel load balancing method with a principled motivation (enabling convergence near local minima through continuous, momentum-smoothed updates rather than discrete sign-based steps). The RSDB is presented as an infrastructure innovation that reduces batch-level noise.
Training recipe contributions: The use of Muon at frontier scale, the learning rate adjustment rule (Equation 41), and the specific schedule (cosine decay to 1/10 peak LR, then continued decay through context extension without rewarming) constitute a validated recipe.
Transparency contribution: The paper provides unusual detail on failures and fixes (Section 6 is essentially an honest post-mortem of early training instability), giving the community insight into real-world training dynamics that are typically absent from polished technical reports.

The paper does not claim to outperform all models on all benchmarks. Instead, it claims to demonstrate that extreme sparsity—when paired with the right architectural and training choices—is a viable path to frontier model quality in the open-weight regime. The evaluation section (Section 5.1) benchmarks against comparable models rather than against the absolute state of the art, consistent with the positioning as an "open-weight foundation" rather than a capability-maximizing proprietary system.

In summary, the gap this paper addresses is the scaling-efficiency problem in language model deployment, and its answer is a systematic integration of architectural, optimization, and data innovations that enables extreme MoE sparsity to work reliably at the 400B-parameter, 13B-active scale. The paper's value proposition is not a single algorithmic breakthrough but rather the demonstration that this particular combination of techniques works together—and the detailed documentation of why each component matters and how they interact.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

What is being built: The Trinity family consists of three decoder-only transformer language models—Nano, Mini, and Large—that use a sparse Mixture-of-Experts (MoE) design to activate only a small fraction of their total parameters per input token, combined with a hybrid attention mechanism that alternates between position-aware local attention and position-agnostic global attention. The core problem this solves: How do you scale a model to 400B total parameters for capacity while keeping per-token inference cost equivalent to a 13B model—roughly 30× cheaper than the total parameter count would suggest? The solution's "shape" is an architectural integration exercise: combine (1) extreme MoE sparsity with a new load-balancing method that prevents expert collapse, (2) a custom tokenizer optimized for numerical and multilingual efficiency, (3) an interleaved local/global attention pattern that handles long contexts efficiently, and (4) the Muon optimizer for improved sample efficiency during training, all wrapped in a training recipe validated stepwise through the Nano → Mini → Large scaling ladder.

3.2 Big-Picture Architecture (Diagram in Words)

The model processes text through six major stages, each feeding into the next:

Tokenizer — Converts raw text into integer token IDs using a custom 200,000-token BPE vocabulary with multi-stage pretokenization (digit chunking, script isolation, word/punctuation splitting, byte-level fallback). This is the only component that sees raw text; everything downstream operates on token sequences.
Embedding layer — Maps each token ID to a $d_{\text{model}}$ -dimensional vector via a learned lookup table, then scales the output by $\sqrt{d_{\text{model}}}$ following Takase et al. (2025). Output: initial token representations with positional information to be added later via attention.
Transformer layers (56–60 stacked) — Each layer contains an attention sub-layer followed by a feed-forward sub-layer, connected via depth-scaled sandwich normalization. The attention sub-layer uses one of two patterns depending on position in the stack: local layers (3 out of every 4) use sliding window attention with RoPE positional embeddings, while global layers (1 out of every 4) use full causal attention with no positional embeddings (NoPE). Both use grouped-query attention (GQA), QK-normalization, and gated attention. The feed-forward sub-layer is either a dense SwiGLU MLP (in the first 2–6 layers) or a Mixture-of-Experts module with $N_r$ routed experts, $N_s$ shared experts, sigmoid routing, and auxiliary-loss-free load balancing (or SMEBU for Trinity Large).
Final normalization — An RMSNorm applied after the last transformer layer, before the output projection.
Language modeling head — A linear projection from $d_{\text{model}}$ to the vocabulary size (200,000), producing logits over the token vocabulary.
Cross-entropy loss with z-loss — During training, the cross-entropy between predicted logits and ground-truth next tokens is the primary objective, optionally augmented with a small z-loss term (weight $1 \times 10^{-6}$ for Trinity Large mid-training stabilization) that penalizes large logit magnitudes.

Information flows as: raw text → tokenizer → token IDs → embedding → [layer 1 attention → layer 1 FFN → normalization → layer 2 attention → ...] → final RMSNorm → LM head → logits → loss. During inference, logits are sampled or greedily decoded to produce output tokens one at a time, with the KV-cache storing attention keys/values to avoid recomputation.

3.3 Roadmap for the Deep Dive

The order below builds from pre-training infrastructure (data) through architecture (tokenizer → attention → MoE → normalization → initialization) to optimization (Muon, training hyperparameters, context extension), reflecting the actual pipeline that an input sees and the training concerns that ensure that pipeline works stably:

First, the data pipeline and RSDB (Section 3.4.1): Since training data is the fuel and the RSDB is a novel contribution that addresses a subtle but consequential minibatch correlation problem—understanding this up front is essential because it affects the noise characteristics of every subsequent training step.
Second, the tokenizer (Section 3.4.2): This is the first transformation applied to raw text and determines the granularity at which the model sees language; the digit chunking and script isolation choices directly impact downstream capabilities (arithmetic, multilingual performance).
Third, the attention mechanism (Section 3.4.3): Explaining how local/global interleaving, GQA, QK-norm, and gating interact—and why this specific combination was chosen—is foundational because attention is the primary computational bottleneck and long-context enabler.
Fourth, the Mixture-of-Experts layer (Section 3.4.4): The core efficiency mechanism; this covers routing, load balancing, and the novel SMEBU method introduced for Trinity Large.
Fifth, normalization and initialization (Section 3.4.5): The "glue" that holds the architecture together and ensures gradients flow stably; the depth-scaled sandwich norm and the embedding multiplier are small details with outsized impact on training dynamics.
Sixth, the Muon optimizer and training hyperparameters (Section 3.4.6): Covers why Muon was chosen over AdamW for hidden layers, the learning rate adjustment rule, and the specific schedules used for each model size, including how context extension was integrated into the learning rate decay.
Seventh, context extension (Section 3.4.7): Explains the counterintuitive finding that extending only global attention layers (not local) works better, the training-at-longer-than-target-length strategy, and the rapid iterative evaluation approach using MK-NIAH.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems-building paper whose core idea is that extreme MoE sparsity, when paired with careful attention design, a novel stable load-balancing method, the right optimizer, and massive-scale synthetic data, can produce open-weight models competitive with much larger active-parameter dense models. The contribution is not a single algorithm but the validated integration of multiple architectural and training choices at the 400B-parameter, 13B-active scale, with unusual transparency about failures and the specific interventions that resolved them.

3.4.1 Data Pipeline and the Random Sequential Document Buffer (RSDB)

The Trinity models were trained on data mixtures curated by DatologyAI, with two distinct mixes: a 10 trillion token mix for Nano and Mini (distributed as 7T phase 1, 1.8T phase 2, 1.2T phase 3) and a 20 trillion token mix for Large (distributed as 13T phase 1, 4T phase 2, 3T phase 3), from which 17 trillion tokens were sampled proportionally. Each mix shifts toward higher-quality and domain-specific data in later phases, boosting the proportion of math and code data in phases 2 and 3, following the mid-training data annealing recipe established by Blakeney et al. (2024).

Synthetic data at scale. A key differentiator is that over 8 trillion tokens of the training data are synthetic—the paper claims this as one of the largest publicly documented synthetic pretraining data efforts. The synthetic web data (approximately 6.5T tokens) was generated using rephrasing approaches building on BeyondWeb (Maini et al., 2025): high-quality seed documents were selected from web corpora, then rephrased using diverse strategies including format transformation (e.g., converting web content into question-answer pairs), style modification (e.g., enhancing pedagogical tone), and content restructuring to improve information density. Approximately 1T tokens of synthetic multilingual data were generated similarly. Approximately 800B tokens of synthetic code data were generated by selecting and rephrasing high-quality code files with strategies maximizing diversity in task and style. The DatologyAI curation stack uses Ray and vLLM on Kubernetes for scalable generation.

Mid-training data annealing. The paper details that "both shifting the data mix away from general web to more code, math, and science over the course of the phases as well as shifting to higher quality and more relevant data within these categories" was applied, following the general principle that exposing models to higher-quality, more-specialized data in later training stages improves downstream performance on those domains.

On-the-fly tokenization and the inter-batch correlation problem. Documents were tokenized on the fly and sequence-packed to construct training sequences, with multiple dataloader workers interleaving documents across batches. For Nano and Mini, no intra-document attention masking was used. For Large, intra-document attention masking was applied (in Phase 1, as part of the stabilization fixes described in Section 6). The paper observed that on-the-fly tokenization with standard sequential packing can cause long documents to dominate consecutive batches when document lengths follow a lognormal distribution, introducing "unbalanced domain biases at the minibatch level." The mechanism is:

"informally, if we treat each unbalanced minibatch as an ideally-representative sample from its own target data generator, we can see that we effectively optimize a different target distribution per step. This not only causes additional overhead, as the network must 'unlearn' the small domain imbalances every step, but also introduces a source of long-tail noise due to the lognormal distribution of sequence lengths."

The paper hypothesizes that this effect is more severe: (i) in limited-batchsize settings, and (ii) as models become more data-efficient per step (which tends to increase with model size, following Kaplan et al., 2020), because larger models are more sensitive to step-to-step fluctuations in the data distribution. They further hypothesize that this is "one potential contributor to training instability as models grow larger."

Random Sequential Document Buffer (RSDB). To address this, the paper introduces the RSDB, which was deployed during Phase 3 of Trinity Large training. The mechanism operates as follows:

After tokenizing a document, load the entire token sequence as an entry in the RSDB and initialize a read head at index 0.
Continue tokenizing and inserting documents until the RSDB is full.
When populating a single sequence buffer for a minibatch, randomly sample a document index from the RSDB and read tokens from that document at the current read head position into the sequence buffer until either the sequence buffer is full or the document is exhausted.
If the document is exhausted, randomly select another document index and continue reading.
Update read head positions after each read.
For runtime efficiency, maintain an internal buffer size of twice the user-specified value, and refill to this larger value as soon as the buffer drops to the user-specified value. Purge old documents and load new ones in bulk during refill steps.

The paper argues this addresses the inter-batch correlation in several ways: it reduces fragmentation by not pre-chunking documents, retains truly random selection across documents to better respect IID sampling, and is memory-efficient enough to not bottleneck training. For Phase 3 of Trinity Large, the RSDB was configured with a buffer size of 8192 per GPU (user-specified 4096) with 4 workers per GPU, splitting the RSDB evenly across workers.

Batch Heterogeneity (BatchHet) metric. The paper introduces a quantitative metric to measure batch imbalance:

$\text{BatchHet}_t = \max_i L^{(t)}_i - \frac{1}{M} \sum_{i=1}^{M} L^{(t)}_i$

where $L^{(t)}_i$ is the loss of microbatch $i$ at training step $t$ , and $M$ is the number of microbatches per step.

What it computes: the difference between the worst-case microbatch loss and the average microbatch loss at each training step. A high BatchHet indicates that at least one microbatch is significantly harder than the others, which the paper claims correlates with gradient norm instability.

Why this form: it captures per-step imbalance as a single scalar that can be monitored during training. The paper does not claim it is the only possible metric, but notes that "in small scale internal experiments, we observe that this metric correlates strongly with gradient norm instability."

The paper reports that enabling RSDB reduced BatchHet by 46× (more precisely, a factor of 4.23× reduction in Section 3.2, though the small-scale experiments report a 46% reduction which appears to be a different baseline comparison, not the reduction factor) and improved loss over a sequential packing baseline. The small-scale experiments found that the RSDB-enabled network had gradient norm kurtosis of 14.6 compared to 187 for the baseline. Matching the step-to-step loss variance of the RSDB-enabled network required roughly a 2× increase in batch size for the baseline; matching the BatchHet required a 7× batch size increase. The paper emphasizes "these improvements happen without dropping any tokens." When integrated into Phase 3 of Trinity Large, the RSDB reduced BatchHet by a factor of 4.23× and step-to-step loss variance by a factor of 2.4×, visible in Figure 1's training loss curve.

3.4.2 Tokenizer Design

Vocabulary training. The tokenizer uses a custom 200,000-token BPE vocabulary trained with the tokenizers library on approximately 48GB (~10B tokens) of data drawn primarily from the Nano/Mini pretraining corpus, supplemented with multilingual text from C4 non-English splits and a mix of instruction-following, reasoning-trace, and code data. The larger multilingual corpus for Trinity Large was not finalized at tokenizer training time, so non-English languages are less well-represented than in the final pretraining data.

Vocabulary size selection. The paper reports an empirical comparison across vocabulary sizes. Because BPE merges are deterministic and greedy, truncating a larger vocabulary by merge rank yields identical tokenization to training at the smaller size, so they trained the full 200k vocabulary and derived smaller variants (including 131k) for ablation. Fertility measurements and small-scale loss curves confirmed consistent gains from the larger vocabulary, with "the most pronounced improvements in CJK languages and French—languages most constrained by smaller vocabularies."

The paper also evaluated SuperBPE (Liu et al., 2025a), which trains an initial vocabulary with standard whitespace splitting, truncates it, then resumes training without the whitespace constraint to learn multi-word tokens. SuperBPE achieved substantially better compression (~29% fewer tokens on English, ~27% fewer tokens on reasoning traces) but did not produce a corresponding improvement in downstream model performance at the experimental scale, so standard BPE was adopted.

Pretokenization pipeline. The pretokenizer uses a multi-stage pipeline with several notable features:

Digit isolation and place-aligned chunking. Contiguous digit runs are separated from surrounding text, then split into right-aligned groups of three with a leading group of one to two digits (e.g., 1234567 → 1|234|567). This ensures each three-digit token consistently represents a fixed place value, following Singh and Strouse (2024)'s finding that place-aligned tokenization materially improves arithmetic. The paper initially adopted the zero-width lookahead regex (?=(\d{3})+(?!\d)) from Liu et al. (2025a), but discovered during early training that this regex "exhibits catastrophic backtracking on documents containing long uninterrupted digit strings, causing tokenization of individual samples to spike to minutes." The fix replaces it with a three-stage pipeline: cap digit runs at 510 characters, peel off a leading group of 1–2 digits, then chunk the remainder into exact groups of three. The two approaches produce identical splits on inputs up to 510 digits, so this was applied post-hoc without retraining.
Script-aware isolation. Extends DeepSeek V3's CJK isolation to additionally cover Thai, Lao, Khmer, Myanmar, and Korean hangul/jamo—scripts that, like CJK, lack whitespace-delimited word boundaries. Isolating these script runs ensures BPE learns merges within each script independently rather than forming cross-script tokens.
Word and punctuation splitting. Adopts the DeepSeek V3 main text regex directly, handling leading-space word attachment, punctuation-prefixed tokens (covering contractions, code sigils, and markup), and whitespace/newline normalization.
Byte-level fallback. Ensures full coverage of any byte sequence without unknown tokens.

Tokenizer efficiency. Table 1 reports bytes-per-token (B/T) and characters-per-token (C/T) across languages and domains compared to DeepSeek R1 (128k), Qwen 3 (152k), Llama 3 (128k), and GPT-OSS (200k). The Trinity tokenizer achieves the strongest compression among standard tokenizers on English (4.84 B/T on C4-en) and French (3.98 B/T). CJK compression trails DeepSeek V3 and Qwen 3, which the paper attributes to the training data timing constraint.

3.4.3 Attention Mechanism

The attention mechanism combines four modifications to standard Multi-Head Attention (MHA), chosen for a mix of training stability, inference efficiency, and long-context performance.

Grouped-Query Attention (GQA). Let $h_q$ be the number of query heads and $h_{kv}$ be the number of key/value heads, with $h_{kv} < h_q$ . Each query head $i$ is mapped to a key/value head index $j(i) = \lfloor i \cdot h_{kv} / h_q \rfloor$ , such that multiple query heads share the same KV head. This reduces KV-cache size by a factor of $h_q / h_{kv}$ relative to MHA while roughly matching performance. The paper's specific configurations: Nano uses $h_q=8$ , $h_{kv}=2$ (4× reduction); Mini uses $h_q=32$ , $h_{kv}=4$ (8× reduction); Large uses $h_q=48$ , $h_{kv}=8$ (6× reduction). All models use per-head dimension $d_h = 128$ , giving $d_{\text{model}} = h_q \times d_h$ .

QK-Normalization. Before scaled dot-product attention, RMSNorm is applied to queries and keys independently. Given the raw linear projections $\mathbf{q}'_{t,i} = \mathbf{W}^Q_i \mathbf{x}_t$ and $\mathbf{k}'_{t,j} = \mathbf{W}^K_j \mathbf{x}_t$ , the normalized versions are:

$\mathbf{q}_{t,i} = \text{RMSNorm}(\mathbf{q}'_{t,i})$

$\mathbf{k}_{t,j} = \text{RMSNorm}(\mathbf{k}'_{t,j})$

What it computes: each query and key vector is re-normalized to unit RMS before the dot-product attention logits are computed. This constrains the logit magnitudes from growing with the norm of the input representations.

Why this form: the paper explicitly states QK-Norm was adopted "primarily for training stability reasons, especially due to our use of the Muon optimizer." They cite Liu et al. (2025b) and Kimi Team et al. (2025a) as prior work identifying growing maximum attention logit values as more of a concern with Muon versus AdamW. GLM-4.5 Team et al. (2025) is cited for showing that QK-Norm "effectively stabilizes the range of maximum attention logit values through a full training run." Without QK-norm, the paper implies, loss spikes become more likely.

Local/Global attention pattern. All models use a 3:1 ratio of local to global attention layers. Local layers use sliding window attention (SWA) with RoPE positional embeddings, restricting each token to attend only to the $w$ most recent tokens. Global layers use full causal attention with NoPE (no positional embeddings), attending to all previous tokens. The pattern repeats for the full depth of the model, following Yang et al. (2025).

For local layers, the valid attention positions are:

$S_t^{\text{local}} = \{ s \mid \max(1, t - w + 1) \leq s \leq t \}$

For global layers:

$S_t^{\text{global}} = \{ s \mid 1 \leq s \leq t \}$

The window sizes: Nano $w = 2048$ (pre-training at sequence length 4096), Mini $w = 2048$ (pre-training at sequence length 4096), Large $w = 4096$ (pre-training at sequence length 8192). Notice the window is always half the pre-training sequence length.

Why local/global interleaving: the paper cites Yang et al. (2025) for the finding that local and global layers learn complementary roles—local layers handle position-sensitive pattern matching while global layers handle long-range information integration. The paper reports practical benefits: "the combination of this scheme and gated attention allowed the model to recover performance in a comparatively lower number of steps when training at longer sequence lengths, and also resulted in observed length extrapolation for Trinity Large." The length extrapolation claim is backed by the fact that Trinity Large, trained at 256K, achieved an MK-NIAH score of 0.976 at 512K without additional training at that length.

RoPE for local layers, NoPE for global layers. The queries and keys used in attention are:

$\hat{\mathbf{q}}_{t,i} = \begin{cases} \text{RoPE}(\mathbf{q}_{t,i}), & \text{local layer} \\ \mathbf{q}_{t,i}, & \text{global (NoPE) layer} \end{cases}$

$\hat{\mathbf{k}}_{t,j} = \begin{cases} \text{RoPE}(\mathbf{k}_{t,j}), & \text{local layer} \\ \mathbf{k}_{t,j}, & \text{global (NoPE) layer} \end{cases}$

Why RoPE only in local layers: the paper follows the finding in Yang et al. (2025) that local layers benefit from position encoding for their pattern-matching role, while global layers, responsible for long-range integration, do not need it. The choice to not use positional embeddings in global layers also makes context extension easier, as only local layers need RoPE base frequency adjustments.

Scaled dot-product attention with GQA. The attention weights for query head $i$ (using shared KV head $j(i)$ ) are:

$\alpha_{t,i,s} = \text{Softmax}_{s \in S_t} \left( \frac{\hat{\mathbf{q}}_{t,i}^\top \hat{\mathbf{k}}_{s,j(i)}}{\sqrt{d_h}} \right)$

$\mathbf{o}_{t,i}^{\text{sdpa}} = \sum_{s \in S_t} \alpha_{t,i,s} \mathbf{v}_{s,j(i)}$

Gated attention. Following Qiu et al. (2025), an elementwise gating is applied to the attention output before the output projection:

$\mathbf{g}_t = \sigma(\mathbf{W}^G \mathbf{x}_t)$

$\mathbf{g}_{t,i} = \text{split}_{h_q}(\mathbf{g}_t)_i$

$\tilde{\mathbf{o}}_{t,i} = \mathbf{o}_{t,i}^{\text{sdpa}} \odot \mathbf{g}_{t,i}$

$\mathbf{u}_t = \mathbf{W}^O [\tilde{\mathbf{o}}_{t,1}; \tilde{\mathbf{o}}_{t,2}; \ldots; \tilde{\mathbf{o}}_{t,h_q}]$

where $\sigma(\cdot)$ is the sigmoid function, $\text{split}_{h_q}(\cdot)$ partitions $\mathbf{g}_t \in \mathbb{R}^d$ into $h_q$ contiguous vectors of size $d_h$ , $\odot$ is elementwise multiplication, $\mathbf{W}^G \in \mathbb{R}^{d \times d}$ is the gate projection, and $\mathbf{W}^O \in \mathbb{R}^{d \times d}$ is the output projection.

What it computes: for each attention head, a learned gating scalar (between 0 and 1) is applied to each element of the attention output, allowing the model to selectively suppress or amplify individual attention outputs. The gate is computed from the same input $\mathbf{x}_t$ that feeds the query/key/value projections, making it context-dependent.

Why gated attention: Qiu et al. (2025) identified that gated attention reduces attention sinks (tokens that absorb excessive attention weight), reduces overly large activations, improves performance on evaluations, improves long-sequence generalization, and—critically for this paper—"appears to stabilize training and reduce the occurrence of loss spikes during the training process." This stabilization property is explicitly called out as a key motivation for adoption.

3.4.4 Mixture-of-Experts Layer

The MoE layers follow the DeepSeekMoE (Dai et al., 2024) design: fine-grained routed experts and an always-active shared expert, using SwiGLU activation. The paper makes several specific design choices to enable extreme sparsity and training stability.

When MoE kicks in. The first $k$ transformer layers use dense feed-forward layers instead of MoE. Nano and Mini use $k=2$ ; Large uses $k=6$ . The paper states dense layers are used "to stabilize early representations," and increasing $k$ from 3 to 6 was one of the stabilization interventions applied during Trinity Large training.

MoE formulation. For input $\mathbf{u}_t \in \mathbb{R}^d$ at token $t$ , the MoE output is:

$\mathbf{h}'_t = \mathbf{u}_t + \sum_{i=1}^{N_s} \text{FFN}^{(s)}_i(\mathbf{u}_t) + \sum_{i=1}^{N_r} g_{i,t} \text{FFN}^{(r)}_i(\mathbf{u}_t)$

where $\text{FFN}^{(s)}_i(\cdot)$ is the $i$ -th shared expert (always active), $\text{FFN}^{(r)}_i(\cdot)$ is the $i$ -th routed expert (only active if selected), $N_s$ is the number of shared experts, $N_r$ is the number of routed experts, and $g_{i,t} \in [0, 1]$ is the routing weight for expert $i$ at token $t$ .

What it computes: the $\mathbf{u}_t$ add inside the residual is critical—it means the MoE layer is a residual block where the input is always preserved, and the expert contributions are additive. Every token passes through all $N_s$ shared experts (guaranteeing some minimum computation) plus the top- $K_r$ selected routed experts (with their contributions weighted by the gating scores). Experts not in the Top-K have $g_{i,t} = 0$ .

Expert configurations. The paper uses dramatically different expert granularity across model sizes. The key tradeoff is between granularity (smaller experts, more of them, more activated per token) and throughput (larger experts, fewer of them, fewer activated per token):

Nano: $N_r = 128$ , $K_r = 8$ activated per token, expert size = 256, route scale = 2.826, 1 shared expert
Mini: $N_r = 128$ , $K_r = 8$ activated per token, expert size = 1024, route scale = 2.826, 1 shared expert
Large: $N_r = 256$ , $K_r = 4$ activated per token, expert size = 3072, route scale = 2.448, 1 shared expert

Why Large uses coarser experts: the paper states explicitly that Large opted "to activate 4 routed experts and make each expert larger, reducing the granularity, due to throughput requirements." Activating fewer, larger experts means fewer expert forward passes per token (4 rather than 8), which is more efficient for inference batching. The total sparsity (total-to-active parameter ratio) is much higher: approximately 30:1 for Large versus roughly 4:1 for Nano.

Sigmoid routing. Instead of the standard softmax-based routing (where all expert scores sum to 1, creating competition), the paper uses normalized sigmoid routing following Wang et al. (2024a). For expert $i$ with router vector $\mathbf{e}_i \in \mathbb{R}^d$ :

$s_{i,t} = \sigma(\mathbf{u}_t^\top \mathbf{e}_i), \quad i \in \{1, \ldots, N_r\}$

where $\sigma(\cdot)$ is the sigmoid function. The raw router scores $s_{i,t}$ are between 0 and 1 but do not sum to 1.

Why sigmoid over softmax: sigmoid-based scores are independent per expert—a token can have high affinity for multiple experts without one suppressing the others, which is claimed to "allow for more stable router logits." Softmax routing creates a zero-sum competition where making one expert more likely necessarily makes all others less likely, which can amplify small perturbations into large routing changes.

Top-K selection with decoupled expert bias. The actual Top-K selection uses the sum of the router score and a learned expert bias $b_i$ , but the gating weights applied to expert outputs use only the router scores:

$g'_{i,t} = \begin{cases} s_{i,t}, & \text{if } s_{i,t} + b_i \in \text{TopK}(\{s_{j,t} + b_j\}_{j=1}^{N_r}, K_r) \\ 0, & \text{otherwise} \end{cases}$

$g_{i,t} = \frac{g'_{i,t}}{\sum_{j=1}^{N_r} g'_{j,t}}$

What it computes: the expert bias $b_i$ influences which experts get selected (by adding to the score used for Top-K ranking), but does not affect the gating weight applied to the selected expert's output. The selected gating scores are L1-normalized to sum to 1 across the $K_r$ selected experts.

Why decoupled bias: this separation (following Wang et al., 2024a) means the load balancing mechanism can aggressively adjust biases to steer token routing without distorting the model's actual mixing weights. The biases are "updated in a decoupled way" (outside the main gradient flow), so they don't receive gradients from the language modeling loss—they are purely a load-balancing mechanism.

Auxiliary-loss-free load balancing (Nano and Mini). For Nano and Mini, the paper uses the standard auxiliary-loss-free formulation from Wang et al. (2024a) with re-centering. Let $n_i$ be the number of tokens routed to expert $i$ in the current training step, and $\bar{n}$ the mean load:

$\bar{n} = \frac{1}{N_r} \sum_{i=1}^{N_r} n_i$

$\Delta b_i = \gamma \cdot \text{sign}(\bar{n} - n_i)$

$b_i = b_i + \Delta b_i$

$b_i = b_i - \frac{1}{N_r} \sum_{j=1}^{N_r} b_j$

where $\gamma$ is the bias update speed (not explicitly specified in the paper for Nano/Mini).

What it computes: for each expert, if its load is below average ( $\bar{n} - n_i > 0$ ), the bias is increased by $+\gamma$ , making it more likely to be selected in subsequent steps. If above average, the bias is decreased by $-\gamma$ . After updating all biases, the mean bias is subtracted to keep the bias vector zero-centered, preventing drift.

Why re-centering matters: without re-centering, all biases could drift upward over time (a form of parameter drift), eventually saturating the routing and defeating the purpose of the aux-loss-free mechanism. Zero-centering means the biases represent relative, not absolute, preferences.

The fundamental problem with sign-based updates. The paper provides a key insight about why the standard aux-loss-free method can cause instability, especially at scale:

"Under the assumption that the ideal expert bias value is a fixed value, we note that the standard aux-free load balancing cannot precisely converge on that value, as each local update under the sign(·) operator is always ±λ. We hypothesize that this is a cause of instability for MoE training with the standard aux-loss-free objective."

The magnitude of each update is constant ( $\pm\gamma$ ) regardless of how far the bias is from its optimal value. As the system approaches balance, the updates still oscillate with the same magnitude, preventing convergence. Worse, as the number of experts increases, "the per-layer norm of the bias step also increases," amplifying the oscillation.

SMEBU: Soft-clamped Momentum Expert Bias Updates (Trinity Large). Trinity Large introduces SMEBU to address the convergence problem. The method replaces the discrete sign-based update with a continuous tanh-based update that can smoothly decay to zero near balance, plus momentum for noise dampening.

First, compute the normalized per-expert violation (independent of sequence length and batch size):

$v_i = \frac{\bar{n} - n_i}{\bar{n}}$

$\tilde{v}_i = \tanh(\kappa \cdot v_i)$

where $\kappa$ is a tunable scale controlling saturation speed. Trinity Large uses $\kappa = 2$ .

What it computes: $v_i$ represents the relative overload/underload of expert $i$ —a value of 0 means the expert is exactly at the mean load, positive means underloaded (needs more tokens), negative means overloaded (needs fewer tokens). The tanh soft-clamps this to $[-1, 1]$ , but unlike sign, it provides a continuous gradient: small imbalances produce proportionally small updates, and large imbalances saturate toward $\pm 1$ .

Why tanh and not linear: the paper reports that "preliminary tests using a linear, unclamped update quickly reduced MaxVio early on in training but resulted in instability later in training." The tanh provides bounded updates even for extreme imbalances, preventing the bias from being driven to large magnitudes that could destabilize the network.

Momentum smoothing. The per-step update is computed and then smoothed with momentum:

$\Delta b_i = \lambda \tilde{v}_i$

$\Delta b_i = \Delta b_i - \frac{1}{N_r} \sum_{j=1}^{N_r} \Delta b_j$

$m_i = \beta m_i + (1 - \beta) \Delta b_i$

$b_i = b_i + m_i$

where $\lambda = 5 \times 10^{-4}$ is the load-balance learning rate, $\beta = 0.5$ is the momentum factor, and $m_i$ is the maintained momentum buffer for expert $i$ . Trinity Large uses $\lambda = 5 \times 10^{-4}$ and $\beta = 0.5$ .

What it computes: the raw update $\Delta b_i$ is proportional to the normalized violation $\tilde{v}_i$ (not a constant $\pm\lambda$ ), then zero-centered. The momentum buffer $m_i$ is an exponential moving average of $\Delta b_i$ , with $\beta$ controlling how quickly old updates decay. The actual bias update $b_i = b_i + m_i$ uses the smoothed value.

Why momentum: the paper makes an analogy to SGD with momentum: "consistent with the hypothesis that behavior near convergence is one of the causes of instability in MoE aux-loss-free load balancing, and that noise near a local minima should be roughly decorrelated over time, we introduce momentum as a form of noise dampening." If the expert load naturally fluctuates around a well-balanced state, the per-step violations $\tilde{v}_i$ are zero-mean noise. Momentum averages this noise out, allowing the bias to settle rather than oscillate.

Why SMEBU as a whole: it enables three things the standard method cannot: (1) updates proportional to imbalance magnitude, allowing convergence; (2) bounded updates via tanh to prevent large bias drift; (3) noise dampening via momentum to reduce oscillation near the balanced state. The paper presents SMEBU as critical to Trinity Large's training stability (Section 6): "We adopted our new load balancing method SMEBU" was the first of six stabilization interventions applied together.

Sequence-wise auxiliary loss. In addition to aux-loss-free load balancing, the paper uses a complementary sequence-wise load balance loss following DeepSeek-V3 (DeepSeek-AI et al., 2025a):

$\mathcal{L}_{\text{Bal}} = \alpha \sum_{i=1}^{N_r} f_i P_i$

$f_i = \frac{N_r}{K_r T} \sum_{t=1}^{T} \mathbb{1}\left(s_{i,t} + b_i \in \text{TopK}(\{s_{j,t} + b_j\}_{j=1}^{N_r}, K_r)\right)$

$\tilde{s}_{i,t} = \frac{s_{i,t}}{\sum_{j=1}^{N_r} s_{j,t}}$

$P_i = \frac{1}{T} \sum_{t=1}^{T} \tilde{s}_{i,t}$

where $\alpha$ is a small coefficient, $T$ is the sequence length, $f_i$ is the fraction of tokens routed to expert $i$ , and $P_i$ is the average normalized router probability for expert $i$ . The paper states this was introduced with "a small weight ( $1 \times 10^{-4}$ )" as one of the stabilization interventions for Trinity Large.

What it computes: for each expert, the product $f_i P_i$ is minimized when the fraction of tokens routed to expert $i$ equals the average router probability for that expert—i.e., when routing decisions are consistent with router confidence. Summing over all experts penalizes concentrated routing. The sequence-level formulation means the loss is computed per sequence, promoting balance within each sequence rather than only across the batch.

Why sequence-wise: the paper follows DeepSeek-V3's rationale that within-sequence balance is important for training efficiency, as imbalanced sequence-level routing can cause some GPUs to process more tokens than others in model-parallel setups.

MaxVio monitoring metric. The paper uses MaxVio (from Wang et al., 2024a) as the primary expert balance monitoring metric:

$\text{MaxVio} = \max_i \frac{\text{Load}_i - \text{Load}}{\text{Load}}$

where $\text{Load}$ is the mean expert load and $\text{Load}_i$ is the load on expert $i$ . This measures the relative worst-case overload—a value of 0.5 means the most overloaded expert is handling 50% more tokens than the average expert. The paper reports that during initial unstable runs, "MaxVio would remain stable and then experience a sudden climb as the experts collapsed."

3.4.5 Normalization and Initialization

Depth-scaled sandwich norm. Each transformer sub-layer uses a simplified depth-scaled sandwich norm following Yin et al. (2025), Ding et al. (2021), and Kim et al. (2025). For layer $\ell$ with sub-layer module $\mathcal{M}_\ell(\cdot)$ (attention, FFN, or MoE), the computation is:

$\mathbf{y}_\ell = \mathbf{x}_\ell + \text{RMSNorm}^{(2)}_\ell \left( \mathcal{M}_\ell \left( \text{RMSNorm}^{(1)}_\ell(\mathbf{x}_\ell) \right) \right)$

What it computes: the input $\mathbf{x}_\ell$ is first normalized by $\text{RMSNorm}^{(1)}_\ell$ , then passed through the sub-layer module $\mathcal{M}_\ell$ , then normalized again by $\text{RMSNorm}^{(2)}_\ell$ , then added back to the original input. This is a "sandwich" because the sub-layer is wrapped between two normalizations—pre-norm ( $\text{RMSNorm}^{(1)}_\ell$ ) and post-norm ( $\text{RMSNorm}^{(2)}_\ell$ ).

Depth scaling. The second RMSNorm's gain parameter is initialized with depth-dependent scaling. Let $L$ be the total number of layers:

$\gamma\left(\text{RMSNorm}^{(1)}_\ell\right) = 1$

$\gamma\left(\text{RMSNorm}^{(2)}_\ell\right) = \frac{1}{\sqrt{L}}$

Why depth scaling: without it, the variance of the residual stream would grow with depth because each layer's output variance adds to the running sum. Scaling the post-norm gain by $1/\sqrt{L}$ roughly compensates for this, making the effective contribution of each layer $\mathcal{O}(1/\sqrt{L})$ rather than $\mathcal{O}(1)$ . This is a simple alternative to more complex initialization schemes that aim to control residual variance growth.

Final normalization. An RMSNorm is applied before the language modeling head:

$\mathbf{z} = \text{RMSNorm}_{\text{LM}}(\mathbf{h}_L)$

Initialization. All trainable parameters are initialized from a zero-mean truncated normal distribution with width-scaled standard deviation:

$\theta \sim \text{TruncNormal}(0, \sigma^2; [-3\sigma, 3\sigma])$

$\sigma = \frac{0.5}{\sqrt{d}}$

where $d$ is the model dimension. The paper notes this "roughly follows" Takase et al. (2025), which recommends $\sigma = \sqrt{2/(5d)}$ . The specific values: $\sigma = 0.016$ for Nano ( $d=1024$ ), $\sigma = 0.011$ for Mini ( $d=2048$ ), $\sigma = 0.009$ for Large ( $d=3072$ ). The paper notes this aligns with DeepSeek-V3's initialization scale $0.5 / \sqrt{7168} = 0.006$ .

Why truncated normal: truncation at $\pm 3\sigma$ prevents extreme initial values that could cause activation or gradient spikes early in training. This is a standard practice for large model initialization.

Why $\sigma = 0.5 / \sqrt{d}$ : this provides width-dependent scaling that keeps the initial output variance roughly constant regardless of model width, following standard neural network initialization theory. Without width scaling, wider models would have proportionally larger initial activations, increasing the risk of training instability.

Embedding multiplier. During the forward pass, the embedding layer's activations are scaled by $\sqrt{d}$ :

$\mathbf{e}_t = \sqrt{d} \cdot \mathbf{E}(\text{tok}_t)$

What it computes: each embedding vector is multiplied by $\sqrt{d}$ before being fed into the transformer. This means the initial token representations have scale $\mathcal{O}(\sqrt{d})$ rather than $\mathcal{O}(1)$ .

Why $\sqrt{d}$ : the paper follows Takase et al. (2025) and notes that both Grok-1 (xAI, 2024) and Grok-2 (xAI, 2025) have embedding_multiplier_scale set to $\sqrt{d}$ in their HuggingFace checkpoints, and the first two generations of Gemma models (Gemma Team et al., 2024a,b) refer to this multiplier as a "normalizer." The purpose is to compensate for the $1/\sqrt{d}$ scaling in the attention mechanism: if embeddings are $\mathcal{O}(1)$ , the dot products in the first attention layer would be $\mathcal{O}(1)$ , but the $1/\sqrt{d_h}$ scaling in attention assumes inputs are $\mathcal{O}(\sqrt{d})$ . This mismatch is corrected by the embedding multiplier.

3.4.6 The Muon Optimizer and Training Hyperparameters

Why Muon over AdamW. The paper uses the Muon optimizer (Jordan et al., 2024a) for all hidden layers (attention projections, FFN weights, MoE expert weights) and AdamW for the embedding and output layers only. The stated rationale is that Muon "enables a larger critical batch size and has higher sample efficiency than the widely used AdamW optimizer." The larger critical batch size means training can scale to more GPUs before hitting diminishing returns from data parallelism, which is crucial for the 2048-GPU Trinity Large run.

Learning rate adjustment rule. Unlike Liu et al. (2025b), who rescaled Muon update RMS to match AdamW update RMS, this paper uses a different approach following Jordan et al. (2024b):

$\text{lr}_{\text{adjusted}} = \text{lr} \sqrt{\max\left(1, \frac{\text{fan}_{\text{out}}}{\text{fan}_{\text{in}}}\right)}$

What it computes: the base learning rate $\text{lr}$ is multiplied by the square root of the maximum of 1 and the fan-out to fan-in ratio. For a weight matrix of shape $d_{\text{out}} \times d_{\text{in}}$ , $\text{fan}_{\text{in}} = d_{\text{in}}$ and $\text{fan}_{\text{out}} = d_{\text{out}}$ . When $d_{\text{out}} > d_{\text{in}}$ , the learning rate is increased by $\sqrt{d_{\text{out}} / d_{\text{in}}}$ ; otherwise it's unchanged.

Why this adjustment: the paper states that "empirically, we observe that this adjustment enables optimal learning rate transfer when scaling model width, for Muon." Without it, wider layers (where $d_{\text{out}} \gg d_{\text{in}}$ ) would effectively have a lower per-parameter learning rate relative to narrower layers, requiring manual re-tuning when scaling width.

Muon implementation details. The paper uses "a modification of the Muon implementation in the Dion repository (Ahn et al., 2025) for efficient distributed Muon." Two implementation optimizations are noted: (1) gradients for experts are orthogonalized in a batched fashion without flattening, and (2) this "simplifies the expert-parallel implementation, since gradients do not need to be gathered across the expert-parallel group."

Training hyperparameters for each model:

Trinity Nano:

Peak learning rates: $1.0 \times 10^{-3}$ (Muon), $3.0 \times 10^{-4}$ (AdamW)
Linear warmup: 2000 steps
Global batch size: 4096 (sequence length 4096), increased to 8192 when scaling from 256 to 512 GPUs to improve utilization
Linear decay to zero during decay stage
Context extension: linear re-warmup to $1/10$ peak LR, then linear decay to zero; trained to 256K for inference at 128K

Trinity Mini:

Peak learning rates: $1.0 \times 10^{-3}$ (Muon), $2.0 \times 10^{-4}$ (AdamW)
Linear warmup: 2000 steps
Global batch size: 4096, increased to 8192 when scaling from 64 to 512 GPUs
Linear decay to zero during decay stage
Context extension: linear re-warmup to $1/10$ peak LR, linear decay to zero; trained to 128K for inference at 128K

Trinity Large:

Peak learning rates: $8.0 \times 10^{-4}$ (Muon), $2.0 \times 10^{-4}$ (AdamW)
Linear warmup: 2000 steps
Global batch size: 12288 (sequence length 8192), increased to 16384 after 4.9T tokens "to improve throughput, roughly following MiniMax et al. (2025)"
Cosine decay to $1/10$ peak LR ( $8.0 \times 10^{-5}$ Muon, $2.0 \times 10^{-5}$ AdamW)
Context extension: continued cosine decay from $1/10$ to $1/20$ of peak LR ( $8.0 \times 10^{-5}$ to $4.0 \times 10^{-5}$ Muon; $2.0 \times 10^{-5}$ to $1.0 \times 10^{-5}$ AdamW); trained to 256K for inference at 512K

Why cosine decay for Large but linear for Nano/Mini: the paper doesn't explicitly justify this, but the cosine decay's smoother trajectory (ending at $1/10$ peak LR rather than zero) enables continuous decay through context extension without a re-warmup step. The paper states this explicitly: "which allows us to proceed into context extension without re-warming." For Nano and Mini, the linear decay reaches zero, requiring a separate re-warmup for context extension.

Z-loss for logit stabilization. Z-loss (Wortsman et al., 2023) penalizes large logit magnitudes. The paper initially planned a weight of $1 \times 10^{-4}$ for Trinity Large, but due to a training code bug it was not applied. When logit instability was observed mid-training, z-loss was introduced with weight $1 \times 10^{-6}$ because "larger values destabilized the network." The paper reports this "effectively stabilized the trend in maximum logit as well as mean logits."

3.4.7 Context Extension

Only extending global attention layers. The paper's key finding for context extension is that adjusting only global attention layers (keeping local layers at their pre-trained window size) results in "a much quicker loss recovery, aligning with the findings in Yang et al. (2025), and further suggesting that local and global layers learn complementary roles." This is practically significant because "not extending the window size of the local layers allows for more efficient inference."

Training at longer-than-target sequence lengths. Consistent with Gao et al. (2025) and NVIDIA et al. (2025a), the paper finds that training at a sequence length longer than the target context window improves performance at the target length. For Nano: training at 128K for a target of 128K yielded MK-NIAH@128K of 0.38; training at 256K for a target of 128K yielded 0.548. Iterating on data and hyperparameters yielded a final 0.864. For Mini: trained at 128K for target 128K, achieved 0.888. For Large: trained at 256K for target 256K, achieved 0.994 at 256K and—without additional training—0.976 at 512K and 0.42 at 1M.

Why training longer than target works: the paper implies that exposing the model to sequences longer than the deployment context forces it to learn more robust long-range attention patterns, which transfer back to the shorter deployment length.

Rapid iterative evaluation using MK-NIAH. Due to time and compute constraints, the paper used the Multi-Key Needle-in-a-Haystack task from RULER (Hsieh et al., 2024) at the target context length as a rapid proxy for context extension quality. This allowed rapid cycling: evaluate a checkpoint, decide what to adjust, and restart.

No benefit from progressive extension. The paper reports: "we did not find any improvements from progressively increasing the context window size as opposed to training directly at the longest sequence length, beginning from the final pre-trained checkpoint." This is a practical finding that simplifies the context extension recipe—jump directly to the final length rather than staging through intermediate lengths.

Long-context dataset composition. The context extension dataset comprises approximately 117B tokens across 35.6M documents, blending samples from the original pretraining distribution with targeted long-context sources. For the pretraining component, a length-biased sampling strategy scales probabilities with document length (from a 1% floor up to 90% for 128K-token documents). Additional sources include OCR-derived PDF data (olmOCR from Poznanski et al., 2025; FinePDF-edu from Kydlíček et al., 2025), ProLong datasets (Gao et al., 2025) regenerated at full sequence length, instruction-style data (FLAN, math, code), AutoMathText (Zhang et al., 2025), and curated arXiv/textbook sources from ProLong.

Trinity Large length extrapolation. The paper's most striking result: Trinity Large achieved MK-NIAH@512K of 0.976 despite only being trained at 256K, and MK-NIAH@1M of 0.42, "suggesting that it would be possible to push later versions of the model to have strong performance at a context window of 1M." The paper attributes this extrapolation ability to the interleaved local/global attention design, which handles position information differently in local versus global layers.

4. Key Insights and Innovations

Innovation 1: Diagnosing MoE Training Instability as a Convergence Problem, Not Just a Load-Balancing Problem

The paper's most conceptually distinctive contribution is a reframing of why Mixture-of-Experts training becomes unstable at scale. Prior work—and the standard auxiliary-loss-free load balancing method from Wang et al. (2024a)—treated expert collapse primarily as a load-balancing problem: the router needs a mechanism to steer tokens toward underutilized experts. The solution was to add a bias term that gets nudged up or down depending on whether an expert is under- or over-loaded. This framing implicitly assumes that if you can just keep the loads roughly balanced, the rest of training will take care of itself.

The Trinity paper identifies a subtler problem hiding inside this framework. The sign-based update rule Δb_i = γ · sign(n̄ − n_i) is fundamentally incapable of converging to a stable routing configuration. The update magnitude is always ±γ, regardless of how close the system is to balance. Near the optimal bias values, the updates oscillate with constant amplitude rather than decaying. The paper's diagnostic framing is worth quoting directly (Section 2.3):

"Under the assumption that the ideal expert bias value is a fixed value, we note that the standard aux-free load balancing cannot precisely converge on that value, as each local update under the sign(·) operator is always ±λ. We hypothesize that this is a cause of instability for MoE training with the standard aux-loss-free objective."

This reframes the problem from "how do we balance loads?" to "how do we make the balancing mechanism converge?" — a shift from a control problem to an optimization problem. The distinction matters because it suggests different solutions. If the issue is load balancing, you add stronger balancing signals. If the issue is convergence, you need the update rule itself to support settling to a fixed point.

The paper supports this reframing with two pieces of mechanistic reasoning that go beyond the standard collapse narrative. First, they note that "as the total number of experts increases, the per-layer norm of the bias step also increases" — the oscillation amplitude grows with model scale, making larger MoE models intrinsically less stable under sign-based updates. Second, they view the behavior near the balanced state through the lens of noisy optimization: "noise near a local minima should be roughly decorrelated over time," motivating momentum as variance reduction for the bias updates. This is an optimization-theoretic argument applied to what had been treated as a routing heuristic.

The significance of this reframing extends beyond Trinity itself. Most prior work on MoE stability focused on auxiliary loss design (Shazeer et al., 2017; DeepSeek-AI et al., 2025a; GLM-4.5 Team et al., 2025) or architectural changes like the number of dense layers. The Trinity paper's diagnosis suggests an entire dimension of the design space — the dynamics of the balancing mechanism itself — that had been underexplored. This is a conceptual contribution that opens a new axis for future work: not just what balancing signal to use, but how to make that signal's update dynamics well-behaved as an optimization process.

The evidence for this reframing is indirect but consistent. The paper reports that on early Trinity Large runs, "MaxVio would remain stable and then experience a sudden climb as the experts collapsed" (Section 6). This non-gradual failure mode — stability followed by sudden divergence — is exactly what you would expect from a system that cannot settle near its equilibrium and eventually crosses a threshold where the routing feedback loop becomes self-reinforcing. SMEBU, designed explicitly to enable convergence through continuous, proportional updates with momentum, resolved the issue. While the paper applied multiple stabilization fixes simultaneously and therefore cannot isolate SMEBU's contribution, the conceptual argument stands independently: the convergence limitation of sign-based updates is a structural problem, not an implementation bug.

Innovation 2: Batch Heterogeneity as a Diagnosable, Measurable Contributor to Training Instability

The paper introduces BatchHet (Batch Heterogeneity) as a quantitative metric for minibatch-level data imbalance, and in doing so, elevates a previously anecdotal concern — "are my training batches representative?" — into a measurable, monitorable, and improvable quantity. This is a diagnostic innovation: the paper provides a tool and a vocabulary for reasoning about a phenomenon that practitioners have long suspected matters but rarely quantified.

Before this work, the standard approach to document-level correlation in training was either to ignore it (sequential packing with simple shuffling) or to apply blunt fixes (larger batch sizes, more aggressive shuffling) without measuring the problem directly. The default assumption, often unstated, was that as long as documents were shuffled at some level, batch-level noise would average out over many steps. The Trinity paper challenges this assumption with a specific mechanistic hypothesis (Section 3.2):

"We hypothesize that imbalanced minibatches have particular impact in at least two scenarios: (i) in limited-batchsize settings, and (ii) in regimes where the model becomes more data efficient per step."

The key claim is that this is not just a small-sample statistical issue. Rather, the paper argues that minibatch imbalance effectively changes the optimization target at each step: "if we treat each unbalanced minibatch as an ideally-representative sample from its own target data generator, we can see that we effectively optimize a different target distribution per step." Under this framing, batch heterogeneity creates a form of objective function drift — the loss landscape the model is descending shifts from step to step, not because the true data distribution changed, but because the sampling procedure creates correlated minibatches that are unrepresentative in correlated ways.

The significance of framing batch heterogeneity as objective drift rather than just noisy gradients is that it predicts different scaling behavior. Noisy gradients (from IID sampling) average out with more steps; objective drift (from correlated sampling) creates systematic biases that persist. The paper reports that matching the step-to-step loss variance of the RSDB-enabled network required a 2× batch size increase for the baseline, while matching BatchHet required a 7× increase. This asymmetry is striking: variance can be reduced by averaging over more data, but the structured imbalance captured by BatchHet is more stubborn, requiring much larger batches to dilute the correlation.

The paper supports this with concrete metrics that go beyond hand-waving. The gradient norm kurtosis comparison — 14.6 for RSDB-enabled versus 187 for the baseline — is particularly telling. Kurtosis measures tail heaviness: a kurtosis of 187 indicates that gradient norms experience extreme outlier spikes far more frequently than a normal distribution would predict. The paper connects this directly to training stability, noting that "training runs without the RSDB exhibit much higher values in the higher-order moments of the running loss distribution, which we believe to correlate with network instability during training." This creates a causal chain: document-level correlation → batch heterogeneity → gradient norm spikes → training instability.

The BatchHet metric itself — max_i L_i − (1/M)Σ_i L_i — is deliberately simple. It measures the gap between the hardest microbatch and the average, capturing the worst-case deviation in difficulty within a training step. While the paper does not claim this is the optimal metric, its simplicity makes it immediately adoptable. More importantly, the paper shows it is actionable: the RSDB reduced BatchHet by 4.23×, and this reduction was visible in training dynamics (Figure 1 shows markedly reduced loss variance after RSDB deployment in Phase 3). This moves batch heterogeneity from a vague concern to an engineering target with a measurable success criterion.

Innovation 3: Extreme Sparsity as a Practical Frontier Strategy — The System-Level Argument

The Trinity Large configuration — 400B total parameters, 13B activated per token, roughly a 30:1 total-to-active ratio — represents an extreme point in the MoE design space that prior open-weight models had not explored. DeepSeek-V3 (DeepSeek-AI et al., 2025a) activates 37B of 671B total (~18:1 ratio). Kimi K2 (Kimi Team et al., 2025a) activates ~12B of ~500B total (~42:1 ratio, though the paper was concurrent and not cited as direct precedent). Mixtral 8×7B activates ~13B of ~47B (~3.6:1). The Trinity Large ratio of ~30:1 pushes sparsity well beyond what most prior models demonstrated, and the paper provides evidence that this extreme sparsity does not collapse performance.

The intellectual contribution here is not the concept of sparsity itself — MoE models have been getting sparser for years. Rather, it's the system-level argument that extreme sparsity is viable when paired with the right architectural and training choices. The paper effectively demonstrates that sparsity is not a free parameter you can dial up arbitrarily; it interacts with attention design, load balancing, optimization, and data quality in non-obvious ways. The key claim is that Trinity Large achieves competitive performance with GLM 4.5 Base (Figure 3) "despite having 4× higher degree of sparsity and a roughly 2.5× lower active parameter count," suggesting that the Trinity architecture extracts more capability per activated parameter.

This finding matters because it pushes against a natural intuition: that extremely sparse models will underperform denser models of equivalent active parameters because the routing mechanism introduces an additional source of error. If the router occasionally sends tokens to suboptimal experts, the model should perform worse than a dense model where every token sees all parameters. The Trinity results suggest that with sufficient total capacity (400B parameters) and stable routing (SMEBU), this degradation is manageable — at least at the scale tested.

The inference throughput data (Figure 4) provides the practical motivation. Trinity Large with FP8 quantization delivers higher throughput than comparably-sized models, which the paper attributes directly to "its extreme sparsity and interleaved local/global attention." This closes the loop on the paper's opening argument: the architecture was designed from the start for deployment efficiency, and the engineering choices (extreme sparsity, local/global attention, GQA) translate into measurable inference gains. The intellectual move is treating inference throughput not as an afterthought but as a co-equal design constraint with training efficiency and benchmark performance.

However, this innovation should be understood as empirical validation rather than theoretical advance. The paper does not derive conditions under which extreme sparsity works or provide a scaling law relating sparsity to performance. It demonstrates one successful configuration at one scale. The generalizability to even sparser configurations (say, 100:1) or to different model sizes is unknown. The paper acknowledges this implicitly by not making strong claims about optimal sparsity ratios. This is fundamentally an existence proof: "extreme sparsity can work at the 400B/13B scale with these specific architectural choices," not a universal principle.

Innovation 4: The Scaling Ladder as a Validation Methodology

A less flashy but methodologically significant contribution is the paper's use of a scaling ladder — training three models at increasing scale (Nano → Mini → Large) to validate the data pipeline, architecture, training recipe, and infrastructure before committing to the largest run. The paper states this explicitly: Trinity Nano and Mini "were developed as smaller form factors to provide immediately usable open models as well as serve as steps in our scaling ladder that validate the data pipeline, architecture, training recipe, and infrastructure required for Trinity Large" (Section 1).

This methodology addresses a practical challenge in frontier model development that is rarely discussed in technical reports: how do you de-risk a large training run when you cannot afford multiple attempts? Training Trinity Large on 17 trillion tokens with 2048 B300 GPUs represents a substantial compute investment. Running the same architecture (with scaled-down dimensions and expert counts) on 10 trillion tokens with 512 H200 GPUs provides confidence that the fundamental recipe works before scaling it up.

The paper demonstrates the value of this approach in two ways. First, the smaller runs discovered stable hyperparameter ranges (learning rates, batch sizes, warmup durations) that transferred to the larger run. The learning rate adjustment rule (Equation 41) was validated on the smaller models before being trusted at scale. Second, the Nano and Mini results serve as evidence that the architectural choices — the attention pattern, the MoE design, the normalization scheme — are not brittle artifacts of a particular scale. The fact that all three models completed training with zero loss spikes (stated in the abstract) is a strong signal that the architecture is fundamentally stable, not just lucky at one scale.

The intellectual contribution here is elevating what might be seen as "just engineering" into a principled methodology for resource-constrained frontier training. In an ideal world with infinite compute, you would run full-scale ablations. In practice, you cannot. The scaling ladder approach — validate architecture, optimizer, and data at 1/50th scale (Nano), confirm at 1/15th scale (Mini), then commit to full scale — is a structured risk-reduction strategy. The paper does not theorize about this methodology or claim it as a formal contribution, but its effectiveness is demonstrated by the outcome: Trinity Large trained successfully with zero loss spikes, a claim the authors clearly take pride in given its prominent placement in the abstract.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on a standard suite of benchmarks spanning code (MBPP+, Liu et al., 2023), math (Minerva MATH500, Lewkowycz et al., 2022), commonsense reasoning (HellaSwag, Zellers et al., 2019; WinoGrande, Sakaguchi et al., 2019), knowledge (MMLU, Hendrycks et al., 2021; MMLU-Pro, Wang et al., 2024b; TriviaQA, Joshi et al., 2017), reasoning (BBH, Suzgun et al., 2022; GPQA Diamond, Rein et al., 2023), and factuality (SimpleQA, Wei et al., 2024), plus the AIME25 competition math benchmark. For context extension evaluation, the Multi-Key Needle-in-a-Haystack (MK-NIAH) task from RULER (Hsieh et al., 2024) is used as a rapid proxy metric at the target context length. Tokenizer efficiency is evaluated by bytes-per-token and characters-per-token across multiple languages using C4 and reasoning corpus subsets (Table 1).
Base model(s). The Trinity family consists of three decoder-only sparse Mixture-of-Experts models: Trinity Nano (6B total, 1B activated per token, 56 layers, $d_{\text{model}}=1024$ ), Trinity Mini (26B total, 3B activated per token, 32 layers, $d_{\text{model}}=2048$ ), and Trinity Large (400B total, 13B activated per token, 60 layers, $d_{\text{model}}=3072$ ). All use the same architecture family (interleaved local/global attention with gating, sigmoid-routed MoE, depth-scaled sandwich norm) but differ in scale, expert granularity, and training data volume. The scaling ladder—validating the architecture and recipe at smaller scales before committing to the 400B-parameter run—is itself a methodological choice the paper explicitly motivates.
Metrics. Benchmark performance is reported as accuracy scores on each evaluation (exact match for math problems; standard grading for multiple-choice tasks), with few-shot prompting configurations specified per benchmark (e.g., 5-shot for MMLU, 0-shot for ARC Challenge). For context extension, MK-NIAH accuracy at the target context length is the primary metric. For inference benchmarks, throughput (tokens per second) is measured on 8xH200 GPUs with FP8-quantized models using vLLM. During training, MaxVio (Equation 42) tracks expert load imbalance, and the novel BatchHet metric (Equation 40) quantifies per-step microbatch loss heterogeneity. Training stability is assessed qualitatively through loss curve smoothness (Figure 1) and the presence or absence of loss spikes.
Baselines. The paper compares Trinity Large Base against other open-weight base models (Figure 3), including GLM 4.5 Base, DeepSeek-V3 Base, Kimi K2 Base, Llama 3.1 405B, and Nemotron 3 Nano. For inference throughput (Figure 4), comparisons include DeepSeek-R1, DeepSeek-V3, Llama 3.1 405B, Llama 3.3 70B, and Qwen 3 235B. For tokenizer efficiency (Table 1), comparisons include DeepSeek R1 (128k vocab), Qwen 3 (152k vocab), Llama 3 (128k vocab), and GPT-OSS (200k vocab). The paper does not report direct training-time comparisons (e.g., loss vs. FLOPs curves) against other models trained with different optimizers or architectures, which limits our ability to assess the "higher sample efficiency" claim for Muon quantitatively.
Generation budget / compute accounting. For training, total tokens processed is the primary compute metric: 10 trillion tokens for Nano and Mini, 17 trillion tokens for Large. For inference benchmarking, all models are quantized to FP8 and evaluated on identical 8xH200 hardware to provide a fair comparison of throughput at equivalent precision. For context extension evaluation, compute is measured by target context length and total training tokens during the extension phase (~117B tokens across 35.6M documents). There is no formal "FLOPs-matched" comparison between the Trinity models and comparably-sized dense models—the efficiency argument is made through architectural design (activated vs. total parameters) and throughput measurements, not through controlled FLOPs accounting.
Cross-validation / statistical protocol. The paper does not report cross-validation, confidence intervals, or statistical significance testing for any benchmark results. Benchmark scores are reported as single numbers per evaluation. For training dynamics metrics (MaxVio, BatchHet, gradient norm kurtosis), only point estimates are provided without error bars or multiple-seed statistics. The context extension evaluations use MK-NIAH as a single-number proxy without reporting variance across evaluation instances. This is consistent with common practice in large-scale training reports (where running multiple seeds is prohibitively expensive), but it means that differences of a few percentage points between models in Figure 3 and Table 3 should be interpreted cautiously—we cannot distinguish genuine capability differences from evaluation noise from these reports alone.

Main Quantitative Results

Benchmark Performance and Resource Efficiency (Figure 3, Tables 3–4)

The headline result is that Trinity Large Base achieves competitive scores with models that have substantially higher active parameter counts, while maintaining extreme sparsity. The specific numbers from Table 3:

Evaluation	Score
MMLU (5-shot)	82.58
MMLU-Pro (5-shot)	66.02
Minerva MATH500	65.20
MBPP+	88.62
HellaSwag (5-shot)	90.11
GPQA Diamond (5-shot)	43.94
BBH (few-shot)	65.70

Figure 3 places these in context visually. The comparison the paper emphasizes is against GLM 4.5 Base: "Trinity Large Base achieves competitive scores with GLM 4.5 Base, despite having 4× higher degree of sparsity and a roughly 2.5× lower active parameter count." Since Figure 3 is a bar chart without exact numerical labels, readers must cross-reference Table 3 with publicly available GLM 4.5 scores to verify this claim. The paper does not provide a tabular head-to-head with exact numbers for all comparison models, which makes precise comparison difficult.

What the paper can claim from these results: At 400B total parameters with 13B activated, Trinity Large Base achieves scores on MMLU and related benchmarks that place it among other open-weight models in the 30–70B active-parameter range. The MMLU score of 82.58 is solid but not state-of-the-art (frontier models now exceed 90 on MMLU). The more interesting result is the efficiency: achieving these scores with only 13B active parameters per token represents a substantially better performance-per-activated-parameter ratio than dense models of comparable total size, and competitive with sparse models that use higher active counts.

What the paper cannot claim without additional experiments: The efficiency argument relies on an implicit counterfactual—that a dense 13B model trained on the same data would underperform Trinity Large. The paper does not run this ablation (a dense 13B model trained identically). It also does not compare against a sparse model with equivalent total parameters but higher active count (e.g., a 400B/25B-active configuration) to test whether the extreme 30:1 ratio is near-optimal or overshot. These would be expensive experiments, but their absence means the sparsity-efficiency relationship is demonstrated at one point rather than characterized across a range.

Trinity Large Preview (instruct-tuned, Table 4):

Evaluation	Score
MMLU	87.21
MMLU-Pro	75.25
GPQA Diamond	63.32
SimpleQA	23.92
AIME25	24.36

The paper explicitly caveats these as preliminary: "Because we allocated most of our cluster time to pretraining, we were constrained to a relatively light post-training phase. As a result, the model we report here, Trinity-Large-Preview, is best viewed as a preliminary release rather than a fully post-trained model" (Section 4). The SimpleQA score of 23.92 is notably low, and the AIME25 score of 24.36, while respectable for a base model with light instruction tuning, is not competitive with reasoning-optimized models. These scores are consistent with the paper's self-assessment: they represent a starting point for further post-training, not a final capability claim.

Tokenizer Efficiency (Table 1)

The custom 200k-token vocabulary achieves the strongest compression among standard (non-SuperBPE) tokenizers on English (4.84 B/T on C4-en) and French (3.98 B/T), and competitive CJK compression given the training data timing constraint. The quantitative results:

English C4 B/T: Trinity 4.84 vs. DeepSeek R1 4.70 vs. Qwen 3 4.64 vs. Llama 3 4.73 vs. GPT-OSS 4.79
Reasoning B/T: Trinity 3.67 vs. DeepSeek R1 3.61 vs. Qwen 3 3.51
French C4 B/T: Trinity 3.98 vs. next best GPT-OSS at 3.97 (essentially tied)
CJK C/T (zh/ja/ko): Trinity at 1.38/1.39/1.26; trails DeepSeek R1 on ja/ko, trails Qwen 3 on zh, but exceeds Llama 3 on all three

The efficiency gains from the larger vocabulary are real but the practical significance depends on downstream model performance rather than compression alone. The paper's ablation found that SuperBPE achieved ~29% better English compression and ~27% better reasoning compression, but "we were unable to reproduce a corresponding improvement in downstream model performance at our experimental scale." This is an important negative result: better compression does not guarantee better modeling. The paper does not report whether the 200k vocabulary improved downstream performance over smaller vocabulary sizes at equal training tokens, only that "fertility measurements and small-scale loss curves confirmed consistent gains."

Context Extension Capabilities (Section 3.5)

The MK-NIAH results demonstrate strong and scaling-dependent long-context performance:

Trinity Nano: 0.864 at 128K after iterative improvement (initial 128K training: 0.38; 256K training for 128K target: 0.548)
Trinity Mini: 0.888 at 128K (trained at 128K, no overshoot training needed)
Trinity Large: 0.994 at 256K (trained target); 0.976 at 512K (zero-shot extrapolation); 0.42 at 1M (zero-shot extrapolation)

The progressive improvement with model scale is a robust finding: larger models extend context more easily and extrapolate further. The Nano result is particularly instructive as an ablation: training at twice the target length (256K for 128K target) improved MK-NIAH from 0.38 to 0.548, and further data/hyperparameter iteration pushed it to 0.864. This validates the "train longer than target" strategy from Gao et al. (2025) and NVIDIA et al. (2025a) at a new scale, but the paper does not isolate which changes (data composition, learning rate schedule, training duration) contributed how much to the 0.864 final score.

The Large extrapolation to 512K (0.976 without training at that length) is the paper's most striking individual result and, if replicable, substantially exceeds what would be expected from prior length generalization literature. However, MK-NIAH is a narrow probe of retrieval capability—it does not measure reasoning, synthesis, or instruction-following at long contexts. The paper does not report RULER's full suite (which includes variable tracking, aggregation, and question-answering tasks at long contexts) or any long-document QA benchmark. The 0.976 at 512K should therefore be understood as evidence that the attention mechanism can retrieve information at that length, not that the model can reason over 512K-token contexts. The 0.42 at 1M is suggestive of further scalability but well below usable thresholds for most applications.

Inference Throughput (Figure 4)

The throughput measurements on 8xH200 with FP8 quantization show Trinity Large's practical deployment characteristics. The paper does not provide a table of exact numbers, but the bar chart in Figure 4 is interpreted in the text as showing "strong performance" attributed to "extreme sparsity and interleaved local/global attention." The specific throughput advantage over comparably-sized models cannot be precisely quantified from the paper alone without the underlying data. This is a significant omission for a paper whose central motivation is inference efficiency—readers cannot independently assess whether the throughput gains are meaningful (e.g., 10% vs. 2×) without extracting numbers from the figure or running the models themselves.

Training Dynamics: BatchHet Reduction with RSDB (Section 3.2)

The RSDB results are among the paper's most carefully quantified findings:

RSDB reduced BatchHet by a factor of 4.23× during Phase 3 of Trinity Large training
Step-to-step loss variance reduced by a factor of 2.4× (visible in Figure 1's smoothed loss curve after Phase 3)
Small-scale experiments: RSDB reduced BatchHet by 46%, reduced gradient norm kurtosis from 187 to 14.6
Matching the baseline's step-to-step loss variance required 2× batch size increase; matching BatchHet required 7× increase

These are substantial effects that demonstrate RSDB is not a marginal improvement. The batch size multiplier (7× to match BatchHet) is particularly informative: it quantifies how much extra compute would be needed to achieve equivalent batch balance through naive scaling, giving a concrete efficiency argument for the RSDB approach.

Training Stability and the Stabilization Interventions (Section 6)

The paper provides unusual transparency about training failures and fixes. The key empirical claim is that the six simultaneous interventions (SMEBU, disabling MXFP8, z-loss, sequence-wise aux loss, increased dense layers from 3 to 6, intra-document masking) collectively resolved the expert collapse and MaxVio divergence observed in early runs. The evidence is:

MaxVio stopped diverging after the fixes
Expert utilization remained balanced throughout the remaining training
Loss continued to converge smoothly (Figure 1 supports this visually)
Zero loss spikes across all three models (stated in the abstract)

However, the paper explicitly acknowledges: "Because the fixes were introduced together to unblock training, we did not have time to run controlled ablations to attribute stabilization to any individual change." This means we cannot determine whether SMEBU alone would have sufficed, or whether the stabilization came primarily from, say, the increased dense layers or the intra-document masking. The paper's candidness about this limitation is admirable, but it means the claims about SMEBU's effectiveness—while mechanistically well-motivated—are not experimentally validated in isolation.

Ablation Studies and Robustness Checks

Vocabulary size: The paper reports an empirical comparison across vocabulary sizes (200k, 131k mentioned) using "fertility measurements and small-scale loss curves." Specific loss values or performance deltas are not reported. The finding that "consistent gains from the larger vocabulary" were observed, with "most pronounced improvements in CJK languages and French," is stated qualitatively without quantitative support in the paper.

SuperBPE vs. standard BPE: Training a SuperBPE variant (which learns multi-word tokens by resuming BPE training without whitespace constraints) achieved ~29% fewer English tokens and ~27% fewer reasoning tokens but did not produce downstream performance improvements at the experimental scale. This negative result is methodologically sound and well-motivated—the paper tested a recently proposed technique (Liu et al., 2025a) that showed promise on compression metrics, found it didn't translate to model quality improvements, and proceeded with standard BPE. The result is not quantified beyond the compression percentages.

Muon optimizer: The paper claims Muon "enables a larger critical batch size and has higher sample efficiency than AdamW," but does not present a head-to-head training comparison between Muon and AdamW-trained models at any scale in the Trinity family. This is a significant omission for a paper that prominently features Muon as a key design choice. Readers must take the sample efficiency claim on faith or reference external work (Jordan et al., 2024a; Liu et al., 2025b). The learning rate adjustment rule (Equation 41) is stated to enable "optimal learning rate transfer when scaling model width" based on empirical observation, but the supporting evidence is not presented.

Context extension strategies: Two key ablations were performed: (1) adjusting only global layers vs. adjusting both local and global layers, and (2) progressive context extension vs. training directly at the target length. For (1), "adjusting only the global layers resulted in a much quicker loss recovery," stated qualitatively. For (2), "we did not find any improvements from progressively increasing the context window size," which is a clean negative result. Both findings are practically useful but are reported without quantitative loss comparisons or multiple-seed verification.

Training-at-longer-than-target-length: For Trinity Nano, the ablation is quantified: training at 128K target → MK-NIAH 0.38; training at 256K for 128K target → 0.548; after iteration → 0.864. This three-point comparison is the paper's best-controlled scaling result and clearly demonstrates the benefit of overshoot training.

Dense layers count: Increasing initial dense layers from 3 to 6 was one of the six simultaneous stabilization fixes. No ablation isolating this change is reported. The paper states the rationale ("to further stabilize representations") but provides no evidence that 6 is better than 3 independently of the other fixes.

Local/global attention pattern: The 3:1 ratio is stated to follow Yang et al. (2025), but no alternative ratios were tested or reported. The finding that the combination of this pattern with gated attention "allowed the model to recover performance in a comparatively lower number of steps when training at longer sequence lengths" is stated qualitatively without step-count comparisons.

QK-normalization: Adopted "primarily for training stability reasons, especially due to our use of the Muon optimizer." No ablation comparing Muon training with and without QK-norm is presented. The paper cites external work (Liu et al., 2025b; GLM-4.5 Team et al., 2025) for evidence but does not reproduce the comparison in the Trinity setting.

Z-loss weight: Initially planned at $1 \times 10^{-4}$ but not applied due to a bug. When introduced mid-training to stabilize logit growth, $1 \times 10^{-6}$ worked while "larger values destabilized the network." The paper does not report what "larger values" were tested or what specific instability was observed, but the sensitivity finding (effective weight range spanning less than two orders of magnitude) is practically useful.

RSDB vs. standard packing: The most carefully quantified ablation in the paper. Small-scale experiments compare gradient norm kurtosis (14.6 vs. 187), BatchHet (46% reduction), and batch-size multipliers needed for parity (2× for loss variance, 7× for BatchHet). The Phase 3 Trinity Large deployment provides real-scale validation: 4.23× BatchHet reduction and 2.4× loss variance reduction. See Main Quantitative Results above for specific numbers.

Intra-document attention masking: Adopted as one of the six stabilization fixes "to prevent token positions from attending to token positions from a different document, to reduce noise in the learning objective." No ablation is reported. This is standard practice in many training pipelines, and the paper does not claim novelty—it is presented as a stabilization measure adopted alongside other changes.

SMEBU vs. sign-based updates: Despite being presented as one of the paper's key innovations, SMEBU was adopted as part of the six simultaneous stabilization fixes, and no controlled comparison with the sign-based method at equal scale is reported. The small-scale tests that motivated SMEBU's design are referenced ("we had briefly tested at very small scale") but not presented with quantitative results. The conceptual argument for why SMEBU should improve over sign-based updates is well-articulated in Section 2.3, but the experimental evidence is limited to the observation that Trinity Large trained stably after the combined fixes were applied. This is the paper's most significant missing ablation.

Critical Assessment

Does Trinity Large actually demonstrate that extreme sparsity works at the 400B scale?

The paper's central claim—that a 400B-parameter, 13B-activated MoE model can achieve competitive performance with much larger active-parameter models—is supported by the benchmark numbers in Table 3 and Figure 3, but only in a specific sense that requires careful qualification. The evidence shows that Trinity Large Base achieves scores comparable to other open-weight base models on a standard suite of academic benchmarks. It does not show that the extreme 30:1 sparsity ratio is near-optimal, nor that the same total parameter budget could not have been used more efficiently with a different active-parameter count. The comparison models in Figure 3 have different total parameters, different active parameters, different training data, different tokenizers, and different training recipes—any of which could account for observed differences. The paper's claim is best understood as an existence argument: "it is possible to get this level of performance with this degree of sparsity," not "this is the optimal sparsity configuration" or "sparsity causes the performance."

A controlled test of the sparsity claim would compare Trinity Large against an identically-trained dense model with the same active parameters (13B) to quantify how much the extra 387B parameters contribute. It would also sweep the sparsity ratio at fixed total compute to identify whether 30:1 is near a Pareto frontier. These experiments are extremely expensive, and their absence is not a criticism of the paper's quality—it is a statement about what we can and cannot conclude from the presented results. The paper is transparent about its positioning: it presents Trinity as a systems demonstration, not a scientific study of sparsity scaling laws.

Is the "zero loss spikes" claim meaningful?

The paper prominently claims that "all three models completed training with zero loss spikes" (abstract), and Figure 1 shows a smooth training loss curve for Trinity Large. This is an engineering achievement worth reporting—training a 400B MoE model stably on new hardware (B300 GPUs) and a new optimizer (Muon) is genuinely difficult. However, "zero loss spikes" is a binary property that depends on the definition of a "spike." The paper reports that early runs experienced MaxVio divergence and expert collapse—these were terminated and the fixes were applied. The successful run had zero spikes after the fixes. It is unclear whether models from other organizations reporting "stable training" use the same implicit threshold for what counts as a spike. The claim is better understood as "we achieved training stability through architectural and algorithmic choices" rather than a comparative claim against other models, whose training dynamics are not equally documented.

Are the benchmark results sufficient to assess model quality?

The evaluation suite (Table 3) is standard but limited. Ten benchmarks, mostly multiple-choice or short-answer, with 0-shot or 5-shot prompting. The paper does not report:

Long-context evaluation beyond MK-NIAH (no long-document QA, no long-context reasoning)
Multilingual benchmark results despite claiming multilingual data curation
Coding benchmarks beyond MBPP+ (no HumanEval, no LiveCodeBench)
Safety evaluations or bias assessments
Perplexity on held-out text, which would provide a more continuous measure of language modeling quality
Robustness to prompt format variations

For a model positioned as an "open-weight foundation," the multilingual gap is notable—the paper acknowledges that the tokenizer training data underweighted non-English languages, which likely affects multilingual performance, but no non-English evaluation is reported to quantify the impact.

The post-training results (Table 4) add five more benchmarks, but the paper explicitly frames these as preliminary. The SimpleQA score of 23.92 is low enough to suggest that the model may hallucinate frequently on factual queries, which would be a significant deployment concern. No analysis of failure modes is provided.

Does the inference throughput data support the efficiency claims?

Figure 4 shows throughput comparisons but is not accompanied by a table of exact measurements. The text states the efficiency gains come from "extreme sparsity and interleaved local/global attention," but without isolating these factors. For example, GQA reduces KV-cache size, the local/global pattern reduces attention FLOPs at long sequences, and MoE sparsity reduces FFN FLOPs per token—but the relative contribution of each to measured throughput is not analyzed. Whether the throughput advantage over, say, DeepSeek-V3 comes from architecture, quantization quality, vLLM implementation maturity, or a combination of factors cannot be determined from the presented data.

The throughput measurements are taken on 8xH200 with FP8 quantization. Performance on different hardware (A100, H100, B200), at different batch sizes, or at long sequence lengths (where the local/global attention advantage should be most pronounced) is not reported. For a paper that positions inference efficiency as a primary motivation, this is a thin evaluation.

Is the RSDB contribution validated?

The RSDB results are the paper's most rigorous quantitative analysis. The BatchHet metric is clearly defined, the small-scale experiments provide controlled comparisons, and the Phase 3 deployment shows real-scale validation. The gradient norm kurtosis reduction (187 → 14.6) is a compelling signal that the mechanism addresses a real source of training noise. The batch size multiplier analysis (2× for loss variance matching, 7× for BatchHet matching) is particularly informative.

However, the paper does not report whether the RSDB improved final model quality or only training dynamics metrics. Did the RSDB-enabled network achieve lower final loss or better downstream benchmarks? The Phase 3 integration makes this difficult to assess (since Phase 3 also changed the data mixture), but small-scale experiments could have reported final loss comparisons. The paper reports "improved loss over a sequential packing baseline" qualitatively but without terminal loss values. Without this, we know RSDB smooths training but not whether that smoothness translates to better models.

Can we conclude that SMEBU is an improvement over standard load balancing?

This is the paper's weakest evidential link. The conceptual argument for SMEBU—continuous, proportional updates with momentum replacing constant-magnitude sign-based updates—is mechanistically sound and well-motivated. The integration into Trinity Large's successful training run is consistent with SMEBU being an improvement. But SMEBU was one of six simultaneous fixes, and no controlled ablation isolates its contribution. A small-scale experiment comparing SMEBU against sign-based updates (at equal compute and model size) would have been feasible and informative. Its absence means SMEBU should be viewed as a well-motivated proposal with preliminary validation, not an empirically established improvement.

Overall assessment of the experimental evidence:

The paper succeeds at its primary task: documenting the Trinity family's architecture, training recipe, and benchmark performance with sufficient transparency that others can adopt or adapt the approach. The strength lies in the systems-level detail—training hyperparameters, stability interventions, data pipeline decisions, and infrastructure configuration are reported with unusual thoroughness.

The weakness lies in causal inference. The paper demonstrates that a specific combination of architectural and training choices can produce stable training and competitive benchmarks. It does not demonstrate why or how much each choice matters. The paper cannot decompose the contributions of its innovations: How much does SMEBU improve over sign-based updates? How much does Muon improve over AdamW for MoE training? What is the performance gap between the 30:1 sparsity of Trinity Large and a hypothetical 15:1 configuration? These are expensive questions to answer, and their absence is characteristic of large-scale training reports—but the paper would be stronger if it acknowledged these specific open questions explicitly rather than implying that the integrated result validates each component independently.

The context extension results (particularly Trinity Large's 0.976 MK-NIAH at 512K zero-shot) are individually striking and, if independently verified, would represent a meaningful contribution to our understanding of length generalization in hybrid attention architectures. The RSDB contribution is well-quantified and addresses a real, previously under-discussed problem in training infrastructure. The tokenizer efficiency data is thorough and practically useful. The negative result on SuperBPE is valuable and well-contextualized. These are substantial contributions that do not depend on isolating individual factors.

The paper's most important experimental contribution may be its demonstration that extreme MoE sparsity training can be made stable through careful architecture and training recipe design, even on new hardware and with a less-common optimizer. The zero-loss-spike record across three model scales is meaningful engineering validation, even if we cannot attribute it to individual components.

6. Limitations and Trade-offs

6.1 SMEBU Cannot Be Isolated as the Causal Factor in Training Stability

The assumption or constraint. SMEBU is presented as one of the paper's key innovations—a novel load balancing method motivated by a principled convergence argument (Section 2.3). However, it was deployed as one of six simultaneous stabilization interventions applied when early Trinity Large runs experienced expert collapse and MaxVio divergence. The paper is explicit about this (Section 6):

"We applied six changes at once, all targeted at increasing the stability of the run... Because the fixes were introduced together to unblock training, we did not have time to run controlled ablations to attribute stabilization to any individual change."

The other five changes were: disabling MXFP8 kernels, adopting z-loss, adding sequence-wise auxiliary loss, increasing initial dense layers from 3 to 6, and adopting intra-document masking.

The consequence. A practitioner deciding whether to adopt SMEBU for their own MoE training pipeline cannot determine whether SMEBU was necessary or sufficient for the observed stability. It is possible that the stabilization came primarily from the increased dense layers (providing more stable early representations), from the intra-document masking (reducing noise in the learning objective), or from the combination of z-loss and sequence-wise aux loss (both of which directly constrain logit magnitudes and routing balance respectively). SMEBU's conceptual motivation—replacing constant-magnitude sign-based bias updates with continuous, proportional, momentum-smoothed updates—is well-articulated, but the empirical evidence that this mechanism matters in practice consists entirely of the observation that a run including SMEBU was stable, while earlier runs lacking SMEBU (and also lacking the other five fixes) were not. This is a correlation-at-best, and a practitioner who implements SMEBU but omits the other fixes might encounter the same instability that early runs experienced. Conversely, a practitioner who implements the other five fixes without SMEBU might achieve stability with the simpler sign-based method.

What evidence exists in the paper. The paper references "small scale internal experiments" where SMEBU was "briefly tested at very small scale" (Section 6), but no quantitative results, loss curves, or MaxVio trajectories from these experiments are reported. The only documented evidence is the successful Trinity Large training run, which incorporated all six fixes simultaneously. There is no ablation isolating SMEBU's contribution at any scale.

Mitigation status. The paper does not attempt to mitigate this limitation—it transparently acknowledges it and does not claim otherwise. No controlled comparison between SMEBU and sign-based bias updates is reported or promised as future work. The limitation is one of experimental design under resource constraints (limited time to unblock a large-scale training run), and the paper's candor about it is commendable, but the evidential status of SMEBU remains: it is a well-motivated proposal with preliminary validation, not an empirically established improvement. Future work—ideally at a scale where controlled ablations are affordable—is needed to establish whether the convergence argument translates to measurable stability or performance gains.

6.2 Performance Evaluated on a Narrow Benchmark Suite Without Long-Context or Multilingual Validation

The assumption or constraint. The paper evaluates Trinity Large Base on ten standard benchmarks (Table 3) spanning code, math, commonsense reasoning, knowledge, and graduate-level reasoning, plus five additional benchmarks for the instruct-tuned Preview model (Table 4). However, two capability domains that the paper explicitly claims as design targets are either absent or minimally evaluated:

Long-context performance beyond retrieval. The paper emphasizes that the architecture was designed for "efficient inference" at long contexts (Section 1, Section 2.2) and reports MK-NIAH scores as evidence of context extension success (Section 3.5). However, MK-NIAH measures retrieval—the ability to find a fact placed somewhere in a long context—and does not measure reasoning, synthesis, instruction-following, or multi-hop capabilities at those lengths. The paper does not report performance on any long-context reasoning benchmark (e.g., LongBench, L-Eval, ZeroSCROLLS, RULER's full suite beyond NIAH) that would demonstrate whether the model can actually use long contexts for complex tasks.
Multilingual performance. The paper states that the Trinity Large data mix includes "multilingual data curation, targeting Arabic, Mandarin, Japanese, Spanish, German, French, Italian, Portuguese, Indonesian, Russian, Vietnamese, Hindi, Korean, and Bengali" (Section 3.1). However, no multilingual benchmark results are reported. The tokenizer efficiency evaluation (Table 1) measures compression rates across languages, but compression is a tokenizer property, not a model capability metric. The paper acknowledges a training data timing constraint: "non-English languages are less well represented relative to their proportion in the final pretraining data" (Section 2.1). The consequence of this under-representation on actual multilingual task performance is never measured.

The consequence. A practitioner evaluating Trinity Large for deployment in a long-context application (e.g., document understanding over 100K+ tokens, repository-scale code analysis, multi-document summarization) has no evidence that the model can reason effectively at those lengths. The MK-NIAH result of 0.976 at 512K is impressive as a retrieval signal but does not guarantee that the model can, for example, answer a question that requires integrating information from multiple locations in a 500K-token document. The gap between retrieval and reasoning at long contexts is well-documented in the literature (Hsieh et al., 2024), and the paper's exclusive reliance on MK-NIAH for long-context validation leaves this gap unaddressed.

Similarly, a practitioner deploying Trinity Large for multilingual applications—particularly in the targeted languages beyond English—cannot assess whether the model meets a minimum quality bar. The tokenizer under-representation of non-English languages during vocabulary training, combined with the absence of any multilingual downstream evaluation, means that multilingual performance is entirely uncharacterized. A model that achieves 82.58 on English MMLU could plausibly score anywhere from competitive to near-random on, say, Arabic MMLU, and the paper provides no way to distinguish these scenarios.

What evidence exists in the paper. For long-context: MK-NIAH scores at target and extrapolated lengths (Section 3.5). Full stop. No long-context reasoning, QA, summarization, or code understanding benchmarks are reported. For multilingual: tokenizer compression rates (Table 1). No downstream task results in any non-English language are reported. The paper does not acknowledge this gap as a limitation—the absence is presented as a scope constraint rather than a known unknown.

Mitigation status. The paper does not attempt to mitigate either gap. For long-context, it suggests that "it would be possible to push later versions of the model to have strong performance at a context window of 1M" (Section 3.5), implying future work but not committing to it. For multilingual evaluation, no future work is proposed. Both gaps are significant for a model positioned as an "open-weight foundation" intended for broad deployment.

6.3 The Sample Efficiency Claim for Muon Is Unsubstantiated by In-Paper Comparisons

The assumption or constraint. The paper makes a specific, testable claim about the Muon optimizer (Section 1):

"the Muon optimizer, which enables a larger critical batch size and has higher sample efficiency than the widely used AdamW optimizer."

This claim is central to the paper's training efficiency argument—if Muon does not actually provide higher sample efficiency, the justification for adopting a less-common optimizer with potential stability risks (the paper itself notes that QK-norm was needed "especially due to our use of the Muon optimizer," citing prior work that identified growing attention logit values as more of a concern with Muon) is weakened. However, the paper provides no head-to-head comparison between Muon and AdamW-trained Trinity models at any scale. The claim is supported entirely by citations to external work (Jordan et al., 2024a; Liu et al., 2025b).

The consequence. A practitioner deciding whether to adopt Muon for their own training pipeline cannot assess its benefit from this paper's evidence. The sample efficiency claim matters because Muon requires additional implementation complexity (the orthogonalization step, the distributed gradient handling described in Section 3.3, the learning rate adjustment rule in Equation 41) and introduces stability concerns that necessitated QK-normalization (Section 2.2). If Muon's sample efficiency advantage over AdamW is, say, 5% for MoE models at this scale, the engineering cost of adoption might outweigh the benefit. If it is 30%, it is clearly worth it. The paper provides no basis for making this judgment.

Furthermore, the paper never reports what the critical batch size actually is for its Muon-trained models. The "larger critical batch size" claim is stated without measurement. The batch size increases during training (e.g., Trinity Large from 12288 to 16384 sequence length 8192, yielding roughly 100M to 134M tokens per batch) are attributed to throughput optimization "roughly following MiniMax et al. (2025)" (Section 3.4.2), not to having reached a critical batch size limit with the previous configuration.

What evidence exists in the paper. Zero direct evidence. The paper reports training curves (Figure 1), benchmark scores (Tables 3–4), and training hyperparameters (Section 3.4.2), but no comparison against an AdamW-trained equivalent. The learning rate adjustment rule (Equation 41) is stated to "enable optimal learning rate transfer when scaling model width, for Muon" based on empirical observation, but the observations are not presented. The external citations (Jordan et al., 2024a; Liu et al., 2025b) provide evidence for Muon's efficiency in other settings, but transfer to the specific architecture, data mixture, and scale of Trinity is assumed, not demonstrated.

Mitigation status. The paper does not acknowledge this as a limitation. The claim is presented as established fact based on prior work, and the paper's contribution is demonstrating that Muon works at scale (i.e., training completes without divergence) rather than that it outperforms AdamW. This is a reasonable but implicit scope constraint—the paper could strengthen its claims by explicitly noting that Muon-vs-AdamW comparisons at Trinity scale were not performed and that the efficiency claims should be understood as motivation for the choice rather than findings of the work.

6.4 Inference Throughput Gains and Efficiency Claims Are Not Quantitatively Decomposed

The assumption or constraint. The paper's opening motivation (Section 1) positions inference efficiency as a primary design goal, citing "a growing need for inference-time efficiency as workflows and contexts over which the LLMs are required to operate on grow larger and larger." The architecture is explicitly designed for efficient inference through extreme MoE sparsity, interleaved local/global attention, and GQA. However, the throughput evaluation (Figure 4) is presented as a single bar chart without a table of numerical values, without decomposition of which architectural components contribute how much to the measured throughput, without reporting batch size or sequence length conditions, and without ablation of individual efficiency features.

The consequence. A practitioner cannot answer several deployment-critical questions from the paper alone:

How much does each architectural choice contribute to throughput? Is the throughput advantage primarily from MoE sparsity (activating only 13B of 400B parameters), from the local/global attention pattern (reducing attention FLOPs at long sequences), from GQA (smaller KV-cache), or from a combination? Without decomposition, a practitioner cannot prioritize which features to adopt or adapt for their own architecture.
How does throughput scale with sequence length? The local/global attention pattern's efficiency advantage should be most pronounced at long sequences (where sliding window attention in local layers provides O(w) rather than O(T) cost, with w ≪ T). The paper does not report throughput at different sequence lengths, so this expected scaling behavior cannot be verified. At short sequences (where the models were pre-trained), the local/global pattern provides no computational advantage over full attention, and the overhead of the pattern switching might even be a slight negative.
How does throughput compare at equivalent precision without quantization? All measurements use FP8 quantization. The paper does not report BF16 throughput, making it unclear whether the efficiency gains are amplified or masked by quantization effects.
What are the actual numbers? Without extracting values from Figure 4 (which may not be precise), readers cannot compare Trinity Large's throughput to, say, DeepSeek-V3's in absolute terms—only in relative bar heights.

What evidence exists in the paper. Figure 4 shows a bar chart comparing throughput for several models measured on 8xH200 with FP8 quantization using vLLM. The text states that Trinity Large's "extreme sparsity and interleaved local/global attention results in strong performance" (Section 5.2). No table of numerical values, no throughput-at-varied-sequence-lengths, no ablation of attention pattern or sparsity effects, and no BF16 comparison are provided.

Mitigation status. The paper does not acknowledge the thinness of the throughput evaluation as a limitation. The inference benchmark section (Section 5.2) is approximately one paragraph long in a 29-page report. The models and code are open-weight and open-source (available on HuggingFace), so independent benchmarking is possible—but a report that foregrounds inference efficiency as a core motivation should provide more detailed characterization. The paper would benefit from explicitly noting what throughput experiments were and were not performed, and committing the detailed results (with numerical values, sequence length sweeps, and precision comparisons) to future releases or community evaluation.

6.5 The RSDB Is Evaluated on Training Dynamics Metrics, Not Final Model Quality

The assumption or constraint. The Random Sequential Document Buffer (RSDB) is presented as an infrastructure innovation that reduces batch-level document correlation and improves training stability. The evaluation in Section 3.2 is thorough for training dynamics: BatchHet reduced by 4.23× during Phase 3 of Trinity Large training, step-to-step loss variance reduced by 2.4×, gradient norm kurtosis reduced from 187 to 14.6 in small-scale experiments, and batch-size multipliers of 2× (for loss variance) and 7× (for BatchHet) needed for a sequential packing baseline to match RSDB. However, the paper does not report whether these improved training dynamics translate to improved final model quality—lower terminal loss, better downstream benchmark performance, or faster convergence to a given performance level.

The consequence. A practitioner evaluating whether to implement RSDB in their own training pipeline has strong evidence that RSDB smooths training metrics but no evidence that this smoothness matters for the outcome they care about: model quality. It is possible that the training dynamics improvements are cosmetic—the loss curve looks smoother (as in Figure 1), but the terminal loss after equivalent compute is unchanged, and the downstream benchmarks are identical. If the RSDB's primary benefit is reducing the probability of training instability (loss spikes, expert collapse), that is valuable but should be framed as a stability intervention rather than an efficiency improvement. If the RSDB also enables faster convergence or better final quality (by reducing the "overhead" the paper hypothesizes comes from the network having to "unlearn" domain imbalances each step), that is a stronger claim requiring different evidence.

The batch-size multiplier analysis (7× larger batches needed to match BatchHet) suggests RSDB provides substantial effective data efficiency, but without terminal quality comparisons, we do not know whether the 7× multiplier translates to actual compute savings. If a 7× larger batch size trained for the same number of tokens achieves equivalent final loss, then RSDB is providing a 7× effective data efficiency improvement. If the larger batch size saturates earlier or converges to a worse minimum despite matching BatchHet, the apparent efficiency gain is illusory.

What evidence exists in the paper. The small-scale experiments report "improved loss over a sequential packing baseline" qualitatively (Section 3.2) but without terminal loss values. The Phase 3 deployment occurred simultaneously with a data mixture change (shifting to Phase 3 data), making it impossible to attribute loss improvements to RSDB versus data quality changes. No downstream benchmark comparison between RSDB-enabled and baseline training is reported, even at small scale.

Mitigation status. The paper does not acknowledge this gap. The RSDB is evaluated exclusively on training dynamics metrics, and the implicit claim is that better training dynamics (lower BatchHet, lower loss variance, lower gradient norm kurtosis) are inherently valuable because they correlate with stability. The paper would be strengthened by explicitly noting that final model quality comparisons are not available (for the large-scale run, due to the Phase 3 data mixture confound; for small-scale runs, due to the experiments not having been run to convergence or not being reported) and that the demonstrated benefits should be understood as stability improvements whose impact on model quality remains to be quantified.

6.6 Post-Training Is Acknowledged as Preliminary, Leaving Open Questions About Production Readiness

The assumption or constraint. The paper is transparent that the instruct-tuned model, Trinity Large Preview, received minimal post-training due to compute constraints (Section 4):

"Because we allocated most of our cluster time to pretraining, we were constrained to a relatively light post-training phase. As a result, the model we report here, Trinity-Large-Preview, is best viewed as a preliminary release rather than a fully post-trained model."

The supervised fine-tuning used a blend of public and custom instruction data (the custom portion generated by stronger teacher models and filtered for quality), plus agentic coding trajectories collected through OpenCode. A short RL stage was run using prime-rl with verifiable rewards where available and a learned reward model otherwise. No details on training duration, data volume, or RL hyperparameters are provided.

The consequence. The instruct-tuned evaluation results (Table 4) cannot be interpreted as indicative of what a fully post-trained Trinity Large could achieve. The SimpleQA score of 23.92 is notably low—for context, GPT-4o scores approximately 39.5 on SimpleQA (Wei et al., 2024), and even smaller open models with more thorough post-training often exceed 30. This suggests significant room for improvement through more extensive instruction tuning and RL, but it also means that practitioners evaluating Trinity Large Preview for deployment requiring factual reliability should expect substantial hallucination on short-form factual queries. The AIME25 score of 24.36, while respectable for a lightly post-trained model, is also well below what reasoning-optimized models achieve (e.g., o1 reportedly exceeds 70 on AIME 2024), indicating that the base model's reasoning capabilities are not fully surfaced through the current post-training recipe.

More broadly, the paper's evaluation of the Preview model is limited to five benchmarks (MMLU, MMLU-Pro, GPQA Diamond, SimpleQA, AIME25) without safety evaluations, bias assessments, instruction-following benchmarks (e.g., IFEval, MT-Bench), or coding benchmarks in the instruct format. For a model released as open-weight and positioned for enterprise deployment, the absence of safety and instruction-following characterization is a significant gap.

What evidence exists in the paper. Table 4 provides five benchmark scores. The paper does not report SFT data volume, RL training steps, reward model details, or any post-training ablations. No safety, bias, or instruction-following evaluations are reported. The paper explicitly frames the Preview as preliminary.

Mitigation status. The paper fully acknowledges this as a scope limitation and commits to future work: "In the next iteration, we plan to build upon this foundation with further, more extensive post-training" (Section 4). This is appropriate framing—the Preview is released as an intermediate checkpoint, not a final product—but practitioners should treat the Preview benchmarks as lower bounds on what better-post-trained versions could achieve, not as definitive capability measurements. The absence of safety and instruction-following evaluation is not explicitly noted as a limitation; the paper would benefit from acknowledging this gap and committing to such evaluations in future releases.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around MoE training from "can we prevent collapse?" to "how do we make the routing dynamics converge?" — a subtle but consequential reframing. Prior work on MoE stability focused heavily on auxiliary loss design (Shazeer et al., 2017; DeepSeek-AI et al., 2025a; GLM-4.5 Team et al., 2025) and architectural safeguards like dense initial layers or router formulations. The implicit assumption was that if you apply a strong enough balancing signal, expert collapse is prevented, and the rest of training proceeds normally. The Trinity paper's diagnostic contribution — articulated most clearly in the SMEBU motivation (Section 2.3) — is that even a correct balancing signal can cause instability if the update dynamics cannot converge. The sign-based bias update is "always ±λ" regardless of proximity to the optimal bias values, creating persistent oscillation that, the paper hypothesizes, can trigger the self-reinforcing feedback loop of expert collapse when it crosses a threshold. This is not a load-balancing problem; it is an optimization problem applied to the balancing mechanism itself.

The magnitude of this shift should be understood precisely. This is not a paradigm shift — sparse MoE remains the dominant architecture for efficient scaling, and the core routing mechanism is unchanged. It is, however, a diagnostic reframing that opens a new axis of the design space. Before this work, a practitioner facing MoE instability would think: "I need stronger balancing — increase the aux loss weight, adjust the bias update speed, add more shared experts." After this work, the practitioner should also think: "Are my bias updates capable of settling? Do I need proportional updates that decay near equilibrium? Would momentum smooth the residual noise?" This is an additional lever, not a replacement for existing ones, but it is a lever that had been largely invisible in the literature.

The paper's BatchHet metric (Section 3.2) makes a parallel contribution at the data infrastructure level. Prior training infrastructure work treated document-level correlation as an inconvenience to be mitigated by aggressive shuffling or larger batch sizes, without measuring the problem directly. BatchHet provides a simple, computable quantity — max_i L_i − mean(L) — that quantifies per-step minibatch imbalance and correlates with gradient norm instability (kurtosis of 14.6 with RSDB vs. 187 without in small-scale experiments). This moves batch heterogeneity from anecdote to diagnosable target. The practical implication is that training infrastructure engineers now have a lightweight metric to monitor and optimize, analogous to how MaxVio provides a window into expert load balance. The paper's evidence that BatchHet is harder to fix than simple loss variance (7× batch size multiplier needed to match BatchHet vs. 2× for loss variance) suggests it captures a qualitatively different and more stubborn form of training noise — what the paper frames as "objective function drift" from correlated minibatch sampling.

In terms of reconciling prior contradictions, the paper provides a potential explanation for why some MoE training runs are stable at moderate scale but become unstable as models grow. The paper notes that "as the total number of experts increases, the per-layer norm of the bias step also increases" under sign-based updates (Section 2.3). A 128-expert model (like Nano and Mini) may have small enough per-layer bias step norms that the oscillation remains below the threshold that triggers collapse. A 256-expert model (like Large) may cross that threshold. This would explain why the standard aux-loss-free method worked at smaller scales for Wang et al. (2024a) but required replacement for Trinity Large. If this hypothesis is correct, it implies that sign-based load balancing has a fundamental scaling limit tied to expert count, and that continuous update schemes (like SMEBU) become necessary beyond some crossover point. The paper does not prove this, but it provides a coherent mechanistic story that future work can test.

The scaling ladder methodology — validating architecture, data, and training recipe at Nano (6B/1B) and Mini (26B/3B) scales before committing to Large (400B/13B) — is not itself a conceptual contribution, but its effectiveness as demonstrated here may influence how resource-constrained teams approach frontier training. The fact that all three models completed training with zero loss spikes, and that the architectural choices transferred across a 67× range of total parameters and a 13× range of activated parameters, provides validation that the design is not brittle to scale. This contrasts with a narrative, sometimes implicit in the literature, that training stability requires heroic per-model tuning. The Trinity experience suggests that a well-designed architecture can be stable across scales when the right stability mechanisms are baked in from the start.

The paper also makes extreme sparsity (30:1 total-to-active ratio) a more credible option in the design space for practitioners balancing capability against inference cost. Before this work, the most prominent open-weight MoE models (DeepSeek-V3 at ~18:1, Mixtral at ~3.6:1) were substantially less sparse, and it was unclear whether pushing sparsity further would cause unacceptable performance degradation. Trinity Large's benchmark scores (Figure 3) — competitive with GLM 4.5 Base at 4× higher sparsity and 2.5× lower active parameters — provide an existence proof that 30:1 can work. This does not mean 30:1 is optimal; the paper does not sweep sparsity ratios. But it means that teams designing MoE models now have an empirically validated point on the sparsity frontier that was previously speculative. This concrete data point may encourage exploration of even sparser configurations (50:1, 100:1), especially for deployment scenarios where inference cost dominates total cost.

The negative result on SuperBPE — achieving ~29% better English compression and ~27% better reasoning compression but no downstream performance improvement — is a small but practically valuable contribution. It suggests that tokenizer compression efficiency and model quality are not as tightly coupled as one might assume from first principles, and it may save other teams from investing engineering effort in SuperBPE adoption without commensurate capability gains. This result aligns with a broader pattern in the literature where tokenizer innovations that improve compression metrics do not reliably improve modeling metrics, but the paper provides a concrete, quantified data point at a specific scale and vocabulary size.

Research directions that become more attractive after this work include: (1) exploring convergence-aware load balancing mechanisms beyond SMEBU (e.g., adaptive update magnitudes, learned balancing policies, annealing schedules for bias update rates), (2) developing more comprehensive batch quality metrics beyond BatchHet (e.g., metrics that capture distribution shift rather than just per-step heterogeneity, or metrics that predict gradient norm spikes), and (3) characterizing the sparsity-performance frontier across a range of total-to-active ratios rather than at isolated points. Directions that become less attractive (or at least, whose limitations are better understood) include: (1) expecting sign-based aux-loss-free balancing to scale to very large expert counts without modification, (2) assuming that better tokenizer compression reliably improves downstream model quality, and (3) treating document packing as a solved problem whose details do not affect training dynamics.

Follow-Up Research This Work Enables

Isolating SMEBU's contribution through controlled small-to-medium-scale comparison. The paper's most significant evidential gap is that SMEBU was deployed as one of six simultaneous fixes. A controlled experiment comparing SMEBU against sign-based bias updates at a scale where multiple seeds are affordable — say, a 1B-parameter MoE model with 64-128 experts, trained on 100-500B tokens, with all other architectural and training choices held constant — would establish whether the convergence argument translates to measurable differences in MaxVio trajectories, expert collapse probability, training loss smoothness, and final model quality. The key measurements: (a) MaxVio over time for both methods with the same initialization and data order, (b) number of experts that collapse (load fraction < 0.1/N_r) by end of training, (c) terminal validation loss, and (d) downstream benchmark performance. A strong result would show SMEBU maintaining lower MaxVio variance and zero expert collapse while sign-based updates exhibit occasional or frequent collapse events. A weak or null result — SMEBU providing no measurable benefit over sign-based updates at this scale — would suggest the stabilization in Trinity Large came primarily from the other five fixes, and would redirect attention toward those interventions. Either outcome is informative given the current ambiguity.

Characterizing the scaling limit of sign-based load balancing. The paper hypothesizes that sign-based bias updates become problematic as expert count increases because "the per-layer norm of the bias step also increases" (Section 2.3). This can be tested directly: train MoE models at fixed total parameters and fixed architecture but with varying numbers of routed experts (e.g., 32, 64, 128, 256, 512), using standard sign-based aux-loss-free balancing, and measure at what expert count MaxVio variance begins to increase and expert collapse begins to occur. The experiment should control for total FLOPs per token (by adjusting expert size inversely with expert count) to isolate the routing dynamics from changes in model capacity. The prediction: there exists a threshold expert count beyond which sign-based updates become unstable, and this threshold should be predictable from the bias update magnitude relative to the routing score distribution. If no such threshold is found — if sign-based updates remain stable even at 512 or 1024 experts with proper hyperparameter tuning — then the paper's diagnostic is incorrect and the instability it observed had other causes. This experiment would provide the first empirical characterization of a design constraint that, if real, affects all future work on very-large-expert-count MoE models.

BatchHet as a predictor of training instability: a prospective study. The paper demonstrates that RSDB reduces BatchHet and that lower BatchHet correlates with smoother loss curves and lower gradient norm kurtosis (14.6 vs. 187). But this is a retrospective correlation on a single architecture and data mixture. A prospective study would: (a) implement BatchHet monitoring in a training framework, (b) intentionally induce batch heterogeneity of varying severity (by manipulating document shuffling, packing strategy, or data mixture granularity) in small-scale training runs, and (c) measure whether BatchHet at early training steps predicts later training outcomes — loss spikes, gradient norm explosions, or expert collapse. The key question: is BatchHet a leading indicator of instability, or merely a concurrent correlate? If it is predictive, training runs could use BatchHet thresholds to trigger automatic interventions (e.g., temporarily reducing learning rate, increasing batch size, or switching to a more aggressive document shuffling strategy) before instability manifests. If it is only concurrent, its practical value is limited to post-hoc analysis. The Trinity paper provides the metric and the initial correlation evidence; the prospective study would establish whether it is actionable. A strong result would show that BatchHet in the first 1-10% of training predicts instability events in the later stages with useful accuracy, enabling preventative rather than reactive stabilization.

Decomposing inference throughput by architectural component. The paper attributes Trinity Large's inference throughput to "extreme sparsity and interleaved local/global attention" (Section 5.2) but provides no decomposition. A systematic throughput ablation would measure tokens-per-second on identical hardware (8xH200, FP8, vLLM) for: (a) Trinity Large as-is, (b) Trinity Large with all attention layers converted to global attention (removing the local/global pattern), (c) Trinity Large with all layers converted to dense (removing MoE sparsity but keeping total parameters constant — likely infeasible at 400B, so perhaps a scaled-down version), (d) Trinity Large with GQA replaced by MHA, and (e) Trinity Large at sequence lengths from 1K to 128K. The output would be a breakdown of what fraction of the throughput advantage comes from each architectural choice, and how those fractions change with sequence length. The paper's hypothesis — that the local/global pattern provides the most benefit at long sequences — would be directly tested. This experiment would give practitioners concrete guidance on which architectural features to prioritize for their own throughput-constrained deployments, rather than having to adopt the entire Trinity design without knowing which components matter most.

Sparsity sweep at fixed total training compute. The Trinity Large configuration (30:1 total-to-active ratio) is one point in a design space. To understand whether this point is near-optimal, train a family of MoE models at fixed total training FLOPs (or fixed total training tokens with identical data) but varying the total-to-active parameter ratio while keeping total parameters constant. For example, with 400B total parameters, compare: 50B activated (8:1 ratio, fewer total experts), 25B activated (16:1), 13B activated (30:1, the Trinity Large point), and 7B activated (~57:1, even sparser). Each configuration would require different expert sizes and counts, and potentially different load balancing hyperparameters. The key measurement: downstream benchmark performance as a function of sparsity ratio, at constant total training compute. The prediction from standard scaling intuition: performance should degrade as sparsity increases because routing error accumulates. The Trinity results suggest this degradation may be milder than expected at least up to 30:1. This experiment would map the sparsity-performance Pareto frontier and identify whether Trinity Large is near the knee of the curve (where further sparsity would sharply degrade performance) or on a relatively flat region (suggesting even sparser models could work). Given the training cost, this would likely need to be done at smaller scale (e.g., 50B-100B total parameters) with the expectation that the qualitative shape of the frontier transfers.

Stress-testing length extrapolation with reasoning benchmarks at 512K. The paper's most striking individual result is Trinity Large's MK-NIAH score of 0.976 at 512K — zero-shot, without training at that length. However, MK-NIAH measures retrieval, not reasoning. The natural stress test is to evaluate Trinity Large on long-context reasoning benchmarks that require: (a) multi-hop reasoning over information distributed across a 512K-token context (e.g., the multi-hop NIAH variant from RULER, or custom benchmarks that require integrating facts from distant locations), (b) long-document question answering where the answer requires synthesizing information from multiple sections of a book-length document, and (c) repository-scale code understanding where the model must answer questions about a codebase whose files total hundreds of thousands of tokens. If Trinity Large performs well on these tasks at 512K (not just retrieval), the local/global attention design has demonstrated a genuinely useful length generalization capability that would influence architecture design for long-context models. If it performs poorly — the retrieval works but reasoning collapses at lengths beyond training — then the MK-NIAH result is a narrow capability that does not translate to practical long-context applications, and the field should be cautious about interpreting NIAH scores as evidence of general long-context competence. Either result is valuable; the current paper provides only the NIAH data point, leaving the practical significance ambiguous.

Practical Applications and Downstream Use Cases

Cost-sensitive batch inference on large document collections. An organization processing millions of documents — legal contracts, scientific papers, financial reports, customer support transcripts — faces a direct tradeoff between model quality and inference cost. Trinity Large's 30:1 sparsity ratio means the per-token inference cost is approximately that of a 13B dense model, while the benchmark scores (MMLU 82.58, GPQA Diamond 43.94 in base form; MMLU 87.21, GPQA Diamond 63.32 in preview form from Tables 3-4) are competitive with models in the 30-70B active parameter range. For a document processing pipeline that must run continuously at scale, the throughput advantage measured in Figure 4 (quantified on 8xH200 with FP8) translates directly to lower hardware requirements or higher throughput per GPU. A concrete calculation: if a 70B dense model achieves comparable accuracy but requires 2-3× more GPU-hours for the same document volume (due to higher per-token FLOPs and larger KV-cache), switching to Trinity Large could reduce inference costs by 50-67% at equivalent quality, or alternatively allow 2-3× more documents processed within the same hardware budget. The key caveat is that this assumes the Preview model's quality on the specific document processing task matches the general benchmarks — verification on task-specific evaluation data is essential.

On-device or edge deployment with extreme compression. The Trinity Nano configuration — 6B total parameters, 1B activated per token — combined with the local/global attention pattern (which enables efficient processing of long contexts by limiting local attention window to 2048 tokens) is explicitly designed for resource-constrained environments. The 1B activated parameter count is small enough to run on consumer GPUs, edge accelerators, or potentially high-end mobile devices with appropriate quantization beyond FP8. The MK-NIAH score of 0.864 at 128K context for Nano demonstrates that even the smallest model in the family can locate information in long documents, enabling applications like on-device document search, privacy-preserving email or message summarization (where data never leaves the device), or real-time transcription analysis. The practical benefit over a dense 1B model: Nano has 6B total parameters, giving it substantially more representational capacity than a 1B dense model, while the inference cost is comparable. A deployment scenario where the user's data cannot leave the device for privacy or regulatory reasons — healthcare, legal, financial services — can run Nano locally for routine document processing and annotation, with optional escalation to cloud-hosted Trinity Large only for complex queries that exceed Nano's capability threshold.

Self-improvement data generation pipelines. The clean training stability (zero loss spikes across all three models) and the scaling ladder methodology enable a specific downstream use case: using Trinity models to generate high-quality training data for fine-tuning or distillation. A team building a domain-specific model can use Trinity Large to generate synthetic training examples, reasoning traces, or instruction-following demonstrations, then fine-tune a smaller model (including Trinity Nano or Mini) on the generated data. The advantage over using a proprietary API for data generation: (a) no per-token cost, enabling generation of billions of tokens without budgetary constraints, (b) full control over output distribution (temperature, sampling parameters, system prompts) without API limitations, and (c) data provenance clarity — all generated data has a documented open-weight source, simplifying licensing and compliance for the downstream model. The Preview model's agentic coding trajectories (generated through OpenCode, Section 4) provide a template: the harness collects full interaction traces including edits, tool calls, and test outcomes, which can be used to train models that learn the edit-run-test loop rather than only final completions. A practical pipeline: generate 100M coding trajectories with Trinity Large Preview using OpenCode, filter for trajectories where all tests pass, and fine-tune Trinity Mini on the filtered trajectories to produce a code-specialized 3B-active model that approximates the coding behavior of the 13B-active teacher at a fraction of the inference cost.

Long-context research and development platforms. Trinity Large's demonstrated length extrapolation to 512K (0.976 MK-NIAH) with the potential for further extension to 1M (Section 3.5) makes it a practical platform for developing and testing long-context applications. The model is open-weight and available on HuggingFace, meaning researchers can download it, run it on their own hardware, and experiment with long-context prompting, retrieval strategies, and reasoning techniques without API rate limits, token costs, or context length restrictions. For researchers exploring questions like "what is the effective reasoning horizon of current architectures?" or "how does instruction-following degrade with context length?", having a model that can process 256K tokens (and plausibly 512K) without proprietary access barriers enables experimental designs that would be prohibitively expensive or simply impossible with API-gated models. The 0.42 MK-NIAH at 1M, while below practical usability, also provides a starting point for research into what architectural or training changes could push usable context toward the million-token regime. The transparency of the architecture (detailed in Section 2) means researchers can directly study how the local/global attention pattern handles length extrapolation, rather than treating it as a black box.

When to Prefer This Method

The paper does not position Trinity against a specific named alternative with a clear decision rule. Rather, it presents Trinity as a demonstration that a particular combination of architectural choices (extreme MoE sparsity, interleaved local/global attention, gating, Muon optimizer, SMEBU load balancing, depth-scaled sandwich norm, and large-scale synthetic data) can produce competitive open-weight models with strong inference efficiency. The implicit comparison is against: (a) dense models of comparable capability (which would have much higher active parameter counts and inference costs), and (b) other sparse MoE models with lower sparsity ratios (which would have higher active parameter counts for equivalent total capacity). Since the paper does not articulate explicit criteria for choosing Trinity's specific recipe over, say, the DeepSeek-V3 architecture or a dense model trained with AdamW, a formal "prefer A when X, prefer B when Y" tradeoff matrix would be fabricating a comparison the authors did not make.

The closest the paper comes to an articulated tradeoff is the discussion of expert granularity in Section 2.3: Trinity Large uses coarser experts (4 activated, expert size 3072) compared to Nano and Mini (8 activated, expert sizes 256 and 1024 respectively), motivated by "throughput requirements." This implies a general principle — prefer fewer, larger activated experts when inference throughput is the binding constraint; prefer more, smaller activated experts when model capacity per activated parameter is the binding constraint — but the paper does not test or quantify this tradeoff directly, and it is specific to one architectural axis rather than a holistic comparison between alternative approaches.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Scaling Model Capacity Without Scaling Inference Cost

Why Sparse Mixture-of-Experts and Why Now

Where Existing MoE Approaches Fall Short

The Open-Weight Imperative

How This Paper Positions Itself

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 Data Pipeline and the Random Sequential Document Buffer (RSDB)

3.4.2 Tokenizer Design

3.4.3 Attention Mechanism

3.4.4 Mixture-of-Experts Layer

3.4.5 Normalization and Initialization

3.4.6 The Muon Optimizer and Training Hyperparameters

3.4.7 Context Extension

4. Key Insights and Innovations

Innovation 1: Diagnosing MoE Training Instability as a Convergence Problem, Not Just a Load-Balancing Problem

Innovation 2: Batch Heterogeneity as a Diagnosable, Measurable Contributor to Training Instability

Innovation 3: Extreme Sparsity as a Practical Frontier Strategy — The System-Level Argument

Innovation 4: The Scaling Ladder as a Validation Methodology

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Benchmark Performance and Resource Efficiency (Figure 3, Tables 3–4)

Tokenizer Efficiency (Table 1)

Context Extension Capabilities (Section 3.5)

Inference Throughput (Figure 4)

Training Dynamics: BatchHet Reduction with RSDB (Section 3.2)

Training Stability and the Stabilization Interventions (Section 6)

Ablation Studies and Robustness Checks

Critical Assessment

6. Limitations and Trade-offs

6.1 SMEBU Cannot Be Isolated as the Causal Factor in Training Stability

6.2 Performance Evaluated on a Narrow Benchmark Suite Without Long-Context or Multilingual Validation

6.3 The Sample Efficiency Claim for Muon Is Unsubstantiated by In-Paper Comparisons

6.4 Inference Throughput Gains and Efficiency Claims Are Not Quantitatively Decomposed

6.5 The RSDB Is Evaluated on Training Dynamics Metrics, Not Final Model Quality

6.6 Post-Training Is Acknowledged as Preliminary, Leaving Open Questions About Production Readiness

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

When to Prefer This Method

Adjacent in domain

Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters

ERNIE 5.0 Technical Report