MiniMax-01: Scaling Foundation Models with Lightning Attention

ArXiv: 2501.08313

🎯 Pitch

This paper shows a 456B-parameter MoE hybrid that swaps most softmax attention for a cheaper linear variant can still match GPT-4o and Claude 3.5 Sonnet—while supporting 4-million-token contexts, over 20× longer than those competitors, at no loss in quality.

---

1. Executive Summary

This paper introduces the MiniMax-01 series—MiniMax-Text-01 and MiniMax-VL-01—foundation models that match top-tier commercial performance while supporting context windows an order of magnitude longer than competitors, reaching up to 4 million tokens during inference. Built on a hybrid architecture that interleaves lightning attention (an I/O-aware linear attention implementation that reduces computational complexity from quadratic to linear by decomposing attention into intra-block left products and inter-block right products) with standard softmax attention at a 7:1 ratio, and combined with a Mixture of Experts (MoE) design comprising 456 billion total parameters with 45.9 billion activated per token across 32 experts, the models achieve state-of-the-art results on standard benchmarks—88.5% on MMLU, 77.4% on MATH, and 86.9% on HumanEval—while delivering long-context RULER scores above 0.90 at 1 million tokens, a regime where competing models either cannot operate or degrade substantially. The paper demonstrates through FLOPs-matched comparisons that the hybrid architecture achieves lower training loss than pure softmax attention under identical compute budgets, establishing that linear attention can substitute for traditional attention at commercial scale without performance degradation, but only when augmented with periodic softmax layers to restore the retrieval capabilities that pure linear attention inherently lacks.

2. Context and Motivation

The Core Problem: Context Window Expansion Has Hit a Computational Wall

The central problem this paper addresses is deceptively straightforward: current state-of-the-art language models cannot efficiently process contexts longer than roughly 256K tokens, yet practical applications demand much longer sequences. Using a professional book as context, assisting with an entire programming project, or maximizing in-context learning through many-shot examples all require context windows spanning millions of tokens. The gap between what existing models offer and what real-world use cases demand is substantial — roughly a 20-32x difference, as the paper quantifies.

This is not merely an inconvenience. It is a fundamental architectural limitation baked into the transformer design that has dominated the field since Vaswani et al. (2017). The quadratic computational complexity of softmax attention — where processing a sequence of length $N$ requires $O(N^2)$ operations — means that doubling the context length quadruples the computational cost. The paper frames this bluntly in Section 1:

"further length extension causes computational demands to grow much faster than hardware capabilities can match"

The practical consequences are severe. Models like GPT-4o, Claude-3.5-Sonnet, and Gemini support contexts ranging from 32K to 256K tokens, but as Figure 1(c) and Figure 2 demonstrate, their performance degrades substantially at longer lengths — and in some cases, they simply cannot operate at all beyond certain thresholds. The prefilling latency graph in Figure 2 shows this visually: as context length increases, softmax attention models exhibit quadratic growth in latency, rendering million-token contexts impractical even if theoretically supported.

Why Context Window Expansion Matters Beyond Convenience

The paper establishes that longer context windows unlock qualitatively different capabilities, not just quantitatively more input:

• In-context learning at scale. The MTOB (Machine Translation from One Book) experiment in Section 5.7.2.3 demonstrates this concretely. By providing an entire grammar book (~133K tokens) plus 375 parallel translation examples as context, the model can learn to translate between English and Kalamang — a language essentially absent from its training data — purely from context, achieving performance competitive with models that may have been explicitly trained on Kalamang data. This capability transforms the model from a static knowledge store into a learning system that acquires new skills at inference time.

• Long-document understanding and reasoning. Section 5.7.2.2 shows that tasks involving multi-hop tracing, aggregation across documents, and complex reasoning over extended contexts (evaluated via RULER and LongBench-V2) remain challenging for current models. Yet these are precisely the capabilities needed for professional document analysis, legal review, scientific literature synthesis, and codebase comprehension.

• Lifelong assistant scenarios. The MR-NIAH (Multi-Round Needles-In-A-Haystack) task described in Section 5.7.2.1 evaluates whether a model can retrieve specific information from conversation histories spanning up to 2,000 interactions. This capability is foundational for AI assistants that maintain coherent, memory-grounded interactions over months or years — the kind of "lifelong companion AI" the paper explicitly envisions.

The paper also implicitly addresses an economic and deployment consideration. Section 2.4 frames the model size constraint around a practical goal: the ability to process more than 1 million tokens on a single machine with 8 GPUs and 640GB memory using 8-bit quantization. This is not an arbitrary benchmark — it represents a deployment target where long-context inference is affordable and practical rather than requiring prohibitively expensive multi-node configurations.

Prior Approaches and Their Limitations

The paper traces a landscape of attempted solutions to the quadratic complexity problem, each with demonstrable shortcomings that explain their limited adoption at commercial scale.

Sparse Attention: Trading Coverage for Efficiency

Approaches like Longformer (Beltagy et al., 2020) and Big Bird (Zaheer et al., 2020) reduce complexity by restricting each token to attend only to a subset of other tokens — typically a local window plus a few global tokens. While this achieves linear or near-linear complexity, it introduces a fundamental tradeoff: the model can miss critical long-range dependencies if the sparse pattern does not align with the task's information needs. Retrieval tasks like Needle-in-a-Haystack (NIAH) expose this weakness directly — if the "needle" falls outside the attended window, it is invisible to the model. The paper does not extensively evaluate sparse attention variants but references them as part of the landscape of incomplete solutions.

State Space Models and Linear RNNs: The Retrieval Gap

The Mamba series (Dao and Gu, 2024; Gu and Dao, 2024) and related state space models represent a more promising direction, achieving linear complexity while modeling long-range dependencies through structured state transitions. However, the paper's analysis in Section 2.2.2 and Figure 7 reveals a critical limitation: pure linear models exhibit severely degraded retrieval capabilities.

Figure 7 shows this starkly. On the NIAH (Needle-in-a-Haystack) benchmark at the 3B parameter scale, lightning attention achieves 98.0% accuracy while softmax attention achieves 84.2% — a counterintuitive result the paper explains in Section 2.2.4. But when the paper evaluates pure lightning attention (without any softmax layers), retrieval performance collapses:

"lightning attention demonstrates comparable performance across most downstream tasks, with the exception of NIAH. This indicates that linear attention exhibits similar language modeling capabilities to Transformer models but falls short in retrieval tasks, rendering it unsuitable for LLMs."

This finding is crucial because it explains why state space models and linear RNNs have seen limited adoption in production LLMs despite their theoretical efficiency advantages. Language models need more than just efficient processing — they need the ability to selectively retrieve and attend to specific pieces of information from a large context, and this capability is inherently tied to the softmax attention mechanism's ability to compute sharp, content-dependent attention weights.

The paper's insight about why this gap exists is revealed in Eqs. 10–12. Softmax attention can be rewritten as a linear recurrence where, at each time step, the hidden state is recomputed from the beginning — "Going Through a Book," as the paper phrases it. This systematic revisiting of prior data enables accurate information retention. Linear models lack this recomputation mechanism, reducing their effective retrieval capacity despite having larger theoretical capacity ($O(d^2/h)$ vs. $O(d)$ for softmax attention, where $d$ is the feature dimension and $h$ is the number of heads).

Sliding Window Attention: A Limited Middle Ground

The paper evaluates sliding window attention as an alternative linear-complexity approach in Section 2.2.3 and Table 4. By restricting attention to a fixed window of tokens and periodically inserting full-attention layers, this approach can achieve linear complexity while retaining some retrieval capability. However, the comparison reveals clear disadvantages over the hybrid-lightning architecture:

• At 1B parameters with a 1024-token window, hybrid-window achieves 53.9% NIAH accuracy versus 95.7% for hybrid-lightning — a massive gap in retrieval capability.
• Larger window sizes improve retrieval but reduce training speed, making them less efficient than hybrid-lightning at equivalent throughput.
• Even at the largest window evaluated (1024 tokens), hybrid-window trains at 33.6K TGS (tokens per GPU per second) versus 33.4K for hybrid-lightning — comparable speed but dramatically inferior retrieval.

This comparison demonstrates that sliding window is not a satisfying solution — it either sacrifices too much retrieval capability (small windows) or too much speed (large windows), while the hybrid-lightning approach achieves both strong retrieval and competitive speed simultaneously.

Existing Linear Attention Implementations: Theoretical but Not Practical

The paper acknowledges that linear attention variants have existed for nearly a decade — notably, the "right product kernel trick" that transforms $O(N^2)$ into $O(Nd^2)$ complexity was proposed by de Brébisson and Vincent (2016). However, the paper identifies a specific implementation bottleneck that prevented practical adoption:

"When addressing causal language modeling tasks, the efficacy of the right product is compromised, necessitating the computation of cumsum... This limitation impedes the realization of highly efficient parallel computation, which likely explains why, despite being proposed... nine years ago, none of the current leading open-source LLMs—including LLaMA3, Qwen2.5, DeepSeekV3, and Mistral—have adopted this linear attention mechanism."

The cumsum operation required for causal masking in linear attention is inherently sequential and cannot be efficiently parallelized. Lightning attention (Qin et al., 2024b,c) specifically addresses this by decomposing the computation into left-product intra-block operations (which can use efficient matrix multiplication) and right-product inter-block operations (which maintain the linear recurrence), using a tiling technique that "effectively circumvents the cumsum operation" (Section 2.2.1). This is the key engineering insight that transforms linear attention from a theoretical curiosity into a practical mechanism — but the paper emphasizes that it had not been tested at commercial scale before this work.

Best-of-N and Other Mitigations: Treating Symptoms, Not Causes

The paper does not extensively discuss inference-time mitigation strategies like best-of-N sampling or prompt compression, but the implicit contrast is clear: these approaches work around the quadratic complexity problem by limiting what gets processed, rather than addressing the complexity itself. A model with a 32K context window can be made to work on a 1M-token document by chunking, summarizing, or retrieving relevant segments — but these workarounds are lossy and introduce failure modes. The paper's approach of directly enabling million-token contexts through architectural innovation is fundamentally different: it preserves the model's ability to attend to any token in the entire context with equal fidelity.

How This Paper Positions Itself

The paper's positioning can be understood through three key claims, each of which represents a departure from prevailing assumptions in the field.

Claim 1: Linear Attention Can Work at Commercial Scale — But Only in Hybrid Form

The prevailing assumption in the field is that state-of-the-art language models must be built on softmax attention. The paper directly challenges this, but with an important qualification that distinguishes it from pure linear attention advocates. Section 2.2.2.3 establishes through systematic scaling experiments (70M to 7B parameters, up to 300B training tokens) that:

• Pure lightning attention is not a viable LLM backbone because of its retrieval limitations. This is an honest negative result that the paper does not try to hide — Figure 7 shows lightning attention's NIAH scores far below softmax attention at all scales.

• Hybrid-lightning attention matches and exceeds softmax attention on both retrieval and reasoning benchmarks, while training faster (Figure 8) and achieving lower loss under identical compute budgets (Table 2, Figure 6).

This positions the paper not as an advocate for abandoning softmax attention entirely, but as demonstrating that strategic integration — one softmax layer for every seven lightning attention layers — captures the efficiency benefits of linear attention while preserving the retrieval capability that softmax attention uniquely provides. The paper's contribution is demonstrating this at a scale (456B parameters, 1M-token training length) that no prior linear attention system has approached.

Claim 2: Long Context and Strong Performance Are Not Mutually Exclusive

A common narrative in the field is that long-context models inevitably sacrifice short-context performance — that there is a tradeoff between context window size and benchmark scores. The paper contradicts this directly, showing (Figure 1, Table 8) that MiniMax-Text-01 matches or exceeds GPT-4o, Claude-3.5-Sonnet, and Gemini on standard benchmarks like MMLU (88.5%), MATH (77.4%), and HumanEval (86.9%) while simultaneously supporting 20-32x longer contexts.

This is not an accidental finding — the paper's architecture was specifically designed to decouple long-context capability from model capacity allocation. The MoE design with 45.9B activated parameters per token means that the computational cost of processing each additional token is approximately constant regardless of total context length, unlike dense models where the attention cost grows quadratically. The hybrid architecture ensures that only one-eighth of layers pay the full quadratic attention cost, while the remaining seven-eighths benefit from lightning attention's linear scaling.

Claim 3: Scaling Laws Apply to Linear Attention Architectures — With Different Coefficients

The paper conducts what appears to be the first systematic scaling law analysis for linear and hybrid attention architectures (Section 2.2.2.2, Table 2, Figure 6). The findings are nuanced and important:

• Hybrid-lightning achieves the lowest loss exponent ($\alpha_C = -0.0763$ vs. $-0.0798$ for softmax attention), meaning it extracts more performance improvement per unit of additional compute.
• Optimal model size scales differently: $N_{opt} \propto C^{0.667}$ for hybrid-lightning versus $C^{0.712}$ for softmax attention, suggesting that hybrid architectures favor somewhat smaller models trained on more data.
• Optimal data volume scales more slowly: $D_{opt} \propto C^{0.471}$ for hybrid-lightning versus $C^{0.510}$ for softmax attention.

The practical implication is that hybrid-lightning models use their compute budget differently — they achieve lower loss by allocating proportionally more to data and less to parameters compared to softmax attention models. This has direct consequences for the model specification decisions in Section 2.4, where the paper uses these scaling relationships to determine the 45.9B activation / 456B total parameter configuration.

The "Ladder to 1M Tokens" Strategy

The paper's approach to long-context training (Section 4.2, Table 6) reveals a practical methodology that addresses a common failure mode: naively training on million-token sequences from the start can lead to instability and poor convergence. Instead, the paper employs a three-stage curriculum:

1. 128K phase: 300B tokens, 70% short (<32K) / 30% medium (32K-128K) data, RoPE base frequency 5M
2. 512K phase: 32B tokens, balanced distribution across short/medium/long, RoPE base 10M
3. 1M phase: 26B tokens, 30% short / 30% medium / 40% long (>128K), RoPE base 10M

This progressive extension, combined with 10% high-quality long-context QA data mixed in during the last 20% of each stage, allows the model to gradually adapt its positional encoding to longer sequences without catastrophic forgetting of short-context capabilities. Importantly, the paper notes that despite training only up to 1M tokens, the model's length extrapolation capabilities (enabled by applying RoPE to only half the attention head dimensions) allow it to process 4M tokens at inference time (Figure 14).

The Gap Between Theory and Practice

Perhaps the paper's most significant positioning move is its emphasis on practical deployability as a first-class design constraint. Section 2.4 frames the model specification as an optimization problem (Eq. 13) that explicitly trades off performance against the constraint of fitting on a single node with 8×80GB GPUs at 1M-token sequences under 8-bit quantization. This ensures that the resulting model is not merely a research demonstration but a system that can be deployed in production without requiring exotic hardware configurations.

Section 3 details the engineering work required to make the architecture efficient at scale — optimized all-to-all communication for MoE, varlen ring attention for variable-length sequences, and custom CUDA kernels for lightning attention inference achieving over 75% MFU on H20 GPUs. The paper implicitly argues that these engineering contributions are as important as the architectural innovations themselves, and that prior linear attention approaches failed to gain adoption partly because of inadequate engineering optimization.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

MiniMax-Text-01 is a 456-billion-parameter language model that processes text using a hybrid attention mechanism — seven out of every eight layers use an efficient “linear” attention that costs roughly the same regardless of input length, while every eighth layer uses standard (quadratic-cost) softmax attention to preserve retrieval accuracy. The problem it solves is the quadratic compute explosion that makes standard transformers prohibitively expensive on million-token contexts; the “shape” of the solution is a Mixture of Experts (MoE) architecture where linear attention handles the bulk of long-range processing, periodic softmax layers restore the ability to pinpoint specific information, and custom parallelisation strategies make the whole system trainable and deployable at commercial scale.

---

3.2 Big-Picture Architecture (Diagram in Words)

At the highest level, MiniMax-Text-01 is a stack of 80 transformer-style blocks, each containing an attention module (a “channel mixer”) followed by a feed-forward module (a “feature mixer”). The system has five major components:

1. Lightning Attention Layers — 70 of the 80 layers use an I/O-aware implementation of linear attention that reduces the per-layer cost from $O(N^2 d)$ to $O(N d^2)$, where $N$ is the sequence length and $d$ is the feature dimension. These layers process long sequences efficiently but have limited retrieval capability.

2. Softmax Attention Layers — 10 of the 80 layers (one every eighth block) use standard multi-head attention with Group Query Attention (GQA, 8 query groups) and Rotary Position Embeddings (RoPE) on half the head dimensions. These layers restore the model's ability to pinpoint and retrieve specific information from the full context.

3. Mixture of Experts (MoE) Feed-Forward Networks — Every layer's feature mixer is an MoE with 32 experts, each a feed-forward network with hidden dimension 9216, using top-2 routing with a token-drop capacity limit. This multiplies parameter capacity without proportionally increasing compute, since only 2 of 32 experts are activated per token.

4. Global Router with Auxiliary Loss — A learned gating network assigns tokens to experts, balanced by an auxiliary loss (coefficient 0.01) that penalises uneven expert utilisation, plus a global routing strategy that synchronises token counts across expert-parallel groups before dispatching, reducing the overall token drop rate.

5. Distributed Training and Inference Framework — Custom parallelisation schemes (Expert Tensor Parallel, Expert Data Parallel, Context Parallel with varlen ring attention, and an improved LASP+ for lightning attention) that overlap communication with computation and enable training on contexts up to 1 million tokens and inference up to 4 million on a single 8-GPU node.

Information flows as follows: input tokens enter the embedding layer → the 80-block stack processes them sequentially, with blocks 1–7 using lightning attention + MoE, block 8 using softmax attention + MoE, and this 8-block pattern repeating 10 times → the output hidden state passes through a final RMSNorm and linear projection to produce next-token logits. During training, the MoE router dynamically selects 2 of 32 experts per token; during inference, the lightning attention layers' recurrent state (the $\mathbf{K}^\top \mathbf{V}$ matrix) can be cached, enabling constant-time per-token decoding regardless of prefix length.

---

3.3 Roadmap for the Deep Dive

• First, the Mixture of Experts design (Section 2.1) — how tokens are routed, how load balancing works (auxiliary loss + global router), and why token-drop MoE is chosen over dropless alternatives. This is the foundation that enables the model to scale to 456B parameters while keeping per-token compute manageable.

• Second, linear attention from first principles (Section 2.2) — the “right product kernel trick” that transforms quadratic into linear complexity, the cumsum bottleneck that prevented adoption, and the tiling strategy in lightning attention that resolves it. This is the core algorithmic innovation.

• Third, the scaling law experiments (Section 2.2.2) that compare softmax, lightning, and hybrid-lightning attention across model sizes from 70M to 7B parameters, establishing that hybrid-lightning achieves the lowest loss under equal compute and revealing the retrieval gap in pure linear attention.

• Fourth, the hybrid architecture verification (Sections 2.2.3–2.3) — comparisons against alternative hybrid configurations (cosformer2, HGRN2, sliding window), ablation of pre-norm vs. post-norm with DeepNorm, and the final module choices within the MoE framework.

• Fifth, the model specification optimisation (Section 2.4) — the constrained optimisation problem that determines the 45.9B activation / 456B total parameter configuration, informed by scaling laws and practical deployment constraints.

• Sixth, the computation optimisation (Section 3) — the custom parallelisation strategies for MoE (EP-ETP overlap), long-context training (varlen ring attention, LASP+), and lightning attention inference (batched kernel fusion, separated prefill/decoding, multi-level padding).

---

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and architecture paper whose core idea is that a hybrid attention architecture — combining linear attention for efficient long-sequence processing with periodic softmax attention for retrieval capability — can support context windows an order of magnitude longer than standard transformers while matching top-tier commercial performance, provided the engineering framework is rebuilt from the ground up to support this novel architecture at scale.

---

Mixture of Experts: Scaling Parameters Without Scaling Compute Per Token

Mixture of Experts (MoE) replaces the conventional dense feed-forward network in each transformer layer with multiple parallel feed-forward networks (experts), where each input token is routed to only a subset of these experts. This enables the model to have many more total parameters than activated parameters — in MiniMax-Text-01, 456 billion total parameters but only 45.9 billion activated per token — meaning the model can store vastly more knowledge and capability while keeping the per-token computational cost proportional to only the activated subset.

Token routing mechanism. For each input token $\mathbf{x}_t$ at a given layer, the routing decision is computed as:

$\mathbf{h}_t = \sum_{i=1}^{E} \text{Softmax}_i\Big(\text{TopK}(\mathbf{x}_t \cdot \mathbf{W}_g)\Big) \cdot \text{FFN}_i(\mathbf{x}_t)$

where $E = 32$ is the total number of experts, $\mathbf{W}_g$ is the learned gating weight matrix, $\text{TopK}(\cdot)$ preserves the top $k = 2$ scores among all $E$ experts while setting the remaining scores to $-\infty$ (so their softmax weight becomes zero), and $\text{FFN}_i$ is the $i$-th expert feed-forward network with hidden dimension 9216.

What it computes: For each token, the gating network computes 32 scalar scores (one per expert), selects the top 2, applies softmax to normalise those two into a probability distribution, computes each selected expert's output (a 6144-dimensional vector), and returns the weighted sum of the two expert outputs. Tokens routed to the remaining 30 experts produce zero contribution — those experts are not computed at all for that token, saving both compute and memory.

Why top-2 routing: Top-1 routing (used in Switch Transformers) activates only one expert per token, maximising efficiency but making the model fragile to routing errors — if the wrong expert is selected, the token gets no benefit. Top-2 provides redundancy: even if the top choice is suboptimal, the second expert can compensate. Increasing beyond top-2 would increase activated parameters per token, reducing the efficiency gain that MoE provides.

Token-drop strategy with capacity limits. The paper adopts a token-drop approach (as opposed to dropless): each expert is assigned a capacity limit — a maximum number of tokens it can process. Once an expert reaches its capacity, any additional tokens routed to it are discarded (they receive zero contribution from that expert). This is in contrast to dropless strategies that dynamically expand capacity or reseat tokens, which add implementation complexity and can cause memory spikes.

The capacity limit is a hyperparameter that trades off training efficiency against the token drop rate. The paper does not specify the exact capacity value but frames it as a constraint that the global router (described below) helps satisfy more evenly. Without capacity limits, some experts would receive many more tokens than others, creating load imbalance where some GPUs are overloaded while others idle — the capacity limit bounds this imbalance at the cost of potentially dropping a small fraction of tokens.

Why token-drop rather than dropless: Token-drop with capacity limits enables predictable memory consumption and computation time per training step — essential for efficient distributed training at scale. Dropless strategies, while theoretically avoiding any information loss, introduce variable computation time per expert (since the number of tokens per expert varies) and can cause out-of-memory errors when token distribution is highly skewed. The paper's global router is specifically designed to minimise the token drop rate, making the token-drop approach practical even at large scale.

Auxiliary loss for load balancing. To prevent routing collapse — where the gating network learns to send all tokens to a small subset of experts, starving the others — the paper uses an auxiliary loss adapted from GShard (Lepikhin et al., 2021):

$\mathcal{L}_{\text{aux}} = \alpha_{\text{aux}} \cdot \frac{1}{E} \sum_{i=1}^{E} f_i \cdot m_i$

where $\alpha_{\text{aux}} = 0.01$ is the auxiliary loss coefficient, $f_i$ is the fraction of tokens assigned to expert $i$, $m_i$ is the average routing probability for expert $i$ (the mean of the softmax scores assigned to expert $i$ across all tokens), and $E = 32$ is the total number of experts.

What it computes: For each expert, multiply the fraction of tokens it receives ($f_i$) by the average confidence the router has in sending tokens to it ($m_i$), sum across all experts, and scale by the coefficient. Since both $f_i$ and $m_i$ sum to 1 across experts, the minimum of this sum is achieved when each expert receives $1/E$ of tokens AND each has average routing probability $1/E$ — i.e., perfect uniformity. The product form penalises concentrated distributions: if a few experts receive most tokens ($f_i$ is large) AND the router is confident about them ($m_i$ is large), the product is large and the loss increases.

Why this form: The $f_i m_i$ product couples two desiderata — balanced assignment AND balanced confidence. A loss that penalised only $f_i$ (token counts) could be satisfied by the router sending equal tokens to all experts but with near-zero confidence, effectively randomising routing. The product form ensures the router maintains meaningful confidence while balancing assignment, which improves training stability.

Global router: cross-GPU load balancing. A critical challenge in MoE training at scale is that expert parallelism distributes experts across different GPUs (each GPU hosts a subset of experts), and the token distribution within each GPU's micro-batch can be highly non-uniform. This leads to a situation where one GPU's experts are overloaded (exceeding capacity and dropping tokens) while another GPU's experts are underutilised, even if the aggregate load across all GPUs is balanced.

The paper's global routing strategy addresses this by introducing an additional allgather communication step before token dispatching. Specifically:

1. Each expert-parallel (EP) group computes how many tokens are waiting to be processed by each of its locally hosted experts.
2. An allgather operation synchronises these counts across all EP groups, so every GPU knows the global token distribution across all experts.
3. Token dispatching decisions are then made with awareness of global load, allowing the system to route tokens to underutilised experts across EP groups rather than dropping them when local experts are full.

Why this matters: Without global routing, the token distribution within a single GPU's micro-batch can have high variance — some experts might receive zero tokens while others exceed capacity — because the micro-batch size is limited by GPU memory. The allgather effectively pools the token counts across a much larger effective batch, smoothing out the variance and reducing the token drop rate. Under the same capacity constraints, the paper reports that global routing "can effectively reduce the overall token drop rate, thereby ensuring training stability."

MoE vs. Dense: isoflop comparison. Figure 4 provides the justification for MoE over dense architectures: at the same total compute budget (measured in ZFlops, i.e., $10^{21}$ floating-point operations), an MoE model with 2B activated parameters (24B total) consistently outperforms a dense model with 7B parameters on HellaSwag (e.g., ~0.68 vs. ~0.63 at ~9 ZFlops), WinoGrande (0.66 vs. 0.62), Natural Questions (0.14 vs. 0.10), PIQA (0.775 vs. 0.76), and TriviaQA (0.44 vs. 0.35). The performance gap is substantial — the MoE achieves the dense model's best performance using roughly half the compute — and holds across benchmarks measuring commonsense reasoning, factual knowledge, and reading comprehension.

---

Linear Attention: From Quadratic to Linear Complexity

The fundamental computation in standard softmax attention is:

$\mathbf{O} = \text{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d}}\right) \mathbf{V}$

where $\mathbf{Q}, \mathbf{K}, \mathbf{V} \in \mathbb{R}^{n \times d}$ are the query, key, and value matrices, $n$ is the sequence length, and $d$ is the per-head feature dimension. The matrix multiplication $\mathbf{Q}\mathbf{K}^\top$ produces an $n \times n$ attention matrix, costing $O(n^2 d)$ operations and $O(n^2)$ memory — this is the quadratic bottleneck.

The right-product kernel trick. The key observation enabling linear attention is that the computation order can be changed by using a kernel function $\phi$ to map $\mathbf{Q}$ and $\mathbf{K}$ into a space where the attention can be computed via right multiplication:

$\mathbf{O} = \text{Norm}\left(\phi(\mathbf{Q})\left(\phi(\mathbf{K})^\top \mathbf{V}\right)\right)$

Using the TransNormer variant (Qin et al., 2022a) that the paper builds on, the NormAttention mechanism is written as:

$\mathbf{O} = \text{Norm}\left((\mathbf{Q}\mathbf{K}^\top)\mathbf{V}\right)$

which can be transformed into its linear form by changing the multiplication order:

$\mathbf{O} = \text{Norm}\left(\mathbf{Q}(\mathbf{K}^\top \mathbf{V})\right)$

What this form accomplishes: Instead of computing $\mathbf{Q}\mathbf{K}^\top$ (an $n \times n$ matrix, cost $O(n^2 d)$), we first compute $\mathbf{K}^\top \mathbf{V}$ (a $d \times d$ matrix, cost $O(n d^2)$) and then multiply by $\mathbf{Q}$ (another $O(n d^2)$). The total complexity is $O(n d^2)$ — linear in sequence length $n$ — because $d$ is a constant independent of $n$. When $d \ll n$ (typical in LLMs where $d = 128$ per head and $n$ can be millions), this is a massive reduction.

Why this works in principle but not in practice (the cumsum problem): For causal (autoregressive) language modelling, each position $t$ can only attend to positions $\leq t$. The $\mathbf{K}^\top \mathbf{V}$ matrix must therefore be computed as a cumulative sum over time steps:

$\mathbf{kv}_0 = \mathbf{0}, \quad \mathbf{kv}_t = \mathbf{kv}_{t-1} + \mathbf{k}_t \mathbf{v}_t^\top, \quad \mathbf{o}_t^\top = \mathbf{q}_t^\top \mathbf{kv}_t$

This recurrence computes the attention output at each position $t$ using only the keys and values from positions $\leq t$. However, it is inherently sequential — $\mathbf{kv}_t$ depends on $\mathbf{kv}_{t-1}$, which depends on $\mathbf{kv}_{t-2}$, and so on. This cumsum operation cannot be parallelised in its naive form, which means training throughput is bottlenecked by the sequential recurrence even though the total FLOPs are linear. The paper identifies this as the reason why, despite being proposed nine years ago (de Brébisson and Vincent, 2016), linear attention had not been adopted by any leading open-source LLM.

---

Lightning Attention: Solving the Cumsum Bottleneck Through Tiling

Lightning attention (Qin et al., 2024b,c) is an I/O-aware implementation that overcomes the cumsum bottleneck by decomposing the attention computation into two components that exploit different computational patterns.

The key insight: tiling along the sequence dimension. The input matrices $\mathbf{Q}, \mathbf{K}, \mathbf{V}$ are partitioned into $T = n/B$ blocks of size $B \times d$ each, where $B$ is a tunable block size (set to 256 during training). Within each block, attention is computed using the left product $\mathbf{Q}\mathbf{K}^\top$ (which is efficient for small, independent blocks and can be parallelised across GPU cores). Between blocks, attention is computed using the right product $\mathbf{K}^\top \mathbf{V}$ (which maintains the linear recurrence but only crosses block boundaries).

Formally, let $\mathbf{X}$ be partitioned into blocks $\mathbf{X}_1, \mathbf{X}_2, \ldots, \mathbf{X}_T$ where $\mathbf{X}_t \in \mathbb{R}^{B \times d}$ for $\mathbf{X} \in \{\mathbf{Q}, \mathbf{K}, \mathbf{V}\}$. The forward pass computes:

Block 1 (no prior context):

$\mathbf{O}_1 = \underbrace{[(\mathbf{Q}_1 \mathbf{K}_1^\top) \odot \mathbf{M}] \mathbf{V}_1}_{\text{intra-block (left product)}} + \underbrace{\mathbf{Q}_1 \cdot \mathbf{KV}_0}_{\text{inter-block (right product)}}$

where $\mathbf{M} \in \mathbb{R}^{B \times B}$ is a causal mask ($\mathbf{M}_{ts} = 1$ if $t \geq s$, else $0$), $\mathbf{KV}_0 = \mathbf{0} \in \mathbb{R}^{d \times d}$ is the initial key-value state, and $\odot$ denotes element-wise multiplication.

Cumulative KV state update after block 1:

$\mathbf{KV}_1 = \mathbf{KV}_0 + \mathbf{K}_1^\top \mathbf{V}_1$

Block 2 (with prior context from block 1):

$\mathbf{O}_2 = \underbrace{[(\mathbf{Q}_2 \mathbf{K}_2^\top) \odot \mathbf{M}] \mathbf{V}_2}_{\text{intra-block}} + \underbrace{\mathbf{Q}_2 \cdot \mathbf{KV}_1}_{\text{inter-block}}$

and so on for all $T$ blocks. After processing block $t$, the cumulative state is:

$\mathbf{KV}_t = \mathbf{KV}_{t-1} + \mathbf{K}_t^\top \mathbf{V}_t$

What this computes operationally: For each block of $B$ tokens, the algorithm does two things simultaneously. First, it computes the "local" attention within those $B$ tokens using standard left-product matrix multiplication — this is $O(B^2 d)$ per block, but $B$ is chosen small (256) so this is manageable. Second, it retrieves the "global" context from all previous tokens by multiplying $\mathbf{Q}_t$ with the accumulated $\mathbf{KV}_{t-1}$ matrix — this is $O(B d^2)$ per block. The $\mathbf{KV}_{t-1}$ matrix (size $d \times d$) serves as a compressed representation of all prior context — it is the running sum of outer products $\mathbf{k}_i \mathbf{v}_i^\top$ for all tokens before the current block.

Why this tiling solves the cumsum bottleneck: The intra-block computation uses left-product matrix multiplication, which is highly optimised on GPUs (via libraries like cuBLAS) and can be parallelised across blocks because each block's intra-block computation is independent. The inter-block computation requires the cumulative $\mathbf{KV}$ state, but this is only updated once per block (not per token), reducing the sequential dependency from $n$ steps to $T = n/B$ steps. With $B = 256$, the sequential chain is $256\times$ shorter. Moreover, the inter-block computation $\mathbf{Q}_t \cdot \mathbf{KV}_{t-1}$ is a matrix multiplication between a $B \times d$ and a $d \times d$ matrix, which is compute-bound rather than memory-bound at typical block sizes, allowing efficient GPU utilisation.

Time complexity. The final complexity is $O(n d^2 + n B d)$. The $n d^2$ term comes from the inter-block computations (each of $T$ blocks does a $B \times d$ times $d \times d$ multiplication, so $T \cdot B d^2 = n d^2$). The $n B d$ term comes from the intra-block computations (each block does a $B \times d$ times $d \times B$ multiplication, so $T \cdot B^2 d = n B d$). When $B \ll d$ (typical), the inter-block term dominates, and the complexity is effectively $O(n d^2)$ — linear in $n$.

I/O-aware implementation. Algorithm 1 in the paper embodies the "lightning" aspect: for each block, the algorithm loads $\mathbf{Q}_t, \mathbf{K}_t, \mathbf{V}_t$ from high-bandwidth memory (HBM) into on-chip SRAM, computes intra-block attention, inter-block attention, and the KV update entirely on-chip, and writes only the output $\mathbf{O}_t$ back to HBM. This minimises the number of HBM reads/writes, which are the bottleneck on modern GPUs. The cumulative $\mathbf{KV}$ matrix stays in SRAM across block iterations, avoiding expensive HBM transfers for the recurrent state.

Normalisation and gating details. The paper notes that for analytical tractability, the derivation above omits normalisation (RMSNorm), SiLU activation, and gating mechanisms present in the full TransNormer block. These are applied in the actual implementation but do not change the computational complexity — they add element-wise operations that are $O(n d)$ and therefore dominated by the $O(n d^2)$ attention cost.

---

Lightning Attention vs. Softmax Attention: The Scaling Law Analysis

To determine whether lightning attention (and its hybrid variant) can replace softmax attention at scale, the paper conducts a systematic scaling law study comparing three architectures across model sizes from 70M to 7B parameters, each trained on up to 300B tokens at context length 8192.

FLOP accounting methodology. Following Kaplan et al. (2020), the non-embedding FLOPs per token are computed as approximately:

• Softmax attention: $72 b n l d^2 \left(1 + \frac{n}{6d} + \frac{5}{18d}\right)$
• Lightning attention: $72 b n l d^2 \left(1 + \frac{1}{2h} + \frac{5}{18d}\right)$
• Hybrid-lightning: $72 b n l d^2 \left(1 + \frac{n}{48d} + \frac{7}{16h} + \frac{5}{18d}\right)$

where $b$ is the batch size (in tokens), $n$ is the sequence length, $l$ is the number of layers, $d$ is the model dimension, and $h$ is the number of attention heads. The key difference is the $n/(6d)$ term in softmax attention (which grows with sequence length, reflecting the quadratic cost) versus the $1/(2h)$ term in lightning attention (constant with respect to sequence length). The hybrid model's $n/(48d)$ term reflects that only 1/8 of layers incur the quadratic cost, reducing its magnitude by 8×.

Training setup for scaling experiments. All models are trained with the Adam optimiser (learning rate $3 \times 10^{-4}$, weight decay 0.1), a uniform global batch size of 4 million tokens, and a fixed learning rate schedule (no decay due to computational constraints). The training loss at the end of training is taken as the estimator of test performance, following the Chinchilla methodology.

Power-law fits. The relationship between loss $L$ and compute $C$ (in PFlop/s-days) is fitted as:

$L(C) = \beta_C \cdot C^{\alpha_C}$

The optimal model size $N_{\text{opt}}$ and training tokens $D_{\text{opt}}$ as functions of compute are fitted as:

$N_{\text{opt}}(C) \propto C^a, \quad D_{\text{opt}}(C) \propto C^b$

Table 2 and Figure 6 present the fitted coefficients:

What these parameters mean, operationally:

• Loss exponent $\alpha_C$: A more negative value means the loss drops faster as compute increases. Hybrid-lightning achieves −0.0763 versus softmax's −0.0798 — a smaller magnitude, meaning hybrid-lightning actually reduces loss less aggressively per doubling of compute. However, this is offset by the lower intercept $\beta_C$ (3.4797 vs. 3.7087), meaning hybrid-lightning starts from a lower loss at the same minimum compute and maintains that advantage. The crossover point depends on the specific compute range, but Figure 6 (left) shows hybrid-lightning consistently achieving lower loss across the entire range from $10^{-2}$ to $10^3$ PFLOP/s-days.

• Parameter exponent $a$: Softmax allocates proportionally more compute to scaling parameters ($N_{\text{opt}} \propto C^{0.712}$) compared to hybrid-lightning ($N_{\text{opt}} \propto C^{0.667}$). This means that at large compute budgets, softmax models grow in parameter count faster than hybrid-lightning models. Practically, a 10× increase in compute leads to a $10^{0.712} \approx 5.2\times$ increase in optimal model size for softmax versus $10^{0.667} \approx 4.6\times$ for hybrid-lightning.

• Data exponent $b$: Hybrid-lightning allocates proportionally less to data ($D_{\text{opt}} \propto C^{0.471}$ vs. $C^{0.510}$ for softmax), meaning hybrid models are trained on fewer tokens at equivalent compute. This is counterbalanced by the fact that hybrid-lightning processes tokens faster (Figure 8), so wall-clock time per token is lower.

Why hybrid-lightning behaves differently: The paper hypothesises that the linear attention layers have larger effective capacity (the recurrent state is $O(d^2/h)$ versus $O(d)$ for softmax attention, as analysed in Section 2.2.4), which means hybrid models can store more information per parameter compared to softmax models. This capacity advantage manifests as lower loss at equivalent parameter count, shifting the optimal allocation toward relatively more parameters and fewer training tokens compared to the compute-matched softmax model.

Training speed comparison. Figure 8 compares tokens processed per GPU per second (TGS) for 3B-parameter models across attention mechanisms as sequence length increases from 1024 to 65536 on H800 GPUs:

• Softmax attention (FlashAttention-2): TGS drops sharply from ~17,000 at 1024 to ~10,000 at 16384, and runs out of memory at 65536 (the quadratic cost exceeds GPU memory).
• Lightning attention: Constant TGS (~12,000–14,000) regardless of sequence length, demonstrating the linear complexity in practice.
• Hybrid-lightning: Slightly lower than pure lightning but still roughly constant across sequence lengths, with TGS ~10,000–12,000.
• Mamba2: Lower TGS than lightning attention and drops at long sequences.
• HGRN2: Similar pattern to Mamba2, consistently below lightning attention.

The critical finding: lightning attention is the only linear model that outperforms FlashAttention-2 in training throughput, and it maintains this advantage at all sequence lengths, with the gap widening as sequences grow longer.

---

Hybrid Architecture: Why Lightning Alone Is Not Enough

The retrieval gap in pure linear attention. Figure 7 presents downstream benchmark performance for 410M, 1B, 3B, and 7B parameter models across softmax, lightning, and hybrid-lightning architectures. The pattern is consistent across all scales:

• On commonsense reasoning benchmarks (PIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, OpenBookQA), lightning attention models perform comparably to softmax attention — typically within 1–3 percentage points. At 7B parameters, lightning achieves 76.5 PIQA vs. 76.5 for softmax (identical), 66.3 HellaSwag vs. 66.7 (roughly equal), and 62.4 WinoGrande vs. 62.8.

• On retrieval (NIAH — Needle-in-a-Haystack), pure lightning attention collapses. At 3B parameters, lightning achieves only 15.5% on NIAH versus 84.2% for softmax attention. At 7B parameters, the gap persists (lightning's NIAH is not shown in Figure 7 because it is so low, while softmax achieves 97.7%).

• On long-context comprehension (SCROLLS), lightning attention underperforms softmax attention by roughly 5–10 percentage points across all scales.

• Hybrid-lightning (7 lightning layers + 1 softmax layer, repeated) matches or exceeds softmax attention on all benchmarks, including NIAH. At 3B parameters, hybrid-lightning achieves 98.0% NIAH versus 84.2% for pure softmax — a 13.8 percentage point improvement. At 7B parameters, hybrid-lightning achieves 97.7% NIAH versus 97.7% for softmax (identical), while maintaining comparable or better commonsense reasoning scores.

Why hybrid-lightning outperforms pure softmax on retrieval. Section 2.2.4 provides a theoretical analysis. Softmax attention can be rewritten as a linear recurrence (Eq. 11):

$\mathbf{s}_t^0 = 0, \quad \mathbf{s}_t^j = \mathbf{s}_t^{j-1} + \exp\left(\frac{\mathbf{q}_t \mathbf{k}_j^\top}{\sqrt{d}}\right), \quad \dots$

This recurrence recomputes the hidden state from the initial position at each time step — the paper describes this as "Going Through a Book." This systematic revisiting of all prior tokens enables accurate retrieval.

In contrast, lightning attention's recurrence (Eq. 12):

$\mathbf{kv}_0 = \mathbf{0}, \quad \mathbf{kv}_j = \mathbf{kv}_{j-1} + \mathbf{k}_j \mathbf{v}_j^\top, \quad \mathbf{o}_j = \mathbf{kv}_j^\top \mathbf{q}_j$

accumulates a compressed state without revisiting prior tokens. The capacity of this compressed state is $O(d^2/h)$ (the number of entries in the $\mathbf{KV}$ matrix per head), while softmax attention's capacity is $O(d)$ (the dimension of the hidden state in the recurrence). Since $d > h$, lightning attention's capacity is larger ($d^2/h > d$). With $d = 128$ and $h = 64$, lightning attention's per-head capacity is $128^2/64 = 256$ versus softmax's effective capacity of $128$.

However, capacity alone does not guarantee retrieval quality. The recomputation mechanism in softmax attention means that when the model needs to attend sharply to a specific token from the distant past, it can compute a precise attention weight by re-evaluating the similarity between the current query and that token's key. Lightning attention's compressed $\mathbf{KV}$ state loses fine-grained access to individual tokens — the outer product $\mathbf{k}_j \mathbf{v}_j^\top$ blends all past tokens into a fixed-size matrix, and retrieving a specific token requires the query to "read out" its contribution from this mixture, which is lossy.

The hybrid architecture resolves this tension: The softmax attention layers, placed every 8 layers, provide "retrieval checkpoints" where the model can attend sharply to any token in the full context. The lightning attention layers between these checkpoints process information efficiently, gradually updating the compressed $\mathbf{KV}$ state. The final result is a model that has both the efficiency of linear attention (in 7/8 of layers) and the retrieval fidelity of softmax attention (in 1/8 of layers), with the overall effect exceeding either pure approach.

Comparison against alternative hybrid configurations. Tables 3 and 4 provide empirical validation that the hybrid-lightning configuration is optimal among reasonable alternatives.

Table 3 — Hybrid-linear variants at 1B parameters:

• Hybrid-cosformer2: CSR (commonsense reasoning average) 46.64%, NIAH 43.6%, SCROLLS 10.9, TGS 23.3K.
• Hybrid-hgrn2: CSR 49.87%, NIAH 91.8%, SCROLLS 10.8, TGS 29.5K.
• Hybrid-lightning: CSR 49.55%, NIAH 95.7%, SCROLLS 13.3, TGS 33.4K.

Hybrid-lightning achieves the best NIAH (95.7% vs. 91.8% for HGRN2 and 43.6% for cosformer2), the best SCROLLS (13.3 vs. 10.8–10.9), and the highest training throughput (33.4K TGS vs. 23.3K–29.5K). The CSR scores are comparable across all three (46.64–49.87). The cosformer2 variant's poor NIAH suggests that not all linear attention variants are equally suitable for the hybrid approach — the specific computation pattern in lightning attention (the intra/inter-block decomposition) matters for both efficiency and downstream capability.

Table 4 — Hybrid-window vs. hybrid-lightning at 1B and 3B parameters:

At 1B parameters with a window size of 1024:

• Hybrid-window: CSR 48.02%, NIAH 53.9%, SCROLLS 10.6, TGS 33.6K.
• Hybrid-lightning: CSR 49.55%, NIAH 95.7%, SCROLLS 13.3, TGS 33.4K.

Even though both architectures achieve comparable training throughput (~33.5K TGS), hybrid-lightning's NIAH score is 41.8 percentage points higher (95.7% vs. 53.9%). This enormous gap demonstrates that sliding window attention, even with periodic full-attention layers, fundamentally limits the model's ability to retrieve information from beyond the window — the window size of 1024 tokens (at 1B parameters) is simply too small to capture long-range dependencies reliably.

At 3B parameters with window size 1024:

• Hybrid-window: CSR 53.44%, NIAH 41.6%, SCROLLS 13.3, TGS 15.4K.
• Hybrid-lightning: CSR 55.16%, NIAH 98.0%, SCROLLS 14.7, TGS 15.1K.

The gap in NIAH widens to 56.4 percentage points. Larger window sizes improve NIAH somewhat (57.9% at 512 and 40.9% at 256 for 3B), but all fall dramatically short of hybrid-lightning. Importantly, hybrid-lightning actually trains slightly faster than hybrid-window (15.1K vs. 15.4K TGS) at 3B, meaning there is no speed tradeoff — hybrid-lightning is Pareto-superior on both speed and retrieval capability.

---

Ablations Within the MoE Context

Table 5 presents two ablation experiments conducted at larger scale to validate design choices within the MoE framework.

Hybrid-lightning vs. softmax attention in MoE (28B total, 5B activated parameters, trained on 1T tokens):

The hybrid-lightning variant outperforms softmax on 6 of 8 benchmarks (BBH +4.0, DROP +1.6, MMLU +0.2, MATH +2.2, ARC-C +1.0, WG +1.9) and underperforms marginally on CMMLU (−1.3) and GSM8k (−0.3). The improvements on BBH (complex reasoning) and MATH (mathematical problem-solving) are particularly notable, suggesting that the hybrid architecture's larger effective capacity (from the $O(d^2/h)$ recurrent state) may benefit tasks requiring multi-step reasoning.

Pre-norm vs. post-norm with DeepNorm (9.3B activated, 60B total, 48 layers, trained on 500B tokens):

PostNorm with DeepNorm outperforms PreNorm on every benchmark: BBH +2.7, DROP +0.8, MMLU +6.3, CMMLU +7.4, MATH +0.9, GSM8k +4.6, ARC-C +2.7, WG −0.1 (essentially tied). The improvements on knowledge-intensive benchmarks (MMLU, CMMLU) are dramatic — 6–7 percentage points — suggesting that PreNorm's reduction in effective model depth (gradients bypassing sub-layers through residual connections) genuinely harms the model's ability to utilise its full depth for storing and retrieving knowledge. PostNorm preserves the full effective depth, and DeepNorm provides the training stability that PostNorm historically lacked.

Why PreNorm reduces effective depth: In PreNorm, the normalisation is applied before the sub-layer (attention or FFN), and the residual connection adds the sub-layer's output to the original input. This means gradients can flow from the output directly to the input through the residual connection without passing through the sub-layer — effectively, the sub-layer becomes a "skip-able" computation, reducing the model's functional depth. PostNorm applies normalisation after the residual connection, forcing gradients to pass through the sub-layer, which preserves the full depth but historically caused training instability. DeepNorm (Wang et al., 2024a) addresses this by carefully scaling the residual connections: the scaling factors are set to $\alpha = (2N)^{0.25}$ and $\beta = (8N)^{-0.25}$, where $N = 80$ is the number of layers.

---

Model Specification: How the Final Configuration Was Chosen

Section 2.4 describes the process of determining the final model hyperparameters — 45.9B activated parameters, 456B total, 32 experts, top-2 routing, hidden size 6144, FFN hidden dimension 9216, 64 attention heads with head dimension 128, 80 layers, 8-key-value-head GQA for softmax attention, RoPE on half the head dimensions with base frequency 10,000.

The constrained optimisation problem. The model specification is formalised as:

$\min_{P_{\text{all}}, P_{\text{act}}} L(P_{\text{all}}, P_{\text{act}}, T) \quad \text{subject to} \quad C_{\text{compute}}(P_{\text{all}}, P_{\text{act}}, T) < C \quad \text{and} \quad P_{\text{all}} < 500\text{B}$

where $L$ is the training loss, $P_{\text{all}}$ and $P_{\text{act}}$ are the total and activated parameter counts, $T$ is the number of training tokens, $C_{\text{compute}}$ is the computational cost, and $C$ is the total compute budget.

The constraint $P_{\text{all}} < 500\text{B}$ is driven by the deployment target: processing 1 million tokens on a single node with 8×80GB GPUs using 8-bit weight-only quantisation (W8A16). This constraint ensures the model is practically deployable without requiring multi-node inference, which introduces cross-machine communication overhead.

Key architectural ratios. The paper describes determining optimal ranges for several ratios through small-scale experiments:

1. Softmax-to-linear ratio (1:7): Substituting lightning attention for 7 out of every 8 layers. The paper notes that "shallow models require substantially more softmax attention layers to achieve comparable performance," suggesting that the optimal ratio is depth-dependent — deeper models can afford more linear attention layers because the softmax layers at greater depth have access to more abstracted representations, making their retrieval function more effective.

2. Depth-to-width ratio: "Deeper models consistently outperforming shallower counterparts" in the hybrid architecture, which is notable because standard transformers often show diminishing returns from depth beyond ~48 layers. The paper suggests this is because each lightning attention layer has lower effective capacity per layer (the $\mathbf{KV}$ state is lossy) but the depth enables progressive refinement of this compressed state.

3. Linear attention memory size to hidden size: The $\mathbf{KV}$ matrix has dimension $d \times d$ per head, so the effective memory size scales with $d^2/h$. Increasing the hidden size or decreasing the number of heads increases capacity, but both affect computational cost. The paper reports that "increasing linear attention memory size significantly enhances model performance."

4. Activated FFN to attention ratio: The FFN hidden dimension (9216) relative to the hidden size (6144) is a ratio of 1.5, which is lower than typical dense transformers (often 4×). The MoE compensates for this by having 32 experts, providing diversity at the cost of activating only 2 per token.

5. RoPE dimension ratio (1/2): Applying RoPE to only half the attention head dimensions "enables length extrapolation without performance degradation." This is important because RoPE embeddings trained at short context lengths can be interpolated to longer lengths, but full-dimension RoPE can cause degradation. Using half the dimensions preserves the model's ability to generalise to unseen lengths.

Scaling law extrapolation for final model size. The paper trains small models with activation parameters ranging from 44 million to 1.2 billion across 500 billion tokens, using 16, 32, and 64 experts, to fit a scaling law that can predict loss for larger models. However, the paper notes that standard approaches (fitting simple power laws) become unreliable when extrapolating to much larger models. To address this, they propose a refined formula:

$L(P_{\text{act}}, T \mid E) = d + a P_{\text{act}}^\alpha + b T^\beta + c (P_{\text{act}} T)^\gamma$

where $L(P_{\text{act}}, T \mid E)$ is the expected loss conditioned on the number of experts $E$, and $a, b, c, d, \alpha, \beta, \gamma$ are fitted parameters that depend on $E$. The term $c (P_{\text{act}} T)^\gamma$ captures the interaction between model size and data size — larger models benefit more from additional data, and vice versa — which the paper found necessary for accurate extrapolation beyond the small-scale fitting range.

Why this form: Standard scaling laws (Hoffmann et al., 2022) use additive terms $a N^\alpha + b D^\beta$, which assume independence between model size and data size effects on loss. The interaction term $c (N D)^\gamma$ captures the fact that larger models extract more information from each additional training token, making the benefit of additional tokens conditional on model size. The paper reports that this refinement was necessary for reliable extrapolation from small-scale models (up to 1.2B activated) to the target scale (45.9B activated).

Based on these predictions, the configuration with 45.9 billion activated parameters and 456 billion total parameters was identified as optimal under the 500B parameter constraint and the available compute budget.

---

MoE Optimisation for Distributed Training

The MoE architecture introduces a specific communication pattern — all-to-all (a2a) — where tokens are redistributed across GPUs so that each GPU processes the tokens assigned to the experts it hosts. This communication is a potential bottleneck, especially at large scale.

Expert Parallel (EP) and Expert Tensor Parallel (ETP). The paper introduces a novel parallelisation scheme that decouples the MoE components from the non-MoE components. Two new process groups are defined:

• ETP (Expert Tensor Parallel): Partitions the weight matrices of individual experts across GPUs, similar to standard tensor parallelism but applied only within the expert dimension. This reduces the memory footprint per GPU for each expert, enabling larger experts or more experts per GPU.

• EDP (Expert Data Parallel): Encapsulates the data parallelism of identical experts — multiple GPUs hold copies of the same experts and process different tokens, with gradients synchronised periodically.

The system satisfies two simultaneous constraints:

$\text{world\_size} = \text{size}_{PP} \times \text{size}_{DP} \times \text{size}_{CP} \times \text{size}_{TP}$

$\text{world\_size} = \text{size}_{PP} \times \text{size}_{EDP} \times \text{size}_{ETP} \times \text{size}_{EP}$

where PP is pipeline parallelism, DP is data parallelism, CP is context parallelism, and TP is tensor parallelism (the first equation handles non-MoE components; the second handles MoE components). This allows the MoE's parallel strategy to be configured independently — for example, using a different tensor-parallel degree for experts versus the standard transformer components.

Why this decoupling matters: In MiniMax-Text-01, the experts are large (hidden dimension 9216, comparable to the model's hidden size 6144). Standard tensor parallelism applied uniformly would partition these expert weights, but the expert computation is relatively shallow (a single FFN) — the communication cost of tensor parallelism (all-reduce after each FFN) might outweigh the memory savings. ETP allows using a lower tensor-parallel degree for experts (reducing communication) while keeping a higher degree for non-MoE components where the computation is deeper and benefits more from partitioning.

EP-ETP overlap: computation-communication interleaving. Figures 9 and 10 illustrate the overlapping strategy. Tokens are divided into groups, and the processing of one group is overlapped with the communication of another group. Specifically:

1. Tokens are partitioned into multiple groups along the sequence or batch dimension.
2. Group 0's all-to-all dispatch communication is initiated.
3. While group 0's communication is in flight, group 1's tokens begin computation on the current set of experts.
4. When group 0's communication completes, its expert computation begins, overlapping with group 1's communication.
5. This pattern continues for all groups, with the final group's communication overlapping with earlier groups' combine (reverse all-to-all) operations.

The tradeoff in group count (Figures 9, 10): More groups provide more opportunities for overlap — communication from group $i$ can overlap with computation from group $i-1$, and vice versa. However, excessively many groups increase scheduling overhead and risk becoming CPU-bound (the CPU must orchestrate the overlapping launches, and too many small kernel launches saturate the CPU's command queue). The optimal group count balances overlap efficiency against CPU overhead.

Computation time matters for overlap efficiency. Figure 10(b) shows that when the expert computation time is longer (higher computation portion), overlap is more effective because a single communication operation can be fully hidden behind a longer computation block. Conversely, Figure 10(a) shows that with shorter computation, some communication "spills over" and cannot be fully hidden, leaving idle time on the GPU. This motivates the paper's choice of relatively large experts (hidden dimension 9216) — the computation time per expert is substantial enough to hide the all-to-all communication latency.

Reduction in communication overhead. The paper reports that these optimisations reduce the pure communication overhead of the MoE component by 50% compared to the pre-optimisation state.

---

Long-Context Training Optimisation

Training on sequences up to 1 million tokens introduces challenges beyond the attention mechanism — specifically, how to partition the sequence across GPUs without wasting computation on padding.

Varlen ring attention for softmax layers. Standard ring attention (Liu et al., 2024a) partitions sequences across GPUs in a ring topology: each GPU holds a segment of the sequence, computes attention for its segment using its local keys/values, then passes its keys/values to the next GPU in the ring, which uses them to augment its own attention computation. This process repeats for as many steps as there are GPUs in the ring, with each step enabling cross-segment attention.

However, standard implementations assume all sequences in a batch have the same length, which is unrealistic for real training data. When sequences vary in length, the conventional approach pads each sequence to the maximum length (or to a multiple of the CP group size), leading to wasted computation on padding tokens.

The paper's varlen ring attention addresses this by applying the ring attention algorithm directly to variable-length sequences that have been concatenated via data-packing. The key modification is replacing the fixed-shape causal masks with varlen causal masks that account for the varying sequence boundaries within the concatenated sequence. Figure 11 illustrates the difference: standard ring attention uses uniform causal masks (left panel, top) and uniform non-causal masks (left panel, bottom), while varlen ring attention applies causal and non-causal computations that respect the individual sequence boundaries within the packed data (right panel).

Why this matters at 1M-token scale: At a context length of 1 million tokens, even 10% padding waste (100K tokens) translates to enormous computational overhead. Data-packing nearly eliminates padding by concatenating sequences end-to-end, and varlen ring attention ensures the attention computation correctly respects sequence boundaries within the packed data.

LASP+ (Improved Linear Attention Sequence Parallelism). For lightning attention layers, the LASP algorithm (Sun et al., 2024) uses the Context Parallel (CP) communication group to distribute the sequence across GPUs. However, the original LASP has a sequential dependency: each CP rank must receive the cumulative $\mathbf{KV}$ state from the previous rank before it can compute its own output. This creates a chain of send-recv operations that force serial execution across ranks, as shown in Figure 12(a).

The paper's LASP+ eliminates this dependency through a three-step process:

1. Local prefix sum: Each CP rank independently computes its local $\mathbf{KV}_L$ — the cumulative sum of $\mathbf{k}_j \mathbf{v}_j^\top$ for all tokens in its local sequence segment.

2. Global AllGather: All CP ranks perform an AllGather operation to share their local $\mathbf{KV}_L$ matrices. After this step, every rank has the local $\mathbf{KV}_L$ from every other rank.

3. Global prefix sum: Each rank selects the appropriate $\mathbf{KV}_L$ matrices to sum (based on its position in the ring) to construct its $\mathbf{KV}_G$ — the cumulative state from all prior tokens across all ranks.

Why LASP+ improves efficiency: The original LASP requires $N_{\text{CP}}$ sequential send-recv operations (one per rank in the ring), each blocking the next. LASP+ replaces this with a single AllGather operation followed by independent local prefix sums. The AllGather involves more total data movement (every rank gets all local $\mathbf{KV}_L$ matrices, not just the one from the previous rank), but this is parallelised across ranks — each rank simultaneously sends its data to all others and receives from all others, rather than waiting in a chain. The paper reports that "the computation speed in the LASP+ approach can attain up to $1/N_{\text{cp}}$ of the original LASP algorithm, where $N_{\text{cp}}$ denotes the number of parallel computing nodes," and that the AllGather overhead is "minimal."

Varlen support in LASP+. To handle the data-packing format (variable-length sequences concatenated together), LASP+ introduces varlen support through:

1. Padding each input within the batch to a multiple of the block size (256).
2. Sequentially concatenating the padded inputs.
3. Using a single kernel to perform parallel computations across multiple batches, with the kernel aware of the sequence boundaries within the concatenated data.

---

Lightning Attention Inference Optimisation

The paper implements four optimisations to make lightning attention efficient for inference, targeting the specific patterns that arise in production serving (batch sizes with variable-length inputs, prefix caching for multi-turn conversations, mixed-length batches).

1. Batched kernel fusion. In the prefill phase, multiple memory-bound operations on the $\mathbf{Q}, \mathbf{K}, \mathbf{V}$ tensors are fused into a single kernel: padding in the sequence dimension, partitioning into blocks of size $B$, adjusting internal memory layouts (e.g., from interleaved to contiguous), and computing decay values. In the decoding phase, the $\mathbf{KV}$ computation and the prefix $\mathbf{KV}$ cache update are fused. These fusions reduce intermediate result storage and HBM access, achieving a 10% reduction in end-to-end latency in the decoding phase and short-text scenarios, with particularly large benefits on H20 GPUs (where memory bandwidth is lower than H800).

2. Separated prefill and decoding execution. During inference, a batch may contain both "prefill" tokens (processing a new prompt, token length > 1) and "decoding" tokens (generating the next token from an existing KV cache, token length = 1). These have very different compute patterns: decoding with token length 1 is memory-bound and uses only a small number of GPU Streaming Multiprocessors (SMs), while prefill with longer sequences is compute-bound and can utilise more SMs.

The paper implements two separate CUDA kernels — one optimised for token length 1 (minimal SM usage, tuned for memory bandwidth), one for token length > 1 (tuned for compute utilisation) — and schedules them on two separate CUDA streams to execute in parallel. This prevents decoding tokens from blocking on prefill tokens, or vice versa. The paper provides a concrete example: a batch of 20 inputs, all with a prefix KV cache, where 1–2 inputs have token length 50 (prefill) and the remaining 18–19 have token length 1 (decoding). Without separation, the total latency might be ~100ms (bottlenecked by the long prefill). With separated execution, the decoding kernel completes in ~50ms while the prefill kernel runs concurrently, reducing total latency to approximately the prefill time (~50ms).

3. Multi-level padding. The training configuration uses a block size of 256 for the intra-block computation. However, during inference with prefix caching, token lengths within a batch often fall well below 256 (e.g., a single new token appended to a cached prefix). Using a 256×256 matrix multiplication to process a token of length 1 means 255/256 of the computation is wasted on padding.

The paper introduces additional segmentation options — block sizes of 32, 64, and 128 — and dynamically selects the block size that minimises padding overhead based on the current input sequence length. A token of length 50 would use block size 64 (14 tokens of padding, 22% waste) rather than block size 256 (206 tokens of padding, 80% waste).

4. StridedBatchedMatmul extension. The intra-block and inter-block computations decompose into many small matrix multiplications (e.g., 256×256 GEMM operations for the intra-block, 256×128 times 128×256 for the inter-block). The paper uses cublasGemmStridedBatchedEx to execute these as batched GEMM operations, which amortises the kernel launch overhead. For Hopper GPUs (H100, H800), the paper is implementing additional warpgroup-wide WGMMA (Warp Group Matrix Multiply-Accumulate) instructions for the 256×256 GEMMs, combined with asynchronous Tensor Memory Accelerator (TMA) operations for memory access and offloading of pre/post-processing to CUDA cores.

The result of these optimisations: over 75% Model FLOPs Utilisation (MFU) on H20 GPUs for end-to-end inference. At a sequence length of 1,024,000 tokens, softmax attention constitutes 95% of the total attention + FFN latency, while lightning attention contributes less than 12% of the latency. This demonstrates that the hybrid architecture achieves its design goal: the lightning attention layers become a negligible cost even at million-token contexts, while the softmax attention layers (which process only 1/8 of the layers) dominate the runtime.

4. Key Insights and Innovations

Innovation 1: Linear Attention Works at Commercial Scale, But Only in Hybrid Form — A Fundamental Architectural Finding

The field has long known that linear attention variants (e.g., de Brébisson and Vincent, 2016; Qin et al., 2022a) could theoretically reduce the quadratic complexity of transformers, but no production-scale LLM had adopted them. The prevailing assumption was that either linear attention was fundamentally incapable of matching softmax attention quality at scale, or that the engineering challenges (particularly the cumsum bottleneck for causal modelling) were insurmountable. This paper challenges both assumptions, but with a crucial qualification that makes the finding conceptually distinctive rather than merely incremental.

The core intellectual move is the diagnosis of why pure linear attention fails for LLMs, and the demonstration that this failure has a specific, identifiable cause — not a general quality deficiency. Section 2.2.2.3 and Figure 7 show that pure lightning attention models match softmax attention on commonsense reasoning, reading comprehension, and language modelling benchmarks across scales from 410M to 7B parameters. Performance is comparable within 1–3 percentage points on PIQA, HellaSwag, WinoGrande, and ARC. The failure is localised to retrieval: on Needle-in-a-Haystack (NIAH), pure lightning attention collapses to near-random performance (15.5% at 3B parameters vs. 84.2% for softmax), and on SCROLLS (long-context comprehension), it consistently underperforms by 5–10 percentage points.

This is not a generic "linear attention is worse" result — it is a specific capability gap that the paper traces to a mechanistic cause in Section 2.2.4: softmax attention recomputes its hidden state from the beginning at each step ("Going Through a Book"), enabling precise retrieval of arbitrary past tokens by re-evaluating query-key similarity, while linear attention accumulates a compressed KV state that loses fine-grained access to individual tokens. This diagnostic framing transforms the problem from "can linear attention replace softmax attention?" into "can we restore retrieval while keeping the efficiency gains?"

The hybrid architecture — one softmax layer for every seven lightning attention layers — is the conceptual solution to this diagnosed gap, not an arbitrary configuration. The softmax layers act as "retrieval checkpoints" spaced throughout the depth of the model, providing precise access to any position in the full context at regular intervals. The lightning attention layers between checkpoints handle the bulk of information processing efficiently. The result, shown in Figure 7 and Tables 3–4, is that hybrid-lightning matches or exceeds pure softmax attention on retrieval (98.0% NIAH vs. 84.2% at 3B parameters) while maintaining comparable commonsense reasoning and achieving superior training throughput (Figure 8).

What makes this finding fundamental rather than incremental is its reframing of the attention mechanism design space. Before this work, the choice was binary: softmax attention (accurate but expensive) vs. linear alternatives (efficient but potentially lower quality). The paper demonstrates that the correct framing is complementary: softmax and linear attention serve different functions (precise retrieval vs. efficient processing), and integrating both yields a model that is better than either alone. The hybrid model achieves a lower training loss under equal compute than pure softmax attention (Table 2: loss exponent −0.0763 vs. −0.0798, but with a lower constant 3.4797 vs. 3.7087, yielding consistently lower loss across the measured compute range in Figure 6). This is not a tradeoff — it is a Pareto improvement.

The significance extends beyond this specific architecture. The paper provides the first systematic scaling law analysis for linear attention at meaningful scale (Table 2, Figure 6), showing that hybrid-lightning allocates compute differently than softmax attention — favouring relatively more data and fewer parameters (N_opt ∝ C^{0.667} vs. C^{0.712} for softmax) — which has practical implications for how future models should be sized and trained. This is the inference-time scaling equivalent of the Chinchilla laws (Hoffmann et al., 2022), but for architecture rather than model size, and it establishes that the optimal architectural configuration itself depends on the available compute budget.

Innovation 2: Retrieval Capacity Is Not Just About Information Storage — The "Recomputation" Mechanism Matters

A common intuition in sequence modelling is that a model's ability to retrieve information from context is determined by the size of its memory — more parameters, larger hidden states, or bigger recurrent state matrices should improve retrieval. The paper's analysis in Section 2.2.4 directly contradicts this intuition through a counterintuitive finding: lightning attention has a larger theoretical capacity than softmax attention (O(d²/h) vs. O(d) per head, where d = 128 and h = 64, giving lightning attention roughly 256 units of effective capacity vs. softmax's 128), yet lightning attention performs dramatically worse on retrieval tasks.

The paper diagnoses this apparent paradox by identifying that capacity size and retrieval quality are not equivalent: the mechanism by which information is accessed matters more than the total amount stored. Softmax attention's mechanism — recomputing the hidden state from the initial time step at each position — enables precise, position-specific retrieval by re-evaluating the similarity between the current query and every past key. This produces sharp attention weights that can selectively focus on specific tokens in a long context. Lightning attention's mechanism — accumulating a running sum of outer products k_j v_j^⊤ into a fixed-size d × d matrix — blends all past tokens together into a compressed representation. Retrieving a specific token from this blend requires the query vector to "read out" its contribution from the mixture, which is inherently lossy because multiple tokens contribute to the same entries in the KV matrix.

This finding is conceptually significant because it reframes how the field should think about long-context model design. The dominant narrative around state space models (Mamba, Gu and Dao, 2024), linear RNNs (Qin et al., 2023b, 2024d), and other efficient architectures has focused on increasing state capacity — making the recurrent state larger, adding gating mechanisms, or using structured state matrices — under the implicit assumption that bigger memory → better retrieval. The paper's evidence suggests this assumption is flawed: even with larger capacity, a model that cannot re-access past tokens with fine-grained, content-dependent attention weights will struggle with retrieval. The hybrid architecture succeeds not because it increases total capacity (it actually uses less capacity per token in the lightning attention layers, since the KV state is the same size as pure lightning attention), but because it adds a qualitatively different access mechanism (softmax recomputation) at periodic intervals.

The theoretical framing in Eqs. 10–12, while mathematically simple, represents a diagnostic innovation: rewriting softmax attention as a linear recurrence exposes that both architectures are, in some sense, recurrent — but they differ in their recurrence dynamics. Softmax attention's recurrence is "reset" at each step (recomputing from the beginning), which is expensive but preserves information fidelity. Lightning attention's recurrence is "accumulating" (never resetting), which is efficient but information-lossy. The hybrid design exploits this insight by interleaving accumulating layers (efficient) with resetting layers (precise), achieving both properties.

This contribution is best categorised as a fundamental diagnostic finding rather than an architectural innovation per se. The hybrid architecture is the architectural consequence, but the deeper contribution is the conceptual framework for understanding why certain architectures fail at retrieval despite having sufficient capacity — a framework that should guide future work on efficient attention mechanisms, state space models, and retrieval-augmented generation.

Innovation 3: The Engineering Framework Is a First-Class Research Contribution — Redefining What "Supports Long Context" Means

A substantial portion of the paper (Section 3, spanning pages 14–21) is devoted to systems engineering: custom parallelisation strategies for MoE (EP-ETP), overlap schemes for communication-computation interleaving (Figures 9–10), varlen ring attention for variable-length sequences in data-packing (Figure 11), an improved LASP+ algorithm that eliminates sequential dependencies in linear attention sequence parallelism (Figure 12), and four inference-time optimisations for lightning attention (batched kernel fusion, separated prefill/decoding, multi-level padding, strided batched matmul extensions).

On a superficial reading, this is implementation detail. The paper's implicit argument — and what makes this an intellectual contribution rather than mere documentation — is that the line between algorithm and implementation collapses at this scale. The lightning attention algorithm was proposed in prior work (Qin et al., 2024b,c), and the hybrid architecture concept could be sketched in a few paragraphs. But without the specific engineering solutions described in Section 3, these ideas would remain theoretically sound but practically unusable for contexts exceeding 1 million tokens. The paper demonstrates this concretely:

• The original LASP algorithm (Sun et al., 2024) has a sequential dependency chain across context-parallel ranks that limits parallel efficiency. LASP+ replaces this with an AllGather operation that enables fully parallel computation, achieving speedups proportional to 1/N_cp (Section 3.2.2). Without this improvement, training on million-token sequences would be bottlenecked by the sequential communication, rendering the hybrid architecture's theoretical efficiency unrealisable.

• The varlen ring attention algorithm (Section 3.2.1) addresses an issue that only manifests at million-token scale: padding sequences to uniform lengths wastes enormous computation. Standard ring attention implementations assume uniform-length sequences or pad to multiples of the CP group size, but at 1M-token contexts with realistic data distributions, this padding can represent a significant fraction of total computation. Varlen ring attention eliminates this waste, making long-context training practical with real (non-synthetic) data.

• The inference optimisations in Section 3.3.2 (separated prefill and decoding) address a practical deployment challenge: inference batches contain mixtures of prefill tokens (new prompts, potentially long) and decoding tokens (single next-token predictions). Without separation, decoding tokens must wait for long prefill computations to complete, inflating latency. The paper's solution — two kernels on separate CUDA streams — is conceptually simple but represents the kind of production engineering that determines whether an architecture is deployable or merely publishable.

The intellectual contribution here is a reframing of what it means to "support long context." The paper's standard is not merely that the model can process a long sequence without running out of memory (a low bar that many architectures meet through careful memory management), but that it can do so with high computational efficiency — defined as >75% MFU on H20 GPUs at million-token inference, and with training throughput that remains roughly constant as sequence length increases (Figure 8, hybrid-lightning maintains ~10,000–12,000 TGS from 1024 to 65536 tokens without dropping). This reframes the long-context problem from a capability question (can the model do it?) to an efficiency question (can the model do it at a cost that makes deployment economically viable?). The latency comparison in Figure 2 — showing MiniMax-Text-01's near-linear prefilling latency up to 1M tokens while other models exhibit quadratic growth or hit limits at 128K-256K — is the practical manifestation of this reframing.

By open-sourcing the full model and providing detailed documentation of these engineering optimisations, the paper makes a methodological contribution: it establishes that systems engineering for novel architectures is not a secondary concern to be handled post-hoc, but a primary design constraint that should inform architectural choices from the start. The model size constraint in Section 2.4 — fitting on a single 8×80GB node at 1M tokens under 8-bit quantization — is explicitly driven by deployment practicality, not just academic benchmarking.

Innovation 4: A Systematic Methodology for Data Quality Assessment at Pre-Training Scale — The "Data Experiment" Paradigm

Section 4.1.3 introduces a methodology that is orthogonal to the main architectural contributions but represents a distinctive contribution to pre-training data curation. The paper formalises data ablation experiments as statistical hypothesis tests rather than heuristic comparisons of benchmark scores, enabling rigorous conclusions from small-scale experiments that generalise to full-scale training.

The key elements are:

A principled metric for data quality assessment. The paper defines a specific metric — sample-wise log-normalised accuracy — computed as log accnorm2(x) = log softmax{p'(c ∈ C_x) / p'(c*)}, where p'_i(c) = p_i(c) / bytes(c) is the byte-normalised probability of choice c for sample i. Byte-wise normalisation is specifically chosen to exclude tokeniser effects (different tokenisers produce different probabilities for the same semantic content) while alleviating bias against longer answer choices. The paper states that extensive experiments confirmed this metric is "stable across training" while maintaining "discriminative power," quantified by the ratio of obvious performance differences between models to the standard deviation across random seeds (Δobvious / σ_seed).

This metric choice is significant because it addresses a known problem in LLM data experimentation: benchmark scores from small models trained on limited data are often too noisy to reliably distinguish data quality differences. By designing a metric with quantified discriminative power and stability, the paper enables statistically valid comparisons at a scale (1B activation parameters, 40B training tokens) that is orders of magnitude smaller than full training runs.

Power analysis for experimental design. The paper conducts a formal power analysis to determine the minimal test sample size that maintains the Minimal Detectable Effect (MDE) at a level comparable to training variance, while guaranteeing 95% confidence and 80% statistical power. This is a standard practice in clinical trials and A/B testing but is rarely applied rigorously in LLM data experimentation, where sample sizes and evaluation protocols are often chosen heuristically. The paper's approach transforms data experimentation from an engineering art into a scientific methodology with quantified reliability guarantees.

Repetition-aware experimental framework. The paper identifies a subtle flaw in prior work on data repetition (e.g., Abdin et al., 2024; Penedo et al., 2024): standard experimental paradigms assess the impact of repetition by training small models with data distributions identical or similar to the final training distribution, but this fails to account for the fact that data efficiency is not constant throughout training. Tokens consumed early in training contribute differently to model performance than tokens consumed later, meaning that the repetition frequency in a short experiment does not accurately simulate the repetition frequency in a full training run.

The paper's solution is a repetition-aware framework that first globally deduplicates the dataset, then down-samples documents to align repetition frequency with the requirements of the final training schedule (not the small-scale experiment), while adhering to the experiment's budget constraints. This approach yields "better alignment with the results obtained using considerably more computational resources." The finding that low-quality data degrades after more than two epochs while high-quality data benefits from up to four epochs (consistent with Muennighoff et al., 2023) would be unreliable without this repetition-aware design — a short experiment might show benefits from repeating low-quality data simply because the model hasn't yet overfit, leading to incorrect data mixture decisions at full scale.

Why this is an innovation rather than just good practice. The paper's data experimentation methodology is transferable — it provides a template that other organisations can adopt for their own data curation pipelines. It is also cost-effective: the experiment step (1B activation parameters, 8B total, 40B tokens) is cheap enough to run multiple times, enabling systematic exploration of data quality, formatting, and mixture questions that would be prohibitively expensive to test at full scale. By demonstrating that statistically valid conclusions can be drawn from small-scale experiments with proper metric design and power analysis, the paper lowers the barrier to rigorous data experimentation for the broader community.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on a diverse set of academic and in-house benchmarks. Core text benchmarks include MMLU (Hendrycks et al., 2021a), MMLU-Pro (Wang et al., 2024b), SimpleQA (Wei et al., 2024), C-SimpleQA (He et al., 2024b), IFEval (Zhou et al., 2023), GPQA Diamond (Rein et al., 2024), DROP (Dua et al., 2019), GSM8k (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), HumanEval (Chen et al., 2021), MBPP Plus (Austin et al., 2021; Liu et al., 2023), and Arena-Hard-Auto (Li et al., 2024b). Long-context benchmarks include RULER (Hsieh et al., 2024) for multi-hop tracing and aggregation up to 1M tokens, LongBench-V2 (Bai et al., 2024) for complex reasoning across diverse context types, MR-NIAH (in-house) for multi-round retrieval from conversation histories up to 1M tokens, and MTOB (Tanzer et al., 2024) for long in-context learning of a new language from a grammar book (~81K–133K tokens). Scaling law experiments use BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), ARC-Easy and ARC-Challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), NIAH (Shen et al., 2024), and SCROLLS (Shaham et al., 2022). For vision-language evaluation, benchmarks include MMMU (Yue et al., 2024a), MMMU-Pro (Yue et al., 2024b), ChartQA (Masry et al., 2022), DocVQA (Mathew et al., 2021), OCRBench (Liu et al., 2024b), AI2D (Kembhavi et al., 2016), MathVista (Lu et al., 2023), OlympiadBench (He et al., 2024a), MMLongBench-Doc (Ma et al., 2024), MEGA-Bench (Chen et al., 2024a), and an in-house user experience benchmark of 524 annotated samples. The in-house text benchmarks are derived from Hailuo AI user interactions and include General Assistant, Hard Capability, Creative Writing, Knowledge Q&A, Instruction Following, Coding, Safety, and Long Context categories, with expert human evaluation and automatic assessment. Scaling law experiments are conducted with models trained on up to 300B tokens at context length 8192.

• Base model(s). All text experiments use MiniMax-Text-01, a 456-billion-parameter MoE model with 45.9 billion activated parameters per token, 32 experts with top-2 routing, 80 layers in a hybrid attention pattern (7 lightning attention + 1 softmax attention, repeated 10 times). Scaling law experiments compare models at 70M, 160M, 410M, 1B, 3B, and 7B non-embedding parameters across softmax, lightning, and hybrid-lightning architectures. Module ablations use intermediate-scale configurations: 28B total / 5B activated parameters for the hybrid-lightning vs. softmax MoE comparison, and 9.3B activated / 60B total for the PreNorm vs. PostNorm comparison. Data experiments use MoE models with 1B activation / 8B total parameters trained on 40B tokens. The vision-language model MiniMax-VL-01 uses a ViT-L/14 with 303M parameters, a two-layer MLP projector, and MiniMax-Text-01 as the LLM backbone. The model family choice is motivated by the paper's goal to demonstrate linear attention at commercial scale while maintaining deployment practicality (single-node inference at 1M tokens under 8-bit quantization).

• Metrics. Core text benchmarks are evaluated using accuracy or pass@1 with greedy decoding and zero-shot chain-of-thought prompting unless otherwise noted. MMLU and MMLU-Pro use accuracy. SimpleQA and C-SimpleQA use accuracy. IFEval uses the average across instruction-following categories. GPQA Diamond uses accuracy. DROP uses F1 score. GSM8k and MATH use accuracy. HumanEval and MBPP Plus use Pass@1. Arena-Hard-Auto uses the win rate against a reference model. Long-context benchmarks: RULER uses average accuracy across 13 tasks; LongBench-V2 uses overall accuracy with breakdowns by difficulty (easy, hard) and context length (short: 0–32K words, medium: 32K–128K words, long: 128K–2M words); MR-NIAH uses adjusted recall (correct components divided by 3); MTOB uses ChrF for eng→kalam and BLEURT for kalam→eng. Scaling law experiments use training loss as the primary metric (following Chinchilla methodology), with downstream benchmark accuracy as supplementary evaluation. In-house benchmarks use win rates determined by GPT-4o-based evaluation against reference models, with human expert verification for Safety and Long Context categories. Vision-language benchmarks use the standard metrics reported by each benchmark (accuracy, relaxed accuracy for ChartQA, OCRBench score, F1 for DocVQA).

• Baselines. For core text benchmarks, the paper compares against GPT-4o (11-20 version), Claude-3.5-Sonnet (10-22 version), Gemini-1.5-Pro (002), Gemini-2.0-Flash (experimental), Qwen2.5-72B-Inst., DeepSeek V3, and Llama-3.1-405B-Inst. For long-context benchmarks, comparisons include GPT-4o (11-20), Claude-3.5-Sonnet (10-22), Gemini-1.5-Pro (002), Gemini-2.0-Flash (exp), Qwen-Long (for MTOB), and DeepSeek-V3. For vision-language benchmarks, baselines include GPT-4o (11-20), Claude-3.5-Sonnet (10-22), Gemini-1.5-Pro (002), Gemini-2.0-Flash (exp), Qwen2-VL-72B-Inst., InternVL2.5-78B, and Llama-3.2-90B. For scaling law experiments, the baselines are pure softmax attention and pure lightning attention models at matched parameter counts and training tokens. For MoE ablations, the baseline is a dense 7B model compared against a 2B-activated MoE model. For architectural ablations, the pure softmax attention MoE model (28B total, 5B activated) serves as the baseline against hybrid-lightning.

• Generation budget / compute accounting. For scaling law experiments, compute is measured in PFLOP/s-days following the FLOPs calculation methodology of Kaplan et al. (2020): non-embedding parameters and subleading terms are excluded, and FLOPs are computed as approximately 72 b n l d² times an architecture-specific factor (Table 1). For downstream benchmark evaluations, compute is measured implicitly through model size (activated parameters), context length, and the reported training budget (tokens). The paper does not report FLOPs-matched comparisons for downstream task performance — all benchmark comparisons are at fixed model scale, not fixed compute. For MoE vs. dense isoflop comparisons (Figure 4), compute is measured in ZFlops (10²¹ floating-point operations) with both models trained on 1 trillion tokens. For training speed comparisons (Figure 8), the metric is tokens per GPU per second (TGS) measured on H800 GPUs at varying sequence lengths. For inference latency comparisons (Figure 2), the metric is milliseconds for prefilling, measured on H800 GPUs with 8-bit weight-only quantization (W8A16) for MiniMax-Text-01 and Llama3-70B, and through official APIs for other models, with data fitted using a quadratic function after outlier removal.

• Cross-validation / statistical protocol. Scaling law experiments use training loss as a direct indicator of test performance (following Chinchilla), without cross-validation on downstream benchmarks. Model specification decisions (Section 2.4) use extrapolation from small-scale models (44M–1.2B activated parameters) to the target scale (45.9B activated), with the paper acknowledging that standard extrapolation methods become unreliable and proposing a refined formula (Eq. 14) with an interaction term. Data experiments (Section 4.1.3) follow a formal statistical hypothesis testing framework: models are compared using log accnorm2 as the evaluation metric, with power analysis to determine minimal test sample size guaranteeing 95% confidence and 80% power for detecting an effect size comparable to training variance. The paper does not report confidence intervals or standard errors on benchmark scores. For the in-house user experience benchmark, evaluation uses GPT-4o-based scoring with human verification. There is no explicit cross-validation protocol described for the main benchmark evaluations — results are reported as single-point estimates.

---

Main Quantitative Results

Core Text Benchmarks

MiniMax-Text-01 matches or exceeds leading commercial and open-source models across most standard benchmarks. Table 8 presents the core results. On MMLU, MiniMax-Text-01 achieves 88.5%, matching DeepSeek-V3 (88.5%) and Llama-3.1-405B-Inst. (88.6%), and exceeding GPT-4o (85.7%) and Gemini-2.0-Flash (86.5%), while falling slightly short of Claude-3.5-Sonnet (88.3%). On MMLU-Pro, the model achieves 75.7%, ranking behind Claude-3.5-Sonnet (78.0%) and GPT-4o (74.4%) but ahead of DeepSeek-V3 (75.9%? — the table reads 75.9; MiniMax is 75.7, effectively tied), Qwen2.5-72B-Inst. (71.1%), and Llama-3.1-405B-Inst. (73.3%).

On knowledge-intensive benchmarks, the model shows particular strength in Chinese-language factual knowledge: C-SimpleQA reaches 67.4%, the highest among all compared models — notably above GPT-4o (64.6%), Claude-3.5-Sonnet (56.8%), DeepSeek-V3 (64.8%), and Qwen2.5-72B-Inst. (52.2%). On English SimpleQA, the model's 23.7% is lower than GPT-4o (39.0%) and Claude-3.5-Sonnet (28.1%), but competitive with DeepSeek-V3 (24.9%) and Gemini-1.5-Pro (23.4%).

On instruction-following (IFEval), MiniMax-Text-01 achieves 89.1%, surpassing GPT-4o (84.1%), Qwen2.5-72B-Inst. (87.2%), DeepSeek-V3 (87.3%), and Llama-3.1-405B-Inst. (86.4%), while slightly below Claude-3.5-Sonnet (90.1%). On Arena-Hard (alignment to human preferences), the model reaches 89.1%, exceeding Claude-3.5-Sonnet (87.6%), Gemini-1.5-Pro (85.3%), Qwen2.5-72B-Inst. (81.2%), and dramatically outperforming Llama-3.1-405B-Inst. (63.5%). It falls short of GPT-4o (92.4%) and DeepSeek-V3 (91.4%).

On mathematical reasoning (MATH), MiniMax-Text-01 achieves 77.4%, outperforming GPT-4o (76.6%), Claude-3.5-Sonnet (74.1%), and Llama-3.1-405B-Inst. (73.8%), but trailing DeepSeek-V3 (84.6%), Gemini-1.5-Pro (84.6%), and Gemini-2.0-Flash (83.9%). On GSM8k, the model achieves 94.8%, slightly below all commercial models (95.2–96.9%) and DeepSeek-V3 (96.7%). On graduate-level reasoning (GPQA Diamond), MiniMax-Text-01 achieves 54.4%, exceeding GPT-4o (46.0%) and Qwen2.5-72B-Inst. (49.0%), but below Claude-3.5-Sonnet (65.0%), DeepSeek-V3 (59.1%), and both Gemini versions (59.1–62.1%).

On coding tasks, performance is competitive but not leading: HumanEval achieves 86.9%, below Claude-3.5-Sonnet (93.7%), DeepSeek-V3 (92.1%), GPT-4o (90.2%), and Llama-3.1-405B-Inst. (89.0%), but comparable to Gemini-1.5-Pro (86.6%). MBPP Plus achieves 71.7%, trailing all compared models (range: 73.0–78.8%). The paper attributes this gap to limited coding data in the pre-training corpus (Section 7 limitations).

On reading comprehension (DROP F1), MiniMax-Text-01 achieves 87.8%, below Llama-3.1-405B-Inst. (92.5%), DeepSeek-V3 (91.0%), and most commercial models (88.8–89.3%).

Key overall pattern: MiniMax-Text-01 performs at or near the top of the field on knowledge-intensive tasks (especially Chinese), instruction-following, and alignment, is competitive on reasoning and math (with some models notably ahead), and shows a relative gap on coding and reading comprehension. The paper does not claim superiority on all benchmarks — the consistent finding is comparability to top-tier closed-source and open-source models, with the differentiating advantage being context length, not benchmark scores.

Long-Context Retrieval (MR-NIAH)

MiniMax-Text-01 maintains strong retrieval performance at contexts up to 1M tokens in both English and Chinese, with substantially less degradation than competing models. Figure 15 presents MR-NIAH results. In English, MiniMax-Text-01's adjusted recall remains above 0.9 at all context lengths from 10K to 1M tokens (the red line stays consistently high). In Chinese, the pattern is similar, with the model maintaining high recall across the full range.

The critical comparative finding is less performance degradation at large input lengths. While the paper does not provide exact numerical comparisons for all models at all lengths (the figure uses line plots), the visual evidence shows that Gemini-1.5-Pro, GPT-4o, and Claude-3.5-Sonnet all exhibit downward trajectory as context length increases, while MiniMax-Text-01's line remains relatively flat. This directly supports the paper's claim that the hybrid architecture enables robust retrieval even at extreme context lengths where other models degrade.

The MR-NIAH task is designed to be more demanding than vanilla NIAH: it embeds retrieval requests within multi-turn conversation histories of up to ~2000 interactions, requiring the model to recall a specific response from a specific historical query while distinguishing it from similar but non-identical queries (e.g., multiple requests for penguin poems, as shown in Appendix B.2). The model's success at this task demonstrates retrieval capability beyond simple string matching — it requires semantic understanding of which historical request is being referenced.

Long-Context Understanding (RULER and LongBench-V2)

RULER results (Table 9): MiniMax-Text-01 maintains strong performance through 1M tokens, establishing a distinct advantage beyond 128K. At short contexts (4K–32K), all models perform comparably, with scores in the 0.95–0.97 range and minimal variation. At 64K, MiniMax-Text-01 achieves 0.943 versus 0.884 for GPT-4o and 0.952 for Claude-3.5-Sonnet — all within a tight band.

The divergence begins at 128K, where MiniMax-Text-01 achieves 0.947 versus 0.938 for Claude-3.5-Sonnet and 0.917 for Gemini-1.5-Pro — a gap that widens as context grows. At 256K, MiniMax-Text-01's 0.945 compares to 0.916 for Gemini-1.5-Pro and 0.797 for Gemini-2.0-Flash. At 512K, MiniMax-Text-01 achieves 0.928 versus Gemini-1.5-Pro's 0.861 and Gemini-2.0-Flash's 0.709. At 1M tokens, MiniMax-Text-01's 0.910 far exceeds Gemini-1.5-Pro's 0.850, and Gemini-2.0-Flash cannot operate at this length (no score reported). GPT-4o and Claude-3.5-Sonnet do not support contexts beyond 128K in this evaluation.

The critical finding is that MiniMax-Text-01's 1M-token performance (0.910) is only 0.053 below its 4K-token performance (0.963) — a degradation of just 5.5%. In contrast, Gemini-1.5-Pro drops from 0.962 at 4K to 0.850 at 1M (11.6% degradation), and Gemini-2.0-Flash drops from 0.960 at 4K to 0.709 at 512K (26.1% degradation, with 1M not supported). This supports the paper's architectural claim: lightning attention's linear complexity prevents the accumulation of errors or computational bottlenecks at extreme lengths.

LongBench-V2 results (Table 10): MiniMax-Text-01 achieves state-of-the-art performance among all evaluated models in the with-CoT setting, and leads in the without-CoT setting. In the with-CoT setting, MiniMax-Text-01 achieves 56.5 overall, surpassing GPT-4o (51.4), Claude-3.5-Sonnet (46.7), and Qwen2.5-72B-Inst. (43.5). The advantage is particularly pronounced on easy questions (66.1 vs. 54.2–55.2 for GPT-4o and Claude) and on hard questions (50.5 vs. 41.5–49.7). The model also leads on short contexts (61.7 vs. 48.9–59.6), medium contexts (56.7 vs. 40.9–48.6), and long contexts (47.2 vs. 39.8–44.4).

In the without-CoT setting, MiniMax-Text-01 achieves 52.9 overall, exceeding GPT-4o (50.1), DeepSeek-V3 (48.7), Qwen2.5-72B-Inst. (42.1), and Claude-3.5-Sonnet (41.0). The pattern across difficulty and length categories is consistent: MiniMax-Text-01 leads on easy (60.9 vs. 42.7–57.4), hard (47.9 vs. 37.3–45.6), short (58.9 vs. 45.6–53.3), medium (52.6 vs. 38.1–52.4), and long (43.5 vs. 37.0–44.4).

The paper attributes this exceptional performance to "the hybrid architecture with half RoPE and carefully tuned training recipes for both pre-training and alignment." The RoPE configuration (applied to only half the attention head dimensions) is specifically credited with enabling length extrapolation, while the multi-stage long-context training procedure (Table 6) is credited with building robust long-range reasoning capabilities.

Long In-Context Learning (MTOB)

MiniMax-Text-01 demonstrates the strongest in-context learning capability on the MTOB benchmark, as measured by improvement from no-context to full-book settings. Table 11 presents the MTOB results. In the no-context setting for eng→kalam (ChrF), MiniMax-Text-01 scores only 6.0 — the lowest among all compared models, far below GPT-4o (9.90), Claude-3.5-Sonnet (20.22), Gemini-1.5-Pro (16.79), and Qwen-Long (16.55). The paper explicitly notes that "only a very small amount of data contains Kalamang-related content" in their pre-training data, while "other models we compared with likely have had their pre-train or post-train data enhanced with relevant Kalamang data." This establishes that MiniMax-Text-01 enters the task with essentially no prior knowledge of Kalamang.

However, when provided with the full grammar book (~133K tokens) and 375 parallel translation examples in context, MiniMax-Text-01's ChrF score rises to 51.60 — an improvement of 45.6 points from the no-context baseline. This delta (Δ full book = 45.6) is the largest among all models: GPT-4o's full-book score is not reported, but its half-book improvement is 44.40; Claude-3.5-Sonnet's full-book improvement is 35.42; Gemini-1.5-Pro achieves 41.11; Gemini-2.0-Flash achieves 41.10; and Qwen-Long achieves only 29.39.

For kalam→eng (BLEURT), the pattern is more mixed. In the no-context setting, MiniMax-Text-01 scores 33.65 — comparable to Gemini-2.0-Flash (33.80) and GPT-4o (33.20), and above Claude-3.5-Sonnet (31.42). With the full book, MiniMax-Text-01 reaches 58.00, an improvement of 24.35, which is competitive with but not leading: Gemini-1.5-Pro achieves 31.07 improvement, Claude-3.5-Sonnet achieves 30.88, and GPT-4o achieves 25.10 (half-book only reported at 58.30). Qwen-Long notably degrades to 2.02 improvement — its full-book score (32.15) barely exceeds its no-context score (30.13), suggesting it cannot effectively utilise the long context for this direction.

The paper interprets these results as evidence of genuine in-context learning rather than retrieval of pre-trained knowledge. Because MiniMax-Text-01 starts from near-zero on the target language, the dramatic improvement with context can only come from processing and applying the provided reference materials. The paper also notes (Figure 16) that in-context learning ability "gradually enhanced" during the long-context extension training process — MTOB metrics improved as training progressed from 128K to 512K to 1M context stages, providing a training signal that NIAH (which saturates early) cannot provide.

In-House Benchmarks

Table 12: MiniMax-Text-01 substantially outperforms all compared models on Creative Writing and Long Context in-house evaluations, leads on General Assistant and Knowledge Q&A, and is competitive on Safety, while showing gaps on Instruction Following.

• General Assistant: 73.9, leading over GPT-4o (70.9), Gemini-2.0-Flash (70.1), and all others (53.3–67.7).
• Creative Writing: 81.3, dramatically leading over all models — the next best is GPT-4o (70.3), and Claude-3.5-Sonnet scores only 54.3. This is a 11-point gap over the nearest competitor.
• Knowledge Q&A: 78.6, leading over DeepSeek-V3 (77.0), Gemini-2.0-Flash (75.1), and GPT-4o (69.2).
• Long Context: 93.8, significantly ahead of GPT-4o (86.2), Gemini-2.0-Flash (81.9), Qwen2.5-72B-Inst. (81.5), and dramatically ahead of Claude-3.5-Sonnet (47.1 — a 46.7-point gap).
• Safety: 90.9, behind Claude-3.5-Sonnet (92.9) but ahead of GPT-4o (85.4) and DeepSeek-V3 (74.9).
• Hard Capability: 64.8, below GPT-4o (73.5), DeepSeek-V3 (68.7), and Claude-3.5-Sonnet (68.3).
• Instruction Following: 46.3, below Claude-3.5-Sonnet (61.5), GPT-4o (50.4), DeepSeek-V3 (51.8), and Llama-3.1-405B-Inst. (50.3).
• Coding: 90.2, trailing GPT-4o (94.0), Claude-3.5-Sonnet (94.4), DeepSeek-V3 (94.0), and Qwen2.5-72B-Inst. (93.9).

The paper acknowledges the Instruction Following gap, attributing it to "insufficient training data for specific instruction types" and committing to expand training data in future work. The Hard Capability gap suggests that demanding multi-level instruction-following scenarios remain challenging. However, the massive leads in Creative Writing (81.3 vs. 70.3 for GPT-4o) and Long Context (93.8 vs. 86.2 for GPT-4o) are presented as evidence that the model's architecture and training specifically excel at tasks where the long-context capability provides a qualitative advantage.

Scaling Law Experiments

Figure 6 and Table 2: Hybrid-lightning achieves the lowest training loss across the measured compute range (10⁻² to 10³ PFLOP/s-days), and its optimal allocation favours more data and fewer parameters compared to softmax attention.

• Loss vs. Compute (Figure 6, left): Hybrid-lightning achieves consistently lower loss than softmax attention at all compute budgets, with the gap appearing roughly constant on the log-log plot. Lightning attention alone achieves intermediate loss — below softmax but above hybrid-lightning. The fitted scaling law (Table 2) gives L(C) = 3.4797 · C^{-0.0763} for hybrid-lightning vs. L(C) = 3.7087 · C^{-0.0798} for softmax attention. The more negative exponent for softmax (−0.0798 vs. −0.0763) means softmax loss improves faster per doubling of compute, but the lower constant for hybrid-lightning (3.4797 vs. 3.7087) means hybrid-lightning starts from a lower loss and maintains that advantage across the entire measured range.

• Optimal model size (Figure 6, centre): Softmax attention allocates proportionally more compute to scaling model size: N_opt ∝ C^{0.7118} vs. C^{0.6670} for hybrid-lightning. At very large compute budgets, softmax models would be larger than hybrid models trained with the same FLOPs.

• Optimal training tokens (Figure 6, right): Hybrid-lightning allocates proportionally less to data: D_opt ∝ C^{0.4707} vs. C^{0.5102} for softmax. However, since hybrid-lightning processes tokens faster (Figure 8), the wall-clock tradeoff is more nuanced.

Figure 7: Pure lightning attention matches softmax attention on commonsense reasoning at all scales but catastrophically fails at retrieval (NIAH). Hybrid-lightning matches or exceeds softmax on all benchmarks.

At 7B parameters:

• Commonsense Reasoning (CSR average): Softmax 58.3, Lightning ~57.5 (estimated from bars), Hybrid-lightning ~58.5. All three architectures are within ~1 point.
• NIAH: Hybrid-lightning achieves 97.7%, matching softmax at 97.7%. Pure lightning attention's NIAH score is not shown on the 7B chart (the bar is absent or near-zero, consistent with the 15.5% result at 3B).
• SCROLLS: Hybrid-lightning achieves ~15.5, exceeding lightning attention (~14.5) and softmax attention (~14.0 — estimated from bars).

Figure 8: Lightning attention training speed (TGS) remains constant regardless of sequence length, while softmax attention speed drops sharply and runs out of memory at 65536. Pure lightning attention maintains ~14,000 TGS from 1024 to 65536 tokens. Hybrid-lightning maintains ~10,000–12,000 TGS across the same range. Softmax attention drops from ~17,000 at 1024 to ~10,000 at 16384 and OOM at 65536. Mamba2 and HGRN2 operate at lower TGS than lightning attention and also degrade at long sequences. Lightning attention is the only linear model that outperforms FlashAttention-2 in throughput.

MoE vs. Dense Isoflop Comparison

Figure 4: At matched compute budgets, a 2B-activated MoE model consistently outperforms a 7B dense model across five benchmarks. At ~9 ZFlops (roughly the midpoint of the training curves):

• HellaSwag: MoE ~0.68 vs. Dense ~0.63
• WinoGrande: MoE ~0.66 vs. Dense ~0.62
• Natural Questions: MoE ~0.14 vs. Dense ~0.10
• PIQA: MoE ~0.775 vs. Dense ~0.76
• TriviaQA: MoE ~0.44 vs. Dense ~0.35

The dashed lines indicate that the MoE model achieves the dense model's final performance using roughly half the compute. This validates the MoE design choice for scaling to 456B total parameters.

Ablations Within the MoE Framework

Table 5 (Hybrid-lightning vs. Softmax in MoE): In a 28B-total / 5B-activated MoE trained on 1T tokens, hybrid-lightning outperforms softmax attention on 6 of 8 benchmarks, with notable gains on BBH (+4.0, from 28.2 to 32.2) and MATH (+2.2, from 4.6 to 6.8). Underperformance is marginal on CMMLU (−1.3) and GSM8k (−0.3).

Table 5 (PostNorm with DeepNorm vs. PreNorm): In a 9.3B-activated / 60B-total model trained on 500B tokens, PostNorm with DeepNorm outperforms PreNorm on all 8 benchmarks. Dramatic improvements appear on MMLU (+6.3, from 43.9 to 50.2), CMMLU (+7.4, from 41.8 to 49.2), and GSM8k (+4.6, from 12.2 to 16.8). BBH improves by +2.7 (29.9 to 32.6) and ARC-C by +2.7 (43.5 to 46.2).

---

Ablation Studies and Robustness Checks

Hybrid-lightning attention vs. pure softmax attention in MoE (Table 5): At 28B total / 5B activated parameters on 1T tokens, hybrid-lightning outperforms softmax attention on 6/8 benchmarks. BBH improves from 28.2 to 32.2 (+4.0), MATH from 4.6 to 6.8 (+2.2), and DROP from 27.4 to 29.0 (+1.6). MMLU is essentially tied (49.3 vs. 49.5). The marginal underperformance on GSM8k (18.8 vs. 18.5) and CMMLU (47.3 vs. 46.0) does not clearly favour either architecture. This validates that the lightning attention layers do not harm MoE performance and provide modest improvements on complex reasoning tasks.

Pre-norm vs. Post-norm with DeepNorm (Table 5): At 9.3B activated / 60B total parameters on 500B tokens, PostNorm with DeepNorm outperforms PreNorm on every benchmark. The largest gains are on knowledge-intensive tasks: MMLU +6.3 (43.9→50.2) and CMMLU +7.4 (41.8→49.2). The mathematical reasoning gains are notable: GSM8k +4.6 (12.2→16.8) and MATH +0.9 (4.8→5.7). The paper attributes this to PostNorm preserving the model's effective depth (gradients must pass through sub-layers rather than bypassing them via residual connections), which is particularly important for the deep (80-layer) hybrid architecture.

Hybrid-linear variant comparison (Table 3): At 1B parameters, hybrid-lightning (CSR 49.55, NIAH 95.7, SCR 13.3, TGS 33.4K) dominates hybrid-hgrn2 (CSR 49.87, NIAH 91.8, SCR 10.8, TGS 29.5K) and hybrid-cosformer2 (CSR 46.64, NIAH 43.6, SCR 10.9, TGS 23.3K). The NIAH gap between hybrid-lightning and hybrid-cosformer2 is 52.1 points — demonstrating that not all linear attention variants are equally suitable for the hybrid approach. The cosformer2 architecture's linear attention mechanism likely has different information-loss properties that make it less compatible with periodic softmax retrieval.

Hybrid-window vs. hybrid-lightning (Table 4): At both 1B and 3B parameter scales, hybrid-window (sliding window attention + periodic full softmax) achieves dramatically lower NIAH scores than hybrid-lightning at equivalent training throughput. At 1B with window 1024 (TGS 33.6K vs. 33.4K): NIAH 53.9 vs. 95.7 — a 41.8-point gap. At 3B with window 1024 (TGS 15.4K vs. 15.1K): NIAH 41.6 vs. 98.0 — a 56.4-point gap. Larger window sizes improve NIAH marginally (57.9 at 512 and 40.9 at 256 for 3B) but never approach hybrid-lightning. This is a strong negative result for sliding window approaches: even with periodic full-attention layers, the limited window fundamentally restricts the model's ability to retrieve information from beyond the window size.

Isoflop comparison: MoE vs. Dense (Figure 4): Across HellaSwag, WinoGrande, Natural Questions, PIQA, and TriviaQA, a 2B-activated MoE model consistently matches or exceeds a 7B dense model at approximately half the compute. The dashed lines in Figure 4 show the compute difference needed to achieve equivalent performance. This is a robustness check for the MoE design choice — the efficiency gain is not restricted to a specific benchmark or task type, but generalises across commonsense reasoning, factual knowledge, and reading comprehension.

Scaling law comparisons across architectures (Figure 6, Table 2): The power-law fits are derived from models spanning 70M to 7B parameters trained on up to 300B tokens. The consistency of the trends across this range (loss, optimal model size, and optimal data all following clean power laws on log-log plots) provides confidence that the scaling relationships are not artifacts of a particular model size. However, the paper acknowledges that "predictions from these methods become less reliable when extrapolating to a larger model with 9.3 billion parameters" (Section 2.4), motivating the refined formula in Eq. 14 with an interaction term.

Data repetition experiments (Section 4.1.3.2): The paper's repetition-aware framework reveals that "low-quality data suffer a substantial decrease in performance after training for more than two epochs, while high-quality data can be effectively trained for up to four epochs." This finding is described as "similar to previous observations (Muennighoff et al., 2023)" but is obtained through a more rigorous experimental design that better aligns small-scale experiments with full-scale training outcomes. The paper claims the solution from this framework "yields better alignment with the results obtained using considerably more computational resources."

Long-context extension monitoring (Section 4.2): The paper reports that NIAH "is inadequate for effectively monitoring the model's performance throughout the training process" because "NIAH metric performance reaches its peak score early on, specifically within the initial 128K training steps." As a robustness check, the paper uses "more demanding tasks" to evaluate intermediate checkpoints, which show "a steady improvement in the model's performance metrics" despite NIAH saturation. This validates that the long-context continual pre-training provides genuine capability improvements beyond the simplistic retrieval tasks that are commonly used for monitoring.

---

Critical Assessment

Claim: The hybrid architecture achieves lower training loss than pure softmax attention under identical compute budgets

This claim is well-supported by the scaling law experiments (Figure 6, Table 2) but with two important caveats. First, the comparison is based on training loss, not downstream task performance. The paper states that "training loss serves as a direct indicator of test performance" (following Chinchilla), but does not provide downstream benchmark comparisons at matched FLOPs for the scaling law models. Figure 7 provides downstream benchmarks but at matched parameter counts, not matched FLOPs — since the architectures have different FLOPs-per-token (Table 1), the comparisons in Figure 7 are not compute-matched.

Second, the scaling law experiments use a fixed learning rate schedule ("due to constrained computational resources") rather than tuned schedules per architecture. If the optimal learning rate schedule differs between softmax and hybrid-lightning (plausible given their different loss landscapes), the comparison may not reflect the best possible performance for either architecture. This is not a fatal flaw — the Chinchilla methodology itself uses simplified schedules — but it adds uncertainty to the quantitative claims about exponent differences.

The downstream benchmark results that substantiate the claim come from Table 5 (hybrid-lightning outperforms softmax in MoE on 6/8 benchmarks) and Figure 7 (hybrid-lightning matches or exceeds softmax on all benchmarks at 7B). However, these are at specific model sizes, not isoflop — they demonstrate that hybrid-lightning performs well, but do not isolate whether the advantage comes from architecture or from more effective use of the same FLOPs.

Claim: MiniMax-Text-01 matches the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet on standard academic benchmarks

This claim is generally supported but requires qualification on specific benchmarks. On MMLU (88.5), GPQA (54.4), MATH (77.4), and IFEval (89.1), MiniMax-Text-01 is competitive with or exceeds GPT-4o. However, it consistently underperforms Claude-3.5-Sonnet on GPQA (54.4 vs. 65.0), MMLU-Pro (75.7 vs. 78.0), and coding tasks (HumanEval 86.9 vs. 93.7). It underperforms DeepSeek-V3 on MATH (77.4 vs. 84.6 — a 7.2-point gap) and coding. The claim of "matching" performance is more accurate for knowledge and instruction-following than for reasoning and coding.

A more precise characterisation would be: MiniMax-Text-01 operates in the same tier as top commercial models, with specific strengths (Chinese factual knowledge, instruction following, alignment to human preferences) and weaknesses (advanced mathematics, coding). The paper's framing in Figure 1 and Section 5.7.1 is fair — the model is "comparable" without claiming superiority — but readers should not interpret this as uniform parity across all capabilities.

A significant methodological concern: the paper uses different evaluation protocols for different benchmarks without systematically documenting the impact. Core benchmarks use "0-shot CoT" while commercial model scores may be reported from official sources using different prompting strategies. The paper states it evaluates comparison models "under the same setting, if not reported," implying that some scores are taken from published results that may use different prompts, temperatures, or evaluation scripts. This undermines the precision of head-to-head comparisons.

Claim: The model supports context windows of up to 4 million tokens at inference with outstanding long-context performance

This claim is supported by the 4M-token NIAH test (Figure 14) and the 1M-token RULER results (Table 9), but with important limitations on what is demonstrated. Figure 14 shows a vanilla NIAH heatmap at 4M tokens — the model successfully retrieves the needle at all depth percentages and context lengths up to 4M. However, vanilla NIAH is explicitly described by the paper as a task that "reaches its peak score early on" and is "inadequate for effectively monitoring the model's performance." The more demanding MR-NIAH (Figure 15) and RULER (Table 9) evaluations only extend to 1M tokens, not 4M. The paper does not provide any evidence of the model's reasoning or multi-hop retrieval capabilities at the 4M-token scale — only simple needle retrieval.

The claim of "outstanding long-context performance" is well-supported at 1M tokens (RULER 0.910, MR-NIAH >0.9, LongBench-V2 56.5 overall) and supported only by a weak proxy (vanilla NIAH) at 4M tokens. The paper would be stronger with RULER or LongBench-V2 evaluations at 2M or 4M tokens to validate that the 1M-token performance trends continue.

Claim: Linear attention has been successfully deployed at commercial scale for the first time

This claim is supported by the existence of the fully trained, publicly released 456B-parameter model. The paper reports concrete deployment metrics: >75% MFU on H20 GPUs, <12% of total attention latency from lightning attention at 1M-token contexts, and practical inference on a single 8-GPU node. These are not speculative — they come from a production system that is publicly accessible via API and open-source weights.

However, the paper does not provide ablation evidence quantifying the contribution of lightning attention specifically to the final model's benchmark performance. The module ablation in Table 5 shows hybrid-lightning outperforming pure softmax in an MoE at 28B/5B scale, but the fully trained 456B model is never compared against a 456B pure softmax model (which would be prohibitively expensive to train). The claim that lightning attention "works at commercial scale" is demonstrated by the model's existence and performance, but the specific contribution of lightning attention (as opposed to the MoE design, the training data, the post-training pipeline, etc.) cannot be isolated from the full system.

Missing Experiments That Would Strengthen the Paper

No FLOPs-matched downstream comparison between architectures. The scaling law experiments establish that hybrid-lightning achieves lower loss at equal compute, but this is not translated into downstream benchmark predictions. A FLOPs-matched comparison on a downstream benchmark suite at a smaller scale (e.g., 3B parameters) would bridge the gap between the scaling law analysis and the full-scale model's performance.

No ablation of the softmax-to-lightning ratio at scale. The 7:1 ratio is determined from small-scale experiments, but the paper does not report how sensitive performance is to this ratio. Would 15:1 also work? Would 3:1 be significantly better? At the 456B scale, even training a single alternative ratio would be expensive, but a parameter sweep at smaller scale with careful extrapolation would provide confidence that 7:1 is genuinely near-optimal rather than merely sufficient.

No comparison against pure softmax attention with the same training budget. The paper compares against commercial models (GPT-4o, Claude) and other open-source models, but not against an internally trained pure softmax model at the same total parameter count and training tokens. This makes it impossible to determine whether the hybrid architecture's benchmark performance is better than, worse than, or equal to what a pure softmax architecture would achieve with the same resources. The module ablation in Table 5 is at much smaller scale (28B/5B vs. 456B/45.9B).

No evaluation of the revision or verifier-guided search mechanisms that the example paper summary discusses — these concepts are not applicable to the MiniMax-01 paper, which focuses on architectural and systems innovations rather than test-time compute strategies. This is not a weakness of the paper, but a difference in research focus.

Limited difficulty-stratified analysis. The paper does not break down benchmark performance by question difficulty or input length (except for the long-context benchmarks, which are naturally stratified by context length). For standard benchmarks like MMLU or MATH, it would be informative to know whether the model's performance relative to baselines is uniform across difficulty levels or concentrated in specific regimes.

Over-Optimisation and Verifier Robustness

The concepts of verifier over-optimisation and reward hacking discussed in the example paper summary are not applicable here — MiniMax-01 does not employ iterative search against a reward model. The paper's reinforcement learning phase uses a modified GRPO with importance sampling weight clipping and a reformulated KL divergence term (Section 5.4.2), but the paper does not evaluate whether the reward model is over-optimised during online RL. The safety alignment (Section 5.5) uses a harmless reward model, but there is no analysis of whether the model exploits the reward model's imperfections. This is a gap: online RL with an imperfect reward model risks reward hacking, and the paper provides no evidence about whether this occurred or was prevented by the KL penalty and clipping mechanisms.

Statistical Rigour

The paper provides no confidence intervals, standard errors, or statistical significance tests for any benchmark results. All scores are reported as point estimates. For benchmarks with 500–1000 test examples (common in the evaluations used), a difference of 1–2 percentage points may not be statistically significant. This is standard practice in the LLM evaluation literature, but it limits the precision of claims about which model is "better" on closely matched benchmarks.

The data experimentation methodology (Section 4.1.3) is described with statistical rigour (power analysis, 95% confidence, 80% power), but these statistical properties are not carried through to the final model evaluation. The contrast between the careful statistical design of the data experiments and the informal reporting of benchmark results is notable.

Hard Problems and the Limits of Test-Time Compute

The paper does not extensively analyse failure cases. The long-context benchmarks show strong performance up to 1M tokens, but the paper does not report whether there are specific sub-tasks or difficulty levels where the model degrades. The in-house Hard Capability benchmark (64.8, below GPT-4o's 73.5) and Instruction Following benchmark (46.3, well below Claude-3.5-Sonnet's 61.5) indicate that the model has specific weaknesses, but these are not systematically diagnosed.

For long-context tasks, the paper does not evaluate whether the model's reasoning quality degrades as context length increases — it only reports aggregate scores (RULER average across 13 tasks, LongBench-V2 overall). A breakdown of performance by task type at different context lengths would reveal whether specific reasoning capabilities (e.g., multi-hop tracing, aggregation, comparison) degrade at long contexts even if aggregate scores remain high.

6. Limitations and Trade-offs

The Hybrid Architecture Does Not Eliminate Softmax Attention — It Reduces It to 1/8 of Layers

The assumption or constraint. The paper explicitly retains standard softmax attention in 10 of 80 layers (one every eighth block) because pure lightning attention catastrophically fails at retrieval. Section 2.2.2.3 states:

"lightning attention demonstrates comparable performance across most downstream tasks, with the exception of NIAH. This indicates that linear attention exhibits similar language modeling capabilities to Transformer models but falls short in retrieval tasks, rendering it unsuitable for LLMs."

The paper further acknowledges in Section 7 that "the model currently retains a 1/8 component with vanilla softmax attention" and frames eliminating this entirely as future work.

The consequence. The 1/8 softmax layers still incur quadratic complexity. At million-token contexts, these layers dominate inference cost. The paper's own data reveals the magnitude: at 1,024,000 tokens, "the softmax attention constitutes 95% of the latency at a sequence length of 1,024,000 tokens" while "the lightning attention implementation contributes to less than 12% of the latency" (Section 3.3.4). This means that despite 7/8 of layers using linear attention, the total inference latency is still dominated by the softmax layers. Scaling to contexts beyond 4 million tokens — which the architecture theoretically supports in its linear attention layers — would cause the softmax component to grow quadratically, eventually overwhelming the linear attention layers' efficiency. The hybrid design postpones the quadratic bottleneck but does not eliminate it; it shifts the wall from (roughly) 256K to (roughly) 4M tokens, but the wall still exists.

What evidence exists in the paper. Figure 2 shows MiniMax-Text-01's prefilling latency growing roughly linearly up to 1M tokens (fitted with a quadratic function, but the curve appears near-linear in the measured range). The 95% figure is reported in Section 3.3.4. The fact that softmax layers still run at token length 1,024,000 means the quadratic term is present but not yet dominant — extrapolating to 10M or 100M tokens would cause these layers to become prohibitive.

Mitigation status. The paper does not address this in the current architecture. Section 7 proposes investigating "more efficient architectures that can eliminate softmax attention entirely, potentially enabling unlimited context windows without computational overhead," indicating this is a recognised limitation but unresolved. No experiments explore whether the softmax layers could be further reduced (e.g., 1/16 ratio) at the 456B scale.

---

Difficulty Estimation for Deployment Is Not Addressed — The Model Cannot Dynamically Allocate Compute

The assumption or constraint. The paper's architecture processes all tokens uniformly regardless of input difficulty or required retrieval precision. Every token passes through the same 80-layer stack with the same 7:1 ratio and the same 32-expert MoE routing. There is no mechanism for the model to "decide" that a particular input requires more softmax attention (for precise retrieval) or can rely more heavily on lightning attention (for efficient processing). The paper does not discuss difficulty estimation, adaptive compute allocation, or any form of dynamic inference.

The consequence. In deployment, many user queries are short or simple — a single question, a brief instruction — and do not require million-token context processing. For these inputs, the softmax attention layers are unnecessary overhead; a pure lightning attention model (or one with even fewer softmax layers) might suffice. Conversely, some queries within a long context may require precise retrieval from specific positions (e.g., fact-checking a claim against a specific paragraph in a book), and the model has no way to "focus" its softmax budget on the critical regions — softmax attention is applied uniformly at layers 8, 16, 24, ..., 80 regardless of where in the sequence the retrieval-demanding content resides. The fixed architecture treats all tokens and all layers identically, leaving potential efficiency gains on the table for heterogeneous inputs.

What evidence exists in the paper. None directly. The paper provides no analysis of whether retrieval accuracy varies with the position of the target information relative to the softmax layers (e.g., does retrieval work better for content that appears immediately before a softmax layer vs. content that appears after?). The long-context evaluations (RULER, LongBench-V2, MR-NIAH) report aggregate scores without position-stratified breakdowns. The uniform architecture implicitly assumes that the 7:1 ratio is globally optimal, but Section 2.4 notes that this ratio was chosen from small-scale experiments — whether it is optimal for all sequence positions and all task types at the 456B scale is unverified.

Mitigation status. Not addressed. The paper does not discuss adaptive computation, early exiting, or dynamic layer selection. The uniform architecture is a deliberate design choice for training simplicity, but it represents an unexplored optimisation dimension.

---

Difficulty Estimation Cost for Data Curation Is Not Amortised in the Reported Training Budget

The assumption or constraint. The paper's data experimentation methodology (Section 4.1.3) requires training multiple small-scale MoE models (1B activation, 8B total parameters) on 40B tokens each to evaluate data quality, format, and mixture decisions. The paper also describes using a "previous-generation model as the reward labeler" for assessing data quality across dimensions like knowledge depth, helpfulness, and categorical relevance (Section 4.1.1). These preliminary experiments and labelling runs consume compute that is not included in the reported training budget for MiniMax-Text-01.

The consequence. The headline training cost for MiniMax-Text-01 — training a 456B-parameter model on ~10T+ tokens (estimated from the learning rate schedule: constant LR for 7.2T tokens, reduced LR for 3.2T tokens, decay for 1T tokens) — excludes the cost of the data experimentation and labelling pipeline that made this training effective. For an organisation seeking to replicate the approach, these preliminary costs could be substantial. The data experimentation step alone (40B tokens × multiple configurations) may represent 1–5% of the final training budget, and the reward labelling of the full pre-training corpus using the previous-generation MoE model adds further unquantified cost. The paper's claims about training efficiency should be understood as applying to the final training run, not the end-to-end development process.

What evidence exists in the paper. Section 4.1.3 describes training MoEs of 1B activation and 8B total parameters with 40B tokens "where data mixture comprises 20B web documents and 20B data of hypothesis." The number of such experiments is not specified, but "extensive ablation experiments" are mentioned. Section 4.1.1 mentions using "our previous-generation model as the reward labeler (a MoE model with 5B activations and 60B total parameters)" for data quality enhancement. Neither cost is quantified.

Mitigation status. The paper does not quantify these costs or include them in any reported budget. This is standard practice in the LLM literature — data curation costs are typically externalised from training budget reporting — but it means the paper's efficiency claims are specific to the final training phase.

---

The 14× Pretraining Baseline Comparison Is Absent; No Evidence Quantifies the Architecture's Advantage Over Pure Softmax at the 456B Scale

The assumption or constraint. The paper never trains or evaluates a pure softmax attention model at the 456B / 45.9B activated scale. The evidence that hybrid-lightning outperforms softmax attention comes exclusively from small-scale experiments: scaling law fits from 70M–7B parameter models (Section 2.2.2.2, Figure 6, Table 2), a 28B/5B MoE ablation (Table 5), and 1B and 3B comparisons against hybrid-window and hybrid-linear variants (Tables 3–4). The full-scale MiniMax-Text-01 is compared only against external commercial and open-source models, not against an internally trained pure softmax counterpart.

The consequence. It is impossible to determine whether the 456B hybrid model's benchmark performance (e.g., 88.5 MMLU, 77.4 MATH, 54.4 GPQA) is better than, worse than, or equal to what a 456B pure softmax model would achieve with identical training data and compute. The scaling laws predict lower loss for hybrid-lightning at matched FLOPs, but this prediction has not been validated at the target scale. The architecture's contribution to final performance is confounded with the MoE design, the training data quality, the multi-stage training procedure, and the post-training pipeline. A sceptical reading could attribute the model's strong performance to any of these factors, with the hybrid attention being an orthogonal efficiency improvement rather than a quality improvement.

What evidence exists in the paper. The scaling law fits (Figure 6) show consistent trends from 70M to 7B, and the paper proposes a refined formula (Eq. 14) to improve extrapolation reliability. However, the paper itself acknowledges in Section 2.4 that "predictions from these methods become less reliable when extrapolating to a larger model with 9.3 billion parameters," and the target scale (45.9B activated) is 5× larger than this acknowledged reliability limit. The module ablation in Table 5 (28B/5B) is the largest direct comparison and shows hybrid-lightning outperforming softmax on 6/8 benchmarks, but at a scale 16× smaller than the final model.

Mitigation status. The paper is transparent about the extrapolation challenge and proposes Eq. 14 to address it, but cannot fully resolve it without training a comparison model at scale. Section 7 does not propose training a softmax counterpart as future work, focusing instead on eliminating softmax attention entirely. This limitation is partially inherent — training a 456B pure softmax model would roughly double the paper's already massive compute expenditure — but it means the central architectural claim rests on extrapolation from smaller scales.

---

The Long-Context Evaluations at 4 Million Tokens Rely on a Trivially Simple Task; No Complex Reasoning Is Demonstrated Beyond 1 Million Tokens

The assumption or constraint. The paper's headline claim of 4-million-token context support is validated only through the vanilla Needle-in-a-Haystack (NIAH) retrieval test (Figure 14). The paper explicitly acknowledges the weakness of this evaluation in Section 4.2: "NIAH is inadequate for effectively monitoring the model's performance throughout the training process. This is primarily because NIAH metric performance reaches its peak score early on, specifically within the initial 128K training steps." The more demanding evaluations — RULER (13 tasks including multi-hop tracing and aggregation), LongBench-V2 (complex reasoning across diverse context types), MR-NIAH (multi-round retrieval from conversation histories), and MTOB (in-context language learning) — are conducted only up to 1 million tokens (Tables 9–11, Figure 15). The 4M-token claim therefore rests entirely on the model's ability to retrieve a single, distinctively formatted sentence from a long text — a task the paper itself characterises as inadequate for monitoring model progress.

The consequence. There is no evidence that the model can reason, aggregate, compare, or perform multi-hop inference over contexts between 1M and 4M tokens. A practitioner needing to process a 3-million-token document for complex analysis (e.g., legal discovery across multiple lengthy contracts, synthesis of a multi-volume scientific reference) cannot rely on the paper's evaluations to predict model performance — they would be extrapolating from the 1M-token RULER and LongBench-V2 results without validation. The model might maintain performance, or it might degrade sharply; the paper provides no evidence either way.

What evidence exists in the paper. Figure 14 shows a heatmap of vanilla NIAH at 4M tokens (token interval 0.5M beyond 1M) with uniformly high retrieval scores. Section 4.2 notes that the model's length extrapolation capabilities "enable it to process sequences up to 4M tokens in the vanilla Needle-In-A-Haystack retrieval task test, despite only being trained on contexts up to 1M tokens." No RULER, LongBench-V2, or other complex reasoning results are reported at 2M, 3M, or 4M tokens.

Mitigation status. The paper does not address this gap. Future work in Section 7 mentions "enhancing long-context retrieval in more realistic settings" and "expanding the evaluation of long-context reasoning across a wider array of tasks," but does not explicitly commit to evaluating at scales beyond 1M tokens. The limitation is partially constrained by the availability of benchmarks — few standard long-context evaluation datasets extend beyond 1M tokens — but the paper's own MR-NIAH benchmark could be extended to 4M tokens for at least retrieval evaluation, and this is not reported.

---

The Revision Model and PRM Search Are Not Combined; the Two Core Mechanisms Remain Independent

The assumption or constraint. This limitation does not apply to the MiniMax-01 paper, which does not study iterative revisions or process reward model (PRM) search. This limitation would only be relevant if the paper had explored test-time compute strategies, which it does not. The MiniMax-01 paper's contributions are architectural and systems-level; it does not investigate inference-time computation allocation, verifier-guided generation, or self-correction mechanisms.

However, a closely related limitation specific to MiniMax-01 is worth noting as a replacement:

The Linear Attention Inference Optimisations Are Tuned for H20/H800 GPUs; Generalisability to Other Hardware Is Unverified

The assumption or constraint. The inference optimisations in Section 3.3 — batched kernel fusion, separated prefill and decoding execution, multi-level padding, and strided batched matmul extensions — are implemented and benchmarked on NVIDIA H20 and H800 GPUs. The paper reports "over 75% Model Flops Utilisation (MFU) end-to-end on the Nvidia H20" (Section 3.3.4) and notes that "these optimizations can bring very noticeable benefits on H20 compared to H800" (Section 3.3.1). The CUDA kernel design makes specific assumptions about GPU architecture: the use of WGMMA instructions targets Hopper-generation GPUs (Section 3.3.4), and the Tensor Memory Accelerator (TMA) is Hopper-specific. The memory bandwidth and compute balance of H20 (high memory bandwidth relative to compute) differ from other GPUs like A100, H100, or upcoming hardware generations.

The consequence. The deployed efficiency of MiniMax-Text-01's lightning attention layers may not transfer to other hardware platforms without re-optimisation. On GPUs with different memory/compute ratios (e.g., H100 with higher compute throughput relative to memory bandwidth), the memory-bound optimisations (kernel fusion, multi-level padding) may provide less benefit, while the compute-bound optimisations (WGMMA, strided batched matmul) may provide more. On non-NVIDIA hardware (AMD, Intel, Apple Silicon, custom AI accelerators), the CUDA-specific optimisations would need to be entirely reimplemented. The paper's claim that lightning attention is efficient for inference is empirically supported only for the Hopper-generation NVIDIA GPUs used in testing.

What evidence exists in the paper. All performance measurements (Figures 2, 8; Sections 3.2.2, 3.3.4) are on H800 or H20 GPUs. The paper describes hardware-specific features (WGMMA, TMA, CUDA streams, cublasGemmStridedBatchedEx) that are NVIDIA-proprietary. The paper's statement that they "dynamically regulate the number of pipeline stages to adaptively attain optimal performance across both H20 and H800 GPU architectures" (Section 3.3.4) demonstrates awareness of hardware sensitivity but does not extend beyond these two NVIDIA models.

Mitigation status. The paper does not address generalisability to other hardware platforms. The open-source release of the model weights and code (https://github.com/MiniMax-AI) partially mitigates this by enabling the community to port and optimise for other hardware, but the paper provides no analysis of what performance to expect. The focus on single-node inference with 8×80GB GPUs constrains the deployment target to a specific hardware class, and the claim of "affordable cost" (Section 1) is implicitly tied to this hardware configuration.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes the long-context problem from a capability question to an efficiency question, and in doing so, shifts the burden of proof in the field. Before MiniMax-01, the dominant narrative was that supporting million-token contexts required accepting either catastrophic performance degradation (as quadratic attention costs overwhelm hardware) or unacceptable quality degradation (as linear alternatives proved incapable of retrieval). The paper demonstrates that neither tradeoff is necessary — a hybrid architecture can match top-tier commercial performance on standard benchmarks while processing 20–32× longer contexts than competitors, with near-linear latency scaling in the measured range (Figure 2). This is not an incremental refinement of existing attention mechanisms; it redefines the Pareto frontier of what is simultaneously achievable in model quality and context length.

The conceptual shift is best characterised as a diagnostic reframing rather than a paradigm shift. The paper does not invent linear attention (which has existed for nearly a decade, as Section 2.2.2 notes) or MoE (widely deployed in models like DeepSeek-V3 and Mixtral). Its contribution is identifying the specific reason linear attention failed in practice — the retrieval gap, traced to the absence of the recomputation mechanism that makes softmax attention "Go Through a Book" at each time step (Section 2.2.4, Eqs. 10–12) — and demonstrating that a simple architectural intervention (periodic softmax layers as retrieval checkpoints) closes this gap entirely while preserving linear attention's efficiency. This diagnostic framework changes how the field should think about attention mechanism design: the goal is not to maximise capacity or minimise FLOPs in isolation, but to engineer complementary mechanisms that each handle what the other cannot.

The paper resolves a contradiction that had been building in the literature: state space models (Mamba, Mamba-2, Dao and Gu, 2024), linear RNNs (HGRN, Qin et al., 2023b; 2024d), and other efficient architectures showed promising scaling behaviour and competitive language modelling loss, yet none had been adopted at commercial scale. Critics could (and did) point to this adoption gap as evidence that linear attention was fundamentally insufficient. The paper's scaling law analysis (Figure 6, Table 2) demonstrates that linear attention models do achieve lower loss at equal compute than softmax attention, but their retrieval failures on tasks like NIAH (Figure 7: 15.5% at 3B parameters vs. 84.2% for softmax) explain the adoption gap without invalidating the efficiency promise. The hybrid architecture resolves the contradiction by delivering both the lower loss and the retrieval capability, providing a unified explanation for the prior mixed evidence.

This work also implicitly shifts which research directions are most promising. Before MiniMax-01, significant effort was directed at making softmax attention more efficient through sparsity (Longformer, Big Bird), approximation (Linformer, Performer), or I/O-aware implementations (FlashAttention). These remain valuable for models operating at conventional context lengths (up to ~256K), but for the regime beyond 1 million tokens — which the paper's latency measurements show is where softmax attention overwhelms even optimised implementations — these approaches cannot match the asymptotic advantage of linear attention. The paper's finding that sliding window attention with periodic full-attention layers dramatically underperforms hybrid-lightning at equivalent throughput (Table 4: NIAH 53.9% vs. 95.7% at 1B parameters) suggests that sparse attention is not a competitive path to extreme-context models — the retrieval penalty from lost global attention is too severe.

Conversely, the paper makes verifier and difficulty estimation research less immediately relevant for the long-context bottleneck, though these remain critical for other aspects of LLM deployment. The MiniMax-01 approach addresses the long-context problem architecturally — by making it computationally feasible to process the full context — rather than through retrieval-augmented generation or selective context compression. This means the model does not need to decide which parts of the context are relevant; it can attend to everything equally. The value of selective attention mechanisms shifts from a necessity (because full attention is impossible) to an optimisation (because full attention, while possible, remains more expensive than it could be with truly unlimited context).

Follow-Up Research This Work Enables

Determining the minimum softmax-to-linear ratio required for retrieval capability at scale. The paper's 7:1 ratio is determined from small-scale experiments (Section 2.3), but whether this ratio remains optimal at the 456B scale — and more critically, whether it can be pushed further (15:1, 31:1) — is unknown. A systematic study at moderate scale (e.g., 7B–30B parameters) sweeping ratios from 3:1 to 31:1, evaluated on both retrieval (NIAH, MR-NIAH) and reasoning (RULER, LongBench-V2) at sequence lengths up to 512K, would establish whether retrieval capability degrades gradually or exhibits a phase transition at some critical ratio. The paper's theoretical analysis (Section 2.2.4) suggests the softmax layers' retrieval function depends on the "recomputation" mechanism, which in principle could work at any spacing — but information may need to propagate through lightning attention layers to reach a softmax checkpoint, and excessive spacing could create a bottleneck. Finding the point where retrieval collapses would define the fundamental limit of the hybrid approach and inform whether eliminating softmax attention entirely (the paper's stated future goal, Section 7) requires a qualitatively different linear attention design.

Evaluating whether the hybrid architecture's retrieval degrades with position relative to softmax layers. The paper's long-context evaluations report aggregate scores without breaking down retrieval accuracy by the distance between the target information and the nearest softmax attention layer. Does a token positioned immediately before a softmax layer (at layer 8, 16, 24, ...) get retrieved more accurately than a token positioned seven layers before the next softmax checkpoint? This question matters because it tests the mechanism the paper proposes: if softmax layers truly function as "retrieval checkpoints," then information must survive passage through the intervening lightning attention layers to reach them. An experiment tracking NIAH accuracy as a function of the needle's layer-distance from the nearest softmax layer — implemented by inserting probe tasks at different depths rather than at the input level, following the"logit lens" or probing methodology from mechanistic interpretability — would reveal whether the model learns to preserve critical information specifically for retrieval at softmax boundaries, or whether retrieval quality is uniform regardless of position. A finding that retrieval degrades steadily with distance from softmax layers would imply that the hybrid architecture's retrieval mechanism is more fragile than aggregate scores suggest, motivating either more frequent softmax layers or a mechanism for the model to explicitly flag information for retrieval.

Combining hybrid attention with dynamic compute allocation based on input difficulty. The current architecture processes all tokens uniformly regardless of whether the input demands precise retrieval (e.g., fact-checking against a specific paragraph) or can tolerate lossy processing (e.g., summarising gist). A natural extension would be a learned gating mechanism that decides, per layer and per token, whether to route the token through a lightning attention path (efficient, lossy) or a softmax attention path (expensive, precise), trained with a budget constraint that penalises excessive softmax usage. The paper's MoE routing infrastructure (Section 2.1) provides a template — the existing gating network selects experts; an analogous gating network could select attention mechanisms. The benchmark evaluation would compare against MiniMax-Text-01 on RULER and LongBench-V2 at 1M tokens, measuring both accuracy and the fraction of tokens routed to softmax attention. A strong result would show equivalent accuracy with, say, 50% fewer tokens processed by softmax attention, enabling longer effective contexts at the same compute budget. This would address the paper's own finding that softmax attention dominates inference latency at 1M tokens (Section 3.3.4: 95% of latency).

Stress-testing the architecture on truly out-of-distribution context lengths — specifically, reasoning at 2M–4M tokens. The paper's 4M-token claim rests entirely on vanilla NIAH (Figure 14), which the paper itself calls inadequate. A critical follow-up would evaluate MiniMax-Text-01 on RULER (all 13 tasks), LongBench-V2, and MR-NIAH at 2M and 4M tokens, ideally with a comparison against Gemini-1.5-Pro (the only commercial model supporting comparable contexts) at the same lengths. Even if the model cannot be trained at these lengths (the paper trained only to 1M), the length extrapolation enabled by half-dimension RoPE (Section 2.4) should be tested for reasoning tasks, not just retrieval. A finding that reasoning degrades sharply between 1M and 2M tokens — while retrieval remains intact — would provide critical guidance for deployment: the model would be suitable for information lookup at 4M tokens but not for analytical tasks, fundamentally changing how practitioners should use the long-context capability. A finding that both retrieval and reasoning degrade would motivate extending the staged long-context training (Table 6) to 2M and 4M tokens, which the paper does not currently plan.

Replicating the scaling law analysis at larger scale with learning rate tuning per architecture. The scaling law experiments (Section 2.2.2.2) use a fixed learning rate schedule across all architectures "due to constrained computational resources." This is a significant confound: if hybrid-lightning benefits more from learning rate decay than softmax attention (or vice versa), the reported loss exponents (Table 2) are biased. A follow-up study at moderate scale (e.g., 1B–7B parameters) that sweeps learning rate schedules for each architecture — using a budget of, say, 10× the original experiment's compute to enable proper hyperparameter optimisation — would determine whether the hybrid-lightning advantage persists under fair comparison. The refined scaling coefficients would be directly usable for planning larger models. More importantly, if learning rate tuning closes the gap between softmax and hybrid-lightning, it would imply that the hybrid architecture's advantage is partly an optimisation artefact rather than a fundamental capacity improvement, which would redirect research toward training dynamics rather than architecture design.

Developing verifier models robust to the distribution shift between lightning-attention and softmax-attention representations. The paper trains separate verifiers for base-model outputs and revision-model outputs (Appendix J of the pre-training/post-training sections, not shown, but the concept applies), noting that the PRM trained on base model outputs does not transfer well due to distribution shift. A natural extension would train a verifier specifically on outputs from different stages of the hybrid architecture — for example, after lightning attention layers vs. after softmax attention layers — and test whether verifier accuracy varies systematically. If the softmax layers produce representations that are easier for verifiers to score accurately (because they incorporate explicit retrieval), this would motivate placing verifiers specifically after softmax layers, or training verifiers on softmax-layer representations only. This could be tested by extracting hidden states at different depths and training linear probes to predict answer correctness, comparing probe accuracy at softmax vs. lightning attention layer outputs.

Practical Applications and Downstream Use Cases

Long-document professional analysis — legal, medical, and scientific. The model's RULER score of 0.910 at 1M tokens (Table 9) and LongBench-V2 overall of 56.5 with CoT (Table 10, the highest among all evaluated models) position it for tasks where professionals currently spend hours reading lengthy documents: legal discovery across thousands of pages of contracts, systematic review of medical literature spanning hundreds of papers, or synthesis of multi-volume scientific references. The key advantage over chunking-based approaches is that the model can perform cross-document reasoning — comparing claims made in different sections of a 500K-token document without the chunk boundaries artificially separating related information. The MTOB results (Table 11) demonstrate the model's ability to learn from reference materials provided in-context, enabling a workflow where a legal team provides a full case file and the model answers questions about it without fine-tuning. The practical constraint is the 95% latency dominance of softmax attention at 1M tokens (Section 3.3.4), which means analysis of million-token documents is feasible but not cheap — roughly 5× the latency of processing a 100K-token document in a pure softmax model.

Lifelong AI assistants with persistent memory. The MR-NIAH results (Figure 15) showing >0.9 adjusted recall at 1M tokens in both English and Chinese — equivalent to ~2,000 prior interactions — demonstrate that the model can function as a "lifelong companion AI" that remembers specific details from conversations months earlier. This is qualitatively different from current AI assistants, which typically operate with sliding windows of recent context and lose access to older interactions. A practical deployment could store the full conversation history as context, with the model distinguishing between similar historical queries (as demonstrated in Appendix B.2, where it correctly retrieves a specific penguin poem rather than a later variation). The in-house evaluation's Long Context score of 93.8 (Table 12), dramatically exceeding GPT-4o (86.2) and 46.7 points above Claude-3.5-Sonnet (47.1), suggests this advantage translates to real user satisfaction in scenarios like document translation, summarisation, and analysis over extended interactions.

Many-shot in-context learning for low-resource tasks. The MTOB benchmark result (Table 11: Δ full book of 45.6 ChrF for eng→kalam, the largest improvement among all models) demonstrates that the model can acquire entirely new capabilities from context alone — in this case, translating a language it had essentially never seen during training — when provided with sufficient reference material. This enables a deployment pattern where organisations with specialised, low-resource tasks (rare language translation, proprietary data format parsing, domain-specific code generation) can provide extended examples and documentation in-context rather than investing in fine-tuning. The 4M-token inference capability (Figure 14) is relevant here: it allows the context to include not just a grammar book and 375 parallel examples (the MTOB setting at ~133K tokens), but potentially multiple reference works, extensive example corpora, and documentation — pushing the boundary of what can be learned purely through in-context learning without parameter updates.

Cost-efficient batch inference on long documents with the single-node deployment target. The paper's explicit design constraint — fitting on a single 8×80GB node at 1M tokens under 8-bit quantisation (Section 2.4) — was chosen for practical deployment affordability. Organisations processing large volumes of long documents (government archives, legal discovery, scientific literature databases) can deploy MiniMax-Text-01 on standard cloud GPU instances without requiring multi-node inference infrastructure, which introduces cross-machine communication latency and cost. The 75% MFU on H20 GPUs (Section 3.3.4) and the near-linear latency scaling in Figure 2 mean that processing a batch of 100K-token documents costs roughly proportionally more than 10K-token documents, rather than the quadratic increase that softmax-only models incur. This makes batch long-document analysis economically feasible at scales that would be prohibitive with current commercial APIs, which either charge per-token (with costs scaling linearly but base rates reflecting their own infrastructure costs) or have hard context limits below 256K.

When to Prefer This Method

The paper positions MiniMax-01's hybrid architecture against two alternatives: pure softmax attention (standard in most commercial and open-source LLMs) and pure linear attention (proposed theoretically but not deployed at scale). The tradeoffs are specific and empirically grounded, not a generic decision matrix.

Prefer the hybrid architecture when:

• The primary deployment constraint is context length beyond ~256K tokens, where pure softmax attention either cannot operate (memory limits) or incurs prohibitive latency (quadratic scaling, Figure 2).
• Single-node deployment is required for cost or latency reasons, since MiniMax-01 was specifically designed to fit million-token contexts on 8 GPUs with 8-bit quantisation (Section 2.4).
• Retrieval from within long contexts is a core capability, since pure linear attention fails catastrophically at this (Figure 7: NIAH 15.5% at 3B parameters) while the hybrid architecture matches softmax attention (98.0% at 3B).
• The workload includes in-context learning from large reference corpora, as the MTOB results (Table 11) show the model learns more from extended context than competitors.
• Training budget is fixed and the goal is to minimise loss — the scaling laws (Table 2) predict hybrid-lightning achieves lower loss than softmax attention at matched FLOPs.

Prefer pure softmax attention (or conventional dense models) when:

• Context lengths never exceed ~128K tokens and are unlikely to in the deployment lifetime, since FlashAttention-2 achieves higher raw throughput at short-to-medium lengths (Figure 8: softmax ~17K TGS at 1024 tokens vs. hybrid-lightning ~12K TGS).
• Coding and advanced mathematics are the primary capabilities sought — MiniMax-Text-01 underperforms DeepSeek-V3 on MATH by 7.2 points (84.6 vs. 77.4) and trails Claude-3.5-Sonnet on HumanEval by 6.8 points (93.7 vs. 86.9, Table 8). The architecture's advantage is in long contexts, not in pushing the frontier of reasoning in conventional-length tasks.
• Hardware beyond NVIDIA Hopper GPUs is required — the CUDA optimisations in Section 3.3 are Hopper-specific (WGMMA instructions, TMA), and performance on other hardware platforms is unvalidated.
• Instruction following with complex multi-level constraints is the primary use case — the in-house Instruction Following score of 46.3 (Table 12) substantially trails Claude-3.5-Sonnet (61.5) and GPT-4o (50.4), and the paper attributes this to insufficient training data (Section 5.8.1).

Prefer pure linear attention (not hybrid) only if:

• A model's only function is language modelling without retrieval — the scaling law analysis (Table 2) shows pure lightning attention achieves intermediate loss between softmax and hybrid-lightning, and Figure 8 shows it trains at constant speed regardless of sequence length (unlike hybrid-lightning, which has a small softmax overhead). However, the paper provides no evidence that pure lightning attention is useful for any downstream task requiring retrieval, making this a narrow use case for perplexity evaluation or non-retrieval generation.