Mamba-3: Improved Sequence Modeling using State Space Principles

🎯 Pitch

Mamba-3 matches the perplexity of Mamba-2 while using only half the state size, shrinking inference latency with no quality loss. It also solves the synthetic parity task that real-valued linear models fundamentally cannot, introducing complex-valued states for state tracking. A new multi-input/output formulation packs up to 4× more FLOPs into decode without increasing state memory or wall-clock time, pushing the performance–efficiency frontier.

1. Executive Summary

This paper introduces Mamba-3, a state space model that advances the performance-efficiency Pareto frontier for sub-quadratic sequence models through three core methodological improvements inspired by SSM principles: exponential-trapezoidal discretization (a second-order accurate recurrence that induces an implicit convolution on the state-input, replacing the explicit short causal convolution), complex-valued state transitions (a data-dependent rotary embedding that enables state-tracking capabilities absent in prior models like Mamba-2), and a multi-input, multi-output (MIMO) formulation (a rank-expanded state update that increases decode FLOPs up to 4× without increasing state size or wall-clock latency). At the 1.5B scale on 100B FineWeb-Edu tokens, Mamba-3 SISO improves average downstream accuracy by 0.6 points over Gated DeltaNet, while the MIMO variant adds another 1.2 points for a total 1.8-point gain, and across state-size experiments Mamba-3 MIMO with state size 64 matches the perplexity of Mamba-2 with state size 128 — effectively halving latency for the same quality. The complex-valued recurrence enables Mamba-3 to solve synthetic state-tracking tasks like parity that real-valued SSMs fundamentally cannot, establishing that the capabilities of linear-time models can be expanded through theoretically grounded architectural choices rather than sacrificed for efficiency.

2. Context and Motivation

The Core Problem: Sub-Quadratic Models Sacrifice Capability for Efficiency

The fundamental tension this paper addresses is deceptively simple: how do you build a sequence model that is both fast at inference and powerful enough to handle complex reasoning tasks? The standard Transformer architecture (Vaswani et al., 2017) provides excellent model quality, but its inference characteristics make it increasingly impractical as deployment scales grow:

Quadratic compute in sequence length: Self-attention requires computing pairwise interactions between all tokens, costing $O(T^2)$ operations for a sequence of length $T$ . This becomes prohibitive in long-context settings or high-volume serving.
Linear memory growth: The KV cache stores key and value vectors for every token in the context, consuming memory proportional to sequence length. For long conversations or documents, this balloons deployment costs.

These costs matter enormously now because the field has entered what the authors call an "inference-first paradigm." Section 1 explicitly grounds this motivation in contemporary trends:

"Test-time compute has emerged as a key driver of progress in LLMs, with techniques like chain-of-thought reasoning and iterative refinement demonstrating that inference-time scaling can unlock new capabilities... The rapid rise of parallel, agentic workflows has only intensified the need for efficient inference and deployment of such models."

In other words, the practical impact of AI systems increasingly depends not just on what they can learn during training, but on how efficiently they can reason and deploy at scale. Architecture choice directly dictates this efficiency, since it determines the computational and memory requirements of every forward pass.

The Rise and Limitations of Linear-Time Models

This bottleneck has motivated an explosion of work on sub-quadratic models — architectures that replace softmax attention with alternatives requiring only $O(T)$ time and $O(1)$ memory per step. The paper categorizes these into three broad frameworks (Section 5.1):

Linear attention models: Approximate softmax attention through kernel feature maps (Katharopoulos et al., 2020; Choromanski et al., 2022) or raw dot-products modulated by decays (Sun et al., 2023; Yang et al., 2024). More recent variants incorporate delta-rule updates where the memory is modified based on prediction errors (Schlag et al., 2021; Yang et al., 2025a, 2025b).
Test-time training/regression models: View sequence modeling as an online learning problem where the recurrent state represents a compressed summary optimized to minimize a regression objective on past inputs (Sun et al., 2025; Tandon et al., 2025; Wang et al., 2025).
State space models (SSMs): Draw on classical signal processing and dynamical systems theory, modeling sequences through continuous-time ODEs that are discretized into recurrent or convolutional forms. Early versions like S4 (Gu et al., 2022a) used linear time-invariant (LTI) dynamics with structured transition matrices. Mamba-1 (Gu & Dao, 2024) introduced input-dependent selectivity — making the SSM parameters data-dependent — which dramatically improved performance on information-dense modalities like language. Mamba-2 (Dao & Gu, 2024) further refined this through the Structured State Space Duality (SSD) framework, establishing equivalences between SSMs and linear attention that enable hardware-efficient matrix multiplication implementations.

These models have achieved notable real-world adoption. The paper highlights that "models such as Mamba-2 and Gated DeltaNet have recently been incorporated into large-scale hybrid models that match the performance of pure Transformer alternatives with much higher efficiency," citing production systems from Kimi, NVIDIA, Tencent Hunyuan, and Qwen (Section 1).

Three Concrete Gaps: Quality, Capability, and Hardware Efficiency

Despite this progress, the paper identifies three specific deficiencies that prevent linear-time models from fully displacing Transformers. These form the paper's main motivation:

Gap 1: Modeling Quality — The Expressivity Trade-off in Mamba-2

Mamba-2 was designed with a specific priority: improving training speed and simplicity over Mamba-1. The authors are explicit about the cost:

"Mamba-2 was developed to improve training speed and simplicity over Mamba-1, by sacrificing some expressivity and thus performing worse for inference-matched models."

What does this "sacrifice of expressivity" mean concretely? Mamba-2 progressively simplified the state transition matrix: S4 used complex-valued Normal Plus Low Rank (NPLR) matrices; Mamba-1 reduced this to a diagonal of real numbers; Mamba-2 went further to a single scalar times identity — a scalar decay $A_t I$ where $A_t \in \mathbb{R}_{<0}$ (Section 2.2, equation 1). This extreme simplification enables the hardware-efficient "chunked" algorithm (processing sequences in blocks that fit in tensor core memory) but strips away representational capacity. The result: a model that trains faster but leaves model quality on the table at fixed inference cost.

This is not a hypothetical concern. The paper's own experiments (Section 4.3, Figure 3) show that across state sizes — which directly control decode latency — Mamba-2 sits on a distinctly worse Pareto frontier than what Mamba-3 achieves. For a given inference speed, Mamba-2 produces worse perplexity; for a given perplexity target, Mamba-2 requires a larger (slower) state.

Gap 2: State-Tracking Capabilities — A Fundamental Expressivity Failure

The most theoretically significant gap is that real-valued SSMs with non-negative eigenvalues fundamentally cannot represent certain simple computations. Section 3.2 frames this precisely:

"Recent works have shown that the restriction to real, non-negative eigenvalue transitions degrades the capabilities of the model on simple state-tracking tasks — here referring primarily to the solvable-group regime (TC $^0$ ) such as parity — which can be solved by a one-layer LSTM."

The paper cites Grazzi et al. (2025), Merrill et al. (2025), and Sarrof et al. (2024) as establishing this limitation. The mathematical essence: tasks like parity (is the cumulative sum of binary inputs odd or even?) require the hidden state to alternate between two configurations — a rotation. Real-valued scalar transitions can only shrink or preserve state values (since $|\alpha_t| < 1$ for stability), they cannot rotate them. A one-layer LSTM can solve parity because its gating structure effectively implements conditional sign flips. Mamba-2 — with its scalar-times-identity transition — cannot, regardless of how many parameters or how much training data you throw at it.

The paper provides a concrete example (Section 3.2):

"Consider the parity function on binary inputs $\{0, 1\}$ , defined as $\sum_t x_t \bmod 2$ . This task can be performed using update: $\mathbf{h}_t = R(\pi x_t) \mathbf{h}_{t-1}$ , where $R(\cdot)$ is a 2-D rotation matrix. Such rotational dynamics cannot be expressed with real eigenvalues."

This is not merely a theoretical curiosity. It means these models have a hard expressivity ceiling on certain classes of formal reasoning — the kind of structured, multi-step computation that increasingly matters as we push LLMs toward agentic behavior and complex reasoning chains. Mamba-2 and similar real-valued SSMs may perform well on average language modeling metrics while being fundamentally incapable of certain kinds of precise computation.

Gap 3: Hardware-Inefficient Decoding — The Memory-Bound Bottleneck

The third and perhaps most practically consequential gap is that the theoretically linear inference of sub-quadratic models is not hardware-efficient in practice. Section 1 states this bluntly:

"Despite these sub-quadratic models being prized for theoretically efficient inference and thus their widespread adoption, their inference algorithms are not hardware efficient. In particular, because these algorithms were developed from a training perspective, their decoding phase has low arithmetic intensity (the ratio of FLOPs to memory traffic), resulting in large portions of hardware remaining idle."

This is a hardware-aware critique that deserves careful unpacking. Modern GPUs are designed around the reality that computation is cheap and memory movement is expensive. An operation's arithmetic intensity — how many floating-point operations it performs per byte of data transferred — determines whether it saturates the GPU's compute units or whether those units sit idle waiting for data.

Table 2a quantifies the problem: Mamba-2's SISO decode has an arithmetic intensity of approximately 2.5 operations per byte. For comparison, a bfloat16 matrix multiply — the workhorse of Transformer attention — achieves roughly 295 ops/byte on an NVIDIA H100-SXM5. The SSM decode is off by two orders of magnitude, meaning the GPU's tensor cores (the specialized matrix-multiply units) are severely underutilized during autoregressive generation. The computation is so lightweight relative to the data movement that the GPU spends most of its time waiting for memory rather than computing.

Why does this happen? In the SISO recurrence (Section 3.3), the dominant computation at each decode step is an outer product $\mathbf{B}_t \mathbf{x}_t^\top$ where $\mathbf{B}_t \in \mathbb{R}^N$ is the input projection and $\mathbf{x}_t \in \mathbb{R}^P$ is the input vector. This outer product requires $O(NP)$ FLOPs but moves $O(NP)$ bytes — the ratio is constant and small. Meanwhile, the state matrix $\mathbf{H}_t \in \mathbb{R}^{N \times P}$ (which can be thousands of elements) must be read and written at every step, dominating memory traffic.

The crucial insight is that this problem arises because the original SSM algorithms were designed from a training perspective, where the parallel (chunked/SSD) form amortizes these costs by processing many tokens simultaneously. During autoregressive generation, however, you process one token at a time serially, and the design assumptions break down.

How This Paper Positions Itself

Mamba-3 positions itself as addressing all three gaps through a unified perspective: the state space model viewpoint, applied with an inference-first design philosophy. This framing is made explicit in the introduction:

"To develop more performant models from an inference-first paradigm, we introduce three core methodological changes on top of Mamba-2, influenced by an SSM-centric viewpoint of sub-quadratic models."

The emphasis on "SSM-centric" is important because — as Section 5.4 argues in detail — each improvement arises naturally from the SSM framework but would be non-obvious or even meaningless from other perspectives (linear attention, test-time regression). For example:

Discretization theory is native to SSMs (which model continuous ODEs) but absent from associative memory or test-time regression formulations. The exponential-trapezoidal rule (Section 3.1) falls directly out of interpreting the recurrence as a discretized dynamical system.
Complex-valued state transitions have a long history in dynamical systems and signal processing — earlier SSMs like S4 used them extensively — but complex coefficients are "meaningless as the coefficient of a regression objective" in the test-time training framework (Section 5.4).
MIMO formulations are a classical concept from control theory (multi-input, multi-output systems) but do not naturally appear in associative memory frameworks.

This is more than taxonomic posturing. The paper argues that the choice of theoretical framework shapes which design extensions are discoverable. By anchoring in the SSM viewpoint, Mamba-3 finds improvements that a pure linear-attention or TTT perspective would not easily suggest.

The paper also explicitly connects its complex-valued recurrence to the prior Mamba lineage while distinguishing its contribution. Earlier models like S4 used complex SSMs but were LTI (not selective) and targeted vision/audio domains where frequency content is explicit. RetNet (Sun et al., 2023) used a constant complex phase (not data-dependent) implemented via RoPE, but as an LTI model with constant scalar decays. Mamba-3 is:

"the first modern recurrent model with complex-valued state transitions, which were introduced for specific purposes of increasing expressivity and state-tracking ability... the first usage of data-dependent RoPE grounded in theoretical motivations." (Section 5.2)

The MIMO contribution is similarly positioned as breaking from prior work. Previous MIMO SSMs like S5 (Smith et al., 2023) and LRU (Orvieto et al., 2023) used MIMO to simplify the model — replacing many independent SISO SSMs with one MIMO system to reduce state size and ease computation. Mamba-3 uses MIMO in the opposite direction: to increase expressivity by adding rank to the state update, motivated by the hardware observation that the decode step is memory-bound and can absorb extra computation for free.

The paper does not claim to solve all problems. It explicitly acknowledges that these models still struggle on hard retrieval tasks (Section 4.1.2, Table 4), performing poorly on semi-structured and unstructured information extraction compared to Transformers. The hybrid model experiments (interleaving Mamba-3 with self-attention) are presented as a pragmatic acknowledgment that linear layers will likely be used alongside attention in practice, not as pure replacements.

Summary: Three Gaps, Three Solutions, One Framework

The paper's contribution can be understood as mapping each architectural gap to a theoretically motivated solution:

Gap	Root Cause	Mamba-3 Solution	Mechanism
Quality	Mamba-2 sacrificed expressivity for training speed	Exponential-trapezoidal discretization + BC biases	3-term recurrence increases representational capacity; removes need for external convolution
Capability	Real-valued scalar transitions cannot represent rotational dynamics	Complex-valued SSM with data-dependent RoPE	Block-diagonal rotation matrices in transition enable state tracking (parity, modular arithmetic)
Efficiency	SISO decode is memory-bound with low arithmetic intensity	MIMO formulation	Rank- $R$ expansion of state update increases FLOPs without increasing state size or latency

The paper's core thesis — stated implicitly through its methodology — is that these improvements are only discoverable through the SSM lens, and that the SSM viewpoint provides a rich source of theoretically principled extensions that other frameworks (linear attention, TTT) cannot easily access. Whether this thesis holds up depends on whether future work from those other frameworks independently discovers equivalent mechanisms — a question the paper leaves open.

3. Technical Approach

3.1 Reader Orientation

Mamba-3 is a recurrent neural network layer designed to replace self-attention in language models, providing both stronger modeling capabilities than prior sub-quadratic architectures and hardware-efficient inference. The paper does not propose a single new algorithm but rather introduces three complementary methodological improvements — a more expressive recurrence, complex-valued state dynamics, and a multi-input/multi-output formulation — each motivated by viewing the SSM not as a training algorithm but as a dynamical system whose discretization, state space, and input-output structure can be systematically upgraded.

3.2 Big-Picture Architecture (Diagram in Words)

The Mamba-3 layer fits into a standard Llama-style transformer backbone, alternating Mamba-3 blocks with SwiGLU MLP blocks under pre-normalization. Within each Mamba-3 block, there are four major components:

Input Projections: The token embedding is projected into several branches — the SSM parameters ( $\mathbf{B}$ , $\mathbf{C}$ , $\mathbf{X}$ , $\Delta$ , and the complex phase $\boldsymbol{\theta}$ ), plus a gating branch $\mathbf{Z}$ that controls how much of the SSM output passes through. These projections are linear layers with learned weights.
The Core SSM Recurrence: This is where the sequence mixing happens. It takes the projected parameters and processes the input sequence through a linear recurrence that maintains a fixed-size hidden state $\mathbf{H}_t \in \mathbb{R}^{N \times P}$ (state size $N$ by head dimension $P$ ). The recurrence has been upgraded from Mamba-2's 2-term form ( $\mathbf{h}_t = \alpha_t \mathbf{h}_{t-1} + \gamma_t \mathbf{B}_t x_t$ ) to a 3-term form that also incorporates the previous token's state-input, and the state transition now includes complex-valued rotations for state tracking.
BC Normalization and Biases: After projecting $\mathbf{B}$ and $\mathbf{C}$ , RMS normalization is applied (stabilizing training, analogous to QK-Norm in Transformers), and learned per-channel biases are added (providing a data-independent convolution-like component that partially replaces the explicit short causal convolution used in Mamba-2).
Output Gating: The SSM output passes through a SiLU-gated linear layer controlled by $\mathbf{Z}$ , producing the final block output that feeds into the residual stream.

Information flows: token embedding → input projections (producing $\mathbf{B}, \mathbf{C}, \mathbf{X}, \Delta, \boldsymbol{\theta}, \mathbf{Z}$ ) → complex-valued exponential-trapezoidal SSM recurrence (mixing information across sequence positions via the hidden state $\mathbf{H}_t$ ) → gated output projection → residual addition → next block. For the MIMO variant, $\mathbf{B}$ , $\mathbf{C}$ , and $\mathbf{X}$ have an additional rank dimension $R$ that increases computation within the recurrence without expanding the state size.

3.3 Roadmap for the Deep Dive

First, the exponential-trapezoidal discretization (Section 3.1 of the paper), because it establishes the mathematical foundation for how continuous-time SSMs become discrete recurrences — the core mechanic of the entire layer. Understanding the discretization framework reveals why Mamba-1/2's heuristic was mathematically justified and how Mamba-3's 3-term recurrence emerges naturally as a higher-order approximation.
Second, the complex-valued SSM (Section 3.2 of the paper), which extends the real-valued recurrence to support rotational state dynamics. This requires understanding the discretization from the prior section, since the complex extension is applied to the discretized system and the RoPE equivalence proof depends on the discretization form.
Third, the MIMO formulation (Section 3.3 of the paper), which modifies the input-output structure of the recurrence to improve hardware utilization. This is conceptually independent of the first two components but interacts with them in the final architecture — the chunked training algorithm for MIMO builds directly on the SSD framework that the parallel form of the recurrence enables.
Fourth, the architectural integration (Section 3.4 of the paper), showing how these components combine into the complete Mamba-3 block and what supplementary architectural choices (BC norm, biases, removal of the short convolution) were made and why.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methods paper whose core idea is that the state space model viewpoint — interpreting a recurrent layer as the discretization of a continuous-time dynamical system — provides a principled framework for extending sub-quadratic architectures along three axes (discretization accuracy, state expressivity, and input-output structure), each yielding concrete empirical gains while maintaining or improving hardware efficiency.

Exponential-Trapezoidal Discretization

This section explains how to convert a continuous-time state space model into a discrete recurrence that can process token sequences. The paper's contribution is twofold: (1) it provides a derivation framework that formalizes and justifies the heuristic discretization used in Mamba-1 and Mamba-2, and (2) it uses this framework to propose a more accurate discretization (exponential-trapezoidal) that yields a 3-term recurrence with greater expressivity.

The Continuous-Time SSM and Why Discretization Matters

State space models originate from dynamical systems theory and are naturally defined in continuous time:

$\dot{\mathbf{h}}(t) = \mathbf{A}(t) \mathbf{h}(t) + \mathbf{B}(t) x(t)$ $y(t) = \mathbf{C}(t)^\top \mathbf{h}(t)$

where $\mathbf{h}(t) \in \mathbb{R}^N$ is the hidden state vector at time $t$ , $x(t) \in \mathbb{R}$ is the scalar input at time $t$ , $\mathbf{A}(t) \in \mathbb{R}^{N \times N}$ is the state transition matrix (governing how the state evolves on its own), $\mathbf{B}(t) \in \mathbb{R}^N$ is the input projection vector (governing how the input affects the state), and $\mathbf{C}(t) \in \mathbb{R}^N$ is the output projection vector (governing how the state maps to the output). The dot notation $\dot{\mathbf{h}}(t)$ denotes the time derivative of the state.

What this ODE represents: the system describes how a vector-valued quantity $\mathbf{h}$ changes over time. At any instant, $\mathbf{h}$ changes in two ways: proportionally to its current value (through $\mathbf{A}(t)$ ) and in response to external input (through $\mathbf{B}(t)x(t)$ ). The output $y(t)$ is a linear readout of the current state. This is the standard form of a linear time-varying dynamical system.

Why discretization is necessary: real data — text tokens — arrives at discrete positions (word 1, word 2, ...), not as a continuous signal. To apply an SSM to language, we must convert the continuous-time ODE into a discrete recurrence that updates the state once per token. The quality of this conversion matters: a poor discretization introduces approximation errors that accumulate over long sequences, while a better discretization more faithfully captures the intended dynamics.

The Variation of Constants Approach (Proposition 5)

The paper's discretization framework begins from a classical result in ODE theory — the variation of constants formula (Proposition 5 in the paper, attributed to Tenenbaum and Pollard, 1985). The derivation applies an integrating factor to the ODE and integrates over a discrete time interval:

$\mathbf{h}(\tau_t) = \exp\left(\int_{\tau_{t-1}}^{\tau_t} \mathbf{A}(s) ds\right) \mathbf{h}(\tau_{t-1}) + \int_{\tau_{t-1}}^{\tau_t} \exp\left(\int_{\tau}^{\tau_t} \mathbf{A}(s) ds\right) \mathbf{B}(\tau) x(\tau) d\tau$

where $\tau_{t-1}$ and $\tau_t$ are the start and end times of the $t$ -th interval, with step size $\Delta_t = \tau_t - \tau_{t-1}$ . The first term propagates the previous state forward through the state transition dynamics, and the second term (an integral) accounts for the cumulative effect of the input over the interval.

Why this form matters: the variation of constants formula is exact for the continuous system — no approximation has been introduced yet. It separates the problem into two parts: the state transition (which has a clean exponential form) and the state-input integral (which needs numerical approximation). The separation is crucial because these two components have different mathematical character — the transition is dominated by the exponential of $\mathbf{A}$ , while the input integral involves the product of the transition and the input projection.

The key approximation step: The paper approximates the state transition integral by assuming $\mathbf{A}(s)$ is constant over the interval and equal to its value at the right endpoint:

$\mathbf{h}_t \approx \exp(\Delta_t \mathbf{A}_t) \mathbf{h}_{t-1} + \int_{\tau_{t-1}}^{\tau_t} \exp((\tau_t - \tau) \mathbf{A}_t) \mathbf{B}(\tau) x(\tau) d\tau$

where $\mathbf{A}_t \equiv \mathbf{A}(\tau_t)$ . This incurs a local truncation error of order $O(\Delta_t^2)$ — meaning the error scales quadratically with the step size, which is acceptable for small steps. The remaining integral (the state-input integral) still needs to be approximated, and different choices of approximation produce different discretization methods.

The SSM-specific simplification: In Mamba-2's parameterization, $\mathbf{A}_t$ is a scalar times identity: $\mathbf{A}_t = A_t \mathbf{I}_{N \times N}$ where $A_t \in \mathbb{R}_{<0}$ (a negative real number). This means $\exp(\Delta_t \mathbf{A}_t) = e^{\Delta_t A_t} \mathbf{I}$ , which is just a scalar decay factor times the identity matrix. Define:

$\alpha_t \equiv e^{\Delta_t A_t} \in (0, 1)$

The scalar $\alpha_t$ is the discrete-time forgetting factor — it controls how much of the previous state is retained. When $\alpha_t$ is close to 1 (small $\Delta_t$ or $A_t$ near zero), the model retains information for many steps; when $\alpha_t$ is close to 0 (large $\Delta_t$ or very negative $A_t$ ), the model forgets quickly.

The Exponential-Euler Discretization (Mamba-1/-2 Formalized)

Mamba-1 claimed to use zero-order hold (ZOH) discretization, but in practice both Mamba-1 and Mamba-2 implemented something different. The paper's framework reveals what was actually computed and provides the theoretical justification:

Zero-Order Hold (ZOH) applied to LTV systems: The classical ZOH formula for linear time-invariant systems treats $\mathbf{B}(\tau)$ and $x(\tau)$ as constant over the interval, with values fixed at the right endpoint. The paper notes that this formula can be naively extended to time-varying systems by replacing the constant parameters with their time-varying versions:

$\int_{\tau_{t-1}}^{\tau_t} e^{(\tau_t - \tau) A_t} B_t x_t d\tau = A_t^{-1}(e^{\Delta_t A_t} - I) B_t x_t$

This requires computing a matrix inverse or solving a linear system involving $A_t$ , which is computationally inconvenient.

Exponential-Euler approximation: The implementation in Mamba-1 and Mamba-2 instead used a simpler approximation. Rather than the full ZOH integral, it approximates the state-input integral using Euler's rule — the simplest possible numerical integration scheme — while holding the integrand constant at the right endpoint:

$\mathbf{h}_t \approx e^{\Delta_t A_t} \mathbf{h}_{t-1} + (\tau_t - \tau_{t-1}) e^{(\tau_t - \tau_t) A_t} \mathbf{B}_t x_t = e^{\Delta_t A_t} \mathbf{h}_{t-1} + \Delta_t \mathbf{B}_t x_t$

Since $e^{0 \cdot A_t} = 1$ , the exponential term in the input contribution vanishes, leaving just $\Delta_t \mathbf{B}_t x_t$ .

What this computes: The state at time $t$ is the previous state $\mathbf{h}_{t-1}$ scaled down by the decay factor $\alpha_t = e^{\Delta_t A_t}$ (forgetting old information), plus the current input $x_t$ scaled by the step size $\Delta_t$ and projected by $\mathbf{B}_t$ (adding new information). This is the 2-term recurrence shown in equation (1) of the paper:

$\mathbf{h}_t = \alpha_t \mathbf{h}_{t-1} + \gamma_t \mathbf{B}_t x_t$

where $\gamma_t \equiv \Delta_t$ .

Why this form: Euler's rule is a first-order method — it approximates the integral using only the value at one endpoint. Its local truncation error scales as $O(\Delta_t^2)$ . The paper names this "exponential-Euler" because it combines the exact exponential treatment of the state transition (from the integrating factor) with Euler approximation of the state-input integral. This naming clarifies that the heuristic was not arbitrary — it corresponds to a specific, mathematically well-defined numerical scheme.

The Exponential-Trapezoidal Discretization (Mamba-3's Innovation)

The paper's key insight is that the state-input integral can be approximated to higher accuracy by using both endpoints of the interval, not just one. This leads to a 3-term recurrence with greater expressivity.

The generalized trapezoidal approximation: Instead of evaluating the integrand only at the right endpoint (Euler), evaluate it as a convex combination of both endpoints:

$\int_{\tau_{t-1}}^{\tau_t} e^{(\tau_t - \tau) A_t} \mathbf{B}(\tau) x(\tau) d\tau \approx (1 - \lambda_t) \Delta_t e^{\Delta_t A_t} \mathbf{B}_{t-1} x_{t-1} + \lambda_t \Delta_t \mathbf{B}_t x_t$

where $\lambda_t \in [0, 1]$ is a data-dependent scalar (computed by a learned projection of the current token, then passed through a sigmoid). The first term weights the previous input's contribution (propagated through the decay), and the second term weights the current input's contribution.

The resulting 3-term recurrence (Proposition 1):

$\mathbf{h}_t = e^{\Delta_t A_t} \mathbf{h}_{t-1} + (1 - \lambda_t) \Delta_t e^{\Delta_t A_t} \mathbf{B}_{t-1} x_{t-1} + \lambda_t \Delta_t \mathbf{B}_t x_t$

which the paper compactly rewrites as:

$\mathbf{h}_t = \alpha_t \mathbf{h}_{t-1} + \beta_t \mathbf{B}_{t-1} x_{t-1} + \gamma_t \mathbf{B}_t x_t$

where $\alpha_t \equiv e^{\Delta_t A_t}$ (the same decay as before), $\beta_t \equiv (1 - \lambda_t) \Delta_t e^{\Delta_t A_t}$ (the weight on the previous input, which includes decay), and $\gamma_t \equiv \lambda_t \Delta_t$ (the weight on the current input).

What this computes: The state now has three contributions: (1) the decayed previous state, (2) the previous token's state-input $\mathbf{B}_{t-1} x_{t-1}$ passed through the decay and weighted by $1 - \lambda_t$ , and (3) the current token's state-input $\mathbf{B}_t x_t$ weighted by $\lambda_t$ . This means information from token $t-1$ can influence the state at time $t$ through two separate paths: indirectly through $\mathbf{h}_{t-1}$ (which already incorporated $x_{t-1}$ ) and directly through the explicit $\beta_t$ term.

Why this form — the error analysis: The classical trapezoidal rule (with $\lambda_t = 1/2$ ) provides second-order accuracy, with local truncation error scaling as $O(\Delta_t^3)$ rather than $O(\Delta_t^2)$ for Euler. This is proven in Appendix A.2: under the assumption that $\lambda_t = 1/2 + c_t \Delta_t$ (i.e., the data-dependent $\lambda_t$ is close to 1/2), the leading error term of order $\Delta_t^2$ cancels, leaving $O(\Delta_t^3)$ error. However, Remark 3 notes that their ablations show not constraining $\lambda_t$ to be close to 1/2 empirically performs better — the model learns the optimal weighting from data, and the strict 1/2 that classical theory prescribes is suboptimal for language modeling.

Remark 2 — relationship to prior work: The exponential-trapezoidal rule generalizes both (a) the classical trapezoid rule (recovered when $\lambda_t = 1/2$ ) and (b) Mamba-2's exponential-Euler rule (recovered when $\lambda_t = 1$ , which eliminates the $\beta_t$ term entirely). When $\lambda_t = 1$ , the model cannot explicitly attend to the previous input beyond what the state already captures.

Parallel representation under SSD (Section 3.1.3): The 3-term recurrence can be expressed in the Structured State Space Duality (SSD) framework as a matrix multiplication with a structured mask. Unrolling the recurrence from $\mathbf{h}_0 = \gamma_0 \mathbf{B}_0 x_0$ through time $T$ and computing the output $\mathbf{y}_t = \mathbf{C}_t^\top \mathbf{h}_t$ , the full sequence output can be written as:

$\mathbf{Y} = (\mathbf{L} \odot \mathbf{C} \mathbf{B}^\top) \mathbf{X}$

where $\mathbf{L} \in \mathbb{R}^{T \times T}$ is now a product of two structured matrices:

$\mathbf{L} = \begin{bmatrix} 1 & & & \\ \alpha_1 & 1 & & \\ \alpha_2 \alpha_1 & \alpha_2 & 1 & \\ \vdots & & \ddots & \end{bmatrix} \begin{bmatrix} \gamma_0 & & & \\ \beta_1 & \gamma_1 & & \\ 0 & \beta_2 & \gamma_2 & \\ \vdots & & \ddots & \end{bmatrix}$

What this means structurally: The first matrix is the standard 1-semiseparable matrix from Mamba-2 (equation 3 in the paper) — it captures the cumulative decay over time. The second matrix is a 2-band matrix (non-zero only on the diagonal and the first sub-diagonal) that encodes the trapezoidal weighting. Multiplying them together produces a matrix whose $(t, s)$ entry is the contribution of token $s$ to the state at time $t$ . The paper notes this is "a special case of a 2-semiseparable matrix."

Why the parallel form matters: For training, this matrix multiplication formulation enables the use of GPU tensor cores, which are optimized for exactly this kind of dense matrix-matrix multiply. The SSM can be computed in chunks — processing a block of tokens simultaneously using the parallel form — while using the recurrent form only across chunks to handle long sequences.

The Implicit Convolution Interpretation

A crucial insight that connects the discretization to architectural design choices: the exponential-trapezoidal recurrence is equivalent to applying a data-dependent convolution of size 2 on the state-input $\mathbf{v}_t = \mathbf{B}_t x_t$ before passing it into the standard recurrence. In a normal SSM, you compute $\mathbf{v}_t$ , then do $\mathbf{h}_t = \alpha_t \mathbf{h}_{t-1} + \gamma_t \mathbf{v}_t$ . In Mamba-3, you first convolve $\mathbf{v}$ with weights $(\beta_t, \gamma_t)$ to get $\tilde{\mathbf{v}}_t = \beta_t \mathbf{v}_{t-1} + \gamma_t \mathbf{v}_t$ , then do $\mathbf{h}_t = \alpha_t \mathbf{h}_{t-1} + \tilde{\mathbf{v}}_t$ .

Remark 4 explains why this is different from standard convolutions: Most recurrent models (Mamba-1, Mamba-2, Gated DeltaNet) include an explicit short causal convolution applied to the input $x_t$ outside the core recurrence — a separate operation with its own activations and parameters. The exponential-trapezoidal convolution operates on the state-input within the recurrence, with weights that are data-dependent (derived from $\lambda_t$ and $\Delta_t$ ) and tightly coupled to the SSM dynamics. This is a more parsimonious mechanism: the convolution is an organic consequence of the discretization rather than a bolted-on component.

Empirical consequence — the short convolution becomes optional: As shown in the ablation in Table 5a, Mamba-3 with exponential-trapezoidal and BC biases achieves a perplexity of 15.72 at the 440M scale, while adding the explicit short causal convolution (as used in Mamba-2) slightly degrades performance to 15.85. The internal convolution induced by the discretization, combined with the BC biases (which provide a data-independent convolutional component — see Section 3.4), makes the external convolution redundant.

Complex-Valued State Space Models

This section explains how Mamba-3 extends the real-valued state space to support complex-valued dynamics, enabling state tracking, and how this can be implemented efficiently through an equivalence with data-dependent rotary position embeddings (RoPE).

The Expressivity Bottleneck: Why Real Eigenvalues Can't Represent Rotation

Mamba-2's transition matrix is a scalar times identity: $\mathbf{A}_t = A_t \mathbf{I}$ where $A_t \in \mathbb{R}_{<0}$ . After discretization, the state update becomes $\mathbf{h}_t = e^{\Delta_t A_t} \mathbf{h}_{t-1} + \ldots$ where $e^{\Delta_t A_t} \in (0, 1)$ . This means the state transition can only scale the previous state — it always shrinks each component of $\mathbf{h}_{t-1}$ toward zero. It cannot rotate state components or change their signs in a structured way.

Why this fails for parity: The parity function (is the sum of binary inputs even or odd?) requires the state to alternate between two configurations — representing "even" and "odd" — based on the input. A rotation matrix can do this: start with state $[1, 0]^\top$ (representing even), and apply a 180° rotation $R(\pi) = \begin{bmatrix} -1 & 0 \\ 0 & -1 \end{bmatrix}$ when the input is 1 to flip to $[-1, 0]^\top$ (representing odd). A scalar decay cannot achieve this alternation because it can only shrink values, not flip their signs in a controlled way. This is formalized in Theorem 1 of Grazzi et al. (2024), which the paper cites: restricting eigenvalues to real numbers prevents representing rotational hidden state dynamics.

The Complex-Valued SSM and Its Discretization

The paper begins with a complex-valued SSM where the state, transition, and projections all have imaginary components:

$\dot{\mathbf{h}}(t) = \text{Diag}(\mathbf{A}(t) + i \boldsymbol{\theta}(t)) \mathbf{h}(t) + (\mathbf{B}(t) + i \hat{\mathbf{B}}(t)) x(t)$ $y(t) = \text{Re}\left((\mathbf{C}(t) + i \hat{\mathbf{C}}(t))^\top \mathbf{h}(t)\right)$

where $\mathbf{h}(t) \in \mathbb{C}^{N/2}$ (the state dimension is halved since complex numbers store two real values), $\boldsymbol{\theta}(t) \in \mathbb{R}^{N/2}$ is the imaginary part of the transition (the rotation frequency for each state channel), $\mathbf{B}(t), \hat{\mathbf{B}}(t) \in \mathbb{R}^{N/2}$ are the real and imaginary parts of the input projection, $\mathbf{C}(t), \hat{\mathbf{C}}(t) \in \mathbb{R}^{N/2}$ are the real and imaginary parts of the output projection, and $x(t), \mathbf{A}(t) \in \mathbb{R}$ are real-valued (the decay and input are real). The output takes only the real part of the complex readout.

What the imaginary transition $i \boldsymbol{\theta}(t)$ does: In continuous time, a purely imaginary eigenvalue $i\theta$ produces oscillations: the state component rotates in the complex plane at frequency $\theta$ . When combined with a real decay $A(t)$ , the dynamics are a damped rotation — the state both shrinks (from the real part) and rotates (from the imaginary part). This directly enables the rotational state dynamics needed for tasks like parity.

Proposition 2: Equivalence to a Real Block-Diagonal SSM

The paper proves that, under exponential-Euler discretization, this complex SSM is mathematically equivalent to a real-valued SSM with doubled state dimension and a specific block-diagonal structure:

$\mathbf{h}_t = e^{\Delta_t A_t} \mathbf{R}_t \mathbf{h}_{t-1} + \Delta_t \mathbf{B}_t x_t$ $y_t = \mathbf{C}_t^\top \mathbf{h}_t$

where now $\mathbf{h}_t \in \mathbb{R}^N$ (real-valued, dimension $N$ ), the projections stack real and imaginary parts:

$\mathbf{B}_t = \begin{bmatrix} \mathbf{B}_t \\ \hat{\mathbf{B}}_t \end{bmatrix} \in \mathbb{R}^N, \quad \mathbf{C}_t = \begin{bmatrix} \mathbf{C}_t \\ -\hat{\mathbf{C}}_t \end{bmatrix} \in \mathbb{R}^N$

and the transition matrix $\mathbf{R}_t \in \mathbb{R}^{N \times N}$ is block-diagonal with $2 \times 2$ rotation blocks:

$\mathbf{R}_t = \text{Block}\left(\{R(\Delta_t \boldsymbol{\theta}_t[i])\}_{i=1}^{N/2}\right), \quad R(\theta) = \begin{bmatrix} \cos(\theta) & -\sin(\theta) \\ \sin(\theta) & \cos(\theta) \end{bmatrix}$

What this means computationally: The complex-valued transition $e^{\Delta_t (A_t + i \boldsymbol{\theta}_t)}$ has been decomposed into a scalar decay $e^{\Delta_t A_t}$ (which scales each dimension independently) and a block-diagonal rotation matrix $\mathbf{R}_t$ (which mixes pairs of dimensions). Each pair of state dimensions $[h_{2i}, h_{2i+1}]$ undergoes a 2D rotation by angle $\Delta_t \boldsymbol{\theta}_t[i]$ — exactly the operation needed for state tracking.

Why this mapping works: The complex exponential splits via Euler's formula: $e^{\Delta_t (A_t + i \theta_t)} = e^{\Delta_t A_t} (\cos(\Delta_t \theta_t) + i \sin(\Delta_t \theta_t))$ . In real coordinates (representing a complex number as a 2D vector), multiplication by $e^{i\Delta_t \theta_t}$ is exactly the 2D rotation matrix $R(\Delta_t \theta_t)$ . Stacking $N/2$ such 2D rotations gives the block-diagonal $\mathbf{R}_t$ .

Proposition 3: The RoPE Trick — Efficient Computation Without Explicit Rotations

Computing the block-diagonal rotation $\mathbf{R}_t$ and applying it to the state $\mathbf{h}_{t-1}$ at every step would add significant computational overhead. The paper shows how to avoid this through a change of variables — the "RoPE trick":

$\tilde{\mathbf{h}}_t = e^{\Delta_t A_t} \tilde{\mathbf{h}}_{t-1} + \left(\prod_{i=0}^t \mathbf{R}_i^\top\right) \Delta_t \mathbf{B}_t x_t$ $y_t = \left[\left(\prod_{i=0}^t \mathbf{R}_i^\top\right) \mathbf{C}_t\right]^\top \tilde{\mathbf{h}}_t$

where $\tilde{\mathbf{h}}_t \equiv \left(\prod_{i=0}^t \mathbf{R}_i^\top\right) \mathbf{h}_t$ is a rotated version of the original state.

What this computes: Rather than applying rotations to the state at each step, we absorb the rotation into the $\mathbf{B}$ and $\mathbf{C}$ projections by pre-multiplying them with the cumulative product of rotation matrices. The state update itself reverts to the simple scalar-transition form $\tilde{\mathbf{h}}_t = e^{\Delta_t A_t} \tilde{\mathbf{h}}_{t-1} + \ldots$ , but with rotated $\mathbf{B}$ and $\mathbf{C}$ .

Why this is efficient: The rotation matrices $\mathbf{R}_t$ are block-diagonal with 2×2 blocks — applying them to vectors is cheap (just 2D rotations for each pair of dimensions). More importantly, the cumulative product $\prod_{i=0}^t \mathbf{R}_i^\top$ can be computed incrementally: each new $\mathbf{R}_t$ is multiplied onto the running product. Since the $\mathbf{R}_t$ matrices commute ( $\mathbf{R}_i \mathbf{R}_j = \mathbf{R}_j \mathbf{R}_i$ , because they operate on disjoint pairs of dimensions), the order of multiplication doesn't affect correctness.

Connection to standard RoPE: Standard Rotary Position Embeddings (Su et al., 2023) apply data-independent rotations to query and key vectors in attention, where the rotation angles follow a fixed frequency schedule $\theta[i] = 10000^{-2i/N}$ . In Mamba-3, the rotations are data-dependent — $\boldsymbol{\theta}_t$ is produced by a learned projection of the current token, so the rotation angle adapts to the input content. This is a generalization: standard RoPE encodes absolute positional information through fixed rotations; data-dependent RoPE can additionally encode content-dependent rotational dynamics.

The paper claims this is "the first usage of data-dependent RoPE grounded in theoretical motivations" (Section 5.2), distinguishing it from prior uses of RoPE in SSMs (like RetNet's constant complex phase) which lacked the theoretical connection to expressivity.

Proposition 4: Complex SSM with Exponential-Trapezoidal Discretization

Extending the complex SSM to use the exponential-trapezoidal rule (rather than exponential-Euler) yields a 3-term recurrence with rotations on all input terms:

$\mathbf{h}_t = \alpha_t \mathbf{h}_{t-1} + \beta_t \left(\prod_{i=0}^{t-1} \mathbf{R}_i^\top\right) \mathbf{B}_{t-1} x_{t-1} + \gamma_t \left(\prod_{i=0}^t \mathbf{R}_i^\top\right) \mathbf{B}_t x_t$ $y_t = \left[\left(\prod_{i=0}^t \mathbf{R}_i^\top\right) \mathbf{C}_t\right]^\top \mathbf{h}_t$

where $\alpha_t = e^{\Delta_t A_t}$ , $\beta_t = (1 - \lambda_t) \Delta_t e^{\Delta_t A_t}$ , and $\gamma_t = \lambda_t \Delta_t$ as before. The rotations are applied to both the previous input term (with cumulative product up to $t-1$ ) and the current input term (with cumulative product up to $t$ ). This is the final SSM recurrence used in Mamba-3.

What changes from the real-valued exponential-trapezoidal recurrence: Only the $\mathbf{B}$ and $\mathbf{C}$ projections are rotated by the cumulative rotation products. The decay $\alpha_t$ remains a real scalar, and the structure of the 3-term recurrence (state decay + previous input + current input) is unchanged. The RoPE trick thus cleanly decouples the rotational expressivity from the discretization scheme — you can upgrade either independently.

Multi-Input, Multi-Output (MIMO) Formulation

This section explains how Mamba-3 modifies the input-output structure of the SSM recurrence to increase computational intensity during decoding, making better use of GPU hardware without slowing down generation.

The Memory-Bound Nature of SISO Decoding

In the standard (SISO) formulation used by Mamba-1 and Mamba-2, a single SSM head with head dimension $P$ operates as follows at each decode step:

$\mathbf{H}_t = \alpha_t \mathbf{H}_{t-1} + \Delta_t \mathbf{B}_t \mathbf{x}_t^\top$ $\mathbf{y}_t = \mathbf{H}_t^\top \mathbf{C}_t$

where $\mathbf{H}_t \in \mathbb{R}^{N \times P}$ is the state matrix, $\mathbf{B}_t \in \mathbb{R}^N$ is the input projection, $\mathbf{x}_t \in \mathbb{R}^P$ is the input vector, and $\mathbf{C}_t \in \mathbb{R}^N$ is the output projection. The critical computation is the outer product $\mathbf{B}_t \mathbf{x}_t^\top$ , which creates an $N \times P$ matrix from an $N$ -vector and a $P$ -vector.

Why this is memory-bound: The arithmetic intensity (FLOPs per byte of data transferred) for this SISO decode is approximately 2.5 ops/byte (Table 2a in the paper). For context, an H100 GPU's bfloat16 matrix multiply unit achieves about 295 ops/byte. The SSM decode is two orders of magnitude below the GPU's compute capability, meaning the tensor cores sit idle while waiting for data movement between GPU memory and compute units. The bottleneck is the state matrix $\mathbf{H}_t$ , which must be read and written at every step and dominates memory traffic ( $2NP$ bytes moved for $O(NP)$ FLOPs).

The MIMO Transformation: Adding Rank Without Adding State Size

The key insight is that the outer product $\mathbf{B}_t \mathbf{x}_t^\top$ is computationally cheap but $\mathbf{B}_t$ and $\mathbf{x}_t$ are small vectors. By expanding them to have an additional rank dimension $R$ , the computation becomes a matrix-matrix multiply while the state size $N \times P$ remains unchanged:

SISO (original): $\mathbf{B}_t \in \mathbb{R}^N$ , $\mathbf{x}_t \in \mathbb{R}^P$ , producing one $N \times P$ update matrix via outer product.

MIMO (Mamba-3): $\mathbf{B}_t \in \mathbb{R}^{N \times R}$ , $\mathbf{x}_t \in \mathbb{R}^{P \times R}$ , producing a sum of $R$ outer products that form the $N \times P$ update matrix.

The state update becomes:

$\mathbf{H}_t = \alpha_t \mathbf{H}_{t-1} + \Delta_t \mathbf{B}_t \mathbf{x}_t^\top$

where now $\mathbf{B}_t \mathbf{x}_t^\top = \sum_{r=1}^R \mathbf{B}_t^{(r)} [\mathbf{x}_t^{(r)}]^\top$ is the sum of $R$ outer products (each $\mathbf{B}_t^{(r)} \in \mathbb{R}^N$ and $\mathbf{x}_t^{(r)} \in \mathbb{R}^P$ ). The state $\mathbf{H}_t$ still has shape $N \times P$ — unchanged — but the computation to update it now involves $R$ times more FLOPs.

What this does to arithmetic intensity (Table 2): For SISO, the arithmetic intensity is: $\text{AI}_{\text{SISO}} = \frac{5NP - P}{2(1 + 2N + P + NP)} \approx 2.5 = \Theta(1)$

For MIMO with rank $R$ , the arithmetic intensity scales as: $\text{AI}_{\text{MIMO}} = \Theta(\min(N, P, R))$

When $R \ll N, P$ (the typical case; the paper uses $R = 4$ ), this is $\Theta(R)$ , meaning a $4\times$ increase in arithmetic intensity for $R = 4$ . The practical consequence: the GPU's tensor cores, which are optimized for matrix-matrix multiplication, can now be engaged on the $\mathbf{B}_t \mathbf{x}_t^\top$ operation (which is now a matmul between an $N \times R$ and an $R \times P$ matrix), while the memory traffic for $\mathbf{H}_t$ remains the dominant cost and doesn't increase significantly.

Why this doesn't hurt decode latency: Since the SISO decode is memory-bound (compute units idle waiting for data), adding more computation to the same memory movement essentially comes "for free" — the extra FLOPs can be executed while waiting for the state matrix to be read/written. Table 6 confirms this empirically: Mamba-3 MIMO with $R = 4$ at $\text{dstate}=128$ has a decode latency of 0.179 ms versus 0.156 ms for SISO — a 15% increase in wall-clock time for a $4\times$ increase in decode FLOPs and significantly better model quality.

Expressing MIMO as Multiple SISO SSMs (for Training)

While MIMO improves decode hardware utilization, training MIMO models requires a different computational strategy since training processes many tokens in parallel and doesn't suffer from the same memory-bound bottleneck. The paper shows that a MIMO SSM can be decomposed into $R^2$ independent SISO SSMs:

For each output channel $i \in \{0, \ldots, R-1\}$ and input channel $j \in \{0, \ldots, R-1\}$ :

$\mathbf{h}_t^{(j)} = \alpha_t \mathbf{h}_{t-1}^{(j)} + \Delta_t \mathbf{B}_t^{(j)} x_t^{(j)}$ $\mathbf{H}_t = \sum_{j=0}^{R-1} \mathbf{h}_t^{(j)}$ $y_t^{(i)} = [\mathbf{C}_t^{(i)}]^\top \mathbf{H}_t$

where each $\mathbf{B}_t^{(j)}, \mathbf{C}_t^{(i)} \in \mathbb{R}^N$ is a column of the MIMO projection matrices, and each $x_t^{(j)} \in \mathbb{R}$ is a scalar input channel. The total output is a vector $\mathbf{y}_t \in \mathbb{R}^R$ .

What this decomposition means: The MIMO SSM is equivalent to: (1) run $R$ independent SISO SSMs on $R$ input channels, (2) sum their states into a shared state $\mathbf{H}_t$ , and (3) read out $R$ output channels via separate $C$ projections, each applied to the shared state. This is $R^2$ SISO SSM invocations total: each of the $R$ outputs reads from a state that aggregates all $R$ inputs.

Why this enables training: The SISO SSD algorithm (the chunked parallel form from Section 2.3) can be invoked $R^2$ times as a black box to compute the MIMO SSM output during training. This reuses the highly optimized SISO kernel implementations without requiring new MIMO-specific training algorithms.

The Chunked Training Optimization: Reducing $O(R^2)$ to $O(R)$

A naive implementation training a MIMO SSM as $R^2$ SISO SSMs would be $R^2$ times slower than SISO training. The paper shows how the chunked SSD algorithm can be adapted to reduce this to $O(R)$ :

The key observation is that the intra-chunk computation (within each chunk of $C$ tokens processed in parallel) involves the term $\mathbf{C}^\top \mathbf{B}$ , which increases by $R^2\times$ in FLOP count. However, by reducing the chunk size proportionally — setting $C_{\text{MIMO}} = C_{\text{SISO}} / R$ — the total intra-chunk FLOPs remain unchanged while the number of chunks increases by $R$ . The inter-chunk recurrence (across chunks) increases by $R\times$ since it involves state projections. Overall, the training FLOPs increase by $R\times$ rather than $R^2\times$ .

Intuition: The $R^2$ overhead in the naive approach comes from computing all pairs of input-output interactions within each chunk. By making chunks $R$ times smaller, each chunk processes fewer tokens and thus fewer interaction pairs, keeping the per-chunk cost constant. The additional cost comes only from having $R$ times more chunks, each requiring state passing, which costs $O(RN P)$ per chunk boundary — an $O(R)$ total overhead rather than $O(R^2)$ .

The paper states in Section 3.3 that their released Triton kernels for Mamba-3 MIMO "only incur a slowdown of 2× when $R = 4$ , as compute latency can be parallelized with memory movement" — the actual slowdown is even less than the $R\times$ theoretical bound because the extra computation overlaps with memory movement.

MIMO Parameterization Choices (Appendix C)

A naive conversion from SISO to MIMO by simply expanding all projection dimensions by $R$ would multiply the parameter count by $R$ , which is prohibitive at scale. The paper makes specific design choices to keep the parameter count comparable:

Components that get the full $R\times$ expansion: $\mathbf{B}$ and $\mathbf{C}$ projections go from $\mathbb{R}^D \rightarrow \mathbb{R}^N$ to $\mathbb{R}^D \rightarrow \mathbb{R}^{N \times R}$ . These are shared across heads in Mamba's multi-value attention (MVA) structure, so the parameter increase is from $D N$ to $D N R$ for the entire layer — modest because $N$ is small (state size 128).

Components that use a lightweight expansion: The input $\mathbf{x}$ and gate $\mathbf{z}$ , which are per-head and dominate parameter count, use a two-step projection: first a standard SISO projection $\mathbb{R}^D \rightarrow \mathbb{R}^P$ , then an element-wise scaling to size $R$ via a learned data-independent vector. This changes the parameter count from $D P$ to $D P + P R$ per head — the $P R$ term is negligible since $R \ll P$ (e.g., $R = 4$ , $P = 64$ ).

Parameter matching: To ensure fair comparisons with SISO baselines, the MLP inner dimension is slightly reduced in MIMO models. At 1.5B scale, the MLP hidden dimension decreases from 4096 to 3824 — a 6.6% reduction (Appendix C).

Mamba-3 Architectural Integration

This section explains how the three methodological improvements are combined into the complete Mamba-3 layer and what auxiliary architectural choices were made.

The Overall Block Structure

The Mamba-3 block (Figure 2 in the paper) follows the same high-level layout as Mamba-2 but with several modifications. The input token embedding first passes through input projections to produce all SSM parameters:

Projections for $\mathbf{B}$ , $\mathbf{C}$ : Linear layers map the token embedding to $\mathbb{R}^N$ (SISO) or $\mathbb{R}^{N \times R}$ (MIMO). These are followed by BC normalization (RMSNorm) and BC biases (learnable per-channel additions).
Projections for $\mathbf{X}$ (the SSM input), $\mathbf{Z}$ (the gate): Linear layers produce per-head input and gating signals.
Projections for $\Delta$ and $\boldsymbol{\theta}$ : The step size $\Delta_t$ (which controls the decay $\alpha_t = e^{\Delta_t A_t}$ and the input weights $\beta_t, \gamma_t$ ) and the rotation frequencies $\boldsymbol{\theta}_t$ (which control the complex rotation) are produced by learned linear projections from the token embedding.
The core SSM recurrence: The exponential-trapezoidal recurrece with complex-valued transitions (Proposition 4) processes the sequence, producing an output $\mathbf{y}_t \in \mathbb{R}^P$ (SISO) or $\mathbb{R}^{P \times R}$ (MIMO).
Output gating: $\mathbf{y}_t$ is element-wise multiplied by $\text{SiLU}(\mathbf{z}_t)$ (the SiLU-gated linear unit), then projected back to the model dimension $D$ .

BC Normalization (BCNorm / QKNorm)

RMS normalization is applied to the $\mathbf{B}$ and $\mathbf{C}$ projections after the linear layer. Section 3.4 states this "mirroring the QKNorm commonly used in modern Transformers" and cites recent linear models that also adopt this practice (Hu et al., 2025; Yang et al., 2025a).

What it does: For each token, the $\mathbf{B}$ and $\mathbf{C}$ vectors are normalized to have unit RMS (root-mean-square) before being used in the SSM recurrence. This prevents the magnitude of these projections from growing or shrinking in ways that could destabilize the recurrence — since $\mathbf{B}$ and $\mathbf{C}$ directly determine how much information enters and exits the state, their scale interacts multiplicatively with the decay dynamics, and uncontrolled growth can cause training instability.

Practical consequence: The paper reports that BCNorm stabilizes large-scale training runs and, critically, allows removal of the post-gate RMSNorm that Mamba-2 introduced for stability. In pure Mamba-3 models (no self-attention), this removal simplifies the architecture. However, in hybrid models (Mamba-3 interleaved with attention layers), the post-gate RMSNorm is retained because it "is crucial for long-context extrapolation" (Section 3.4, Table 4).

BC Biases (Learnable Per-Channel Additions)

After BC normalization, learned bias vectors are added to $\mathbf{B}$ and $\mathbf{C}$ . These biases are:

Head-specific: each attention head has its own bias vector
Channel-wise: a separate scalar bias for each of the $N$ state dimensions
Data-independent: the bias is a fixed learned parameter, not computed from the input
Initialized to all ones (Table 10a shows this initialization performs best)

What this does (conceptual): Adding a constant offset to $\mathbf{B}$ means the state-input becomes $(\mathbf{B}_t^{\text{data}} + \mathbf{B}^{\text{bias}}) x_t$ . The data-dependent part $\mathbf{B}_t^{\text{data}} x_t$ captures content-specific information, while the bias term $\mathbf{B}^{\text{bias}} x_t$ provides a data-independent, fixed response to the input — analogous to a convolution filter with fixed weights. Similarly, adding bias to $\mathbf{C}$ provides a fixed readout that doesn't depend on the token content.

Connection to the short convolution removal: The exponential-trapezoidal discretization already induces a data-dependent convolution of width 2 on the state-input (the $\beta_t \mathbf{B}_{t-1} x_{t-1} + \gamma_t \mathbf{B}_t x_t$ term). Adding BC biases introduces a data-independent convolutional component. Together, these provide sufficient local mixing to make the explicit short causal convolution (used in Mamba-2 and most other recurrent models) redundant. Table 5a confirms: Mamba-3 with both exponential-trapezoidal and BC biases but without the external convolution achieves 15.72 perplexity; adding the convolution back yields 15.85 (slightly worse).

Ablation on which biases to include (Table 10b): The paper tests combinations: no biases (16.52 perplexity, worst), B-bias only (16.68, worse than none — consistent with Yu and Erichson's 2025 block-biased formulation being insufficient alone), C-bias only (15.98, improvement), both B and C biases (15.69, best). The B and C biases exhibit synergy — both are needed for the full benefit.

Why all-ones initialization? Table 10a ablates different initializations. All-ones achieves 15.72 perplexity. Zero initialization degrades to 16.57 — the model needs a non-zero starting point for the biases to be useful. Uniform random in $[0, 1]$ achieves 15.76 (similar to all-ones), while uniform in $[-1, 1]$ degrades to 16.07 — negative initial biases are harmful, likely because they can cause sign flips in the state-input that interfere with the decay dynamics.

The Complex-Valued $\mathbf{A}$ Parameterization

The paper makes a specific choice about $\mathbf{A}$ that's noted in Remark 1: "In Mamba-3, we empirically found that data-dependent $A_t$ has similar performance to data-independent $A_t$ , and chose the former as a default for consistency so that all SSM parameters are data-dependent." In Mamba-2, $A_t$ was a learned scalar constant (not computed from the input), and the data-dependence came only through $\Delta_t$ which modulates how $A_t$ affects the recurrence. Mamba-3 makes $A_t$ itself a function of the input (in addition to $\Delta_t$ ), making the full discrete decay $\alpha_t = e^{\Delta_t A_t}$ doubly data-dependent. The paper finds this doesn't hurt and keeps the design principle uniform.

For the complex extension, the SSM parameter $\mathbf{A}$ is partitioned into:

Real part $A_t$ : controls the scalar decay, passed into the SSD computation as in Mamba-2
Imaginary part $\boldsymbol{\theta}_t$ : controls the rotational dynamics, implemented via the RoPE trick (cumulative rotation applied to $\mathbf{B}$ and $\mathbf{C}$ )

This clean separation allows reusing the existing Mamba-2 SSD infrastructure for the decay computation, with the rotational component added as a lightweight modification to the input/output projections.

Key Architectural Differences from Mamba-2

The paper's Figure 2 contrasts the Mamba-2 and Mamba-3 blocks visually. The modifications are:

No explicit short causal convolution in Mamba-3 (removed because exponential-trapezoidal + BC biases provide sufficient local mixing)
Additional projections for $\mathbf{A}$ (real and imaginary parts), with the imaginary part feeding into the RoPE computation
BC normalization applied after $\mathbf{B}$ and $\mathbf{C}$ projections
BC biases (learnable, data-independent per-channel additions) added after normalization
Optional MIMO projections that expand $\mathbf{B}$ , $\mathbf{C}$ , $\mathbf{X}$ with rank $R$

The element-wise gating structure (SiLU gate on the SSM output) remains unchanged from Mamba-2, as does the alternating pattern of Mamba-3 and SwiGLU MLP blocks in the full model.

Design Principle: Inference-First Thinking

The paper emphasizes that all three methodological improvements arise from an "inference-first" design philosophy — thinking about what constraints and opportunities exist during autoregressive generation, not just during training:

Exponential-trapezoidal discretization increases the FLOPs of the recurrence (from 2-term to 3-term), which would slow training if done naively, but the parallel SSD form absorbs this cost during training. During decode, the extra term adds negligible latency but significantly increases expressivity.
Complex-valued state transitions add rotations that are computed via the RoPE trick — a few 2D rotations per state dimension, which is cheap. During training, the rotations are pre-computed and applied to $\mathbf{B}$ and $\mathbf{C}$ before the main SSD computation.
MIMO is explicitly motivated by the memory-bound nature of SISO decode: "to improve FLOP efficiency during decoding, we switch from an outer-product–based state update to a matrix-multiplication–based state update" (Section 3.3). The $R\times$ increase in decode FLOPs comes essentially for free in wall-clock time because the computation was previously idle waiting for memory.

The paper's view is that training algorithms (which process many tokens in parallel) and inference algorithms (which process one token at a time) have fundamentally different computational bottlenecks, and architectures should be designed to exploit the slack in both regimes.

4. Key Insights and Innovations

Innovation 1: Inference-Time Architecture Design as a First-Class Design Principle

The deepest conceptual shift in this paper is methodological, not technical: it treats autoregressive decoding as the primary design constraint rather than an afterthought, and derives all three architectural improvements from thinking about what happens when you generate one token at a time.

What the field did before: Prior sub-quadratic models — including Mamba-1 and Mamba-2 — were designed primarily from a training perspective. Mamba-2's core contribution was a chunked parallel algorithm (SSD) that made training fast by exploiting GPU tensor cores for matrix multiplication. The decode behavior was treated as a secondary concern — you simply ran the same recurrence one step at a time. The paper is explicit about the consequence: "their inference algorithms are not hardware efficient... because these algorithms were developed from a training perspective, their decoding phase has low arithmetic intensity... resulting in large portions of hardware remaining idle" (Section 1). This was true not just of Mamba-2 but of essentially all modern linear-time models — linear attention, Gated DeltaNet, test-time training models all inherit this training-first design pattern.

What Mamba-3 does differently: The paper inverts the priority. The MIMO formulation is the clearest example: it starts from the observation that SISO decode achieves ~2.5 ops/byte on hardware capable of ~295 ops/byte — a 100× gap between actual and potential compute utilization. The solution (adding rank to the state update) is motivated entirely by the memory-bound characteristics of autoregressive generation, not by any training-time consideration. In fact, MIMO makes training more expensive (by up to 2× in practice, Section 3.3), but the paper accepts this cost because the decode benefit — better model quality at the same wall-clock latency — is worth it in deployment. This is a genuine inversion of the standard tradeoff calculus.

The exponential-trapezoidal discretization and complex-valued state transitions similarly reflect inference-first thinking, though more subtly. Both add computation to the recurrence (a third term in the update, 2D rotations on B and C projections) that is negligible during serial decode but would be painful if the recurrence were computed sequentially at training time. The paper gets away with this because the parallel SSD training form absorbs the extra work — during training, you compute the full sequence in chunks via matrix multiplication, and the additional terms fold into the structured mask L without changing the asymptotic complexity. This is design by exploiting the asymmetry between training and inference computational bottlenecks: training is compute-bound (you can always use more FLOPs), inference is memory-bound (extra FLOPs are free if they don't add memory traffic).

Why this is a fundamental shift, not incremental: Treating inference as the primary design target changes what counts as a "good" architectural modification. From a training perspective, a change that adds FLOPs without commensurate perplexity improvement is a loss. From an inference perspective, a change that adds FLOPs during memory-bound decode while improving quality is a pure win — you're converting idle compute capacity into better modeling. The MIMO result encapsulates this: 4× more decode FLOPs for a 15% latency increase and a 0.11 perplexity improvement at 1.5B scale (Tables 3, 6). A training-first designer would never have explored this direction because the training slowdown (2×) would have been flagged as unacceptable before the decode benefits were measured.

Evidence: The state-size Pareto frontier in Figure 3 is the clearest demonstration. Mamba-3 MIMO with state size 64 achieves better perplexity than Mamba-2 with state size 128 — and since decode latency scales with state size, this means identical quality at roughly half the latency. The entire MIMO section is a worked example of inference-first reasoning: identify the bottleneck (memory-bound outer product), observe that the hardware has idle compute (tensor cores sitting at 1% utilization), and restructure the operation to use that idle capacity (from outer product to matmul).

Innovation 2: Discretization as a Source of Architectural Expressivity, Not Just Numerical Accuracy

The paper's discretization framework introduces a conceptual shift in how we think about converting continuous-time SSMs to discrete recurrences. Prior work treated discretization as a necessary implementation step — something you do to make the math work on discrete token sequences — and focused on numerical accuracy (minimizing truncation error). Mamba-3 treats discretization as a design space for architectural expressivity.

What the field did before: Earlier SSMs used standard discretization formulas from control theory, primarily zero-order hold (ZOH) and bilinear/trapezoidal methods (S4, S4D, S5). These were applied to linear time-invariant systems where the discretization formula is analytically derived and the goal is to faithfully reproduce the continuous dynamics. Mamba-1 and Mamba-2 departed from this by making the SSM time-varying (data-dependent), but their discretization was driven by implementation convenience — a heuristic approximation (what the paper retrospectively names exponential-Euler) was used without theoretical justification. The paper notes that "Mamba-1 adapted the zero-order hold method to LTV systems without proof... the complexity associated with selective SSMs prompted the use of an additional heuristic approximation that lacked theoretical justification" (Section 3.1). In other words, the field had a gap between rigorous ZOH (which applies to LTI systems and doesn't cleanly extend to LTV) and the heuristic that practitioners actually used.

What Mamba-3 contributes conceptually: The paper provides the missing theoretical bridge — Proposition 5 derives a variation-of-constants formula for LTV systems that separates the state transition (treated exactly via the exponential) from the state-input integral (available for approximation). This framework doesn't just justify the prior heuristic; it opens up a family of possible discretizations parameterized by how you approximate the state-input integral. Exponential-Euler (Mamba-1/2) is the simplest member — a first-order method using one endpoint. Exponential-trapezoidal is a second-order member — using both endpoints with a learned weighting λ_t.

The conceptual leap is recognizing that the approximation error properties are less important than the architectural properties. Classical numerical analysis would prescribe λ_t = 1/2 (the trapezoidal rule) to achieve O(Δ³_t) local truncation error. The paper explicitly rejects this constraint: "our ablations indicate that not enforcing this constraint is better for empirical performance" (Remark 3). Letting λ_t be data-dependent and learned — even though it technically degrades the convergence order — produces a more expressive recurrence because the model can learn to adaptively mix the previous and current inputs based on content. The discretization has become a learnable component of the architecture, not just a numerical scheme.

The implicit convolution insight as a reframing: The observation that the exponential-trapezoidal recurrence is equivalent to applying a width-2 convolution on the state-input before the recurrence (Section 3.1.2, Remark 4) is more than a mathematical curiosity. It explains why the discretization helps empirically — it provides local temporal mixing — and it aligns the SSM discretization with a mechanism (convolutions) that the field already understands to be useful. More importantly, it distinguishes this internal convolution from the external short convolutions used in Mamba-2, GDN, and most recurrent models. The external convolution is an independent operation applied to x_t before the SSM, requiring separate parameters and activations. The exponential-trapezoidal convolution operates on B_t x_t within the recurrence, with weights that are data-dependent and theoretically coupled to the SSM dynamics through Δ_t and λ_t. This is a more unified design: the local mixing emerges from the discretization theory rather than being bolted on.

Why this matters beyond the specific method: The framing suggests a general research direction: rather than searching over ad-hoc architectural modifications (should we add convolutions? of what width? with what activation?), derive them from the discretization of higher-order dynamical systems. The exponential-trapezoidal rule is the trapezoidal approximation applied to a first-order ODE — one could imagine extending this to higher-order ODEs (producing wider implicit convolutions) or to other integration schemes (producing different recurrence structures). The paper doesn't explore this space, but it provides the conceptual scaffolding for doing so.

Evidence: The ablation in Table 5a is the key result. Mamba-3 with exponential-trapezoidal discretization and BC biases (but no external convolution) achieves perplexity 15.72. Adding the explicit short convolution degrades to 15.85. Removing exponential-trapezoidal (reverting to exponential-Euler, i.e., Mamba-2's recurrence) with biases yields 16.68. The discretization isn't just a numerical detail — it's worth 0.96 perplexity points, about the same gap as exists between completely different model families in Table 3. And it enables removing a component (the external convolution) that prior work considered essential, simplifying the architecture while improving performance.

Innovation 3: Data-Dependent Rotary Embeddings as a Principled Solution to the State-Tracking Problem

The complex-valued SSM contribution is notable not just for adding state-tracking capability — Gated DeltaNet had already shown that negative eigenvalues could recover parity (Grazzi et al., 2025) — but for connecting a practical capability gap to a specific mathematical mechanism through a theoretical equivalence, and for providing the first principled justification for data-dependent RoPE.

What the field knew before: The state-tracking limitation of real-valued SSMs was established by prior theoretical work (Merrill et al., 2025; Grazzi et al., 2024, 2025; Sarrof et al., 2024). The diagnosis: restricting state transition eigenvalues to real numbers (specifically, non-negative reals in the scalar-times-identity case) prevents the model from representing rotational dynamics needed for tasks like parity and modular arithmetic. Several fixes had been proposed: allowing negative eigenvalues (Yang et al.'s Gated DeltaNet with eigenvalue range [-1,1]), channel-specific biases (Yu & Erichson, 2025), or block decomposition. These approaches worked empirically but were somewhat ad-hoc — "allow negative eigenvalues" is a parameter range expansion, not a mechanistic explanation of what computation becomes possible.

The theoretical connection between complex SSMs and RoPE was partially anticipated. S4 and other early SSMs used complex-valued state transitions, but these were linear time-invariant (LTI) systems where the complex parameters were fixed after training. The complex values were a mathematical convenience for representing certain normal-plus-low-rank matrices efficiently, not a mechanism for content-dependent computation. RetNet (Sun et al., 2023) used RoPE within an SSM-like model, but the rotations were data-independent (standard sinusoidal position encodings) and the underlying dynamics were LTI with constant scalar decays — closer to linear attention than to selective SSMs.

What makes Mamba-3's contribution distinctive: The paper establishes a three-way equivalence that had not been previously articulated: complex-valued SSM (with data-dependent imaginary transition) ⇔ real-valued SSM with block-diagonal rotation matrices ⇔ standard scalar-transition SSM with data-dependent RoPE applied to B and C. This isn't just a computational trick — it's a theoretical statement about what representational capacity is being added and why it solves the state-tracking problem.

The critical insight is in Proposition 2: the discretized complex SSM's transition matrix becomes $e^{\Delta_t A_t} \mathbf{R}_t$ — a scalar decay times a block-diagonal rotation. The rotation $\mathbf{R}_t$ operates on pairs of state dimensions, mixing them via 2D rotation matrices with angles $\Delta_t \boldsymbol{\theta}_t[i]$ that are data-dependent (produced by a learned projection of the current token). This is precisely the mechanism needed for parity: one pair of state dimensions can encode "even/odd" as two configurations related by a 180° rotation, with the rotation angle depending on the current input bit. The scalar-decay component (from the real part $A_t$ ) provides forgetting, while the rotation component (from the imaginary part $\boldsymbol{\theta}_t$ ) provides state transitions that preserve information magnitude while changing its representation.

The RoPE trick (Proposition 3) is the observation that you can absorb these rotations into the B and C projections rather than applying them to the state at each step. This is a computational convenience but also a conceptual unification: it shows that data-dependent RoPE — which might seem like an arbitrary architectural hack — has a precise interpretation as the imaginary component of a complex-valued SSM transition. The paper claims this is "the first usage of data-dependent RoPE grounded in theoretical motivations" (Section 5.2), and the claim is credible: prior uses of RoPE in SSMs (RetNet) or attention (Su et al., 2023) were for positional encoding with fixed frequency schedules, not for content-dependent state tracking.

A diagnostic justification, not just a capability gain: The synthetic experiments in Table 5b serve a specific purpose: they validate that the proposed mechanism actually solves the diagnosed problem. Mamba-3 (with data-dependent RoPE) achieves 100% accuracy on parity and 98.51% on modular arithmetic without brackets — tasks that Mamba-2 and Mamba-3 without RoPE fail at random-guessing levels (0.90–2.27%). This isn't about language modeling benchmarks; it's about confirming that the theoretical connection (complex eigenvalues → rotational dynamics → state tracking) holds in practice. The fact that Mamba-3 with standard (data-independent) RoPE also fails (1.56% on parity) is particularly important — it shows that the benefit comes specifically from the data-dependence of the rotations, not from having any rotation at all. Standard RoPE provides fixed position-dependent rotations that don't help with content-dependent state tracking.

Why this is fundamental rather than incremental: The complex-valued SSM contribution doesn't just add a capability — it provides a principled mechanism for adding that capability that integrates cleanly with the existing SSM framework. The discretization theory from Section 3.1 extends naturally to complex parameters (Proposition 4 shows the exponential-trapezoidal version of the complex recurrence). The RoPE trick makes the implementation cheap enough to be practical. And the theoretical connection (complex SSM ⇔ real SSM with rotations ⇔ SSM with RoPE) provides a vocabulary for reasoning about what kinds of computation different SSM variants can and cannot perform. This is a sharp contrast with approaches like "allow negative eigenvalues in the parameter range," which improve empirical capability but don't provide mechanistic understanding of why.

Evidence: Table 5b is the headline, but the theoretical architecture (Propositions 2-4) is the substance. The paper doesn't just show that Mamba-3 solves parity — it explains why parity requires rotational dynamics, how complex eigenvalues provide those dynamics, and what the efficient implementation looks like. This is a worked example of using theory to guide architecture design, rather than discovering architecture through empirical search and retroactively justifying it.

Innovation 4: MIMO as a Computational, Not Mathematical, Innovation — Breaking the State-Size/Quality Tradeoff

The MIMO contribution is conceptually distinct from the other two because it is not about improving the mathematical expressivity of the SSM — it's about exploiting hardware characteristics to get better models without paying the usual latency cost. This is an engineering insight disguised as a methodological one, and it reveals something important about how sub-quadratic models should be designed.

What the field assumed before: The standard design axis for trading off model quality against inference speed in recurrent models is the state size $N$ . A larger state can store more information, leading to better perplexity and downstream performance, but decode latency scales with $N$ (more bytes to read/write at each step). The quality-speed Pareto frontier is thus assumed to be determined by $N$ : if you want a better model, you increase the state size and accept slower generation. This assumption holds for Mamba-1, Mamba-2, Gated DeltaNet, and essentially all fixed-state recurrent models.

What Mamba-3's MIMO reveals: There is a second, independent axis for improving model quality that does not increase state size: the rank of the state update. By expanding the input and output projections from vectors (SISO, rank 1) to matrices with $R$ columns (MIMO, rank $R$ ), you increase the computational work done at each decode step by $R\times$ without increasing the size of the state matrix $\mathbf{H}_t \in \mathbb{R}^{N \times P}$ that must be read and written. The key empirical observation is that this extra computation is essentially free in wall-clock time because the SISO decode is so severely memory-bound that the GPU's compute units were idle anyway.

This is a genuinely surprising result if you come from a training-centric mindset, where FLOPs are the primary cost metric and anything that adds FLOPs must justify itself with commensurate quality improvement. From an inference-centric mindset, the equation is different: if an operation is memory-bound, you can add FLOPs up to the point where it becomes compute-bound without increasing latency. The paper quantifies this: SISO achieves ~2.5 ops/byte, while the GPU's matmul units achieve ~295 ops/byte — there's a 100× gap to fill before compute becomes the bottleneck. MIMO with $R=4$ increases arithmetic intensity by roughly 4×, still leaving enormous headroom.

Why this is non-obvious from other frameworks: Section 5.4 explicitly argues that MIMO "does not naturally appear in associative memory (linear attention or test-time training) frameworks." In the SSM perspective, MIMO is a classical concept — moving from single-input single-output to multi-input multi-output dynamical systems. The SSM viewpoint makes the design space legible in a way that other frameworks do not. A linear attention researcher thinking in terms of keys, queries, and values would not naturally arrive at "increase the rank of the key-value outer product to exploit idle tensor cores" because the associative memory framework doesn't highlight the state update as the computational bottleneck or the outer product as the operation to modify.

The broader implication: The MIMO result suggests that sub-quadratic models have been under-designed for inference — there is a large, unexplored space of architectural choices that add computation to the memory-bound decode step without affecting latency. The paper's specific instantiation (rank- $R$ state update) is one point in that space, but the principle generalizes: any operation that increases decode FLOPs without proportionally increasing memory traffic can shift the quality-latency Pareto frontier. This is a different design philosophy from the standard "simplify the recurrence to make it faster" approach that dominated Mamba-2's development.

Evidence: Table 3 shows Mamba-3 MIMO ( $R=4$ ) improving average downstream accuracy by 1.2 points over SISO at 1.5B scale (57.6 vs. 56.4). Figure 3 shows Mamba-3 MIMO with state size 64 matching Mamba-2's perplexity with state size 128 — a 2× state size reduction that roughly halves decode latency. Table 6 confirms the latency claim: MIMO at dstate=128, bf16 takes 0.179 ms per decode step versus SISO's 0.156 ms — a 15% increase, not the 4× you'd expect from the FLOP count. The gap between 4× FLOPs and 1.15× latency is the exploitation of the memory-bound idle time.

A note on the tradeoff: MIMO is not free. Training is slower (2× in practice for $R=4$ , Section 3.3), and the model requires parameter matching adjustments (reducing MLP width) to keep total parameters comparable. But the paper argues, implicitly, that this is the right tradeoff for a deployment-focused design: pay a one-time training cost to get a model that is both better and equally fast at inference. This is the inference-first philosophy made concrete.

5. Experimental Analysis

Evaluation Methodology

Dataset. All language modeling experiments use the FineWeb-Edu dataset (Penedo et al., 2024), a filtered subset of CommonCrawl web data curated for educational quality. Models are pretrained on 100 billion tokens with the Llama-3.1 tokenizer (Grattafiori et al., 2024) at a 2K context length. Downstream evaluation uses the standard suite from LM Evaluation Harness (Gao et al., 2024): LAMBADA (OpenAI version, Paperno et al., 2016), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2019), Arc-Easy and Arc-Challenge (Clark et al., 2018), WinoGrande (Sakaguchi et al., 2019), and OpenBookQA (Mihaylov et al., 2018). Retrieval tasks add cloze-formatted versions of SWDE, SQuAD, FDA, TriviaQA, NQ, and DROP, plus synthetic Needle-in-a-Haystack (NIAH) tasks from Hsieh et al. (2024). State-tracking synthetic tasks use Parity and two versions of Modular Arithmetic from Grazzi et al. (2025), trained with a length curriculum up to 160 and evaluated at length 256.
Base model(s). All models are built on a Llama-style backbone (Grattafiori et al., 2024) alternating Mamba-3 or baseline sequence mixing blocks with SwiGLU MLP blocks under pre-normalization. Experiments span four parameter scales: 180M, 440M, 880M, and 1.5B parameters. The Mamba family uses an expand factor of 2, state size 128, and head dimension 64. The Transformer baseline follows the configuration from Dao and Gu (2024). Gated DeltaNet baselines follow Yang et al. (2025a) with query-key dimension 128 and value dimension 256. The model family choice is deliberate: PaLM 2-S* from the reference example is replaced by these open-weight-architecture models to ensure reproducibility.
Metrics. The primary quality metric is pretraining perplexity on held-out FineWeb-Edu validation data, reported as the standard exponentiated cross-entropy loss. Downstream evaluation uses zero-shot accuracy (exact match for LAMBADA and QA tasks, normalized accuracy for HellaSwag, and standard accuracy for the remainder). The paper reports an average accuracy across the seven downstream tasks as a summary statistic. For retrieval tasks, exact match accuracy is reported on cloze-format questions. For synthetic state-tracking, scaled accuracy (percentage of correct predictions, with chance = 50% for parity, lower for modular arithmetic) is used. For inference benchmarks, wall-clock latency in milliseconds per decode step and per prefill operation is measured on an NVIDIA H100-SXM5 GPU.
Baselines. The paper compares against five model families: (1) Transformer — standard Llama-style architecture with RoPE and SwiGLU MLP, following Dao and Gu (2024); (2) Mamba-2 (Dao and Gu, 2024) — the immediate predecessor, using scalar-times-identity transitions, exponential-Euler discretization, and an explicit short causal convolution; (3) Gated DeltaNet (GDN) (Yang et al., 2025a) — a strong contemporary linear model that replaces the additive state update with a delta-rule recurrence, using multi-head attention rather than multi-value attention; (4) Mamba-3 SISO — the paper's single-input, single-output variant, serving as both a contribution and a baseline for ablating MIMO benefits; and (5) Mamba-3 MIMO — the multi-input, multi-output variant with rank R=4, parameter-matched to SISO by reducing MLP inner dimension. For hybrid model experiments, each linear baseline is interleaved with NoPE self-attention at a 5:1 ratio of linear layers to attention layers.
Generation budget / compute accounting. For the quality-speed Pareto analysis (Section 4.3, Figure 3), the paper uses state size ( $d_{\text{state}}$ ) as a proxy for decode latency — larger state size means more bytes to read/write per step and thus slower generation. This is justified by the analysis in Section 3.3 showing that state memory traffic dominates decode runtime for memory-bound SISO operations. Models are compared at fixed state sizes (16, 32, 64, 128) and at fixed parameter counts (440M), all trained to 2× Chinchilla-optimal tokens (20× tokens per parameter, corresponding to ~8.8B tokens for 440M models). For MIMO, the rank R = 4 adds FLOPs without changing state size, and the comparison keeps total parameters matched by reducing MLP width (e.g., MLP dim reduced from 2048 to 1792 for 440M, from 4096 to 3824 for 1.5B). For kernel benchmarks (Table 6), compute is measured in milliseconds per decode step at batch size 128 on a single H100-SXM5, with separate reporting for FP32 and BF16 precision and for $d_{\text{state}} \in \{64, 128\}$ . For end-to-end benchmarks (Table 7), prefill latency and prefill+decode latency are measured across sequence lengths 512 to 16384 at batch size 128.
Cross-validation / statistical protocol. The paper does not report confidence intervals or statistical significance tests for perplexity or accuracy numbers. For the state-tracking synthetic experiments, 8 learning rates are swept logarithmically between $10^{-4}$ and $10^{-2}$ , and $d_{\text{model}} \in \{32, 64\}$ is swept, with the best validation accuracy reported. For main pretraining experiments, all models at a given scale use identical training hyperparameters and procedures, so differences in perplexity and accuracy reflect architectural choices rather than optimization noise — though without error bars, the reliability of small differences (e.g., 0.1-0.3 perplexity gaps) is uncertain. The two-fold cross-validation protocol from the reference example (for compute-optimal strategy selection) does not apply to this paper, since Mamba-3 does not involve difficulty-adaptive test-time strategy selection — the model architecture itself is the contribution, and evaluation is standard held-out accuracy/perplexity on fixed architectures.

Main Quantitative Results

Language Modeling Quality Across Scales

Table 3 is the central result table. At every model scale from 180M to 1.5B, Mamba-3 SISO achieves the best validation perplexity among non-MIMO models, and Mamba-3 MIMO (R=4) achieves the best perplexity overall. The specific numbers at 1.5B scale:

Mamba-3 SISO: 10.35 perplexity, 56.4 average accuracy Mamba-3 MIMO: 10.24 perplexity, 57.6 average accuracy Gated DeltaNet: 10.45 perplexity, 55.8 average accuracy Mamba-2: 10.47 perplexity, 55.7 average accuracy Transformer: 10.51 perplexity, 55.4 average accuracy

The accuracy gaps are meaningful: Mamba-3 SISO improves over the next-best non-MIMO model (GDN) by 0.6 percentage points (56.4 vs. 55.8), and MIMO adds another 1.2 points (57.6 vs. 56.4) for a total improvement of 1.8 points over GDN and 2.2 points over the Transformer. These gains are not uniform across tasks: on Arc-Easy, Mamba-3 MIMO achieves 76.5% vs. 74.0% for Transformer (a 2.5 point gap); on Arc-Challenge, 44.5% vs. 40.4% (a 4.1 point gap); on WinoGrande, 60.6% vs. 58.7%. However, on PIQA the differences are smaller (75.3% vs. 73.8%), and on LAMBADA Mamba-3 SISO (49.4%) is slightly below GDN (49.2%) — though the absolute differences are within typical variance for these benchmarks. The consistent pattern across scales (180M, 440M, 880M, 1.5B) with monotonic improvements as models grow suggests the architectural advantage is robust, not a fluke of a particular scale or random seed.

A notable detail in Table 3: the second-best numbers are underlined, excluding MIMO variants. This means the paper treats MIMO as a separate tier — the fair SISO-to-SISO comparison is Mamba-3 SISO vs. GDN vs. Mamba-2 vs. Transformer, and MIMO is presented as an additional upgrade on top of the already-best SISO model. This is honest reporting, since MIMO's parameter matching (reducing MLP width) creates a slightly different computational profile for training.

Retrieval Capabilities

Table 4 evaluates retrieval at 1.5B scale on both real-world and synthetic tasks, with all models trained at 2K context length and evaluated at sequence lengths from 1024 to 4096. The results reveal a sharp capability gap between pure and hybrid models on certain tasks:

On structured QA tasks (SQuAD, TriviaQA, NQ): Pure Mamba-3 SISO achieves scores of 40.1, 64.5, and 26.5 respectively — competitive with Mamba-2 (39.1, 64.3, 25.1) and GDN (40.0, 63.5, 25.7), but substantially behind the Transformer (46.6, 67.5, 31.7). This deficit is expected for fixed-state models: they must compress context into the state, while attention can directly attend to earlier tokens.
On semi-structured and unstructured data (SWDE, FDA): The gap is even larger. Pure Mamba-3 SISO achieves only 28.5 on SWDE vs. 48.9 for Transformer, and 23.4 on FDA vs. 58.4 for Transformer. This is the most significant weakness the paper acknowledges — these tasks require extracting specific facts from structured tables (SWDE) or unstructured documents (FDA) where the answer location is not predictable, and the fixed-size state simply cannot retain all the relevant details.
On synthetic NIAH (needle-in-a-haystack): The story flips. Mamba-3 SISO achieves 100% on NIAH-Single-1 and NIAH-Single-2 at training length (1024 tokens), and crucially maintains 88.2% at 2048 and 50.6% at 4096 tokens for NIAH-Single-2 — far exceeding Mamba-2 (53.8% and 11.8%) and the Transformer (0% at all lengths beyond training due to RoPE length generalization failure). For NIAH-Single-3 (the hardest variant), Mamba-3 SISO reaches 34.2% at 4096 tokens vs. 13.4% for Mamba-2 and 0% for Transformer. This length generalization advantage is a key practical benefit: Mamba-3 degrades gracefully beyond its training context while the Transformer fails catastrophically (0% accuracy on all out-of-distribution lengths).
Hybrid models close most retrieval gaps. Adding self-attention at a 5:1 ratio boosts SWDE from 28.5 (pure) to 58.6 (hybrid with pre-gate grouped RMSNorm), closely matching the Transformer's 48.9 — and on FDA, the hybrid Mamba-3 SISO achieves 52.4, still below the Transformer's 58.4 but dramatically better than the pure model's 23.4. The hybrid also preserves the NIAH length generalization: with the right normalization (pre-gate grouped RMS), Mamba-3 SISO achieves 100% across all lengths on NIAH-Single-1 and -2, and 56.8% at 4096 on NIAH-Single-3.

The normalization sensitivity in hybrid models is notable (Table 9 in Appendix E). Five normalization configurations are tested: no additional norm, post-gate default RMS, pre-gate default RMS, post-gate grouped RMS, and pre-gate grouped RMS. The results show a tradeoff between in-context retrieval and length generalization: no additional norm achieves the best in-context real-world retrieval (58.5 SWDE, 65.9 FDA), but the worst length generalization (36.2% NIAH-Single-2 at 2048). Pre-gate grouped RMS achieves the best length generalization (100% NIAH-Single-2 at 2048, 96.0% at 4096) but sacrfices FDA performance (52.4 vs. 65.9). The paper doesn't resolve this tradeoff — it simply reports the configurations and notes that "hybrid models and their exact characteristics and dynamics are complex and oftentimes unintuitive" (Section 4.1.2, citing Cabannes et al., 2025).

State-Tracking Capabilities

Table 5b shows results on three formal language tasks from Grazzi et al. (2025), using models trained from scratch on these specific tasks (not the pretrained language models):

Parity: Mamba-3 achieves 100.00% scaled accuracy, matching GDN's 100.00% and starkly contrasting with Mamba-2 (0.90% — essentially random guessing), Mamba-3 with standard RoPE (1.56%), and Mamba-3 without RoPE (2.27%).
Modular Arithmetic without brackets: Mamba-3 achieves 98.51% vs. GDN's 99.25%. Mamba-2 scores 47.81% (above chance but far from solving), while Mamba-3 without RoPE collapses to 1.49% and Mamba-3 with standard RoPE to 20.70%.
Modular Arithmetic with brackets: This is the hardest variant, and Mamba-3 achieves 87.75% vs. GDN's 93.50%. Mamba-2 and ablations all score near chance (0.88–2.62%).

The critical diagnostic result is the comparison between Mamba-3 (100% parity) and Mamba-3 with standard RoPE (1.56%). Standard RoPE applies data-independent sinusoidal rotations — these encode positional information but do not adapt to the content of each token. The fact that standard RoPE fails completely while data-dependent RoPE succeeds confirms that the benefit is specifically from the content-dependent rotational dynamics, not from having rotations in general. This validates the theoretical claim from Section 3.2: the complex-valued transition (Proposition 2) provides data-dependent rotations that enable the state to conditionally flip between configurations — exactly the mechanism needed for parity. Standard RoPE's fixed rotations cannot implement this conditional logic.

The Mamba-2 result (0.90% on parity) is also informative: Mamba-2's scalar-times-identity transition with positive real decay fundamentally cannot represent the sign-flipping dynamics that parity requires. Even with negative eigenvalues, a real-valued scalar transition cannot rotate two state dimensions relative to each other — it can only scale them. This is a genuine expressivity ceiling, not a training difficulty issue.

Quality vs. Inference Speed: The State-Size Pareto Frontier

Figure 3 (and its expanded version in Appendix E, Figure 6 including GDN) plots the tradeoff between pretraining perplexity (y-axis, lower is better) and relative total state size (x-axis, lower means faster decode). Models are 440M parameters, trained to 2× Chinchilla-optimal tokens, with $d_{\text{state}}$ swept across {16, 32, 64, 128} (Mamba family) or head dimension swept across {32, 64, 128} (GDN, adjusting to match parameter count). The key finding:

Mamba-3 MIMO with state size 64 achieves better perplexity than Mamba-2 with state size 128.

Since decode latency scales approximately linearly with state size (the state matrix $\mathbf{H}_t \in \mathbb{R}^{N \times P}$ is the dominant memory traffic), this means Mamba-3 MIMO at state size 64 is roughly 2× faster than Mamba-2 at state size 128 while achieving the same or better quality. The Pareto frontier shows a clear downward shift at each step: Mamba-2 is the worst (highest perplexity for a given state size), Mamba-3 SISO shifts down (better for the same speed), and Mamba-3 MIMO shifts down further (even better). The GDN comparison in Figure 6 shows Mamba-3 MIMO generally below or comparable to GDN, though the comparison is complicated by the structural differences in how state size is defined for GDN's multi-head vs. Mamba's multi-value architecture.

The practical interpretation: for a fixed inference latency budget (determined by state size), Mamba-3 provides strictly better model quality than Mamba-2, and Mamba-3 MIMO provides another increment of quality without increasing latency at all (since MIMO adds FLOPs to the memory-bound decode step without changing state size). This is the central inference-first argument made concrete: MIMO improves quality and Mamba-3 SISO improves quality, but both maintain or improve the speed at which that quality is delivered.

Figure 4 (in the paper, showing length extrapolation) deserves mention: Mamba-3 at 1.5B demonstrates substantially better length extrapolation than Mamba-2 when evaluated beyond the 2K training context. At 32K context (16× training length), Mamba-3 maintains a perplexity around 10.7, while Mamba-2 degrades sharply to above 10.9. GDN shows intermediate behavior. This is an additional practical benefit not directly related to the three core methodological improvements — it may be due to the BC normalization, the exponential-trapezoidal discretization, or some interaction — but it's an empirically observed property of the complete architecture.

Inference Kernel Efficiency

Table 6 benchmarks per-token decode latency for a single decode step at batch size 128 on an H100-SXM5, comparing Mamba-2, GDN, Mamba-3 SISO, and Mamba-3 MIMO (R=4) across precision (FP32, BF16) and state size (64, 128). The models are 1.5B parameters, model dimension 2048. The headline result:

At the commonly used BF16, $d_{\text{state}} = 128$ setting, Mamba-3 SISO achieves 0.156 ms per decode step — faster than Mamba-2 (0.203 ms) and GDN (0.257 ms).

This is significant because Mamba-3 SISO adds computation (3-term recurrence instead of 2-term, BC normalization and biases, data-dependent RoPE) yet the implementation is faster than Mamba-2. The paper's released kernels are competitive independent of the architectural improvements. For MIMO (R=4), the latency at BF16, $d_{\text{state}} = 128$ is 0.179 ms — 15% slower than SISO but 12% faster than Mamba-2. The paper's claim that MIMO "incurs only a marginal latency cost" (Section 3.3) is supported: 4× more decode FLOPs yields 1.15× the latency of SISO and 0.88× the latency of Mamba-2.

The FP32 numbers show a somewhat different pattern: Mamba-3 SISO at $d_{\text{state}} = 128$ achieves 0.399 ms vs. Mamba-2's 0.409 ms (essentially tied) and GDN's 0.423 ms. At $d_{\text{state}} = 64$ , Mamba-3 SISO (0.310 ms) is slightly slower than Mamba-2 (0.295 ms). This precision dependence suggests the kernels have different optimization characteristics at different datatypes — a detail not explored in the paper.

Table 7 provides end-to-end latency including prefill, across sequence lengths 512 to 16384. The key insight: recurrent mixers scale much more gently with context length than Transformers. At 16384 tokens, vLLM Llama-3.2-1B takes 976.5 ms for prefill+decode (estimated at batch size 128 equivalent), while Mamba-3 SISO takes 140.6 ms — a 7× speedup. Even at shorter lengths (2048 tokens), Mamba-3 SISO (17.57 ms) is slightly faster than Mamba-2 (18.62 ms) and the Transformer (20.37 ms). The MIMO prefill overhead is visible: at 2048 tokens, MIMO prefill takes 19.44 ms vs. 17.57 ms for SISO — the 2× training slowdown from the chunked algorithm (Section 3.3) manifests in prefill, which uses the same parallel computation as training.

Ablation Studies and Robustness Checks

Core SSM component ablation (Table 5a): At the 440M scale, Mamba-3 SISO with all components achieves 15.72 perplexity on the test set. Ablating both the exponential-trapezoidal discretization (reverting to exponential-Euler) and BC biases yields 16.68 — a degradation of 0.96 perplexity, roughly the gap between Mamba-2 and Mamba-3 SISO at this scale. Adding back exponential-trapezoidal but keeping biases removed yields 16.49 — a 0.19 improvement, showing the discretization alone provides meaningful benefit. Adding biases but keeping exponential-Euler yields... the paper doesn't report this exact combination in Table 5a (which shows "-bias" and "-bias -trap" but not the cross of "+bias, -trap" vs. "+bias, +trap, +conv"), but Table 10b in the bias ablation (discussed below) partially fills this gap. Adding the external short causal convolution to the full Mamba-3 yields 15.85 — slightly worse than without it (15.72), confirming that the implicit convolution from exponential-trapezoidal + BC biases makes the explicit convolution redundant and even slightly harmful.

BC bias parameterization (Table 10a): The bias initialization was ablated across: all-ones (15.72, default), all-zeros (16.57 — much worse), trainable constant initialized to 1.0 (15.80 — slightly worse than the learned per-channel default), uniform [0,1] (15.76 — similar), uniform [-1,1] (16.07 — significantly worse). The takeaway: the biases need to be positive and learnable; zero initialization prevents the model from benefiting from them, and negative initial values are harmful (likely because they create sign instabilities in the state-input path). The relative insensitivity to exact positive initialization (all-ones vs. uniform [0,1]) suggests the model can adapt the bias values during training as long as they start in a reasonable range.

B vs. C bias presence (Table 10b): Testing all four combinations: no biases (16.52 perplexity), B-bias only (16.68 — slightly worse than no biases, consistent with the block-biased finding from Yu and Erichson, 2025, not transferring well to Mamba-3), C-bias only (15.98 — substantial improvement), both B and C biases (15.69 — best, slightly better than the 15.72 reported in Table 5a, suggesting run-to-run variance or slight hyperparameter differences). The synergy between B and C biases is important: B-bias alone doesn't help, C-bias alone helps considerably, but together they provide more than the sum of individual benefits.

Exponential-trapezoidal λ_t parameterization (Table 8 in Appendix A.3): Three configurations were tested at 440M Chinchilla scale: default data-dependent λ_t = σ(u_t) (15.72 perplexity), fixed λ_t = 1/2 (15.76, the classical trapezoidal rule — slightly worse), and λ_t = 1 (15.81 — equivalent to exponential-Euler since the β_t term vanishes). The classical trapezoidal rule's inferiority to the learned data-dependent λ_t confirms Remark 3's claim: the second-order accuracy property is less important for language modeling than the ability to adaptively weight the previous vs. current input based on content. The gap between λ_t = 1 (Euler) and the other two shows that having a 3-term recurrence at all is beneficial, even with fixed λ_t = 1/2, compared to the 2-term Euler version.

Complex-valued SSM capability diagnostic (Table 5b): Discussed in detail above in Section 5.2. The key negative results: Mamba-3 without RoPE achieves 2.27% on parity (random guessing is 50%, but the model consistently fails to learn the pattern — 2.27% may reflect training collapse rather than chance-level performance). Mamba-3 with standard RoPE (Su et al., 2023, using fixed sinusoidal frequencies) achieves 1.56% — also failing completely. The specificity of the benefit to data-dependent RoPE is the critical robustness check.

Retrieval normalization in hybrid models (Table 9): Five normalization configurations tested: no additional norm, post-gate default RMS, pre-gate default RMS, post-gate grouped RMS, and pre-gate grouped RMS. Average LM accuracy is similar across all (55.9–56.5), but retrieval behavior diverges sharply. Pre-gate grouped RMS achieves 100% NIAH-Single-2 at 2048 and 56.8% at 4096 (the best length generalization), while no additional norm achieves only 36.2% and 8.8% respectively. However, pre-gate grouped RMS suffers on FDA (52.4 vs. 65.9) and SWDE (58.6 vs. 58.5, minor). The paper doesn't identify a single "best" configuration — it presents both and lets the use case determine the tradeoff.

Training dynamics (Figure 5 in Appendix E): Validation perplexity curves for the 1.5B models show Mamba-3 SISO and MIMO consistently below Mamba-2 and GDN throughout training, with Mamba-3 MIMO achieving the lowest final perplexity. The gap between Mamba-3 SISO and MIMO is visible from early in training (around step 20,000) and widens steadily. This suggests MIMO's benefit is not just a final-convergence effect — it improves learning dynamics throughout training.

Length extrapolation (Figure 4 in Appendix E): Mamba-3 at 1.5B shows stable perplexity degradation from 2K to 32K context, while Mamba-2 shows a sharper increase. GDN is intermediate. This robustness check confirms that Mamba-3's architectural changes don't inadvertently harm long-context behavior — they improve it.

Critical Assessment

Do the experiments support the claim that Mamba-3 improves over Mamba-2 and other linear models?

Supported for language modeling quality. Table 3 shows consistent improvements across four model scales (180M, 440M, 880M, 1.5B) with monotonic gains as scale increases — this pattern of consistency across scales is more persuasive than a single-scale result would be. The 1.5B Mamba-3 SISO improves average accuracy by 0.7 points over the best non-MIMO baseline (GDN), and MIMO adds another 1.2 points. However, the absolute magnitude of the SISO gain (0.6-0.7 points over GDN) is modest relative to the variance typical in these benchmarks — without confidence intervals, it's difficult to assess whether this would replicate across different random seeds or data orders. The MIMO gain (1.2 points over SISO) is larger and more clearly outside typical noise ranges. A missing experiment: training Mamba-2 or GDN with equivalent parameter-matched expansions (e.g., increasing GDN's head dimension to add FLOPs during decode) would test whether MIMO's benefit is specific to the Mamba-3 structure or generalizable to other linear models. The paper's argument (Section 5.4) that MIMO is natural in the SSM framework but not in associative memory frameworks suggests this transfer might not be straightforward, but the experiment isn't run.

Supported for state-tracking capability. Table 5b provides clear evidence that the data-dependent RoPE mechanism solves tasks Mamba-2 fundamentally cannot. The diagnostic comparison against standard RoPE (which fails) isolates the benefit to data dependence specifically, not to rotation in general. The 100% parity result is binary — the model either learns the task or doesn't — which makes it robust to statistical noise. However, the paper doesn't evaluate state tracking on the pretrained language models — only on small models trained from scratch on these specific synthetic tasks. An open question: does the complex-valued state transition help with state tracking in the context of language modeling, or does it only manifest on isolated synthetic tasks? The language modeling benchmarks don't directly test state tracking, so the practical benefit of the complex-valued recurrence for real-world tasks remains unproven by the experiments in this paper. A missing experiment: probing the pretrained language models on synthetic state-tracking tasks (parity embedded in natural language, counting, or multi-step reasoning) to test whether the complex-valued recurrence provides benefits that transfer to language.

Supported for inference efficiency. Table 6 and Table 7 convincingly demonstrate that Mamba-3 SISO is faster at decode than Mamba-2 and GDN (despite adding the 3-term recurrence and RoPE), and that MIMO adds only ~15% latency for a significant quality improvement. The Pareto analysis in Figure 3 (matching Mamba-2 quality with 2× smaller state size) is a strong practical result because it directly translates to deployment cost savings. However, the kernel implementation is not controlled: Mamba-3 uses newly written CuTe DSL decode kernels, while Mamba-2 uses the original Triton kernels. The speed comparison partially reflects implementation quality, not just algorithmic efficiency. The paper acknowledges this indirectly by releasing the kernels, but a fairer comparison would re-implement all baselines at the same level of optimization, or at minimum verify that the Mamba-2 Triton kernel is competitive with a CuTe reimplementation.

What key experiments are missing?

No combined ablation of the three contributions. The paper ablates BC biases and exponential-trapezoidal discretization (Table 5a), and it ablates the complex-valued recurrence (Table 5b), but it never ablates all three simultaneously — for example, training Mamba-3 with exponential-trapezoidal but without complex-valued recurrence (to measure the isolated contribution of complex values to language modeling), or with complex-valued recurrence but without exponential-trapezoidal. The state-tracking experiments (Table 5b) isolate the complex-valued contribution, but on synthetic tasks only, not on language modeling perplexity. This means the reader cannot determine how much of the language modeling quality gain comes from each component individually.

No long-context language modeling beyond 2K training. The retrieval experiments (Table 4) test length extrapolation up to 4K on synthetic NIAH tasks, and Figure 4 (Appendix E) shows perplexity extrapolation to 32K, but there are no extended-context language modeling benchmarks (e.g., LongBench, Scrolls, or standard long-context QA) that would test whether the length extrapolation advantage translates to real downstream tasks. The hybrid model results with pre-gate grouped RMSNorm are promising for NIAH but suffer on in-context retrieval tasks (FDA drops from 65.9 to 52.4). The practical value of the length-extrapolation advantage is unclear without testing on tasks that require reasoning over long contexts, not just retrieving a single fact.

No scaling beyond 1.5B parameters. The largest model is 1.5B parameters trained on 100B tokens — this is modest by modern standards and well below the scale where emergent capabilities or training instability issues typically appear. The paper claims the three methodological improvements arise from an SSM-centric viewpoint that generalizes, but this claim is untested at scale. The BC normalization is specifically noted to stabilize large-scale runs and enable removal of Mamba-2's post-gate RMSNorm (Section 3.4), but this stability claim is only tested up to 1.5B. Larger-scale experiments (7B, 13B) would be needed to validate the paper's architectural recommendations for production-scale deployment.

No comparison against strong hybrid baselines. The hybrid experiments compare Mamba-3 SISO interleaved with NoPE self-attention against similarly hybridized Mamba-2 and GDN, but not against state-of-the-art production hybrids like Nemotron-H (NVIDIA et al., 2025) or Kimi Linear (Kimi Team et al., 2025) which the paper cites. This is understandable given computational constraints, but it means the paper cannot claim Mamba-3 is competitive with the best production linear models — only that it improves over Mamba-2 and GDN in a controlled comparison.

No training efficiency benchmarks. The paper focuses on inference efficiency but does not report training throughput or memory usage for Mamba-3 vs. baselines, despite MIMO's acknowledged training slowdown (2× for R=4, Section 3.3). Table 6 and 7 are inference-only. A practitioner choosing between Mamba-3 SISO and Mamba-3 MIMO needs to know not just that MIMO provides +1.2 accuracy at equal inference speed, but also what the training cost premium is in wall-clock hours or GPU-hours.

Do the experiments support the claim that these improvements arise specifically from the SSM viewpoint?

This is a methodological claim, not directly tested experimentally. The paper argues in Section 5.4 that the three improvements are natural from the SSM perspective but non-obvious or meaningless from linear attention or test-time training perspectives. This is a philosophical argument, not a falsifiable hypothesis. The experiments demonstrate that the improvements work, but they don't test whether a researcher starting from the linear attention or TTT framework could have discovered equivalent improvements through different reasoning. For example, a linear attention researcher might independently discover that increasing the rank of the key-value outer product improves quality without slowing decode (the MIMO insight) — the paper's claim that this is non-obvious from that framework is untestable without a controlled study of what different research groups actually discover. The paper's strength is in demonstrating that the SSM perspective did lead to these specific improvements; the counterfactual claim about other frameworks is speculative.

What are the genuine weaknesses of the experimental design?

No error bars or statistical testing. Perplexity improvements of 0.1-0.3 points (e.g., Mamba-3 MIMO vs. SISO at 1.5B) are reported without confidence intervals. Given the well-known variance in language model training across random seeds (especially at smaller scales), it's unclear whether these gaps are statistically reliable or within the noise floor. The consistency across four scales mitigates this concern somewhat — a false positive would have to occur at all four scales independently — but proper statistical reporting would strengthen the results.

The 100B token training budget is small relative to model scale. At 1.5B parameters, 100B tokens is only ~67× the parameter count — well below Chinchilla-optimal (~20× for compute-optimal training, but 100B/1.5B ≈ 67× for these specific experiments). The models are undertrained relative to what practitioners would deploy, and the relative ranking of architectures could shift with more training. Figure 5 (Appendix E) shows validation curves that have not fully converged at 90K steps — Mamba-3 MIMO is still descending, suggesting further training could change the relative gaps.

Single dataset for pretraining. All language modeling experiments use FineWeb-Edu. This is a high-quality filtered dataset, but architectural advantages can be dataset-specific — a model that excels on clean educational text might not generalize its advantage to noisier web text, code, or multilingual data. The retrieval tasks provide some domain diversity but are all English-language benchmarks.

MIMO parameter matching is approximate. Reducing the MLP inner dimension to compensate for the MIMO projection parameters ensures equal total parameter count, but it changes the ratio of sequence-mixing to channel-mixing capacity. A MIMO model has more parameters devoted to the SSM (the sequence mixer) and fewer to the MLP (the channel mixer) compared to a SISO model. If sequence mixing is the bottleneck for these benchmarks, this reallocation could be the source of MIMO's gain rather than the MIMO structure itself. A control experiment: keeping total parameters fixed but increasing the SSM parameter count in the SISO model (e.g., by expanding state size and reducing MLP width proportionally) would test whether the benefit comes from MIMO specifically or from devoting more parameters to sequence mixing in general.

The 38% reversion issue from the reference example does not apply here — Mamba-3 does not have a revision model and thus doesn't suffer from correct-to-incorrect reversion. However, Mamba-3 has its own stability concern: training instability when scaling up, mentioned in Section 3.4 as the motivation for BC normalization. The paper reports that BC normalization stabilizes large-scale runs and enables removal of Mamba-2's post-gate RMSNorm in pure models, but provides no quantitative evidence (e.g., training loss spikes, gradient norm plots, or failure rates with and without BCNorm) to support this claim. This is a gap — the stabilization benefit is asserted but not demonstrated.

The "inference-first" narrative is somewhat overstated relative to the experiments. The MIMO section (3.3) provides a clean inference-first motivation, and Figure 3/Table 6 support it. But the exponential-trapezoidal and complex-valued contributions are primarily quality improvements that happen to be inference-cheap — they weren't motivated by an inference bottleneck analysis. The paper's framing as uniformly "inference-first" blurs the distinction between MIMO (which genuinely emerged from thinking about decode hardware utilization) and the other two contributions (which emerged from thinking about SSM dynamics and expressivity theory). This doesn't weaken the empirical results, but it's a rhetorical overclaim that a critical reader should note.

6. Limitations and Trade-offs

The Difficulty Estimation Problem Is Inherited by Any Future Adaptive Deployment

The assumption or constraint. The paper’s core architectural improvements — exponential-trapezoidal discretization, complex-valued state transitions, and MIMO — are presented as static architectural upgrades: you train a Mamba-3 model, and it is better than a Mamba-2 model at the same inference cost. However, the paper’s framing in Section 1 invokes test-time compute scaling as the motivating paradigm shift: "Scaling inference-time compute has emerged as an important driver of LLM performance... The rapid rise of parallel, agentic workflows has only intensified the need for efficient inference." This framing implies a future where inference strategy is adapted per-query (e.g., spending more compute on harder problems, as in the reference example’s compute-optimal framework). The paper does not develop such an adaptive strategy for Mamba-3 — it builds a better static architecture — but the motivating context naturally raises the question: how would Mamba-3’s design interact with adaptive test-time compute allocation?

The specific gap: difficulty estimation is expensive and architecturally entangled. If a deployment wanted to adaptively decide whether to run Mamba-3 SISO (fast, lower quality) or Mamba-3 MIMO (slower prefill, higher quality) based on estimated prompt difficulty, it would need a reliable difficulty estimator. The paper provides no such estimator and does not discuss how one might be built for Mamba-3’s architecture. The state size / MIMO rank cannot be dynamically changed at inference time — they are baked into the model weights — so adaptive allocation would require serving multiple model variants or accepting that the architecture itself determines the fixed quality-speed tradeoff point.

The consequence. The paper’s contribution is a better point on the quality-speed Pareto frontier, but it does not enable the kind of per-query adaptive computation that the introduction’s motivating context envisions. A practitioner deploying Mamba-3 gets a strictly better model than Mamba-2 at equivalent latency, but they cannot dynamically trade speed for quality on a per-prompt basis without serving multiple model copies (SISO and MIMO, or models with different state sizes). This limits the practical applicability of the "inference efficiency" narrative: the architectural improvements make inference cheaper in absolute terms but do not provide a mechanism for spending that saved budget selectively on problems that need it. In contrast, a Transformer with speculative decoding or a model with early-exit capability can adapt computation to input difficulty — Mamba-3 cannot.

What evidence exists in the paper. The paper shows (Table 3, Figure 3) that Mamba-3 SISO and MIMO occupy different points on the quality-latency frontier, but there is no experiment where a single deployment switches between them based on input characteristics. Section 1’s invocation of test-time compute scaling sets up an expectation that the paper’s methods would enable adaptive inference, but Section 3.3 explicitly acknowledges the tradeoff: MIMO "trades off slower training for even stronger modeling power, while maintaining competitive inference efficiency compared to Mamba-2" — the adaptation is architectural at training time, not dynamic at inference time.

Mitigation status. Not addressed and not proposed as future work. The paper does not position itself as solving the adaptive inference problem, so this limitation is more about the gap between the motivating narrative and the actual contribution than a failure of the method itself. A fair characterization: Mamba-3 provides better static efficiency; the adaptive deployment layer remains future work, and it is not obvious how Mamba-3’s fixed-state design would support it.

State-Tracking Gains Are Demonstrated Only on Isolated Synthetic Tasks — Transfer to Language Modeling Is Unmeasured

The assumption or constraint. The complex-valued state transition is the paper’s most theoretically distinctive contribution, with Section 3.2 providing a formal proof that data-dependent rotational dynamics enable state tracking (Proposition 2) and an efficient implementation via the RoPE trick (Proposition 3). The synthetic experiments in Table 5b validate the theory: Mamba-3 achieves 100% accuracy on parity and 98.51% on modular arithmetic without brackets, tasks that Mamba-2 fails at random-guessing levels (0.90%, 47.81%). However, these experiments use small models trained from scratch on the synthetic tasks themselves — they do not test whether the complex-valued recurrence provides state-tracking benefits that transfer to language modeling.

The consequence. A practitioner reading this paper might reasonably conclude that Mamba-3’s complex-valued recurrence improves language model performance by enabling better tracking of syntactic structure, entity states, or multi-step reasoning in natural language. The paper provides no evidence for this. All language modeling results (Table 3, Figure 3, Figure 5) compare the full Mamba-3 architecture (which includes complex-valued transitions) against baselines, but there is no ablation that isolates the contribution of the complex-valued recurrence to language modeling perplexity or downstream accuracy. The state-tracking capability is proven on synthetic tasks; whether it improves language modeling remains an open empirical question. The paper’s language modeling gains over Mamba-2 could be entirely due to the exponential-trapezoidal discretization and BC biases, with the complex-valued recurrence contributing nothing at all to language tasks — the experiments cannot distinguish these possibilities.

This is particularly consequential because prior work (cited in Section 5.2) found that "complex numbers have been empirically found to be unhelpful for language modeling, and hence were phased out in Mamba-1 and successors." The paper acknowledges this history but argues that "Mamba-3 represents the first modern recurrent model with complex-valued state transitions, which were introduced for specific purposes of increasing expressivity and state-tracking ability." The "specific purposes" framing suggests the complex recurrence is solving a known expressivity gap — but if that gap doesn’t affect language modeling performance on standard benchmarks, the theoretical contribution may have limited practical impact for language applications.

What evidence exists in the paper. Only Table 5b — synthetic tasks with models trained specifically for those tasks. The paper never reports a "Mamba-3 without complex-valued recurrence" ablation on language modeling perplexity. The BC bias and exponential-trapezoidal ablation (Table 5a) compares full Mamba-3 against versions with those components removed, but all versions presumably include the complex-valued recurrence (the ablation is described as targeting "core SSM components" without mentioning removal of the complex extension). This means the language modeling benchmarks cannot attribute any fraction of the gain to the complex-valued recurrence specifically.

Mitigation status. Not addressed. The paper presents the complex-valued recurrence as a theoretically motivated improvement and validates it on synthetic state-tracking tasks, but does not close the loop by measuring its contribution to the primary language modeling results. This is a significant gap because the language modeling experiments are where the paper claims practical superiority over baselines.

All Experiments Use a Single Model Family and a Single Pretraining Dataset — Generalizability Is Unestablished

The assumption or constraint. Every language modeling result in the paper (Table 3, Table 4, Figure 3, Figure 4, Figure 5, Table 5a) uses the same base architecture family (Llama-style backbone with Mamba-3 or baseline sequence mixers) pretrained on exactly one dataset: 100B tokens of FineWeb-Edu. The paper cites this explicitly (Section 4.1): "All models are pretrained with 100B tokens of the FineWeb-Edu dataset." The largest model scale is 1.5B parameters — modest by current standards and well below the scale where architectural differences might interact with emergent capabilities.

The consequence. A practitioner considering Mamba-3 for a production deployment needs to know whether the architectural advantages transfer to: (1) different data distributions (code, multilingual text, domain-specific corpora); (2) larger model scales (7B, 13B, 70B+); (3) different training recipes (longer training, different optimizers, different context lengths). The paper provides no evidence on any of these axes. The finding that BC biases and exponential-trapezoidal discretization make the short causal convolution unnecessary (Table 5a, 15.72 vs. 15.85 perplexity) might be specific to FineWeb-Edu’s text characteristics — a dataset with different local statistics (e.g., code with long-range syntactic dependencies) might benefit from the explicit convolution that the paper recommends removing. Similarly, the claim that BC normalization stabilizes large-scale training and enables removal of Mamba-2’s post-gate RMSNorm (Section 3.4) is untested beyond 1.5B — training instability often manifests only at larger scales (the "small-scale proxies" problem that Wortsman et al., 2023, which the paper cites in the context of QKNorm, was specifically about).

The hybrid model results (Table 4, Table 9) add some data diversity through retrieval tasks (SWDE, FDA, SQuAD, etc.), but these are evaluated zero-shot on the FineWeb-Edu-pretrained models — they test transfer, not training on diverse data. A model pretrained primarily on educational web text may have limited exposure to the structured table formats of SWDE or the medical terminology of FDA, making it unclear whether retrieval deficits reflect architecture limitations or data distribution mismatch.

What evidence exists in the paper. The consistency of results across four model scales (180M, 440M, 880M, 1.5B) provides some evidence against scale-specific effects at these sizes, but 1.5B is still far from production scale. All scales use the same dataset, same tokenizer (Llama-3.1), same training recipe, and same hyperparameter configuration. The paper states that "all models at each scale follow the same procedure and were trained with bfloat16" (Appendix D), which controls for training variance but does not test robustness to recipe changes.

Mitigation status. Not explicitly addressed as a limitation. The paper states that Mamba-2 and Gated DeltaNet "have recently been incorporated into large-scale hybrid models that match the performance of pure Transformer alternatives" (Section 1), implying that the Mamba lineage scales effectively, but this is about Mamba-2, not Mamba-3. The three methodological innovations in Mamba-3 — particularly the exponential-trapezoidal recurrence (which adds a term that could affect gradient flow) and the complex-valued recurrence (which introduces oscillatory dynamics that might interact differently with large-scale optimization) — have unknown scaling properties.

MIMO’s Training Slowdown and Parameter Reallocation Are Controlled but Not Fully Characterized

The assumption or constraint. The MIMO formulation is motivated entirely by inference efficiency: increasing decode FLOPs without increasing state size to exploit idle GPU compute during memory-bound autoregressive generation (Section 3.3). The paper acknowledges a training cost: "the training algorithms for MIMO SSMs... have been typically developed for SISO models" and the chunked training algorithm has "a slowdown of 2× when R = 4, as compute latency can be parallelized with memory movement" (Section 3.3). To keep total parameters comparable to SISO baselines, the paper reduces the MLP inner dimension (e.g., from 4096 to 3824 at 1.5B scale, a 6.6% reduction — Appendix C).

The consequence. Two unquantified factors affect the fairness and interpretability of the SISO-to-MIMO comparison:

First, the training cost premium is not reported in wall-clock time or GPU-hours. The paper states a 2× slowdown for MIMO with R=4 relative to SISO (Table 7 shows this in prefill latency — 19.44 ms vs. 17.57 ms at 2048 tokens — but does not report total training time). A practitioner considering MIMO faces a decision: is +1.2 points of average downstream accuracy worth approximately 2× the training cost? The paper provides no data to inform this decision. The inference-first framing (Section 1: "inference efficiency a central focus of model design") emphasizes deployment benefits but obscures the training cost. For organizations that train models once and serve them at high volume, a 2× training cost increase may be easily amortized; for organizations that retrain frequently or have limited compute budgets, it may be prohibitive.

Second, the MLP width reduction confounds the MIMO comparison. Reducing the MLP inner dimension changes the ratio of sequence-mixing capacity (SSM parameters) to channel-mixing capacity (MLP parameters). The MIMO variant devotes a larger fraction of its total parameters to the SSM and a smaller fraction to the MLP compared to the SISO variant. If SSM capacity is the bottleneck for these language modeling benchmarks, the MIMO gain might be partly or entirely attributable to this parameter reallocation rather than to the MIMO structure itself. A control experiment — increasing SSM parameter count in the SISO model while proportionally reducing MLP width to match the MIMO allocation — would distinguish these explanations, but is not reported.

What evidence exists in the paper. Table 3 shows MIMO accuracy gains at all scales; Table 6 shows decode latency (MIMO ~15% slower than SISO, still faster than Mamba-2); Table 7 shows prefill latency (MIMO ~15% slower than SISO at 2048 tokens, consistent with the "2×" training claim since prefill uses the parallel training computation). The training time in GPU-hours is not reported anywhere. The MLP width reduction is documented in Appendix C but its effect is not ablated.

Mitigation status. The paper is transparent about the MLP width reduction but does not control for the parameter allocation confound. The training slowdown is acknowledged qualitatively but never quantified in absolute terms that would enable a cost-benefit calculation. This is a significant gap for a paper whose primary claimed contribution is advancing the "performance-efficiency Pareto frontier" — training efficiency is part of total cost of ownership, and ignoring it makes the Pareto analysis incomplete.

The Removal of the Short Convolution Is Not Stress-Tested Beyond FineWeb-Edu Perplexity

The assumption or constraint. One of the paper’s most architecturally consequential claims is that the combination of exponential-trapezoidal discretization and BC biases eliminates the need for the explicit short causal convolution that has been "empirically identified as an important component in many performant linear models" (Section 4.1, citing Allen-Zhu, 2025; Gu and Dao, 2024; Yang et al., 2025a). The supporting evidence is a single ablation at 440M scale (Table 5a): Mamba-3 without the convolution achieves 15.72 perplexity, and Mamba-3 with the convolution achieves 15.85 — slightly worse. Section 3.1.2 provides a theoretical rationale: the exponential-trapezoidal recurrence induces an implicit convolution on the state-input, and BC biases add a data-independent convolutional component. Together, these supposedly make the external convolution redundant.

The consequence. The external short convolution has been a nearly universal component of performant recurrent language models. Removing it simplifies the architecture and reduces parameter count, which is a genuine contribution if it generalizes. But the evidence for this claim is thin: one perplexity number at one scale on one dataset. Short convolutions provide local temporal smoothing that may be especially important for: (1) modeling tasks that depend on precise local token order (e.g., code, where character-level syntax matters); (2) handling noisy or irregularly spaced inputs; (3) compensating for discretization errors at very long context lengths; or (4) stabilizing training at larger model scales. None of these scenarios is tested. The paper’s retrieval results (Table 4) show Mamba-3 struggling on semi-structured and unstructured information extraction (SWDE: 28.5 vs. Transformer 48.9; FDA: 23.4 vs. Transformer 58.4) — while these deficits are primarily attributed to the fixed-state bottleneck, the missing convolution could be a contributing factor, since local pattern matching (e.g., identifying field-value pairs in semi-structured text) is exactly what short convolutions excel at.

The theoretical argument that the exponential-trapezoidal convolution replaces the external one (Remark 4) relies on a conceptual equivalence — both are convolutions — but they operate on different signals (state-input vs. raw token embeddings) with different receptive fields and different parameterizations. The external convolution in Mamba-2 is a depthwise convolution applied to the token embedding before the SSM; the exponential-trapezoidal convolution is applied to $\mathbf{B}_t x_t$ within the recurrence. These are not functionally equivalent, and the ablation in Table 5a tests them jointly only on perplexity, not on tasks where local feature extraction is critical.

What evidence exists in the paper. Table 5a, one number. The paper does not ablate the convolution’s presence/absence on downstream tasks, on longer contexts, on different data distributions, or at larger scales. The retrieval results (Table 4) are from the full Mamba-3 architecture without convolution, but there is no "Mamba-3 + convolution" retrieval comparison to isolate whether the missing convolution contributes to the retrieval deficits.

Mitigation status. Not addressed beyond the single perplexity ablation. The paper treats the convolution removal as a settled finding, integrating it into the final architecture (Figure 2: "Mamba-3 Block" has no convolution, while "Mamba-2 Block" shows it explicitly). Given the convolution’s ubiquity in the literature and the paper’s own limited evidence, this is an under-validated design decision that practitioners should treat as provisional.

Hybrid Model Normalization Sensitivity Is Documented but Unresolved — There Is No "Best" Configuration

The assumption or constraint. Section 4.1.2 acknowledges that pure linear models struggle with certain retrieval tasks, and proposes hybrid models (interleaving Mamba-3 with self-attention at a 5:1 ratio) as the practical deployment path. However, hybrid model behavior turns out to be highly sensitive to normalization configuration in ways the paper does not resolve. Table 9 (Appendix E) tests five normalization variants — no additional norm, post-gate default RMS, pre-gate default RMS, post-gate grouped RMS, and pre-gate grouped RMS — on the 1.5B hybrid Mamba-3 SISO model. The results reveal a fundamental tradeoff with no uniformly best choice:

No additional norm achieves the best in-context retrieval on real-world tasks (SWDE: 58.5, FDA: 65.9, SQuAD: 47.0) but catastrophic length generalization on NIAH (NIAH-Single-2 at 2048: 36.2%, at 4096: 9.4%).
Pre-gate grouped RMS achieves near-perfect length generalization (NIAH-Single-2 at 2048: 100%, at 4096: 96.0%) but substantially degraded FDA performance (52.4 vs. 65.9) and SQuAD (47.3 vs. 47.0, minor).
Pre-gate default RMS balances these but does not match the best of either extreme (NIAH-Single-2 at 4096: 90.2%, FDA: 54.4).

The consequence. A practitioner building a hybrid Mamba-3 model must choose a normalization configuration without clear guidance. If the application requires long-context retrieval (e.g., document QA, conversation history), pre-gate grouped RMS is necessary. If the application primarily involves structured information extraction at training-length contexts (e.g., entity extraction from fixed-format documents), no additional norm is superior. The paper states this explicitly: "the ideal norm type (grouped vs default) and its placement (pre- vs post-gate) is still unclear due to competing tradeoffs" (Section 4.1.2). This is honest, but it means the practitioner cannot simply "use Mamba-3 hybrid" — they must run their own sensitivity analysis to determine which configuration works for their specific task distribution.

More concerning, the paper does not provide a mechanistic explanation for why normalization configuration has such dramatic effects on length generalization. The post-gate RMSNorm was originally introduced in Mamba-2 for training stability; the paper removes it in pure Mamba-3 (Section 3.4: "BCNorm is also able to stabilize large-scale runs, resulting in the removal of the post-gate RMSNorm layer in our pure Mamba-3 models") but finds it "crucial for long-context extrapolation" in hybrid models. This suggests a complex interaction between the Mamba-3 layer’s internal dynamics and the attention layer’s signal characteristics — an interaction that is observed but not understood.

What evidence exists in the paper. Table 9 is thorough in its sweep of configurations and tasks. The pattern is clear: normalization placement and type create a tradeoff between in-context accuracy and out-of-distribution length generalization. The paper cites Cabannes et al. (2025) as having noted that "hybrid models and their exact characteristics and dynamics are complex and oftentimes unintuitive" (Section 4.1.2), framing the sensitivity as a known challenge rather than a Mamba-3-specific defect.

Mitigation status. Partially addressed through documentation. The paper explicitly presents both strong configurations and acknowledges the unresolved tradeoff, which is more than many papers do. However, the lack of a mechanistic understanding means this limitation cannot be engineered around — it can only be empirically measured for each deployment scenario. This substantially increases the practical cost of adopting Mamba-3 in hybrid architectures.

7. Implications and Future Directions

How This Work Changes the Landscape

Mamba-3 is not a paradigm shift in the sense of introducing a new model class — it remains firmly within the structured state space model lineage — but it is a methodological reframing of how to design sequence model architectures. Its most important contribution is the demonstration that a principled, mathematically grounded approach to architecture improvement — starting from the continuous-time SSM viewpoint and progressively upgrading discretization accuracy, state expressivity, and input-output structure — systematically shifts the quality-efficiency Pareto frontier without requiring ad-hoc architectural tricks or massive empirical search. This stands in contrast to how many recent linear model improvements were discovered: through empirical trial-and-error (trying different gating mechanisms, convolution widths, normalization placements) rather than through theoretical derivation.

The paper's most field-shifting implication is the argument — made implicitly through its methodology and explicitly in Section 5.4 — that the theoretical framework you adopt shapes which architectural extensions you can discover. The three Mamba-3 innovations all arise naturally from the SSM perspective but would be non-obvious or even incoherent from the linear attention or test-time training perspectives. The exponential-trapezoidal discretization falls out of numerical integration theory applied to a continuous ODE; the complex-valued recurrence is a natural extension of dynamical systems to rotational state dynamics; MIMO is a classical control theory concept. If this thesis holds — and the paper provides compelling existence proof but not a controlled test — then it implies that the SSM framework is not just one of several equivalent ways to describe recurrent models, but a generative design toolkit with richer extensibility than its alternatives. This shifts research attention toward theoretical foundations: rather than treating mathematical frameworks as interchangeable descriptions of the same computations, the paper suggests they have different "forward reach" — different capacities to suggest novel and productive architectural modifications.

The paper also resolves a prior contradiction in the SSM literature about complex-valued states. Earlier SSMs (S4, S4D) used complex numbers extensively and were found empirically unhelpful for language modeling, leading to their removal in Mamba-1 and successors. Mamba-3 reintroduces complex-valued transitions and demonstrates they are useful — but for a specific purpose (state-tracking capability via data-dependent rotations) that was not the purpose they served in earlier models (efficient representation of normal-plus-low-rank matrices for LTI systems). The resolution: complex values are not categorically good or bad for language; their utility depends on how they are used — as fixed, data-independent frequency encodings (ineffective) versus as data-dependent rotational dynamics grounded in expressivity theory (effective). This reframes the research question from "should SSMs use complex numbers?" to "what specific computational role do complex numbers play in the architecture, and is that role useful for the target task?"

The paper also changes the default expectation around short causal convolutions in recurrent models. These convolutions have been a nearly universal architectural component — Mamba-1, Mamba-2, Gated DeltaNet, H3, and many others all include them, and theoretical work (Allen-Zhu, 2025) has argued for their necessity. Mamba-3 shows they can be removed with a performance improvement if the core recurrence is made sufficiently expressive through better discretization and learned biases. This doesn't prove convolutions are unnecessary in general, but it shifts the burden of proof: future recurrent architectures should justify their external convolution rather than including it by default. The implicit convolution induced by the exponential-trapezoidal discretization (on the state-input, with data-dependent weights, inside the recurrence) is a more parsimonious mechanism that the paper argues achieves the same functional role.

The inference-first design philosophy — particularly the MIMO formulation — represents a conceptual shift in how to evaluate architectural modifications. The standard approach in deep learning architecture design is to evaluate changes by their training-cost-to-quality-improvement ratio: does adding FLOPs or parameters at training time yield proportional perplexity or accuracy gains? Mamba-3's MIMO turns this logic on its head: MIMO makes training more expensive (2× slower for R=4) while delivering a modest quality improvement (+1.2 points accuracy at 1.5B), which would be a negative result under the training-centric evaluation. But the evaluation metric that matters is decode-latency-to-quality, not training-FLOPs-to-quality, and on that metric MIMO is a clear win because the extra decode FLOPs are absorbed by idle compute during the memory-bound generation step. This suggests that the field should report both training cost and inference cost when introducing architectural modifications, with the relative weighting depending on the deployment scenario (high-volume serving prioritizes inference cost; one-time training for many users amortizes training cost). Currently, most architecture papers report only training perplexity curves and perhaps inference latency as an afterthought; Mamba-3 argues — through its design process rather than through explicit methodological prescription — that inference characteristics should be the primary axis of comparison.

Finally, the paper's diagnostic approach to capability gaps is worth noting as a methodological contribution. Rather than simply claiming that Mamba-3 improves language modeling and leaving it at that, the paper identifies a specific theoretical limitation of prior architectures (the inability of real-valued scalar transitions to represent rotational dynamics, Theorem 1 of Grazzi et al., 2024), proposes a specific mechanism to address it (data-dependent complex rotations via the RoPE trick), and validates that the mechanism actually solves the diagnosed problem on synthetic tasks (Table 5b: 100% parity for Mamba-3 vs. 0.90% for Mamba-2). This "diagnose → propose mechanism → validate mechanism on controlled task → integrate into full system" pipeline is more rigorous than the common approach of proposing architectural changes and evaluating them holistically on benchmarks without isolating whether the intended mechanism actually functions. It doesn't prove the mechanism helps on real-world tasks (the paper does not ablate the complex recurrence's contribution to language modeling), but it provides stronger evidence that the theoretical reasoning is sound.

Follow-Up Research This Work Enables

Quantify the contribution of data-dependent RoPE to language modeling quality through controlled ablation. The paper proves that the complex-valued recurrence enables state tracking on synthetic tasks (Table 5b) but never measures its isolated contribution to language modeling perplexity or downstream accuracy. A direct follow-up would train Mamba-3 models at the same scales reported in Table 3, with and without the complex-valued recurrence (keeping all other components — exponential-trapezoidal discretization, BC biases, BC normalization — identical), and measure the perplexity gap at each scale. If the gap is near zero, the complex recurrence is solving a capability problem that does not manifest on standard language benchmarks (suggesting either that the benchmarks don't test state tracking, or that language doesn't require it at these scales). If the gap is measurable, it quantifies the practical value of the theoretical expressivity improvement. A stronger version of this experiment would test on language tasks that plausibly require state tracking: multi-step reasoning (e.g., GSM8K, StrategyQA), coreference resolution across long distances, or synthetic language tasks where the model must track a hidden variable (e.g., "the character is now in location X" through a narrative). This would test whether the mechanism that solves isolated parity and modular arithmetic transfers to embedded state tracking in natural language.

Develop continuous-time interpretation of linear attention and test-time training models to enable cross-framework discretization improvements. Section 5.4 argues that the exponential-trapezoidal discretization "falls out of an improved discretization method" for continuous-time SSMs but "as associative memory methods do not use discretization, it is not obvious how to interpret a 3-term recurrence such as exponential-trapezoidal under alternate viewpoints." A natural follow-up is to ask whether linear attention or test-time training models can be reinterpreted as discretizations of continuous dynamical systems, and if so, whether higher-order discretization methods (like exponential-trapezoidal) can be ported to those frameworks. Concretely: can the delta-rule update in Gated DeltaNet be derived as the discretization of a continuous ODE? If so, does applying a trapezoidal correction (adding a term that depends on the previous key-value pair, weighted by a learned mixing coefficient) improve GDN's performance in the same way it improves Mamba-2's? This would test the paper's thesis that the SSM viewpoint provides unique extensibility — if equivalent improvements can be discovered from other frameworks once the connection is made, the thesis is weakened; if the improvements resist porting, the thesis is strengthened.

Characterize the scaling behavior of the exponential-trapezoidal + BC bias combination on diverse data modalities and at larger scales. The paper's claim that the short causal convolution can be removed rests on a single 440M model trained on FineWeb-Edu (Table 5a: 15.72 without convolution vs. 15.85 with). A follow-up should test this at 7B+ parameters on at least three diverse datasets: (1) code (e.g., The Stack), where precise local syntax might benefit from explicit convolutions; (2) multilingual text, where character-level patterns differ from English; and (3) a long-context dataset (e.g., 8K+ tokens), where local smoothing might help with positional generalization. For each, train matched models with and without the external convolution, and report both perplexity and a downstream task metric. A negative result (convolution removal hurts on code or long contexts) would establish boundary conditions for the architectural simplification; a positive result would make the removal a reliable default for Mamba-3 architectures. Additionally, the scaling behavior of the exponential-trapezoidal λ_t parameterization should be studied: at larger scales, does the learned data-dependent λ_t naturally converge toward 1/2 (the classical trapezoidal rule, which would suggest the second-order accuracy property becomes beneficial with more training), or does it remain far from 1/2 (suggesting the data-dependent mixing is genuinely architecturally useful and the numerical accuracy interpretation is secondary)?

Combine MIMO with adaptive state-size or rank selection at inference time for difficulty-aware efficiency. The paper shows that Mamba-3 SISO and MIMO occupy different points on the quality-latency Pareto frontier (Table 3, Figure 3), but the choice between them must be made at training time. A natural extension is to develop a single model that can dynamically adjust its effective rank or state size at inference time based on input difficulty. One approach: train a Mamba-3 MIMO model with R=4, but add a lightweight difficulty predictor (a small MLP or linear head that takes the token embedding or the SSM's Δ_t values as input) that outputs a scalar indicating how much of the MIMO rank to use. At inference time, easy tokens could use R=1 (reducing to SISO computation), while hard tokens could use R=4. The challenge is that the state dimensionality must remain compatible — one approach is to always maintain the full MIMO state but mask or down-weight the additional rank dimensions when the predictor indicates they're unnecessary. This would enable per-token or per-sequence adaptive computation without serving multiple model copies. The difficulty estimator could be trained with a regularization term that penalizes using high rank, creating a natural speed-quality tradeoff controlled by the regularization coefficient. This directly addresses the gap between the paper's motivating context (test-time compute scaling) and its actual contribution (static architecture improvements).

Stress-test the complex-valued recurrence on adversarial or pathological input sequences that require precise state tracking. The synthetic experiments in Table 5b use clean, well-structured formal language tasks (parity on binary sequences, modular arithmetic with and without brackets). A more diagnostic follow-up would test Mamba-3 on adversarially constructed sequences that are designed to break state-tracking capabilities: (1) parity with long runs of zeros between relevant bits (testing whether the decay dynamics cause the model to "forget" the tracked state during long gaps); (2) nested or hierarchical state-tracking tasks (e.g., tracking multiple independent binary variables simultaneously, requiring the model to maintain several orthogonal rotational dynamics); (3) state-tracking under distractors (e.g., parity where the input sequence is interleaved with irrelevant tokens that the model must ignore). These experiments would characterize the practical limits of the complex-valued recurrence's state-tracking capability — the paper proves it can represent rotational dynamics in principle, but doesn't test how robust that representation is to realistic interference. A particularly informative negative result would be if the model fails on hierarchical tracking, which would suggest that the fixed number of rotational channels (determined by the state size N) imposes a hard ceiling on how many independent state variables can be tracked.

Develop hardware-aware MIMO kernels that fully close the gap between theoretical arithmetic intensity and realized throughput. Table 6 shows Mamba-3 MIMO (R=4) achieving 0.179 ms decode latency vs. 0.156 ms for SISO at BF16, dstate=128 — a 15% increase for a 4× increase in decode FLOPs. This is good but leaves room for improvement: the theoretical arithmetic intensity scaling (Table 2) suggests the compute overhead should be almost entirely hidden in the memory-bound regime. The paper's CuTe DSL decode kernels are described as "competitive" but not necessarily optimal. A kernel engineering follow-up would profile exactly where the MIMO latency overhead originates (is it the additional loads for the expanded B and C projections? the extra computation in the gate? synchronization overhead from the increased chunk count?) and optimize accordingly. The goal would be to reduce MIMO decode latency to within 5% of SISO, at which point MIMO becomes a strictly dominant choice — better quality at negligible latency cost. This is a practical engineering contribution that would directly impact deployment adoption.

Practical Applications and Downstream Use Cases

High-volume conversational AI serving. A deployment serving millions of conversations per day — chatbots, customer support, voice assistants — is the canonical inference-first scenario. The dominant cost is autoregressive token generation, and latency directly impacts user experience. Mamba-3 SISO provides a direct upgrade path: at BF16, dstate=128, it decodes at 0.156 ms/token vs. Mamba-2's 0.203 ms/token (Table 6) — a 23% latency reduction — while achieving 0.7 points higher average accuracy at 1.5B scale (Table 3: 56.4 vs. 55.7) and better perplexity (10.35 vs. 10.47). For a deployment currently using Mamba-2, switching to Mamba-3 SISO is strictly better on both speed and quality. For a deployment using Mamba-3 MIMO (R=4), the tradeoff is +1.2 points accuracy (57.6 vs. 56.4) for +15% latency (0.179 vs. 0.156 ms) — still faster than Mamba-2 while being substantially more accurate. The total cost of ownership improvement (better accuracy per millisecond of latency, and better accuracy per GPU-hour of serving) makes Mamba-3 directly actionable for production systems. The length extrapolation advantage (Figure 4, 32K context at ~10.7 perplexity for Mamba-3 vs. >10.9 for Mamba-2) additionally means conversations can extend longer before quality degrades, reducing the need for context windowing tricks.

On-device or edge deployment with strict latency budgets. Edge devices (phones, laptops, embedded systems) have severely constrained compute and memory, and the fixed memory footprint of recurrent models (no KV cache) is a critical advantage over Transformers. Mamba-3's Pareto shift (Figure 3: matching Mamba-2's quality with half the state size) directly translates to edge deployment: state size 64 can replace state size 128, roughly halving the memory bandwidth requirement for the state update — which is often the bottleneck on low-power hardware. The removal of the external short causal convolution (Table 5a) further simplifies the architecture, reducing the number of kernel launches and the model's memory footprint. A concrete deployment: a Mamba-3 model with state size 64 and SISO, running on a phone-class GPU or NPU, could achieve quality comparable to a Mamba-2 model with state size 128 while requiring substantially less memory bandwidth, enabling real-time text generation on hardware that would otherwise be too slow for competitive quality. The complex-valued recurrence adds negligible compute (the RoPE trick reduces it to cheap 2D rotations) while enabling state-tracking capabilities that might matter for tasks like following cooking recipes or navigating multi-step instructions — common on-device use cases.

Large-scale batch inference for data processing pipelines. Organizations that run batch inference over large document collections — for summarization, entity extraction, classification, or data labeling — care primarily about throughput (tokens processed per GPU-hour) and total cost. The prefill+decode latencies in Table 7 show that Mamba-3 SISO processes 2048-token sequences in 17.57 ms (vs. 20.37 ms for the Transformer baseline via vLLM), and the gap widens dramatically at longer sequences (16384 tokens: 140.61 ms vs. 976.50 ms — a ~7× speedup). For a pipeline processing millions of 16K-token documents, this difference is transformational: a single GPU can process ~7× more documents per hour with Mamba-3 than with a Transformer, directly reducing hardware costs and energy consumption. Mamba-3 MIMO at the same sequence length takes 151.81 ms — still a ~6.4× speedup over Transformer. The retrieval results (Table 4) suggest caution for extraction-heavy pipelines (pure Mamba-3 struggles on SWDE and FDA), but for summarization or classification tasks where local context suffices, the throughput advantage is compelling. The hybrid model option (5:1 Mamba-3 to attention) provides a middle ground: better retrieval at the cost of some speed, configurable based on the task's dependence on precise information extraction.

Self-improvement and synthetic data generation loops. The paper's observation that Mamba-3 demonstrates better length generalization (Figure 4, Table 4 NIAH results) and competitive language modeling quality at reduced inference cost makes it attractive for pipelines that use LLMs to generate training data for themselves (STaR, ReST^EM, rejection sampling). In these loops, a model generates many candidate outputs, which are then filtered by a verifier or used to fine-tune an improved model. The generation step dominates compute cost, and Mamba-3 SISO's combination of low latency and competitive quality (especially relative to larger Transformer models that might be used in the same role) could reduce the cost per generated token by 2-5× while maintaining output quality sufficient for training data. The length extrapolation advantage means the model can be trained on shorter contexts (cheaper) and deployed to generate longer contexts during data generation (more useful training examples). A specific pipeline: train a Mamba-3 1.5B model on 2K-context FineWeb-Edu, then use it to generate 8K-context reasoning chains (leveraging the length generalization shown in Figure 4) that are filtered and used to train a larger model. The state-tracking capability (Table 5b) might improve the quality of generated multi-step reasoning — the model can track intermediate conclusions more reliably than Mamba-2 — though this is speculative without the language-modeling ablation of the complex-valued recurrence.