Test-Time Training with KV Binding Is Secretly Linear Attention

🎯 Pitch

Replacing gradient descent with gradient ascent in test-time training (TTT) models preserves or even improves performance—a finding that overturns the prevailing view of TTT as online memorization. The authors prove that TTT is mathematically equivalent to a learned form of linear attention, explaining these paradoxes and enabling massive speedups with minimal accuracy loss.

1. Executive Summary

This paper analyzes test-time training with KV binding, challenging the prevailing interpretation of TTT as an online meta-learning mechanism that memorizes key–value associations at test time. Through systematic experiments on language modeling with LaCT-LLM (760M parameters), novel view synthesis with LaCT-NVS, and image classification with ViTTT-B, the authors identify four empirical anomalies—including the observation that replacing gradient descent with gradient ascent preserves or even improves performance, and that substituting queries with keys causes negligible degradation—that fundamentally contradict a memorization-based account. The paper provides an alternative formulation, showing analytically that a broad class of TTT architectures can be exactly rewritten as a learned linear attention operator (with effective queries, keys, and values emerging from the inner-loop update dynamics rather than from similarity-based retrieval), which resolves all observed paradoxes and enables a progressive reduction of complex TTT variants to standard linear attention with only minor performance loss (+0.4 perplexity on language modeling, −0.2 dB PSNR on novel view synthesis) while yielding up to 4.0× inference throughput improvement through parallelization. The analysis establishes that TTT functions as a structured feature mixer rather than a test-time memory system, though the reduction to linear attention requires the inner-loop final layer to be linear and bias-free—a boundary condition that identifies when the proposed parallel formulation is applicable.

2. Context and Motivation

The Prevailing Interpretation of TTT as Memorization

Test-time training (TTT) has recently emerged as a powerful architectural primitive for sequence modeling, offering linear-time computation and constant memory usage during autoregressive inference—properties that make it an attractive alternative to standard softmax attention in transformers. Among TTT variants, methods that optimize a self-supervised key-value binding objective in an inner loop (referred to as TTT-KVB by Tandon et al., 2025) have gained particular traction. The inner-loop update is conceptually simple: for each token, the model takes the key $k$ as input, uses the value $v$ as a regression target, performs one or more steps of gradient descent to minimize a reconstruction loss (e.g., $\mathcal{L} = \|f_\theta(k) - v\|^2$ or a dot-product variant), and then processes the query $q$ through the updated function $f_\theta$ to produce the output.

The dominant interpretation of this mechanism—found across essentially all prior TTT-KVB literature—casts it as online meta-learning or test-time memorization (Finn et al., 2017; Metz et al., 2018; Sun et al., 2025). Under this view, the inner loop "stores" observed key–value pairs into a parametric function (typically an MLP), and the subsequent query step "retrieves" this stored information. This storage-and-retrieval framing has been the primary conceptual driver of architectural design in the TTT literature. Researchers motivated by this perspective have:

Increased inner-loop complexity: Deploying multi-layer MLPs instead of single linear layers to improve memorization fidelity (Han et al., 2025; Behrouz et al., 2024).
Adopted sophisticated optimizers: Incorporating momentum, per-token learnable learning rates, and gradient orthogonalization (e.g., Muon in LaCT; Zhang et al., 2025) to achieve better convergence on the inner-loop objective.
Added normalization schemes: Applying weight normalization after each inner-loop update to stabilize the stored representations (Zhang et al., 2025).
Explored alternative regression targets: Investigating different key-value binding losses to enhance the quality of memorized associations (Han et al., 2025; Behrouz et al., 2025b).

Each of these design choices was explicitly justified by the goal of improving TTT's ability to faithfully memorize key–value mappings at test time. The storage-and-retrieval interpretation has been so influential that TTT architectures are often described with language borrowed directly from memory systems: fast weights "store" information, the inner loop "memorizes" context, and queries "retrieve" the stored content.

Why This Interpretation Matters—and What's at Stake

The memorization interpretation is not merely a philosophical stance; it has direct and consequential implications for how the field develops TTT architectures:

Architectural complexity increases. If TTT genuinely functions as a memory system, then improving memory fidelity—through deeper inner-loop networks, more powerful optimizers, or better normalization—should improve task performance. This logic has driven a steady increase in the complexity of TTT designs, making them harder to implement, analyze, and optimize for hardware efficiency.

Recurrent implementations are the default. A storage-and-retrieval mechanism is inherently sequential: each new observation updates the memory, and retrieval can only occur after the relevant information has been stored. Consequently, TTT implementations are universally recurrent, processing tokens one at a time in strict sequence. This prevents the kind of parallelization that has made standard attention so computationally efficient on modern hardware.

The design space is siloed from the linear attention literature. If TTT operates fundamentally differently from attention—using gradient-based learning rather than similarity-based retrieval—then insights from the extensive literature on linear attention variants (Katharopoulos et al., 2020; Schlag et al., 2021; Gu and Dao, 2024) are largely irrelevant to TTT design. This conceptual separation has prevented cross-pollination between two research communities that, as this paper demonstrates, are working on mathematically equivalent mechanisms.

Empirical anomalies go unexplained. As this paper documents in Section 4, TTT models exhibit puzzling behaviors that make no sense under a memorization framework—yet these anomalies had not been systematically identified or analyzed prior to this work. The absence of a coherent alternative interpretation meant that contradictory evidence was either overlooked or dismissed rather than investigated.

Resolving whether TTT truly functions as memorization is therefore not just an academic exercise in interpretation. The answer determines whether the field should continue investing in increasingly complex inner-loop optimizers and deeper MLPs (the natural trajectory under the memorization view), or whether a fundamentally different design philosophy—one based on learned feature mixing rather than explicit storage—is more appropriate.

The Known Connection: TTT ⇔ Linear Attention (Special Case)

This is not the first time a connection between TTT and linear attention has been observed. Prior work had already established that in the restricted special case of a single linear inner-loop layer with zero initialization and mean squared error (MSE) loss, TTT is exactly equivalent to linear attention (Sun et al., 2025). Specifically, when $f_\theta(x) = xW$ and $\mathcal{L} = \|kW - v\|^2$ , one step of gradient descent on the inner-loop objective yields:

$W_{t+1} = W_t - \eta \cdot k_t^\top (k_t W_t - v_t)$

With zero initialization ( $W_0 = 0$ ) and a learning rate of $\eta = 1$ , this reduces to:

$o_t = q_t W_{t+1} = q_t \sum_{i=0}^t k_i^\top v_i$

which is exactly the standard linear attention form $o_t = q_t \sum_{i=0}^t k_i^\top v_i$ (Katharopoulos et al., 2020).

However, this known connection was treated as an isolated special case rather than a general principle. The prevailing view was that this equivalence breaks down as soon as the inner loop becomes more sophisticated—when the MLP has multiple layers, when momentum is used, when the loss function deviates from MSE, or when normalization is applied. The field operated under the implicit assumption that these additional complexities were exactly what made TTT different from and more powerful than linear attention. The storage-and-retrieval interpretation provided the conceptual rationale for why these complexities should help: deeper networks can store more information, momentum smooths the memory formation process, and normalization stabilizes the stored representations.

This paper challenges that assumption head-on, demonstrating that the equivalence generalizes far beyond the single-linear-layer case.

The Gap This Paper Fills

The central gap the paper addresses can be stated precisely: the prevailing memorization-based interpretation of TTT is both empirically falsifiable and analytically incomplete, and the field lacks a unified alternative that can explain observed behaviors, guide architecture design, and connect TTT to the broader linear attention literature.

This gap manifests in four specific ways the paper identifies and addresses:

1. No systematic empirical stress-testing of the memorization hypothesis. Prior work presented the storage-and-retrieval interpretation as conceptually intuitive and then designed architectures to optimize for that interpretation, but no prior study had systematically subjected the memorization hypothesis to direct empirical tests. What happens if we deliberately break the memorization objective? What does the relationship between inner-loop optimization quality and task performance actually look like? What are the distributional properties of queries and keys in converged TTT models, and are they consistent with retrieval semantics? These questions had not been asked.

2. No general analytical framework connecting complex TTT variants to linear attention. The known equivalence for single linear layers was viewed as a trivial boundary case rather than a hint of a deeper structural relationship. The analytical tools for handling multi-layer inner loops, momentum, and arbitrary differentiable loss functions within a unified linear-attention framework did not exist.

3. No bridge between the TTT and linear attention research communities. If TTT truly functions as a form of linear attention, then TTT variants can be systematically compared to and improved by the extensive body of work on efficient attention mechanisms, chunk-parallel formulations, and hardware-optimized implementations. Without this bridge, TTT research operates in isolation.

4. No principled framework for simplifying TTT architectures. The memorization interpretation naturally encourages architectural complexity (deeper MLPs, better optimizers, more sophisticated normalization). Without an alternative conceptual framework, there is no principled way to distinguish which components are genuinely valuable and which are unnecessary. The field needs a lens that reveals the essential computational mechanism of TTT, enabling systematic ablation and simplification.

How the Paper Positions Itself

The paper's positioning is ambitious but precise: it aims not merely to propose an alternative interpretation, but to systematically dismantle the prevailing memorization-based account and replace it with a mathematically rigorous alternative that explains anomalous behaviors, enables practical simplifications, and connects TTT to linear attention.

The strategy proceeds in three stages:

Stage 1: Empirical refutation (Section 4). Rather than arguing against the memorization interpretation in the abstract, the paper identifies four specific, testable predictions that the memorization hypothesis makes, and then shows empirically that each prediction fails in practice. These four contradictions—the inverse relationship between inner-loop optimization and task performance (Section 4.1), the effectiveness of gradient ascent (Section 4.2), distributional asymmetry between queries and keys (Section 4.3), and the negligible impact of replacing queries with keys (Section 4.4)—are chosen to collectively rule out both the "storage" and "retrieval" aspects of the memorization interpretation. If TTT doesn't store information (gradient ascent preserves performance) and doesn't retrieve information (replacing Q with K doesn't matter), then the storage-and-retrieval account is fundamentally incorrect regardless of any auxiliary assumptions.

Stage 2: Analytical reconstruction (Section 5). The paper provides a constructive alternative by showing that TTT can be exactly rewritten as a linear attention operator, even for complex inner-loop configurations involving multi-layer MLPs (Theorem 5.1), sequential processing of multiple tokens (Theorem 5.2), and momentum-based optimization (Theorem 5.3). The framework introduces the crucial concept that the inner loop's gradient descent induces a learned linear attention form where the effective queries ( $\hat{q} = \phi_{t+1}(q)$ ), keys ( $\hat{k} = \phi_t(k)$ ), and values ( $\hat{v} = g_t(k)$ ) emerge from the optimization dynamics rather than being directly provided as inputs. This reframes TTT not as explicit memorization, but as a mechanism for dynamically constructing query, key, and value representations through a structured optimization process—fundamentally different from standard attention but mathematically expressible in the same framework.

Stage 3: Practical validation (Section 6). To demonstrate that the linear-attention perspective is not merely theoretically elegant but practically useful, the paper shows that TTT architectures can be progressively simplified by removing components that are only justified under the memorization interpretation (per-token learning rates, momentum, weight normalization, gradient orthogonalization, deep inner-loop MLPs), ultimately reducing to standard linear attention with minimal performance degradation. Furthermore, by recognizing that the simplified formulations correspond to an associative state update, the paper derives a fully parallel implementation that achieves up to $4.0\times$ inference throughput improvement.

Relation to Prior Work

The paper explicitly distinguishes its contribution from several related lines of work:

From the original TTT-as-memorization work (Sun et al., 2025): This paper acknowledges the prior observation that TTT with a single linear layer reduces to linear attention but argues that this was treated as a special case rather than a general principle. The current work extends the analytical framework to handle the actual complex architectures used in practice (multi-layer MLPs, momentum, arbitrary losses) and, crucially, uses the resulting perspective to explain previously puzzling behaviors and derive practical simplifications.

From LaCT (Zhang et al., 2025) and ViTTT (Han et al., 2025): These are the specific TTT variants analyzed in the paper. The paper does not propose a new TTT architecture; instead, it provides a reinterpretation of existing ones, showing that their apparent complexity masks a fundamentally simpler underlying mechanism.

From Titans (Behrouz et al., 2024) and related work on test-time memory: The paper's empirical findings directly challenge the design philosophy behind architectures that invest heavily in sophisticated inner-loop optimization. The observation that gradient ascent preserves performance (Table 1) is particularly devastating for any approach whose core value proposition rests on the quality of inner-loop gradient-based learning.

From the linear attention literature (Katharopoulos et al., 2020; Schlag et al., 2021): The paper bridges TTT to this literature by showing that TTT is not an alternative to linear attention but rather a specific learned parameterization of it. This connection is bidirectional: it explains TTT's behavior in terms of known attention mechanisms, and it suggests that linear attention research—particularly around chunk-parallel formulations and decay factors—can directly inform TTT design.

What the Paper Does Not Address

The paper is explicit about its scope limitations, which are important for understanding where the analysis applies and where it doesn't:

Only TTT-KVB, not TTT-E2E. The analysis applies to TTT variants that optimize a key-value binding objective in the inner loop, not to end-to-end methods that backpropagate from the final task loss through the inner loop (Tandon et al., 2025). These are fundamentally different mechanisms.
Linear, bias-free final layer required. The analytical reduction to linear attention assumes the inner-loop function has a linear final layer without bias terms (Section 5.1). Extending the analysis to fully nonlinear architectures remains future work.
Not a proposal for a new architecture. The paper's contribution is analytical and empirical clarification, not a new TTT variant. The practical implications (Section 6) derive from simplifying existing architectures by removing redundant components, not from introducing new ones.

In summary, this paper addresses a fundamental question that has shaped TTT architecture design since its inception: does TTT genuinely function as test-time memorization, or does it operate through a different mechanism? By providing both empirical evidence against the memorization hypothesis and a constructive alternative in the form of a unified linear-attention framework, the paper aims to redirect TTT research away from increasingly complex storage-and-retrieval machinery and toward principled design based on learned feature mixing.

3. Technical Approach

This is primarily an analytical reconstruction paper whose core idea is that TTT with KV binding does not function as test-time memorization but rather as a learned linear attention operator, and that this perspective enables both a unified explanation of previously puzzling empirical behaviors and concrete practical simplifications of complex TTT architectures.

3.1 Reader Orientation

The paper develops a mathematical framework for rewriting TTT inner-loop dynamics as linear attention, then uses this framework to explain why TTT behaves the way it does and to progressively strip away unnecessary architectural components. The system being analyzed is the TTT sequence modeling layer itself—the paper does not propose a new system but rather provides a lens for understanding and simplifying existing ones. The problem it solves is conceptual: the prevailing memorization-based interpretation of TTT leads to increasingly complex architectures and prevents cross-pollination with the linear attention literature, but this interpretation is empirically false and analytically unnecessary. The solution is a constructive alternative that shows exactly how TTT's inner loop induces a linear attention operator, with the effective queries, keys, and values emerging from the optimization dynamics rather than being manually specified.

3.2 Big-Picture Architecture (Diagram in Words)

The analysis framework has three major components:

Forward pass decomposition: The TTT inner-loop function $f(x) = \phi(x; \Theta) W$ is factored into a kernel function $\phi(\cdot; \Theta)$ (the hidden representation, potentially multi-layer and nonlinear) and a linear final layer $W$ (the weight matrix mapping from hidden dimension to output dimension). This decomposition is the key analytical move—it separates the contribution of the final linear layer from everything else.
Gradient unrolling: One or more steps of inner-loop gradient descent on the key-value binding loss are algebraically expanded to reveal how the final-layer weight matrix $W$ accumulates outer products of (transformed keys) and (gradient-derived values). This unrolling is performed for three progressively more complex settings: single-step SGD, multi-step sequential SGD, and SGD with momentum.
Linear attention identification: The unrolled expression is recognized as having the form $o = \hat{q} (S_0 + \sum_i \hat{k}_i^\top \hat{v}_i)$ , which is exactly a linear attention operator with state matrix $S_0$ , effective queries $\hat{q}$ , effective keys $\hat{k}_i$ , and effective values $\hat{v}_i$ . These effective representations are not the raw inputs $Q$ , $K$ , $V$ but are instead computed from them through the kernel function and gradient dynamics.

Information flows as follows: raw inputs $(q_t, k_t, v_t)$ enter the TTT layer → the kernel function transforms the raw key into $\phi_t(k_t)$ (the effective key) and computes gradients through the loss → the gradient with respect to the final layer yields $g_t(k_t)$ (the effective value) → the weight matrix update accumulates the outer product $\phi_t(k_t)^\top g_t(k_t)$ → the query passes through the updated kernel function to produce $\phi_{t+1}(q_t)$ (the effective query) → the output is computed as $\phi_{t+1}(q_t)$ times the accumulated state. At no point does the system explicitly "store" and "retrieve" key-value pairs; instead, the gradient descent dynamics implicitly construct a structured feature mixing operation.

3.3 Roadmap for the Deep Dive

First, the general analytical framework (Theorems 5.1–5.3): The paper establishes that TTT inner loops can be rewritten as linear attention operators under increasingly general conditions. Theorem 5.1 handles a single gradient step; Theorem 5.2 handles sequential processing of multiple tokens; Theorem 5.3 extends to momentum-based optimization. These theorems form the mathematical backbone of the paper and are required to understand every subsequent claim.
Second, the explanation of empirical anomalies (Section 5.2): With the mathematical framework in place, the paper revisits each of the four empirical contradictions from Section 4 and shows how the linear-attention perspective provides a mechanistic explanation for each one. This connects the analytical contribution to the empirical motivation.
Third, the concrete reduction of LaCT to linear attention (Section 5.3): The paper applies the general framework to a specific, representative TTT variant (LaCT; Zhang et al., 2025), showing how its SwiGLU MLP, Frobenius inner product loss, momentum, and Muon gradient orthogonalization all fit within the linear-attention form. This demonstrates that the framework handles real-world architectural complexity, not just toy cases.
Fourth, the concrete reduction of ViTTT to linear attention (Section 5.4): The paper applies the same framework to a second TTT variant (ViTTT; Han et al., 2025), demonstrating generality across model families and task domains.
Fifth, the progressive ablation trajectory (Section 6.1): Building on the analytical framework, the paper defines a six-step reduction from complex TTT variants (LaCT, ViTTT) to standard linear attention, identifying which components are redundant and which provide genuine benefit. This is where the theoretical perspective translates to practical architecture simplification.
Sixth, the parallel formulation (Section 6.2): Recognizing that the simplified TTT formulations correspond to associative state updates, the paper derives a parallel implementation that replaces token-by-token recurrence with a chunk-parallel prefix scan, yielding throughput improvements.

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 Theorem 5.1: Linearization of a Single Inner-Loop Update

Theorem 5.1 is the foundational result that the entire analytical framework builds upon. It shows that for any TTT model whose inner-loop function has a linear, bias-free final layer, a single step of gradient descent on the key-value binding loss produces an output that can be expressed exactly as a linear attention operator.

Setup and assumptions. The theorem considers a TTT inner-loop function of the form:

$f(x) = \phi(x; \Theta) W$

where $\phi(x; \Theta) \in \mathbb{R}^{D_h}$ is the hidden representation produced by all layers of the inner-loop network except the final layer, parameterized by $\Theta$ , and $W \in \mathbb{R}^{D_h \times D_{\text{out}}}$ is the weight matrix of the final linear layer. The function $f$ takes an input $x$ (which will be either a key $k$ during inner-loop updates or a query $q$ during output computation), passes it through the potentially deep and nonlinear kernel function $\phi$ to produce a $D_h$ -dimensional hidden vector, and then applies the linear transformation $W$ to produce the $D_{\text{out}}$ -dimensional output.

The key structural assumption is that the final layer is linear and bias-free. This assumption holds for essentially all TTT-KVB architectures in the literature: LaCT uses a SwiGLU MLP where the final operation is multiplication by $W_1$ , ViTTT uses a gated linear unit with a final linear projection, and Titans-family architectures use MLPs with linear output layers. The assumption does not restrict the architecture of the kernel function $\phi(\cdot; \Theta)$ itself, which can be arbitrarily deep and nonlinear.

Gradient descent step. At step $t$ , the inner loop performs one step of gradient descent on a loss $\mathcal{L}$ that measures the discrepancy between the model's output for the key input $k$ and some target (typically the value $v$ , but the theorem is general with respect to the loss function). The update rule is:

$(W_{t+1}, \Theta_{t+1}) = (W_t, \Theta_t) - \eta \nabla_{(W_t, \Theta_t)} \mathcal{L}(f_t(k))$

where $f_t(\cdot) \triangleq \phi(\cdot; \Theta_t) W_t$ is the model at step $t$ (using the notation $f_t$ as shorthand for the function with current parameters), and $\phi_t(\cdot) \triangleq \phi(\cdot; \Theta_t)$ is the kernel function at step $t$ . The gradient is taken with respect to all trainable parameters—both the final-layer weights $W_t$ and the kernel function parameters $\Theta_t$ —so both components are updated in this step.

Gradient of the final layer. The critical algebraic manipulation is applying the chain rule to the final-layer gradient. Because $f_t(k) = \phi_t(k) W_t$ is linear in $W_t$ , the gradient with respect to $W_t$ takes a particularly simple form:

$\nabla_{W_t} \mathcal{L} = \phi_t(k)^\top \frac{\partial \mathcal{L}}{\partial f_t(k)}$

where $\frac{\partial \mathcal{L}}{\partial f_t(k)} \in \mathbb{R}^{D_{\text{out}}}$ is the gradient of the loss with respect to the model's output (a row vector when written as $\phi_t(k)^\top$ times a column vector of output gradients). This is the standard outer product form for the gradient of a linear layer: the outer product of the input activation (as a column vector) and the upstream gradient (as a row vector).

The effective value vector. The paper defines a crucial quantity—the effective value vector—as:

$g_t(k) \triangleq -\eta \frac{\partial \mathcal{L}}{\partial f_t(k)} \in \mathbb{R}^{D_{\text{out}}}$

This vector is the negative of the upstream gradient scaled by the learning rate. The negative sign arises from the gradient descent update rule: $W_{t+1} = W_t - \eta \nabla_{W_t} \mathcal{L} = W_t + \phi_t(k)^\top (-\eta \frac{\partial \mathcal{L}}{\partial f_t(k)})$ . The effective value $g_t(k)$ is what gets "accumulated" into the weight matrix during the update.

The final update expression. Substituting the gradient expression into the gradient descent update yields:

$W_{t+1} = W_t + \phi_t(k)^\top g_t(k)$

What this computes: the weight matrix after the update equals the weight matrix before the update plus the outer product of the effective key $\phi_t(k)^\top$ (a column vector of dimension $D_h$ ) and the effective value $g_t(k)$ (a row vector of dimension $D_{\text{out}}$ ). This outer product is a rank-1 matrix of dimension $D_h \times D_{\text{out}}$ that gets added to $W_t$ .

Why this form matters: this is exactly the form of a state update in linear attention. In standard linear attention (Katharopoulos et al., 2020), the state matrix accumulates key-value outer products: $S_{t+1} = S_t + k_t^\top v_t$ . Here, $W_t$ plays the role of the state $S_t$ , $\phi_t(k)$ plays the role of the key $k_t$ , and $g_t(k)$ plays the role of the value $v_t$ . The weight matrix of the TTT inner loop is structurally identical to the state matrix of a linear attention layer.

Output computation. After the parameter update, the model is evaluated on the query $q$ using the updated parameters $(W_{t+1}, \Theta_{t+1})$ :

$o = \phi_{t+1}(q) W_{t+1}$

where $\phi_{t+1}(\cdot) = \phi(\cdot; \Theta_{t+1})$ is the kernel function with updated parameters. Substituting the expression for $W_{t+1}$ :

$o = \phi_{t+1}(q) (W_t + \phi_t(k)^\top g_t(k))$

Identification as linear attention. This expression is directly recognizable as a linear attention operator of the form:

$o = \hat{q} (S_0 + \hat{k}^\top \hat{v})$

where:

$\hat{q} = \phi_{t+1}(q)$ is the effective query: the query transformed by the updated kernel function after the gradient step
$S_0 = W_t$ is the initial state: the weight matrix before the update, which encodes any prior accumulated information
$\hat{k} = \phi_t(k)$ is the effective key: the key transformed by the kernel function before the gradient step
$\hat{v} = g_t(k)$ is the effective value: the negative scaled upstream gradient

Why this is not just a trivial relabeling: the crucial insight is that the effective queries, keys, and values are not the raw input vectors $q$ , $k$ , $v$ but are instead computed from them through learned transformations and gradient dynamics. The kernel function $\phi$ maps raw inputs to a hidden space where the linear attention computation occurs, and the loss function $\mathcal{L}$ determines how raw values $v$ are transformed into effective values $g_t(k)$ through the gradient computation. This means TTT does not perform explicit key-value storage and retrieval; rather, it uses the inner-loop optimization as a mechanism to dynamically construct query, key, and value representations that are then combined through linear attention.

Handling the kernel function parameter update. The theorem's update rule also updates $\Theta$ (the parameters of $\phi$ ), which changes the kernel function from $\phi_t$ to $\phi_{t+1}$ . This is why the effective query uses $\phi_{t+1}(q)$ while the effective key uses $\phi_t(k)$ —they are computed by the same parametric function evaluated at different parameter states. This asymmetry is not a bug but a feature: it means the query and key representations are computed under different parameter configurations, which naturally explains why their distributions can differ (as observed in Section 4.3) and why replacing $q$ with $k$ does not collapse the attention mechanism (Section 4.4).

3.4.2 Theorem 5.2: Unrolling Multiple Sequential Inner-Loop Updates

Theorem 5.2 extends Theorem 5.1 from a single update to the case where the TTT model processes a sequence of tokens sequentially, performing one gradient descent step per token. This is the standard operating mode of TTT during autoregressive inference and training.

Setup. Given a sequence of query-key pairs $\{(q_0, k_0), (q_1, k_1), \ldots, (q_t, k_t)\}$ (where each token provides both a key for the inner-loop update and a query for the output computation), the TTT model performs one gradient descent step per input in sequence. The parameters after processing token $i$ are:

$(W_{i+1}, \Theta_{i+1}) = (W_i, \Theta_i) - \eta \nabla_{(W_i, \Theta_i)} \mathcal{L}(f_i(k_i))$

After processing all tokens up through $t$ , the parameters $(W_{t+1}, \Theta_{t+1})$ reflect the cumulative effect of $t+1$ gradient steps.

Unrolling the recurrence. By repeatedly applying the single-step update from Theorem 5.1, the weight matrix after $t+1$ updates can be expressed as a sum over history:

$W_{t+1} = W_0 - \eta \sum_{i=0}^{t} \nabla_{W_i} \mathcal{L}(f_i(k_i)) = W_0 + \sum_{i=0}^{t} \phi_i(k_i)^\top g_i(k_i)$

where $W_0$ is the initial weight matrix, $\phi_i(k_i) = \phi(k_i; \Theta_i)$ is the kernel function evaluated at the parameters after $i$ updates, and $g_i(k_i) = -\eta \frac{\partial \mathcal{L}}{\partial f_i(k_i)}$ is the effective value at step $i$ .

What this computes: the weight matrix after processing $t+1$ tokens equals the initial weight matrix plus the sum of rank-1 updates from all previous tokens. Each token $i$ contributes the outer product of its effective key $\phi_i(k_i)^\top$ and its effective value $g_i(k_i)$ , where both the effective key and effective value are computed using the kernel function parameters $\Theta_i$ as they existed at step $i$ (before the update from token $i$ itself).

Why the sum form matters: this is a cumulative sum—exactly the structure that enables parallel prefix scan computation. If the $\phi_i$ functions were all identical (i.e., if the kernel function parameters $\Theta$ were not updated in the inner loop), then the state would be a simple unweighted sum over history: $W_{t+1} = W_0 + \sum_{i=0}^t \phi(k_i)^\top g_i(k_i)$ , which is the standard linear attention form and is trivially parallelizable. The fact that $\phi_i$ changes with $i$ introduces a sequential dependency: each $\phi_i$ depends on all previous updates through $\Theta_i$ . This is the fundamental reason why TTT with full inner-loop parameter updates (including kernel function parameters) is not trivially parallelizable—a point the paper returns to in Section 6.2 and Appendix I.

Output computation. Evaluating the model on query $q_t$ after processing token $t$ yields:

$o_t = \phi_{t+1}(q_t) W_{t+1} = \phi_{t+1}(q_t) \left( W_0 + \sum_{i=0}^{t} \phi_i(k_i)^\top g_i(k_i) \right)$

Identification as extended linear attention. This has the form:

$o_t = \hat{q}_t \left( S_0 + \sum_{i=0}^{t} \hat{k}_i^\top \hat{v}_i \right)$

where:

$\hat{q}_t = \phi_{t+1}(q_t)$ is the effective query for token $t$ , computed using the kernel function after all updates through token $t$
$S_0 = W_0$ is the initial state
$\hat{k}_i = \phi_i(k_i)$ is the effective key for historical token $i$ , computed using the kernel function parameters as they existed at step $i$
$\hat{v}_i = g_i(k_i)$ is the effective value for historical token $i$

The causal structure: the sum runs from $i=0$ to $i=t$ , meaning each output can only attend to tokens that have been processed before it (including itself, since the query $q_t$ is processed after the update from key $k_t$ ). This is exactly the causal masking structure of autoregressive attention. The formulation is inherently causal because the weight matrix $W_{t+1}$ only contains contributions from tokens $0$ through $t$ .

The asymmetry in query and key computation: the effective query $\hat{q}_t = \phi_{t+1}(q_t)$ uses the kernel function with parameters $\Theta_{t+1}$ (after the update from token $t$ ), while the effective keys $\hat{k}_i = \phi_i(k_i)$ use the kernel function with parameters $\Theta_i$ (before the update from token $i$ itself, but after all previous updates). This means the effective query is always computed with strictly more parameter updates than any effective key, creating a systematic asymmetry that explains why the distributions of $\hat{q}$ and $\hat{k}$ can differ substantially (Section 4.3). This is the mathematical basis for the paper's claim that distributional mismatch is "expected rather than pathological" (Section 5.2).

3.4.3 Theorem 5.3: Gradient Descent with Momentum

Theorem 5.3 extends the framework to handle momentum-based optimization, which is used in several TTT variants including LaCT. The result shows that momentum does not fundamentally alter the linear-attention structure; it only changes how historical gradient information is aggregated into the effective value vector.

Momentum formulation. The momentum-augmented gradient accumulator is defined as:

$(\Delta W_t, \Delta \Theta_t) = \nabla_{(W, \Theta)} \mathcal{L}(f_t(k_t)) + \alpha_t (\Delta W_{t-1}, \Delta \Theta_{t-1})$

where $\alpha_t$ is the (possibly token-dependent) momentum factor at step $t$ . The momentum accumulator $\Delta W_t$ is a weighted combination of the current gradient and the previous accumulator. The model parameters are then updated using this momentum-smoothed gradient:

$(W_{t+1}, \Theta_{t+1}) = (W_t, \Theta_t) - \eta (\Delta W_t, \Delta \Theta_t)$

What momentum does: standard SGD updates parameters using only the current gradient, which can be noisy and cause high-variance parameter trajectories. Momentum smooths the update by maintaining an exponentially weighted moving average of past gradients. The momentum factor $\alpha_t$ controls how much weight is given to historical gradients versus the current one: $\alpha_t = 0$ recovers standard SGD (no momentum), while $\alpha_t$ close to 1 gives high persistence to past gradient directions.

Unrolling the momentum recurrence. The key to the proof is unrolling the momentum accumulator to express it as a weighted sum of all past gradients. Define the cumulative momentum coefficient from step $i$ to step $j$ as:

$\beta_{ij} \triangleq \begin{cases} \prod_{s=i+1}^{j} \alpha_s & \text{if } i < j, \\ 1 & \text{if } i = j. \end{cases}$

where $\beta_{ij}$ is the product of all momentum factors between steps $i+1$ and $j$ inclusive. This coefficient captures how much of the gradient from step $i$ survives in the momentum accumulator at step $j$ —each intermediate step multiplies the surviving contribution by its momentum factor $\alpha_s$ .

What this notation achieves: $\beta_{ij}$ measures the "persistence" of the gradient from step $i$ through step $j$ . If all $\alpha_s = 0.9$ , then $\beta_{i,i+2} = 0.9 \cdot 0.9 = 0.81$ , meaning the gradient from step $i$ contributes 81% as much to the momentum accumulator at step $i+2$ as it did at step $i$ . The $\beta_{ij}$ notation allows writing the momentum accumulator at step $t$ as a single sum over all past steps with appropriate decay weights.

The momentum accumulator in unrolled form. For the final-layer weights, the momentum accumulator at step $t$ is:

$\Delta W_t = \sum_{i=0}^{t} \beta_{it} \cdot \phi_i(k_i)^\top \frac{\partial \mathcal{L}}{\partial f_i(k_i)}$

This expresses the momentum accumulator as a weighted sum of the raw gradients from all steps $0$ through $t$ , where the gradient from step $i$ is weighted by the cumulative momentum coefficient $\beta_{it}$ .

The weight matrix update. The weight matrix after processing token $t$ is:

$W_{t+1} = W_0 - \eta \sum_{j=0}^{t} \Delta W_j$

Substituting the unrolled form of $\Delta W_j$ and exchanging the order of summation (summing over $i$ first, then $j$ ) yields:

$W_{t+1} = W_0 + \sum_{i=0}^{t} \phi_i(k_i)^\top \underbrace{\left( g_i(k_i) \cdot \sum_{j=i}^{t} \beta_{ij} \right)}_{m_i(k_i)}$

where $g_i(k_i) = -\eta \frac{\partial \mathcal{L}}{\partial f_i(k_i)}$ is the unscaled effective value as before.

The momentum-weighted effective value. The quantity:

$m_i(k_i) \triangleq g_i(k_i) \cdot \sum_{j=i}^{t} \beta_{ij}$

is the momentum-weighted effective value for token $i$ . It equals the raw effective value $g_i(k_i)$ multiplied by the sum of cumulative momentum coefficients from step $i$ to step $t$ (inclusive). The sum $\sum_{j=i}^{t} \beta_{ij}$ represents the total influence that the gradient from step $i$ has on all subsequent momentum accumulators up through step $t$ .

What this computes: the momentum-weighted effective value is larger for recent tokens (where $i$ is close to $t$ , so $\sum_{j=i}^t \beta_{ij}$ is dominated by $\beta_{it} \approx 1$ and a few additional terms) and smaller for distant tokens (where $i \ll t$ , so the sum includes many $\beta_{ij}$ terms that have decayed substantially). This implements a recency bias in the linear attention computation: recent key-value pairs are weighted more heavily than distant ones because their gradients have persisted through fewer momentum decay steps.

Output computation and linear attention form. Evaluating on query $q_t$ :

$o_t = \phi_{t+1}(q_t) \left( W_0 + \sum_{i=0}^{t} \phi_i(k_i)^\top m_i(k_i) \right)$

This is the linear attention form with effective values $\hat{v}_i = m_i(k_i)$ instead of $g_i(k_i)$ . The form is identical to that of Theorem 5.2, with the only difference being that the effective values are momentum-weighted rather than raw.

Why momentum does not break the linear attention equivalence: momentum only changes how the effective values $m_i(k_i)$ are computed from the raw gradients $g_i(k_i)$ . The outer product structure $\phi_i(k_i)^\top m_i(k_i)$ and the linear reading operation $\phi_{t+1}(q_t) W_{t+1}$ remain unchanged. From the perspective of the linear attention operator, momentum simply remixes the historical value vectors into a single weighted representation. Since both the keys and values are already learnable parameters (through the kernel function $\phi$ and the value projection), this additional mixing is unlikely to provide meaningful benefits beyond what the learned projections can already achieve—a point the paper verifies empirically in the ablation study (Table 2, Variant 5, where removing momentum actually slightly improves perplexity).

The significance of $\beta_{ij}$ : this notation formalizes the recency bias in TTT with momentum. In standard linear attention without decay, all historical tokens contribute equally to the state (subject to the outer product magnitudes). TTT with momentum implicitly introduces a form of token-dependent decay where more recent tokens have greater influence on the current state. This connects TTT to the broader literature on linear attention with decay factors (e.g., RetNet, Mamba, GLA), where explicit decay mechanisms are a central design component. From the TTT perspective, decay emerges automatically from the momentum dynamics rather than being manually specified.

3.4.4 Explanation of TTT Empirical Behaviors (Section 5.2)

With the linear-attention framework established, the paper revisits each empirical anomaly from Section 4 and provides a mechanistic explanation.

More inner-loop steps degrade performance (Section 4.1). Under the memorization interpretation, more inner-loop steps should improve key-value fitting and thus improve task performance. Under the linear-attention view, the number of inner-loop steps is a hyperparameter of the attention operator—it controls the effective query, key, and value mappings. Training is performed with a specific number of inner-loop steps, which induces a specific linear attention operator that the downstream task loss optimizes over. At inference time, changing the number of inner-loop steps produces a different attention operator than the one the model was trained for, creating a train-test mismatch. This mismatch naturally degrades performance, exactly as observed in Figure 1. The degradation is not because memorization fails but because the inference-time attention operator differs from the training-time one.

Gradient ascent preserves performance (Section 4.2). Replacing gradient descent with gradient ascent flips the sign of the effective value vector: $g_t(k)$ becomes $-g_t(k)$ (since $g_t(k) = -\eta \frac{\partial \mathcal{L}}{\partial f_t(k)}$ , and ascent uses $+\eta$ instead of $-\eta$ , so the effective value becomes $+\eta \frac{\partial \mathcal{L}}{\partial f_t(k)} = -g_t(k)$ ). From the linear-attention perspective, this sign flip is absorbed into the learned value projection. The model learns to map raw values $v$ to effective values that produce correct behavior under the sign convention used during training. The sign of the gradient step is just one part of the learned mapping and does not affect the fundamental structure of the operator. This explains why gradient ascent works despite being catastrophic under a memorization interpretation.

Distributional asymmetry between Q and K (Section 4.3). In standard attention, queries and keys inhabit the same space because attention weights are computed as dot-product similarities: similar query-key pairs produce large attention weights. This forces queries and keys to have overlapping distributions for the mechanism to function. In TTT-as-linear-attention, the query and key influence different components of the operator: $q$ determines the effective query $\phi_{t+1}(q)$ through the kernel function applied to the raw query, while $k$ determines the effective key $\phi_t(k)$ through the kernel function applied to the raw key. Since the kernel function parameters change between the time $k$ is processed (step $t$ ) and the time $q$ is processed (step $t+1$ ), the effective query and key are computed by different functions. Moreover, the effective query is used for reading (left-multiplying the state matrix), while the effective key is used for writing (forming outer products that are added to the state matrix). These are fundamentally different roles that do not require distributional similarity. The observed asymmetry is therefore not a bug but a natural consequence of the mechanism.

Replacing Q with K has negligible effect (Section 4.4). When replacing queries with keys, the effective query becomes $\phi_{t+1}(k)$ instead of $\phi_{t+1}(q)$ . Despite the raw inputs being identical, the effective query and effective key remain distinct because they are computed at different parameter states: $\phi_{t+1}(\cdot)$ versus $\phi_t(\cdot)$ . The kernel function can map the same raw input to different representations at different points in the parameter trajectory, preserving the functional distinction between query and key roles. This explains why performance is preserved: the model is not relying on similarity between $q$ and $k$ for retrieval; it is relying on the learned kernel function to produce appropriate effective representations regardless of whether the raw input is nominally a query or a key.

3.4.5 Concrete Reduction: LaCT as Linear Attention (Section 5.3)

The paper demonstrates the generality of its framework by applying it to LaCT (Zhang et al., 2025), a representative TTT variant with substantial architectural complexity. LaCT uses a bias-free SwiGLU MLP, Frobenius inner product loss, per-token learning rates, momentum, and Muon-style gradient orthogonalization—all components that, under a memorization interpretation, are justified for improving storage fidelity. The paper shows that despite this complexity, LaCT's inner loop reduces exactly to a linear attention form.

LaCT architecture. The inner-loop function is a SwiGLU MLP parameterized by three weight matrices:

$f(x) = \left( \text{silu}(x W_0) \odot (x W_2) \right) W_1$

where $W_0, W_2 \in \mathbb{R}^{D_h \times D_k}$ and $W_1 \in \mathbb{R}^{D_h \times D_v}$ . The SwiGLU activation applies a silu-gated linear unit: the input is projected by $W_0$ , passed through the silu activation, and then element-wise multiplied by the input projected by $W_2$ . The result is then projected by $W_1$ to produce the output. The kernel function is:

$\phi_t(x) = \text{silu}(x W_{0,t}) \odot (x W_{2,t})$

where $W_{0,t}$ and $W_{2,t}$ are the current values of the weight matrices at step $t$ . This maps an input $x$ to a $D_h$ -dimensional hidden vector, which is then multiplied by $W_{1,t}$ to produce the output. This fits the general form $f(x) = \phi(x; \Theta) W$ with $\Theta = \{W_0, W_2\}$ and $W = W_1$ .

Inner-loop loss. LaCT uses the Frobenius inner product as its key-value binding objective:

$\mathcal{L}(f(k), v) = -\langle f(k), v \rangle$

where $\langle \cdot, \cdot \rangle$ denotes the Frobenius inner product (sum of element-wise products). This is not a standard regression loss—it is a dot-product loss that encourages the model output to align with the value vector in direction, without constraining magnitude. The negative sign makes this a minimization objective: the loss is minimized when $f(k)$ points in the same direction as $v$ .

What the Frobenius inner product loss implies for the gradient: since $\mathcal{L} = -f(k) \cdot v$ (as a dot product), the gradient with respect to the model output is simply:

$\frac{\partial \mathcal{L}}{\partial f_t(k_t)} = -v_t$

This is crucial: the upstream gradient is exactly the negative of the value vector, independent of the model's current output. This means the effective value vector is:

$g_t(k_t) = -\eta_t \frac{\partial \mathcal{L}}{\partial f_t(k_t)} = \eta_t v_t$

The effective value is simply the raw value scaled by the learning rate. The sign from the loss function and the sign from the gradient descent update cancel out, leaving a positive scaling of the value vector.

Why this loss is used: the Frobenius inner product loss makes the effective value computation trivial—it doesn't depend on the kernel function or the model's current parameters. This is mathematically convenient and computationally efficient because the effective value can be computed without additional forward or backward passes through the kernel function. It also means that gradient ascent and gradient descent differ only in the sign of $v_t$ in the effective value, which gets absorbed into the learned value projection.

Momentum in LaCT. LaCT uses per-token momentum $\alpha_t$ and learning rate $\eta_t$ . The update rule for each weight matrix $W_{i,t}$ (for $i \in \{0, 1, 2\}$ ) is:

$\Delta W_{i,t} = \nabla_{W_{i,t}} \mathcal{L}(f_t(k_t), v_t) + \alpha_t \Delta W_{i, t-1}$ $W_{i, t+1} = W_{i, t} - \eta_t \mathcal{M}(\Delta W_{i,t})$

where $\mathcal{M}(\cdot)$ is the Muon gradient orthogonalization operator (described below).

Applying Theorem 5.3 to LaCT. Following the same derivation as Theorem 5.3, the update for the final-layer weight matrix $W_{1,t}$ is:

$W_{1, t+1} = W_{1, 0} + \sum_{i=0}^{t} \mathcal{M}\left( \phi_i(k_i)^\top m_i \right)$

where the momentum-weighted effective value is:

$m_i \triangleq \eta_i v_i \cdot \sum_{j=i}^{t} \beta_{ij}$

with $\beta_{ij}$ as defined in Theorem 5.3.

What changes with momentum: the effective value $m_i$ is now a scalar multiple of the raw value $v_i$ , scaled by both the learning rate $\eta_i$ and the cumulative momentum sum $\sum_{j=i}^t \beta_{ij}$ . Recent tokens (with $i$ close to $t$ ) have larger cumulative momentum sums and thus larger effective values; distant tokens have smaller effective values due to momentum decay.

Muon gradient orthogonalization $\mathcal{M}(\cdot)$ . LaCT applies Muon-style gradient orthogonalization (Jordan et al., 2024) to the momentum accumulator before using it for the weight update. The operator $\mathcal{M}(\cdot)$ orthogonalizes the gradient matrix, typically through QR decomposition or a spectral normalization procedure. From the linear-attention perspective, this corresponds to applying an orthogonalization operator to each key-value outer product before accumulating it into the state:

$\mathcal{M}(\phi_i(k_i)^\top m_i)$

The orthogonalization ensures that each update contributes a matrix with orthonormal structure, which can improve training stability and prevent rank collapse in the state matrix.

Output computation. Evaluating the updated model on query $q_t$ :

$o_t = \phi_{t+1}(q_t) W_{1, t+1} = \phi_{t+1}(q_t) \left( W_{1, 0} + \sum_{i=0}^{t} \mathcal{M}\left( \phi_i(k_i)^\top m_i \right) \right)$

Identification as linear attention. This is exactly the linear attention form with:

Effective query: $\hat{q}_t = \phi_{t+1}(q_t)$
Effective key: $\hat{k}_i = \phi_i(k_i)$
Effective value: $\hat{v}_i = m_i = \eta_i v_i \cdot \sum_{j=i}^{t} \beta_{ij}$
State update operator: $\mathcal{M}(\cdot)$ applied to each outer product

Weight normalization (additional component). LaCT also applies weight normalization to $W_1$ after each update:

$W_{1, t+1} = \text{Norm}\left( W_{1, t} - \eta_t \mathcal{M}(\Delta W_{1,t}) \right)$

where $\text{Norm}(\cdot)$ applies channel-wise $\ell_2$ normalization. The paper notes that this does not break the linear attention interpretation—it simply normalizes the state matrix after each update—but it does break associativity. The state at step $t+1$ becomes a nested normalization:

$S_{t+1} = \text{Norm}\left( \text{Norm}\left( S_{t-1} + \phi_{t-1}(k_{t-1})^\top m_{t-1} \right) + \phi_t(k_t)^\top m_t \right)$

Because normalization is not associative ( $\text{Norm}(A+B) \neq \text{Norm}(A) + \text{Norm}(B)$ ), this creates a strict sequential dependency that prevents parallelization (discussed in detail in Appendix I.2).

Key takeaway from the LaCT reduction: despite the apparent complexity of LaCT—SwiGLU MLP, Frobenius inner product loss, per-token learning rates, momentum, Muon orthogonalization, and weight normalization—the core computation is a linear attention operator where the gradient descent dynamics implicitly construct the effective queries, keys, and values. The additional components (momentum, orthogonalization, normalization) modify how the key-value outer products are computed and accumulated but do not change the fundamental linear-attention structure. This demonstrates that the reduction to linear attention is not limited to toy cases; it handles production-grade TTT architectures with all their practical complexities.

3.4.6 Concrete Reduction: ViTTT as Linear Attention (Section 5.4)

The paper further demonstrates generality by reducing ViTTT (Han et al., 2025), a TTT variant designed for vision tasks with a different architectural structure than LaCT. ViTTT has two independent fast-weight components: a simplified gated linear unit (GLU) and a depthwise convolution layer.

GLU component. ViTTT's GLU is defined as:

$f(x) = \text{silu}(x W_0) \odot (x W_1)$

where $W_0, W_1 \in \mathbb{R}^{D_h \times D_h}$ are both square matrices. Unlike LaCT's SwiGLU, this is a true GLU without the final projection: the gating output $\text{silu}(x W_0)$ is element-wise multiplied with the linear projection $x W_1$ to directly produce the output. The kernel function is:

$\phi_t(x) \triangleq \text{silu}(x W_{0,t})$

This is the gating component. The inner-loop loss uses the Frobenius inner product as in LaCT.

Gradient of the linear projection $W_1$ . The gradient with respect to $W_1$ is:

$\nabla_{W_{1,t}} \mathcal{L} = k_t^\top \left( \frac{\partial \mathcal{L}}{\partial f_t(k_t)} \odot \phi_t(k_t) \right) = -k_t^\top (v_t \odot \phi_t(k_t))$

where $\odot$ denotes element-wise multiplication. The gradient is the outer product of the raw key $k_t^\top$ and the gated value $v_t \odot \phi_t(k_t)$ . Here, $\phi_t(k_t)$ acts as a multiplicative gate on the value vector before accumulation.

Linear attention form for the GLU. After one step of gradient descent:

$W_{1, t+1} = W_{1, t} + \eta \cdot k_t^\top (v_t \odot \phi_t(k_t))$

Evaluating on query $q_t$ :

$o_t = \phi_{t+1}(q_t) \odot \left( q_t W_{1, t+1} \right) = \phi_{t+1}(q_t) \odot \left( q_t W_{1, t} + \eta \cdot (q_t k_t^\top) (v_t \odot \phi_t(k_t)) \right)$

What this means: the term $\eta \cdot (q_t k_t^\top) (v_t \odot \phi_t(k_t))$ computes a scalar attention weight $\langle q_t, k_t \rangle$ (the dot product of query and key) and uses it to scale the gated value vector $v_t \odot \phi_t(k_t)$ . The kernel function appears in two places: $\phi_t(k_t)$ gates the value before accumulation, and $\phi_{t+1}(q_t)$ gates the final output. This is a form of gated linear attention where the gating mechanism is learned through the inner-loop optimization dynamics.

Depthwise convolution component. ViTTT additionally includes a $3 \times 3$ depthwise convolution layer with fast weights that are also updated in the inner loop. The forward pass computes:

$f_t(K) = \text{Conv}_{3\times 3}(K; W_t)$

where $W_t \in \mathbb{R}^{C \times 1 \times 3 \times 3}$ is the depthwise convolution kernel. The inner-loop loss is again the Frobenius inner product between the convolution output and a spatial value tensor $V$ .

Gradient of the convolution kernel. For depthwise convolution, the gradient with respect to $W_t$ can be written as a cross-correlation between the input key tensor and the upstream gradient:

$\nabla_{W_t} \mathcal{L} = K \star \frac{\partial \mathcal{L}}{\partial f_t(K)} = -K \star V$

where $\star$ denotes cross-correlation. For each channel $c$ and spatial offset $(\delta_y, \delta_x) \in \{-1, 0, 1\}^2$ :

$[\nabla_{W_t} \mathcal{L}]_{c, \delta_y, \delta_x} = -\sum_{i, j} K_{c, i+\delta_y, j+\delta_x} \cdot V_{c, i, j}$

Linear attention form for the convolution. After one step of gradient descent and evaluating on a query tensor $Q$ , the output at spatial position $(i, j)$ is:

$O_{c, i, j} = [\text{Conv}(Q; W_t)]_{c, i, j} + \eta \sum_{i', j'} \underbrace{\left( \sum_{\delta_y, \delta_x} Q_{c, i+\delta_y, j+\delta_x} \cdot K_{c, i'+\delta_y, j'+\delta_x} \right)}_{\text{spatial attention weight}} \cdot V_{c, i', j'}$

What this computes: the output at each spatial position is the convolution of the query with the current kernel (the "initial state" term) plus a weighted sum over all spatial positions of the value tensor, where the weight between query position $(i, j)$ and key position $(i', j')$ is the sum of element-wise products over the $3 \times 3$ neighborhood offsets. This is a form of spatially-local linear attention: each output position attends to all key-value positions, but the attention weight is computed using only local $3 \times 3$ neighborhoods rather than global dot products. The convolution is effectively a sliding-window linear attention mechanism.

Why both components reduce to linear attention: the GLU reduces to gated linear attention where the gating functions are learned through gradient dynamics; the depthwise convolution reduces to spatially-local linear attention with $3 \times 3$ neighborhood interactions. Since both components independently admit linear-attention formulations, their combination (used together in ViTTT) also induces a linear-attention-like operator. The paper establishes that ViTTT, despite its dual-component architecture and domain-specific design (vision rather than language), falls within the same unified framework as LaCT.

3.4.7 Progressive Ablation: Reducing TTT to Linear Attention (Section 6.1)

The paper's practical contribution is a six-step ablation trajectory that progressively removes components from complex TTT variants, ultimately reducing them to standard linear attention. Each step is justified by the linear-attention perspective and tested empirically across three tasks (language modeling with LaCT-LLM, novel view synthesis with LaCT-NVS, image classification with ViTTT-B). Table 2 reports results for all variants.

Step 1: Update only the last-layer parameters. In the full TTT formulation, all parameters of the inner-loop model (including the kernel function parameters $\Theta = \{W_0, W_2\}$ in LaCT, or $\{W_0\}$ in ViTTT) are updated in the inner loop. This makes the kernel function $\phi_t(\cdot)$ history-dependent, since $\phi_t$ changes with each token. The paper proposes restricting the inner-loop update to only the final-layer weight matrix $W_1$ (or $W_1$ in ViTTT), keeping the kernel function parameters $\Theta$ fixed. From the linear-attention perspective, this makes $\phi(\cdot)$ a static learnable kernel function, and the effective queries and keys become $\phi(q)$ and $\phi(k)$ —constant functions of the raw inputs rather than history-dependent ones. The state update becomes a simple sum: $W_{t+1} = W_0 + \sum_{i=0}^t \phi(k_i)^\top g_i(k_i)$ . Empirically, Variant 1 improves performance on all three tasks: perplexity drops from 16.43 to 15.93 on language modeling, PSNR increases from 25.94 to 25.97 on novel view synthesis, and accuracy increases from 79.34% to 79.63% on image classification. This suggests that updating kernel function parameters in the inner loop is not only unnecessary but actually harmful—it introduces a sequential dependency (through the changing $\phi_t$ ) without providing representational benefits beyond what a static learned kernel can achieve.

Step 2: Remove weight normalization. TTT variants like LaCT apply weight normalization to all learnable parameters after each inner-loop update. After Step 1, normalization on $\Theta$ is a no-op since $\Theta$ is fixed. Normalization on the final-layer weight matrix $W_1$ corresponds to normalizing the state matrix $S_t$ in the linear attention view. Since state normalization is uncommon in the linear attention literature (standard linear attention does not normalize its accumulated state), the paper removes it. Empirically, removing weight normalization has minimal impact: perplexity increases from 15.93 to 16.31, PSNR drops slightly from 25.97 to 25.93, and accuracy is unchanged at 79.63%. A crucial side effect of this step: with static kernel function and no weight normalization, the state update becomes associative, enabling the parallel formulation described in Section 6.2.

Step 3: Reduce multi-layer MLP to single linear layer. TTT variants use multi-layer MLPs as inner-loop functions (e.g., LaCT's SwiGLU with a hidden dimension). From the linear-attention perspective, the MLP depth simply determines the complexity of the kernel function $\phi(\cdot)$ . When raw queries and keys already have sufficient representational capacity (as they do, being the output of preceding transformer layers), a deeper kernel function is unlikely to provide benefit. The paper replaces the multi-layer MLP with a single linear layer, effectively setting $\phi(x) = x$ (removing the kernel function altogether). This exposes the true queries and keys as the raw inputs: $\hat{q} = q$ , $\hat{k} = k$ . Empirically, this simplification has task-dependent effects: language modeling perplexity improves slightly from 16.31 to 16.23, novel view synthesis PSNR drops from 25.93 to 25.71 (suggesting the kernel function does provide some benefit for this task), and image classification accuracy drops slightly from 79.63% to 79.39%. The overall performance impact is modest, confirming that deep inner-loop MLPs are not essential to the core TTT mechanism.

Step 4: Remove per-token learnable learning rates. TTT methods like LaCT and Titans use a per-token learnable learning rate $\eta_t$ , typically produced by a small neural network that takes the current token as input. Under the Frobenius inner product loss used by LaCT and ViTTT, the effective value is $m_i = \eta_i v_i \cdot \sum_{j=i}^t \beta_{ij}$ . The per-token learning rate $\eta_i$ simply scales the raw value $v_i$ before accumulation, which can be absorbed into a learned value projection. The paper removes the per-token learning rate, using a fixed $\eta$ instead. Empirically, this has minimal impact: perplexity improves from 16.23 to 16.12, PSNR is unchanged at 25.70, and accuracy is unchanged at 79.39%. The paper notes that ViTTT independently found that a constant learning rate of 1.0 suffices, consistent with this result.

Step 5: Remove momentum from SGD. As shown in Theorem 5.3, momentum remixes historical gradients into a weighted effective value $m_i = g_i(k_i) \cdot \sum_{j=i}^t \beta_{ij}$ instead of using the raw effective value $g_i(k_i)$ . With the Frobenius inner product loss, $g_i(k_i) = \eta v_i$ , so removing momentum (setting $\alpha_t = 0$ for all $t$ ) recovers $m_i = \eta v_i$ —the effective value becomes exactly the scaled raw value. Empirically, removing momentum has minimal or slightly beneficial impact: perplexity drops from 16.12 to 15.97, PSNR is unchanged at 25.70, and accuracy is unchanged at 79.39%. This confirms that momentum's recency-weighting effect is not providing meaningful benefits beyond what the learned value projection can already achieve.

Step 6: Remove gradient orthogonalization. LaCT applies Muon-style gradient orthogonalization $\mathcal{M}(\cdot)$ to the momentum accumulator. From the linear-attention perspective, this applies an orthogonalization operator to each key-value outer product before accumulation: $\mathcal{M}(\hat{k}_i^\top \hat{v}_i)$ . The paper removes this operation. After this final step, the TTT formulation reduces exactly to standard linear attention:

$o = q \left( W + \sum_i k_i^\top v_i \right)$

where $q$ , $k_i$ , and $v_i$ are the raw query, key, and value vectors (possibly with learned linear projections as in standard attention), $W$ is a learned initial state, and the sum is over all previous tokens in the sequence. Empirically, removing orthogonalization has task-dependent effects: language modeling perplexity increases from 15.97 to 16.80 (the largest single-step degradation, suggesting orthogonalization is genuinely beneficial for language modeling), novel view synthesis PSNR improves slightly from 25.70 to 25.73, and image classification accuracy increases from 79.39% to 79.54% (note: ViTTT does not use gradient orthogonalization; the asterisk in Table 2 indicates that a different normalization was ablated instead, with accuracy improving upon its removal). The overall performance of Variant 6 (standard linear attention) compared to the original LaCT/ViTTT baselines shows a perplexity increase of 0.37 (16.80 vs. 16.43), a PSNR drop of 0.21 dB (25.73 vs. 25.94), and an accuracy increase of 0.20% (79.54% vs. 79.34%). The paper characterizes this as "only minor performance degradation" relative to the substantial simplification achieved.

Why this ablation trajectory is principled: each step is motivated by the linear-attention perspective rather than by arbitrary trial-and-error. The framework identifies which components correspond to the core linear attention computation and which are additional machinery that modifies how queries, keys, and values are computed or accumulated. By stripping away the latter while preserving the former, the paper demonstrates that the essential computation of TTT is linear attention, and the additional components provide at most marginal benefits on specific tasks.

3.4.8 Parallel Formulation of TTT (Section 6.2)

A major practical benefit of the linear-attention perspective is that it enables parallel implementations. Under the original storage-and-retrieval interpretation, TTT is inherently sequential: each token's inner-loop update must complete before the next token can be processed. But the paper shows that when certain conditions are met (corresponding to Variants 2–6 in the ablation), TTT admits a fully parallel formulation.

Conditions for parallelization. The key requirement is that the state update must be associative—that is, the order of accumulation must not affect the final result. This holds when:

The kernel function is static (only $W_1$ is updated, not $\Theta$ ), so $\phi(\cdot)$ does not depend on sequence history.
Weight normalization is removed, so the state is a simple cumulative sum rather than a nested normalization.

Under these conditions, the state at step $t$ is:

$S_{t+1} = S_0 + \sum_{i=0}^{t} \phi(k_i)^\top m_i$

This is an associative operation: the sum can be broken into chunks, each chunk's contribution can be computed independently, and the chunk results can be combined. This enables parallel prefix scan (also called parallel scan or associative scan), a standard parallel algorithm for computing all prefix sums of a sequence in logarithmic depth.

Chunk-parallel formulation. The paper implements the parallel formulation by splitting the sequence of $N \cdot L$ tokens into $N$ chunks of size $L$ . Define the following quantities for a batched computation:

$\mathbb{Q}, \mathbb{K} \in \mathbb{R}^{(NL) \times D_k}$ : concatenated query and key tensors
$\mathbb{V} \in \mathbb{R}^{(NL) \times D_v}$ : concatenated value tensor
$\Phi(\mathbf{X}) = \text{silu}(\mathbf{X} W_0) \odot (\mathbf{X} W_2)$ : batched kernel function applied to all tokens simultaneously
$\mathbf{B} \in \mathbb{R}^{(NL) \times (NL)}$ : block-diagonal matrix with $N$ identity blocks of size $L \times L$ (used for chunk-internal computation)
$\boldsymbol{\eta} \in \mathbb{R}^{(NL) \times 1}$ : per-token learning rates (removed in later variants)
$\tilde{\mathbb{V}} = \mathbb{V} \odot \boldsymbol{\eta}$ : scaled values (when using per-token learning rates)
$\mathcal{A} \in \mathbb{R}^{N \times N}$ : chunk-level momentum matrix where $\mathcal{A}_{ij} = \prod_{s=j+1}^i \alpha_s$ for $i \geq j$ (captures momentum decay across chunks)
$\mathbf{M} \in \{0, 1\}^{N \times N}$ : causal mask where $\mathbf{M}_{ij} = 1$ if $i \geq j$ (ensures chunk $i$ can only attend to chunks $j \leq i$ )

The output is computed as:

$\mathbb{O} = \Phi(\mathbb{Q}) W_{1,0} + \left( \left( \Phi(\mathbb{Q}) \Phi(\mathbb{K})^\top \right) \odot (\mathcal{A} \odot \mathbf{M})^{\uparrow L} \right) \tilde{\mathbb{V}}$

where $(\cdot)^{\uparrow L}$ denotes the Kronecker product with $\mathbf{1}_{L \times L}$ to expand the $N \times N$ chunk-level mask to the $(NL) \times (NL)$ token-level attention matrix.

What this computes: the first term $\Phi(\mathbb{Q}) W_{1,0}$ is the contribution from the initial state (the same for all tokens). The second term computes chunk-wise linear attention: $\Phi(\mathbb{Q}) \Phi(\mathbb{K})^\top \in \mathbb{R}^{(NL) \times (NL)}$ is the full token-level attention matrix (dot products between all effective queries and all effective keys), the mask $(\mathcal{A} \odot \mathbf{M})^{\uparrow L}$ zeros out future-chunk and enforces momentum-based decay, and the matrix multiplication with $\tilde{\mathbb{V}}$ aggregates the value vectors weighted by attention scores. This is recognizable as standard linear attention with a causal and momentum-weighted mask.

Proof of equivalence (Appendix H.2). The paper provides a proof that this parallel formulation is equivalent to the sequential recurrence. The proof works by unrolling the momentum recurrence (showing that the momentum accumulator at chunk $t$ is a weighted sum of all past gradients), unrolling the weight recurrence (showing that the weight matrix is the sum of momentum-accumulated outer products), and matching the resulting output expression with the parallel form's matrix multiplication. The key insight is that the momentum matrix $\mathcal{A}$ exactly captures the cumulative momentum coefficients $\beta_{ij}$ from Theorem 5.3, and the causal mask $\mathbf{M}$ enforces the temporal ordering.

Inference throughput improvements (Table 2). The paper measures the inference throughput of each ablation variant's TTT layer (in tokens per second, single batch) for the language modeling task. The recurrent implementations show progressively better throughput as components are removed: from 4.30M tokens/sec for the original LaCT baseline, to 10.60M for Variant 1 (updating only last layer), to 11.02M for Variant 2 (removing weight normalization), up to 89.67M for Variant 6 (standard linear attention). The parallel implementations (applicable from Variant 2 onward, since Variant 1 still uses weight normalization which breaks associativity) further improve throughput: Variant 2 parallel achieves 30.18M tokens/sec (2.7× over recurrent), Variant 6 parallel achieves 124.6M tokens/sec (1.39× over recurrent Variant 6, and 29× over the original LaCT baseline). The paper highlights the $4.0\times$ improvement specifically for the parallel form of Variant 6 compared to the recurrent form of the original LaCT (124.6M vs. 30.18M, though the text states "up to $4.0\times$ inference throughput" comparing Variant 2 parallel to the original LaCT recurrent: 30.18/4.30 ≈ 7.0×, suggesting the $4.0\times$ figure may refer to a different comparison or a different batch size setting—the paper does not specify the exact comparison for this claim).

Training speedup (Figure 4). Beyond inference, the paper shows that the parallel formulation also accelerates training. Comparing the original LaCT-TTT (recurrent) with the parallel form of Variant 2 (static kernel, no weight normalization), the parallel form achieves a $1.19\times$ end-to-end training speedup while maintaining comparable convergence (training loss vs. wall-clock time). This demonstrates that the efficiency benefits extend beyond inference to the training phase, where recurrent TTT implementations have been a significant bottleneck.

Why not all TTT variants are parallelizable (Appendix I). The paper analyzes two cases that prevent parallelization:

Dynamic kernel function (Appendix I.1): When $W_0$ and $W_2$ are updated in the inner loop (as in the original LaCT), the kernel function $\phi_t(\cdot)$ becomes history-dependent. The gradient update for $W_0$ involves $\text{silu}'(K_t W_{0,t})$ , which depends on $W_{0,t}$ , which in turn depends on previous updates involving $\text{silu}'(K_{t-1} W_{0, t-1})$ , creating nested nonlinear dependencies. This prevents expressing the output as a simple sum over history with fixed weights, breaking associativity.
Weight normalization (Appendix I.2): Even when only $W_1$ is updated, weight normalization creates a nested structure $S_{t+1} = \text{Norm}(S_t + \text{update})$ where each step's normalization depends on the fully normalized previous state. Because normalization is not associative ( $\text{Norm}(A+B) \neq \text{Norm}(A) + \text{Norm}(B)$ ), the computation cannot be parallelized via prefix scan.

These analyses clarify the boundary conditions for the parallel formulation: it applies when the kernel function is static and weight normalization is absent—precisely the conditions of Variants 2–6 in the ablation trajectory. This provides a concrete guideline for practitioners: to obtain parallelizable TTT, fix the kernel function parameters and remove weight normalization, accepting the small performance trade-off shown in Table 2.

4. Key Insights and Innovations

Innovation 1: The Memorization Hypothesis Is Empirically Falsifiable—and Falsified

The paper's most distinctive intellectual move is not the linear-attention reduction itself, but the prior step: systematically subjecting the prevailing memorization-based interpretation of TTT to direct empirical tests and showing that it fails every one. This is a diagnostic contribution rather than a constructive one, and it changes the conversation by establishing that the dominant conceptual framework in the TTT literature is not merely imprecise but fundamentally incorrect.

What the field assumed. Prior to this work, the storage-and-retrieval interpretation of TTT was essentially unchallenged. The inner loop was described as "memorizing" key–value associations (Sun et al., 2025; Behrouz et al., 2024), architectural complexity was justified by the goal of improving "memorization fidelity" (Zhang et al., 2025; Han et al., 2025), and the mechanism was framed as online meta-learning where the model learns to store information at test time (Finn et al., 2017; Metz et al., 2018). This was the default conceptual starting point for essentially all TTT-KVB research. No prior work had asked: what would constitute evidence against this interpretation, and do converged TTT models exhibit the properties that memorization implies?

What the paper does differently. Rather than arguing against the memorization view in the abstract, the paper identifies four specific, testable predictions that follow directly from the storage-and-retrieval hypothesis and tests each one empirically:

If the inner loop performs memorization, then better inner-loop optimization (lower loss) should improve task performance. Result: the opposite holds—more inner-loop steps consistently degrade downstream performance (Figure 1, across both language modeling and novel view synthesis).
If the inner loop performs memorization via gradient descent, then replacing descent with ascent should catastrophically break the mechanism. Result: gradient ascent preserves or slightly improves performance across all three tasks tested (Table 1), despite increasing inner-loop loss.
If queries retrieve stored key-value information, then queries must inhabit the same semantic space as the keys used during storage. Result: queries and keys exhibit pronounced distributional mismatch in converged TTT models (Figure 2, t-SNE visualizations), meaning queries are evaluated out-of-distribution relative to the optimization domain.
If queries perform retrieval, then replacing queries with keys should collapse the attention mechanism (as it does in standard attention). Result: the substitution has negligible effect on performance (Table 1).

Why this is a fundamental contribution, not incremental. This is not a refinement of existing understanding—it is a refutation. Collectively, these four results rule out both the "storage" aspect of memorization (gradient ascent preserves performance, so faithful key–value fitting is unnecessary) and the "retrieval" aspect (replacing Q with K doesn't matter, and Q/K distributions don't overlap). A mechanism cannot be described as storage-and-retrieval if it functions identically when the storage objective is inverted and the retrieval signal is removed. This forces the field to either abandon the memorization interpretation entirely or fundamentally redefine what "memorization" means—the paper argues for the former.

The significance extends beyond correcting a conceptual error. The memorization interpretation has been the primary driver of architectural complexity in the TTT literature: deeper inner-loop MLPs, sophisticated optimizers, per-token learning rates, and normalization schemes were all justified by the goal of improving storage fidelity. By demonstrating that this entire conceptual foundation is empirically unsupported, the paper opens the door to a fundamentally different design philosophy—one based on learned feature mixing rather than explicit memory formation—which is exactly what the linear-attention framework provides.

A note on methodology. The paper's falsification strategy is notable for its completeness: rather than identifying a single anomaly and arguing that the memorization interpretation must therefore be wrong, it systematically rules out both the storage and retrieval components independently. This makes the argument robust to auxiliary assumptions—one cannot defend the memorization view by, say, claiming that gradient ascent somehow still enables storage through a different mechanism, because the retrieval tests also fail. The interpretation fails on both ends of the storage-and-retrieval pipeline.

Innovation 2: The Inner Loop as a Structured Attention Operator Constructor

The paper's analytical contribution—that TTT can be rewritten as linear attention—is not itself novel as a mathematical observation (the single-linear-layer case was known; Sun et al., 2025). What is novel is the conceptual reframing this enables: the inner loop is not performing optimization in service of a downstream task, but is instead constructing a linear attention operator whose effective queries, keys, and values are dynamically computed through the gradient descent dynamics. This shifts TTT from being understood as a learning algorithm to being understood as a representation construction mechanism.

What prior work recognized. The known equivalence between single-layer TTT with MSE loss and linear attention was treated as a boundary case—a trivial algebraic coincidence that breaks down once the inner loop becomes more sophisticated. The field's implicit assumption was that the additional complexity of multi-layer MLPs, momentum, and alternative loss functions was exactly what made TTT different from and more powerful than linear attention. Under the memorization interpretation, these components were essential for increasing storage capacity and optimization quality.

What the paper contributes conceptually. The paper demonstrates that the linear-attention equivalence is not a fragile special case but a structural property of the gradient-descent inner loop that persists through multi-layer kernel functions (Theorem 5.1), sequential processing (Theorem 5.2), and momentum-based optimization (Theorem 5.3). The key insight is that the gradient descent update on the final linear layer always induces an outer-product accumulation of the form (effective key)ᵀ × (effective value), regardless of what happens in the kernel function or how the gradient is computed. The additional components—deeper MLPs, momentum, per-token learning rates, gradient orthogonalization—modify how the effective keys and values are computed, but do not change the structure of the computation.

This reframing matters because it reveals that TTT's apparent complexity is largely superficial. The SwiGLU MLP in LaCT, with its three weight matrices, silu activations, and Muon orthogonalization, is not a fundamentally different mechanism from standard linear attention—it is a particular learned parameterization of it. The inner loop's contribution is not to "store" information but to define a mapping from raw inputs (q, k, v) to effective representations (q̂, k̂, v̂) that are then combined through a linear attention computation. This is a fundamentally different conceptual model that redirects attention from optimization quality (how well does the inner loop converge?) to representation quality (what effective attention operator does the inner loop induce?).

Why this distinguishes the paper from prior reductions. Prior work that identified the single-layer equivalence (Sun et al., 2025) presented it as a special case—an interesting observation about a simplified setting. This paper promotes the observation to a unifying principle that explains the behavior of production-grade TTT architectures. The difference is analogous to recognizing that a specific linear regression is a special case of least squares versus recognizing that all linear models with squared-error loss have the same normal equations regardless of basis function complexity. The latter is a structural insight that generalizes; the former is a coincidence.

The practical consequence—demonstrated in Section 6.1—is that many components added to TTT architectures under the memorization interpretation are redundant precisely because they do not change the underlying attention structure. This is a rigorous explanation for why simplifications work, not just an empirical finding that they do.

Innovation 3: Verifier Over-Optimization in TTT—A New Diagnostic Concept

This paper does not use the term "verifier over-optimization," but the concept it identifies in Section 4.1—that improving the inner-loop objective degrades downstream performance—is structurally identical to the reward hacking phenomenon documented in RLHF and the PRM over-optimization identified in test-time compute scaling research. The paper's contribution is to identify this phenomenon in a new context (TTT inner loops) and provide a mechanistic explanation for it: the degradation arises from a train-test mismatch in the induced attention operator rather than from any failure of memorization.

What the field assumed. Under the memorization interpretation, the inner-loop loss is a proxy for storage quality. More gradient steps should produce better key–value fitting, which should improve task performance. If this assumption held, one would expect a monotonically increasing (or at least non-decreasing) relationship between inner-loop optimization and downstream accuracy—saturating at worst, but certainly not degrading.

What the paper shows. Figure 1 demonstrates a clear inverse relationship: as the number of inner-loop steps increases, inner-loop loss decreases (better fitting) but task performance consistently degrades (worse perplexity on language modeling, lower PSNR on novel view synthesis). This is not saturation or diminishing returns—it is active harm from optimization.

The mechanistic explanation (from the linear-attention perspective). The number of inner-loop steps is a hyperparameter of the attention operator—it determines how many gradient updates are applied to the kernel function parameters Θ before the effective query is computed. During training, the model is optimized with a specific number of inner-loop steps, which induces a specific linear attention operator that the downstream loss optimizes over. At inference time, changing the number of steps produces a different attention operator—different effective queries, keys, and values—than the one the model was trained for. The degradation is not because memorization fails but because the inference-time computation diverges from the training-time computation.

Why this is a diagnostic innovation, not just an empirical observation. The concept of "over-optimization" in TTT inner loops provides a principled explanation for why increasing inner-loop capacity (more steps, deeper MLPs, better optimizers) can be counterproductive. It identifies a mismatch as the root cause rather than a capacity limitation. This diagnostic has direct practical implications: it suggests that the path to better TTT performance is not to improve the inner-loop optimizer but to ensure consistency between training-time and inference-time inner-loop configurations—a design principle that is invisible under the memorization interpretation.

The finding also connects TTT to broader phenomena in machine learning where optimizing a proxy objective past a certain point degrades performance on the true objective (reward hacking in RLHF, PRM over-optimization in verifier-guided search). This connection had not been previously drawn for TTT, and it suggests that lessons from those domains—such as early stopping, KL regularization, or training with the same optimization budget used at inference—may transfer to TTT design.

Innovation 4: Decomposition into Core Linear Attention + Optional Enhancements

The paper's ablation trajectory (Section 6.1) is more than an empirical study—it is a principled decomposition of TTT into an essential linear attention core and a set of optional enhancements, each of which can be justified (or not) independently. This decomposition is novel because it answers a question that the memorization interpretation could not even frame: which components of TTT are essential to the mechanism and which are accidents of the memorization-based design philosophy?

What prior work lacked. Under the memorization interpretation, architectural components were justified by their contribution to storage fidelity. A deeper MLP stores more information; momentum smooths the memory formation process; weight normalization stabilizes the stored representations. There was no framework for distinguishing which of these justifications were valid and which were post-hoc rationalizations. As a result, the design space was explored through trial-and-error, with components added based on intuitive appeal rather than structural necessity.

What the decomposition reveals. The six-step ablation trajectory identifies three categories of components:

The essential core: a linear final layer updated via gradient descent on key-value pairs, which induces a linear attention operator. This is present in all TTT variants and is what makes TTT functionally equivalent to linear attention.
Context-dependent enhancements: components that modify the effective query/key/value computation and provide genuine but task-specific benefits. The kernel function (multi-layer MLP) provides modest gains on novel view synthesis (PSNR drops 0.22 dB when removed; Table 2, Variant 3 vs. Variant 2) but negligible benefit on language modeling. Gradient orthogonalization provides meaningful gains on language modeling (perplexity increases 0.83 when removed; Table 2, Variant 6 vs. Variant 5) but is not used in ViTTT.
Redundant components: elements that are functionally absorbed into learned projections or that provide benefits that can be achieved through simpler means. Per-token learning rates are absorbed into learned value projections (Variant 4). Momentum's recency-weighting is unnecessary when keys and values are already learned (Variant 5). Weight normalization is uncommon in linear attention and provides minimal benefit (Variant 2).

Why this is a fundamental contribution, not just an ablation study. The decomposition transforms TTT architecture design from an artisanal process guided by the memorization metaphor into a structured engineering discipline guided by the linear-attention framework. A practitioner building a new TTT variant can now start from standard linear attention (Variant 6), add only the enhancements that provide genuine benefit for their specific task (kernel function if representational capacity is needed, orthogonalization if training stability matters), and avoid the redundant components entirely. This is a principled design methodology rather than a collection of empirical findings.

The decomposition also clarifies the relationship between TTT and the broader linear attention literature. The essential core is linear attention; the optional enhancements are exactly the design axes along which TTT variants differ from standard linear attention and from each other. This means improvements from the linear attention literature (better decay mechanisms, more efficient chunk-parallel formulations, hardware-optimized kernels) can be directly incorporated into TTT, and conversely, lessons from TTT (the benefits of gradient orthogonalization, the utility of learned kernel functions) can inform linear attention design.

Anchoring to evidence. Table 2 provides the quantitative basis for this decomposition: it shows the cumulative effect of removing each component and identifies which removals cause performance changes and which do not. The fact that Variant 6 (standard linear attention) achieves comparable performance to the original LaCT baseline on novel view synthesis (-0.21 dB PSNR) and ViTTT on image classification (+0.20% accuracy) while performing moderately worse on language modeling (+0.37 perplexity) demonstrates both that the essential core is linear attention and that the optional enhancements have genuine but domain-specific value. This is a more nuanced and practically useful picture than either "TTT is just linear attention" or "TTT is fundamentally different from linear attention."

Innovation 5: Associativity as the Boundary Condition for TTT Parallelization

The paper's analysis of when TTT admits parallel implementation (Section 6.2 and Appendix I) identifies a precise and non-obvious boundary condition: TTT becomes parallelizable when and only when the state update is associative. This is not a performance optimization trick—it is a structural insight that connects the mathematical properties of the inner-loop update to the computational properties of the implementation.

What prior work assumed. Under the memorization interpretation, TTT is inherently sequential: each token must be "stored" before the next can be processed, and retrieval occurs only after storage. The default implementation is therefore recurrent, token-by-token. The possibility of parallelization was either not considered or assumed impossible due to the sequential nature of the storage process. The known single-layer equivalence to linear attention (Sun et al., 2025) was recognized as parallelizable, but this was seen as a special property of the simplified case that didn't generalize.

What the paper shows. The paper proves that the parallelizability boundary is determined by two specific architectural choices: whether the kernel function parameters Θ are updated in the inner loop, and whether weight normalization is applied. When Θ is fixed (only the final-layer weight is updated) and normalization is absent, the state update becomes a simple cumulative sum of outer products, which is trivially associative and thus parallelizable via prefix scan. When Θ is updated, the kernel function becomes history-dependent, introducing nested nonlinearities that break associativity. When normalization is applied, the nested Norm(·) operations break associativity even if Θ is fixed.

Why this is a conceptual innovation, not just an engineering result. The paper identifies associativity—a mathematical property of the update operation—as the fundamental criterion for parallelization. This is far more general than the specific architectures tested: any future TTT variant, regardless of its inner-loop complexity, will be parallelizable if and only if its state update satisfies the associative property. This provides a principled test for parallelizability that does not depend on the memorization interpretation or any specific architecture.

The analysis also reveals a tradeoff that was previously invisible: updating the kernel function parameters (which enables dynamic, history-dependent effective query/key representations) comes at the cost of parallelizability (Appendix I.1). This is a concrete engineering decision—do you want the representational flexibility of a dynamic kernel, or the throughput of parallel execution?—that practitioners can now make explicitly rather than implicitly inheriting the recurrent default.

Distinguishing from prior parallel linear attention work. Chunk-parallel formulations exist for various linear attention variants (Yang et al., 2024b; the DeltaNet literature). What distinguishes this paper's contribution is the demonstration that the parallelizability boundary in TTT is architectural rather than algorithmic—it depends on which parameters are updated in the inner loop, not on which parallel algorithm is used. This means parallelizability can be designed into TTT architectures from the start (by fixing Θ and removing normalization) rather than retrofitted onto inherently sequential designs.

Anchoring to evidence. Table 2 shows the inference throughput consequences: Variant 2 (static kernel, no normalization—associative) achieves 30.18M tokens/sec in parallel form versus 11.02M recurrent (2.7× improvement), while the original LaCT (dynamic kernel, normalization—non-associative) cannot be parallelized at all. The 1.19× end-to-end training speedup (Figure 4) demonstrates that associativity benefits extend beyond inference to the training phase, where recurrent TTT implementations have been a significant practical bottleneck.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses three distinct tasks and datasets. For language modeling, training is performed on 100B tokens from the FineWeb-Edu dataset (Penedo et al., 2024), with evaluation on 2.5B tokens from the Book-3 dataset (Gao et al., 2020). For novel view synthesis, training and evaluation use the RealEstate10K dataset (Zhou et al., 2018) with 2 input views and 6 target views for training, and 2 input views and 3 target views for evaluation, all at 128×128 resolution. For image classification, training and evaluation use the ImageNet-1K dataset (Deng et al., 2009) at standard resolution. The choice of three diverse tasks—language modeling, 3D vision, and image classification—is deliberate: it tests whether the paper's claims about TTT's underlying mechanism generalize across modalities and problem types, not just within a single benchmark.
Base model(s). For language modeling, the baseline is LaCT-LLM (Zhang et al., 2025) with 760M parameters, trained on 8 NVIDIA A100 GPUs with a batch size of 4 per GPU for 20K iterations (~56 hours). For novel view synthesis, the baseline is LaCT-NVS (Zhang et al., 2025) with 12 layers, 768 hidden dimension, and 114M parameters, trained on 4 NVIDIA A100 GPUs with batch size 128 per GPU for 20K iterations (~38 hours). For image classification, the baseline is ViTTT-B (Han et al., 2025) with 90M parameters, trained on 2 NVIDIA H100 GPUs with batch size 256 per GPU for 60 epochs (~16 hours). These baselines represent the state of TTT-KVB architectures at the time of writing—LaCT being the most prominent language/vision TTT variant and ViTTT being the leading vision-specific TTT design. The paper does not introduce new architectures; it reinterprets and simplifies existing ones. All hyperparameters follow the original published configurations unless explicitly ablated.
Metrics. Three task-specific metrics are reported. For language modeling: perplexity (lower is better), computed on 2.5B tokens from Book-3. For novel view synthesis: Peak Signal-to-Noise Ratio, or PSNR, in dB (higher is better), computed between rendered and ground-truth novel views. For image classification: Top-1 accuracy as a percentage (higher is better), computed on the ImageNet-1K validation set. These metrics are standard in their respective domains and directly comparable to prior published results for LaCT and ViTTT. The paper also reports tokens per second (both recurrent and parallel implementations) as an efficiency metric for the LLM task, measured on the TTT layer specifically (not end-to-end model throughput).
Baselines. The primary baselines are the original implementations of LaCT (Zhang et al., 2025) for language modeling and novel view synthesis, and ViTTT (Han et al., 2025) for image classification. These serve as the "unmodified TTT" reference points against which all ablations are compared. The paper does not introduce external baselines (e.g., standard transformers, other linear attention variants) because the goal is not to claim that TTT outperforms other architectures, but rather to demonstrate that complex TTT variants can be simplified to standard linear attention with minimal performance loss. The relevant comparison is therefore: complex TTT versus progressively simplified TTT, not TTT versus attention.
Generation budget / compute accounting. There is no generation budget in the standard sense because the paper does not study test-time compute scaling. Instead, the relevant resource metrics are: (1) training compute, reported as GPU-hours for each task; (2) inference throughput, measured in tokens per second for the TTT layer specifically (Table 2, both recurrent and parallel implementations); and (3) end-to-end training wall-clock time (Figure 4). The ablation trajectory (Section 6.1) holds model size and training data constant across variants, so performance differences are attributable to architectural changes rather than compute differences. The efficiency comparisons in Section 6.2 are apple-to-apple: the recurrent and parallel implementations process the same inputs and produce mathematically equivalent outputs (proven in Appendix H.2), so throughput differences reflect genuine implementation efficiency rather than algorithmic approximations.
Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. All results in Table 2 and Figures 1, 3, and 4 are single-run measurements. For the language modeling task, perplexity is evaluated on 2.5B tokens, which provides a stable estimate (the effective sample size is large enough that variance across evaluation tokens is negligible). For novel view synthesis and image classification, the test sets are fixed (RealEstate10K test split and ImageNet-1K validation set, respectively), and results are reported as single-point estimates. The paper does not discuss statistical significance or run-to-run variance, which is a limitation: for the smaller-magnitude differences in the ablation study (e.g., Variant 3 vs. Variant 4 on language modeling: 16.23 vs. 16.12 perplexity), it is unclear whether the observed differences exceed training noise. The paper implicitly treats perplexity differences below ~0.1 as within noise, though this threshold is not formally justified.

Main Quantitative Results

The paper's empirical contributions are organized around two axes: (1) the four contradiction experiments that refute the memorization interpretation (presented in Section 4, but with results quantified here), and (2) the progressive ablation trajectory that reduces complex TTT to standard linear attention (Section 6.1 and Table 2). Unlike most ML papers, the "main results" are not a single headline number but rather a systematic demonstration that complex TTT variants are functionally equivalent to linear attention plus optional task-specific enhancements.

The Four Contradiction Experiments (Section 4)

Inner-loop optimization versus task performance (Figure 1). The paper varies the number of inner-loop gradient steps at inference time for pretrained LaCT models and measures both inner-loop loss and downstream task performance. For language modeling with LaCT-LLM, perplexity consistently increases (worsens) as the number of inner-loop steps increases, despite inner-loop loss monotonically decreasing. The exact perplexity values are not reported as a table, but Figure 1 (left panel) shows a clear monotonic degradation with increasing steps. For novel view synthesis with LaCT-NVS, PSNR consistently decreases as inner-loop steps increase (Figure 1, right panel), again despite lower inner-loop loss. This inverse relationship holds across both tasks and constitutes the paper's first empirical contradiction: if the inner loop were performing useful memorization, better fitting should not systematically degrade downstream performance.

Gradient ascent experiment (Table 1). The paper replaces gradient descent with gradient ascent in the inner loop for three separate models and tasks. Quantitative results from Table 1:

LaCT-LLM (language modeling): Baseline perplexity 16.43 → Gradient ascent 16.19 (a 0.24 perplexity improvement, not a degradation).
LaCT-NVS (novel view synthesis): Baseline PSNR 25.94 → Gradient ascent 25.85 (a 0.09 dB decrease, essentially within noise).
ViTTT (image classification): Baseline Top-1 accuracy 79.34% → Gradient ascent 79.61% (a 0.27 percentage point improvement).

The paper notes that gradient ascent consistently increases inner-loop loss (as expected—it's optimizing in the wrong direction), making the preservation or improvement of task performance particularly striking. These results are the most direct refutation of the memorization hypothesis: a mechanism that genuinely relies on faithful key–value storage should fail catastrophically when the storage objective is inverted, yet TTT performance is essentially unchanged.

Distributional asymmetry between Q and K (Figure 2). The paper collects query and key vectors across all tokens and all layers from a pretrained LaCT model on the NVS task, then visualizes their distributions using t-SNE. Figure 2 shows t-SNE plots for Q/K pairs and V/O (value/output) pairs across layers. The visualization reveals a pronounced and consistent mismatch: Q and K inhabit different regions of the t-SNE space at every layer. For a retrieval mechanism to function, the function being queried must be evaluated on inputs from the same distribution it was optimized on—otherwise the outputs are out-of-distribution and cannot be interpreted as reliable retrieval. The paper does not report a quantitative distributional distance metric (e.g., KL divergence, maximum mean discrepancy), which is a limitation—the evidence is visual rather than statistical. However, the qualitative separation is sufficiently stark (the t-SNE clusters for Q and K are clearly disjoint) that the paper treats it as definitive evidence of distributional mismatch.

Replacing Q with K (Table 1). In standard attention, replacing queries with keys collapses the mechanism because attention weights become dominated by self-similarity (each key attending most strongly to itself). The paper tests whether TTT exhibits the same sensitivity. Quantitative results from Table 1:

LaCT-LLM: Baseline perplexity 16.43 → Q replaced with K 16.18 (a 0.25 perplexity improvement).
LaCT-NVS: Baseline PSNR 25.94 → Q replaced with K 25.95 (a 0.01 dB improvement, within noise).
ViTTT: Baseline accuracy 79.34% → Q replaced with K 79.18% (a 0.16 percentage point decrease, within noise).

The near-perfect preservation of performance when queries are replaced with keys demonstrates that TTT does not rely on query-key similarity for its operation. This contradicts the retrieval aspect of the memorization interpretation directly: if queries are not functioning as retrieval cues (since replacing them with keys—which are not "queries" in any semantic sense—doesn't matter), then the retrieval framing is incorrect.

Collective interpretation. The four experiments collectively rule out both the storage and retrieval aspects of the memorization hypothesis. Gradient ascent rules out storage (the objective doesn't need to be minimized). Replacing Q with K and the distributional asymmetry together rule out retrieval (queries don't need to be semantically similar to keys, and replacing them with keys doesn't change behavior). The inner-loop optimization paradox (Figure 1) rules out the assumed relationship between inner-loop quality and downstream performance. Together, these results leave essentially no room for a storage-and-retrieval interpretation: TTT does not store key-value information and does not retrieve it using queries.

Progressive Ablation: Reducing TTT to Linear Attention (Section 6.1, Table 2, Figure 3)

The core empirical contribution of Section 6 is the six-step ablation trajectory, summarized in Table 2, with additional detail in Figure 3 (perplexity across multiple sequence lengths for the LLM task). The paper reports results for all three tasks (LLM, NVS, image classification) at each ablation step.

Step 1: Update only the last-layer parameters. The paper restricts the inner-loop update to only the final-layer weight matrix W₁, keeping the kernel function parameters Θ fixed. This is the paper's proposed first simplification and, notably, it improves performance across all three tasks:

LaCT-LLM: Perplexity drops from 16.43 (baseline) to 15.93 (a 0.50 improvement).
LaCT-NVS: PSNR increases from 25.94 to 25.97 (a 0.03 dB improvement).
ViTTT: Accuracy increases from 79.34% to 79.63% (a 0.29 point improvement).

This is a striking result: removing the ability to update kernel function parameters in the inner loop—which, under the memorization interpretation, reduces the model's capacity to adapt its representations to each token—actually improves performance. From the linear-attention perspective, this makes sense: fixing Θ makes the kernel function static, which eliminates a source of train-test mismatch (the kernel function behaves the same way at every token position during training and inference). The concurrent throughput improvement is substantial: from 4.30M tokens/sec (baseline recurrent) to 10.60M tokens/sec (Variant 1 recurrent), a 2.5× speedup from this simplification alone.

Step 2: Remove weight normalization. After Step 1, normalization on Θ is a no-op (Θ is fixed). The paper removes the remaining normalization on W₁. Results:

LaCT-LLM: Perplexity increases from 15.93 to 16.31 (a 0.38 increase, but still 0.12 better than baseline).
LaCT-NVS: PSNR drops from 25.97 to 25.93 (a 0.04 dB decrease, within noise).
ViTTT: Accuracy unchanged at 79.63%.

The performance impact is minimal. The critical side effect is that this step makes the state update associative, enabling parallelization. The parallel implementation achieves 30.18M tokens/sec (vs. 11.02M recurrent), a 2.7× improvement from parallelization alone. This step is where the practical benefits of the linear-attention perspective begin to materialize: the recognition that weight normalization is unnecessary for performance but prevents parallelization enables an explicit engineering trade-off.

Step 3: Reduce multi-layer MLP to single linear layer. The kernel function φ(·) is removed entirely, exposing raw queries and keys. Results show task-dependent effects:

LaCT-LLM: Perplexity improves slightly from 16.31 to 16.23 (the kernel function was providing no benefit for language modeling).
LaCT-NVS: PSNR drops from 25.93 to 25.71 (a 0.22 dB decrease, suggesting the kernel function provides meaningful representational capacity for this 3D task).
ViTTT: Accuracy drops from 79.63% to 79.39% (a 0.24 point decrease).

The throughput improvements are substantial: recurrent throughput reaches 12.95M tokens/sec, parallel reaches 49.69M tokens/sec. The task-dependent performance impact is informative: the kernel function matters more for the novel view synthesis task, which involves spatial reasoning and 3D geometry—domain where the additional nonlinear transformations may genuinely help. For language modeling and image classification, raw query/key representations appear sufficient.

Step 4: Remove per-token learnable learning rates. The per-token learning rate η_t is replaced with a constant η. Results show essentially no impact:

LaCT-LLM: Perplexity improves from 16.23 to 16.12 (a marginal 0.11 improvement).
LaCT-NVS: PSNR unchanged at 25.70 (note: the value is reported as 25.70 here but 25.71 in Step 3; this 0.01 discrepancy may be a rounding artifact in the table).
ViTTT: Accuracy unchanged at 79.39%.

The paper notes this is consistent with ViTTT's independent finding that a constant learning rate of 1.0 suffices. The result confirms the linear-attention perspective's prediction: the per-token learning rate is absorbed into the learned value projection and provides no independent benefit.

Step 5: Remove momentum from SGD. Setting α_t = 0 recovers standard SGD without momentum. Results again show minimal impact, and in one case a slight improvement:

LaCT-LLM: Perplexity improves from 16.12 to 15.97 (a 0.15 improvement—removing momentum actually helps).
LaCT-NVS: PSNR unchanged at 25.70.
ViTTT: Accuracy unchanged at 79.39%.

The finding that removing momentum improves language modeling perplexity is notable: it suggests that momentum's recency-weighting effect is not merely unnecessary but slightly harmful, perhaps because it introduces a form of temporal bias that conflicts with the learned attention patterns. Throughput reaches 14.40M tokens/sec (recurrent) and 57.28M tokens/sec (parallel).

Step 6: Remove gradient orthogonalization. This eliminates the Muon operator ℳ(·) from the LaCT variants (ViTTT does not use Muon, so a different normalization is ablated—see table note). Results show the largest single-step performance impact:

LaCT-LLM: Perplexity increases from 15.97 to 16.80 (a 0.83 increase—the largest degradation in the entire trajectory).
LaCT-NVS: PSNR increases from 25.70 to 25.73 (a 0.03 dB improvement—orthogonalization was slightly harmful for NVS).
ViTTT: Accuracy increases from 79.39% to 79.54% (a 0.15 point improvement—the ablated normalization was slightly harmful).

The language modeling result (0.83 perplexity increase) is the most significant performance degradation in the entire trajectory and identifies gradient orthogonalization as genuinely important for language modeling with TTT. The paper's linear-attention framework explains this: orthogonalization ensures that each key-value outer product added to the state matrix has orthonormal structure, preventing rank collapse and maintaining well-conditioned state representations. For language modeling, where the state matrix must capture long-range dependencies across thousands of tokens, this conditioning likely matters more than for vision tasks with shorter effective sequence lengths. Throughput after this step reaches 89.67M tokens/sec (recurrent) and 124.6M tokens/sec (parallel).

The final reduction (Variant 6) compared to original baselines. After all six steps, the fully reduced TTT model (which is now exactly standard linear attention: o = q(W + Σ k_i^T v_i)) achieves:

LaCT-LLM: Perplexity 16.80 vs. baseline 16.43 (+0.37, or 2.3% degradation).
LaCT-NVS: PSNR 25.73 vs. baseline 25.94 (−0.21 dB, or 0.8% degradation).
ViTTT: Accuracy 79.54% vs. baseline 79.34% (+0.20 point improvement).

The paper characterizes this as "only minor performance degradation." For language modeling, a 0.37 perplexity increase on a 760M parameter model trained on 100B tokens is indeed modest—it represents roughly the difference one might expect from a slightly suboptimal learning rate or a marginally smaller model. For NVS and image classification, the differences are within what might be considered run-to-run variance. The practical implication is clear: practitioners can use standard linear attention (with all its implementation advantages, including parallel prefix scan, hardware-optimized kernels, and mature codebases) and expect performance within a few percent of complex TTT formulations.

Sequence length scaling (Figure 3). The paper evaluates perplexity at multiple sequence lengths (8k, 16k, 32k, and implied intermediate lengths) to test whether the simplifications affect TTT's ability to handle long contexts. Figure 3 shows perplexity curves for all six variants plus the baseline, evaluated on 2.5B tokens from Book-3. The curves are largely parallel: the ranking of variants is consistent across sequence lengths, with no variant showing disproportionate degradation at longer contexts. This is important because one of TTT's claimed advantages is linear-time computation with constant memory—if the simplifications broke this property, one would expect performance to diverge at longer sequences. The consistent gaps across sequence lengths suggest the simplifications do not fundamentally alter the sequence-length scaling behavior.

Throughput improvements from parallelization (Table 2, rightmost columns). The paper reports both recurrent and parallel inference throughput for each variant. Key comparisons:

Original LaCT (baseline): 4.30M tokens/sec, recurrent only (no parallel form possible due to weight normalization and dynamic kernel).
Variant 2 (static kernel, no normalization): 11.02M recurrent, 30.18M parallel → 2.7× improvement from parallelization.
Variant 6 (standard linear attention): 89.67M recurrent, 124.6M parallel → 1.4× improvement from parallelization, and 29× improvement over original LaCT recurrent.

The paper claims "up to 4.0× inference throughput" in the abstract and Section 6.2. The exact comparison for this figure is not explicitly stated in the text, but the largest ratio in Table 2 is 124.6M / 30.18M ≈ 4.1× (Variant 6 parallel versus Variant 1 parallel? but Variant 1 doesn't have a parallel form). The 4.0× figure likely refers to the Variant 2 parallel versus original LaCT recurrent comparison (30.18 / 4.30 ≈ 7.0×, which is larger) or perhaps to a specific batch size setting not detailed in the table. The paper would benefit from explicitly stating which comparison yields the 4.0× figure.

Training speedup (Figure 4). Beyond inference, the paper demonstrates that the parallel formulation also accelerates training. Figure 4 plots training loss versus wall-clock time for three configurations: original LaCT-TTT (recurrent), Variant 2 recurrent, and Variant 2 parallel. The parallel form achieves a 1.19× end-to-end training speedup while reaching comparable loss values at equivalent wall-clock times. The loss curves are reported for the LLM task only. This is a practically significant result: TTT training has been slow due to the sequential inner-loop updates, and the recognition that parallelization is possible (under the linear-attention perspective) directly translates to reduced training time.

Ablation Studies and Robustness Checks

Weight normalization removal (Steps 1→2): The removal of weight normalization is tested implicitly as part of the ablation trajectory (Table 2, Variant 1 → Variant 2). The performance impact is small: +0.38 perplexity on LLM, −0.04 dB on NVS, no change on image classification. The paper does not ablate this independently of Step 1 (i.e., it does not test removing weight normalization from the original LaCT with full inner-loop updates), so the possibility remains that weight normalization is more important when kernel function parameters are also being updated—a reasonable hypothesis given that weight normalization was originally introduced to stabilize training of deep inner-loop MLPs. The paper's Appendix I.2 provides the theoretical analysis of why weight normalization prevents parallelization, but the empirical question of whether it provides benefits in the full-update case is left open.

Multi-layer MLP depth (Step 3): The paper replaces the multi-layer SwiGLU MLP with a single linear layer. The effect is task-dependent: negligible for language modeling (+0.08 perplexity improvement from Variant 2 to Variant 3, though this "improvement" is within noise), meaningful for novel view synthesis (−0.22 dB), and modest for image classification (−0.24% accuracy). The paper does not explore intermediate depths (e.g., 2-layer vs. 3-layer vs. single-layer) or alternative kernel function architectures. A more granular ablation of MLP depth would strengthen the claim that the kernel function is genuinely optional rather than simply requiring a different architecture. The finding that NVS benefits from the kernel function while language modeling does not is consistent with the intuition that spatial reasoning tasks may benefit from learned input transformations, but the paper does not investigate why this asymmetry exists.

Per-token learnable learning rate (Step 4): Removed with essentially zero impact across all tasks. This is the cleanest redundancy result in the trajectory: the linear-attention framework predicts that η_t is absorbed into the learned value projection, and the empirical results confirm this prediction exactly. The paper does not test whether per-token learning rates provide benefit in the original LaCT configuration (with full inner-loop updates), but the fact that ViTTT independently converged on a constant learning rate suggests the redundancy is general.

Momentum removal (Step 5): Removed with minimal impact (+0.15 perplexity improvement on LLM, no change on NVS and classification). This result directly supports Theorem 5.3's implication that momentum merely remixes historical value vectors in a way that learned projections can already achieve. The slight improvement on language modeling when momentum is removed is an interesting negative result: it suggests that momentum's implicit recency bias may interfere with the learned attention patterns rather than helping them. The paper does not test different momentum factors (α values) to see whether a smaller α might be beneficial, nor does it test whether momentum provides benefits in the full-update case.

Gradient orthogonalization removal (Step 6): The most impactful ablation on language modeling (+0.83 perplexity increase). This is the strongest evidence that at least one component of the complex TTT formulation provides genuine, non-redundant benefits—specifically for language modeling. The paper does not test alternative orthogonalization schemes (e.g., spectral normalization, simple L2 normalization, or orthogonalization frequency—applying Muon every k steps rather than every step) to understand what aspect of the orthogonalization is important. The fact that orthogonalization helps language modeling significantly but is slightly harmful for NVS and not used in ViTTT at all suggests that its benefits are tied to the properties of long-range sequence modeling (maintaining well-conditioned state matrices over thousands of tokens) rather than being generally useful.

Static versus dynamic kernel function (Step 1 and Appendix I.1): The improvement from fixing kernel function parameters (Step 1: +0.50 perplexity improvement) is one of the paper's most striking results and is tested only on LaCT. The paper does not ablate this on ViTTT (which has a different kernel architecture) or other TTT variants (e.g., Titans). The analysis in Appendix I.1 explains theoretically why a dynamic kernel prevents parallelization, but the empirical finding that it hurts performance even in the recurrent setting is not fully explained—the linear-attention framework predicts train-test mismatch as the cause, but verifying this would require, e.g., measuring the consistency of effective query/key representations across training and inference. This is left as a hypothesis rather than a tested mechanism.

Parallel versus recurrent equivalence (Section 6.2 and Appendix H.2): The paper provides a formal proof (Appendix H.2) that the parallel formulation is mathematically equivalent to the sequential recurrence for Variants 2–6. This is not empirically verified beyond the fact that the parallel and recurrent implementations produce the same outputs by construction—the proof is analytical, not experimental. However, the paper does validate that the parallel implementation achieves the expected throughput improvements (Table 2, rightmost columns) without degrading model quality (Figure 4, where the parallel training curve matches the recurrent curve). A direct empirical check that recurrent and parallel implementations produce bitwise-identical outputs would strengthen confidence, though the mathematical proof is sufficient for the paper's claims.

Multiple sequence lengths (Figure 3): The paper evaluates all ablation variants at multiple sequence lengths on the LLM task. This serves as a robustness check: if any simplification disproportionately harmed long-range modeling, it would appear as diverging perplexity curves at longer lengths. Figure 3 shows consistent gaps across all lengths, confirming that the simplifications do not fundamentally alter TTT's sequence-length scaling behavior. The longest evaluated length is 32k tokens (implicit from the text discussing 32k sequence length). Testing at even longer lengths (e.g., 64k, 128k) would further validate the claim that TTT's linear-complexity property is preserved through the simplifications, but 32k is a reasonable practical upper bound for models of this scale.

ViTTT-specific ablations: The paper notes (Table 2 footnote) that ViTTT does not use gradient orthogonalization, so the Step 6 ablation for ViTTT removes gradient normalization instead. The finding that this removal improves accuracy (79.39% → 79.54%) suggests the original ViTTT's normalization was providing no benefit, consistent with the broader theme that normalization is unnecessary once the kernel function is simplified. However, the paper does not detail what "gradient normalization" entails in ViTTT or how it differs from LaCT's weight normalization, making this ablation harder to interpret.

Summary of non-obvious findings from ablations:

Restricting updates to the final layer improves performance (Step 1)—contrary to the intuition that more adaptable inner-loop parameters should help.
Momentum is slightly harmful for language modeling (Step 5)—counter to the widespread use of momentum in optimization.
Gradient orthogonalization is the single most important non-core component, but only for language modeling (Step 6)—identifying domain-specific rather than universal benefits.
The kernel function matters for 3D vision but not language (Step 3)—suggesting representational demands differ by modality.
Parallelization yields 2.7× throughput at minimal performance cost (Variant 2)—making a strong practical case for the simplified formulation even if one values the kernel function (one could keep Step 1's improvements and still parallelize by removing weight normalization).

Critical Assessment

The experimental design in this paper serves a different purpose than in typical ML papers that propose new methods. The goal is not to demonstrate state-of-the-art performance but to validate a conceptual framework. Consequently, traditional evaluation criteria (Is the model better than baselines? Are the gains statistically significant?) are less relevant than: Do the experiments genuinely support the claim that TTT is reducible to linear attention? Are the observed phenomena explained by the framework? Are there alternative explanations that were not ruled out? Below, I assess each major claim against the reported evidence.

Claim: The memorization-based interpretation of TTT is empirically falsified.

What the experiments demonstrate: The four contradiction experiments (Section 4) show that converged TTT models do not behave as a storage-and-retrieval system would. Gradient ascent preserves performance (Table 1), more optimization harms performance (Figure 1), queries and keys are distributionally distinct (Figure 2), and replacing queries with keys doesn't matter (Table 1). These are strong negative results against the memorization hypothesis.
What the experiments do not demonstrate: The experiments test what pretrained TTT models do at inference time. They do not test what constraints the training process imposes—it is possible, for instance, that the inner-loop optimization was important during training for establishing good representations, even if at inference time the exact direction of the gradient doesn't matter. The gradient ascent experiment shows that the sign of the gradient doesn't matter for a trained model, but it doesn't rule out that gradient descent (rather than ascent) was necessary during training to learn the projections that make this possible. This is not a weakness of the falsification argument per se—the storage-and-retrieval interpretation makes claims about what TTT does, not just about how it was trained—but it does mean that the memorization hypothesis might still have training-time validity even if it lacks inference-time validity.
What would strengthen the claim: Training a TTT model from scratch with gradient ascent in the inner loop. If the resulting model achieved comparable performance to gradient-descent-trained TTT, that would definitively rule out any role for gradient-based memorization even during training. Additionally, training a TTT model with random inner-loop updates (not gradient-based at all) would test whether any structured update is sufficient, regardless of the loss function. The paper does not run these experiments, likely due to computational cost (training 760M parameter LLMs is expensive), but they would substantially strengthen the falsification argument.

Claim: Complex TTT variants can be analytically rewritten as linear attention operators.

What the experiments demonstrate: The Theorems of Section 5 are mathematical proofs, not empirical results. The experimental validation is indirect: the progressive ablation (Table 2) shows that removing components not present in standard linear attention has minimal impact on performance, which is consistent with (but does not prove) the claim that those components were implementing a linear attention computation all along.
What the experiments do not demonstrate: The paper does not empirically verify that the analytical rewriting is correct for any specific model instance. For example, it does not take a trained LaCT model, compute the effective queries/keys/values according to the formulas in Section 5.3, and show that the resulting linear attention computation produces the same outputs as the original TTT inner loop. Such a verification would require instrumenting the model to capture intermediate values (φ_t(k), g_t(k), etc.) and comparing the computed outputs, which is implementationally straightforward but was apparently not done. The analytical proofs stand on their own mathematically, but empirical verification would protect against implementation errors or unstated assumptions.
Caveat on the "general" nature of the reduction: The analytical framework assumes a linear, bias-free final layer (Section 5.1). While this holds for LaCT and ViTTT, it does not hold for all TTT variants—some architectures use nonlinear output layers or include bias terms. The paper is explicit about this limitation, but readers familiar with the broader TTT literature should be aware that the reduction does not automatically apply to all TTT-KVB methods.

Claim: The simplifications reduce complex TTT to standard linear attention with only minor performance degradation.

What the experiments demonstrate: Table 2 shows that Variant 6 (standard linear attention) achieves perplexity 16.80 vs. 16.43 for LaCT-LLM (+2.3%), PSNR 25.73 vs. 25.94 for LaCT-NVS (−0.8%), and accuracy 79.54% vs. 79.34% for ViTTT (+0.25%). These are indeed modest differences, especially considering the dramatic simplification (from multi-layer MLP with momentum, orthogonalization, and normalization to a single linear transformation).
What the experiments do not demonstrate: The paper tests this claim on exactly two TTT architectures (LaCT and ViTTT) and three tasks. Whether the simplification generalizes to other TTT variants (Titans, TTT with different inner-loop architectures) and other tasks (code generation, long-range document understanding, multi-turn dialogue) is an open question. The language modeling experiment uses a 760M parameter model trained on 100B tokens—at this scale, many architectural choices have modest impact on perplexity. A larger model (e.g., 7B parameters) or a more data-constrained setting might reveal larger gaps between complex TTT and standard linear attention. The paper's claim is appropriately qualified ("only minor performance degradation") but should be contextualized: it has been demonstrated for the specific architectures, tasks, and scales tested.
The language modeling gap deserves scrutiny: A 0.37 perplexity increase is not negligible in absolute terms. At the 760M parameter scale, this roughly corresponds to the gap between models trained with slightly different learning rates or data mixtures. A practitioner deciding whether to use LaCT-TTT or standard linear attention for a language modeling application should weigh this 0.37 perplexity against the implementation simplicity and throughput gains of standard linear attention. The paper frames this trade-off correctly ("only minor degradation") but does not explore whether the gap might be closed by, e.g., tuning the linear attention variant's hyperparameters more carefully. Since the linear attention variant (Variant 6) was obtained by subtraction (removing TTT components), not by optimization (tuning the linear attention configuration for best performance), it is possible that a properly optimized linear attention baseline would close or eliminate the gap entirely.

Claim: The parallel formulation achieves up to 4.0× inference throughput improvement.

What the experiments demonstrate: Table 2 shows substantial throughput improvements from parallelization. Variant 2 parallel achieves 30.18M tokens/sec versus 11.02M recurrent (2.7×). Variant 6 parallel achieves 124.6M tokens/sec versus 89.67M recurrent (1.4×). The overall improvement from original LaCT recurrent (4.30M) to Variant 6 parallel (124.6M) is roughly 29×, though this combines effects of simplification (removing components that slow down the recurrent implementation) and parallelization.
What requires clarification: The paper's headline figure of "4.0×" does not cleanly correspond to any single comparison in Table 2. The largest parallel-to-recurrent ratio for a single variant is 30.18/11.02 ≈ 2.7× (Variant 2). The ratio 124.6/30.18 ≈ 4.1× compares Variant 6 parallel to Variant 2 parallel—different architectural variants, not recurrent vs. parallel implementations of the same variant. The 4.0× figure may refer to Variant 6 parallel versus original LaCT recurrent (124.6/30.18? no, that's 4.1× and the denominator is wrong) or to a specific batch size configuration not reported in the main text. This ambiguity weakens the headline claim slightly.
The throughput measurements are for the TTT layer only. The paper reports tokens per second for the TTT layer in isolation, not end-to-end model throughput. Since a transformer model contains many other operations (feed-forward layers, layer normalization, embedding, etc.), the end-to-end speedup from parallelizing the TTT layer will be smaller than the TTT-layer-specific speedup. The 1.19× end-to-end training speedup reported in Figure 4 provides a more realistic picture: even a nearly 3× improvement in the TTT layer specifically translates to a roughly 19% end-to-end improvement, consistent with Amdahl's law (the TTT layer is only one component of the model).

Missing experiments that would strengthen the paper:

Direct verification of the analytical reduction. Take a trained LaCT model, extract the effective queries, keys, and values according to the formulas, and verify that the linear attention computation produces identical outputs. This would provide empirical validation of the analytical framework beyond the indirect evidence of the ablation trajectory.
Comparison to optimized linear attention baselines. The ablation trajectory reduces TTT to linear attention, but does not compare against carefully tuned standard linear attention variants (e.g., with learned decay factors, multi-head formulations, etc.). If a well-tuned linear attention baseline matches or exceeds Variant 6, that would strengthen the claim that TTT offers no fundamental advantage over linear attention.
Testing at larger scale. All experiments use models under 1B parameters. At the 7B+ scale, the relative importance of different architectural components may shift. The paper's claims about which components are "redundant" are necessarily specific to the scale tested.
Training TTT with gradient ascent from scratch. The most definitive test of the memorization hypothesis would be to train a model where the inner loop uses gradient ascent throughout training. If such a model achieves comparable performance to standard TTT, the memorization interpretation would be definitively refuted even for training.
Measuring the train-test mismatch hypothesis. The paper attributes the degradation from more inner-loop steps (Figure 1) to a mismatch between training and inference configurations. This could be tested by training models with varying numbers of inner-loop steps and measuring whether the optimal inference-time step count matches the training-time count. No such experiment is reported.
Statistical characterization of results. The paper reports single-point estimates without confidence intervals or multiple seeds. For the ablation trajectory, where differences are sometimes small (0.01 dB, 0.08 perplexity), it is difficult to distinguish genuine effects from noise without multiple training runs.

Overall assessment of experimental support:

The experiments strongly support the paper's negative claim: TTT does not function as memorization. The four contradiction experiments are well-designed, cover both storage and retrieval aspects, and produce consistent results across three different models and tasks. This is the paper's most robust contribution.

The experiments provide moderate but not definitive support for the positive claim that TTT is equivalent to linear attention. The analytical framework (Theorems 5.1–5.3) is mathematically sound, and the ablation trajectory shows that removing components not present in linear attention has minimal performance impact. However, the paper does not directly verify the equivalence on trained model instances, does not test whether optimized linear attention baselines close the remaining performance gap, and does not explore whether the equivalence holds at larger scales or for other TTT variants.

The practical claims (4.0× throughput improvement, parallelizability, simplification benefits) are well-supported by the reported measurements but require careful interpretation: the throughput improvements combine effects of simplification and parallelization, the end-to-end speedup is modest (~19%), and the performance cost of simplification (0.37 perplexity) is non-zero and may matter in some applications.

The paper's greatest experimental strength is the systematic nature of the ablation: six steps, three tasks, consistent methodology, with each step justified by the analytical framework rather than trial-and-error. The greatest experimental weakness is the single-scale, single-family nature of the evaluation—all results are on models under 1B parameters, and conclusions about what is "redundant" or "minor" may not transfer to larger models or different architectures. The paper is appropriately cautious in its claims (e.g., "only minor performance degradation on these tasks at this scale"), but readers should be aware of the boundary conditions.

6. Limitations and Trade-offs

1. Difficulty Estimation Cost Is Not Amortized in the Efficiency Calculations

The assumption or constraint. The paper's headline efficiency claims—particularly the 4× inference throughput improvement from the parallel formulation (Section 6.2, Table 2)—are measured for the TTT layer in isolation after the model has been trained and the architecture has been simplified. The paper does not account for the cost of training the complex TTT model (LaCT, ViTTT) in the first place, nor for any additional cost that might be incurred if one wanted to determine which components to remove for a given task and model scale. The ablation trajectory (Section 6.1) requires training multiple model variants to identify which components are redundant—a cost that is externalized from the efficiency analysis.

The consequence. The 4× throughput improvement is a post-hoc benefit realized by simplifying an already-trained architecture, but a practitioner starting from scratch would need to either (a) train the full complex TTT model and then incrementally ablate components (paying the full training cost of the original model plus the cost of ablation experiments), or (b) trust that the ablation trajectory generalizes to their architecture, task, and scale without verification. The latter is risky because the paper's own results show that component importance is task-dependent: the kernel function matters for novel view synthesis but not language modeling (Table 2, Variant 3: PSNR drops 0.22 dB vs. perplexity unchanged), and gradient orthogonalization matters for language modeling but not for vision tasks (Variant 6: perplexity up 0.83 vs. PSNR up 0.03). A practitioner cannot blindly apply the paper's exact ablation steps to a new domain without empirical validation, which reintroduces the training cost.

What evidence exists in the paper. The paper does not report the total compute cost of the ablation experiments. From Appendix A, training the 760M LaCT-LLM baseline takes approximately 56 hours on 8 A100 GPUs (~448 GPU-hours). Training all six ablation variants (or at minimum a subset to identify which simplifications work) would multiply this cost. The paper evaluates the ablations on three tasks (LLM, NVS, classification), each with its own training pipeline and compute requirements (38 GPU-hours for NVS, 32 GPU-hours for classification), making the full empirical validation a non-trivial investment. The paper does not present a predictive method for determining which components are redundant without retraining.

Mitigation status. The paper does not address this cost. Section 6 presents the ablation as a demonstration of the linear-attention framework's explanatory power, not as a practical recipe that practitioners should replicate from scratch for every new TTT variant. The implicit mitigation is that the framework provides principled guidance: components that are functionally absorbed into learned projections (per-token learning rates, momentum) should be redundant in general, while components that provide structural benefits (gradient orthogonalization for long sequences, kernel function for complex input modalities) may be worth keeping. However, the boundary between these categories is not crisply defined by the analytical framework alone—the paper relies on empirical validation to determine, e.g., that gradient orthogonalization matters for LLMs but not NVS. A practitioner in a new domain would face uncertainty about which side of the boundary their application falls on.

2. The Reduction to Linear Attention Requires a Linear, Bias-Free Final Layer

The assumption or constraint. The entire analytical framework (Theorems 5.1–5.3) depends on the inner-loop function having a linear, bias-free final layer: $f(x) = \phi(x; \Theta)W$ , where $W$ is a weight matrix with no bias term. The paper is explicit about this requirement:

"Extending these insights to nonlinear final layers, and exploring deeper connections between TTT and modern linear attention mechanisms in both directions, remain important avenues for future work." (Section 7)

The consequence. TTT variants whose inner-loop MLP has a nonlinear output layer (e.g., with an activation function after the final projection), a bias term in the final layer, or a structure that cannot be factorized as a linear function of some hidden representation $\phi(x)$ followed by a weight matrix $W$ do not fall within the paper's analytical framework. The paper demonstrates that this assumption holds for LaCT (SwiGLU MLP with final linear projection $W_1$ ; Section 5.3) and ViTTT (GLU with final linear projection; Section 5.4), and asserts that it "holds for essentially all TTT-KVB architectures in the literature" (our Section 3.4.1). However, this is an empirical claim about the TTT literature, not a mathematical necessity. A future TTT proposal could use a fully nonlinear inner-loop function (e.g., an MLP where all layers use nonlinear activations and the final output is not a linear readout of any hidden state), and the paper's framework would not apply to it. Similarly, architectures that use bias terms in the final layer produce gradient expressions that include additional terms (the bias gradient) that do not cleanly factor as key-value outer products.

What evidence exists in the paper. The paper does not test its framework on any architecture with a nonlinear final layer or bias terms. The analytical derivations (Appendices B–G) all assume the linear, bias-free form. The empirical validations (Section 6) all start from LaCT and ViTTT, both of which satisfy the assumption. There is no "failure case" experiment where the paper attempts to apply its framework to a non-conforming architecture and demonstrates that the reduction fails—the boundary is asserted analytically but not tested empirically.

Mitigation status. The paper acknowledges the limitation explicitly (quoted above) and frames it as future work. No attempt is made to extend the framework to nonlinear final layers within the current paper. This is a genuine scope boundary: the paper's claims apply to a large and important subset of TTT architectures (those with linear output layers) but not to all possible or future TTT designs. Practitioners evaluating whether the framework applies to their architecture should verify that their inner-loop function has a linear, bias-free final layer before relying on the paper's conclusions.

3. Single Scale Regime: No Evidence at 7B+ Parameter Models

The assumption or constraint. All experiments in the paper use models at a single scale: 760M parameters for language modeling, 114M for novel view synthesis, and 90M for image classification (Appendix A). No experiments are conducted at larger scales (e.g., 1B, 7B, 13B, or beyond). The paper implicitly assumes that the findings—particularly the redundancy of components like momentum, per-token learning rates, and weight normalization—generalize to larger models.

The consequence. The relative importance of architectural components often changes with scale. At 760M parameters, many design choices have modest impact on perplexity (the entire six-step ablation changes perplexity by only 0.37, or 2.3%; Table 2). At 7B or 70B parameters, the same architectural differences could be amplified (if the components provide benefits that compound with model capacity) or further diminished (if the increased representational capacity of a larger model makes the kernel function even more redundant). The paper's conclusion that gradient orthogonalization is important for language modeling (+0.83 perplexity; Table 2, Variant 6) might not hold at larger scales where alternative mechanisms for maintaining well-conditioned representations (e.g., better initialization, different normalization schemes) could be more effective. Similarly, the finding that the kernel function matters for novel view synthesis but not language modeling (Table 2, Variant 3) might be an artifact of the specific model sizes tested—a 7B parameter model for NVS might find the kernel function redundant, or a 100M parameter LLM might benefit from it more.

What evidence exists in the paper. None. The paper does not run any scaling experiments (varying model size while holding architecture constant), nor does it cite scaling studies from the TTT or linear attention literature that would support the claim that these findings are scale-invariant. The only evidence that sequence length does not interact with the ablation findings is Figure 3, which shows consistent perplexity gaps across sequence lengths (up to 32k) for the 760M model—but this tests context length scaling, not model size scaling.

Mitigation status. Not addressed. The paper does not claim that its findings are scale-invariant, but neither does it caution that they may not be. The abstract and conclusion present the results as general properties of TTT, without qualifying the model sizes tested. A practitioner considering TTT for a large-scale deployment should be aware that the evidence base is limited to sub-1B parameter models and that the optimal simplification strategy at larger scales may differ from the one identified here.

4. No Direct Empirical Verification of the Analytical Reduction on Trained Models

The assumption or constraint. The paper's central analytical claim—that TTT can be rewritten as a linear attention operator (Theorems 5.1–5.3, Sections 5.3–5.4)—is supported by mathematical derivation, not by direct empirical verification. The paper does not take a trained LaCT or ViTTT model, instrument it to capture the intermediate quantities defined by the framework ( $\phi_t(k_t)$ , $g_t(k_t)$ , $W_t$ , etc.), and demonstrate that the linear attention computation produces numerically identical outputs to the original TTT inner loop.

The consequence. The analytical framework could, in principle, contain an algebraic error or an unstated assumption that is not satisfied by the actual implementations of LaCT and ViTTT. The ablation trajectory (Section 6.1) provides indirect evidence: if TTT were not equivalent to linear attention, one would expect removing TTT-specific components to cause larger performance degradation than what is observed in Table 2. However, indirect evidence is weaker than direct verification—it is possible that the ablation succeeds for reasons unrelated to the linear-attention equivalence (e.g., the removed components were simply poorly tuned or trained). A direct verification would involve: (1) running a trained LaCT model on a test input, (2) computing the effective queries, keys, and values according to the formulas in Section 5.3, (3) performing the linear attention computation $o = \hat{q}(S_0 + \sum \hat{k}_i^\top \hat{v}_i)$ , and (4) confirming that the output matches the model's actual output to within numerical precision.

What evidence exists in the paper. The only empirical validation of the analytical framework is the ablation trajectory (Table 2), which shows that components not present in linear attention (momentum, per-token learning rates, normalization, deep MLPs) can be removed with minimal performance impact. This is consistent with the framework's predictions but does not constitute a direct test. The parallel formulation (Section 6.2, Appendix H) includes a proof of equivalence to the sequential recurrence, but this proof is for the simplified TTT variants (Variants 2–6), not for the original LaCT or ViTTT with all components present. The equivalence of the original complex TTT to linear attention is asserted analytically but never checked numerically.

Mitigation status. Not addressed. The paper relies on the mathematical derivations standing on their own, which is a legitimate approach for a theoretical contribution. However, for practitioners who want to be certain that their specific TTT implementation falls within the framework, the absence of direct empirical verification means they must trust the mathematical analysis or perform their own validation. Given the complexity of modern deep learning codebases—where implementation details (numerical precision, operation ordering, subtle differences from the published algorithm) can affect behavior—direct verification would significantly strengthen confidence in the framework's practical applicability.

5. No Combination of Search and Revisions; Components Studied in Isolation

The assumption or constraint. The ablation trajectory (Section 6.1, Table 2) evaluates each simplification step in isolation: the paper starts from the full LaCT/ViTTT and removes components one at a time, measuring performance at each step. The paper does not evaluate combinations of simplifications that skip intermediate steps, nor does it test whether the optimal simplified architecture might include a subset of components that were removed at different steps. For example, Variant 3 (single linear layer) removes the MLP depth while still using momentum, per-token learning rates, and weight normalization—but the paper does not test whether removing the MLP and momentum simultaneously (without the intermediate steps) might yield different results due to interaction effects.

The consequence. The linear, cumulative nature of the ablation trajectory implicitly assumes that component contributions are approximately additive—that the effect of removing momentum is similar regardless of whether the kernel function is a deep MLP or a single linear layer. If there are interactions between components (e.g., momentum is more important when the kernel function is deep because it smooths the optimization of the kernel parameters; weight normalization is more important when momentum is present because momentum can cause state norm growth), the sequential ablation might misattribute importance. In the extreme case, a component that appears redundant in Step 5 (after four prior simplifications have been applied) might have been essential in Step 1 (before those simplifications), and vice versa.

What evidence exists in the paper. The paper does not test for interaction effects. The ablation trajectory is strictly sequential: Step 1 → Step 2 → Step 3 → Step 4 → Step 5 → Step 6. There is no factorial experiment testing subsets of simplifications. The paper does not discuss the possibility of non-additive interactions or provide theoretical reasons to believe they are absent. Table 2 presents the results as a linear progression, which implicitly encourages the reader to interpret each step's delta as the independent contribution of the removed component.

Mitigation status. Not addressed. This is a standard limitation of sequential ablation studies and is not unique to this paper, but it is particularly relevant here because the paper's goal is to identify which components are "redundant" and which are "essential" to guide practitioners. A statement that the additivity assumption has not been tested would help practitioners interpret the ablation results more cautiously. Testing interaction effects would be computationally expensive (requiring factorial combinations of the six binary choices, or $2^6 = 64$ model variants), which likely explains why it was not done, but the limitation remains unacknowledged.

6. Latency-Wall-Clock Time Tradeoff Not Analyzed for the Parallel Formulation

The assumption or constraint. The paper's efficiency analysis (Section 6.2, Table 2, Figure 4) measures throughput (tokens per second) for the TTT layer in isolation and for end-to-end training. Throughput is a measure of total work completed per unit time and is maximized by processing many tokens in parallel. However, throughput is not the only relevant efficiency metric for deployment. Latency—the time to process a single token or a single request—can be equally important for interactive applications (chatbots, real-time translation, live video processing). The parallel formulation processes chunks of $L$ tokens simultaneously, which improves throughput but may increase the latency for producing the first output token compared to a recurrent formulation that can begin emitting tokens immediately upon receiving the first input.

The consequence. A practitioner deploying TTT in a latency-sensitive setting needs to understand the trade-off between throughput and time-to-first-token. The paper's claim of "up to 4× inference throughput" (Section 6.2) does not distinguish between throughput improvements that come from better hardware utilization (which may not affect latency) and improvements that come from algorithmic parallelism (which may increase latency due to chunking overhead). For autoregressive generation, the recurrent formulation can produce output $i+1$ as soon as token $i$ has been processed; the parallel formulation must wait until an entire chunk of $L$ input tokens has been accumulated before producing any outputs for that chunk. This introduces a latency penalty of up to $L$ token-processing times at the start of each chunk.

What evidence exists in the paper. The paper reports only throughput metrics (tokens per second in Table 2; wall-clock training time in Figure 4). Latency—time to first token, per-token generation latency, or end-to-end request latency—is not measured or discussed. The chunk size $L$ for the parallel formulation is not specified in the main text or Appendix H, making it impossible to estimate the latency penalty from the reported results. The paper does not discuss the throughput-latency tradeoff or provide guidance on choosing $L$ to balance the two.

Mitigation status. Not addressed. The paper frames the parallel formulation as a pure improvement over the recurrent implementation (Section 6.2 presents it as enabling "more efficient implementation" and "significant efficiency gains"), without acknowledging that this efficiency comes at a potential latency cost. This is a significant omission for practitioners, as the appropriate implementation strategy (recurrent vs. parallel, and if parallel, what chunk size) depends heavily on the deployment scenario. A latency-sensitive chatbot might prefer the recurrent implementation despite its lower throughput, while a batch inference pipeline might prefer the parallel implementation. The paper's silence on this trade-off means practitioners cannot make an informed decision based on the reported results.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper triggers a fundamental reframing of test-time training with KV binding: TTT is not a memorization mechanism but a learned linear attention operator whose inner-loop gradient descent dynamically constructs effective queries, keys, and values through a structured optimization process. The magnitude of this shift is substantial—not because the paper proposes a new architecture that outperforms existing ones, but because it dismantles the conceptual foundation upon which an entire subfield of sequence modeling research has been built and replaces it with a mathematically rigorous alternative that explains previously puzzling behaviors and enables principled architecture design.

From memorization-as-metaphor to attention-as-mechanism. Prior to this work, the TTT literature operated under a unifying metaphor: the inner loop "stores" key-value pairs into fast weights, and queries "retrieve" the stored information. This metaphor was productive—it motivated an active line of research and generated genuinely novel architectures—but it was also constraining. It led researchers to invest in increasingly complex inner-loop optimizers (momentum, per-token learning rates, Muon orthogonalization) and deeper MLPs under the assumption that better optimization fidelity would improve memory quality. It kept TTT implementations recurrent because storage-and-retrieval is inherently sequential. It separated TTT conceptually from the linear attention literature, preventing cross-pollination. The paper demonstrates that each of these consequences of the memorization metaphor was empirically and analytically unnecessary: gradient ascent preserves performance (Table 1), removing momentum slightly improves language modeling perplexity (Table 2, Variant 5), the kernel function can be a single linear layer with minimal degradation (Variant 3), and the simplified TTT admits a fully parallel formulation (Section 6.2).

The reframing matters because it redirects the field's energy. Under the memorization view, the path to better TTT was to improve the inner-loop optimizer—make it converge faster, store more faithfully, stabilize better. Under the linear attention view, the path is to design better effective query, key, and value mappings through the kernel function φ(·) and the loss function ℒ, while recognizing that the gradient descent dynamics are just one way (and not necessarily the best way) to induce a structured attention operator. This opens the entire design space of linear attention variants—learned decay factors, multi-head formulations, gating mechanisms, chunk-parallel implementations—to the TTT community, and conversely suggests that TTT's innovations (gradient orthogonalization for well-conditioned state matrices, learned kernel functions for input-dependent transformations) may transfer back to the linear attention literature.

Reconciling the known special case with the general phenomenon. The prior observation that single-layer TTT with MSE loss reduces to linear attention (Sun et al., 2025) was correctly interpreted as a mathematical equivalence, but its significance was underappreciated because it was seen as a fragile boundary case. The field implicitly assumed that the equivalence broke down as soon as complexity was added—that multi-layer MLPs, momentum, and alternative loss functions were precisely what made TTT different from and more powerful than linear attention. This paper proves that assumption wrong: the linear-attention structure persists through all of these complications (Theorems 5.1–5.3), and the additional components modify how the effective queries, keys, and values are computed without changing the structure of the computation. This turns a known coincidence into a unifying principle.

Resolving why prior work reached contradictory conclusions about inner-loop optimization. The TTT literature contained an implicit tension: some papers emphasized the importance of careful inner-loop optimizer design (Behrouz et al., 2024; Zhang et al., 2025), while others found that simple configurations sufficed (ViTTT used constant learning rates; Han et al., 2025). Under the memorization view, this was a puzzle—if the inner loop performs storage, why doesn't better optimization consistently help? The linear-attention perspective resolves this: the inner-loop optimizer is not optimizing a storage objective in any meaningful sense; it is defining a mapping from raw inputs to effective representations. Different optimizer configurations produce different mappings, but the model learns to compensate for these differences through the learned projections (queries, keys, values). This explains why gradient ascent works (the sign flip is absorbed into the value projection; Section 5.2), why momentum is redundant (its recency-weighting is already learnable; Theorem 5.3), and why more gradient steps hurt (train-test mismatch in the induced attention operator; Section 5.2).

Making test-time compute research more skeptical of metaphor-driven architecture design. Beyond TTT specifically, the paper serves as a cautionary example of how a compelling metaphor can drive a research agenda in directions that are empirically unsupported. The storage-and-retrieval interpretation was not malicious or obviously wrong—it was intuitive, aligned with the gradient-descent update semantics, and consistent with the meta-learning framing that TTT emerged from. But it was also untested: no prior work had asked what would constitute evidence against it. The paper's systematic falsification strategy—identify predictions the metaphor makes, test them empirically, and show they fail—provides a template for evaluating other metaphor-driven architecture claims in the sequence modeling literature. This is a methodological contribution that extends beyond the specifics of TTT.

Which research directions become more attractive. The linear-attention perspective makes several lines of work newly promising:

Combining TTT with modern linear attention mechanisms: TTT's learned kernel function φ(·) and gradient-derived value mappings can be integrated with explicit decay factors (as in Mamba, GLA, or RetNet), chunk-parallel formulations (Yang et al., 2024b), and hardware-optimized attention kernels. The paper's demonstration that gradient orthogonalization provides meaningful benefits for language modeling (Table 2, Variant 6: +0.83 perplexity when removed) suggests that this particular TTT innovation could improve standard linear attention implementations, which typically do not orthogonalize state updates.
Designing loss functions to directly specify the attention operator: The paper shows that the inner-loop loss function ℒ determines the effective value vector through the gradient computation (e.g., Frobenius inner product yields g_t(k) = η·v_t; Section 5.3). This opens the possibility of designing loss functions not for their optimization properties, but for the specific attention operators they induce. For example, a loss function that produces g_t(k) = v_t − f_t(k) (the residual between value and current prediction) would induce an attention operator that naturally performs error correction—a potentially useful inductive bias for certain tasks.
Training-time and inference-time co-design of the attention operator: The paper identifies train-test mismatch as the root cause of degradation with more inner-loop steps (Section 5.2). This suggests that the number of inner-loop steps should be treated as a hyperparameter that is fixed during both training and inference, rather than varied at inference time in hopes of improved memorization. More broadly, it suggests that the entire inner-loop configuration (number of steps, optimizer, loss function) should be treated as part of the architecture specification, with training and inference using identical configurations—a design principle that directly contradicts the "test-time adaptation" framing but follows naturally from the linear-attention perspective.

Which research directions become less attractive. The paper's results cast doubt on several active areas of TTT research:

Developing more sophisticated inner-loop optimizers: If momentum, per-token learning rates, and weight normalization are all redundant or marginally beneficial (Table 2, Variants 2–5), then further investment in optimizer complexity (e.g., second-order methods, adaptive per-parameter learning rates, learned optimization algorithms) is unlikely to yield meaningful improvements beyond what can be achieved through better query/key/value projections. The paper does not rule out that some optimizer innovations might help—gradient orthogonalization does matter for language modeling—but it shifts the burden of proof: any new optimizer component must demonstrate benefits beyond those achievable through learned projections.
Increasing inner-loop MLP depth for improved storage capacity: The memorization interpretation's natural prescription—deeper MLPs store more information—is directly contradicted by Step 1 of the ablation, where restricting updates to the final layer improves performance (Table 2, Variant 1: perplexity drops from 16.43 to 15.93). This suggests that the complexity of the inner-loop function is better invested in the kernel function φ(·) (which transforms raw inputs into effective queries and keys) rather than in the optimization dynamics (which merely accumulate outer products into the state). Future TTT designs should separate these concerns: make φ(·) as expressive as needed for the task, but keep the state update simple (linear, associative where possible).
Framing TTT as test-time adaptation rather than as an architectural choice: The paper's results undermine the "test-time" aspect of test-time training. If the inner-loop optimization can be replaced with a static linear attention computation (Variant 6) with minimal performance loss, then the gradient-descent-at-inference-time framing is not just inaccurate—it's unnecessary. TTT's value proposition is not that it adapts at test time but that it provides a particular form of linear attention with certain representational properties (learnable kernel functions, optional orthogonalization). This suggests that TTT research should be reframed as a subfield of efficient attention design rather than as a continuation of the test-time adaptation literature from which it originated.

Follow-Up Research This Work Enables

Direct empirical verification of the linear-attention equivalence on trained TTT models. The paper provides analytical proofs that TTT can be rewritten as linear attention (Theorems 5.1–5.3, Sections 5.3–5.4) and indirect empirical evidence through the ablation trajectory (Table 2). What is missing is a direct test: take a pretrained LaCT model, instrument it to capture the intermediate quantities defined by the framework (φ_t(k_t), g_t(k_t), W_t, etc.) at each token position, compute the linear attention output ô = φ_{t+1}(q_t)(W_0 + Σ_i φ_i(k_i)ᵀ g_i(k_i)), and verify that ô matches the model's actual output to within numerical precision. This experiment would definitively validate the analytical framework and protect against the possibility that the proofs rely on an unstated assumption violated by the actual implementation (e.g., precision, operation ordering, or subtle differences between the published algorithm and the codebase). A strong follow-up would run this verification on both LaCT and ViTTT, across multiple layers and sequence lengths, and report the maximum absolute deviation between analytical and actual outputs. Any non-zero deviation (beyond floating-point noise) would identify cases where the framework's assumptions are violated.

Training TTT from scratch with gradient ascent in the inner loop. The paper's gradient ascent experiment (Table 1) tests pretrained models: it takes a model trained with standard gradient descent in the inner loop and shows that replacing descent with ascent at inference time doesn't degrade performance. This demonstrates that inference-time gradient direction doesn't matter, but it doesn't rule out that gradient descent was necessary during training to establish good representations. The definitive test would be to train a LaCT-LLM from scratch with gradient ascent in the inner loop for the entire training run (all 20K iterations, 100B tokens). If the resulting model achieves comparable perplexity to the gradient-descent-trained baseline, that would rule out any essential role for gradient-based key-value fitting even during training. If it underperforms significantly, that would identify a boundary condition: gradient descent matters for establishing the learned projections, even though it becomes unnecessary once they are established. This experiment is computationally expensive (requiring a full 56-hour training run on 8 A100 GPUs) but feasible for a well-resourced research group, and it would definitively answer the most aggressive form of the paper's core claim.

Scaling the ablation trajectory to 7B+ parameter models. The paper's conclusions about component redundancy (momentum, per-token learning rates, weight normalization) and component importance (gradient orthogonalization for language modeling, kernel function for 3D vision) are based on models at 90M–760M parameters. The relative importance of architectural components often changes with scale—components that appear redundant at 760M might become important at 7B (if their benefits compound with model capacity), or vice versa (if increased capacity makes them even more redundant). A strong follow-up would replicate the six-step ablation (or a targeted subset focusing on Steps 1, 3, and 6) at the 7B parameter scale, using a dataset like FineWeb-Edu or C4, and measuring both perplexity and downstream task performance. The key question is whether the performance gap between Variant 6 (standard linear attention) and the original LaCT widens or narrows with scale. This experiment would also test whether gradient orthogonalization's importance for language modeling persists at larger scales, where alternative mechanisms for maintaining well-conditioned state matrices (e.g., better initialization, different normalization schemes) might be more effective.

Combining TTT's kernel function with modern linear attention decay mechanisms. The paper establishes that TTT's kernel function φ(·) is a learned transformation of raw inputs into effective queries and keys (Section 5.3), and that the effective values include a recency bias when momentum is used (Theorem 5.3, the Σ_j β_{ij} term). However, TTT's recency bias is an emergent property of the momentum dynamics, not an explicitly designed component. Modern linear attention variants (Mamba, GLA, RetNet) use carefully designed data-dependent decay factors that have been shown to be important for in-context learning and long-range reasoning. A natural synthesis would be to augment Variant 2 (static kernel function, simplified state update) with an explicit, learned decay factor on the state matrix—replacing the simple sum S_{t+1} = S_t + φ(k_t)ᵀ v_t with S_{t+1} = γ_t · S_t + φ(k_t)ᵀ v_t, where γ_t is a data-dependent gating scalar (e.g., computed from the current token). This would combine TTT's learned kernel function (which transforms raw inputs into effective representations) with the proven benefits of explicit decay from the linear attention literature. A strong follow-up would compare this hybrid against both pure TTT and pure linear attention baselines on standard long-range benchmarks (e.g., Long Range Arena, SCROLLS, or long-context language modeling). The hypothesis is that the kernel function provides representational benefits that explicit decay cannot replicate alone, and vice versa—the two mechanisms are complementary rather than redundant.

Exploring loss functions as a design space for attention operators. The paper shows that the inner-loop loss ℒ determines the effective value vector through g_t(k) = −η·∂ℒ/∂f_t(k) (Theorem 5.1). With the Frobenius inner product loss used by LaCT and ViTTT, g_t(k) = η·v_t—the effective value is simply a scaled version of the raw value. This is computationally convenient but not the only possibility. A different loss function would produce a different effective value computation, potentially inducing attention operators with useful inductive biases. For example:

A squared-error loss ℒ = ‖f_t(k) − v‖² would yield g_t(k) = −2η(v_t − f_t(k)), making the effective value proportional to the prediction error—an error-correcting attention operator.
A contrastive loss involving multiple values would yield g_t(k) that depends on the relationship between f_t(k) and multiple candidate values—inducing a form of competitive attention. A systematic exploration of loss functions as a design space for attention operators—cataloging the effective value computations they produce and measuring their impact on downstream task performance—would be a natural extension of the paper's framework. The key insight is that the loss function is no longer about "memorizing better"; it's about defining the computation that the attention operator performs.

Testing whether the reduction to linear attention generalizes to nonlinear final layers. The paper's analytical framework assumes a linear, bias-free final layer (Section 5.1), and all empirical validation is on architectures satisfying this assumption. The authors acknowledge this limitation and flag it as future work (Section 7). A constructive follow-up would analyze whether TTT with a nonlinear final layer—e.g., f(x) = σ(φ(x)W) where σ is a nonlinear activation—can still be expressed as some form of attention operator, or whether the equivalence genuinely breaks down. One approach: unroll the inner-loop update for the final layer using the chain rule, which would produce g_t(k) = −η·σ′(φ_t(k)W_t)·(∂ℒ/∂f_t(k)), introducing a state-dependent scaling factor σ′(·) that depends on the current weight matrix W_t. This would not be standard linear attention (the effective value would depend on the current state), but it might correspond to a gated or input-dependent attention variant. Alternatively, fully nonlinear inner-loop functions without a separable final layer might genuinely not reduce to attention, which would establish a crisp boundary: TTT with linear output layers is linear attention; TTT with nonlinear output layers is something different. This experiment would both extend the framework and define its scope.

Practical Applications and Downstream Use Cases

Simplifying and accelerating TTT-based language models for batch inference pipelines. The paper's central practical finding is that LaCT-based language models can be reduced to standard linear attention (Variant 6) with a perplexity increase of only 0.37 (16.43 → 16.80; Table 2) while achieving a 29× improvement in TTT-layer inference throughput (4.30M → 124.6M tokens/sec; Table 2, recurrent baseline vs. Variant 6 parallel). For organizations running large-scale batch inference—evaluating models on text corpora, generating training data, or scoring candidate generations—this translates directly to cost savings: the same model quality (within ~2.3% in perplexity) at a fraction of the inference compute cost. A deployment engineer could take a pretrained LaCT-LLM checkpoint, apply the simplifications described in Section 6.1 (fix kernel function parameters, remove weight normalization and momentum, optionally remove orthogonalization if the ~0.8 perplexity cost is acceptable), and deploy the resulting linear attention model using existing optimized kernels (FlashLinearAttention, chunk-parallel implementations) rather than maintaining a custom TTT inference stack. The 1.19× end-to-end training speedup (Figure 4) further reduces the cost of fine-tuning or continued pretraining on domain-specific data.

On-device deployment of TTT-based vision models with reduced memory and latency. For the ViTTT image classification model (90M parameters), the simplified linear attention variant (Variant 6) actually improves Top-1 accuracy by 0.20 percentage points (79.34% → 79.54%; Table 2) while eliminating the need for per-token gradient computations and weight updates during inference. This is particularly relevant for on-device or edge deployment, where the original ViTTT's inner-loop optimization adds computational overhead (gradient computations, weight updates, normalization) that consumes memory, energy, and time. The simplified variant replaces all inner-loop operations with a single linear attention computation that can be implemented via standard matrix multiplications—operations that are heavily optimized on mobile GPUs and NPUs. The throughput improvement is not directly measured for ViTTT (Table 2 reports tokens/sec only for the LLM task), but the structural simplification (removing multi-layer MLPs, momentum, per-token learning rates) implies substantial latency and energy reductions. A mobile vision application (e.g., real-time object recognition, augmented reality) using the simplified ViTTT variant would benefit from both the accuracy improvement and the reduced compute footprint.

Incorporating TTT-derived gradient orthogonalization into standard linear attention implementations for long-context language modeling. The ablation identifies gradient orthogonalization (Muon) as the single most important non-core component for language modeling: removing it increases perplexity by 0.83 (15.97 → 16.80; Table 2, Variant 6 vs. Variant 5). This component is not specific to TTT—it is an operation applied to the state update matrix (the key-value outer product) before accumulation. Standard linear attention implementations (Katharopoulos et al., 2020; various fast-transformers implementations) typically perform simple additive accumulation S_{t+1} = S_t + k_t^T v_t without any orthogonalization. The paper's results suggest that adding Muon-style orthogonalization to standard linear attention's state update could improve its performance on long-context language modeling, potentially closing or eliminating the gap between complex TTT and standard linear attention. Implementing this would be straightforward: after computing the outer product k_t^T v_t, apply the Muon operator (e.g., via Newton-Schulz iteration or QR decomposition) before adding to the state. The computational overhead of orthogonalization would need to be balanced against the perplexity benefit, but the throughput data in Table 2 suggests this overhead is manageable: Variant 6 (with orthogonalization removed) achieves 124.6M tokens/sec, while Variant 2 (with normalization but before orthogonalization removal) achieves 30.18M tokens/sec (parallel)—a substantial gap, but orthogonalization specifically may be cheaper than the full normalization that Variant 2 includes, since Muon can be applied efficiently via Newton-Schulz iteration. Benchmarking the perplexity-vs-throughput trade-off for orthogonalized linear attention on standard long-context evaluation suites (e.g., LongBench, L-Eval) would be a practical contribution building directly on this paper's findings.

A principled recipe for converting existing TTT codebases to efficient implementations. The paper's six-step ablation trajectory (Section 6.1) provides a concrete, ordered recipe that practitioners can follow to simplify their TTT implementations. The trajectory is designed to be cumulative: each step builds on the previous one, and the performance impact is measured at each stage (Table 2). A practitioner with an existing TTT codebase (LaCT, ViTTT, or a custom variant with a linear final layer) can:

First, restrict inner-loop updates to only the final-layer weight matrix (Step 1). This alone improves performance and throughput (Table 2: perplexity 16.43 → 15.93, throughput 4.30M → 10.60M tokens/sec).
If parallelization is desired, additionally remove weight normalization (Step 2) to make the state update associative. The performance cost is minimal (perplexity 15.93 → 16.31) and enables 2.7× further throughput improvement (11.02M → 30.18M tokens/sec parallel).
For maximum simplicity, continue through Steps 3–6, removing MLP depth, per-token learning rates, momentum, and orthogonalization, accepting the accumulated performance trade-offs documented in Table 2. At each step, the practitioner can stop if further simplification would cause unacceptable performance degradation for their specific task (e.g., a novel view synthesis application might stop at Step 2 to preserve the kernel function, which provides 0.22 dB PSNR; a language modeling application might stop at Step 5 to preserve gradient orthogonalization, which provides 0.83 perplexity).

This recipe is immediately actionable and requires no new model training—it can be applied to existing pretrained checkpoints by modifying the inference code. The paper's Appendix H provides the parallel formulation that replaces the sequential inner loop, and the throughput measurements in Table 2 provide realistic estimates of the efficiency gains at each step. A practitioner could implement this conversion in a matter of days and immediately realize the documented throughput improvements without waiting for new model training.