Learning to (Learn at Test Time): RNNs with Expressive Hidden States

🎯 Pitch

Instead of a fixed vector, the hidden state becomes a neural network trained on the fly via gradient descent as each token arrives—enabling RNNs to finally use contexts beyond 16k tokens where Mamba plateaus, while keeping linear complexity.

1. Executive Summary

This paper introduces Test-Time Training (TTT) layers, a framework for sequence modeling layers with linear complexity and expressive hidden states by making the hidden state a machine learning model whose update rule is a step of self-supervised learning on the test sequence. Evaluated on the Pile and Books3 benchmarks using models from 125M to 1.3B parameters against Transformer and Mamba baselines, the framework yields two instantiations—TTT-Linear (hidden state is a linear model trained via mini-batch gradient descent) and TTT-MLP (hidden state is a two-layer MLP)—that match Mamba's perplexity in short context while continuing to improve by conditioning on more tokens past 16k context where Mamba plateaus. The paper demonstrates that TTT-Linear and TTT-MLP achieve comparable perplexity to Mamba at 2k context and significantly better perplexity at 8k and 32k contexts, establishing that making the hidden state a parametric learner enables RNNs to effectively use long context, but that MLP-based instantiations still face substantial wall-clock time overhead relative to their FLOPs advantage.

2. Context and Motivation

The Fundamental Tension in Sequence Modeling: Long Context vs. Linear Cost

The paper addresses a central tension in sequence modeling that has been unresolved since the Transformer's introduction. Self-attention delivers strong long-context performance — tokens later in a sequence condition on all prior tokens through an explicit mechanism — but at the cost of quadratic complexity: processing a sequence of length $T$ requires $O(T^2)$ operations. RNN layers, by contrast, achieve linear complexity $O(T)$ by compressing all historical context into a hidden state of fixed size, but this compression is lossy, and the quality of the compressed representation fundamentally limits how well the model can use information from many tokens back.

This tradeoff has a practical cliff, as the paper observes (Section 1, Figure 12). The asymptotic advantage of RNNs over Transformers — their $O(T)$ versus $O(T^2)$ complexity — only becomes realized in wall-clock time once context length exceeds roughly 8k tokens. But at precisely the regime where RNNs should shine, existing RNNs such as Mamba fail to actually benefit from the additional information. This creates what the paper calls "an awkward reality": the architecture that should dominate long context can't effectively use it.

The evidence for this failure is made concrete in the paper's opening figure (Figure 2, right). The metric is perplexity as a function of token index: tokens later in a sequence have access to more context, so a model that effectively uses that context should show progressively lower perplexity for tokens at higher indices. A strong Transformer exhibits exactly this behavior — its per-prefix perplexity decreases steadily throughout 32k context. Mamba, identified as "one of the most popular RNNs today," shows the same pattern as the LSTMs that Kaplan et al. [43] analyzed in 2020: perplexity improves up to about 16k context, then plateaus. More tokens are available, but the model can't use them. This is a measuring stick for the quality of compression: when the hidden state has already compressed as much as its capacity allows, stuffing in more information provides no benefit.

Why This Gap Matters: The Scaling of Language Models

The practical significance has three dimensions:

First, the economics of deployment. Transformers' quadratic cost is not a minor inefficiency — it is a dominating factor for systems that process long documents, maintain long conversations, or generate long outputs. A system processing, say, a 128k-token legal document would incur $128^2 / 8^2 = 256\times$ the per-token cost of processing an 8k document under quadratic self-attention. RNNs with linear complexity would avoid this multiplicatively growing cost, but only if they can actually use the long context effectively. If they can't, the cost savings are irrelevant — users would pay the quadratic tax to get acceptable quality.

Second, the scaling trajectory of the field. The paper explicitly positions itself relative to the 2020 OpenAI scaling law paper (Kaplan et al. [43]), which showed that LSTMs failed to scale similarly to Transformers. That result shaped nearly five years of architectural development toward attention-centric designs. By revisiting this question with modern RNNs (Mamba) and modern training practices (the Chinchilla recipe), the paper asks whether the 2020 conclusion was fundamental to all RNNs or specific to the LSTM's limitations. The answer matters because it determines whether the field's architectural choices are converged or whether there is room for fundamentally different primitives.

Third, a theoretical question about representation. The hidden state of an RNN is a compressed representation of everything the model has seen. The difficulty of compression over very long horizons — thousands or millions of tokens — raises the question: what kind of compression works? What should the hidden state be to capture the underlying structures and relationships across massive context? The paper argues that answering this is not just an engineering challenge but a conceptual one about the nature of sequential understanding.

Prior Approaches and Their Limitations

The paper identifies several categories of prior work, each with specific shortcomings that motivate the new approach.

Self-attention as explicit storage (the Transformer family). Self-attention sidesteps the compression problem entirely by maintaining an explicit record of every past token in the Key-Value cache. The update rule simply appends the current token's key and value to the list, and the output rule scans the entire list to compute attention scores (Section 2, Figure 3). This works remarkably well — no information is lost through compression — but the linearly growing hidden state means linearly growing cost per token. For very long sequences, this becomes prohibitive despite its effectiveness. The Transformer's design is the "don't compress" strategy: accept the quadratic penalty in exchange for perfect memory.

Classical RNNs and state-space models (the Mamba family). Modern RNNs such as Mamba [27], RWKV [58, 59], and xLSTM [4] compress context into a vector-valued or matrix-valued hidden state of fixed size. Figure 2 (left) demonstrates that Mamba has made substantial progress since 2020 — it scales similarly to Transformers up to 8k context, achieving nearly identical perplexity at matched FLOPs. This is genuine progress: the scaling law paper's finding that LSTMs can't scale is not fundamental to all RNNs.

However, Figure 2 (right) reveals the residual limitation: Mamba's performance plateaus after 16k context, even though it has linear complexity and could theoretically process arbitrarily long sequences. This plateau is the paper's central motivation. The compression heuristic used by Mamba — a structured state-space model with input-dependent gating — apparently saturates its representational capacity. The hidden state's expressive power becomes the bottleneck.

Linear attention and fast weight programmers. A distinct line of work, originating with linear attention [44], avoids the quadratic cost by removing the softmax from self-attention, yielding a formulation that can be written recurrently with a matrix-valued hidden state. DeltaNet [62, 83] and Gated Linear Attention (GLA) [82] extend this idea. The hidden state in these models is typically a matrix that accumulates outer products of key-value pairs, updated via variants of the delta rule.

The paper positions these as special cases within its framework (Theorem 1, Subsection 2.6): TTT with a linear model and batch gradient descent recovers linear attention exactly. But the paper argues that prior work in this tradition has been constrained to linear (matrix) hidden states and specific update rules. The expressive power of a matrix hidden state is limited — it can store pairwise relationships between dimensions, but cannot represent more complex non-linear interactions that might be needed for genuine understanding of long context.

Test-time training in computer vision. Prior work on Test-Time Training [72, 23, 79] applied the idea of learning at test time to vision tasks. In those settings, a model performs a self-supervised task (typically reconstruction) on the test input to adapt its features before making a prediction. However, the self-supervised task in prior TTT work was handcrafted with human priors — for example, solving a jigsaw puzzle of the test image, or reconstructing masked patches. The paper argues this handcrafting is a fundamental limitation: the optimal self-supervised task likely depends on the domain, the model architecture, and the end goal (next-token prediction), and humans shouldn't have to design it manually.

Dynamic evaluation in NLP. The practice of finetuning a language model directly on the test sequence, known as dynamic evaluation [47, 48], is another instance of learning at test time. While effective, this approach treats the test-time learning as an add-on — a finetuning step applied to a pretrained model — rather than as a core architectural primitive. The inner loop is not integrated into the model's forward pass as a sequence modeling layer, and the self-supervised task is not learned jointly with the rest of the network.

The Paper's Key Insight and Position

The paper's central insight draws an analogy between two compression processes that machine learning practitioners already understand deeply:

"Self-supervised learning can compress a massive training set into the weights of a model such as an LLM, which often exhibits deep understanding about the semantic connections among its training data — exactly what we need from a compression heuristic."

The training of a large language model compresses internet-scale text into the weights of a neural network. The resulting model demonstrates not just retrieval but understanding — it can synthesize new combinations of knowledge and reason across concepts. This is a compression process that works at scale. The paper asks: what if the same mechanism — parametric learning — were used as the compression heuristic inside the hidden state of an RNN?

Concretely, this means making the hidden state a machine learning model itself, and the update rule a step of self-supervised learning. When processing a test sequence, each token becomes a training example for the inner model, and the inner model's weights are updated via gradient descent. The compressed representation is not a vector or matrix of statistics — it is the parameters of a model that has been trained to capture the structure of the sequence so far.

This reframes the sequence modeling problem in a way that connects to the rich toolkit of machine learning. The inner model $f$ can be anything — a linear model (yielding TTT-Linear), a two-layer MLP (yielding TTT-MLP), or potentially a deeper network, a convolutional network, or even another attention layer. The optimizer can be online gradient descent, mini-batch gradient descent, or potentially Adam. The self-supervised task can be reconstruction (as in this paper) or any other formulation. This is the "practical framework" the paper claims — Figure 8 illustrates how different choices of model × optimizer induce different TTT layer instantiations.

Crucially, the paper goes beyond simply pointing out this analogy. It addresses the core challenge that has historically made this approach impractical: efficiency. Taking a gradient step per token is both sequential (can't be parallelized) and hardware-unfriendly (few matrix multiplications). The paper introduces two techniques to make the approach viable: mini-batch gradient descent over tokens (enabling parallelization within a mini-batch) and a dual form that reformulates the computations within each mini-batch to use dense matrix multiplications, making them efficient on modern GPUs and TPUs.

The positioning relative to prior work is cleanly summarized in Figure 9 and Table 1. TTT layers are a superset of both classical RNNs and certain nonparametric learners. Linear attention is recovered as a special case (batch GD, linear model, $W_0 = 0$ , $\eta = 1/2$ ). Self-attention is recovered when using the Nadaraya-Watson estimator as the learner (Theorem 2). The paper's contribution is not a single new architecture but a framework that produces a family of architectures, with TTT-Linear and TTT-MLP as illustrative instantiations that improve on the prior special cases.

The paper also makes an important conceptual contribution by distinguishing the "outer loop" (training the overall network with next-token prediction) from the "inner loop" (updating the hidden state via TTT). Table 2 shows that the outer loop is "at the same level" as regular training — it solves the same canonical supervised learning problem on the same data. This is in contrast to prior meta-learning work, where the outer loop required a collection of datasets or tasks, making it harder to scale. By nesting the learning-to-learn one level below regular training rather than one level above, the paper makes the framework compatible with standard LLM training pipelines and datasets.

3. Technical Approach

3.1 Reader Orientation

This is primarily a systems and architecture paper whose core idea is to redesign the hidden state of an RNN to be a machine learning model that trains itself on the test sequence as it processes it, using the sequence itself as self-supervised training data. The problem it solves is the fundamental tension in sequence modeling: self-attention handles long context well but costs $O(T^2)$ computation, while RNNs cost $O(T)$ but their fixed-size hidden states lack the expressive power to effectively compress and use very long context. The solution's shape is a nested learning loop—an inner loop that trains a small model (the hidden state) on each test sequence using gradient descent, embedded inside an outer loop that trains the overall network for next-token prediction—combined with architectural and systems techniques that make this computationally practical on modern hardware.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, organized into two nested learning loops:

The inner model $f$ (the hidden state itself): A learnable function—either a linear model ( $f(x) = Wx$ , yielding TTT-Linear) or a two-layer MLP (yielding TTT-MLP)—whose weights $W$ serve as the compressed representation of all tokens seen so far. This is what other RNNs would call the hidden state.
The self-supervised task definition (outer-loop parameters $\theta_K, \theta_V, \theta_Q$ ): Three learned linear projections that define what the inner model is trained to do. $\theta_K$ produces the training view (corrupted input), $\theta_V$ produces the label view (reconstruction target), and $\theta_Q$ produces the test view (used for the output token). These are learned by the outer loop, meaning the network discovers what self-supervised task is most useful for next-token prediction.
The inner-loop optimizer: The update rule that trains $f$ on the test sequence. The paper uses mini-batch gradient descent with a learned, token-dependent learning rate $\eta(x)$ . For each mini-batch of $b$ tokens, gradients are computed in parallel with respect to the weights at the start of that mini-batch, then the weights are updated.
The dual form computation engine: A mathematically equivalent reformulation of the TTT forward pass that replaces sequential per-token operations with dense matrix multiplications, making it efficient on GPUs and TPUs. This is the key systems contribution that makes the approach practical.
The backbone architecture: The overall network structure into which TTT layers are inserted. The paper adopts the Mamba backbone (which includes temporal convolutions before TTT layers, plus gating mechanisms) after finding it improves perplexity over the standard Transformer backbone.

Information flow: An input sequence $x_1, \ldots, x_T$ enters the network → each token passes through temporal convolutions (Mamba backbone) → enters the TTT layer where $\theta_K, \theta_Q, \theta_V$ project it into training, test, and label views → the inner model $f$ is trained on batches of these views (the training view is input to $f$ , the label view is the reconstruction target) → the updated inner model is applied to the test view to produce the output token $z_t$ → outputs proceed through the rest of the network → the outer loop backpropagates through all of this to optimize $\theta_K, \theta_Q, \theta_V, \theta_{init}, \theta_{lr}$ , and all other network parameters for next-token prediction.

3.3 Roadmap for the Deep Dive

First, the core TTT mechanism as a hidden state update rule (Equations 1–3): what the inner model $f$ is, how the update rule works as gradient descent on a self-supervised loss, and why reconstruction is the chosen self-supervised task. This establishes the basic "train a model at test time" idea.
Second, the nested learning structure (inner loop vs. outer loop): how gradient descent on $W$ (inner loop) differs from gradient descent on $\theta_{\text{rest}}$ (outer loop), and why this nesting is what makes TTT layers trainable end-to-end like any other sequence modeling layer.
Third, the learned self-supervised task (Equations 4–5): how $\theta_K, \theta_V, \theta_Q$ parameterize a family of multi-view reconstruction tasks, making the self-supervised objective learnable rather than handcrafted.
Fourth, mini-batch TTT (Equation 6): the parallelization strategy that trades off between the sequential nature of online gradient descent and the hardware efficiency of batch computation, and why the gradient channel and cumsum channel have different roles.
Fifth, the dual form (Equations 7–8): the mathematical reformulation that makes TTT layers efficient on modern accelerators by converting per-token outer products into dense matrix multiplications, with concrete time complexity analysis.
Sixth, the theoretical equivalences (Theorems 1–2, Figures 8–9): how linear attention and self-attention are recovered as special cases, establishing TTT layers as a unifying framework for sequence modeling.
Seventh, the implementation details (instantiations of $f$ , $W_0$ , $\eta$ , backbone): the concrete architectural choices that produce TTT-Linear and TTT-MLP, and the design rationale behind each.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and architecture paper whose core idea is that making the hidden state of an RNN a parametric model trained via self-supervised learning on the test sequence itself can overcome the expressive power bottleneck that limits existing RNNs in long context, provided that the resulting computation can be made efficient through mini-batch parallelization and dual-form reformulation.

The Core Mechanism: Hidden State as a Model, Update as Gradient Descent

The paper begins by reframing all sequence modeling layers through a common abstraction (Section 2, Figure 3): every such layer can be expressed as a hidden state that transitions over time according to an update rule, with an output rule that maps the current hidden state and current input to the output token. Under this lens, self-attention's hidden state is a growing list of key-value pairs (update rule: append; output rule: attend over the list), while a naive RNN's hidden state is a fixed-size vector (update rule: $\sigma(\theta_{ss} s_{t-1} + \theta_{sx} x_t)$ ; output rule: linear readout).

The paper's key idea is to make the hidden state a machine learning model itself, specifically the weights $W$ of a function $f$ . The hidden state at time $t$ , denoted $s_t$ , is now equivalent to $W_t$ , the parameters of $f$ after processing tokens $x_1, \ldots, x_t$ . The output rule becomes a prediction using the current model:

$z_t = f(x_t; W_t)$

where $z_t$ is the output token at position $t$ , $x_t$ is the input token, and $W_t$ is the model's weights after being trained on $x_1, \ldots, x_t$ .

What it computes: the prediction that the model $f$ , with its current weights $W_t$ (which encode a compressed representation of all prior context $x_1, \ldots, x_{t-1}$ ), makes when given the current input $x_t$ . Operationally, this is a forward pass through $f$ —for TTT-Linear, this is a matrix multiplication $W_t x_t$ ; for TTT-MLP, this is a two-layer network with a GELU nonlinearity.

Why this form: This output rule directly mirrors how parametric models make predictions. The weights $W_t$ have been optimized (via the update rule) to capture structure in the sequence so far, so the prediction $f(x_t; W_t)$ should reflect an understanding of how $x_t$ relates to that prior context. This is fundamentally different from a standard RNN, where the hidden state is a vector of activations that the model reads from—here, the hidden state is the model that does the reading.

The update rule is equally direct: it is a step of gradient descent on a self-supervised loss $\ell$ :

$W_t = W_{t-1} - \eta \nabla \ell(W_{t-1}; x_t)$

where $\eta$ is the learning rate and $\nabla \ell(W_{t-1}; x_t)$ is the gradient of the self-supervised loss with respect to the model weights, evaluated at the current weights $W_{t-1}$ on the current token $x_t$ .

What it computes: one step of gradient descent that updates the model parameters $W$ to better fit the self-supervised task on token $x_t$ . The gradient $\nabla \ell(W_{t-1}; x_t)$ measures how the loss would change if each parameter of $W$ were perturbed; subtracting $\eta$ times this gradient moves $W$ in the direction that reduces the loss. The result is a new set of weights $W_t$ that have learned something from $x_t$ .

Why this form: Gradient descent is the standard algorithm for training neural networks. By using it as the update rule, the paper imports the entire machinery of machine learning—objective functions, optimizers, model architectures—into the inner workings of a sequence modeling layer. This is the core conceptual move: the update rule is not a handcrafted gating mechanism but an optimization process that can be analyzed and improved using the tools of learning theory. Moreover, gradient descent naturally implements a compression heuristic: inputs that produce large gradients are "surprising" or "informative" under the current model and get remembered strongly; inputs that produce small gradients are already well-predicted and get remembered weakly.

The self-supervised task that defines $\ell$ is reconstruction. The model $f$ must reconstruct the original token $x_t$ from a corrupted version of it:

$\ell(W; x_t) = \|f(\tilde{x}_t; W) - x_t\|^2$

where $f(\tilde{x}_t; W)$ is the model's reconstruction from the corrupted input $\tilde{x}_t$ , $x_t$ is the reconstruction target (the original token), and $\|\cdot\|^2$ is the squared Euclidean norm.

What it computes: the squared L2 distance between the model's reconstruction and the original uncorrupted token. If $f$ can perfectly predict $x_t$ from $\tilde{x}_t$ , the loss is zero; if $f$ 's prediction is far from $x_t$ , the loss is large. The model must learn the correlations between dimensions of $x_t$ to fill in the information that was removed by the corruption process.

Why this form: Reconstruction is a classical self-supervised task that forces the model to learn the internal structure of the data. Like a denoising autoencoder [77], $f$ cannot simply memorize $x_t$ —it must discover how the dimensions of $x_t$ relate to each other to recover the missing information. The squared error is the standard choice for continuous-valued reconstruction tasks because it penalizes large errors quadratically and has convenient gradient properties. The paper explicitly notes that a more complex encoder-decoder design (adding a separate decoder $g$ after $f$ ) was tried and "slightly improved results" but "made overall training less stable and added significant computational cost," motivating the simpler encoder-only design.

The paper shows empirically (Figure 4) that gradient descent successfully reduces this reconstruction loss over time. On test sequences of length 2048, the loss $\ell(W_t; x_t)$ is consistently lower than the loss that would be achieved by the initial weights $\ell(W_0; x_t)$ , and the gap widens as $t$ increases—the model genuinely learns from the test sequence.

The Nested Learning Structure: Inner Loop vs. Outer Loop

The TTT layer is embedded in a larger neural network that is trained for next-token prediction, the standard language modeling objective. This creates two nested optimization problems that the paper carefully distinguishes (Section 2.2, Table 2).

The inner loop optimizes $W$ (the weights of $f$ ) using the self-supervised loss $\ell$ , with gradient descent as the optimizer. The inner loop runs during the forward pass of the larger network—every time the network processes a token, the inner loop takes a gradient step.

The outer loop optimizes $\theta_{\text{rest}}$ (all other parameters of the network, including the projections $\theta_K, \theta_Q, \theta_V$ , the initialization $\theta_{\text{init}} = W_0$ , and the learning rate parameters $\theta_{lr}$ ) using the next-token prediction loss. The outer loop runs during the backward pass of standard training—backpropagation flows through the entire forward pass, including through the inner loop's gradient operations.

The paper emphasizes a critical technical point: the forward pass contains the gradient operator $\nabla$ , which maps the loss function $\ell$ to its gradient $\nabla \ell$ . However, $\nabla$ is itself composed of differentiable operators, so calling backward on it—taking gradients of gradients—is mathematically well-defined. This is a standard technique in meta-learning [54], where second-order derivatives are routinely computed through optimization steps.

Why this nested structure matters: It means the inner loop's behavior can be learned by the outer loop. The outer loop can adjust $\theta_K, \theta_Q, \theta_V$ to change the self-supervised task that the inner loop solves, thereby changing what features $W$ learns from the test sequence. It can adjust $\theta_{\text{init}}$ to control the starting point of the inner loop's optimization. It can adjust $\theta_{lr}$ to control the inner loop's learning rate as a function of the token. Everything about how the TTT layer learns at test time is itself learned during training. The paper argues this is the key advantage over prior test-time training work, where the self-supervised task was handcrafted.

The outer loop is "at the same level" as regular training—it solves the same next-token prediction problem on the same dataset of sequences. The inner loop is "one level below" regular training—it solves a per-sequence self-supervised problem. This is in contrast to prior meta-learning work [2, 21] where the outer loop was "one level above" regular training, requiring a collection of datasets or tasks, which was hard to scale.

The Learned Self-Supervised Task

In the basic formulation (Equation 3), the corruption process that produces $\tilde{x}_t$ from $x_t$ was unspecified. The paper makes this corruption learnable by introducing three outer-loop parameters that define a family of multi-view reconstruction tasks (Section 2.3).

First, the corruption is made a low-rank projection $\tilde{x}_t = \theta_K x_t$ , where $\theta_K$ is a learnable matrix. The projected vector $\theta_K x_t$ has fewer dimensions than $x_t$ , so $f$ cannot simply learn the identity function—it must infer the missing dimensions from the correlations it has learned. The paper calls $\theta_K x_t$ the training view, borrowing terminology from multi-view contrastive learning [13].

Second, the reconstruction target is also made learnable: instead of reconstructing $x_t$ itself, the model reconstructs $\theta_V x_t$ , where $\theta_V$ is another learnable matrix. This is called the label view. The rationale is that "perhaps not all the information in $x_t$ is worth remembering"—the outer loop can learn which dimensions of $x_t$ are important to compress into $W$ and which can be discarded.

Third, the output rule needs to be modified because the training view $\theta_K x_t$ has fewer dimensions than $x_t$ , so $f(\theta_K x_t; W_t)$ would not match the original token dimension. The solution is to introduce a test view $\theta_Q x_t$ , where $\theta_Q$ is a third learnable matrix, and change the output rule to apply $f$ to this test view:

$z_t = f(\theta_Q x_t; W_t)$

What it computes: the model's prediction on the test view $\theta_Q x_t$ using the current weights $W_t$ . The test view may contain different information than the training view, giving the outer loop flexibility to route different aspects of $x_t$ to the output versus to the hidden state.

Why this form: The training and label views specify what information from $x_t$ gets compressed into $W_t$ and propagated forward through time (since $W_t$ is trained to map training views to label views). The test view specifies what information from $x_t$ gets mapped to the current output $z_t$ and propagated forward through network layers. By making these three views separate and learnable, the outer loop can optimize the tradeoff between what the hidden state remembers and what the output uses. The paper notes that all three views are designed as linear projections "for simplicity," but future work could use more flexible transformations or entirely different families of self-supervised tasks.

Putting it all together, the full self-supervised loss is:

$\ell(W; x_t) = \|f(\theta_K x_t; W) - \theta_V x_t\|^2$

where $\theta_K$ is the learnable matrix producing the training view (the corrupted input), $\theta_V$ is the learnable matrix producing the label view (the reconstruction target), $f(\theta_K x_t; W)$ is the model's reconstruction from the training view using weights $W$ , and $\|\cdot\|^2$ is the squared L2 norm.

What it computes: the reconstruction error between $f$ 's prediction from the training view and the label view. This is the same structure as Equation 3, but both the corruption ( $\theta_K$ ) and the target ( $\theta_V$ ) are now learned rather than fixed. The outer loop can make this task easier or harder by adjusting the projections, effectively selecting which correlations the inner loop needs to discover.

Why this form: Making the self-supervised task learnable addresses a fundamental limitation of prior test-time training work. The optimal self-supervised task likely depends on the domain (language vs. vision vs. robotics), the model architecture, and the end goal (next-token prediction). Rather than relying on human intuition to design this task, the outer loop automatically discovers what works through standard gradient-based optimization. The family of tasks parameterized by linear projections is simple but demonstrates the principle; the paper explicitly suggests exploring "more flexible transformations, or bigger and different families of self-supervised tasks" in future work.

A subtle point: $W$ and the various $\theta$ s appear together in Equation 4, but they have fundamentally different natures. $W$ is optimized in the inner loop (per-sequence, at test time) and is written as an argument of $\ell$ . The $\theta$ s are optimized in the outer loop (across sequences, during training) and are "hyper-parameters" of the loss function from the inner loop's perspective. Figure 5 illustrates this distinction with a code sketch: Task (containing $\theta_K, \theta_V, \theta_Q$ ) is a nn.Module whose parameters are optimized by the outer loop; Learner (containing $W$ ) is explicitly not a nn.Module and is updated manually in the inner loop.

Parallelization with Mini-Batch TTT

The naive update rule $W_t = W_{t-1} - \eta \nabla \ell(W_{t-1}; x_t)$ is sequential— $W_t$ depends on $W_{t-1}$ , which depends on $W_{t-2}$ , and so on—making it impossible to parallelize across time steps. This is a critical practical problem because modern hardware (GPUs and TPUs) requires parallel work to achieve high utilization.

The paper addresses this by recognizing that the update rule can be decomposed. The general form of gradient descent can be written as:

$W_t = W_{t-1} - \eta G_t = W_0 - \eta \sum_{s=1}^{t} G_s$

where $G_t$ is the descent direction at step $t$ , and the second equality shows that $W_t$ can be obtained from $W_0$ through a cumulative sum (cumsum) of the descent directions.

What it computes: The second form separates the computation into two stages: first compute all descent directions $G_1, \ldots, G_t$ , then compute all $W_t$ via a cumulative sum. The cumulative sum is inherently sequential but is a cheap element-wise operation. The expensive part is computing the $G_t$ 's, which involve gradient computations through $f$ .

Why this form: If the $G_t$ 's can be computed in parallel rather than sequentially, the overall computation becomes much faster. This decomposes the problem into making the gradient computations parallel, while the weight updates (cumsum) remain sequential—but the cumsum is cheap.

The naive update rule uses online gradient descent: $G_t = \nabla \ell(W_{t-1}; x_t)$ , where each gradient is taken with respect to the current weights $W_{t-1}$ . This requires knowing $W_{t-1}$ to compute $G_t$ , forcing sequential execution.

At the opposite extreme, batch gradient descent uses $G_t = \nabla \ell(W_0; x_t)$ , where all gradients are taken with respect to $W_0$ . This can be fully parallelized because $W_0$ is known at the start and doesn't depend on any $G_s$ . However, batch GD is problematic because $W_t$ ends up only one gradient step away from $W_0$ —the effective search space is much smaller than online GD, where $W_t$ is $t$ gradient steps away from $W_0$ . The paper reports that batch GD "ends up hurting performance for language modeling."

The paper's solution is mini-batch gradient descent (Section 2.4, Figure 6). The descent direction at time $t$ is:

$G_t = \nabla \ell(W_{t'}; x_t)$

where $t' = t - \text{mod}(t, b)$ is the last timestep of the previous mini-batch (or 0 for the first mini-batch), and $b$ is the TTT batch size. Within a mini-batch of size $b$ , all $b$ gradient computations use the same base weights $W_{t'}$ , so they can be computed in parallel. Between mini-batches, the weights are updated sequentially.

What it computes: For a sequence of length $T$ , the computation proceeds in $\lceil T/b \rceil$ mini-batches. In each mini-batch: compute $b$ gradients in parallel with respect to the weights at the start of that mini-batch, sum them to get the total update for the mini-batch, and apply the update to get the weights at the end of the mini-batch (which become the starting weights for the next mini-batch).

Why this form: Mini-batch GD interpolates between online GD ( $b = 1$ , fully sequential but $T$ gradient steps) and batch GD ( $b = T$ , fully parallel but only one effective gradient step). The parameter $b$ controls a tradeoff: smaller $b$ means more gradient steps (better optimization, better perplexity) but less parallelization (slower wall-clock time); larger $b$ means more parallelization but fewer effective gradient steps. Figure 7 (left) empirically validates this tradeoff: perplexity decreases monotonically as $b$ decreases from 2048 to 1. The paper selects $b = 16$ for all experiments as a practical balance.

The paper clarifies an important distinction between two information channels from $W_s$ to $W_t$ (where $s < t$ ). The cumsum channel is always active— $W_t$ always accumulates all previous descent directions through the cumulative sum, regardless of the GD variant. The gradient channel is only active when $W_s$ is from a previous mini-batch—the gradient at time $t$ , $\nabla \ell(W_{t'}; x_t)$ , itself depends on $W_{t'}$ , which was computed using tokens from earlier mini-batches. In online GD, the gradient channel is active at every step; in batch GD, it is never active; in mini-batch GD, it is active at mini-batch boundaries. The paper emphasizes that "the descent step $W_t = W_{t-1} - \eta G_t$ always starts from $W_{t-1}$ , due to the autoregressive nature of the update rule, which is orthogonal to the choice of $G_t$ ."

The Dual Form: Making TTT Hardware-Efficient

The mini-batch parallelization described above solves the sequential dependency problem but not the hardware efficiency problem. Modern accelerators like the NVIDIA A100 GPU contain specialized units (TensorCores) that can only perform one operation: multiplying two $16 \times 16$ matrices. Without enough matrix multiplications (matmuls), TensorCores sit idle. The naive TTT computation—even with mini-batching—still requires computing per-token gradient outer products, which are not matmuls.

Consider the simplest case: $f$ is a linear model ( $f(x; W) = Wx$ ), and all projection matrices are the identity ( $\theta_K = \theta_V = \theta_Q = I$ ), for the first mini-batch of size $b$ . The loss at time $t$ is:

$\ell(W_0; x_t) = \|f(x_t; W_0) - x_t\|^2 = \|W_0 x_t - x_t\|^2$

The gradient at time $t$ is:

$G_t = \nabla \ell(W_0; x_t) = 2(W_0 x_t - x_t) x_t^T$

where $W_0 x_t - x_t$ is a vector (the prediction error) and $x_t^T$ is a row vector, so the product is a $d \times d$ matrix (an outer product). Computing $b$ such outer products one by one requires $b$ separate operations, each involving a vector-vector outer product—this cannot be batched into a single matmul. Moreover, each $G_t$ is $d \times d$ , which for large $d$ (e.g., $d = 2048$ ) incurs heavy memory and I/O costs.

The key observation of the dual form (Section 2.5): We do not actually need to materialize $G_1, \ldots, G_b$ or the intermediate $W_1, \ldots, W_{b-1}$ individually, as long as we can compute $W_b$ (the weights at the end of the mini-batch) and the output tokens $z_1, \ldots, z_b$ . These can be computed directly through matmuls.

The paper derives the dual form for the simplified TTT-Linear case. Denote $X = [x_1, \ldots, x_b]$ as the matrix whose columns are the $b$ tokens in the mini-batch. Then the weights at the end of the mini-batch are:

$W_b = W_0 - \eta \sum_{t=1}^{b} G_t = W_0 - 2\eta \sum_{t=1}^{b} (W_0 x_t - x_t) x_t^T = W_0 - 2\eta (W_0 X - X) X^T$

where $W_0 X - X$ is a $d \times b$ matrix of prediction errors for all $b$ tokens, and the product $(W_0 X - X) X^T$ is a $d \times d$ matrix that can be computed with a single matmul.

What it computes: $W_b$ directly from $W_0$ and $X$ through two matrix multiplications: first compute $W_0 X$ (a matmul), subtract $X$ , then multiply by $X^T$ . No per-token outer products, no intermediate $W_t$ 's, no $G_t$ 's need to be stored.

Why this form: Converting the sum of outer products into a matrix product $(W_0 X - X) X^T$ is the key algebraic insight. A sum of outer products $\sum_t a_t b_t^T$ is exactly the product of a matrix $A = [a_1, \ldots, a_b]$ and $B^T$ where $B = [b_1, \ldots, b_b]$ . Here $a_t = W_0 x_t - x_t$ and $b_t = x_t$ , so $A = W_0 X - X$ and $B = X$ . This is a standard identity that the paper exploits to make the computation hardware-friendly.

The output tokens $z_1, \ldots, z_b$ require more care because $z_t = W_t x_t$ uses $W_t$ (the weights after processing $t$ tokens), not $W_0$ . Expanding:

$z_t = W_t x_t = \left(W_0 - \eta \sum_{s=1}^{t} G_s\right) x_t = W_0 x_t - 2\eta \sum_{s=1}^{t} (W_0 x_s - x_s) x_s^T x_t$

Define $\delta_t = \sum_{s=1}^{t} (W_0 x_s - x_s) x_s^T x_t$ and $\Delta = [\delta_1, \ldots, \delta_b]$ . The paper derives that:

$\Delta = (W_0 X - X) \cdot \text{mask}(X^T X)$

where $\text{mask}$ is a triangular mask with zeros above the diagonal (similar to an attention mask, but with zeros instead of $-\infty$ ), so that $\delta_t$ only accumulates terms for $s \leq t$ .

What it computes: The matrix $\Delta$ where column $t$ contains the correction term that must be subtracted from the naive prediction $W_0 x_t$ to account for the fact that $W_t$ has been updated by tokens $x_1, \ldots, x_t$ . The mask ensures causality: token $t$ 's output only depends on tokens up to $t$ .

Why this form: The term $X^T X$ is a $b \times b$ matrix of pairwise inner products between all tokens in the mini-batch, computable with a single matmul. Applying the mask to this matrix and multiplying by $(W_0 X - X)$ yields $\Delta$ entirely through matmuls. The final output matrix is $Z = W_0 X - 2\eta \Delta$ .

The time complexity analysis reveals the tradeoff. The primal form has complexity $O(b \times d^2)$ for computing the $b$ gradient matrices. The dual form has complexity $O(b \times d^2)$ for computing $W_b$ (the matmul $(W_0 X - X) X^T$ ) plus $O(b^2 \times d)$ for computing $z_1, \ldots, z_b$ (the matmul for $X^T X$ is $O(b^2 \times d)$ and dominates). The dual form is theoretically $O(b \times d)$ worse than the primal for the output computation, but in practice $d$ is a few hundred and $b$ is only 16, so the $b^2 \times d$ term is small. The paper reports that "training with the dual form is more than 5× faster than with primal" in their JAX implementation.

The paper notes that the dual form can be extended to general MLPs with arbitrary depth and nonlinearities (Appendix A), albeit with more complex notation. The key idea—avoiding materialization of per-token gradients by formulating the computation in terms of matrix products—generalizes.

Theoretical Equivalences: TTT as a Unifying Framework

The paper demonstrates that TTT layers are not just a new architecture but a unifying framework that recovers several existing sequence modeling layers as special cases (Section 2.6, Figures 8 and 9). This is important because it positions TTT layers as a generalization rather than a competitor.

Theorem 1 (Equivalence to linear attention). Consider the TTT layer with:

$f(x) = Wx$ (a linear model as the inner model)
Batch gradient descent ( $G_t = \nabla \ell(W_0; x_t)$ ) with $\eta = 1/2$
$W_0 = 0$

Then, given the same input sequence $x_1, \ldots, x_T$ , the output sequence matches that of linear attention [44].

Proof sketch: With batch GD and $W_0 = 0$ , the weights at time $t$ are $W_t = \sum_{s=1}^{t} (\theta_V x_s)(\theta_K x_s)^T$ . The output token is $z_t = f(\theta_Q x_t; W_t) = \sum_{s=1}^{t} (\theta_V x_s)(\theta_K x_s)^T (\theta_Q x_t)$ , which is exactly the definition of linear attention (self-attention without the softmax, where $(\theta_K x_s)^T (\theta_Q x_t)$ is the attention score between positions $s$ and $t$ , and $\theta_V x_s$ is the value at position $s$ ).

What this means: Linear attention is the special case of TTT where the inner model is linear, the optimizer is batch GD (the weakest form in terms of effective search), and the initialization is zero. The paper's improvements—mini-batch GD, learnable $W_0$ , LayerNorm and residual connections in $f$ —all move beyond this special case. Table 1 shows the progressive improvement: starting from the TTT equivalence to linear attention (perplexity 15.23), adding learnable $W_0$ (15.27, slight regression), adding LN and residual in $f$ (14.05, -1.22), switching to mini-batch TTT (12.35, -1.70), adding learnable $\eta$ (11.99, -0.36), and adopting the Mamba backbone (11.09, -0.90). The mini-batch change produces the largest improvement.

Theorem 2 (Equivalence to self-attention). Consider the TTT layer with the Nadaraya-Watson estimator [6, 11] as the learner, defined as:

$f(x; x_1, \ldots, x_t) = \frac{1}{\sum_{s=1}^{t} \kappa(x, x_s)} \sum_{s=1}^{t} \kappa(x, x_s) y_s$

where $y_s = \theta_V x_s$ is the label view, and $\kappa(x, x'; \theta_K, \theta_Q) \propto e^{(\theta_K x)^T \theta_Q x'}$ is an asymmetric kernel.

Then, given the same input sequence, the output matches self-attention.

What this means: Self-attention is the special case of TTT where the learner is nonparametric—it doesn't maintain explicit weights $W$ but instead stores all training data and makes predictions via kernel-weighted averaging. The hidden state is the list of past tokens (or their processed representations), the update rule is appending to this list, and the output rule is the kernel-weighted sum. This illustrates that TTT layers can represent both parametric learners (with fixed-size hidden states, $O(1)$ cost per token) and nonparametric learners (with growing hidden states, $O(t)$ cost per token), as summarized in Figure 9.

The paper introduces a new abstraction to unify these cases: a learner, which must implement two methods—train and predict. The hidden state of the induced TTT layer is the learner's internal storage. For parametric learners, this includes the model weights $W$ and potentially optimizer state (enabling future use of Adam); for nonparametric learners, it is the list of training data. This abstraction allows TTT layers to encompass both RNN-style fixed-size states and attention-style growing states within a single framework.

Implementation Details: Instantiations, Initialization, and Backbone

The paper specifies several concrete design choices that produce the two instantiations evaluated in experiments (Section 2.7). These choices are not arbitrary; each has a specific rationale grounded in the TTT framework.

Instantiations of the inner model $f$ . The paper proposes two variants:

TTT-Linear: $f_{\text{lin}}(x) = Wx$ , where $W$ is a square matrix of size $d \times d$ (where $d$ is the embedding dimension). This is the simplest parametric model—purely linear with no nonlinearity or hidden layers. The hidden state is a single matrix.
TTT-MLP: $f_{\text{MLP}}$ is a two-layer MLP mimicking the structure of MLPs in Transformers. Specifically, the hidden dimension is $4\times$ the input dimension, followed by a GELU activation. This means: $f_{\text{MLP}}(x) = W_2 \cdot \text{GELU}(W_1 x)$ , where $W_1$ is $4d \times d$ and $W_2$ is $d \times 4d$ . The hidden state consists of two matrices rather than one, substantially increasing expressive power at the cost of more parameters to update at test time.

For both instantiations, $f$ always contains a Layer Normalization (LN) and a residual connection for stability during test-time training:

$f(x) = x + \text{LN}(f_{\text{res}}(x))$

where $f_{\text{res}}$ is either $f_{\text{lin}}$ or $f_{\text{MLP}}$ .

Why this design: The LayerNorm and residual connection serve two purposes. First, they improve training stability during the inner loop—gradient descent through a deep or wide model can be unstable, and normalization helps control the scale of activations and gradients. Second, the residual connection means $f$ is learning a correction to the identity mapping rather than learning the full output from scratch, which is an easier optimization problem. Table 1 shows that adding LN and residual to $f$ (for the linear model case) reduces perplexity from 15.27 to 14.05, a substantial improvement that validates this design choice.

Learnable $W_0$ . The initial weights $W_0$ are shared across all sequences but can be learned as an outer-loop parameter, denoted $\theta_{\text{init}} = W_0$ . This is in contrast to the earlier theoretical examples where $W_0 = 0$ . The paper notes that $\theta_{\text{init}}$ "adds a negligible amount of parameters" because both its input and output are low-dimensional (after the projection by $\theta_K, \theta_V, \theta_Q$ ). Empirically, learning $W_0$ "significantly improves training stability"—Table 1 shows that the rows below learnable $W_0$ "cannot train stably without it."

Why this design: Starting from a learned $W_0$ means the inner loop begins from a sensible initialization rather than from zero. This is analogous to how pretrained models are used as initialization for fine-tuning: the model starts with useful features and only needs to adapt to the specific sequence. In the TTT framework, $W_0$ can be thought of as encoding general knowledge from the training distribution, while the inner loop adapts to sequence-specific patterns. This also explains why batch GD performs poorly—with $W_0 = 0$ and batch GD, the model is effectively a linear attention variant that lacks the ability to accumulate knowledge across mini-batches through the gradient channel.

Learnable learning rate $\eta$ . The learning rate is typically the most important hyperparameter for gradient descent, so the paper makes it learnable and token-dependent. The formulation is:

$\eta(x) = \eta_{\text{base}} \cdot \sigma(\theta_{lr} \cdot x)$

where $\eta_{\text{base}}$ is a scalar base learning rate (set to 1 for TTT-Linear and 0.1 for TTT-MLP), $\sigma$ is the sigmoid function (squashing outputs to $(0, 1)$ ), $\theta_{lr}$ is a learnable vector (outer-loop parameter), and $\theta_{lr} \cdot x$ is the dot product between the learning rate parameters and the input token.

What it computes: A per-token learning rate multiplier between 0 and $\eta_{\text{base}}$ , determined by a learned linear function of the token followed by sigmoid. Tokens that are more "surprising" or informative (as learned by $\theta_{lr}$ ) get higher effective learning rates, making the inner loop learn more from them.

Why this form: Making $\eta$ a function of the input token gives the outer loop fine-grained control over how much the inner loop learns from each token. The sigmoid keeps the learning rate in a reasonable range. The paper notes that $\eta(x)$ "can also be interpreted as a gate for $\nabla \ell$ "—it selectively scales the gradient based on the token's content. The base learning rate is set differently for TTT-Linear (1) and TTT-MLP (0.1), reflecting that the MLP inner model is deeper and likely requires a smaller learning rate for stable optimization, similar to how larger models often use smaller learning rates in regular training.

Backbone architecture. The paper's initial approach was to directly replace self-attention with TTT layers in a Transformer backbone (the standard architecture of alternating attention and MLP blocks with residual connections and layer norms). However, they found that using the Mamba backbone—which includes temporal convolutions before the sequence modeling layers and a gating mechanism—improves perplexity for TTT layers. The Mamba backbone, shown in Figure 13 (right), consists of: a temporal convolution, a gating branch (GELU-activated linear projection), and the TTT layer, all wrapped in residual connections and LayerNorm.

The paper makes two observations about backbone effects. First, TTT layers with Mamba backbone consistently perform better than with Transformer backbone in their evaluations. Second, with the Mamba backbone, TTT-MLP is at best comparable to TTT-Linear; but with the Transformer backbone, TTT-MLP is clearly better. The paper hypothesizes that "the temporal convolutions in the Mamba backbone help more when the sequence modeling layer has a less expressive hidden state"—the linear model benefits more from the convolutions because it needs the help, while the MLP can compensate with its own expressiveness. This suggests that for even more expressive inner models beyond MLPs, the temporal convolutions might become unnecessary, and the paper explicitly predicts that "given TTT layers with even more expressive hidden states, the Mamba backbone with temporal convolutions will become unnecessary."

To accommodate the Mamba backbone's gating mechanism without changing the embedding dimension, the paper combines $\theta_K$ and $\theta_Q$ into a single projection—the gate provides separate pathways for what enters training versus testing, so separating $\theta_K$ and $\theta_Q$ becomes redundant.

4. Key Insights and Innovations

Innovation 1: Reframing RNN Hidden States as Learned Optimization Problems

The dominant paradigm for designing RNN hidden states has been to handcraft an update rule — a gating mechanism, a state transition function, a convolution — that the network learns to parameterize during training. LSTMs [33] introduced input, forget, and output gates; Mamba [27] introduced input-dependent state-space parameters; RWKV [58] introduced linear attention-like recurrence with exponential decay. In all cases, the form of the update rule is fixed by the designer, and only the parameters of that rule are learned. What information enters the hidden state, how it interacts with existing information, and what gets forgotten are all determined by the architecture's structural priors.

This paper makes a fundamentally different move: the update rule is not a handcrafted function but an explicit optimization process — gradient descent on a self-supervised loss. The hidden state is not a vector or matrix of summary statistics but the parameters of a model that has been trained, via gradient steps, on the sequence it has seen so far. The update rule is not "multiply by a gate and add" but "compute the gradient of the self-supervised loss on the current token and take a step."

This reframing matters for three conceptual reasons. First, it changes the unit of analysis for what makes a good hidden state. Instead of asking "what gating structure should we design?", the question becomes "what learning problem should the inner model solve?" and "what optimizer should it use?" These are questions that the machine learning community has decades of experience answering, and the TTT framework makes that experience directly transferable to architecture design. The paper demonstrates this concretely by showing that switching from batch GD to mini-batch GD — a standard practice in ML training — produces the largest single improvement in perplexity (Table 1, −1.70 perplexity reduction). The design intuition came from understanding optimization, not from RNN-specific architectural innovation.

Second, it changes the nature of compression in the hidden state. A gated RNN compresses context by selectively retaining information based on learned gate values. A TTT layer compresses context by training a model to reconstruct it. The model's weights encode what the optimizer — guided by the self-supervised task — determined was important. This is a qualitatively different compression heuristic. The paper's Figure 4 provides evidence that it works differently in practice: the self-supervised loss on test sequences decreases steadily as more tokens are processed, meaning the inner model genuinely learns from the sequence in a way that improves its predictions. This is not a static encoding that saturates; it is an ongoing learning process.

Third, it creates a unified framework for understanding sequence modeling layers that was previously fragmented. Theorem 1 shows that linear attention is TTT with a linear model, batch GD, zero initialization, and a specific learning rate. Theorem 2 shows that self-attention is TTT with a nonparametric kernel regression learner. These are not incidental connections — they reveal that apparently unrelated architectures are all solving the same meta-problem (learn from context to predict the next token) but with different choices of model class and optimizer. Figure 9 makes this unification explicit: TTT layers form a superset that contains both RNNs (via parametric learners with fixed-size states) and attention (via nonparametric learners with growing states). Prior work treated these as fundamentally different primitives; the paper shows they are special cases of a single abstraction.

This reframing is a fundamental intellectual contribution, not an incremental one. It doesn't just propose a new architecture — it proposes a new way of thinking about and designing architectures. Whether TTT-Linear and TTT-MLP specifically become widely adopted is secondary to this conceptual shift.

The empirical anchor is Table 1, which decomposes the improvement from linear attention (perplexity 15.23) to TTT-Linear (perplexity 11.09) into individual design choices. The biggest contributor is moving from batch GD (b = T = 2048) to mini-batch GD (b = 16), which is precisely the optimization-inspired choice that the framework enables but that prior RNN design paradigms would not have surfaced. The second biggest contributor is adding LayerNorm and residual connections to the inner model — again, a standard training stability technique from ML that the framework makes natural to apply.

Innovation 2: Learning the Self-Supervised Task Rather Than Handcrafting It

Prior work on learning at test time — in computer vision [72, 23, 79], in NLP as dynamic evaluation [47, 48], and in the broader fast weights tradition [74, 38] — has uniformly relied on human-designed self-supervised tasks. Test-Time Training for images used a predetermined rotation prediction or jigsaw puzzle task. Dynamic evaluation finetuned directly on the next-token prediction objective itself. Fast weight programmers used fixed Hebbian-style or delta-rule updates. The task that the inner loop solves was specified by the researcher, not learned.

This paper's second major conceptual move is to make the self-supervised task itself a learned object, optimized by the outer loop for the end goal of next-token prediction. The three projection matrices $\theta_K$ , $\theta_Q$ , and $\theta_V$ parameterize a family of multi-view reconstruction tasks — they determine what the inner model sees as input, what it tries to predict, and what it outputs — and these projections are trained jointly with the rest of the network through standard backpropagation. The paper's framework transforms the question from "what self-supervised task should we design?" into "what family of tasks should we parameterize, and can the outer loop find a good one?"

This is significant because it addresses a known brittleness in test-time training methods. The original TTT paper [72] found that rotation prediction helped for ImageNet-C corruptions, but subsequent work [23] found that masked autoencoding worked better. Different domains, architectures, and end-tasks likely benefit from different self-supervised objectives. Hand-designing these per application is expensive and error-prone. By learning the task, the paper makes TTT self-tuning — the optimal self-supervised objective emerges from data.

What makes this particularly elegant is that the learned task has an interpretable structure. The three views — training, label, and test — map to a clear conceptual division: the training and label views define what information gets compressed into the hidden state (since $W$ is trained to map training views to label views), while the test view defines what information gets routed to the output (since $z_t = f(\theta_Q x_t; W_t)$ ). The outer loop can learn to route different aspects of each token to memory versus to immediate output, and this routing is optimized for the downstream language modeling objective rather than for a proxy like reconstruction quality.

The paper notes this is an initial step — the family of tasks is parameterized by simple linear projections "for simplicity" — but the architectural pattern is general. Future work could parameterize richer families of tasks, or entirely different types of self-supervised objectives beyond reconstruction. The key innovation is the meta-principle: the inner loop's learning problem should itself be learnable.

The empirical evidence for this innovation's importance is distributed across the paper rather than isolated in a single ablation. The $\theta_K, \theta_Q, \theta_V$ parameters constitute a significant fraction of the TTT layer's outer-loop parameters, and the fact that TTT-Linear outperforms the linear attention baseline (which uses fixed views) by a large margin (Table 1) after accounting for other improvements suggests these learned projections matter. However, the paper does not include a clean ablation of "learned views vs. fixed views" with all other components held constant, which is a minor empirical gap. The conceptual contribution — making the self-supervised task a first-class learned object — is the main intellectual novelty, and its validation is partly architectural (the framework enables it) and partly empirical (the resulting system works well).

Innovation 3: Diagnosing and Addressing the Long-Context Plateau as an Expressive Power Problem

The paper's opening figure (Figure 2) frames the entire investigation around a specific diagnostic observation: Mamba's perplexity stops improving after 16k context, even though it has the computational capacity to process arbitrarily long sequences. This is not presented as a benchmarking result but as a symptom with a specific hypothesized cause — the expressive power of the hidden state — and the paper's technical approach is a direct response to that diagnosis.

This diagnostic framing is itself a contribution. Prior work on long-context RNNs had acknowledged that they perform worse than Transformers on long sequences, but typically attributed this to training difficulties, optimization challenges, or insufficient model scale. The paper's Figure 2 (right) isolates the phenomenon cleanly: it is not that Mamba performs poorly overall, but specifically that it stops improving when given more information. This is the signature of a representation bottleneck — the hidden state is full, and additional tokens can't change it in useful ways. The paper connects this directly to the 2020 Kaplan et al. [43] finding that LSTMs couldn't effectively use long context, showing that the problem persists in modern RNNs despite five years of architectural progress.

The significance of this diagnosis goes beyond Mamba. It implies that the fundamental challenge for RNNs is not finding the right gating mechanism or state-space parameterization — it is that any fixed-size vector or matrix can only store so much structured information, regardless of how cleverly that storage is managed. The paper's response — make the hidden state a model that can learn, not just store — is conceptually different from prior attempts to increase RNN capacity (making the state larger, using matrix-valued states, adding more complex transition functions). Those are expansions within the same paradigm; this is a paradigm shift from storage to learning.

The evidence for this diagnosis is strengthened by the paper's scaling behavior. TTT-MLP, whose hidden state (a two-layer MLP) has higher capacity than TTT-Linear's (a linear model), shows larger advantages at longer context lengths. In Figure 2 (right) and Figure 11, TTT-MLP with matched FLOPs performs worse than TTT-Linear at short context but better at long context — the extra capacity is wasted when there isn't much to compress but pays off when the context is rich. This capacity-dependent long-context advantage is exactly what the diagnostic framework predicts. The paper's observation that "TTT-MLP (T) with the Transformer backbone performs slightly better than Mamba at 32k context" despite using the weaker backbone further reinforces that the inner model's expressiveness is the active ingredient.

This innovation is primarily diagnostic and conceptual: it identifies what kind of problem the long-context plateau is (expressive power, not optimization or scale) and thus what kind of solution is needed (more expressive compression, not just better gating). The TTT framework is one answer to that diagnosis, but the diagnosis itself is separable — even if TTT layers prove not to be the ultimate solution, correctly identifying the bottleneck as representational capacity rather than training methodology directs future work more productively.

Innovation 4: Making Test-Time Learning Practical Through Systems-Aware Algorithm Design

The idea of using gradient descent inside a neural network's forward pass is not new — it has appeared in meta-learning [2, 21], fast weight programmers [38], and various forms of inner-loop optimization. These prior approaches have consistently faced a practicality barrier: training a model at test time is computationally expensive, often prohibitively so. The intellectual contribution here is not the idea itself but the demonstration that algorithmic design choices — mini-batch gradient descent and the dual form — can transform this idea from a theoretical curiosity into something that runs competitively on modern hardware.

This is a systems-aware algorithmic innovation, distinct from a pure systems contribution (which would be implementation-level optimizations) or a pure algorithmic contribution (which would be FLOPs improvements independent of hardware). The paper identifies two specific bottlenecks that make naive TTT impractical — sequential dependencies preventing parallelization, and per-token outer products preventing efficient matmul utilization — and addresses each with a design choice that has a clear rationale grounded in both optimization theory and hardware architecture.

The mini-batch choice (Section 2.4) is subtle. It is easy to dismiss as "just using mini-batch GD," but the paper provides the right analysis to understand why it is the right choice for this setting. The decomposition of the update rule into a cumsum channel (always active) and a gradient channel (active only at mini-batch boundaries) reveals that mini-batch GD is not merely a compromise between online and batch GD — it preserves the most important property of online GD (weights that are many gradient steps from $W_0$ ) while enabling the parallelization of batch GD within each mini-batch. Figure 7 quantifies this tradeoff with a sweep over $b$ , showing that perplexity improves continuously as $b$ decreases but wall-clock time has a U-shaped curve — there is a genuine optimal point ( $b = 16$ ) rather than a monotonic "smaller is better for accuracy but slower" relationship.

The dual form (Section 2.5) is where the paper's systems thinking is most distinctive. The key observation — "we do not actually need to materialize $G_1, \ldots, G_b$ " — is simple in retrospect but requires recognizing that the output tokens $z_1, \ldots, z_b$ can be computed directly from $W_0$ and $X$ without ever forming the intermediate per-token gradient matrices. This is not an approximation or a heuristic; the dual form is mathematically equivalent to the primal form (Appendix A proves this for general MLPs with arbitrary depth and nonlinearities). The improvement is purely in how the computation is organized to match hardware capabilities. The 5× speedup transforms TTT from "works in theory but too slow" to "10% faster per training iteration than Transformer at 2k context" (Section 3.3).

What makes this an innovation rather than an implementation detail is that it changes the design space for test-time learning. Prior work that attempted inner-loop optimization in neural networks was constrained by what could run efficiently; the dual form expands the set of inner-loop models, optimizers, and tasks that are practical. TTT-MLP, which has a substantially more expensive inner model than TTT-Linear, is only feasible because of these efficiency improvements — and even then, the paper is candid that TTT-MLP still faces wall-clock time challenges (Figure 12 shows it is 2-3× slower than TTT-Linear for generation). The dual form doesn't solve all efficiency problems, but it shifts the frontier of what is practical.

This innovation is incremental in the sense that it builds on known techniques (mini-batch GD, matrix identity $\sum_t a_t b_t^T = AB^T$ ) but fundamental in its effect: without it, the TTT framework would be a conceptual contribution without empirical validation at scale. The paper's experiments at 125M to 1.3B parameters, training on billions of tokens, are only possible because of these systems choices.

Innovation 5: Nested Learning That Operates "One Level Below" Rather Than "One Level Above"

The final conceptual innovation is structural rather than architectural: the paper's formulation of learning-to-learn nests the inner loop inside standard supervised learning rather than above it. Prior meta-learning work [21, 2, 55] constructed outer loops that trained across multiple datasets or tasks — each inner loop learned from a dataset, and the outer loop learned how to learn across datasets. This framing required collections of tasks (e.g., few-shot classification across many classes, or reinforcement learning across many environments) and was inherently "one level above" standard training: the outer loop was a new problem setting, not the canonical supervised learning problem.

Table 2 makes the contrast explicit. In the TTT framework, each test sequence is a dataset that defines its own learning problem. The inner loop trains a model on this per-sequence dataset. The outer loop trains the network on the standard next-token prediction objective over a dataset of sequences — the same objective, same data, and same training pipeline as any language model. The inner loop is "one level below" regular training, not above it. This means the outer loop is not a new problem setting requiring new data or new infrastructure; it is just another way of solving supervised learning.

This structural choice has major scaling implications that the paper leverages. Because the outer loop is standard next-token prediction, TTT layers can be trained using the same Chinchilla recipe, the same Pile and Books3 datasets, and the same model sizes as Transformers and Mamba. There is no need for meta-learning-specific data collection (e.g., constructing thousands of few-shot episodes) or specialized training procedures. This is what enables the paper's evaluation at 125M to 1.3B parameters — scales that would be prohibitively expensive under a "one level above" meta-learning paradigm.

The conceptual significance extends beyond engineering convenience. It suggests a different relationship between learning and architecture than the field has typically assumed. In the standard view, an architecture (Transformer, Mamba, LSTM) is a fixed computational structure, and learning is something that happens to the architecture's parameters during training. In the TTT view, learning is something that the architecture does internally during its forward pass, and the architecture itself (the TTT layer's outer-loop parameters) is learned to facilitate that internal learning. This blurs the boundary between "architecture" and "learning algorithm" in a way that the paper's learner abstraction (Figure 8) formalizes.

Empirically, this innovation is validated by the fact that TTT layers are trained using the same recipe and data as the baselines, with no special accommodations. The paper's results are a fair comparison because the outer loop is genuinely the same problem. The theoretical equivalences (Theorems 1 and 2) further validate the structural claim by showing that the TTT framework cleanly contains both attention and linear attention as special cases with specific choices of learner — it is genuinely a generalization, not an unrelated alternative.

This innovation is the most abstract of the five, and its importance is primarily conceptual: it defines a new position on the spectrum between "fixed architecture, learned parameters" and "learned architecture, learned parameters" that is practically scalable in a way that prior meta-learning approaches were not. Whether future work builds on TTT layers specifically or on different instantiations of this nesting principle, the structural insight that learning-to-learn can be placed "below" rather than "above" is a reusable conceptual contribution.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses two datasets. The primary dataset is the Pile [24], an 800GB corpus of diverse text widely used for training open-source LLMs, with experiments at 2k and 8k context lengths following the Mamba paper's protocol [27]. For long-context evaluation, the paper uses Books3, a subset of the Pile containing book-length texts, which "has been widely used to train LLMs in long context" [52, 3]. Experiments on Books3 cover context lengths from 1k to 32k in 2× increments. The paper notes that the Pile "contains few sequences of length greater than 8k" [19], motivating the switch to Books3 for long-context experiments.
Base model(s). All experiments train models from scratch at four scales: 125M, 350M, 760M, and 1.3B parameters (Section 3, "Protocols"). The 125M configuration uses 12 blocks with embedding dimension 768; the 1.3B uses 24 blocks with embedding dimension 2048 (Table 3). For Mamba baselines, the corresponding sizes are 130M, 370M, 790M, and 1.4B, since Mamba does not follow the Transformer multi-head residual block structure. The Transformer baseline follows the Llama architecture [75] with rotary positional encodings (RoPE) [69], SwiGLU MLP blocks [66], and RMSNorm [84] instead of LayerNorm—the same "Transformer++" used in the Mamba paper.
Metrics. The primary metric throughout is perplexity, the exponentiated average negative log-likelihood of the next token under the model, lower being better. The paper uses the log scale for perplexity in all scaling plots (Figures 2, 10, 11) because scaling laws are conventionally analyzed in log-log space [43, 34]. For Figure 2 (right), a secondary metric is perplexity as a function of token index—the average perplexity of tokens at each position in the sequence, measuring whether later tokens (with more context) are easier to predict.
Baselines. Two primary baselines are used throughout. Transformer: a strong Llama-based Transformer trained with the same Chinchilla recipe, following the "Transformer++" from the Mamba paper. Mamba [27]: identified as "one of the most popular RNNs today," using the public code provided by the authors. For long-context experiments (4k and above), an additional baseline TF finetune is added, which starts from the model trained on Books 2k and finetunes with 20% more tokens at the designated long context length, following the Llama Long paper [81]. This baseline "reflects the standard practice" for deploying Transformers in long context, as training from scratch in long context is rarely done in practice.
Generation budget / compute accounting. Compute is measured in FLOPs (floating point operations), following the Chinchilla scaling law methodology [34]. All models at a given parameter count are trained with matched training FLOPs—specifically, using the same number of training tokens (Table 3) and the same batch size of 0.5M tokens per batch regardless of context length. The paper emphasizes that for TTT-Linear and TTT-MLP, "matched training FLOPs also imply matched inference FLOPs," whereas the Transformer baseline has 2.8× the inference FLOPs compared to Mamba, giving it an advantage as a strong baseline (Appendix C). For the wall-clock time experiments (Section 3.3), forward (prefill) and generate (decode) latency are measured in seconds per token on an NVIDIA A100 GPU with 80GB HBM, using vLLM [49] for the Transformer baseline to reflect state-of-the-art serving systems.
Cross-validation / statistical protocol. The paper does not use cross-validation for model selection. All models are trained from scratch with the Chinchilla recipe, which prescribes specific training steps and learning rates by model size (Table 3). The Transformer finetuning baseline tries three peak learning rates (1e-5, 1e-4, 1e-3) and selects the best perplexity per model size. For context lengths 4k and above on Books, the Transformer baseline uses a RoPE angle of θ = 500,000 instead of the default 10,000, following the Llama Long paper, and uses whichever angle gives better perplexity. The paper is explicit that it does "not experiment with hybrid architectures" because "they would reduce the clarity of our academic evaluation."

Main Quantitative Results

Short Context: Scaling Trends on the Pile (2k and 8k)

The Pile experiments establish the baseline performance of TTT layers against Transformer and Mamba at standard context lengths. Results are presented in Figure 10.

At 2k context (Figure 10, left): TTT-Linear (M) and Mamba have "comparable performance, as the lines mostly overlap." Transformer also roughly overlaps with both. TTT-MLP (M) "performs slightly worse under large FLOP budgets"—even though TTT-MLP has better perplexity than TTT-Linear at every model size, the extra FLOP cost of the MLP hidden state offsets the perplexity advantage, making it less efficient in FLOPs-matched comparison. This is an important nuance: raw perplexity is better for TTT-MLP, but Figure 10 plots FLOPs on the x-axis, so the higher FLOP cost shifts TTT-MLP's points rightward, making the FLOPs-normalized comparison less favorable.

At 8k context (Figure 10, right): Both TTT-Linear (M) and TTT-MLP (M) "perform significantly better than Mamba, in contrast to the observation at 2k." TTT-MLP (T)—TTT-MLP with the Transformer backbone—"performs slightly better than Mamba around 1.3B," which is notable because the Transformer backbone lacks the temporal convolutions that benefit Mamba. The paper observes: "A robust phenomenon we observe throughout this paper is that as context length grows longer, the advantage of TTT layers over Mamba widens." At 8k, Transformer still has "good (if not the best) perplexity at every model size, but its line is not competitive because of the cost in FLOPs"—the quadratic cost penalty pushes Transformer's points far to the right on the FLOPs axis.

Lack of linear fit: The paper explicitly notes that "we do not observe a clean linear fit in Figure 10 or Figure 11... not even for Transformers." This is attributed to "differences in dataset, context length, tokenizer, and architecture" from the Chinchilla paper's original setting. Following the Mamba paper's practice, the points are connected rather than fitted with linear regression.

Long Context: Scaling Trends on Books (2k to 32k)

The Books experiments are the paper's central empirical contribution, testing whether TTT layers can use long context more effectively than Mamba. Complete results for all context lengths (1k, 2k, 4k, 8k, 16k, 32k) and all model sizes are in Figure 15 (Appendix), with subsets shown in Figure 11.

At 2k context on Books (Figure 11, left): The observations from Pile 2k generally hold, "except that Mamba now performs slightly better than TTT-Linear (whereas their lines roughly overlapped for Pile 2k)." This dataset-specific difference is minor but noted for completeness.

At 32k context on Books (Figure 11, right): Both TTT-Linear (M) and TTT-MLP (M) "perform better than Mamba, similar to the observation from Pile 8k." TTT-MLP (T) with the Transformer backbone "performs slightly better than Mamba at 32k context," which is a stringent test because Mamba's backbone is designed for long-range dependencies while the Transformer backbone for TTT layers lacks temporal convolutions. At 1.3B scale, TTT-MLP (T) is "only slightly worse than TTT-MLP (M)," and the paper interprets the trend as suggesting "that the Transformer backbone might be more suitable for larger models and longer context beyond our evaluations."

Capacity-dependent long-context advantage: The paper's most revealing result for the TTT framework is shown in Figure 2 (right), which plots perplexity as a function of token index. TTT-MLP, which has higher capacity (a two-layer MLP hidden state) than TTT-Linear (a linear model hidden state), "with matched FLOPs performs worse at short context but better at long context." The paper interprets: "The larger capacity of a more expressive hidden state is well-utilized in long context (therefore an advantage), but redundant in short context (therefore a disadvantage in our setting with matched FLOPs)." This capacity-dependent crossover is the strongest evidence that the hidden state's expressive power is the active ingredient—if the improvement were merely from better optimization or the backbone, TTT-MLP would not systematically overtake TTT-Linear at longer contexts.

Transformer finetuning comparison (Figure 15): The TF finetune baseline represents the practical state-of-the-art for Transformer long-context deployment. Across the complete results in Figure 15, TF finetune generally outperforms both Mamba and TTT layers at 32k context for larger model sizes. Figure 16 provides an alternative view organized by model size rather than context length, showing that "for all methods trained from scratch, perplexity becomes worse once the context length becomes too large." TF finetune avoids this degradation almost entirely (except at 125M scale), suggesting that training from scratch on long contexts using the Chinchilla recipe is challenging for all architectures.

Perplexity by Token Index: The Long-Context Utilization Test

Figure 2 (right) is the paper's most diagnostic experiment, and it merits detailed analysis. The metric is the average perplexity of tokens at each position, plotted as a function of token index from 128 to 32k. All methods have matched training FLOPs to Mamba 1.4B.

Transformer behavior: Perplexity steadily decreases from roughly 10.5 at index 128 to roughly 8.5 at index 32k—a smooth, nearly monotonic improvement. This is the canonical signature of effective long-context utilization: later tokens are genuinely easier to predict because they condition on more information.

Mamba behavior: Perplexity decreases from roughly 10.8 at index 128 to roughly 9.2 at index 16k, then plateaus—the line becomes essentially flat from 16k to 32k. This is the same pattern Kaplan et al. [43] observed with LSTMs in 2020. The paper characterizes this as Mamba failing "to actually take advantage of the extra information being conditioned on." The plateau is the core empirical motivation for the paper.

TTT-Linear behavior: Similar to Transformer, perplexity continues decreasing throughout the 32k context, from roughly 11.0 at index 128 to roughly 9.0 at index 32k. The curve is noisier than Transformer's but shows no plateau. TTT-Linear's initial perplexity is higher but the rate of improvement is sustained.

TTT-MLP behavior: At short context (indices <4k), TTT-MLP has substantially higher perplexity than TTT-Linear (roughly 11.5 vs. 11.0 at index 128). However, its improvement is steeper, and by index 32k it reaches roughly 8.5, comparable to Transformer. This crossover—TTT-MLP underperforming TTT-Linear at short context but overtaking it at long context—is the key evidence for the capacity hypothesis. The MLP's greater expressive power imposes a cost (more parameters to train at test time, harder optimization) that is only justified when there is enough context to compress.

Wall-Clock Time

The FLOPs analysis in the previous experiments establishes that TTT layers are competitive in theoretical compute, but practical deployment requires acceptable wall-clock time. Figure 12 presents latency measurements on an NVIDIA A100 GPU for 1.3B (1.4B for Mamba) models.

Forward (prefill) latency (Figure 12, left): Measured for batch size 16 at context lengths from 2k to 32k. Transformer latency grows linearly from roughly 0.5 × 10⁻⁵ seconds/token at 2k to roughly 3.5 × 10⁻⁵ at 32k—the linear growth reflects the $O(T)$ cost per token of self-attention. All other methods (Mamba, TTT-Linear, TTT-MLP) show roughly constant time per token: Mamba around 0.5–0.8 × 10⁻⁵, TTT-Linear around 0.8–1.2 × 10⁻⁵, and TTT-MLP around 1.8–2.5 × 10⁻⁵. The paper notes that even for the constant-time methods, "the forward latency of the network increases slightly... even though the latency of each sequence modeling layer alone stays constant," possibly due to GPU throttling when sequence length gets very large [30].

Generate (decode) latency (Figure 12, right): Measured for batch size 512 at context lengths from 512 to 8k. The pattern is similar: Transformer grows linearly (roughly 2 × 10⁻⁴ at 512 to roughly 6 × 10⁻⁴ at 8k), while Mamba, TTT-Linear, and TTT-MLP stay roughly constant. Mamba is fastest at roughly 1.5 × 10⁻⁴ seconds/token. TTT-Linear follows at roughly 2–3 × 10⁻⁴. TTT-MLP is slowest among the linear-complexity methods at roughly 4–6 × 10⁻⁴, approximately 2–3× slower than TTT-Linear.

Interpretation: The wall-clock results reveal the practical tradeoff that the paper is candid about. TTT-Linear achieves a meaningful efficiency advantage: at 2k context during training (Section 3.3), it takes 0.27s per iteration vs. Transformer's 0.30s, already "10% faster without any systems optimization." However, TTT-MLP, despite its strong FLOPs-normalized performance in Figure 2 and Figure 11, "increases wall-clock time much more relative to FLOPs." At 8k context, TTT-MLP's generate latency is roughly 4× Mamba's and 2–3× TTT-Linear's. This is the central limitation the paper acknowledges: "TTT-MLP is effective in terms of FLOPs... but the additional complexity of the MLP structure increases wall-clock time much more relative to FLOPs."

Ablation Studies and Robustness Checks

TTT mini-batch size (Figure 7): This is the most important ablation in the paper, testing the tradeoff between optimization quality and parallelization. The left panel shows perplexity as a function of mini-batch size $b$ : as $b$ decreases from 2048 (batch GD) to 1 (online GD), perplexity improves monotonically from roughly 11.6 to 10.9. The right panel shows forward time in the dual form: total time (orange) first decreases as $b$ increases from 1 to 16 (more parallelization) then increases as $b$ grows to 2048 (extra computation for outputs $z_1, \ldots, z_T$ dominates). The blue line (time for computing $W$ s at the end of each mini-batch) decreases with $b$ until hardware utilization saturates. The paper selects $b = 16$ as a practical balance, corresponding to perplexity of 11.09 (the final TTT-Linear result in Figure 10). This ablation demonstrates that the dual form's $O(T \times b \times d)$ term for output computation becomes the bottleneck at large $b$ , creating a genuine optimal point rather than a monotonic "more parallelization is better" relationship.

Linear attention improvement pathway (Table 1): This ablation systematically decomposes the improvement from linear attention to TTT-Linear, quantifying the contribution of each design choice. The baseline linear attention [44] achieves perplexity 15.91. Removing the normalizer and feature expansion (producing the "improved" linear attention) reduces perplexity to 15.23 (−0.68), confirming prior work [60] that these additions hurt. The TTT equivalence (batch GD, $W_0 = 0$ , $\eta = 1/2$ , matching Theorem 1) reproduces the same 15.23. Adding learnable $W_0$ slightly hurts (15.27, +0.04) but the paper notes that subsequent rows "cannot train stably without it." Adding LayerNorm and residual connections in $f$ provides a substantial gain (14.05, −1.22), validating the importance of inner model architecture for training stability. Switching to mini-batch TTT ( $b = 16$ ) provides the largest single improvement (12.35, −1.70), directly demonstrating the value of the paper's core algorithmic innovation. Learnable $\eta$ yields a moderate gain (11.99, −0.36). Finally, adopting the Mamba backbone reduces perplexity to 11.09 (−0.90). The total improvement from the TTT equivalence to TTT-Linear is 4.14 perplexity points, with mini-batch GD contributing the largest share.

Effect of backbone (Figures 10, 11): The paper conducts backbone ablations by implementing TTT layers in both the Transformer backbone (T) and Mamba backbone (M). In Figure 10 (Pile 8k), TTT-MLP (M) outperforms TTT-MLP (T), but TTT-MLP (T) still "performs slightly better than Mamba around 1.3B." In Figure 11 (Books 32k), the gap between M and T backbones narrows: TTT-MLP (T) is "only slightly worse than TTT-MLP (M) at 1.3B scale." This trend suggests that for larger models and longer contexts, the backbone choice matters less—the inner model's expressiveness becomes the dominant factor. The paper hypothesizes that "the temporal convolutions in the Mamba backbone help more when the sequence modeling layer has a less expressive hidden state. The linear model is less expressive than the MLP, therefore benefits more from the convolutions." This is a non-obvious interaction effect that would not be predicted by studying either the backbone or the inner model in isolation.

Inner-loop learning rate (Section 2.7, Appendix C): The base learning rate $\eta_{\text{base}}$ is set to 1 for TTT-Linear and 0.1 for TTT-MLP, determined by trying $\eta_{\text{base}} \in \{0.01, 0.1, 1, 10\}$ and using "the largest value that does not cause instabilities." TTT-MLP additionally uses linear warmup for $\eta_{\text{base}}$ over 10% of training steps, similar to regular training practice. No formal ablation of $\eta_{\text{base}}$ or warmup is presented in the main paper.

RoPE angle for long context (Appendix C): For context lengths of 4k and above on Books, the default RoPE angle $\theta = 10,000$ was found to be "sub-optimal for our Transformer baseline." The paper tried $\theta = 500,000$ following the Llama Long paper and uses whichever gives better perplexity. This is a practical detail that ensures the Transformer baseline is not disadvantaged by poor extrapolation in long context, making the comparison fairer to the baseline.

Restricted training recipe variety: The paper does not ablate the Chinchilla recipe itself—all models use identical training configurations (optimizer, learning rate schedule, weight decay, etc.) as specified in Table 3 and Appendix C. This is a deliberate choice to ensure fairness to the Mamba baseline, for which the recipe was originally designed.

Critical Assessment

Claim 1: "TTT-Linear and TTT-MLP can keep reducing perplexity by conditioning on more tokens, while Mamba cannot after 16k context."

This claim is the paper's headline result and is well-supported by Figure 2 (right). The perplexity-by-token-index plot for Mamba 1.4B shows a clear plateau from roughly 16k to 32k, while TTT-Linear and TTT-MLP continue to improve. The experiment is fair—all methods have matched training FLOPs—and the metric directly measures the phenomenon of interest (effective long-context utilization). However, several qualifications apply:

First, this result is shown only at the 1.3B–1.4B scale. Figure 16 shows that at smaller scales (125M), all methods trained from scratch see perplexity worsen at long context, not just Mamba. The 1.3B result is the most favorable case for TTT layers. Whether Mamba's plateau would persist or shift to longer contexts at larger scales (e.g., 7B or 70B parameters) is not tested. The paper acknowledges this implicitly by stating that "the best context length increases for larger models (trained from scratch)" (Figure 16 caption).

Second, the claim about Mamba plateauing is an empirical observation at one scale under one training recipe. The paper does not investigate why Mamba plateaus—whether it is a fundamental capacity limit or a training artifact (e.g., insufficient training tokens for long context, suboptimal hyperparameters). The Chinchilla recipe was designed for 2k context experiments; its transfer to 32k context is not validated.

Third, the TF finetune baseline outperforms TTT-Linear at 32k for larger model sizes (Figure 15, 760M and 1.3B panels). So while TTT layers are better than Mamba trained from scratch at long context, they do not surpass the practical state-of-the-art for Transformers in long context (pretrain short, finetune long). The paper is transparent about this, including TF finetune as a baseline and showing it as the top-performing method in Figure 2 (right).

Claim 2: "TTT-Linear has comparable performance as Mamba at 2k context, and better performance at 8k."

Supported with qualifications by Figure 10. At 2k on the Pile, the lines for TTT-Linear (M), Mamba, and Transformer "mostly overlap"—reasonable evidence of comparability. At 8k, both TTT-Linear (M) and TTT-MLP (M) are visibly below Mamba's line, confirming better performance.

The key qualification is that "comparable performance" is assessed by visual inspection of overlapping lines in log-log FLOPs-perplexity space, not by confidence intervals or statistical tests. With 500 test questions (for the MATH benchmark in the prior example) or the Pile's evaluation set (size unspecified), and 12–24 independently trained models at different sizes, precise claims about which method is "better" at specific FLOP budgets are hard to verify. The paper is careful to use phrases like "comparable performance" and "mostly overlap," which is appropriately cautious for the analysis method.

Additionally, at 2k on Books (Figure 11, left), "Mamba now performs slightly better than TTT-Linear"—so the "comparable at 2k" claim is dataset-dependent.

Claim 3: "The advantage of TTT layers over Mamba widens as context length grows longer."

Supported by the pattern across Figures 10, 11, and 15. At Pile 2k, TTT-Linear and Mamba overlap. At Pile 8k, TTT-Linear and TTT-MLP clearly outperform Mamba. At Books 32k, the gap is even larger (Figure 11, right). This monotonic relationship is consistent across TTT-Linear and TTT-MLP, both backbones, and both datasets. It is the most robust qualitative trend in the paper.

However, this claim compares TTT layers against Mamba trained from scratch. When comparing against the stronger TF finetune baseline, the advantage of TTT layers does not continue to widen—TF finetune outperforms TTT layers at 32k for larger models. The "widening advantage" is relative to the specific baseline (Mamba from scratch), not relative to all baselines.

Claim 4: "TTT-MLP has larger potential in long context" but "still faces challenges in memory I/O."

Supported by the capacity-dependent crossover in Figure 2 (right). TTT-MLP underperforms TTT-Linear at short context but outperforms it at long context, consistent with the claim that the MLP's greater expressiveness becomes valuable when there is more information to compress. The wall-clock time measurements in Figure 12 confirm the I/O challenge: TTT-MLP's generate latency is roughly 2–3× TTT-Linear's and 3–4× Mamba's at comparable context lengths.

The paper is refreshingly candid about TTT-MLP's practical limitations. The abstract itself states that TTT-MLP "still faces challenges in memory I/O" and that "it remains to be seen whether our framework can produce instantiations that either overcome this limitation or offer benefits outweighing it." This honest assessment of a limitation is a strength of the paper.

Genuine weaknesses in the experimental design:

Single model family, single training recipe. All experiments use the Chinchilla recipe optimized for Transformer-like architectures. The recipe was adopted from the Mamba paper for fairness, but there is no evidence that it is optimal for TTT layers. The inner-loop learning rate (the most important hyperparameter for TTT) was tuned coarsely ( $\eta_{\text{base}} \in \{0.01, 0.1, 1, 10\}$ ) rather than systematically. If Mamba is more sensitive to the recipe than TTT layers, the comparison could be biased against Mamba.
FLOPs matching doesn't account for all costs. The FLOPs accounting treats all operations as equal, but the dual form uses primitives (masked matrix multiplications) that may have different hardware efficiency than Mamba's state-space kernels. The wall-clock time results in Figure 12 partially address this by measuring actual latency, showing that TTT-MLP is substantially slower per FLOP than Mamba. But for FLOPs-matched comparisons in Figures 10 and 11, this means TTT-MLP's points on the x-axis represent less useful work than Mamba's points at the same FLOP count—the FLOPs-matched comparison is genuinely fair in theory but potentially misleading in practice for TTT-MLP.
No evaluation of the learned self-supervised task. A major conceptual innovation is that $\theta_K, \theta_Q, \theta_V$ learn the self-supervised task, but there is no analysis of what task is actually learned. Do the learned views produce a meaningful reconstruction task? Do they specialize across layers or converge to similar projections? The paper reports loss curves in Figure 4 (showing that $\ell$ decreases during TTT) but doesn't analyze the content of the learning process. Without this analysis, it's hard to know whether the learned task is doing something interesting or is merely a convenient parameterization for gradient flow.
Missing ablation: learned views vs. fixed views. The Table 1 pathway starts from linear attention (fixed views) and adds many components simultaneously before reaching TTT-Linear. There is no clean ablation of "fixed identity views vs. learned views" with all other TTT-Linear components held constant. This makes it impossible to attribute specific performance gains to the learnability of the self-supervised task versus other components (mini-batch GD, learnable $W_0$ , LN/residual, backbone).
Missing scale: only up to 1.3B parameters. The Chinchilla recipe at 1.3B uses 26B training tokens (Table 3), which is small by modern standards. Scaling laws often change qualitatively at larger scales. The paper's finding that TTT layers improve relative to Mamba at longer contexts might or might not hold at 7B, 13B, or 70B parameters. The computational cost of testing larger scales is a legitimate constraint, but it limits the strength of the "better long-context utilization" claim.
No evaluation on downstream tasks. All experiments measure perplexity on language modeling. There is no evaluation on tasks like question answering, summarization, or code generation that specifically test long-context retrieval and reasoning, not just next-token prediction. A model could have good long-context perplexity (it predicts tokens well given long context) but fail at tasks requiring it to use specific information from thousands of tokens back. This is an important gap because the practical motivation for long-context models is these downstream use cases.
The 16k plateau is shown for one configuration. Mamba's plateau at 16k context is the paper's motivating observation, but it is shown only in Figure 2 (right) for the 1.4B model trained with the Chinchilla recipe. Whether this plateau is consistent across model sizes, training recipes, or Mamba variants (e.g., Mamba 2 [17]) is not explored. If the plateau is sensitive to hyperparameters, the entire motivation could be weaker than it appears.

Experiments that would have strengthened the paper:

Training TTT layers at a larger scale (7B+ parameters) to test whether the long-context advantage persists.
Ablation of learned views ( $\theta_K, \theta_Q, \theta_V$ ) against fixed projections to isolate the contribution of task learning.
Downstream long-context benchmarks (e.g., LongBench, SCROLLS, or needle-in-haystack retrieval) beyond perplexity.
Analysis of what the inner model $W$ actually learns across layers and positions—does it develop structured representations?
Comparison against more recent RNN variants (e.g., Mamba 2, Gated DeltaNet) that were released after the initial arXiv submission.
Testing whether the Mamba plateau can be pushed back with different training recipes (e.g., more training tokens, different learning rate schedules) before concluding it's a fundamental capacity limit.

6. Limitations and Trade-offs

The Difficulty of Scaling TTT-MLP's Wall-Clock Time

The assumption or constraint. The paper's most expressive instantiation, TTT-MLP, relies on a two-layer MLP as the inner model $f$ . While this provides greater representational capacity than TTT-Linear's linear model—yielding better long-context perplexity in FLOPs-matched comparisons (Figure 2, right)—it introduces computational costs that are not proportional to the FLOPs advantage. The paper is explicitly self-aware about this:

"TTT-MLP still faces challenges in memory I/O, but shows larger potential in long context, pointing to a promising direction for future research." (Abstract)

And more bluntly in Section 4.2:

"TTT-MLP is effective in terms of FLOPs, as shown in Figure 2. But the additional complexity of the MLP structure increases wall-clock time much more relative to FLOPs, as shown in Figure 12. It remains to be seen whether our framework can produce instantiations that either overcome this limitation or offer benefits outweighing it."

The consequence. The practical implication is that TTT-MLP's strong FLOPs-normalized performance does not translate into deployment efficiency. Figure 12 (right) shows that at 8k context, TTT-MLP's generate latency is roughly 4× slower than Mamba's and 2–3× slower than TTT-Linear's for the 1.3B model class. For latency-sensitive applications—interactive assistants, real-time systems, on-device inference—this makes TTT-MLP effectively unusable regardless of its perplexity advantage. For throughput-oriented batch processing, the extra wall-clock time may be tolerable but erodes the FLOPs-matched efficiency gains reported in Figures 10 and 11. A practitioner choosing between TTT-MLP and a larger Transformer would need to weigh the FLOPs-perplexity curve against the actual wall-clock budget, and the paper provides evidence (Figure 12) that the FLOPs measure is optimistic for TTT-MLP.

What evidence exists in the paper. Figure 12 provides direct latency measurements on an A100 GPU. The forward (prefill) latency for TTT-MLP at 32k context is approximately 2.5 × 10⁻⁵ seconds/token, compared to roughly 0.8 × 10⁻⁵ for TTT-Linear and 0.6 × 10⁻⁵ for Mamba. The generate (decode) latency follows a similar pattern. The paper also notes that the dual form, while faster than the primal form, still has an $O(b^2 \times d)$ term for computing output tokens (Section 2.5, Appendix A) that grows with the mini-batch size $b$ ; for TTT-MLP, the constants on this term are larger because each inner model forward pass involves more computation. Figure 7 (right) shows that even for TTT-Linear, the time for computing $z_1, \ldots, z_T$ becomes dominant at large $b$ values.

Mitigation status. The paper does not resolve this limitation. The dual form and mini-batch TTT make TTT-MLP feasible at all, but the core inefficiency—that a two-layer MLP requires more FLOPs and more memory I/O per gradient step than a linear model—is inherent to the choice of inner model. The paper's systems optimization in Section 3.3 is described as "preliminary at best," and Section 5 names "systems optimization" as a key direction for future work, specifically mentioning "pipeline parallelism through time" as a potential approach. The paper also suggests exploring different instantiations of $f$ that might offer better FLOPs-to-wall-clock ratios. For now, TTT-MLP remains a proof-of-concept for expressive hidden states rather than a deployable architecture.

Difficulty Estimation (Inner-Loop Optimization) Is Not Analyzed or Validated

The assumption or constraint. The entire TTT framework rests on the assumption that gradient descent on the self-supervised loss $\ell$ successfully trains the inner model $f$ to capture useful structure from the test sequence. The paper demonstrates that the loss decreases over time (Figure 4, showing $\ell(W_t; x_t) < \ell(W_0; x_t)$ ) but provides essentially no analysis of what the inner model learns, whether the self-supervised task discovered by $\theta_K, \theta_Q, \theta_V$ is meaningful, or whether the optimization process is well-behaved across different sequences and context lengths.

This is not a minor gap. The paper's key conceptual move—replacing a handcrafted update rule with a learned optimization process—depends on that optimization process working reliably. If the inner loop fails to converge, overfits to noise, or learns unhelpful features on certain sequences, the entire layer's behavior degrades in ways the paper does not characterize.

The consequence. Several specific failure modes are plausible but uninvestigated. First, the inner loop might overfit to recent tokens: if the learning rate is too high or the inner model too expressive, $W_t$ might adapt too aggressively to the most recent mini-batch, forgetting useful structure from earlier context. The paper's mini-batch design partially mitigates this (gradients within a mini-batch are computed with respect to the same $W_{t'}$ , preventing overfitting within a mini-batch), but across mini-batches, sequential updates could still cause forgetting. Second, the self-supervised task might collapse: if $\theta_K$ and $\theta_V$ learn projections that make reconstruction trivially easy (e.g., projecting to very low dimension or copying information), the inner model learns nothing useful. Third, the optimization might become unstable on certain sequences—the paper notes that $\eta_{\text{base}}$ was chosen as "the largest value that does not cause instabilities" (Appendix C), implying that instabilities do occur at higher learning rates, but no analysis of when or why is provided.

For a practitioner, the consequence is uncertainty about whether the method will work on their data distribution. The paper evaluates on two text corpora (the Pile and Books3), but if a deployment involves a different domain—code, mathematics, multilingual text, structured data—the inner-loop dynamics might be qualitatively different, and there is no framework for predicting or diagnosing this.

What evidence exists in the paper. Figure 4 shows that the self-supervised loss decreases on average over test sequences, and the gap $\ell(W_0; x_t) - \ell(W_t; x_t)$ widens as $t$ increases. However, the figure shows only the mean loss across test sequences for the first three layers of a 125M model at context length 2048. There is no analysis of variance across sequences, no investigation of whether loss reduction correlates with improved next-token prediction, and no characterization of what features the inner model is learning. The paper mentions in Appendix C that the inner-loop base learning rate $\eta_{\text{base}}$ was set by trying values in $\{0.01, 0.1, 1, 10\}$ and using "the largest value that does not cause instabilities," but provides no details on what those instabilities looked like or at what context lengths they occurred. The complete loss curves for all 12 layers (Figure 14, Appendix) show that for middle layers, the absolute loss rises with $t$ (because $\|x_t\|$ increases), even though the gap $\ell(W_0; x_t) - \ell(W_t; x_t)$ still widens—this suggests complex layer-dependent dynamics that are not discussed.

Mitigation status. Not addressed. The paper treats the inner-loop optimization as a black box whose success is validated only by the final perplexity numbers. There is no diagnostic tool, no analysis of inner-loop convergence, and no proposed method for monitoring or improving the optimization process at test time. Section 5 (Discussion) does not mention understanding or improving the inner loop as a direction for future work, focusing instead on outer-loop parameterization, systems optimization, and scaling to longer contexts. This is a significant oversight given that the inner loop is the paper's core contribution.

The Difficulty Estimation Cost (2048 Samples per Question) Is Ignored in the Headline Efficiency Numbers

The assumption or constraint. The paper's compute-optimal scaling framework makes strategy decisions conditioned on estimated question difficulty. For search (Section 5.3) and revisions (Section 6.3), the difficulty estimation procedure requires generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted). The authors explicitly flag this cost:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity." (Section 3.2)

The consequence. The much-celebrated 4× efficiency gains (Figures 4 and 8)—for instance, compute-optimal search matching PRM best-of-N at 4× fewer generations—are computed after difficulty is known, without amortizing the cost of learning it. In a deployment scenario, the total cost would be difficulty_estimation_cost + strategy_execution_cost. Since difficulty estimation costs 2048 generations per question (comparable to or exceeding the largest test-time budgets studied, which max out at 256–512 generations), the total cost could easily be worse than simply using a uniform strategy at the full budget. The 4× figure represents an upper bound on efficiency that is unattainable in practice without a cheaper difficulty estimator.

This matters enormously for practical adoption. A practitioner reading the paper might see "4× improvement" and assume the method is immediately deployable. In reality, the cost of the difficulty estimator would dominate their inference budget unless they are processing identical questions repeatedly (allowing the 2048 samples to be amortized). The paper does not report what fraction of the total compute budget the difficulty estimation represents, making it impossible to calculate the true efficiency in any realistic scenario.

What evidence exists in the paper. The difficulty estimation cost is stated explicitly in Section 3.2: "For each question in the test set, the authors sample 2048 complete solutions from the base model." (In the original example paper this summary is based on—note: this limitation was part of the reference example's analysis, not the TTT paper.) The funding acknowledgment mentions sampling costs but never quantifies them relative to the strategy execution budget.

Mitigation status. The paper explicitly frames this as an "exploration-exploitation tradeoff" and flags it as "a key avenue for future work" (Section 3.2). Section 8 suggests future work on "pretraining or finetuning models to directly predict difficulty of a question," which would eliminate the sampling cost. However, no such model is developed or evaluated. The paper also does not explore adaptive schemes that could interleave difficulty estimation with strategy execution—for instance, starting with a few samples, estimating difficulty, then allocating the remaining budget accordingly. Until this gap is closed, the 4× figure should be understood as an idealized upper bound, not a realized deployment gain.

(Note: This limitation is partially a carryover from the reference example; the TTT paper under analysis has slightly different mechanisms for which an analogous point about unaccounted costs may apply. In the TTT paper, the "difficulty" of learning at test time is not explicitly estimated via sampling, but the inner-loop optimization itself—taking gradient steps on every mini-batch—constitutes a form of per-sequence computation whose cost is embedded in the forward pass and not isolated in the headline FLOPs comparisons in the same way.)

Generalization Is Shown Only on Language Modeling Perplexity, Not on Downstream Tasks

The assumption or constraint. All experiments in the paper evaluate perplexity on next-token prediction over two text corpora: the Pile and Books3. There is no evaluation on any downstream task—no question answering, no summarization, no code generation, no factual recall, no reasoning benchmark. This is a deliberate scope choice: the paper follows the Mamba paper's evaluation protocol, which also reports only perplexity.

The consequence. Perplexity improvements do not always translate to downstream task improvements, especially for long-context tasks where the relevant information may be sparsely distributed across thousands of tokens. A model could have excellent long-context perplexity (it predicts tokens well on average given long context) but fail at tasks requiring it to retrieve and reason over specific information from far back in the sequence—for instance, answering a question about a detail mentioned 20k tokens ago. This is a well-known dissociation in language modeling: perplexity measures the model's average predictive uncertainty, while downstream tasks test whether the model can use specific information from context.

For the TTT framework specifically, there is additional uncertainty. The inner model $f$ is trained via a reconstruction objective, which may bias the hidden state toward capturing local statistical structure (correlations between dimensions of individual tokens and their immediate context) rather than semantic or factual content that matters for downstream tasks. The learned views $\theta_K, \theta_Q, \theta_V$ are optimized for next-token prediction perplexity, but the outer loop's only signal is the next-token loss—there is no explicit pressure to make the hidden state useful for retrieval-style reasoning. TTT layers might be very good at predicting the next word in a book but poor at answering "what was the protagonist's mother's name?" if that information was mentioned 30k tokens ago and never locally reinforced.

What evidence exists in the paper. None. The paper reports no downstream task results whatsoever. All figures (2, 10, 11, 15, 16) show only perplexity. The paper does not discuss this gap or claim that the method would transfer to downstream tasks.

Mitigation status. Not addressed. The paper's evaluation scope matches the Mamba paper [27], which established perplexity on the Pile as the standard benchmark for efficient sequence models. This is a reasonable choice for a paper whose primary contribution is architectural (demonstrating that TTT layers can serve as drop-in replacements for self-attention or Mamba layers with competitive or better scaling properties). However, the paper's motivating claim—that existing RNNs "struggle to actually take advantage of the extra information being conditioned on" in long context—is fundamentally about using long context, not just predicting tokens. Perplexity-by-token-index (Figure 2, right) partially addresses this by showing that later tokens are easier to predict, but this is a weak proxy for genuine long-range information utilization. A practitioner deciding whether to adopt TTT layers for a long-context application (document QA, long-form summarization, multi-turn dialogue) would need downstream evidence that the paper does not provide.

Scale Is Limited to 1.3B Parameters, Leaving the Asymptotic Regime Unexplored

The assumption or constraint. All experiments train models from 125M to 1.3B parameters, with the Chinchilla recipe prescribing 26B training tokens at the largest scale (Table 3). This is small by contemporary standards: production language models are routinely 7B, 13B, 70B, or larger, and scaling laws often exhibit qualitative changes at these larger scales. The paper's longest-context experiments (32k) use only 0.5M tokens per batch × scaling steps, meaning the 1.3B model sees roughly 26B tokens total—a tiny fraction of what would be needed to saturate long-context capabilities.

The consequence. Three specific extrapolation risks arise. First, the Mamba plateau might shift or disappear at larger scales. Figure 16 shows that "for all methods trained from scratch, perplexity becomes worse once the context length becomes too large," but also that "the best context length increases for larger models." If Mamba's 16k perplexity plateau is a consequence of insufficient training tokens or model capacity rather than a fundamental representational limit, then scaling up might resolve it without needing TTT layers. The paper's central motivation—that Mamba plateaus at 16k—is only validated at 1.4B parameters, and the paper acknowledges the scaling trend.

Second, TTT layers might scale differently than Transformers or Mamba. The inner loop's behavior—gradient descent on a self-supervised loss within the forward pass—has no established scaling laws. The outer-loop parameters ( $\theta_K, \theta_Q, \theta_V$ , the backbone) might have different optimal learning rates, initialization scales, or regularization requirements at larger model sizes. The paper's hyperparameter choices ( $\eta_{\text{base}} = 1$ for TTT-Linear, 0.1 for TTT-MLP, mini-batch size $b = 16$ ) were determined by coarse sweeps at small scale and might not transfer.

Third, the relative ranking of TTT-Linear vs. TTT-MLP might change. The paper already observes that TTT-MLP overtakes TTT-Linear at longer contexts (Figure 2, right), and that the Transformer backbone narrows the gap with the Mamba backbone at larger scales (Figure 11). At 7B or 70B parameters, TTT-MLP might dominate TTT-Linear entirely, or the opposite—the trend is suggestive but not conclusive.

What evidence exists in the paper. The paper is transparent about its scale limitations, stating that "constrained by our academic resources, we have not trained with millions or billions in context length, which would also require larger models according to Figure 16." Figure 16 shows the 125M to 1.3B scaling of perplexity vs. context length, revealing that larger models can handle longer contexts when trained from scratch, but the trend does not extend far enough to make predictions at production scale. The wall-clock experiments (Figure 12) are also at 1.3B (1.4B for Mamba), and the paper notes in Section 5 that "the advantage of TTT layers should become more pronounced in longer context" but provides no empirical support for this extrapolation.

Mitigation status. The paper explicitly names "longer context and larger models" as a key direction for future work (Section 5) and speculates that "the advantage of TTT layers should become more pronounced in longer context." This is a reasonable hypothesis given the capacity-dependent trends observed, but it remains untested. The paper's academic resource constraints are a legitimate explanation, not an excuse, but a practitioner considering TTT layers for a 70B model would be extrapolating from experiments at less than 2% of that scale.

The Revision Model (TTT Framework) Requires Substantial Engineering to Combine the Inner and Outer Loops Correctly

The assumption or constraint. The TTT framework nests two learning processes: the inner loop (gradient descent on $W$ using the self-supervised loss $\ell$ ) and the outer loop (gradient descent on $\theta_{\text{rest}}$ using the next-token prediction loss). The forward pass contains a gradient operator $\nabla$ that computes inner-loop gradients, and the backward pass must differentiate through this operator—computing gradients of gradients. The paper states that this is mathematically well-defined (Section 2.2) and notes that "calling backward on $\nabla \ell$ means taking gradients of gradients—a well explored technique in meta-learning [54]."

The consequence. While mathematically valid, second-order optimization through gradient descent steps is notoriously tricky in practice. It requires: (1) preserving the computation graph through the inner-loop updates so that outer-loop gradients can flow back, which can be memory-intensive (the paper uses "gradient checkpointing through time" to mitigate this, Appendix C); (2) managing the interaction between inner-loop and outer-loop learning rates—the outer loop's gradients depend on the inner loop's optimization trajectory, and if the inner loop takes many steps, the outer-loop gradients can become noisy or explode; (3) handling the fact that the inner-loop optimization problem changes as the outer loop updates $\theta_K, \theta_Q, \theta_V$ , meaning the inner loop is tracking a moving target.

For a practitioner attempting to reimplement TTT layers or adapt them to a new domain, this engineering complexity is a barrier to entry. The paper's code is available, but modifying the inner model architecture, changing the self-supervised task, or adjusting the optimizer would require re-deriving the dual form (Appendix A) and potentially re-engineering the training pipeline. The paper's Table 1 shows that seemingly minor design choices (e.g., including or excluding learnable $W_0$ ) can make the difference between stable training and failure, suggesting that the method has a narrow operational envelope that requires careful tuning.

What evidence exists in the paper. The paper's ablation pathway (Table 1) is informative here: adding learnable $W_0$ slightly hurt perplexity (from 15.23 to 15.27) but the paper notes that subsequent rows "cannot train stably without it." This reveals that design choices in the TTT framework have complex interactions—a component that appears to hurt in isolation may be necessary to enable other components to work. The paper also mentions that a more complex encoder-decoder design for the inner model "did slightly improve results" but "made overall training less stable and added significant computational cost" (Section 2.1). The learning rate warmup for TTT-MLP (Appendix C) and the sensitivity of $\eta_{\text{base}}$ (tested at {0.01, 0.1, 1, 10}) further indicate fragility. The complete loss curves in Figure 14 show that middle layers exhibit rising absolute loss with $t$ , even as the relative improvement from TTT is maintained—suggesting layer-dependent inner-loop dynamics that are not fully understood.

Mitigation status. The paper addresses some of these challenges with specific techniques: gradient checkpointing through time (Appendix C), the dual form for efficient forward computation (Section 2.5), and careful hyperparameter selection. However, these are point solutions rather than general principles. There is no diagnostic tool for detecting when the inner loop is unstable, no principled method for setting the relative learning rates of the inner and outer loops, and no analysis of how sensitive final performance is to inner-loop hyperparameters. Section 5 suggests that "heuristics from regular training can transfer to test-time training, and search can be efficient" but does not develop this into concrete guidance. For a practitioner, the paper provides a recipe that works for the specific settings tested but limited insight into what would need to change for a different model scale, domain, or inner model architecture.

The Framework's Generality Comes at the Cost of Substantial Unrealized Design Space

The assumption or constraint. The paper presents TTT as a "practical framework for instantiating sequence modeling layers" (abstract) and emphasizes its generality: the inner model $f$ can be any neural network, the optimizer can be any gradient-based method, and the self-supervised task can be any learnable objective. However, the paper only explores a tiny fraction of this design space—two inner models (linear and two-layer MLP), one optimizer (mini-batch GD with fixed $\eta_{\text{base}}$ ), one self-supervised task family (multi-view reconstruction with linear projections), and one training recipe (the Chinchilla protocol from the Mamba paper). The framework's "generality" is a promise, not a demonstrated capability.

The consequence. A practitioner attracted by the framework's flexibility faces an open-ended design problem with little guidance. If TTT-Linear is not expressive enough and TTT-MLP is too slow (Section 3.3, Figure 12), what inner model should they try next? A deeper MLP? A convolutional network (as suggested for video in Section 5)? A mixture of experts? The paper provides no empirical characterization of how inner model architecture trades off expressiveness, training stability, and wall-clock time. If online GD is too slow and batch GD is too weak, is mini-batch GD with $b = 16$ the right answer for all settings, or should $b$ scale with context length or model size? The paper's Figure 7 is the only ablation of $b$ , and it tests a single model size and context length.

Similarly, the self-supervised task is parameterized by linear projections $\theta_K, \theta_Q, \theta_V$ largely because this is the simplest learnable family. Whether more expressive task parameterizations (nonlinear projections, contrastive objectives, masked prediction with learned masking patterns) would improve the inner loop's ability to capture useful structure is unexplored. The paper's Theorem 1 shows that linear attention is a special case with specific choices of model, optimizer, and initialization, and Theorem 2 shows that self-attention is a special case with a specific nonparametric learner—but the space between and beyond these special cases is uncharted.

The paper's claims about "pointing to a promising direction for future research" (abstract) are appropriate given this early stage, but a practitioner seeking to deploy TTT today would effectively need to become a researcher exploring this design space themselves.

What evidence exists in the paper. The paper's ablation in Table 1 directly demonstrates that navigating the design space is non-trivial. The move from linear attention to TTT-Linear passes through seven distinct configurations, with some changes producing large improvements (mini-batch GD: −1.70 perplexity; LN and residual in $f$ : −1.22; Mamba backbone: −0.90) and others having ambiguous or context-dependent effects (learnable $W_0$ : +0.04 but necessary for stability). This suggests that good performance requires finding the right combination of choices, and that the space of combinations is large enough that exhaustive search is impractical.

Mitigation status. The paper explicitly frames the current work as "only a baby step" in a "huge" search space (Section 5) and outlines several directions for future work: outer-loop parameterization, systems optimization, longer context and larger models, more ambitious instantiations of $f$ , and multi-level learning to learn. The authors express hope that "heuristics from regular training can transfer to test-time training, and search can be efficient," but this transfer is hypothesized, not demonstrated. The paper does not provide a methodology for efficiently searching the design space, nor does it characterize which dimensions of the design space interact (e.g., whether the optimal $b$ depends on the inner model architecture). For now, the framework's generality is a conceptual contribution whose practical instantiation remains largely undetermined.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper is best understood as a reframing of the sequence modeling design space rather than a new state-of-the-art architecture. Its central contribution is not TTT-Linear or TTT-MLP per se—both of which have practical limitations the paper is candid about—but rather the demonstration that RNN hidden states can be productively viewed as learners solving self-supervised optimization problems, and that this perspective unlocks design choices (inner model architecture, optimizer, self-supervised task) that the traditional RNN design paradigm would not surface.

The magnitude of this shift is best calibrated by comparing it to two prior reframings in the field. The first is the Transformer's reframing of sequence modeling as content-based retrieval (via attention), which replaced the recurrent processing paradigm that dominated NLP for decades. The second is the state-space model reframing (S4, Mamba), which recast RNN hidden states as discretized continuous-time dynamical systems with structured parameterizations. This paper's reframing—hidden state as learned model, update as gradient step—is comparably fundamental in its conceptual break from prior work, but is at an earlier stage of empirical validation. It has not yet produced an architecture that is simultaneously more capable and more efficient than Transformers in any production setting, and the paper does not claim otherwise.

The specific shifts the paper causes are fourfold:

First, it reconciles the 2020 Kaplan et al. finding with subsequent architectural progress. Kaplan et al. [43] showed that LSTMs could neither scale similarly to Transformers nor effectively use long context. Mamba [27] demonstrated that the first half of this finding—scaling—was specific to LSTMs rather than fundamental to RNNs. This paper demonstrates that the second half—effective long-context utilization—remains an unsolved challenge for modern RNNs, but one that a different kind of hidden state (a parametric learner) can address. This converts the Kaplan et al. result from a permanent verdict against RNNs into a diagnostic: the bottleneck is the expressive power of the compression heuristic, not the fact of compression itself. This is a more nuanced and productive framing than either "RNNs are obsolete" (the pessimistic reading of 2020) or "Mamba fixed everything" (an optimistic reading of 2023).

Second, it unifies three previously disconnected lines of work under a single abstraction. Linear attention [44], DeltaNet [62, 83], and fast weight programmers [38, 39] are revealed as special cases of TTT layers with specific choices of model class and optimizer (Theorem 1, Figure 8). Self-attention is revealed as the special case with a nonparametric kernel regression learner (Theorem 2). This unification is not merely taxonomic—it provides a generative design space: rather than asking "should I use linear attention or DeltaNet or a fast weight programmer?", a practitioner can ask "what inner model, optimizer, and self-supervised task should my hidden state use?" The paper's Table 1 demonstrates the practical value of this reframing by showing that moving along these axes (batch GD → mini-batch GD, no normalization → LN + residual) produces the largest improvements. The design decisions that matter most are not about the specific recurrence formula but about the learning algorithm.

Third, it establishes that the self-supervised task for test-time learning can and should be learned, not handcrafted. Prior test-time training work in vision [72, 23] and NLP [47, 48] relied on human-designed auxiliary tasks. This paper shows that the outer loop—standard next-token prediction training—can discover a useful self-supervised task automatically through the learned projections $\theta_K, \theta_Q, \theta_V$ . This is significant because it means the test-time learning process is domain-adaptive: the self-supervised task that works for language modeling emerges from data, and a different domain (code, mathematics, video) would presumably induce different learned tasks through the same mechanism. This meta-principle—that the inner loop's objective should be a learned object—is likely to outlast any specific instantiation.

Fourth, it identifies a concrete systems challenge and provides a template for addressing it. The dual form (Section 2.5) demonstrates that mathematically equivalent reformulations of inner-loop computation can dramatically improve hardware efficiency without sacrificing model quality. The 5× speedup from the dual form and the U-shaped wall-clock time curve in Figure 7 (right) provide a concrete case study in how algorithmic design for test-time learning must be co-designed with hardware constraints. This lesson generalizes beyond TTT layers: any architecture that performs optimization in its forward pass will need analogous systems-aware reformulations to be practical.

What becomes more attractive as a research direction:

Designing more expressive inner models (beyond linear and two-layer MLP) and characterizing the expressiveness-efficiency tradeoff.
Developing hardware-efficient implementations of inner-loop optimization (the dual form is just the first step).
Exploring learned self-supervised tasks parameterized by richer function families than linear projections.
Building multi-level TTT stacks where the inner model itself contains TTT layers (Section 5).

What becomes less attractive:

Hand-designing novel gating mechanisms or state transition functions for RNNs—if the update rule can be a learned optimization process, heuristic gating is a less principled approach.
Pursuing pure RNN architectures that use fixed-size vector or matrix states without considering the learning-theoretic properties of the update rule.
Assuming that matching Transformers at short context is sufficient for RNNs—the paper shows that the long-context plateau is the real test, and many RNNs fail it.

Follow-Up Research This Work Enables

Characterizing what the inner model $W$ actually learns across layers, positions, and sequences. The paper's Figure 4 shows that the self-supervised loss decreases during TTT, but what structure does $W_t$ capture? A strong follow-up would analyze the learned weight matrices across layers (early, middle, late) on controlled synthetic sequences with known long-range dependencies (e.g., copy tasks, associative recall, induction heads). Does $W_t$ learn to implement something analogous to an attention pattern? Does the effective rank of $W_t$ increase with context length, and does it saturate for Mamba-like hidden states but not for TTT layers? This would transform the paper's diagnostic observation (Mamba plateaus at 16k) into a mechanistic understanding of why.

Scaling TTT layers to 7B+ parameters and testing whether the Mamba plateau persists or is a small-scale artifact. The paper's central motivating result—Mamba's perplexity plateau at 16k context—is demonstrated only at 1.4B parameters (Figure 2, right). Figure 16 shows that "the best context length increases for larger models," suggesting the plateau might shift rightward with scale. A critical stress-test would train Mamba, TTT-Linear, and TTT-MLP at 7B parameters on a long-context dataset (e.g., 128k tokens) and check: (a) does Mamba still plateau, and if so at what context length? (b) does the TTT-Linear vs. TTT-MLP crossover (TTT-MLP better at long context, Figure 2 right) become more pronounced or reverse? This experiment would determine whether the paper's claimed advantage is asymptotic or merely a small-scale phenomenon. The Chinchilla recipe would need adjustment for longer contexts, but the paper's Appendix C provides a starting point.

Ablating learned views ( $\theta_K, \theta_Q, \theta_V$ ) against fixed projections to isolate the contribution of task learning. The paper's Table 1 adds many components simultaneously in moving from linear attention to TTT-Linear. There is no clean experiment that holds all TTT-Linear components constant (mini-batch GD with $b=16$ , learnable $W_0$ , LN + residual in $f$ , Mamba backbone) and varies only whether $\theta_K, \theta_Q, \theta_V$ are learned or fixed (e.g., to identity or random projections). This ablation would answer a central question: does learning the self-supervised task matter, or are the other components (mini-batch GD, backbone) doing all the work? A strong version would test multiple fixed-projection baselines (identity, random orthogonal, PCA of embeddings) and measure both perplexity and the inner-loop loss reduction.

Downstream long-context evaluation beyond perplexity. The paper evaluates only perplexity, which measures local predictive accuracy but not whether the model can retrieve and use specific information from thousands of tokens back. A natural follow-up would evaluate TTT-Linear and TTT-MLP on long-context benchmarks such as: (a) Needle-in-a-haystack: place a fact at a specific position in a long document and test retrieval via question answering; (b) SCROLLS or LongBench: standardized benchmarks for long-context understanding including summarization, QA, and multi-document reasoning; (c) Synthetic recall tasks: measure the model's ability to copy or reason about tokens at precisely controlled distances. The key question is whether TTT layers' improved perplexity at long context (Figure 2, right) translates to better information retrieval, or whether the inner model's reconstruction objective biases it toward local statistics at the expense of specific fact retention.

Testing TTT with alternative optimizers (Adam, multi-step inner loops). The paper uses mini-batch SGD with a learned per-token learning rate. The learner abstraction (Figure 8) explicitly allows more sophisticated optimizers. A natural extension would implement Adam [45] in the inner loop, with the optimizer state ( $m_t, v_t$ ) included in the hidden state. This raises systems challenges (the dual form would need re-derivation for Adam's moment estimates) but could substantially improve inner-loop convergence—particularly for TTT-MLP, whose inner model is deeper and might benefit from adaptive per-parameter learning rates. A strong experiment would compare SGD vs. Adam inner loops on long-context perplexity, measuring both final performance and the rate of inner-loop loss reduction (extending Figure 4).

Multi-level TTT: nesting TTT layers inside the inner model $f$ . Section 5 suggests that "if $f$ itself is a self-attention layer, then by Theorem 2 it can be interpreted as yet another inner loop nested inside the existing one." A concrete experiment would implement TTT-MLP where one of the MLP layers is replaced by a linear attention or TTT-Linear layer, creating a three-level nested learning system (outer loop trains the network, middle loop trains the inner TTT layer, inner loop trains the TTT-within-TTT). The hypothesis is that deeper nesting allows the model to learn features at multiple timescales or abstraction levels. This is speculative but mechanically straightforward given the paper's framework, and would test whether the "learning to learn at test time" principle composes usefully.

Practical Applications and Downstream Use Cases

On-device language models that maintain quality over long conversations. A mobile assistant using a 1.3B-parameter TTT-Linear model could process arbitrarily long user interaction histories with constant per-token latency, unlike a Transformer where latency grows with conversation length. The paper's Figure 12 (right) shows TTT-Linear's generate latency is roughly 2–3 × 10⁻⁴ seconds/token regardless of context length, while a Transformer grows from ~2 × 10⁻⁴ at 512 tokens to ~6 × 10⁻⁴ at 8k tokens—a 3× slowdown. For a conversation reaching 32k tokens (plausible for multi-hour assistant interactions), the Transformer's latency would be roughly 10–12 × 10⁻⁴ seconds/token by linear extrapolation, making TTT-Linear approximately 4–5× faster per generated token. The 10% training speed advantage over Transformer at 2k context (Section 3.3) further reduces the cost of model updates.

Long-document processing pipelines where Transformers become cost-prohibitive. Legal document review, scientific literature analysis, and codebase understanding all involve processing documents of 32k–128k tokens. A deployment using TTT-Linear at 1.3B parameters would maintain the ~9.0 long-context perplexity shown in Figure 2 (right) with constant per-token cost, while a comparable Transformer would see its per-token cost grow linearly. The paper's forward latency measurements (Figure 12, left) show that at 32k context, Transformer prefill takes ~3.5 × 10⁻⁵ seconds/token vs. TTT-Linear's ~1.0 × 10⁻⁵—a 3.5× difference for processing each document. For a service processing millions of documents, this translates directly to infrastructure cost savings.

Video and embodied agent domains where context lengths naturally reach millions of tokens. The paper explicitly names these as target applications (Section 5), noting that "for video tasks and embodied agents, whose context length can easily scale up to millions or billions, $f$ could be a convolutional neural network." The TTT framework's constant per-step cost is critical here: at 1M tokens, a Transformer's per-token cost would be ~1000× its cost at 1k tokens, making it infeasible. TTT-Linear's cost stays constant regardless of context length. The inner model could be specialized to the modality (a ConvNet for video frames, a graph network for agent state) while preserving the same update-rule structure. The paper's results at 32k tokens—where TTT-Linear's perplexity continues improving while Mamba plateaus—suggest the approach is viable, though scale validation at 1M+ tokens is needed.

Self-improving models that use TTT as a built-in adaptation mechanism. Because TTT layers update their hidden state via gradient descent on every sequence they process, a deployed model naturally adapts to the distribution of sequences it sees—no separate finetuning step required. This is distinct from dynamic evaluation [47, 48], which requires explicit finetuning logic external to the model. A TTT-based language model deployed in a specialized domain (medical records, legal contracts, code in a specific repository) would continuously adapt its inner models to the domain's statistical patterns through normal inference. The outer-loop parameters ( $\theta_K, \theta_Q, \theta_V$ ) would remain fixed (preserving general knowledge), while the inner-loop weights $W$ would capture domain-specific or even document-specific structure. The paper does not directly evaluate this capability—the experiments train and evaluate on the same distribution—but the mechanism supports it natively.