End-to-End Test-Time Training for Long Context

ArXiv: 2512.23675

🎯 Pitch

Even Mamba 2 and Gated DeltaNet get worse as context length grows, but this method—which simply continues training the model on the current context—actually scales better with length than full attention, while being 2.7x faster at 128K tokens. The trick is making the test-time learning objective match the original training objective exactly, unlike prior attempts.

---

1. Executive Summary

This paper introduces TTT-E2E, a method that reformulates long-context language modeling as a continual learning problem rather than an architectural design problem, using only a standard Transformer with sliding-window attention but continuing to train the model at test time via next-token prediction on the given context—compressing the context into its weights. The approach is end-to-end in two senses: the inner loop directly optimizes the standard next-token prediction loss (unlike prior long-context TTT methods that use layer-wise key-value binding), and the outer loop meta-learns the model's initialization for test-time training via gradients of gradients (unlike dynamic evaluation, which optimizes for static out-of-the-box performance). Experiments on DCLM and Books with models up to 3B parameters and 164B training tokens demonstrate that TTT-E2E scales with context length in the same way as Transformer with full attention—maintaining a consistent accuracy advantage across 8K to 128K tokens—while others, such as Mamba 2 and Gated DeltaNet, degrade at longer contexts, and it achieves 2.7× faster prefill latency than full attention at 128K context on an H100, establishing that constant-cost inference can match full-attention scaling only when the test-time training objective and the training-time meta-learning objective are aligned end-to-end.

2. Context and Motivation

The Core Problem: Scaling Context Length Without Scaling Cost

The fundamental question this paper tackles is deceptively simple: how can we build a language model that effectively uses very long contexts (128K+ tokens) while maintaining a constant cost per token? This matters because Transformer models with full self-attention—the dominant architecture since Vaswani et al. (2017)—scale poorly in two critical dimensions:

• Compute cost grows linearly with context length. For each new token, full attention must scan through the keys and values of all previous tokens, giving prefill complexity of $O(T^2)$ and decode complexity of $O(T)$ for context length $T$. The paper gives concrete numbers: at 128K context, full attention takes 2.7× longer than constant-cost alternatives on an H100 (Figure 1, right panel).
• Memory for the KV cache also grows linearly. Storing explicit key-value pairs for every token becomes prohibitive at the million-token scales that modern applications demand.

This gap is practically urgent for several reasons that the paper directly engages with:

• Long-document understanding. Legal contracts, scientific papers, and codebases routinely exceed 32K tokens. Models that degrade on longer contexts cannot reliably extract information from these documents.
• Multi-turn dialogue and agent loops. Conversational agents accumulate context over many turns. If the model's effectiveness decays as the conversation grows longer, the user experience degrades.
• Long-chain reasoning. Chain-of-thought and reasoning traces can extend to thousands of tokens. If additional reasoning steps stop improving (or worsen) performance beyond some context threshold, the benefit of compute-time reasoning is capped.
• Streaming and real-time applications. Video, audio, and sensor streams produce unbounded context. A model whose latency grows with the stream length is fundamentally unsuitable for real-time deployment.

The paper's framing in Section 1 is clear about the stakes: self-attention over the full context "readily attends to every detail, but its cost per token grows linearly with context length and quickly becomes prohibitive." The goal is not merely to reduce cost, but to reduce cost while preserving the property that longer context produces better performance—a property that full attention delivers but that many efficient alternatives sacrifice.

---

The Inadequacy of Current Efficient Architectures

Prior to this work, the dominant approaches to efficient long-context modeling could be categorized into three strategies, each with a fundamental limitation that the paper identifies empirically in Figure 1 (left panel):

1. Approximate Attention (Sliding Window + Hybrid)

What they do: Replace dense self-attention with a fixed-size sliding window (e.g., Longformer [8], SWA), optionally interleaving occasional full-attention layers to provide global context (e.g., Gemma's 5:1 sliding-to-full ratio [90]).

Where they fall short: Figure 1 shows that both pure SWA and the hybrid 5:1 approach produce worse loss than full attention, and critically, their loss gets worse relative to full attention as context length increases beyond 32K. The paper shows this directly: the loss ∆ (which subtracts the loss of full attention from the method's loss) trends negative for SWA and hybrid, meaning they fall further behind full attention in longer contexts. These architectures are fundamentally bottlenecked by their fixed window size—no matter how long the context grows, the model can only attend within its window, so additional tokens beyond the window provide no benefit (and can hurt, likely due to gradient noise during fine-tuning on long sequences with fewer sequences per batch).

2. Linear RNNs and State Space Models (Mamba 2, Gated DeltaNet)

What they do: Replace attention entirely with a recurrent hidden state that has constant size. Mamba 2 [21] uses structured state space duality; Gated DeltaNet [104] extends DeltaNet [106] with gating mechanisms inspired by Mamba. Both use a hybrid of their RNN layers interleaved with sliding-window attention layers.

Where they fall short: This is the crucial empirical finding in Figure 1. Mamba 2 and Gated DeltaNet achieve constant inference latency (right panel), but their loss ∆ increases dramatically with context length (left panel). At 128K, Mamba 2's loss ∆ is roughly +0.045 above full attention, and Gated DeltaNet's is approximately +0.035. For reference, TTT-E2E maintains a loss ∆ of approximately −0.01 to −0.02 (consistently below full attention) at all context lengths from 8K to 128K. The RNN baselines are not just worse in absolute terms—their relative disadvantage grows with longer context. The paper's diagnosis is implicit but clear: a fixed-size hidden state (the RNN's memory) has finite capacity, and as the context grows, increasingly important information gets discarded. The model cannot "compress" arbitrarily long context into a fixed-size vector without losing information critical for prediction.

3. TTT with Key-Value Binding (TTT-KVB)

What they do: TTT-KVB [87, 110] uses test-time training to construct a drop-in replacement for self-attention layers. At each layer, a small MLP is trained at test time to predict the value of each token from its key (the KV Binding loss, Equation 7). The updated MLP—essentially a learned associative memory—then produces the layer's output. This approach is closer in spirit to TTT-E2E because it also compresses context into model weights via gradient descent.

Where it falls short: Figure 1 shows TTT-KVB is the third-worst performer at 128K, with a loss ∆ of approximately +0.025. This is better than Mamba 2 but worse than Gated DeltaNet, and far from full attention. The paper identifies a specific reason in Subsection 2.4: TTT-KVB's layer-wise KV binding loss is not end-to-end at test time—it trains each layer independently to reconstruct values from keys, rather than optimizing the single metric that matters (next-token prediction loss at the final output). The paper provides concrete evidence for this diagnosis in Table 1: replacing the layer-wise KVB losses with a single end-to-end next-token prediction loss (moving to "TTT-E2E all layers MH") improves loss from 2.819 to 2.806 for the 760M model at 8K context. The KVB objective is a proxy—it tries to mimic self-attention's key-value associations—but it's the wrong proxy because it doesn't guarantee better final predictions.

An additional limitation of TTT-KVB: As discussed in Subsection 3.7, TTT-KVB must fit its hidden states (the updated MLP weights) onto individual GPU chips because they're treated as per-layer RNN states. This forces extreme compression: multi-head MLPs with LoRA [43] that have much smaller effective capacity than the regular MLPs in TTT-E2E. TTT-E2E, by using standard training infrastructure that shards MLP layers across GPUs, achieves a hidden state 5× larger (88M vs. 18M parameters for the 760M model) while being 2× faster at inference (0.0086 vs. 0.017 sec per 1K tokens for prefill on H100).

---

The Deeper Problem: Mismatched Training and Test-Time Objectives

Beyond the architectural limitations, the paper identifies a conceptual flaw that runs through prior work on trainable sequence models.

Dynamic evaluation mismatches training and test time. The classic approach, dating back to Mikolov et al. [72] and extended by Krause et al. [60], is to continue training the language model at test time on the given context via next-token prediction. This is what the paper calls TTT-naive in Subsection 2.2. The problem is that the model was pretrained to minimize its loss out-of-the-box (Equation 4), not its loss after undergoing test-time training. There is a mismatch: the model is optimized for a static evaluation, but at test time it's asked to adapt. The paper demonstrates this concretely in the right panel of Figure 2: TTT-naive (gray line) performs only slightly better than a Transformer with no attention at all (green), and far worse than full attention (orange) or TTT-E2E (blue). The training-time loss function is not preparing the weights for what will happen at test time.

TTT-KVB matches training and test time, but on the wrong loss. As discussed above, TTT-KVB uses meta-learning (outer loop over inner loop) so the training-time objective matches the test-time behavior—but both are optimizing the layer-wise KV binding loss, not the final prediction loss. This is end-to-end at training time but not at test time (in terms of the actual task). The paper draws this distinction explicitly in Subsection 2.4: "Our primary derivation starts from TTT via next-token prediction, which is E2E at test time, and focused on making it E2E at training time via meta-learning in Subsection 2.2. Our alternative derivation, on the other hand, starts from TTT-KVB, which is E2E at training time, and focused on making it E2E at test time via next-token prediction."

Clark et al. [17] comes closest but doesn't achieve efficiency. This contemporaneous work adds an MLP as fast weights updated via next-token prediction on chunks, trained with meta-learning. The method improves perplexity over the Transformer baseline but does not improve efficiency—the fast-weight MLP is added on top of full attention, so the combined architecture still has quadratic complexity. The paper also notes (Section 4.3) that Clark et al. only add the fast MLP at the end of the model, while TTT-E2E interleaves TTT-updated MLPs with sliding-window attention layers, which "proves to be critical for maintaining the performance gain on top of larger baselines."

---

How This Paper Positions Itself

The paper frames TTT-E2E as a unified solution to both the efficiency problem (sub-quadratic context scaling) and the objective mismatch problem (training-time and test-time alignment). The positioning is sharp and specific:

It is not an architectural innovation. The paper explicitly states (end of Subsection 3.2.1 via the b=8K ablation) that "architecture design plays a minor, supporting role in our method." TTT-E2E uses a standard Transformer with sliding-window attention—the same baseline architecture as SWA and the backbone of the hybrid and RNN baselines. The innovation is purely in the training procedure: what loss is optimized, when, and how the initialization is prepared.

It positions long-context modeling as continual learning. Section 4.1 explicitly frames the entire problem as continual learning rather than architecture design: "Most of today's AI systems remain static after deployment, even though the world keeps changing." The paper argues that the right approach to long context is not to design a better sequence model, but to let the model continue learning from the context it encounters. This reframing is significant because it connects long-context efficiency to a broader research agenda (continual learning, meta-learning, test-time training) that has been largely separate from the architecture-focused sequence modeling literature.

It identifies the specific failure of prior TTT approaches. The paper does not claim to invent test-time training for language modeling—it explicitly credits dynamic evaluation (Subsection 2.1) and TTT-KVB (Subsection 2.4) as direct predecessors. The contribution is identifying why these prior approaches underperform and fixing both ends simultaneously: use next-token prediction for the inner loop (E2E at test time) and meta-learn the initialization for that inner loop (E2E at training time). The alternative derivation in Subsection 2.4 is particularly effective at making this argument: it starts from TTT-KVB, replaces the KVB loss with next-token prediction (the key step), and then simplifies the architecture to use larger hidden states with less compute, reaching TTT-E2E through incremental changes whose individual effects are measured in Table 1.

It draws a direct analogy to biological memory. The paper's opening paragraph compares TTT-E2E's two memory systems—sliding-window attention (short-term, high-fidelity) and TTT-updated MLP weights (long-term, compressed)—to human memory: "You might not recall the instructor's first word during the lecture, but the intuition you learned is probably helping you understand this paper, even if that lecture happened years ago." This is not merely rhetorical. It articulates the core design principle: the model should forget details (which the sliding window discards) but retain understanding (which TTT compresses into the weights). The Needle-in-a-Haystack results in Subsection 3.5, where full attention dramatically outperforms TTT-E2E, empirically validate that TTT-E2E is indeed sacrificing lossless recall—and the language modeling results in Figure 1 show that this sacrifice is the right trade-off for prediction accuracy.

It establishes a clear empirical boundary condition. The paper does not claim universal superiority. In the FLOPs-matched scaling analysis (Figure 5), TTT-E2E's advantage over full attention decreases with more training compute in the small-compute regime, only stabilizing to match full attention's trend at medium-to-large budgets (760M+ models, 48B+ tokens). The paper interprets this as a regime where Transformers "under-perform with insufficient training compute compared to RNNs" (Section 3.3), suggesting that TTT-E2E's benefits are most pronounced when compute is the bottleneck—exactly the scenario where constant-cost inference matters most for deployment.

3. Technical Approach

This is primarily a methods paper with a strong empirical validation component, whose core idea is that long-context language modeling can be formulated as a continual learning problem where a standard Transformer with sliding-window attention continues training at test time via next-token prediction on the given context, and the model's initialization is prepared for this test-time training via meta-learning at training time—making the approach end-to-end in both the inner loop (test-time objective matches the final task) and the outer loop (training-time objective matches the test-time behavior).

3.1 Reader Orientation

TTT-E2E is a training procedure, not a new architecture. At test time, given a long sequence of tokens as context, the system continually updates a subset of the model's MLP weights via gradient descent on the standard next-token prediction loss—effectively compressing the context into the model's parameters. The system solves two problems simultaneously: it achieves constant inference cost per token (like an RNN) while scaling with context length in the same way as a Transformer with full attention (maintaining or improving performance as more context becomes available). The "shape" of the solution is a bi-level optimization: an inner loop that trains the model on the test sequence itself, and an outer loop that meta-learns the initialization so that the inner loop works well.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components, organized into two nested optimization loops:

1. Base Architecture (Transformer with Sliding-Window Attention): A standard Transformer where every full-attention layer is replaced by a sliding-window attention layer with a fixed window size $k = 8\text{K}$ tokens. This provides short-term, high-fidelity memory within each window. The MLP layers serve as the "storage" that TTT will update.

2. Inner Loop (Test-Time Training via Next-Token Prediction): At test time, the model processes the context in mini-batches of size $b = 1\text{K}$ tokens. After each mini-batch, it takes one gradient step on the standard next-token prediction loss—using the tokens it just saw as training data—updating only the MLP weights in the last 1/4 of the Transformer blocks. The updated weights serve as a compressed, long-term memory of the context seen so far.

3. Outer Loop (Meta-Learning the Initialization): At training time, each training sequence is treated as if it were a test sequence: the inner loop runs on it, and the model's loss after inner-loop adaptation is averaged over many training sequences. This average loss is then differentiated with respect to the initial weights (the weights before the inner loop), producing meta-gradients that flow through the inner-loop gradient steps. These meta-gradients update the initialization so that future test-time training will be effective.

4. Answer Selection (Standard Decoding): After the inner loop has processed all context tokens, the model uses its updated weights to decode the next token(s). For decoding multiple tokens, TTT only takes additional gradient steps once enough decoded tokens have accumulated to fill a full mini-batch.

Information flows as follows: a test sequence enters → the inner loop processes it in mini-batches, updating the MLP weights after each batch → after all mini-batches, the final updated weights are used to predict the next token → at training time, this entire process is differentiated through to produce meta-gradients that update the initialization weights.

3.3 Roadmap for the Deep Dive

• First, the inner loop mechanics (Subsection 2.1): How TTT works at test time via next-token prediction, starting from a toy architecture with no attention, to show the effect of TTT in isolation.
• Second, the outer loop and why it matters (Subsection 2.2): The mismatch between TTT-naive (static pretraining) and TTT-E2E (meta-learning), and how the end-to-end training objective is constructed and optimized via gradients of gradients.
• Third, mini-batch TTT and the sliding window (Subsection 2.3): How the efficiency and stability problems of online gradient descent are solved by moving to mini-batch updates, why this creates a "bigram within each batch" problem, and how sliding-window attention resolves it.
• Fourth, the three implementation details (Subsection 2.3.1): Which layers are updated during TTT, how many blocks to update, and how pre-trained knowledge is preserved via dual MLP layers—with ablations justifying each choice.
• Fifth, the alternative derivation from TTT-KVB (Subsection 2.4): How TTT-E2E can be reached by starting from prior work on key-value binding, replacing the layer-wise reconstruction loss with the end-to-end prediction loss, and simplifying the architecture to use larger hidden states with less compute.

3.4 Detailed, Sentence-Based Technical Breakdown

The Inner Loop: Test-Time Training via Next-Token Prediction

The paper begins with a deliberately minimal setup to isolate the effect of TTT. The "toy baseline" is a Transformer with all self-attention layers removed, leaving only the MLP layers. Without attention, this architecture has no mechanism to carry information from previous tokens to the current prediction—it is effectively a bigram model that can only condition on the immediately preceding token. The goal of this toy setup is to understand what TTT contributes when there are no confounding factors from other sequence modeling components.

The test-time training procedure. At test time, the model is given a context of $T+1$ tokens: $x_0, x_1, \ldots, x_T$, where $x_0$ is the Beginning of Sequence (<BOS>) token. The baseline architecture is denoted as $f$ with weights $W$. For each timestep $t = 1, \ldots, T$, the model computes the standard next-token prediction loss:

$\ell_t(W) = \text{CE}(f(x_{t-1}; W), x_t)$

where $\text{CE}$ is cross-entropy, $f(x_{t-1}; W)$ is the model's predicted distribution over the next token given the previous token, and $x_t$ is the ground-truth token.

What it computes: the cross-entropy between the model's prediction (conditioned on the immediately preceding token $x_{t-1}$) and the actual next token $x_t$. Because the toy baseline has no attention, the prediction $f(x_{t-1}; W)$ depends only on $x_{t-1}$ and the current weights $W$—there is no hidden state carrying forward information from earlier tokens.

Why this form: cross-entropy is the standard loss for next-token prediction, and using it directly makes the test-time training objective identical to the test-time evaluation objective. This is the "end-to-end at test time" property: the model is being trained on exactly the metric that will be used to evaluate it.

The key innovation is what happens after computing this loss: the model takes a gradient step to update its weights before moving to the next token:

$W_t = W_{t-1} - \eta \nabla \ell_t(W_{t-1})$

where $\eta$ is a learning rate for the inner loop, $W_{t-1}$ are the weights after processing the previous token, and $W_t$ are the updated weights that will be used to predict the next token. The initial weights at the start of test time are $W_0$.

What it computes: a single step of stochastic gradient descent on the loss at timestep $t$, evaluated at the weights from the previous timestep. The gradient $\nabla \ell_t(W_{t-1})$ measures how the loss would change if each weight were perturbed; subtracting $\eta$ times this gradient moves the weights in the direction that reduces the loss on token $x_t$.

Why this form: gradient descent is the standard optimization algorithm for neural networks, and using it at test time means the same infrastructure (automatic differentiation, optimizer states) can be reused. The per-token update is analogous to online learning, where the model continually adapts to a stream of data. Critically, by updating $W_{t-1}$ rather than $W_0$, information from all previous tokens is accumulated in the weights—the weight update at time $t$ depends on $W_{t-1}$, which itself depends on all gradients from $\nabla \ell_1$ through $\nabla \ell_{t-1}$.

After processing all $T$ context tokens, the model simply outputs $\hat{p}_{T+1} = f(x_T; W_T)$—the prediction for the next token using the final updated weights.

The effect in the toy example. Figure 2 (left panel) illustrates this process for a toy sequence of three tokens: given $x_1$ and $x_2$ as context, predict $x_3$. The forward pass (upward arrows) produces predictions; the backward pass (downward arrows) computes gradients; the horizontal arrows represent the weight update flowing from one timestep to the next. The right panel of Figure 2 shows the result: TTT-E2E with $b=1$ (online gradient descent, blue line) transforms the toy baseline (green line, effectively a bigram) into a model whose test loss decreases over time and approaches that of a Transformer with full attention (orange line). The green line is flat because the bigram cannot use additional context; the blue line slopes downward because each gradient step encodes information from the current token into the weights, where it persists for future predictions.

The Outer Loop: Learning to Learn at Test Time

The inner loop described above assumes we have initial weights $W_0$. The paper's central methodological claim is that these initial weights cannot be obtained through standard pretraining—they must be specifically optimized for the fact that they will be updated at test time.

The problem with TTT-naive. Standard pretraining minimizes the expected loss of the model out-of-the-box, without any test-time updates:

$L_{\text{naive}}(W_0; X) = \frac{1}{T} \sum_{t=1}^{T} \ell_t(W_0)$

where $\ell_t(W_0) = \text{CE}(f(x_{t-1}; W_0), x_t)$ is the loss using the initial weights $W_0$ at every timestep, and $X = (x_1, \ldots, x_T)$ is a training sequence.

What it computes: the average next-token prediction loss over the sequence, but always using the same static weights $W_0$. This is the standard language modeling pretraining objective.

Why this is wrong for TTT: at test time, the weights are not static—they are updated after each token via Equation 2. So $L_{\text{naive}}$ optimizes for a scenario (static weights) that does not match the deployment scenario (adaptive weights). The paper states this bluntly: "we can provide little guarantee that a minimizer of $L_{\text{naive}}$ will also produce low test loss $L$." The right panel of Figure 2 confirms this empirically: TTT-naive (gray line) performs only marginally better than the toy baseline (green), and far worse than TTT-E2E (blue), because its pretrained initialization was not designed to be a good starting point for gradient-based adaptation.

The end-to-end training objective. TTT-E2E instead optimizes exactly the loss that will be observed at test time—the loss after test-time training has been applied:

$L(W_0; X) = \frac{1}{T} \sum_{t=1}^{T} \ell_t(W_{t-1}) = \frac{1}{T} \sum_{t=1}^{T} \text{CE}(f(x_{t-1}; W_{t-1}), x_t)$

where $W_{t-1}$ is not a free variable but is computed from $W_0$ through the inner-loop update rule in Equation 2. That is, $W_1 = W_0 - \eta \nabla \ell_1(W_0)$, $W_2 = W_1 - \eta \nabla \ell_2(W_1)$, and so on. The loss $L$ depends on $W_0$ both directly (through $\ell_1(W_0)$) and indirectly (through all subsequent $W_t$, each of which is a function of $W_0$ via the chain of gradient steps).

What it computes: the average next-token prediction loss, but crucially, the prediction at timestep $t$ uses weights $W_{t-1}$ that have already been updated on tokens $x_1, \ldots, x_{t-1}$. The loss at time $t$ evaluates how well the model predicts $x_t$ after having learned from all previous tokens in the sequence.

Why this form: this is end-to-end training—the training loss exactly matches the test-time behavior. If we can find a $W_0$ that minimizes $L(W_0; X)$ averaged over training sequences, then at test time, starting from that $W_0$ and applying the same inner-loop updates will also produce low loss. The form also captures the sequential nature of the problem: the model must be good at predicting the next token given all previous tokens, where "given" means "after having trained on."

Meta-learning via gradients of gradients. To minimize $L(W_0; X)$ with respect to $W_0$, we need to compute $\nabla L(W_0)$. Because $W_{t-1}$ is itself a function of $W_0$ through the inner-loop gradient steps, computing $\nabla L(W_0)$ requires differentiating through those gradient steps—that is, computing gradients of gradients. The paper notes that modern automatic differentiation frameworks (JAX, PyTorch) can efficiently compute such higher-order gradients "with minimal overhead."

In the terminology of meta-learning, the gradient step on $L$ with respect to $W_0$ is called the outer loop, and the gradient steps on $\ell_t$ with respect to $W_{t-1}$ are called the inner loop. The outer loop optimizes the initialization for the inner loop's learning process. This is the same high-level structure as MAML [27], but applied to the problem of language modeling by treating each sequence as a "task" that the model must adapt to.

The practical workflow. At training time, for each training sequence $X$:

1. Run the inner loop: for $t = 1, \ldots, T$, compute $\ell_t(W_{t-1})$ and update $W_t = W_{t-1} - \eta \nabla \ell_t(W_{t-1})$.
2. Accumulate the losses $\ell_t(W_{t-1})$ to form $L(W_0; X)$.
3. Compute the meta-gradient $\nabla_{W_0} L(W_0; X)$ by backpropagating through the entire inner-loop computation graph.
4. Average meta-gradients over a batch of sequences and apply a standard optimizer (e.g., AdamW) to update $W_0$.

This is computationally more expensive than standard training because of the gradients-of-gradients computation, but the paper argues (and Figure 2 demonstrates) that it is necessary for the inner loop to be effective.

Mini-Batch TTT and the Sliding Window

The online gradient descent formulation in Equation 2 (one gradient step per token) has two practical problems that the paper identifies:

Efficiency. Taking a gradient step after every single token means $T$ sequential forward-backward passes that cannot be parallelized. For $T = 128\text{K}$, this is 128,000 sequential steps, each of which must wait for the previous one to complete before it can begin. Standard training parallelizes over mini-batches of size $b$, processing all $b$ tokens simultaneously and taking one gradient step per batch, reducing the number of sequential steps by a factor of $b$.

Stability. A gradient computed from a single token has very high variance. If the token happens to be unusual or if the model's prediction is particularly poor, the gradient can be large and cause the weights to move in an unhelpful direction. Averaging gradients over $b$ tokens reduces this variance by a factor of $b$.

Mini-batch TTT formulation. The paper generalizes the inner loop to use mini-batch gradient descent. Given the test-time context $x_1, \ldots, x_T$, the sequence is partitioned into $T/b$ batches (assuming divisibility), each of size $b$. The weight update rule becomes:

$W_i = W_{i-1} - \eta \frac{1}{b} \sum_{t=(i-1)b+1}^{ib} \nabla \ell_t(W_{i-1})$

for $i = 1, \ldots, T/b$, where $W_0$ is the initial weights and $W_i$ are the weights after processing the $i$-th mini-batch.

What it computes: for the $i$-th mini-batch (containing tokens $x_{(i-1)b+1}$ through $x_{ib}$), compute the average gradient of the next-token prediction loss over all $b$ tokens, evaluated at the weights $W_{i-1}$ from the previous mini-batch. Then take one gradient step with this averaged gradient. This processes $b$ tokens per weight update instead of one.

Why this form: it follows the standard mini-batch gradient descent recipe used in neural network training. The averaging over $b$ tokens reduces gradient variance and improves stability, while reducing the number of sequential steps from $T$ to $T/b$—a $b$-fold improvement in parallelism.

The outer-loop loss is correspondingly generalized to:

$L(W_0; X) = \frac{1}{T} \sum_{i=1}^{T/b} \sum_{t=(i-1)b+1}^{ib} \ell_t(W_{i-1})$

What it computes: the same as before (average loss over all tokens), but now the prediction at timestep $t$ uses the weights $W_{i-1}$ from the most recent mini-batch update, where $i = \lceil t/b \rceil$. All tokens within the same mini-batch share the same weights $W_{i-1}$.

The mini-batch introduces a memory gap. The problem with mini-batch TTT is illustrated by the red and purple lines in Figure 2 (right panel) for $b=16$. Consider the first mini-batch containing tokens $x_1, \ldots, x_b$. Every prediction $\hat{p}_t = f(x_{t-1}; W_0)$ uses the same initial weights $W_0$—no weight updates occur within the batch. This means the model has no memory of earlier tokens in the batch: $\hat{p}_t$ conditions only on $x_{t-1}$ (the immediately preceding token) via the toy baseline's architecture, plus whatever information is encoded in $W_0$ from pretraining. But $W_0$ hasn't seen any of the test context yet. The result is that within each mini-batch, $\ell_t(W_{i-1})$ increases with $t$ as the model misses more and more context. The paper states: "the only predictions without missing context are the first and second ones inside the mini-batch."

Why this is a problem: the gradients computed at the end of the mini-batch are based on predictions that had access to very little context, so they are poor indicators of what the weights should learn. These poor gradients produce worse weight updates, which propagate forward and degrade performance on subsequent mini-batches. In Figure 2, the purple line ($b=16$) performs worse than the blue line ($b=1$), and mini-batch TTT without the sliding window (the red line for $b=16$ without SWA) performs even worse because the model is a bigram within each batch.

Resolving the gap with sliding-window attention. The paper's solution is to augment the toy baseline with sliding-window attention layers. Instead of removing all attention layers (as in the toy example), TTT-E2E restricts the self-attention layers to a fixed window size $k$. For the main results with $T = 128\text{K}$, the window size is set to $k = 8\text{K}$ and the TTT mini-batch size to $b = 1\text{K}$. The paper states: "It is important to set $k \geq b$ so our model can remember the context within each mini-batch before TTT has a chance to update its weights."

How this works: the sliding-window attention provides short-term memory within each mini-batch. For any token $x_t$, the attention mechanism can attend to the previous $k$ tokens (including those in the current mini-batch), so the prediction $\hat{p}_t$ conditions on up to $k$ tokens of local context. The mini-batch size $b$ is set to $1\text{K}$ and the window size $k$ to $8\text{K}$, so $k \gg b$: the attention window spans multiple mini-batches, providing ample local context. The role of TTT is then to compress information from tokens beyond the sliding window into the updated MLP weights—these tokens cannot be attended to directly, but their information can be retained in the weights through gradient updates.

This creates a two-tier memory hierarchy:

• Short-term memory: the sliding-window attention, which provides lossless recall within the last $k = 8\text{K}$ tokens.
• Long-term memory: the TTT-updated MLP weights, which compress information from all tokens beyond the window through gradient-based learning.

The paper explicitly draws this analogy in the conclusion: "the weights updated at test time can be interpreted as long-term memory and the sliding window as short-term memory," comparing it to biological memory systems.

Default hyperparameters. For all main experiments with $T = 128\text{K}$, the paper uses $k = 8\text{K}$ and $b = 1\text{K}$. The sliding window size ablation (Figure 4, leftmost panel) shows that larger $k$ improves performance for TTT-E2E, SWA, and Gated DeltaNet, but the improvement plateaus. The choice of $k = 8\text{K}$ is motivated by the observation that "a smaller $k$ does not significantly improve runtime"—there is a trade-off between the quality of short-term memory and computational cost, and $8\text{K}$ is the sweet spot.

The mini-batch size ablation (Figure 4, middle panel) shows that larger $b$ significantly hurts performance for both TTT-E2E and TTT-KVB, because the memory gap within each batch grows with $b$. However, "a choice of $b$ smaller than $1\text{K}$ also significantly hurts our hardware utilization and stability, to the point that it becomes difficult to experiment with." The choice of $b = 1\text{K}$ represents the smallest mini-batch size that is computationally practical.

The $b = 8\text{K}$ ablation as a control experiment. When $b = 8\text{K}$, the mini-batch size equals the pre-training context length, so no TTT updates occur within a pre-training sequence. This is equivalent to not doing TTT at all—the model processes the entire sequence with static weights. The paper reports that without TTT, the loss for TTT-E2E (2.825) is almost identical to full attention (2.827), confirming that "architecture design plays a minor, supporting role in our method." The performance gains come from the TTT procedure, not from the slight architectural modifications (dual MLPs, frozen layers).

Three Implementation Details

The paper introduces three implementation details that are "necessary for achieving our reported results," while acknowledging that "it is still possible that they are merely artifacts of our experimental setup, and different design choices could be better suited in other setups." These details are justified with ablations in Section 3.

Detail 1: TTT only the MLP layers. Modern Transformer blocks consist of an attention layer, an MLP layer, and normalization layers. During TTT (the inner loop), the paper freezes the embedding layers, normalization layers, and attention layers, updating only the MLP layers. The justification: "updating them [the other layers] in the inner loop causes instability in the outer loop."

Why this matters: the MLP layers contain the majority of the model's parameters and are responsible for the non-linear transformations that produce the output. By focusing the inner-loop updates on the MLPs, the model can adapt its processing of the current context without destabilizing the attention patterns (which are determined by the attention weights) or the normalization statistics (which are determined by the normalization parameters). The frozen attention layers still use the sliding window to provide short-term memory; the updated MLPs provide the long-term compressed memory.

Detail 2: TTT only 1/4 of the blocks. The paper frames this as a trade-off between storage capacity and computational cost. "In general, less information is lost during compression when we have a larger amount of storage. In our case, the information is the context, and the storage is the updated MLP layers. However, updating more layers also implies more computation to back-propagate the gradients." The paper chooses to update only the last 1/4 of the Transformer blocks (the ones closest to the output).

The ablation evidence (Figure 4, rightmost panel). The paper experiments with updating the last 1/2 (12 layers for the 760M model with 24 total), 1/4 (6 layers), 1/8 (3 layers), and only the final layer. The results, plotted as loss ∆ relative to full attention across context lengths from 8K to 128K, show:

• Updating only 1 or 3 layers: the method does not scale with context length in the same way as full attention—the loss ∆ trends upward as context grows, indicating the compressed memory is insufficient to capture the information in longer contexts.
• Updating 6 layers (1/4): the loss ∆ is flat or slightly negative across all context lengths, matching full attention's scaling behavior.
• Updating 12 layers (1/2): the loss ∆ is also flat across context lengths, but performs at "roughly the same level as 6." There is no additional benefit from updating more layers despite the increased computational cost.

Why 1/4 is the default: 6 layers provide enough storage to compress the context without loss of information (the scaling curves match full attention), while 3 layers are insufficient (the curves diverge). The paper chooses 1/4 for all model sizes regardless of the total number of layers (e.g., for the 125M model with 12 total layers, 1/4 means 3 layers).

Detail 3: Two MLP layers per block in updated blocks. A concern with TTT is "forgetting the knowledge learned during pre-training." When the inner loop updates the MLP weights on the test context, it may overwrite useful general knowledge that the model acquired during pretraining (outer-loop training). The paper adopts "the simplest way to address this concern": in the blocks that are updated during TTT (the last 1/4), each block contains two MLP layers. One is the standard MLP that is updated by TTT (the "learning" MLP). The other is a static, frozen MLP that serves as "a 'safe' storage for pre-trained knowledge."

Parameter accounting. To keep the total number of parameters the same as the baselines for fair comparison, the paper reduces the hidden dimension of the MLPs throughout the entire network (including in blocks that are frozen during TTT). This means the TTT-updated MLPs and the static MLPs in the last 1/4 of blocks are individually smaller than the single MLP in a standard Transformer block, but together they have the same total parameters.

Design rationale: the static MLP provides a fallback—even if the TTT-updated MLP learns something from the test context that conflicts with pre-trained knowledge, the static MLP still retains the original knowledge. The outputs of the two MLPs are presumably combined (the paper does not specify the exact combination mechanism, but it is likely addition or concatenation followed by a linear projection). This dual-MLP design is a form of explicit knowledge preservation, contrasting with the implicit regularization that would come from, for example, a small learning rate or early stopping.

Decoding Multiple Tokens

The paper addresses the extension from prefilling (conditioning on context) to decoding (generating new tokens). At test time, after processing all $T$ prefilled tokens (with $T$ assumed divisible by $b$ for simplicity), the model has taken $T/b$ mini-batch gradient steps and its weights are $W_{T/b}$. The model then uses these weights to decode the next token $x_{T+1}$.

No TTT during decode until a batch fills. The model continues decoding tokens $x_{T+1}, x_{T+2}, \ldots$ using the current weights $W_{T/b}$ without taking any gradient steps, until it has decoded $b$ tokens. Once a full mini-batch of $b$ decoded tokens has accumulated, the model performs one step of TTT on this batch—using exactly the same procedure as during prefill—updating the weights to $W_{T/b + 1}$. The updated weights are then used to decode the next $b$ tokens, and the process repeats.

What this means for latency. As discussed in Subsection 3.7, "before reaching a full batch, our decode latency is the same as that of a regular Transformer with SWA." The decode cost per token is constant (sliding-window attention only) until the batch boundary, at which point there is one TTT step whose cost is the same as the prefill cost for one mini-batch. For long sequences, the TTT steps are amortized over $b$ decoded tokens each, so the average cost per decoded token remains constant plus a small per-batch overhead.

Alternative Derivation: From TTT-KVB to TTT-E2E

The paper provides an alternative derivation of TTT-E2E starting from prior work on long-context TTT based on Key-Value Binding (KVB) [87, 110]. This derivation serves two purposes: it connects TTT-E2E to the existing literature on RNNs and TTT layers, and it isolates the specific changes that lead to improved performance. The derivation proceeds in four steps, with each step's effect measured in Table 1 for 760M models at 8K context length.

Starting point: TTT-KVB. TTT-KVB constructs a sequence modeling layer that serves as a drop-in replacement for self-attention. The core idea is to store key-value associations implicitly in a learned model, rather than explicitly in a KV cache. At each layer $l$ and each timestep $t$, a small model $g$ (usually an MLP) is trained at test time to predict the value of each token from its key:

$\ell_t^{(l)}\left(W_{t-1}^{(l)}\right) = \left\| g\left(\theta_K^{(l)} x_t^{(l)}; W_{t-1}^{(l)}\right) - \theta_V^{(l)} x_t^{(l)} \right\|^2$

where $x_t^{(l)}$ is the input embedding at layer $l$ and timestep $t$, $\theta_K^{(l)}$ and $\theta_V^{(l)}$ are outer-loop parameters (analogous to the key and value projection matrices in Transformers), $W_{t-1}^{(l)}$ are the weights of $g$ after the previous timestep, and $\|\cdot\|^2$ is the squared L2 norm.

After the gradient step updates $W_{t-1}^{(l)}$ to $W_t^{(l)}$, the model $g$ uses the updated weights to produce the output embedding:

$z_t^{(l)} = g\left(\theta_Q^{(l)} x_t^{(l)}; W_t^{(l)}\right)$

where $\theta_Q^{(l)}$ is another set of outer-loop parameters (analogous to the query projection matrix). Each TTT layer operates independently with its own loss and weights. At training time, the outer loop optimizes all outer-loop parameters ($\theta_K$, $\theta_V$, $\theta_Q$, and $W_0$ for all layers) through meta-learning, similar to TTT-E2E's outer loop in Equation 6. TTT-KVB is thus end-to-end at training time (the outer loop optimizes for the inner-loop's behavior), but not at test time (the inner-loop loss is layer-wise KVB, not the final next-token prediction).

Step 1: Simplify the output rule. The output rule in Equation 8 uses the updated weights $W_t^{(l)}$ with a separate query projection $\theta_Q$. The paper observes that this can be simplified to reuse the prediction from the loss computation:

$z_t^{(l)} = g\left(\theta_K^{(l)} x_t^{(l)}; W_{t-1}^{(l)}\right)$

What changed: instead of calling $g$ a second time with the updated weights and the query input, the output embedding is simply $g$'s prediction of the value from the key before the gradient step. The paper provides an intuitive justification: "calling $g$ with the updated weights can be unnecessary if sliding-window attention already provides enough local context, and prior work has argued that the separation between $\theta_K$ and $\theta_Q$ can also be unnecessary."

Effect on performance (Table 1): this simplification causes a negligible change in loss (from 2.818 to 2.819 for the 760M model), which the paper considers "below the threshold of statistical significance" (difference of 0.001). The architecture is now closer to TTT-E2E, as illustrated in the right panel of Figure 3.

Why this matters: this step shows that the sophisticated output mechanism of TTT-KVB (separate query projection, updated weights) is not contributing to performance. The value prediction made during the loss computation is already sufficient as the layer's output.

Step 2 (the key step): Replace KVB loss with next-token prediction loss. Instead of having each layer independently minimize a reconstruction loss (predicting values from keys), the method switches to a single loss at the end of the network: the standard next-token prediction loss. This simultaneously eliminates the need for $\theta_K$ and $\theta_V$ (since there are no more layer-wise reconstruction losses) and makes the inner loop end-to-end at test time. The resulting method is called "TTT-E2E all layers MH," where MH stands for multi-head (the MLPs are still split into multiple heads as in TTT-KVB).

Effect on performance (Table 1): loss drops from 2.819 to 2.806—a substantial improvement of 0.013. This is the single largest improvement among the derivation steps, confirming the paper's central claim that the test-time objective matters crucially.

Why this matters: the KVB loss is a proxy—it tries to approximate self-attention's key-value associations. But the actual goal is to predict the next token correctly. By aligning the inner-loop loss with the final task, the model's test-time training directly optimizes what we care about. This step is what the paper means by "E2E at test time."

Step 3 (the final step): Larger state with less compute. The intermediate method, TTT-E2E all layers MH, still has two architectural features inherited from TTT-KVB:

1. It updates an MLP in every Transformer block (not just the last 1/4).
2. The updated MLPs are split into multiple heads, making them much smaller than regular MLPs (with $H$ heads, each head has $D^2/H$ parameters instead of $D^2$). Additionally, these MLPs are updated with LoRA [43], further reducing effective capacity.

The paper observes that these two choices create a disproportionate trade-off: "In order to update the small multi-head MLP in a block, gradients need to back-propagate through the large MLP above it, let alone the attention layer below for the backward pass to proceed further." The computational cost of preparing upstream gradients is high relative to the benefit of updating a tiny MLP.

The final method, TTT-E2E, removes both restrictions simultaneously:

• It updates only the last 1/4 of blocks (saving compute by not backpropagating through earlier blocks' MLPs).
• It updates regular (non-multi-head, non-LoRA) MLPs in those blocks (using the saved compute to update a larger hidden state).

Quantitative impact: For the 760M model, TTT-E2E has a hidden state 5× larger than TTT-E2E all layers MH (88M vs. 18M parameters) and 2× faster inference latency (0.0086 vs. 0.017 seconds per 1K tokens for prefill on H100). The loss improvement is modest at 8K context (2.806 to 2.805, difference of 0.001), but the context scaling ablation in Figure 4 (rightmost panel) shows that a smaller state (fewer updated layers) leads to worse context scaling—the advantage of larger state becomes apparent in long contexts, not short ones.

The Outer Loop in Practice: Training Recipe and Hyperparameters

The paper provides a detailed basic recipe (Table 3) that covers model configurations, pre-training, and fine-tuning. All experiments are reproducible from the public repository.

Model configurations. The paper experiments with five model sizes: 125M, 350M, 760M, 1.3B, and 2.7B parameters. The configurations follow GPT-3 [14] and Mamba [32]:

All models use the standard Transformer architecture with QK norm [74] and the Llama 3 tokenizer [24]. Rotary Position Embeddings (RoPE) [59] use $\theta = 500\text{K}$ for pre-training at 8K context, following Llama 3.

Pre-training recipe. Following GPT-3 with two modifications from the Mamba 2 paper: 5× the peak learning rate from GPT-3, and half the batch size for the 1B model. The number of pre-training tokens follows the Chinchilla recipe [40] (approximately 20× the number of model parameters). For the 760M model, this means 15B tokens. The learning rate schedule is identical across model sizes: for the first 10% of training, the learning rate increases linearly from 0 to the peak; for the remaining 90%, it decays to $1 \times 10^{-5}$ via a cosine schedule. The (outer-loop) batch size is 0.5M tokens for most models and 1M for the 2.7B model. The peak learning rates are: 3e-3 (125M), 1.5e-3 (350M), 1.25e-3 (760M), 1e-3 (1.3B), 8e-4 (2.7B).

Fine-tuning recipe. For extension to longer contexts (beyond the 8K pre-training length), the paper fine-tunes on the Books dataset [29]. The number of fine-tuning tokens is always 5% of the number of pre-training tokens (e.g., 750M tokens for the 760M model). The batch size is doubled relative to pre-training "so there can be a reasonable number of sequences per batch." The peak learning rate is $4 \times 10^{-4}$ for all model sizes and context lengths—this was determined by sweeping over $[8 \times 10^{-5}, 1 \times 10^{-4}, 2 \times 10^{-4}, 4 \times 10^{-4}, 8 \times 10^{-4}]$ for the full attention baseline, and finding that $4 \times 10^{-4}$ "happens to perform the best across all the model sizes and context lengths." The cosine schedule is restarted at the beginning of fine-tuning. RoPE $\theta$ is increased for full attention following standard practice: $\theta = 1\text{M}$ for 16K, $2\text{M}$ for 32K, $5\text{M}$ for 64K, and $10\text{M}$ for 128K, assuming a log-linear relationship with context length.

Datasets. Pre-training uses DCLM-Baseline [63], a heavily filtered subset of Common Crawl. Documents shorter than 8K tokens are discarded to avoid "resetting the updated MLPs across document boundaries when different documents are packed into the same training sequence, since resetting slows down training for our infrastructure." Fine-tuning uses Books [29], a standard academic dataset for long-context extension. Evaluation uses a held-out partition of Books.

Why a separate fine-tuning stage? The paper states that "today's large-scale runs usually consist of two or more stages: pre-training at short context length on a general dataset containing diverse knowledge, and extending the context length by fine-tuning on a dataset of long sequences." The two-stage design mirrors production workflows and isolates the effect of context extension from the effect of pre-training data diversity.

Why 5% for fine-tuning tokens? The paper does not provide a detailed justification beyond stating this as part of the basic recipe. It likely balances the need for sufficient long-context training data against the computational cost of fine-tuning, and follows conventions from prior long-context extension work.

Stability considerations for the ablations. The paper notes that fine-tuning the 760M model at 64K and 128K context length required doubling the outer-loop batch size beyond what the basic recipe specifies, because "this modification allows us to average over enough sequences so our fine-tuning runs are stable." The doubled batch size is adopted for all context lengths in the number-of-layers ablation to ensure clean comparison.

Computational Efficiency and Infrastructure

Inference latency. The right panel of Figure 1 shows prefill latency for 3B models on an H100. The setup: a constant number of tokens (128K) per outer-loop batch, so at 128K context length each batch contains one sequence, and at 8K each batch contains 16 sequences. TTT-E2E achieves constant inference latency regardless of context length (like SWA and the RNN baselines), making it 2.7× faster than full attention at 128K.

Why TTT-E2E is fast at inference despite doing gradient steps. At test time, TTT-E2E uses standard training infrastructure—the same forward-backward passes and optimizer updates that are highly optimized for training. The paper emphasizes an important advantage: "since our hidden state takes the form of regular MLP layers, it can be sharded across GPUs using standard tools with no custom kernel." In contrast, prior work like Mamba 2 and Gated DeltaNet must fit their hidden states onto individual GPU chips, requiring custom kernels for efficient memory I/O. TTT-KVB must reduce its state size with LoRA for the same reason.

Training latency (a limitation). The left panel of Figure 8 shows training latency on an H200. TTT-E2E's training is slower than standard Transformer training because it computes gradients of gradients. At 128K context, TTT-E2E is 1.2× faster than full attention, but at 8K (the pre-training length), it is 3.4× slower. Since most training compute is spent on pre-training at short context, this is a significant limitation.

Why training latency grows with context length despite constant FLOPs. The right panel of Figure 8 shows that TTT-E2E's FLOPs per token remain constant (the blue line is flat), but the left panel shows latency increases from 8K to 32K before flattening. The paper explains: "This trend arises because we have to increase the amount of gradient checkpointing through time by a factor of $\log(T)$, where $T$ is the context length." Gradient checkpointing [15] is a memory-saving technique that trades compute for memory by recomputing intermediate activations during the backward pass. For TTT, the inner-loop weights $W_1, \ldots, W_T$ would consume prohibitive memory if stored naively, so the paper applies gradient checkpointing through time—recomputing activations as needed during the meta-gradient computation. The $\log(T)$ factor comes from the hierarchical nature of the checkpointing schedule.

Two directions for faster training. The paper identifies two approaches for future work. First, a custom attention kernel that supports gradients of gradients: "Our current implementation cannot use cuDNN FlashAttention [20] at training time because it does not support gradients of gradients." Second, initializing TTT-E2E training from a pre-trained Transformer without TTT, so that the expensive meta-learning phase only accounts for a small portion of total training compute—a technique "often adopted by prior work on RNNs."

Summary of Design Choices and Their Justifications

• Sliding-window attention as the base architecture rather than full attention: reduces per-token cost from $O(T)$ to $O(k)$ constant, with the window providing short-term memory and TTT providing long-term compression.
• Next-token prediction as the inner-loop loss rather than layer-wise KVB: aligns the test-time training objective with the actual evaluation metric, producing a 0.013 loss improvement (Table 1, Step 2).
• Meta-learning (gradients of gradients) for the outer loop rather than static pretraining: prepares the initialization for gradient-based adaptation, without which TTT performs only marginally better than the baseline (Figure 2, TTT-naive vs. TTT-E2E).
• Mini-batch TTT with $b = 1\text{K}$ rather than online ($b = 1$): improves parallelism by $1000\times$ in sequential steps and improves gradient stability through averaging, at the cost of a memory gap within each batch that is filled by the sliding window.
• Window size $k = 8\text{K}$ and batch size $b = 1\text{K}$ with $k \gg b$: ensures the sliding window spans multiple mini-batches, so attention provides sufficient local context within each batch while TTT compresses beyond-window information.
• Updating only the last 1/4 of blocks: the context scaling ablation shows 6 layers (1/4 for the 760M model) are sufficient to match full attention's scaling, while 3 layers (1/8) are insufficient and 12 layers (1/2) provide no additional benefit at higher computational cost.
• Updating only MLP layers (not attention or normalization): prevents instability in the outer loop caused by updating attention patterns or normalization statistics during the inner loop.
• Dual MLPs in updated blocks: the static second MLP preserves pre-trained knowledge against overwriting by TTT, at the cost of reducing the hidden dimension across the entire network to keep total parameters constant.
• Standard Transformer architecture throughout: no custom kernels, no multi-head MLPs, no LoRA—just regular MLPs that can be sharded across GPUs using standard infrastructure. This is both a practical advantage (no custom kernel development) and a conceptual point (the gains come from the training procedure, not the architecture).
• Two-stage training (pre-train at 8K, fine-tune at target length): matches production workflows and isolates the effect of context extension. Fine-tuning uses 5% of pre-training tokens and a fixed learning rate of $4 \times 10^{-4}$ determined by sweeping the full attention baseline.
• RoPE $\theta$ scaling during fine-tuning: $\theta$ is increased log-linearly with context length (1M at 16K, 10M at 128K), following standard practice for long-context extension.

4. Key Insights and Innovations

Innovation 1: Long-Context Language Modeling as a Continual Learning Problem Rather Than an Architecture Design Problem

This paper's most fundamental intellectual move is the reframing of long-context efficiency from an architectural question ("what sequence model design can handle long contexts?") to a learning question ("how should the model continue learning from the context it encounters?"). This is not a semantic distinction—it changes what problems researchers should work on.

What the field assumed before this work. The dominant research program for efficient long-context modeling has been architecture design: invent new sequence mixing layers (linear attention, state space models, gated delta rules, TTT layers with key-value binding) that serve as drop-in replacements for self-attention. The goal was to design a layer that, when stacked in a deep network, could process arbitrarily long sequences with constant per-token cost while matching the representational capacity of full attention. This framing treated the sequence model as a static function—once trained, it processes tokens one at a time through a fixed computation graph. The research question was: what is the right computation graph?

What TTT-E2E does differently. The paper explicitly formulates long-context modeling as continual learning (Section 4.1): "Most of today's AI systems remain static after deployment, even though the world keeps changing. The high-level goal of continual learning is to enable AI systems to keep changing with the world." Under this formulation, the model is not a static function but an adaptive learner that uses the test-time context as training data. The architecture (sliding-window Transformer) provides the mechanism for short-term, high-fidelity memory, while the continual learning process (TTT via next-token prediction) provides the mechanism for long-term, compressed memory. The research question shifts from "what architecture?" to "what learning procedure, applied when, with what initialization?"

Why this is a fundamental shift, not an incremental refinement. The paper demonstrates this shift concretely through the control experiment at $b = 8\text{K}$ (Figure 4, middle panel), where TTT is effectively disabled because the mini-batch size equals the pre-training context length. With TTT turned off, the loss of TTT-E2E (2.825) is essentially identical to that of full attention (2.827). The paper's own words: "this observation suggests that architecture design plays a minor, supporting role in our method." A reader who comes away thinking TTT-E2E is "an RNN with MLP hidden states" has missed the point—the architecture is deliberately standard and unremarkable. The entire contribution is in the learning procedure: when gradient steps are taken (at test time, after each mini-batch of context), what loss they optimize (next-token prediction, the same metric used for evaluation), and how the initialization is prepared (meta-learning that directly optimizes post-adaptation loss).

This reframing connects long-context efficiency to a broader set of research traditions—continual learning, meta-learning, test-time training, fast weights—that had been largely separate from the architecture-focused sequence modeling literature. It implies that progress on long-context models may come not from better recurrent cells but from better learning algorithms for test-time adaptation, better initialization strategies, and better understanding of how to compress sequential information into gradient updates. It also explains, retrospectively, why prior TTT approaches like TTT-KVB underperformed: they were still thinking like architecture designers (building a drop-in replacement for self-attention layers) rather than like continual learning researchers (formulating the right learning problem for the whole network).

---

Innovation 2: The Double-End-to-End Principle—Aligning Both Test-Time and Training-Time Objectives

The paper's title emphasizes "End-to-End" twice, and this is not redundant. The insight is that test-time training methods for sequence modeling have suffered from two distinct misalignments, and fixing only one is insufficient—both must be addressed simultaneously for the approach to work at scale.

The two misalignments, and how prior work addressed at most one of them.

Misalignment at test time (the inner loop): The model's test-time training objective should be the same as its evaluation objective. Prior TTT methods for long context (TTT-KVB [87, 110], MesaNet [98], Titans [7]) use a proxy loss—key-value binding—to update their weights at test time. The proxy loss trains each layer to reconstruct values from keys, which is inspired by self-attention's mechanism but is not the task the model is ultimately evaluated on (next-token prediction). The paper quantifies the cost of this misalignment in Table 1 (the "key step"): replacing the layer-wise KVB losses with a single end-to-end next-token prediction loss improves loss from 2.819 to 2.806 for the 760M model—a 0.013 improvement that is the single largest gain among the derivation steps. This is not merely a better auxiliary loss; it is the difference between optimizing a proxy and optimizing the true objective.

Misalignment at training time (the outer loop): The model's pretraining objective should prepare it for the fact that its weights will be updated at test time. Dynamic evaluation [72, 60], which the paper calls TTT-naive, trains the model to minimize its out-of-the-box loss (Equation 4), then at test time subjects it to gradient-based adaptation. The mismatch is stark: the weights are optimized for a static scenario, but deployed in an adaptive one. The paper shows the consequence in Figure 2 (right panel): TTT-naive (gray line) barely outperforms a Transformer with no attention at all (green line), while TTT-E2E with meta-learning (blue line) approaches full-attention performance. The model was never taught how to be a good learner—its initialization is not a good starting point for gradient-based adaptation.

Why fixing both simultaneously is non-obvious and significant. The two alignments are independent design dimensions. TTT-KVB fixes the training-time alignment (it uses meta-learning so the outer loop optimizes for inner-loop behavior) but breaks the test-time alignment (the inner loop optimizes KVB, not next-token prediction). Dynamic evaluation fixes the test-time alignment (the inner loop optimizes next-token prediction) but breaks the training-time alignment (the outer loop optimizes static performance). The paper's key diagnostic move is identifying that neither alone suffices, and the key methodological move is providing an architecture-agnostic procedure that achieves both: use the standard next-token prediction loss everywhere (inner loop and outer loop), and use meta-learning so that the outer loop directly optimizes post-adaptation performance.

This principle should generalize beyond the specific architecture choices in this paper. Any test-time training method for any sequential task should ask two questions: (1) Is the test-time training loss the same as the evaluation loss? (2) Is the training-time objective aligned with the test-time procedure? A "no" to either question predicts degraded performance relative to a method that answers "yes" to both. The paper provides a template for achieving the double-end-to-end property: make the inner loop optimize the final task loss, make the outer loop optimize the post-inner-loop loss, and connect them through gradients of gradients.

---

Innovation 3: The Memory Hierarchy Interpretation—Why Sliding-Window Attention + TTT Outperforms Either Alone

The paper's architectural choice—a standard Transformer with sliding-window attention, where TTT updates only the MLP layers—is not arbitrary. It instantiates a specific theory about what kinds of memory are needed for long-context processing, and why a single mechanism (all-attention or all-recurrence) is insufficient.

The two mechanisms and their complementary strengths. The sliding-window attention provides lossless, high-fidelity recall within a fixed horizon ($k = 8\text{K}$ tokens). It can attend to exact token identities, positions, and relationships within the window with the full power of self-attention. But it has a hard boundary: tokens beyond the window are invisible. TTT provides lossy, compressed memory of all tokens beyond the window through gradient-based learning. It cannot recall exact details (as the Needle-in-a-Haystack results in Subsection 3.5 demonstrate—full attention dramatically outperforms TTT-E2E on exact retrieval), but it can extract and retain the statistical patterns, intuitions, and understanding that matter for prediction.

Why this division of labor is theoretically motivated, not just empirically convenient. The paper draws an explicit analogy to human memory in the opening paragraph: "You might not recall the instructor's first word during the lecture, but the intuition you learned is probably helping you understand this paper." This is not rhetoric—it articulates a design principle. Exact recall of every detail is computationally expensive and often unnecessary for the task of predicting what comes next. What matters for language modeling is understanding the topic, style, entities, and argumentative structure of the context—precisely the kind of information that gradient-based compression into weights can capture. Conversely, local syntactic and semantic relationships (within a paragraph or section) benefit from exact attention, and the sliding window provides this.

The evidence that the two mechanisms are complementary, not redundant. The window size ablation (Figure 4, leftmost panel) shows that TTT-E2E and SWA (which has only short-term memory, no TTT) both improve with larger window sizes, but TTT-E2E maintains a consistent advantage over SWA at every window size. At $k = 8\text{K}$ (which equals the pre-training context length, so SWA equals full attention in the pre-training setting), TTT-E2E still outperforms full attention by 0.018 in loss. This means TTT is not merely compensating for the absence of full attention—it is providing an orthogonal benefit that full attention does not capture. Conversely, the Needle-in-a-Haystack results (Table 2) show that TTT-E2E performs poorly on tasks requiring exact retrieval of arbitrary details—full attention's lossless recall is necessary for that capability, and TTT's compression cannot substitute for it.

This memory-hierarchy interpretation reframes the design space for efficient sequence models. Rather than searching for a single recurrent state that can simultaneously handle local syntax and long-range dependencies (the approach of Mamba, Gated DeltaNet, and linear attention variants), the paper suggests that the right architecture has two distinct memory systems with complementary properties. The design problem shifts from "how do we build a better recurrent state?" to "what is the right division of labor between exact local memory and compressed long-term memory, and how should they interact?"

---

Innovation 4: Verifier-Free TTT—Eliminating the Heuristic Proxy Objective from Test-Time Training for Sequences

TTT-KVB and its variants (Titans, MesaNet, Nested Learning) are built on the idea that TTT layers should learn to mimic self-attention's key-value associations—essentially, that the test-time training objective should be a form of self-supervised reconstruction that verifies whether the layer's internal model can reproduce the value vectors from the key vectors. This paper makes the case, through direct empirical comparison and ablation, that this verification-style objective is harmful compared to simply optimizing the end-to-end task loss, and that the architectural complexity it introduces (per-layer losses, separate key/value/query projections, multi-head MLPs with LoRA) is unnecessary.

What prior TTT approaches implicitly assumed. The KVB loss is derived from the observation that self-attention computes a weighted sum of value vectors, where the weights come from key-query similarities. The idea was: if a small model can learn to predict each token's value from its key, then that model's weights serve as a compressed representation of the key-value cache. The KVB loss is a reconstruction loss—it measures how well the internal model can reproduce the values. This is conceptually similar to using a verifier or a self-supervised proxy task: train an auxiliary model on an intermediate objective, and hope that improving this auxiliary model translates to better final predictions.

What TTT-E2E shows instead. By replacing the per-layer KVB reconstruction losses with a single next-token prediction loss at the network output (the "key step" in Subsection 2.4.3, quantified in Table 1), the paper demonstrates that the proxy objective is not merely unnecessary—it is worse than the direct objective. The 0.013 loss improvement from this single change is the largest among all derivation steps. The intuitive explanation: the KVB loss encourages each layer to faithfully reproduce the value vectors, but faithful reproduction of values is not the same as good next-token prediction. Some information in the values may be irrelevant or even detrimental to the final task; some information needed for the final task may not be well-captured by value reconstruction. By optimizing the final loss directly, the model's test-time training is free to compress whatever aspects of the context are most useful for prediction, without being constrained by an intermediate reconstruction target.

The broader implication. This finding challenges a common pattern in the test-time training literature—using self-supervised auxiliary tasks (reconstruction, contrastive learning, rotation prediction) as the inner-loop objective. The paper shows that for sequence modeling at least, the auxiliary task is a bottleneck: it limits what the model can learn at test time to what can be captured by that specific proxy. The direct approach—train on the task loss itself—requires no auxiliary task design, no balancing of multiple losses, and no architectural components dedicated to the auxiliary objective. The prerequisite is meta-learning (so the initialization supports test-time training on the task loss), which this paper provides. This is not to say auxiliary tasks are never useful—they may be essential when the task loss cannot be computed at test time (e.g., in unsupervised domain adaptation where test labels are unavailable). But for next-token prediction, where the test loss is computable on the context itself, the paper makes a strong case that auxiliary objectives are strictly worse than the real thing.

5. Experimental Analysis

Evaluation Methodology

• Dataset. Pre-training uses DCLM-Baseline [63], a heavily filtered subset of Common Crawl, with all documents shorter than 8K tokens discarded (to avoid resetting TTT-updated MLPs across document boundaries during training). Extension fine-tuning uses Books [29], a standard academic dataset for long-context extension, with a held-out partition used for language modeling evaluation. Needle-in-a-Haystack evaluation uses the three S-NIAH tasks from RULER [42]. The pre-training dataset construction starts from the 3.8T tokens in DCLM-Baseline, discards documents shorter than 8K, and randomly samples to construct training sets of various sizes. For Books, most DCLM sequences longer than 128K are of low quality, motivating the switch to Books for fine-tuning.

• Base model(s). All experiments use standard Transformer architectures with QK norm and the Llama 3 tokenizer, following model configurations from GPT-3 [14] and Mamba [32]. Five model sizes are explored: 125M (12 blocks, 768 embedding dim, 12 heads), 350M (24 blocks, 1024 dim, 16 heads), 760M (24 blocks, 1536 dim, 16 heads), 1.3B (24 blocks, 2048 dim, 32 heads), and 2.7B (32 blocks, 2560 dim, 32 heads). The largest models used in the main context-scaling experiments are 3B parameters. The paper argues PaLM 2-S* was chosen as "representative of the capabilities of many contemporary LLMs" for prior TTT work, and the GPT-3/Mamba configurations here serve a similar role for the efficient architecture literature.

• Metrics. The primary metric throughout is test loss, reported as either log perplexity or as "loss ∆"—the difference (loss of reported method) − (loss of Transformer with full attention). Loss ∆ is used so that full attention's performance is the flat line at y = 0, making it easy to see which methods outperform or underperform the full-attention baseline across context lengths. For Needle-in-a-Haystack, accuracy (0 to 1) is reported as the fraction of queries where the model successfully retrieves the target string. For the decoding evaluation, log likelihood of the generated text under Qwen-3-8B-Base is reported. Latency is measured in seconds per 1K tokens for prefill on an H100 or H200 GPU. FLOPs per token are reported for training efficiency analysis.

• Baselines. Six baselines represent the state-of-the-art approaches in architecture design for efficient long-context modeling. All baselines with sliding windows use the same window size k = 8K:

  1. Transformer with full attention [95]: Standard Transformer with dense self-attention over the entire context. Uses the GPT-3/Mamba model configurations. This is the gold-standard baseline—the "flat line at y = 0" in all loss ∆ plots.
  2. Transformer with Sliding-Window Attention (SWA) [8]: Every full attention layer replaced by a sliding-window attention layer with window size k. Since the pre-training context length is also 8K, full attention and SWA are identical until extension fine-tuning (where SWA becomes restricted to the window).
  3. Hybrid SWA and full attention (5:1) [90]: Repeating the pattern of five SWA layers followed by one full attention layer, in the style of Gemma. Provides occasional global context through the full-attention layers while maintaining efficiency through SWA layers.
  4. Mamba 2 [21]: A popular RNN that uses a hybrid of Mamba 2 layers and SWA layers. Tested at large scale in Nemotron-H [11]. The paper uses the official code and configurations provided by the authors.
  5. Gated DeltaNet [104]: A popular RNN extending Mamba 2 and DeltaNet [106], using a hybrid of Gated DeltaNet layers and SWA layers. Tested at large scale in Kimi Linear [91]. The paper uses the official code and configurations.
  6. TTT-KVB [110]: A TTT-based RNN using a hybrid of TTT-MLP layers with Key-Value Binding loss [87] and SWA layers. This is the starting point for the paper's alternative derivation in Subsection 2.4 and is the most directly comparable prior TTT method. Variants include MesaNet [98], Titans [7], and Nested Learning [6], which follow similar constructions.

  Baselines 1–3 are implemented in JAX alongside TTT-E2E. Baselines 4–6 use the official PyTorch code and configurations. The paper made two improvements to the baselines: upgrading their attention layers to use FlashAttention 3 [81] (from FlashAttention 2) for fairer latency comparisons, and applying QK norm to baselines with sliding-window attention layers (which improved Gated DeltaNet's loss from 2.814 to 2.809 at 8K pre-training, and from 2.691 to 2.683 at 32K fine-tuning, for the 760M model).

• Generation budget / compute accounting. The paper does not use a "generation budget" in the sense of number of sampled solutions. Instead, compute is measured along three axes: (1) context length (8K to 128K), which determines the length of the test sequence; (2) training compute in terms of model parameters and number of training tokens (following Chinchilla scaling conventions); and (3) inference latency in seconds per 1K tokens and FLOPs per token for training. For latency measurements, a constant number of tokens (128K) per outer-loop batch is used regardless of context length, so at 128K each batch contains one sequence and at 8K each batch contains 16 sequences. Training compute for pre-training follows the Chinchilla recipe, and fine-tuning always uses 5% of the pre-training token count. For the model size scaling experiments, all five model sizes (125M to 2.7B) are used. For token scaling experiments, the 760M model is trained on up to 5× the basic Chinchilla token count, with fine-tuning token count maintaining the 5% ratio.

• Cross-validation / statistical protocol. The paper does not employ cross-validation in the traditional sense. Training consists of a single pre-training run on DCLM at 8K context length, followed by separate fine-tuning runs on Books at each target context length (16K, 32K, 64K, 128K). Each context length gets a separate fine-tuned model from the same pre-trained checkpoint. Evaluation uses a held-out partition of Books for language modeling. For the number-of-layers ablation, all fine-tuning runs (including at shorter contexts) use doubled batch size to ensure stability at 64K and 128K. No statistical significance testing or confidence intervals are reported. The paper notes that a difference of 0.001 in loss is considered "below the threshold of statistical significance" (Table 1 caption).

Main Quantitative Results

The paper's experimental results are organized into four major investigations: scaling with context length (the headline result), scaling with training compute, Needle-in-a-Haystack evaluation, and decoding long sequences. Computational efficiency is analyzed separately across both inference and training.

Scaling with Context Length (Figure 1, Figure 6, Figure 9)

Headline result. For 3B models trained with 164B tokens, TTT-E2E maintains a consistent loss advantage over Transformer with full attention across all context lengths from 8K to 128K, while all other efficient architectures (SWA, Hybrid 5:1, Mamba 2, Gated DeltaNet, TTT-KVB) show worsening loss relative to full attention as context length increases. Simultaneously, TTT-E2E achieves constant inference latency regardless of context length, making it 2.7× faster than full attention at 128K context on an H100.

Detailed loss ∆ analysis (Figure 1, left panel). The paper plots loss ∆ (computed as loss of the reported method minus loss of Transformer with full attention) against context length on a log scale from 8K to 128K:

• Full attention is the flat line at y = 0 by definition.
• TTT-E2E (blue) maintains a negative loss ∆ of approximately −0.01 to −0.02 across the entire range, meaning it consistently outperforms full attention. The line is essentially flat with a slight downward trend beyond 32K, indicating its advantage does not erode—and may slightly grow—with longer context.
• SWA (green) has a loss ∆ starting near 0 at 8K (since k = 8K equals the pre-training context length, making SWA identical to full attention until fine-tuning) and rising steeply to approximately +0.05 at 128K. SWA is the worst performer at 128K.
• Hybrid SWA and full 5:1 has a loss ∆ starting near 0 at 8K and rising to approximately +0.03 at 128K, performing better than pure SWA but still substantially worse than full attention.
• Gated DeltaNet has a loss ∆ starting slightly positive (approximately +0.005) at 8K and rising to approximately +0.035 at 128K. It is the best of the non-TTT-E2E efficient methods.
• Mamba 2 has a loss ∆ starting near 0 at 8K and rising to approximately +0.045 at 128K, performing worse than Gated DeltaNet.
• TTT-KVB has a loss ∆ starting near 0 at 8K and rising to approximately +0.025 at 128K, placing it between Gated DeltaNet and Mamba 2.

The critical pattern is the slope of these lines. TTT-E2E's line is flat or slightly negative; all other efficient methods' lines have positive slope, meaning their performance relative to full attention degrades as context lengthens. The paper's interpretation is that these methods have finite memory capacity that becomes a bottleneck—longer context provides more information, but the fixed-size recurrent state or windowed attention cannot effectively capture it.

Absolute loss values (Figure 9). To complement the loss ∆ plot, the paper provides the raw loss values in log perplexity:

• Full attention: approximately 2.25 at 8K, decreasing to approximately 2.25 at 128K (essentially flat).
• TTT-E2E: approximately 2.24 at 8K, decreasing slightly to approximately 2.24–2.23 at 128K (slightly below full attention throughout).
• SWA: approximately 2.26 at 8K, increasing to approximately 2.33 at 128K.
• Gated DeltaNet: approximately 2.26 at 8K, increasing to approximately 2.32 at 128K.
• Mamba 2: approximately 2.26 at 8K, increasing to approximately 2.33 at 128K.
• TTT-KVB: approximately 2.26 at 8K, increasing to approximately 2.31 at 128K.

Why SWA and the RNNs degrade at longer contexts. The paper offers a concrete explanation for the upward trend in loss beyond 32K for SWA, Mamba 2, Gated DeltaNet, and TTT-KVB: "for longer context, there are fewer training sequences per (outer-loop) mini-batch during extension fine-tuning, so the gradients have higher variance; at the same time, these methods cannot effectively leverage the benefit of longer context, so the harm of the higher variance outweighs the benefit" (Figure 9 caption). In other words, at long context lengths, a single batch contains very few sequences (at 128K with a constant token budget per batch, perhaps only 1–2 sequences), making gradient estimates noisy. Full attention and TTT-E2E overcome this noise because they genuinely benefit from the longer context. The other methods do not benefit enough to offset the increased gradient variance, so their loss goes up.

Loss breakdown by token index (Figure 6). The paper decomposes the aggregate loss for 32K and 128K contexts into per-token-index losses. This reveals where exactly TTT-E2E's advantage comes from:

• TTT-E2E is the only method that achieves lower losses than full attention at every token index throughout the entire context. At both 32K (left panel) and 128K (right panel), the TTT-E2E curve sits below the full attention curve at all token positions from t = 128 to t = 32K or 128K.
• The advantage is largest at early tokens and narrows near the end. At 32K, TTT-E2E's loss advantage over full attention is roughly 0.1–0.2 log perplexity at token indices 128–1K, narrowing to near zero by t = 32K. At 128K, the same pattern repeats—the TTT-E2E curve is well below full attention at earlier indices, and the gap narrows near t = 128K.
• This pattern "stretches" with context length. The right panel (128K) resembles a stretched version of the left panel (32K) rather than a continuation where the curves cross. The paper interprets this as evidence that TTT-E2E maintains the same advantage structure regardless of context length: the model's initial weights produce better predictions for early tokens, and TTT maintains this advantage by adapting as context accumulates.

The paper provides an intuitive explanation for why TTT-E2E outperforms full attention on early tokens: "The weights of full attention must prepare to be good at all future tokens in the context window. Such a task can be very hard, because being good at all possible futures limits the model's capacity to be good at any particular one. But the weights of TTT-E2E only need to be good at the present mini-batch of tokens, since TTT will produce future weights for the future tokens."

The b = 8K control experiment (Figure 4, middle panel). At b = 8K, the mini-batch size equals the pre-training context length, so no TTT steps occur during a sequence—the model processes all tokens with static weights. Both TTT-E2E (loss 2.825) and TTT-KVB (loss 2.826) are essentially identical to full attention (2.827). This confirms that the architectural modifications alone (dual MLPs, frozen layers) contribute negligibly to performance—the gains come from the TTT procedure itself.

Inference latency (Figure 1, right panel). For 3B models at 128K context on an H100:

• Full attention: approximately 0.065 seconds per 1K tokens.
• TTT-E2E: approximately 0.024 seconds per 1K tokens—2.7× faster.
• SWA, Mamba 2, Gated DeltaNet, TTT-KVB: all cluster around 0.022–0.025 seconds per 1K tokens, essentially identical to TTT-E2E's latency.

TTT-E2E and all the efficient methods have flat latency curves as context length grows, while full attention's latency increases linearly. This is expected because TTT-E2E's per-token compute is dominated by the sliding-window attention (constant cost per token) plus a fixed per-mini-batch cost for the TTT gradient step, which amortizes to constant per token.

Scaling with Training Compute (Figure 5)

Motivation. The paper investigates whether TTT-E2E's performance advantage holds up as more compute is invested in training, along two axes: model size (125M to 2.7B) and number of training tokens (up to 5× the Chinchilla recipe for the 760M model). The concern is that efficient architectures sometimes show advantages at small scale that disappear when models are large and well-trained—if TTT-E2E's benefit shrinks to zero at scale, it would be a method for the small-compute regime only.

Setup. For model scaling, all five model sizes are pre-trained on DCLM at 8K and evaluated on both DCLM (left panel a) and Books at 32K after fine-tuning (left panel b). For token scaling, the 760M model is trained on 16B, 32B, 48B, 64B, and 80B tokens (pre-training), with fine-tuning tokens at 5% of each, and evaluated on both DCLM at 8K (right panel c) and Books at 32K (right panel d). Gated DeltaNet is included as the representative RNN baseline.

Key pattern: advantage shrinks in the small-compute regime, stabilizes in the medium-compute regime.

• Model scaling on DCLM (Figure 5a): TTT-E2E's loss ∆ relative to full attention starts at approximately −0.06 for the 125M model, rises to approximately −0.04 at 350M, −0.01 at 760M, near 0 at 1B, and approximately −0.01 at 3B. The advantage shrinks substantially from 125M to 760M but then stabilizes—the curve is essentially flat from 760M to 3B. Gated DeltaNet follows a similar pattern, starting at approximately −0.06 at 125M and rising to near 0 by 760M–1B.

• Model scaling on Books at 32K (Figure 5b): The pattern is similar but compressed. TTT-E2E's loss ∆ starts at approximately +0.02 at 125M (slightly worse than full attention), drops to approximately −0.01 at 350M, rises to approximately +0.01 at 760M, and stabilizes near 0 (−0.005 to +0.005) at 1B and 3B. The fine-tuned results are closer to full attention across the board.

• Token scaling on DCLM (Figure 5c): TTT-E2E's loss ∆ starts at approximately −0.015 at 16B tokens, rises to approximately +0.005 at 32B (briefly worse than full attention), drops slightly, and stabilizes near −0.01 from 48B onward. Gated DeltaNet follows a similar trajectory but ends near 0 at 80B. The vertical dotted line at 48B marks where the paper considers the "medium-compute regime" to begin.

• Token scaling on Books at 32K (Figure 5d): TTT-E2E's loss ∆ starts at approximately +0.005 at 16B, rises to approximately +0.02 at 32B, then drops toward 0 (−0.005 to +0.005) at 48B through 80B. Again, the advantage is smaller after fine-tuning and stabilizes in the medium-compute regime.

The dotted vertical lines mark regime boundaries. The paper places a dotted line at 760M for model scaling and at 48B tokens for token scaling, marking "the boundary for the change of regime." In the small-compute regime (to the left of the dotted line), TTT-E2E's advantage decreases with more compute. In the medium-to-large regime (to the right), the advantage stabilizes. The paper offers two explanations for this pattern:

1. "Our method can also be interpreted as a hybrid RNN... We expect RNNs (sequence models with hidden states of fixed size) to share a similar trend for scaling with training compute." The fact that Gated DeltaNet shows the same pattern supports this interpretation.
2. "Transformers are widely known to under-perform with insufficient training compute compared to RNNs. Our observations can be interpreted as a deficiency of the full attention baseline with small compute, rather than a deficiency of RNNs with large compute." That is, TTT-E2E's large advantage at 125M–350M may reflect full attention being undertrained, not TTT-E2E being especially strong.

The takeaway. The paper's stated conclusion is that TTT-E2E "should produce the same trend as full attention for scaling with training compute in large-budget production runs." The stabilization of the loss ∆ curve at larger model sizes and token counts supports extrapolation to larger scales, though the paper does not actually run larger-scale experiments to verify this extrapolation.

Sensitivity to tokenizer and data quality. The paper reports two anecdotal observations:

• Switching to the Llama 3 tokenizer (2024) from the Llama 2 tokenizer (2023) improved TTT-E2E's advantage over full attention by about 0.01 for 3B models.
• Switching to DCLM (2024) from SlimPajama (2023) enabled TTT-E2E to produce the same trend as full attention for token scaling after 48B. With SlimPajama, "our lines in the right panels of Figure 5 exhibited a small uptick, similar to those in the left panels for scaling with model size." This suggests that data quality and tokenizer recency affect the scaling behavior, and that older/less curated data makes the advantage degrade at scale while newer data preserves it.

Needle in a Haystack (Table 2)

Setup. All 3B models fine-tuned at 128K context length are evaluated on the three S-NIAH tasks from RULER [42]: S-NIAH-1 (pass-key retrieval), S-NIAH-2 (number in haystack), and S-NIAH-3 (UUID in haystack). These tasks require the model to locate a specific target string (the "needle") within a long passage of irrelevant text (the "haystack"). The needle is distinguished by being clearly irrelevant to the surrounding text.

Results. Across all three tasks, Transformer with full attention dramatically outperforms all other methods, including TTT-E2E:

• S-NIAH-1: Full attention achieves 1.00 accuracy at 8K–64K and 0.99 at 128K. All other methods degrade severely: SWA drops from 1.00 at 8K to 0.07 at 128K; Hybrid drops from 1.00 to 0.21; Mamba 2 from 0.99 to 0.07; Gated DeltaNet from 1.00 to 0.07; TTT-KVB from 0.98 to 0.01; TTT-E2E from 1.00 to 0.06. At 128K, TTT-E2E (0.06) is comparable to SWA (0.07), Mamba 2 (0.07), Gated DeltaNet (0.07), and TTT-KVB (0.01), but dramatically worse than full attention (0.99).

• S-NIAH-2: The pattern repeats with slightly better performance for the efficient methods. Full attention: 0.99–1.00 across 8K–64K, 0.86 at 128K. TTT-E2E: 0.99 at 8K, dropping to 0.05 at 128K. Mamba 2 and Gated DeltaNet also drop to 0.05 at 128K. The Hybrid 5:1 method is the best non-full-attention method, dropping only to 0.29 at 128K.

• S-NIAH-3: The hardest variant, requiring retrieval of a random UUID string. Full attention: 0.64 at 8K, 0.64 at 128K (the drop from the other tasks suggests that UUID retrieval is inherently harder). TTT-E2E: 0.77 at 8K (outperforming full attention!), dropping to 0.03 at 128K. Gated DeltaNet: 0.91 at 8K, dropping to 0.03 at 128K. The Hybrid 5:1 drops to 0.06 at 128K.

Interpretation. The paper explicitly states: "This observation, combined with findings from our previous subsections, supports the intuition that the strength of full attention lies in its nearly lossless recall. This strength is inherent to the design of self-attention, which attends to the keys and values of all previous tokens in its cache. In contrast, the key mechanism in our method is compression, which leaves out seemingly irrelevant details, such as the target string." TTT-E2E is designed to compress and forget—the Needle-in-a-Haystack results confirm that this design choice has the expected cost: exact retrieval of arbitrary details is severely impaired compared to lossless attention. The performance at 8K for S-NIAH-3 (where TTT-E2E gets 0.77 vs. full attention's 0.64) is an interesting exception that the paper does not analyze further.

Decoding Long Sequences (Figure 7)

Setup. Evaluating base models on long-form generation is inherently challenging because base models without instruction fine-tuning tend to produce repetitive or degenerate text. The paper uses Qwen-3-8B-Base [92] as an external evaluator: each 3B model (from the context-scaling experiment in Subsection 3.4) is given 8K tokens from Books as context, decodes another 8K tokens as continuation, and the Qwen-8B model's log likelihood is plotted for the concatenated 16K sequence. A repetition penalty of 1.1 is used to prevent degenerate generation, and top-p sampling with p = 0.95 and temperature 1.0 is used following standard practice.

Results (Figure 7). The plot shows Qwen loss (log perplexity) against token index from 0 to 16K, with a dotted vertical line at 8K marking the prefill/decode boundary. For both TTT-E2E and full attention:

• During prefill (tokens 0–8K): TTT-E2E achieves lower Qwen loss than full attention, consistent with the language modeling evaluations.
• At the prefill/decode boundary (token 8K): Both methods show a sharp increase in Qwen loss, jumping from approximately 2.3 to 2.6 log perplexity. The paper attributes this to Qwen being "initially unfamiliar with the generation style of the evaluated method."
• During decode (tokens 8K–16K): Both methods' Qwen loss gradually decreases as more generated content accumulates. TTT-E2E maintains a consistent advantage over full attention throughout the decode phase, with its loss curve running approximately 0.05–0.1 log perplexity below full attention's curve.

The paper states that "we have carefully inspected ≈20 samples of the generated text and found them reasonable," providing qualitative confirmation that the generation is coherent.

How TTT operates during decode. The paper reiterates the mechanism from the end of Subsection 2.3: during the prefill phase, TTT-E2E processes the 8K context in mini-batches of 1K tokens, taking gradient steps after each batch. After prefill, the model decodes tokens one at a time using the final updated weights. Once 1K tokens have been decoded (filling a full mini-batch), TTT takes one gradient step on this batch of self-generated tokens, updating the weights before decoding the next batch. This means the decode performance reflects both the initial context compression and the model's ability to continue learning from its own generated output.

Caveat. The paper acknowledges that "it is inherently challenging to evaluate base models, without the two stages above [instruction fine-tuning and reinforcement learning], in a realistic way." The evaluation with Qwen-8B provides a proxy for generation quality but does not directly measure task success, coherence, or factuality of the decoded text.

Computational Efficiency (Figures 1 and 8)

Inference latency (Figure 1, right panel). Already discussed under scaling with context length. The key number: 2.7× faster than full attention at 128K on an H100.

Training latency (Figure 8, left panel). On an H200, with a constant 128K tokens per batch:

• At 8K context: TTT-E2E training latency is approximately 0.35 seconds per 1K tokens, while full attention is approximately 0.10—TTT-E2E is 3.4× slower.
• At 128K context: TTT-E2E is approximately 0.40 seconds per 1K tokens, while full attention is approximately 0.50—TTT-E2E is 1.2× faster.

The paper flags training latency as a "significant limitation": "Since most of the training compute is typically spent on pre-training with short context, the training latency of our current implementation remains a significant limitation." The higher cost at short context comes from the gradients-of-gradients computation, which is less optimized than standard training.

The cause of increasing latency at constant FLOPs. TTT-E2E's FLOPs per token are constant across context lengths (Figure 8, right panel: the blue line is flat at approximately 2.5 × 10^10 FLOPs per token). But latency increases from 8K to 32K before flattening (left panel). The paper explains: "This trend arises because we have to increase the amount of gradient checkpointing through time by a factor of log(T), where T is the context length." Gradient checkpointing through time is applied hierarchically to manage the memory cost of storing inner-loop weights W_1, ..., W_T, and the log(T) factor in the checkpointing schedule causes the latency increase despite constant FLOPs.

FLOPs comparison (Figure 8, right panel). TTT-E2E's FLOPs per token are constant and sit between the efficient baselines (Mamba 2, Gated DeltaNet, TTT-KVB, all approximately 1.5–2 × 10^10 FLOPs/token) and full attention (which grows linearly with context length, reaching approximately 7 × 10^10 FLOPs/token at 128K). SWA and Hybrid have constant FLOPs similar to the efficient baselines. TTT-E2E's higher constant FLOPs relative to Mamba 2 and Gated DeltaNet reflect the cost of the inner-loop gradient steps (backpropagation through the last 1/4 of the network's MLP layers).

Ablation Studies and Robustness Checks

Sliding window size k (Figure 4, leftmost panel): Larger windows improve performance for TTT-E2E, SWA, and Gated DeltaNet, but the improvement plateaus. For the 760M model evaluated on DCLM after pre-training, TTT-E2E's loss drops from approximately 2.84 at k = 1K to 2.82 at k = 8K, with diminishing returns beyond 4K. The paper notes that TTT-E2E with k = 8K (which equals the pre-training context length) is effectively TTT-E2E on top of full attention in this setting, and it improves loss over full attention by 0.018—indicating TTT provides an orthogonal benefit beyond what full attention captures. The default k = 8K is chosen because "a smaller k does not significantly improve runtime."

TTT mini-batch size b (Figure 4, middle panel): Larger batch sizes significantly hurt performance for both TTT-E2E and TTT-KVB. For the 760M model evaluated on DCLM after pre-training, TTT-E2E's loss rises from approximately 2.805 at b = 1K to 2.825 at b = 8K. TTT-KVB similarly rises from 2.818 to 2.826. The choice of b = 1K is the smallest batch size that does not "significantly hurt our hardware utilization and stability." The b = 8K point serves as the "no TTT" control, showing that without TTT, the architectures perform essentially identically to full attention.

Number of layers updated during TTT (Figure 4, rightmost panel): This is the most consequential ablation. For the 760M model (24 layers total), trained on DCLM and fine-tuned on Books at each context length 8K–128K:

• Updating only 1 or 3 layers: the method does NOT scale with context length in the same way as full attention. The loss ∆ curves trend upward, meaning performance degrades relative to full attention as context lengthens. At 128K, loss ∆ is approximately +0.02 for 1 layer and approximately +0.01 for 3 layers.
• Updating 6 layers (1/4 of total): the loss ∆ curve is approximately flat at −0.01 across all context lengths, matching full attention's scaling behavior and maintaining a small advantage.
• Updating 12 layers (1/2 of total): the loss ∆ curve is also approximately flat, performing at "roughly the same level as 6" layers. The paper concludes that 1/4 of layers is the sweet spot—fewer layers cannot compress enough information to scale with context length, while more layers provide no additional benefit at higher computational cost. This result directly supports the paper's claim about state size (number of updated parameters) being critical for context scaling, as discussed in Subsection 2.4's "Final Step."

Effect of QK norm on baselines: Adding QK norm to Gated DeltaNet improved its loss from 2.814 to 2.809 for pre-training at 8K, and from 2.691 to 2.683 for fine-tuning at 32K (for the 760M model). This improvement is applied to all baselines with sliding-window attention layers to ensure fair comparison.

Data quality sensitivity (anecdotal): The paper reports that with SlimPajama (2023) instead of DCLM (2024), TTT-E2E's token scaling curves exhibited a small uptick (rather than being flat), indicating that data quality affects whether the advantage stabilizes or degrades at larger training budgets. Switching from the Llama 2 tokenizer to Llama 3 improved TTT-E2E's advantage by approximately 0.01 in loss.

Critical Assessment

Does the paper demonstrate that TTT-E2E "scales with context length in the same way as Transformer with full attention"?

Yes, for the specific setting tested: 3B models, 164B training tokens, Books evaluation. Figure 1 (left panel) shows that TTT-E2E's loss ∆ is essentially flat from 8K to 128K, meaning its relative performance compared to full attention neither degrades nor improves substantially as context grows. This is genuinely impressive given that every other efficient method (SWA, Hybrid, Mamba 2, Gated DeltaNet, TTT-KVB) shows worsening loss ∆ with context length. The flatness of TTT-E2E's curve is the single strongest piece of evidence for the paper's central claim.

However, several caveats are important:

1. The evaluation is on a single dataset (Books) in a single domain. Language modeling on books is a specific task with particular statistical properties—narrative coherence, character continuity, thematic consistency. The paper does not test whether TTT-E2E's context scaling holds on other long-context tasks like legal document review, codebase understanding, or multi-turn dialogue.
2. The scaling is demonstrated up to 128K, not to million-token scales. The paper's title says "Long Context," and the experiments go to 128K, which is long by 2024 standards but far from the 1M+ contexts that production models now target. Whether TTT-E2E's flat loss ∆ continues to 256K, 512K, or 1M tokens is not tested. The sliding window is k = 8K, the mini-batch is b = 1K, and only 1/4 of layers are updated—these hyperparameters might need to change for much longer contexts, and the paper provides no guidance on how to scale them.
3. Full attention itself shows essentially zero improvement from longer context on Books (Figure 9). Full attention's loss on Books is approximately 2.25 at both 8K and 128K. If the ground truth is that longer context doesn't help much on this dataset, then "scaling like full attention" means "not degrading like the other methods," not "improving with context." The paper's framing in the introduction—"using longer context to achieve better performance"—is not actually demonstrated for any method on this evaluation set. The loss breakdown in Figure 6 shows that TTT-E2E outperforms full attention at early token indices, but the absence of improvement from longer context for full attention itself limits how strong a claim can be made about "scaling with context."

Does the paper demonstrate that TTT-E2E achieves "2.7× faster than full attention for 128K context"?

Yes, for prefill latency on an H100. The right panel of Figure 1 shows this clearly, and the latency advantage is robust because TTT-E2E's per-token cost is dominated by the sliding-window attention (constant cost). However:

1. Decode latency is not measured separately. The paper argues in Subsection 3.7 that TTT-E2E's decode latency is the sum of SWA decode latency and prefill latency for each mini-batch of b = 1K decoded tokens. For applications that generate very long outputs (e.g., chain-of-thought), the decode cost dominates, and the 2.7× figure only applies to prefill.
2. Training latency is substantially worse at short contexts (3.4× slower at 8K). Since most training compute is spent on pre-training at short context, TTT-E2E is significantly more expensive to train than a standard Transformer at the scales that matter most for total cost. The paper acknowledges this as a "significant limitation" and proposes two mitigations (custom attention kernel for gradients-of-gradients, and initializing from a pre-trained Transformer) that are left to future work.
3. The latency comparison uses FlashAttention 3 for the baselines and cuDNN FlashAttention for TTT-E2E. The paper states they upgraded the PyTorch baselines to FlashAttention 3 to improve their latency, but TTT-E2E "cannot use cuDNN FlashAttention at training time because it does not support gradients of gradients." This means the inference latency numbers reflect different attention implementations—TTT-E2E uses a less optimized kernel than the baselines at inference time, which makes the 2.7× speedup somewhat conservative (TTT-E2E might be even faster with a better kernel).

Does the paper demonstrate that TTT-E2E "scales with training compute in the same way as full attention"?

Partially—the evidence is suggestive but incomplete. Figure 5 shows that TTT-E2E's loss ∆ stabilizes at 760M model size and 48B tokens, supporting extrapolation to larger scales. However:

1. The largest model tested is only 2.7B (described as 3B in the text). Modern production language models are 7B–405B parameters. Extrapolation from 760M–2.7B to 8B–70B is a 10–100× jump that the paper does not validate.
2. The largest token count is 5× the Chinchilla recipe. Chinchilla-optimal for a 760M model is approximately 15B tokens; the paper tests up to 80B. This is still far from the trillion-token scale of production models.
3. Gated DeltaNet shows the same stabilization pattern. The paper interprets this as evidence that the pattern is shared among RNN-like architectures, which is plausible but also means the finding is not specific to TTT-E2E—it may be a general property of hybrid architectures with fixed-size hidden states.
4. The "stabilization" at larger scales could be an artifact of the loss ∆ metric. If both TTT-E2E and full attention improve with more training compute, and their improvement rates are similar, the loss ∆ will be flat. But "similar improvement rates" does not guarantee that TTT-E2E's advantage at 3B will persist at 30B—the curves could diverge again at larger scales. The paper provides no mechanistic reason to expect stabilization to continue indefinitely.

Does the paper demonstrate that TTT-E2E's advantage comes from the "end-to-end" alignment of objectives?

Yes, through the ablation in Table 1. The stepwise derivation from TTT-KVB to TTT-E2E isolates three changes:

1. Simplifying the output rule: negligible effect (loss change from 2.818 to 2.819, difference of +0.001).
2. Replacing KVB loss with next-token prediction loss: large improvement (2.819 to 2.806, −0.013). This is the key step, directly validating the "E2E at test time" claim.
3. Larger state with less compute (updating only 1/4 of layers with regular MLPs): modest improvement at 8K context (2.806 to 2.805, −0.001), but critical for context scaling as shown in the number-of-layers ablation (Figure 4, rightmost panel).

The chain of evidence is clean and well-controlled. The 0.013 improvement from Step 2 is the largest single gain, directly supporting the paper's central methodological claim.

Missing ablation: What if we use end-to-end next-token prediction for the inner loop but train with the naive static loss (no meta-learning)? This is the comparison between TTT-naive and TTT-E2E in Figure 2 (right panel), but only in the toy setting with a 2-block Transformer without attention. The paper never evaluates TTT-naive at scale (760M–3B models on DCLM/Books). This is a significant gap: the toy result shows a large difference between naive and E2E, but the toy architecture is a bigram without attention—TTT is the only mechanism providing memory. In the full architecture with sliding-window attention, the memory gap within each batch is partially filled by the window, so TTT's role is different. It is possible that TTT-naive performs much closer to TTT-E2E in the full architecture, which would weaken the claim that meta-learning is essential. The fact that the paper does not report this comparison at scale is a notable omission.

Does the paper demonstrate that TTT-E2E's memory hierarchy (sliding window + TTT-updated MLPs) is the right division of labor?

The Needle-in-a-Haystack results (Table 2) provide strong validation of the intended trade-off. TTT-E2E performs dramatically worse than full attention on exact retrieval tasks, confirming that compression via TTT sacrifices lossless recall. The language modeling results (Figure 1) show that this sacrifice is worthwhile for next-token prediction, where TTT-E2E outperforms full attention despite losing the ability to recall arbitrary details. The combination of these two results—good on language modeling, bad on exact retrieval—directly supports the paper's design principle that "the key mechanism in our method is compression, which leaves out seemingly irrelevant details."

Missing experiment: Ablation of the dual-MLP design. The paper introduces dual MLPs in updated blocks (a static MLP for pre-trained knowledge, a TTT-updatable MLP for compressed context) as one of three implementation details. However, there is no ablation showing what happens without the dual-MLP design—no experiment comparing TTT-E2E with a single, larger MLP that is fully updated during TTT, versus the dual-MLP configuration with the same total parameter count. This is a significant gap because the dual-MLP design is the paper's proposed solution to the forgetting problem, and without evidence that it actually prevents forgetting, it remains an architectural choice rather than a validated design principle. The paper could have compared TTT-E2E with and without the static MLP on a task that requires pre-trained knowledge (e.g., factual QA within long contexts) to test whether the dual-MLP design preserves knowledge better.

Are there weaknesses in the experimental design that limit the conclusions?

Single evaluation benchmark (Books). The primary evaluation for context scaling and all fine-tuning results uses a held-out partition of Books. While Books is a standard dataset for long-context extension, it represents a narrow domain (narrative text). The paper does not evaluate on code, scientific papers, legal documents, or multi-turn dialogue. Without multi-domain evaluation, we cannot know whether TTT-E2E's advantage over full attention is specific to narrative text or generalizes.

No evaluation on tasks requiring reasoning or information extraction. The only task-specific evaluation is Needle-in-a-Haystack (where TTT-E2E performs poorly), and there is no evaluation on tasks like long-document QA, summarization, or multi-hop reasoning that would test whether the compressed long-term memory actually captures useful information for downstream tasks. Next-token prediction loss (log perplexity) is the sole metric for language modeling quality—it is a standard metric, but improvements in perplexity do not always translate to improvements in task performance.

The decoding evaluation is limited and uses an external model. The decoding evaluation in Figure 7 uses Qwen-8B as the evaluator, which introduces a confound: differences in Qwen's loss may reflect differences in how well Qwen models the two distributions, not differences in generation quality per se. The manual inspection of ~20 samples is anecdotal and not quantified. An ideal evaluation would include human evaluation or task-based metrics for the generated continuations.

No comparison against retrieval-augmented methods or sparse attention methods. The baselines cover sliding window, hybrid, and RNN approaches, but not retrieval-based long-context methods (e.g., caching and retrieving relevant chunks) or sparse attention methods (e.g., BigBird, Longformer with global attention, or the more recent Native Sparse Attention). These represent an alternative approach to long-context efficiency that the paper does not engage with.

Difficulty estimation cost is not analyzed for TTT-E2E (unlike the reference example paper). While the paper discusses computational cost in terms of latency and FLOPs, it does not decompose the cost into components (e.g., forward pass, backward pass, optimizer step) or analyze whether the per-mini-batch gradient step's cost could be reduced. For example, could TTT use a smaller learning rate, fewer optimizer states, or lower precision for the inner loop? The paper treats the inner-loop cost as fixed and does not explore efficiency-accuracy trade-offs within the TTT mechanism itself.

The QK norm and FlashAttention upgrades to baselines may not be sufficient for full fairness. The paper made two improvements to baselines (QK norm and FlashAttention 3), but there may be other optimizations specific to Mamba 2 or Gated DeltaNet that would improve their performance. The paper states they consulted the baseline authors, but the extent of optimization is unclear. The gap between TTT-E2E and the RNN baselines in Figure 1 is large enough (approximately 0.04–0.05 loss ∆ at 128K) that it's unlikely to be closed by hyperparameter tuning, but this remains a possibility.

The training data filtering (discarding DCLM documents shorter than 8K) is not applied to baselines. The paper discards documents shorter than 8K from DCLM for TTT-E2E's pre-training "to avoid resetting the updated MLPs across document boundaries." It's not stated whether baselines are trained on the same filtered data. If baselines were trained on the full DCLM (including short documents), they would have access to more diverse training data, potentially disadvantaging TTT-E2E. The paper should clarify this, and if the filtering was only applied to TTT-E2E, should provide a control experiment with baselines trained on the same filtered data.

No statistical error bars or significance testing. None of the figures report confidence intervals, standard errors, or any measure of statistical significance. The 500-question test set split into five difficulty bins leaves small per-bin sample sizes, and the paper does not assess whether differences between methods are statistically reliable. The one exception is the statement that a loss difference of 0.001 is "below the threshold of statistical significance" (Table 1), but no justification is provided for this threshold.

6. Limitations and Trade-offs

Training Latency at Short Context Lengths

The assumption or constraint. TTT-E2E's outer loop requires computing gradients of gradients—meta-gradients that backpropagate through the inner-loop gradient steps. This operation is significantly less optimized in current deep learning frameworks than standard training. The paper acknowledges this directly in Section 3.7: "At training time, TTT-E2E takes gradients of gradients, which is a much less optimized procedure compared to training a regular Transformer." The consequence is captured in Figure 8 (left panel): at 8K context length (the pre-training length), TTT-E2E's training latency is approximately 0.35 seconds per 1K tokens on an H200, compared to approximately 0.10 seconds for full attention—a 3.4× slowdown.

The consequence. This limitation reshapes the practical economics of adopting TTT-E2E. The paper's headline results on context scaling and inference speed address deployment efficiency, but the training-time cost is disproportionately higher at the short context lengths where the vast majority of training compute is spent in standard two-stage pipelines (pre-training on diverse data at short context, then fine-tuning for extension). For a practitioner evaluating total cost of ownership, the 3.4× training slowdown at pre-training length may dominate the 2.7× inference speedup at 128K, depending on the ratio of training compute to inference queries. The paper's proposed mitigations—a custom attention kernel supporting gradients of gradients, and initializing TTT-E2E from a pre-trained standard Transformer—are described as future work and are not implemented or evaluated.

What evidence exists in the paper. Figure 8 provides the latency numbers. The left panel shows TTT-E2E's latency curve starting at ~0.35 at 8K and rising to ~0.40 at 32K before flattening, while full attention starts at ~0.10 and rises linearly to ~0.50 at 128K. The crossing point (where TTT-E2E becomes faster) is around 64K–128K. The paper also notes that the latency increase from 8K to 32K despite constant FLOPs (right panel) is due to "gradient checkpointing through time by a factor of log(T)" and that "our current implementation cannot use cuDNN FlashAttention at training time because it does not support gradients of gradients."

Mitigation status. The paper proposes two directions but implements neither: (1) a custom attention kernel that supports gradients of gradients, and (2) initializing TTT-E2E training from a pre-trained Transformer without TTT, so the expensive meta-learning phase accounts for only a small fraction of total training compute. The paper notes this second approach is "often adopted by prior work on RNNs" but does not test it. Without either mitigation, the training cost remains a significant barrier to adoption at scales beyond the 3B/164B-token experiments in the paper.

---

The Needle-in-a-Haystack Boundary: Exact Recall Is Fundamentally Sacrificed

The assumption or constraint. TTT-E2E is built on compression—the inner loop compresses the context into the updated MLP weights via gradient descent on next-token prediction. Compression by its nature discards information. The paper is explicit about this design choice in the introduction: "How can we achieve better performance in longer context without recalling every detail, as in the opening example? The key mechanism is compression." The NIAH results in Table 2 reveal the sharp boundary of this trade-off: at 128K context, TTT-E2E achieves 0.06 accuracy on pass-key retrieval (S-NIAH-1), compared to 0.99 for full attention, and 0.03 accuracy on UUID retrieval (S-NIAH-3), compared to 0.64 for full attention. This is not a modest degradation—it is near-total failure on tasks that require locating a specific piece of information in a long context.

The consequence. For any application where users need to ask questions about specific facts buried in long documents—contract review, legal discovery, evidence retrieval from reports, fact-checking against source material—TTT-E2E would be unsuitable in its current form. The model cannot reliably "look up" a particular detail that it saw earlier unless that detail happened to influence the next-token prediction loss strongly enough to be preserved in the compressed weights. This is not a bug that can be fixed by scaling up; it is inherent to the compression mechanism. The paper's argument is that this trade-off is acceptable because language modeling (predicting what comes next) does not require exact recall of arbitrary details, and the NIAH results support the existence of the trade-off without establishing where the boundary lies for more practical retrieval-like tasks (e.g., answering a question about a character mentioned 50K tokens ago in a novel).

What evidence exists in the paper. Table 2 provides comprehensive NIAH results across three task variants and five context lengths. Full attention dominates across the board, with all efficient methods (including TTT-E2E) performing at near-zero accuracy beyond 16K–32K for most variants. The one exception is S-NIAH-3 at 8K, where TTT-E2E achieves 0.77 vs. full attention's 0.64—an unexplained inversion that the paper does not analyze. The paper's interpretation is that "the key mechanism in our method is compression, which leaves out seemingly irrelevant details, such as the target string." This is a candid acknowledgment of the limitation but does not quantify it for tasks that lie between "pure language modeling" and "pure needle retrieval"—for instance, question-answering where the answer is a single sentence in a 100K-token document.

Mitigation status. The paper does not attempt to mitigate this limitation. It presents the NIAH results as evidence that the method behaves as designed—sacrificing lossless recall for better language modeling—and frames this as a feature, not a bug. The hybrid 5:1 baseline (sliding window with occasional full attention) achieves substantially better NIAH performance than pure SWA or TTT-E2E (e.g., 0.21 vs. 0.06–0.07 on S-NIAH-1 at 128K; 0.29 vs. 0.05 on S-NIAH-2 at 128K), suggesting that a hybrid approach combining TTT-E2E with occasional full-attention layers could recover some retrieval capability. The paper does not explore this combination.

---

Difficulty Estimation Cost Is Unaccounted For in the Headline Efficiency Numbers

The assumption or constraint. The paper does not use the term "difficulty estimation" in the sense of the reference paper (estimating per-question difficulty), but an analogous unaccounted cost exists in TTT-E2E: the inner-loop gradient steps at test time. The paper's headline latency comparison (Figure 1, right panel) measures prefill latency for processing context, which includes the cost of running the inner loop (forward pass, loss computation, backward pass, weight update) after each mini-batch of b = 1K tokens. For a 128K context, this means 128 inner-loop gradient steps. These steps are included in the latency measurement, so the "2.7× faster than full attention" figure is honest about test-time cost.

However, there is a subtler unaccounted cost: the paper does not analyze what happens to latency when decoding very long sequences with interleaved TTT steps. As described in Subsection 2.3.2, after every b = 1K decoded tokens, the model pauses to take a gradient step on the self-generated batch before continuing. For a system generating 128K tokens of output, this means 128 additional gradient steps interspersed throughout generation. The paper's decode latency analysis (Subsection 3.7) states that "before reaching a full batch, our decode latency is the same as that of a regular Transformer with SWA," and that the TTT step cost "is the same as that for prefill." However, no end-to-end decode latency numbers are reported. The paper argues that the cost amortizes to constant per token, which is mathematically true, but in interactive or latency-sensitive settings, the periodic gradient steps could cause noticeable pauses every 1K generated tokens.

The consequence. For a practitioner deploying TTT-E2E in a system that generates long outputs (e.g., chain-of-thought reasoning, story generation, code completion), the actual user-perceived latency may include periodic stalls that are not captured by an amortized per-token average. If the TTT gradient step for a 1K-token batch takes, say, 100ms for a 3B model, a user generating 8K tokens would experience 7 pauses of 100ms each (at tokens 1000, 2000, ..., 7000), adding 700ms of stall time to the generation. The paper provides no measurements of this effect and no analysis of whether the TTT step latency is perceptible or problematic at the scales (3B, 1K batch size) they tested.

What evidence exists in the paper. The paper discusses the mechanism in Subsection 2.3.2 and the cost model in Subsection 3.7, but only prefill latency is measured and reported in Figure 1 (right panel). Decode latency is described qualitatively: "our decode latency is the same as that of a regular Transformer with SWA... Once we have a full batch, we need a step of TTT... our latency for this TTT step is the same as that for prefill." No decode latency numbers, no end-to-end generation timing, and no analysis of the stall pattern are provided.

Mitigation status. The paper does not address this as a limitation or propose mitigation strategies. Potential mitigations include: (1) increasing the TTT mini-batch size during decoding to reduce the frequency of gradient steps (at the cost of stale weights for longer stretches); (2) performing TTT steps asynchronously on a separate device or thread while the primary device continues decoding with frozen weights; (3) using smaller, cheaper gradient computations (e.g., updating fewer layers during decode than during prefill). None of these are discussed.

---

Single Benchmark and Domain: No Evidence of Generalization Beyond Narrative Text

The assumption or constraint. All of the paper's primary evaluations—context scaling (Figure 1), loss breakdown (Figure 6), training compute scaling after fine-tuning (Figure 5b,d), and decoding quality (Figure 7)—use Books [29] as the evaluation dataset. Books is a collection of narrative text (fiction and non-fiction books) with particular statistical properties: long-range thematic coherence, character continuity, relatively predictable narrative structure. Pre-training uses DCLM (filtered Common Crawl), but the only metric reported on DCLM is after pre-training at 8K context (where SWA equals full attention), not after fine-tuning at longer contexts. The paper never evaluates on code, scientific papers, legal documents, multi-turn dialogue, or any domain where the nature of long-range dependencies differs from narrative text.

The consequence. A practitioner considering TTT-E2E for a code assistant (where long contexts arise from large codebases), a legal document analyzer (where precise cross-references matter), or a multi-turn chatbot (where the conversation history is the context) cannot determine from this paper whether the method's advantages transfer. The compression mechanism—gradient steps on next-token prediction—preserves whatever information is most predictive of upcoming tokens. In narrative text, this might mean preserving the current topic, writing style, and active characters. In code, it might need to preserve import statements, function signatures, and variable bindings defined thousands of tokens ago. In legal text, it might need to preserve precise definitions and cross-references. The paper provides no evidence about which types of long-range dependencies survive compression and which are lost, beyond the extreme case of NIAH (where essentially everything is lost). The anecdotal observation that switching from SlimPajama (2023) to DCLM (2024) improved token scaling behavior (Section 3.3) hints at data quality sensitivity, but the effect of domain on TTT-E2E's behavior is entirely unexplored.

What evidence exists in the paper. The evaluation datasets are DCLM (pre-training only, 8K context), Books (fine-tuning and all long-context evaluation), and RULER (NIAH tasks). There is no cross-domain evaluation. The paper does not claim generalization to other domains, but this limitation is not discussed as a explicit boundary either—the title and abstract refer to "long-context language modeling" generically, not "long-context language modeling on narrative text."

Mitigation status. The paper does not address domain generalization. Future work should evaluate TTT-E2E on at minimum: code completion and code understanding tasks (e.g., RepoBench, LongCodeArena), long-document QA (e.g., NarrativeQA, QASPER), and multi-turn dialogue benchmarks. The paper also does not test whether TTT-E2E's pre-trained initialization transfers across domains—if the model is pre-trained with TTT-E2E on DCLM and then fine-tuned on Books, does the meta-learned initialization remain effective when the domain shifts, or would separate meta-learning per domain be necessary?

---

No Evaluation on Downstream Tasks Requiring Understanding of Compressed Context

The assumption or constraint. The paper evaluates TTT-E2E almost exclusively through next-token prediction loss (log perplexity). The sole exceptions are the NIAH tasks (which test exact recall, not understanding) and the limited decoding evaluation with Qwen-8B (which is still a perplexity-like metric, just evaluated by a different model). There is no evaluation on tasks that would validate whether the compressed long-term memory actually captures useful semantic information for downstream applications: summarization of long texts, multi-hop question answering across a long document, entity tracking over long narratives, or reasoning tasks that require synthesizing information from widely separated parts of the context.

The consequence. Perplexity improvements do not always translate to task improvements. A model could achieve lower perplexity by being better at predicting short-range syntactic patterns while losing the ability to track entities or events across long distances—exactly the pattern one might expect from a method that compresses context into MLP weights via gradient descent, since the gradient signal from next-token prediction is dominated by local statistics. For a practitioner who cares about whether their model can answer questions about a 100-page document or summarize a long report, the paper provides no evidence that TTT-E2E's perplexity gains correspond to better performance on these tasks. This is particularly concerning given the NIAH results, which show that TTT-E2E loses the ability to retrieve specific information. It is plausible that the method simultaneously improves language modeling perplexity while degrading performance on tasks that require integrating specific facts from distant parts of the context—the compression may be preserving statistical regularities (style, topic distribution) while discarding the specific entities and events that are most relevant for question answering.

What evidence exists in the paper. None. The paper does not report any task-based evaluation beyond NIAH. The decoding evaluation with Qwen-8B measures the external model's log-likelihood of the generated continuation, which is again a perplexity-like metric. The manual inspection of 20 samples is qualitative and described only as finding them "reasonable."

Mitigation status. The paper does not address this gap or propose task-based evaluations as future work. Given that the paper's core claim is about effective use of long context for language modeling, evaluation on tasks that directly test long-range understanding (e.g., zero-shot long-document QA, entity state tracking, narrative coherence judgment) would substantially strengthen the case that TTT-E2E's compression preserves semantically meaningful information, not just statistical regularities.

---

The Revision Model Correct-to-Incorrect Reversion and the Absence of Dynamic Allocation

The assumption or constraint. This limitation has two connected parts. First, TTT-E2E's inner loop operates with a fixed schedule: process b = 1K tokens, take one gradient step, repeat. There is no mechanism for the model to decide whether a gradient step is needed or how much to update based on the content of the current mini-batch. The model always updates, regardless of whether the current batch contains important information worth compressing or is just procedural filler text. This is analogous to the "difficulty estimation" problem in the reference paper: just as the reference paper's method would benefit from allocating more test-time compute to harder problems, TTT-E2E would benefit from allocating more gradient updates to information-dense parts of the context (e.g., character introductions, plot twists, topic shifts) and fewer to routine transitions.

Second, TTT-E2E has no mechanism to recognize when its compressed memory is becoming unreliable or when important information has been overwritten. The 38% correct-to-incorrect reversion rate from the reference paper's revision model has a conceptual parallel here: as TTT-E2E takes gradient steps on later mini-batches, the updates may partially overwrite information from earlier batches that is still relevant for future predictions. The paper provides no analysis of what information is retained versus forgotten in the updated MLP weights, and no mechanism for the model to "rehearse" important earlier information to prevent catastrophic forgetting during test-time training.

The consequence. For very long contexts (beyond the 128K tested), this fixed-update strategy may become increasingly suboptimal. If the later parts of a 500K-token context contain less information-dense content, TTT-E2E would still take 500 gradient steps, potentially overwriting important early information with updates driven by low-signal tokens. Conversely, if the context contains a crucial piece of information at token 200K that the model needs to retain until token 400K, there is no guarantee that the information survives 200 intervening gradient steps. The paper's context scaling results show flat loss ∆ up to 128K, but this does not guarantee the trend continues to 256K or 1M—the number-of-layers ablation (Figure 4, rightmost panel) shows that insufficient state size causes the scaling curve to degrade, and fixed-capacity compression will eventually hit a limit as context length grows.

What evidence exists in the paper. The number-of-layers ablation (Figure 4, rightmost panel) provides indirect evidence: with only 1 or 3 layers updated (small state), the method fails to scale with context length, suggesting that state capacity is a bottleneck. The window size ablation (Figure 4, leftmost panel) shows that TTT-E2E's advantage over SWA is consistent across window sizes, suggesting that TTT is compressing information beyond the window effectively up to 128K. But there is no direct measurement of information retention—no probing of what the updated MLP weights have learned at different points in the context, and no evaluation of whether performance would degrade if the method were pushed to much longer contexts with the same update schedule.

Mitigation status. The paper does not address dynamic allocation or catastrophic forgetting during test-time training. The fixed schedule (b = 1K, update every batch) is treated as a hyperparameter to be tuned (as in the mini-batch size ablation) rather than a policy to be learned. The dual-MLP design (static MLP for pre-trained knowledge) partially addresses forgetting of pre-training knowledge but does not address forgetting of earlier test-time context within the same sequence. The paper does not explore gating mechanisms, importance-weighted updates, or rehearsal strategies that could improve retention of important context information.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around efficient long-context modeling from architecture design to learning procedure design. The field's dominant research program—invent new sequence mixing layers (linear attention, state space models, gated delta rules, TTT layers with key-value binding) that serve as drop-in replacements for self-attention—is implicitly challenged by the paper's central empirical finding: a standard sliding-window Transformer, augmented with the right test-time and training-time learning procedures, matches full attention's context scaling while every purpose-built efficient architecture (Mamba 2, Gated DeltaNet, TTT-KVB) degrades at length (Figure 1, left panel). The paper's own words are blunt: "architecture design plays a minor, supporting role in our method" (from the b = 8K control in Figure 4, middle panel, where TTT-E2E without TTT achieves loss 2.825 vs. full attention's 2.827).

The magnitude of the shift is not a paradigm overthrow but a reframing. The paper does not argue that architecture doesn't matter—it explicitly uses sliding-window attention as a necessary short-term memory component, and the window size ablation (Figure 4, leftmost panel) shows architecture quality affects all methods. Rather, it argues that the bottleneck for efficient long-context modeling is not finding the right recurrent cell, but aligning two objectives: what the model learns at test time (the inner-loop loss) and how the model is prepared for that learning (the outer-loop loss). This reframing redirects research attention from "what computation graph processes the sequence?" to "what learning problem does the model solve on the context, and how is it initialized to solve it well?"

The double-end-to-end principle provides a diagnostic for the mixed results in the prior TTT literature. Before this paper, the literature on test-time training for sequences was divided between works showing TTT helps for language modeling (TTT-KVB [87, 110], Clark et al. [17]) and works showing that naive test-time training doesn't work (the paper's TTT-naive result in Figure 2, right panel, and the broader dynamic evaluation literature's inconsistent results). The paper reconciles this contradiction by showing that the two approaches are fixing different misalignments: TTT-KVB fixes the training-time alignment (outer loop meta-learns for inner-loop behavior) but breaks the test-time alignment (inner loop optimizes a proxy KVB loss, not the final task); dynamic evaluation fixes the test-time alignment (inner loop optimizes next-token prediction) but breaks the training-time alignment (outer loop optimizes static performance). Neither alone suffices—both must be fixed simultaneously. The stepwise derivation in Table 1 provides quantitative confirmation: fixing the test-time alignment (replacing KVB with next-token prediction) yields a 0.013 loss improvement, the largest single gain among the derivation steps.

Directions that become more attractive. The paper makes three research directions newly salient. First, meta-learning for language model initialization becomes a first-class design dimension rather than a niche technique—the outer loop is not an optional enhancement but a necessary component for test-time training to work at all (TTT-naive performs near the toy baseline in Figure 2). Second, understanding what information survives compression in gradient-based updates becomes a core scientific question—the NIAH results (Table 2) show that TTT-E2E loses exact recall, but we don't know what semantic information it retains, and probing this directly (e.g., through designed interventions on the updated MLP weights) would clarify the method's applicability. Third, memory hierarchy design for sequence models shifts from a hardware concern (cache hierarchies) to an algorithmic one: what is the right division of labor between exact short-term memory (the sliding window) and compressed long-term memory (the TTT-updated weights), and how should these two systems interact?

Directions that become less attractive in light of this paper. The paper's results suggest that further incremental improvements to fixed-size recurrent state architectures (Mamba variants, DeltaNet variants, linear attention variants) may face a fundamental bottleneck: "a fixed-size hidden state has finite capacity, and as the context grows, increasingly important information gets discarded" (as argued in Section 2 of the prior analysis). If TTT-E2E's advantage over Mamba 2 and Gated DeltaNet at 128K is approximately 0.04–0.05 loss ∆ (Figure 1, left panel)—a gap that grows with context length—it suggests that RNN research should focus on increasing effective state capacity (through gradient-based updates, larger states, or learned update rules) rather than refining the recurrent cell design within a fixed state budget. Similarly, the paper's finding that TTT-KVB's multi-head MLPs with LoRA have 5× smaller effective capacity than TTT-E2E's regular MLPs (88M vs. 18M parameters for the 760M model, Subsection 3.7) suggests that the constraint of fitting hidden states onto individual GPU chips is a practical bottleneck that architecture research should address by embracing model parallelism rather than compressing states to fit.

---

Follow-Up Research This Work Enables

1. Training a difficulty predictor to make the inner-loop learning rate adaptive. The paper uses a fixed inner-loop learning rate η for all gradient steps within a sequence. However, the "amount to learn" from each mini-batch is not uniform: a batch containing a topic shift or a new character introduction may benefit from a larger update than a batch of routine transition text. A natural extension is to train a lightweight predictor (or learn a gating function) that modulates η per mini-batch based on the content of that batch—for example, by conditioning on the loss value or gradient norm of the current batch. This is analogous to the difficulty estimation problem in the reference paper, where a compute-optimal policy allocates test-time compute per prompt based on estimated difficulty. The paper's difficulty estimation insight (that difficulty can be predicted from the verifier's score distribution without ground-truth labels) translates directly: the inner-loop loss ℓ_t on the current batch provides a signal for how much the model's predictions are deviating from the context, and this signal could be used to modulate update magnitude. A strong follow-up would sweep inner-loop learning rate schedules (constant, loss-dependent, learned) and measure both final language modeling loss and information retention on synthetic tasks where the "important" tokens are known (e.g., a variant of NIAH where the model needs to retain a fact for a downstream prediction, not retrieve it immediately). The key measurement would be whether adaptive learning rates improve context scaling beyond 128K by preventing important early information from being overwritten by later gradient steps.

2. Combining TTT-E2E with occasional full-attention layers for retrieval capability. The NIAH results (Table 2) show a sharp boundary: TTT-E2E sacrifices essentially all retrieval ability (0.06 accuracy at 128K on S-NIAH-1 vs. full attention's 0.99), while the Hybrid 5:1 baseline (sliding window with occasional full attention) retains substantially more retrieval capability (0.21 at 128K). This suggests a natural combination: use TTT-E2E's compressed long-term memory for language modeling (where it outperforms full attention) while inserting occasional full-attention layers at a low frequency (e.g., every 16th or 32nd layer) to provide a "retrieval backbone" for tasks requiring exact recall. The paper's TTT-E2E already interleaves TTT-updated MLPs with sliding-window attention layers in a standard Transformer block structure—adding a full-attention layer every N blocks would be a straightforward architectural modification. A strong follow-up would measure the trade-off between retrieval accuracy (on RULER or a custom long-document QA task) and inference latency as a function of the full-attention frequency, to determine whether a small number of full-attention layers (e.g., one per 16 blocks, adding minimal latency) can recover most of the retrieval capability while preserving TTT-E2E's language modeling advantage. The paper's memory hierarchy framing predicts that such a hybrid would work well: the full-attention layers provide lossless recall for arbitrary details, while the TTT-updated MLPs compress long-range statistical structure that even full attention doesn't capture efficiently (as suggested by TTT-E2E outperforming full attention even when k = 8K equals the full context length in the window size ablation).

3. Probing what information TTT-E2E's updated MLPs retain versus forget. The paper provides no direct measurement of what the inner-loop gradient steps actually encode in the updated weights. This is a critical gap because the method's design relies on the assumption that next-token prediction loss provides a useful compression of context—but we don't know whether the updated MLPs retain entity identity, syntactic patterns, topic information, factual knowledge, or merely local statistics that happen to reduce perplexity. A strong follow-up would design a suite of probing tasks that intervene on the updated weights: after TTT processes a context containing a specific fact (e.g., "Alice is a doctor living in Paris"), probe the updated MLP weights (through linear classifiers or prompting) to measure whether "Alice," "doctor," and "Paris" are encoded, and at what granularity. The experiment could compare TTT-E2E with different inner-loop learning rates, mini-batch sizes, and numbers of updated layers to map the relationship between hyperparameters and information retention. This would directly address the question: does the inner loop learn semantically meaningful compression, or is it merely adjusting to local statistical patterns? The paper's NIAH results suggest the latter (exact facts are lost), but a more fine-grained probing study could reveal whether certain types of information (entity types, sentiment, narrative structure) are preserved while others (exact strings, numbers) are discarded—which would guide practitioners in choosing when TTT-E2E is appropriate for their use case.

4. Initializing TTT-E2E from a pre-trained standard Transformer to reduce training cost. The paper identifies training latency as a "significant limitation" of the current implementation, noting that TTT-E2E is 3.4× slower than full attention at 8K context length during pre-training (Figure 8, left panel). The proposed mitigation—initializing TTT-E2E from a pre-trained Transformer without TTT, so the expensive meta-learning phase accounts for only a small fraction of total training compute—is described but not tested. This is a high-priority follow-up because it directly addresses the main barrier to adopting TTT-E2E at scale. A strong experiment would: (1) pre-train a standard Transformer (with full attention) on DCLM at 8K context using the paper's basic recipe; (2) convert it to the TTT-E2E architecture (adding the dual MLPs in the last 1/4 of blocks, replacing full attention with sliding-window attention in those blocks); (3) continue training with the TTT-E2E outer loop (meta-learning) for a small number of steps (e.g., 5–10% of total training compute); (4) measure whether the final model achieves the same context scaling as a model trained with TTT-E2E from scratch. The paper notes this technique is "often adopted by prior work on RNNs" [54, 10, 99], and its success would make TTT-E2E practical for large-scale training pipelines where pre-training dominates the compute budget. A negative result—where the initialization from a static Transformer does not support effective test-time training—would reveal that the meta-learned initialization is doing something fundamentally different from standard Transformer weights, which would be an important scientific finding about the nature of the outer loop.

5. Stress-testing TTT-E2E at extreme context lengths (256K–1M tokens). All the paper's context scaling experiments go up to 128K. The loss ∆ curve for TTT-E2E is flat from 8K to 128K (Figure 1, left panel), but the number-of-layers ablation (Figure 4, rightmost panel) shows that with insufficient state size (updating only 1 or 3 layers), the curve degrades at longer contexts. This implies that for some state size and some context length, TTT-E2E's compressed memory will saturate and performance will degrade relative to full attention. The pressing question is: where is that saturation point for the current configuration (3B model, updating last 6 of 24 layers, b = 1K, k = 8K)? A strong follow-up would extend fine-tuning and evaluation to 256K, 512K, and 1M tokens on Books (or another long-text dataset that supports these lengths), measuring both language modeling loss and a retrieval-adjacent task (e.g., question-answering where the answer appears at a known distance from the question) to track where the compressed memory breaks down. This experiment would establish the practical context length ceiling for the current method and inform how hyperparameters (number of updated layers, window size, mini-batch size) should scale with context length. A negative result—where scaling breaks down at 256K even with the optimal hyperparameters—would not invalidate the method but would clarify its regime of applicability and motivate research into larger state capacities or different compression mechanisms.

6. Evaluating TTT-E2E on downstream tasks requiring long-range understanding. The paper's sole evaluation metrics are next-token prediction loss (log perplexity) and NIAH retrieval accuracy. Neither directly measures whether the compressed long-term memory captures semantically useful information for tasks that humans care about: summarizing a long document, answering questions that require synthesizing information from different sections, tracking entity states over a narrative, or reasoning about events separated by thousands of tokens. A strong follow-up would evaluate TTT-E2E fine-tuned models on established long-context benchmarks: ZeroSCROLLS (for summarization and QA on 10K+ token documents), LongBench (for multi-hop QA, summarization, and code understanding at various lengths), and L-Eval (for diverse long-context tasks). The key comparison is not just TTT-E2E vs. full attention, but TTT-E2E vs. the efficient baselines (Mamba 2, Gated DeltaNet, Hybrid 5:1) that it outperforms on perplexity—the critical question is whether the perplexity advantage translates to task performance, or whether the compression mechanism that improves language modeling degrades the specific information retrieval and integration that downstream tasks require. The NIAH results suggest the latter may be true for some tasks, but we don't know the boundary. This evaluation would directly address the paper's central claim: that TTT-E2E "scales with context length in the same way as Transformer with full attention." If the method matches full attention on perplexity but falls behind on QA and summarization, the claim is narrower than the paper suggests and requires qualification.

---

Practical Applications and Downstream Use Cases

1. Long-document language modeling and text generation for narrative content. The paper's primary evaluation on Books, combined with TTT-E2E's 2.7× faster prefill latency than full attention at 128K (Figure 1, right panel) and flat loss ∆ across context lengths (Figure 1, left panel), directly supports deployment for applications that process book-length text. A service that generates chapter-by-chapter summaries of novels, continues partial manuscripts, or analyzes narrative structure across hundreds of pages could use TTT-E2E to process the entire document context at constant per-token cost, rather than chunking the document and losing cross-chapter coherence. The window size ablation (Figure 4, leftmost panel) shows that TTT-E2E with k = 8K outperforms even full attention at 8K context (loss improvement of 0.018), suggesting that TTT-E2E is not merely matching full attention but providing an orthogonal benefit—likely better modeling of statistical patterns beyond the attention window. The practical benefit is that a 3B model with TTT-E2E can process a 128K-token book in ~3 seconds of prefill latency on an H100, while a full-attention model of the same size would take ~8 seconds—a difference that compounds for applications processing many documents.

2. Streaming and real-time sequence processing where latency must be bounded. TTT-E2E's constant inference latency regardless of context length (Figure 1, right panel: the blue line is flat while the orange full-attention line grows linearly) makes it suitable for applications where the model must process an unbounded stream of tokens with a hard latency budget. Examples include: real-time transcription and analysis of live audio streams (meetings, lectures, broadcasts), continuous monitoring of sensor logs or system outputs for anomaly detection, and always-on document co-editing where the model maintains context of the entire editing session. The key constraint these applications share is that the context grows without bound (a meeting can go on for hours, a log stream runs indefinitely), and the model cannot afford to have its per-token latency increase over time. The paper's decode mechanism (Subsection 2.3.2) is particularly well-suited here: the model takes TTT gradient steps only once per b = 1K decoded tokens, so the amortized cost per token remains constant, with periodic, predictable pauses for weight updates rather than a linearly growing cost per token. A practitioner deploying TTT-E2E for streaming would need to characterize the latency distribution (the constant per-token cost plus the periodic stall for each TTT step) and ensure the stall duration is acceptable for their real-time budget.

3. On-device long-context processing where model size is constrained by hardware. The paper's finding that TTT-E2E's hidden state "can be sharded across GPUs using standard tools with no custom kernel" (Section 3.7), combined with the observation that TTT-E2E uses regular MLPs rather than the constrained states of Mamba 2 or Gated DeltaNet (which "must fit their hidden states onto the individual chips inside a GPU"), suggests an advantage for deployment on hardware with limited per-device memory but good interconnect. The paper's 5× larger hidden state (88M vs. 18M parameters for the 760M model) is enabled by model parallelism—a capability that consumer devices with multiple compute units (e.g., phones with NPUs or laptops with integrated GPUs) could leverage. For a practitioner building an on-device assistant that needs to process long conversation histories or large personal documents, TTT-E2E offers a path to effective long-context modeling without requiring the memory capacity to store a full KV cache for 128K+ tokens. The trade-off, as the NIAH results make clear, is that the device won't be able to answer arbitrary factual questions about details buried in the context—but for summarization, continuation, style transfer, and conversation modeling, the compression approach may be sufficient and dramatically more memory-efficient.