Less is More: Recursive Reasoning with Tiny Networks

ArXiv: 2510.04871

🎯 Pitch

A 7M-parameter, 2-layer network achieves 45% on ARC-AGI-1 and 87.4% on Sudoku-Extreme, decisively beating models like DeepSeek R1 and o3-mini that have 10,000× more parameters—and it does so by recursively refining its own answers during inference, not by scaling compute.

1. Executive Summary

This paper introduces Tiny Recursive Model (TRM), a simplified recursive reasoning method that uses a single tiny 2-layer neural network to iteratively refine its predictions by alternating between latent reasoning updates (recursively updating a latent feature z given the current answer y and input x) and answer refinement (producing an improved answer y from the updated latent z), achieving substantially higher generalization than the more complex Hierarchical Reasoning Model (HRM). Evaluated on Sudoku-Extreme, Maze-Hard, ARC-AGI-1, and ARC-AGI-2, TRM with only 7M parameters attains 45% test accuracy on ARC-AGI-1 and 8% on ARC-AGI-2—outperforming most large language models like DeepSeek R1, o3-mini, and Gemini 2.5 Pro with less than 0.01% of their parameters—while on Sudoku-Extreme the improvement is 87.4% vs. HRM's 55%. The key architectural simplifications driving these gains—removing the fixed-point theorem assumption, collapsing two networks into one, replacing self-attention with an MLP for small-context tasks, and using EMA for training stability—establish that deep recursive refinement with backpropagation through the full recursion process succeeds where HRM's 1-step gradient approximation underperforms, though the approach remains limited to supervised learning with deterministic outputs and optimal hyperparameter choices vary by task domain.

2. Context and Motivation

The Core Problem: Learning Complex Reasoning from Tiny Datasets

The fundamental challenge this paper addresses is deceptively simple: how can a neural network learn to solve problems that require many sequential reasoning steps when trained on extremely small datasets? This question becomes acute when we consider that state-of-the-art large language models—trained on trillions of tokens, with hundreds of billions of parameters, and augmented with chain-of-thought prompting and test-time compute scaling—still struggle profoundly on certain puzzle-like reasoning tasks.

The numbers reveal the depth of the problem. On Sudoku puzzles, DeepSeek R1 (671B parameters with chain-of-thought) scores 0.0% accuracy. On ARC-AGI-1, a benchmark specifically designed to require abstract reasoning rather than pattern matching against memorized examples, even the best frontier models prior to the bespoke Grok-4 approach achieve only 15–37% accuracy after six years of intensive effort. On the newer ARC-AGI-2 benchmark—designed to be harder and more resistant to memorization—Gemini 2.5 Pro with extensive test-time compute reaches only 4.9%.

These failures are not merely that the models are "not big enough." The paper's motivating observation is more specific: these tasks involve closed-form reasoning (a Sudoku grid has exactly one correct solution; a maze has a determinable shortest path; an ARC puzzle has a specific transformation rule) where the reasoning process itself—not the memorization of facts or the generation of plausible text—is the bottleneck. An LLM auto-regressively generating tokens "thinks" one token at a time, and a single incorrect token can cascade into an entirely wrong solution grid. The model has no built-in mechanism to iterate on a partial solution, detect errors, and refine its output.

Why This Matters: Beyond Scaling Laws

This problem is important for both practical and theoretical reasons that the paper surfaces explicitly and implicitly:

Practical: efficient deployment without massive infrastructure. The ARC-AGI benchmarks carry monetary prizes precisely because they represent capabilities—abstract reasoning, systematic generalization—that current AI systems largely lack. A system that solves ARC puzzles reliably would demonstrate a qualitatively different kind of intelligence from the statistical pattern matching that dominates LLM performance on most benchmarks. Moreover, solving hard reasoning tasks with a 7M-parameter model (which fits on a single modest GPU) rather than a 671B-parameter model (requiring data-center scale infrastructure) has direct economic and accessibility implications. It raises the prospect of on-device reasoning for edge deployment, real-time applications, or settings where API calls to frontier models are impractical.

Theoretical: the depth generalization challenge. The paper is grappling with a deep learning puzzle that has been recognized since at least the 1990s: how to train networks that effectively implement algorithms requiring more sequential operations than the network has layers. A standard 4-layer transformer, given a Sudoku puzzle, cannot solve it in a single forward pass—the computation requires many sequential logical deductions (filling a cell constrains its row, column, and box, which constrains other cells, recursively). The traditional solution is to make the network deeper, but as the paper observes through its ablation experiments (Table 1), simply adding layers to a transformer trained on 1000 Sudoku examples leads to worse generalization due to overfitting.

This creates a tension: deep reasoning needs depth, but depth requires data to train without overfitting, and the hardest reasoning benchmarks have small training sets by design (ARC-AGI explicitly tests generalization from few examples). Breaking this tension—achieving effective depth without requiring massive training data—is the theoretical contribution the paper is working toward.

Prior Approaches and Where They Fall Short

The paper identifies two families of prior approaches, each with specific limitations.

LLMs with Chain-of-Thought and Test-Time Compute

The dominant approach to reasoning tasks in 2024–2025 is scaling: train ever-larger models on ever-larger datasets, then augment them with chain-of-thought prompting (having the model verbalize intermediate reasoning steps) and test-time compute scaling (generating many candidate answers, selecting via verifiers or majority voting). The paper presents this approach's failures not as an indictment of LLMs generally, but as evidence of a specific gap: auto-regressive token generation, even with CoT, treats reasoning as a sequence generation problem where each token is produced once and never revisited. There is no latent state that the model can iteratively refine without committing to surface-form tokens.

The paper's empirical evidence for this gap is stark: DeepSeek R1, Claude 3.7, and o3-mini-high all score 0.0% on Sudoku and Maze puzzles (Table 4). These are not ambiguous or subjective tasks—they have unambiguous correct answers that can be verified mechanically. The LLMs' failure suggests they are not successfully performing the multi-step constraint propagation that solving these puzzles requires, even when prompted to "think step by step."

A crucial point the paper highlights is that test-time compute as practiced in LLMs (best-of-N, majority voting) is wastefully parallel: it generates many independent complete solutions rather than iteratively refining a single solution. Each candidate is generated from scratch, without access to the reasoning process or intermediate state of other candidates. This is computationally expensive and misses the opportunity to learn from "near misses"—partial solutions that are mostly correct but contain localized errors.

Hierarchical Reasoning Model (HRM): The Immediate Predecessor

Wang et al. (2025) proposed the Hierarchical Reasoning Model, which directly addresses the depth-vs-overfitting tension through two key mechanisms:

1. Recursive reasoning with two networks at different frequencies: A "low-level" network $f_L$ updates a latent feature $z_L$ at high frequency (applied $n = 2$ times per cycle), while a "high-level" network $f_H$ updates $z_H$ at low frequency (applied once per cycle, after every $T = 2$ cycles of $f_L$). The latent $z_H$ decodes to the current answer estimate.

2. Deep supervision: The model reuses its own latents $(z_L, z_H)$ from the previous supervision step as initialization for the next, detaching them from the computational graph to avoid backpropagating through time. This allows the model to progressively refine its answer over up to $N_{\text{sup}} = 16$ steps, emulating a very deep network (384 effective layers by the authors' estimate) without the memory cost of backpropagation through that entire depth.

HRM achieved breakthrough results: 55% on Sudoku-Extreme, 74.5% on Maze-Hard, 40.3% on ARC-AGI-1, and 5.0% on ARC-AGI-2—all with a 27M-parameter model trained on small datasets. This demonstrated that the depth-vs-overfitting tension can be broken through architectural innovation rather than scaling alone.

However, the paper identifies three critical weaknesses in HRM that motivated the TRM design:

1. Dubious theoretical foundation: the implicit function theorem and 1-step gradient approximation. HRM justifies backpropagating through only the last two of six function evaluations per supervision step by invoking the Implicit Function Theorem (IFT) with the 1-step gradient approximation from Bai et al. (2019). The theorem states that if a recurrent function $z^* = f(z^*)$ converges to a fixed point, the gradient can be computed by backpropagating through a single application of $f$ at that equilibrium.

The paper's Section 3.1 identifies serious issues with applying this theorem to HRM. First, HRM does not perform fixed-point iteration—it applies $f_L$ and $f_H$ a fixed small number of times ($n = 2$, $T = 2$) rather than iterating to convergence. Second, the specific pattern of operations (two $f_L$ calls, one $f_H$ call, one more $f_L$ call) does not correspond to iterating any single function to a fixed point for both latents simultaneously. Third, the paper points to evidence from Wang et al.'s own Figure 3, which shows forward residuals over time at a much higher setting ($n = 7$, $T = 7$)—and even there, the $z_H$ residual "is clearly well above 0 at every step" and $z_L$ "only becomes closer to 0 after many cycles, but it remains significantly above 0." The paper argues that at the actual $n = 2$, $T = 2$ used in all HRM experiments, a fixed point is almost certainly not being reached when the 1-step gradient approximation is applied.

This matters because getting the gradient wrong—backpropagating through only the last few steps while ignoring the effect of earlier recursive updates on those steps—means the model receives a noisy or biased training signal about how its recursive reasoning contributes to the final answer. The paper's ablation (Table 1) confirms this empirically: replacing the 1-step gradient approximation with full backpropagation through the recursion process takes TRM from 56.5% to 87.4% on Sudoku-Extreme. The IFT shortcut, which was meant to be a memory-saving trick, was actually severely constraining what the model could learn.

2. Excessive complexity without clear justification. HRM uses two separate 4-layer transformer networks ($f_L$ and $f_H$) motivated by biological arguments about hierarchical processing and temporal frequencies in the brain. The paper argues that this biological framing "makes it incredibly hard to parse out why HRM is designed the way it is" and, combined with the lack of ablation studies in the HRM paper, leaves unanswered the question of why two networks are needed rather than one, or three. The paper's reinterpretation (Section 4.2) is cleaner: $z_H$ is simply the embedded current answer estimate (which decodes to the predicted output), and $z_L$ is a latent reasoning trace that captures intermediate computational state. Under this view, the distinction between the two latents is functional (one represents the output, one represents the reasoning that produced it) rather than hierarchical, and a single network should be capable of updating both given that their roles are signaled by whether the input $x$ is included in their respective inputs.

3. Inefficient adaptive computation. HRM's Adaptive Computation Time (ACT) mechanism uses Q-learning to decide when to stop iterating on a training example and move to the next one. The paper reveals—drawing from HRM's code rather than its paper—that the Q-learning objective requires an additional forward pass through the full HRM to compute the "continue" loss. This means each optimization step involves two forward passes, doubling the computational cost. While ACT successfully reduces the average number of supervision steps per example (below 2 on Sudoku-Extreme), the per-step cost remains elevated.

Why Direct Prediction Fails

The paper also reports that a standard "direct prediction" model—a single-forward-pass 27M-parameter transformer trained on the same data—scores 0.0% on Sudoku-Extreme, 0.0% on Maze-Hard, 21.0% on ARC-AGI-1, and 0.0% on ARC-AGI-2. This baseline establishes that the problem is not the training data quality or the model size per se, but the architecture's inability to perform multi-step reasoning through iterative refinement. A model that produces its answer in one shot, with no mechanism to detect and correct errors, cannot handle problems requiring sequential logical deduction beyond what a single forward pass through 4 layers can express.

How This Paper Positions Itself

The paper positions TRM as a simplification and strengthening of HRM rather than a radical departure. It accepts HRM's core insight—that recursive refinement with deep supervision can emulate deep reasoning without overfitting—but argues that HRM's specific implementation choices (two networks, biological justification, IFT-based gradient approximation, Q-learning with two forward passes) are unnecessarily complex and, in the case of the gradient approximation, actively harmful.

The paper's framing is explicitly one of reducing complexity while improving performance. The title "Less is More" signals this: the paper argues that removing architectural components (collapsing two networks into one, reducing from 4 layers to 2, replacing self-attention with an MLP on small contexts) and theoretical overhead (no IFT, no fixed-point assumptions) produces a model that not only generalizes better but is simpler to understand and implement. The ablation table (Table 1) is central to this argument: each simplification (single network, removing 1-step gradient, EMA, attention-free architecture) is evaluated independently, and the cumulative effect is the jump from 55% to 87.4% on Sudoku-Extreme.

The paper also positions itself within a broader research direction that questions the necessity of scale for reasoning. By achieving 45% on ARC-AGI-1 with 7M parameters—compared to 34.5% for o3-mini-high and 37% for Gemini 2.5 Pro—the paper provides evidence that architectural innovation can compensate for massive differences in parameter count and training data scale on certain structured reasoning tasks. This is not presented as a refutation of scaling but as evidence that "when data is too scarce and model size is large, there can be an over-optimization penalty" (referencing Kaplan et al., 2020), and that recursive architectures with deep supervision are a more parameter-efficient way to achieve the effective depth these tasks require.

Finally, the paper implicitly positions TRM within the literature on temporal credit assignment without backpropagation through time, acknowledging that HRM's deep supervision + 1-step gradient approach was itself an attempt to solve this problem. TRM's solution—backpropagating through the full recursion process but keeping it tractable by making the network tiny (2 layers)—is a pragmatic middle ground: it accepts the memory cost of BPTT through the recursion in exchange for eliminating the bias of the approximation, and compensates for the resulting memory constraints by radically reducing model size.

3. Technical Approach

3.1 Reader orientation

Tiny Recursive Model (TRM) is a supervised learning architecture that uses a single tiny 2-layer neural network to solve hard reasoning tasks by alternating between recursively updating a latent reasoning state and refining its predicted answer, repeating this cycle over multiple supervision steps. The system solves the problem of learning complex multi-step reasoning from extremely small datasets (around 1000 training examples) by emulating a very deep network through repeated application of a shallow network, where backpropagation flows through the full recursive cycle at each supervision step, allowing the model to learn how its iterative reasoning contributes to the final answer.

3.2 Big-picture architecture (diagram in words)

The TRM system has five major components that operate in a nested loop structure:

1. Input embedding $f_I$ — transforms the raw input tokens (a Sudoku grid, a maze layout, or ARC puzzle grids) into a continuous representation $x$ of shape $[B, L, D]$ where $B$ is batch size, $L$ is context length, and $D = 512$ is the embedding dimension.

2. Single recursive network $f$ — a tiny 2-layer transformer (or optionally an MLP-Mixer variant) that plays the role of both $f_L$ and $f_H$ from HRM. It takes three inputs — the embedded question $x$, the current answer embedding $y$, and the latent reasoning state $z$ — and can operate in two modes depending on its inputs: when $x$ is included, it updates $z$ (latent reasoning); when $x$ is excluded, it updates $y$ (answer refinement). This single network replaces HRM's two separate 4-layer networks.

3. Latent recursion block — applies $n = 6$ recursive updates to $z$ (conditioned on $x$, the current $y$, and the evolving $z$), followed by one update to $y$ (conditioned on the refined $z$ and the previous $y$, but not $x$). This $n+1 = 7$-step cycle constitutes one "full recursion process," and backpropagation flows through all 7 steps. In the paper's reinterpretation, $y$ (called $z_H$ in HRM) is the embedded current solution estimate that can be decoded to a predicted answer grid, while $z$ (called $z_L$ in HRM) is a latent reasoning trace that does not directly correspond to a valid output but captures the intermediate computational state.

4. Deep supervision loop — reuses $(y, z)$ from the previous supervision step as initialization for the next, running up to $N_{\text{sup}} = 16$ supervision steps. Before each supervision step that carries gradients, $T - 1 = 2$ full recursion processes are run without gradient tracking (inside torch.no_grad()) to improve $(y, z)$ without memory cost, followed by one recursion process WITH gradient tracking. This provides the model with 3 recursion processes per supervision step ($T = 3$), totaling $T(n + 1) \cdot n_{\text{layers}} = 3 \cdot 7 \cdot 2 = 42$ effective layers per supervision step, and $42 \cdot 16 = 672$ effective layers over the full deep supervision chain.

5. Output head and halting mechanism — the output head $f_O$ decodes $y$ to logits over the output vocabulary, and argmax produces the predicted answer grid. A separate Q-head $f_Q$ takes $y$ as input and outputs a single scalar representing the probability that the current answer is correct, trained with a Binary Cross-Entropy loss against a target of 1 if the answer matches the ground truth and 0 otherwise. If $f_Q(y) > 0$ at any supervision step, the model halts early and moves to the next training example.

Information flows as follows: a training example enters → the input is embedded into $x$ → $y$ and $z$ are initialized as learned embeddings → for each of up to 16 supervision steps: (a) run 2 no-gradient recursion processes to refine $(y, z)$, (b) run 1 gradient-tracked recursion process, (c) compute cross-entropy loss on the decoded answer, (d) compute BCE loss on the halting probability, (e) if halted, break to next example; otherwise detach $(y, z)$ and continue → backpropagate through all gradient-tracked operations → update weights with AdamW → apply Exponential Moving Average of weights.

3.3 Roadmap for the deep dive

• First, the core recursion mechanism — how a single network updates $z$ and $y$ in sequence, why the input $x$ is included only in the $z$ updates, and what each latent represents. This is the foundational operation that everything else builds on.

• Second, the deep supervision framework — how multiple recursion processes are chained across supervision steps, the $T-1$ no-gradient + 1 gradient pattern, and the effective depth this achieves. This addresses temporal credit assignment and explains why backpropagation flows through only the last recursion cycle per supervision step.

• Third, the simplified Adaptive Computation Time (ACT) mechanism — how halting decisions are made using a single Q-head and BCE loss, eliminating HRM's second forward pass while still allowing early termination during training.

• Fourth, the network architecture choices — the 2-layer design, the attention-free MLP-Mixer variant for small-context tasks, the single-network-vs-two-networks ablation, and the EMA stabilization technique. These are the "less is more" innovations that reduce parameter count while improving generalization.

• Fifth, the loss functions and optimization setup — the cross-entropy for answer prediction, the BCE for halting, the stable-max loss variant, and the AdamW configuration with EMA, presented with all hyperparameters.

• Sixth, the data augmentation and evaluation protocols — how tiny datasets are expanded via shuffling, dihedral transformations, and color permutations, and how test-time predictions are aggregated (e.g., the 1000-augmentation voting for ARC-AGI).

3.4 Detailed, sentence-based technical breakdown

This is primarily an architectural innovation paper whose core idea is that deep recursive refinement — applying a tiny network many times with backpropagation through the full recursion cycle — achieves better generalization on hard reasoning tasks than complex multi-network architectures with approximate gradient shortcuts, especially when training data is scarce.

The Core Recursion Mechanism: How $z$ and $y$ Are Updated

The fundamental operation in TRM is a single cycle of recursive updates applied by a shared network $f$ parameterized by $\theta$. The network takes three inputs — the embedded question representation $x$, the current answer embedding $y$, and the latent reasoning state $z$ — and performs two distinct types of updates in a fixed sequence.

Latent reasoning updates (updating $z$): For $n = 6$ iterations, the network updates the latent reasoning state by conditioning on all three inputs:

$z \leftarrow f(x + y + z; \theta)$

where $x$ is the embedded input question (shape $[B, L, D]$), $y$ is the embedded current answer estimate (same shape), and $z$ is the latent reasoning state (same shape).

What it computes: At each of the $n = 6$ steps, the network receives the sum of all three signals — the problem specification, the current answer guess, and the reasoning trace so far — and produces an updated reasoning trace $z$. Because $x$ is included, the network has access to the problem constraints (e.g., Sudoku rules, maze layout, ARC pattern demonstrations) and can use them to guide the reasoning update.

Why this form: Including $x$ in the input signals to the network that the task at this step is reasoning about the problem given the current state, not yet producing a final answer. The summation $x + y + z$ (rather than concatenation) is a design choice carried over from HRM that keeps the input dimensionality fixed at $D = 512$ regardless of how many signals are combined, and it forces the network to learn an additive interaction between the question, answer, and reasoning trace. An alternative would be concatenation along the feature dimension, which would increase dimensionality and parameter count — precisely what the "less is more" philosophy seeks to avoid.

Answer refinement update (updating $y$): After the $n = 6$ latent reasoning updates, the network performs one additional forward pass to update the answer estimate:

$y \leftarrow f(y + z; \theta)$

What it computes: The network receives only the previous answer $y$ and the final reasoning trace $z$ (after $n$ recursive refinements), but crucially not the question embedding $x$. It produces an improved answer embedding $y$.

Why $x$ is excluded here but included in $z$ updates: The paper argues that this input signature directly specifies the task: when $x$ is present, the network's job is to reason (update $z$); when $x$ is absent, the network's job is to produce an answer from the reasoning (update $y$). This task disambiguation through input presence/absence is what allows a single network to serve both roles, replacing HRM's two separate networks. If $x$ were included in the $y$ update as well, the network would have no signal distinguishing "refine the reasoning" from "produce the final answer," potentially causing it to continue reasoning when it should instead commit to an answer.

The full recursion process: One complete recursion process (called latent_recursion in the pseudocode, Algorithm 3) consists of $n$ updates to $z$ followed by 1 update to $y$, for a total of $n + 1 = 7$ network evaluations. During a gradient-tracked recursion process, backpropagation flows through all 7 evaluations, meaning the network learns how each of the 6 reasoning steps and the final answer step contribute to the quality of $y$. This is in direct contrast to HRM, where only the last 2 of 6 evaluations carry gradients (under the IFT approximation), leaving the model blind to how early reasoning steps affect the final answer.

The reinterpretation of latents: The paper provides a concrete example (Figure 6) showing a Sudoku instance where $y$ (denoted $z_H$ in HRM) decodes to a nearly-correct Sudoku grid (one cell wrong: the bottom-right corner shows 5 instead of the correct 7), while $z$ (denoted $z_L$ in HRM) decodes to an unintelligible jumble of tokens. This empirically validates the interpretation: $y$ is the current solution estimate (decodeable via the output head to a meaningful answer), while $z$ is an opaque reasoning trace that captures intermediate computation but only becomes meaningful when transformed into $y$ by the network.

Deep Supervision: Chaining Recursion Processes Across Supervision Steps

The core recursion process operates within a larger outer loop called deep supervision, which allows the model to progressively refine its answer over up to $N_{\text{sup}} = 16$ steps. The key design question is: how many recursion processes should run per supervision step, and which should carry gradients?

The $T-1$ no-gradient + 1 gradient pattern: For each supervision step, TRM runs $T = 3$ full recursion processes. The first $T - 1 = 2$ are executed inside a torch.no_grad() context, meaning their computations are performed but not tracked for backpropagation. The third (final) recursion process is executed with gradient tracking enabled. In pseudocode from Algorithm 3:

with torch.no_grad():
    for j in range(T-1):
        y, z = latent_recursion(x, y, z, n)
y, z = latent_recursion(x, y, z, n)

What this accomplishes: The no-gradient recursion processes serve as free iterative improvement — they allow $(y, z)$ to move closer to a good solution without incurring memory cost for backpropagation, exactly as HRM's detached forward passes did, but here with the full $n+1 = 7$ step cycle rather than HRM's partial 4-step detached sequence. The final gradient-tracked recursion process then learns to take whatever state $(y, z)$ the no-gradient processes produced and improve it further. After backpropagation, $(y, z)$ are detached from the computational graph (.detach()) and become the initialization for the next supervision step.

Why $T = 3$: The paper experimented with different values of $T$ and $n$ (Table 3) and found $T = 3, n = 6$ to be optimal for Sudoku-Extreme at 87.4% accuracy. Lower values ($T = 1, n = 1$ at 63.2%, $T = 2, n = 2$ at 81.9%) underperform due to insufficient refinement before the gradient step. Higher values ($T = 4, n = 4$ at 84.2%, $T = 3, n = 6$ extended variants) either overfit or exceed memory limits. Crucially, increasing $n$ too far leads to Out-Of-Memory (OOM) errors because TRM backpropagates through all $n+1$ steps, unlike HRM which only backpropagates through 2 steps regardless of $n$. The memory cost of TRM scales linearly with $n$, making $n = 6$ the practical maximum under the authors' hardware constraints.

Effective depth calculation: The paper estimates that each supervision step applies $T(n + 1) = 3 \cdot 7 = 21$ network evaluations with a 2-layer network, yielding $21 \cdot 2 = 42$ effective layers per supervision step. Across the maximum $N_{\text{sup}} = 16$ supervision steps, this totals $42 \cdot 16 = 672$ effective layers. Compare to HRM: $T(n + 1) \cdot n_{\text{layers}} = 2 \cdot 3 \cdot 4 = 24$ effective layers per supervision step, for $24 \cdot 16 = 384$ effective layers total. TRM achieves deeper effective computation (672 vs. 384 layers) despite using half the layers per network (2 vs. 4), because it uses more recursion cycles ($T = 3, n = 6$ vs. $T = 2, n = 2$) and backpropagates through all of them.

Why deep supervision replaces BPTT: The paper explains that deep supervision combined with the $T-1$ no-gradient + 1 gradient pattern provides an alternative to Backpropagation Through Time (BPTT) for solving the temporal credit assignment problem — teaching the network how its early decisions affect its later performance — without the memory cost of unrolling the entire depth through time. The model learns to take a partially-refined state $(y, z)$ and improve it, and because the gradient flows through the final recursion process, the network receives a training signal about how its recursive operations contribute to answer quality. The no-gradient processes ensure that the state entering the gradient-tracked process is already reasonably good, so the model learns refinement rather than having to simultaneously learn initial reasoning AND refinement in one gradient step.

What happens across supervision steps: At supervision step 1, $(y, z)$ are initialized from learned embedding vectors (separate from the input embedding). The model runs 3 recursion processes (2 no-gradient, 1 gradient), produces a predicted answer, computes loss, and detaches $(y, z)$. At supervision step 2, the detached $(y, z)$ — which now encode the model's best guess after step 1 — are used as initialization for another round of 3 recursion processes, producing a (hopefully) improved answer. This continues until either the model halts (if the Q-head predicts correctness with probability > 0) or $N_{\text{sup}} = 16$ steps are reached. At test time, ACT is disabled and all 16 supervision steps are run, with the final answer taken from the last step.

What happens to the gradient across supervision step boundaries: The .detach() operation between supervision steps means that the computational graph is broken — gradients from supervision step $k+1$ do not flow back to the operations in supervision step $k$. This is intentional: it prevents the memory explosion of full BPTT across 16 × 21 = 336 network evaluations, and it creates a curriculum-like training signal where each supervision step learns "improve the answer from whatever state you're given" rather than "construct the entire reasoning chain from scratch." This is the same principle as the 1-step gradient approximation in HRM, but applied at the coarser granularity of supervision steps rather than individual network evaluations within a recursion.

Simplified Adaptive Computation Time (ACT): Halting Without a Second Forward Pass

During training, running all $N_{\text{sup}} = 16$ supervision steps for every example in every batch is computationally wasteful — many examples converge to the correct answer early, and spending additional compute on them provides no learning signal while delaying exposure to new examples. HRM addressed this with a Q-learning-based ACT mechanism that required two forward passes per optimization step. TRM simplifies this to a single forward pass with a binary classification objective.

The Q-head architecture: An additional small network $f_Q$ (the "Q-head") takes the answer embedding $y$ as input and outputs a single scalar:

$\hat{q} = f_Q(y; \phi)$

where $\phi$ are the parameters of the Q-head (separate from the main network $\theta$), and $\hat{q} \in [0, 1]$ after sigmoid activation represents the model's estimated probability that the current answer $y$ (when decoded) matches the ground truth $y_{\text{true}}$.

The halting loss: The Q-head is trained with a simple Binary Cross-Entropy loss:

$\mathcal{L}_{\text{halt}} = -\left[\mathbb{1}_{y_{\text{pred}} = y_{\text{true}}} \log(\hat{q}) + \mathbb{1}_{y_{\text{pred}} \neq y_{\text{true}}} \log(1 - \hat{q})\right]$

where $y_{\text{pred}} = \arg\max(f_O(y))$ is the decoded answer, $y_{\text{true}}$ is the ground truth, and $\hat{q}$ is the Q-head's predicted probability of correctness.

What it computes: This is a standard binary cross-entropy that encourages the Q-head to output a high probability when the predicted answer matches the ground truth, and a low probability when it does not. The target is constructed from the actual correctness of the current answer — no "what if" counterfactual required.

Why this eliminates the second forward pass: HRM's Q-learning objective required a "continue loss" that asked: "would halting now have been better than continuing to the next step?" Computing this required running an additional forward pass (all 7 steps of the recursion process) to see what answer the model would have produced if it had continued, then comparing the two outcomes. TRM's simplified loss only asks: "is the current answer correct?" which is answerable from the current forward pass alone. The paper reports that this simplification causes "no significant difference in generalization" (Table 1: 87.4% without the extra pass vs. 86.1% with HRM-style ACT), while halving the computational cost per optimization step.

The halting decision: During training, after computing $\hat{q}$ and the total loss, the model checks:

if q_hat > 0:
    break

If the predicted probability of correctness exceeds 0 (i.e., any positive value after sigmoid, which is always the case unless $\hat{q} = 0$ exactly), the deep supervision loop terminates for that example and training proceeds to the next example in the batch. In practice, this means the model halts as soon as it believes the answer is correct, which the paper reports leads to an average of less than 2 supervision steps per example on Sudoku-Extreme (matching HRM's efficiency with ACT).

What happens at test time: ACT is disabled — the Q-head is ignored, and all $N_{\text{sup}} = 16$ supervision steps are executed. The final answer is taken from the output head at step 16. The Q-head exists only to make training more efficient by reducing time spent on already-solved examples; it does not constrain the model's test-time capacity.

Network Architecture: The "Less is More" Design Choices

The network $f$ at the heart of TRM embodies a series of deliberate simplifications relative to HRM, each of which the paper ablates (Table 1) and justifies through improved generalization on small-data regimes.

Two layers instead of four: The base architecture for $f$ uses only $n_{\text{layers}} = 2$ transformer layers, compared to HRM's 4 layers per network (8 layers total across the two networks). The paper reports a counterintuitive finding: "adding layers decreased generalization due to overfitting" (Section 4.4). Specifically, a 4-layer TRM with $n = 3$ recursions (keeping effective depth approximately constant at 48 layers per supervision step) achieves 79.5% on Sudoku-Extreme, while the 2-layer TRM with $n = 6$ (42 effective layers) achieves 87.4%. The deeper-per-application network overfits despite similar total effective depth, consistent with the observation that "when data is too scarce and model size is large, there can be an overfitting penalty" (citing Kaplan et al., 2020). The paper's strategy is to push depth into the recursion count $n$ (where parameters are reused) rather than into the layer count $n_{\text{layers}}$ (where parameters are added), maximizing the ratio of effective depth to parameter count.

Single network instead of two: HRM uses two separate 4-layer transformer networks — $f_L$ (applied $n$ times per cycle to update $z_L$) and $f_H$ (applied once per cycle to update $z_H$) — for a total of $2 \cdot 4 \cdot D^2$-scale parameters dedicated to the recursive modules. TRM collapses both roles into a single 2-layer network $f$, which the paper justifies through the observation that the task distinction (reasoning vs. answer production) is already encoded in the input signature: $f(x + y + z)$ for reasoning, $f(y + z)$ for answer refinement. The ablation confirms this: a TRM variant with separate $f_L$ and $f_H$ networks (matching HRM's two-network design but with TRM's full backpropagation) achieves 82.4% on Sudoku-Extreme vs. 87.4% for the single-network TRM. The single network not only halves the parameter count (5M vs. 10M for the two-network variant on Sudoku) but generalizes better, likely because sharing parameters between reasoning and answer refinement acts as a regularizer in the small-data regime.

Transformer layer components: Each of the 2 layers is a standard transformer block consisting of:

• RMSNorm (Zhang & Sennrich, 2019) for pre-layer normalization — more stable than LayerNorm for small networks
• No bias terms (Chowdhery et al., 2023) — reducing parameter count without affecting representational capacity
• Rotary positional embeddings (RoPE, Su et al., 2024) — encoding position information without learnable parameters
• SwiGLU activation (Hendrycks & Gimpel, 2016; Shazeer, 2020) in the feed-forward sublayer — a gated linear unit variant that the paper finds effective but does not ablate (a failed experiment with Mixture-of-Experts SwiGLU "decrease[d] massively")

The embedding dimension is $D = 512$ throughout, and the feed-forward expansion factor is implicit in the SwiGLU design (typically $4 \times D$ or $\frac{8}{3} \times D$ for SwiGLU, though the paper does not specify the exact expansion factor).

Attention-free architecture for small-context tasks (TRM-MLP): For tasks where the context length $L$ is small and fixed (specifically Sudoku 9×9 grids with $L = 81$), the paper replaces the self-attention sublayer with a multilayer perceptron (MLP) applied along the sequence dimension, inspired by the MLP-Mixer architecture (Tolstikhin et al., 2021). This is a design choice based on a cost-benefit analysis: "when focusing on tasks where $L \leq D$, a linear layer is cheap, requiring only a matrix of $[L, L]$ parameters." On Sudoku-Extreme, TRM-MLP achieves 87.4% accuracy with only 5M parameters, compared to 74.7% for TRM-Att (with self-attention) at 7M parameters. However, on tasks with larger context lengths — Maze-Hard (30×30 grid, $L = 900$) and ARC-AGI (up to 30×30 grids with multiple input-output pairs) — TRM-MLP performs poorly (0.0% on Maze-Hard, 29.6% on ARC-AGI-1) because the $[L, L]$ MLP weight matrix becomes $900 \times 900$, which is both parameter-intensive and lacks the inductive bias that nearby cells interact more than distant ones (an inductive bias self-attention with positional encodings captures naturally). The paper's recommendation is task-dependent: use TRM-MLP when $L$ is small and fixed; use TRM-Att when $L$ is large or variable.

Exponential Moving Average (EMA) of weights: During training, TRM maintains an exponential moving average of the model parameters with decay rate $\beta_{\text{EMA}} = 0.999$. After each optimizer step updates the "online" parameters $\theta$, the EMA parameters $\theta_{\text{EMA}}$ are updated as:

$\theta_{\text{EMA}} \leftarrow \beta_{\text{EMA}} \cdot \theta_{\text{EMA}} + (1 - \beta_{\text{EMA}}) \cdot \theta$

What this accomplishes: EMA smooths the parameter trajectory over training steps, reducing the variance from individual minibatch updates. The paper reports this as critical for stability on small datasets: without EMA, Sudoku-Extreme accuracy drops from 87.4% to 79.9% (Table 1). The paper notes that "on small data, HRM tends to overfit quickly and then diverge" and that EMA "prevents sharp collapse." This is a standard technique from GAN training (Brock et al., 2018) and diffusion models (Song & Ermon, 2020), applied here to the small-data supervised learning setting. At test time, the EMA parameters $\theta_{\text{EMA}}$ are used rather than the raw training parameters.

Why MoE failed (negative result): The paper attempted to increase capacity without increasing layer count by replacing the SwiGLU MLP blocks with SwiGLU Mixture-of-Experts layers (Shazeer et al., 2017; Fedus et al., 2022). This "decrease[d] massively" in generalization. The paper's interpretation: "MoEs clearly add too much unnecessary capacity, just like increasing the number of layers does" — reinforcing the central thesis that on small datasets, capacity is the enemy, not the bottleneck.

Why weight tying failed (negative result): Tying the input embedding matrix and output head projection (a common technique in language models to reduce parameters) "was too constraining and led to a massive generalization drop." The paper does not elaborate on the mechanism, but it is plausible that the input embedding and output decoding require different representational geometries — the input must encode puzzle constraints in a way useful for reasoning, while the output must map from the answer embedding to precise token predictions — and forcing them to share weights constrains the model's ability to learn both mappings.

Loss Functions and Optimization

TRM is trained with two loss components summed together, optimized with AdamW, and stabilized with the stable-max loss variant.

Answer prediction loss (supervised learning objective): The primary loss is the softmax cross-entropy between the predicted answer logits and the ground truth answer:

$\mathcal{L}_{\text{answer}} = -\sum_{l=1}^{L} \sum_{c=1}^{C} y_{\text{true}, l, c} \log\left(\frac{\exp(\text{logit}_{l, c})}{\sum_{c'=1}^{C} \exp(\text{logit}_{l, c'})}\right)$

where $L$ is the sequence length (number of output tokens), $C$ is the vocabulary size, $y_{\text{true}, l, c}$ is a one-hot indicator of the correct token at position $l$, and $\text{logit}_{l, c}$ is the output head's pre-softmax score for token $c$ at position $l$.

What it computes: For each position in the output sequence, the model predicts a probability distribution over the vocabulary; the loss penalizes deviations from the ground truth token. This is summed over all positions. For a Sudoku grid, $L = 81$ (one token per cell) and $C$ includes the digits 1–9 plus empty-cell tokens; for Maze, $L = 900$ and $C$ includes path/empty/wall tokens; for ARC-AGI, $L$ varies by puzzle and $C$ includes 10 colors plus background.

Why cross-entropy: This is the standard maximum-likelihood objective for classification, appropriate because the output is a deterministic function of the input (there is exactly one correct Sudoku grid, one shortest path, one correct ARC output). The paper uses the stable-max loss variant (Prieto et al., 2025) for "improved stability," which modifies the softmax computation to avoid numerical overflow/underflow through a stabilized log-sum-exp implementation, though the mathematical form is equivalent to standard cross-entropy.

Halting loss: As described above, the Q-head is trained with BCE against the binary indicator of answer correctness. The total loss for a supervision step is:

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{answer}} + \mathcal{L}_{\text{halt}}$

where both losses are equally weighted (implicitly, since no weighting coefficient is specified).

What happens across supervision steps: The total loss is accumulated across all supervision steps that execute before halting (or all 16 steps if no halt occurs). Then loss.backward() is called, which backpropagates through all gradient-tracked recursion processes across all executed supervision steps. The optimizer then updates all parameters (input embedding, network $f$, output head, Q-head) jointly.

Optimizer configuration: The paper uses AdamW (Loshchilov & Hutter, 2017; Kingma & Ba, 2014) with the following hyperparameters, quoted verbatim:

• $\beta_1 = 0.9$, $\beta_2 = 0.95$ (standard for transformer training)
• Learning rate: $1 \times 10^{-4}$ for Sudoku-Extreme and Maze-Hard; $1 \times 10^{-4}$ for network parameters but $1 \times 10^{-2}$ for the embeddings on ARC-AGI
• Weight decay: 1.0 for Sudoku-Extreme and Maze-Hard; 0.1 for ARC-AGI
• Batch size: 768
• Learning rate warmup: 2,000 iterations (linear warmup from 0 to the target learning rate)
• Training duration: 60,000 epochs for Sudoku-Extreme and Maze-Hard; 100,000 epochs for ARC-AGI

Why different learning rates for embeddings on ARC-AGI: The 10× higher learning rate for embeddings ($1 \times 10^{-2}$ vs. $1 \times 10^{-4}$) is not explicitly justified, but it likely reflects the higher variance of embedding gradients when the vocabulary is small (10 colors) and the embeddings must rapidly adapt to represent puzzle-specific color mappings that change across tasks. A standard transformer practice is to use higher learning rates for input/output embeddings than for transformer layers, as embeddings are farther from the output and receive smaller gradient magnitudes.

Why different weight decay values: Strong weight decay (1.0) on the smaller datasets (Sudoku-Extreme, Maze-Hard with 1K training examples) provides aggressive regularization to prevent overfitting. Weaker weight decay (0.1) on the larger ARC-AGI dataset (800+ tasks, augmented to ~1M examples) reflects the reduced overfitting risk. The paper does not ablate these choices.

Hardware: Sudoku-Extreme experiments used a single L40S GPU (40GB RAM) for < 36 hours. Maze-Hard used 4× L40S for < 24 hours. ARC-AGI used 4× H100 GPUs (80GB RAM each) for approximately 3 days.

Data Augmentation and Evaluation Protocols

The paper deals with extremely small training sets by applying task-specific heavy data augmentation, effectively multiplying the dataset size by 8 to 1000×.

Sudoku-Extreme: The training set contains only 1,000 Sudoku puzzles. Each puzzle is augmented 1,000 times through shuffling operations that preserve Sudoku constraints — specifically, row permutations within bands, column permutations within stacks, band permutations, stack permutations, and digit relabeling. These transformations do not change the logical structure of the puzzle (a permuted Sudoku has the same solution up to the same permutations), but they create superficially different input-output pairs. Note that the paper says "1000 shuffling (done without breaking the Sudoku rules) augmentations per data example," but this is not conventional data augmentation that produces varied training examples — it produces the same puzzle in different permutations. The model must learn the underlying solving algorithm, not memorize specific grid patterns, because the 423K test examples include unseen puzzle configurations (423× the training set size).

Maze-Hard: The training set of 1,000 mazes is augmented with 8 dihedral transformations per maze — the 8 symmetries of the square (4 rotations × 2 reflections). This is a standard geometric data augmentation that teaches the model invariance to the maze's absolute orientation. Unlike Sudoku, maze solutions (the shortest path) are not invariant under all dihedral transformations — a reflected maze has a reflected shortest path — so the augmentation must transform both input and output consistently.

ARC-AGI: The training set (ARC-AGI-1 training tasks + ConceptARC tasks) is augmented 1,000 times per puzzle. Each augmentation applies three transformations:

1. Color permutation: randomly reassign the 10 colors in the puzzle to different color indices. Since ARC tasks are defined by abstract patterns rather than specific colors (e.g., "fill the shape with the same color as its border"), color permutation preserves the task structure while preventing the model from memorizing color-specific heuristics.
2. Dihedral-group transformations: random 90° rotations, horizontal/vertical flips, and reflections (same 8 symmetries as Maze-Hard). Since ARC tasks often involve spatial relationships, these augmentations teach invariance to absolute orientation while preserving relative spatial structure.
3. Translation transformations: shift the grid contents within the 30×30 canvas. This prevents the model from developing position-specific biases (e.g., "objects in the top-left corner tend to be...") and forces it to attend to relative positions.

Test-time aggregation for ARC-AGI: At test time, each ARC puzzle (comprising 2–3 demonstration input-output pairs and 1–2 test inputs) is presented to the model with all 1,000 augmentations. The model produces 1,000 candidate output grids (one per augmentation), and the most common answer (majority vote) is reported. This is analogous to test-time compute scaling via best-of-N sampling, but using data augmentation rather than stochastic decoding (since TRM produces deterministic outputs for a given input). The paper notes that each puzzle is given a "specific embedding of shape $[0, 1, D]$" to distinguish different tasks within a batch, which is added to the input embedding $x$.

Why majority voting over augmentations works: If the model has learned the underlying transformation rule (e.g., "copy the shape and fill it with blue"), it will produce the correct output for most augmentations, and the incorrect outputs from augmentations that confuse the model (e.g., rare color permutations that create accidental symmetries) will be outvoted. This leverages the same principle as test-time compute scaling — aggregate over multiple independent (or quasi-independent) attempts — but without requiring the model to generate multiple stochastic samples. The tradeoff is that the model must be evaluated 1,000 times per test input, which is computationally expensive but tractable for the ARC-AGI public evaluation set (hundreds of test inputs).

Direct prediction baseline: The paper reports a "Direct pred" baseline with 27M parameters that performs a single forward pass (no recursion, no deep supervision). On Sudoku-Extreme and Maze-Hard, this achieves 0.0% — demonstrating that the problem is not solvable by a single-pass architecture of comparable size, and that the recursive refinement is essential. On ARC-AGI-1, direct prediction achieves 21.0%, showing that some ARC puzzles are solvable in a single forward pass (likely the simpler pattern-completion tasks), but the majority require iterative reasoning. On ARC-AGI-2, direct prediction achieves 0.0%, consistent with ARC-AGI-2 being designed to be harder and more resistant to single-pass pattern matching.

Test-accuracy metric: The paper reports exact-match accuracy: the fraction of test examples for which every token of the predicted output matches the ground truth. For Sudoku, this means all 81 cells must be correct; for Maze, all 900 cells; for ARC-AGI, the full output grid must match exactly (with two attempts allowed per puzzle, matching the official ARC evaluation protocol). This is a strict metric — a Sudoku with 80 correct cells and 1 error counts as a failure — which makes the reported accuracy numbers (87.4% on 423K Sudoku puzzles) more impressive than they might appear.

Summary of Key Design Choices and Their Justifications

• Full backpropagation through $n+1$ recursion steps over the 1-step gradient approximation: eliminates the need for the Implicit Function Theorem and fixed-point assumptions, and empirically provides a 30+ percentage point gain on Sudoku-Extreme (56.5% → 87.4%), because the model receives an accurate gradient signal about how each reasoning step contributes to the final answer.

• Single 2-layer network over two 4-layer networks: reduces parameter count from 27M to 7M (or 5M for TRM-MLP), and the sharing of parameters between reasoning and answer refinement acts as a regularizer that improves generalization on small datasets. The task distinction is communicated through the presence/absence of $x$ in the input, making separate networks unnecessary.

• $T-1$ no-gradient + 1 gradient recursion processes over running all with gradients: provides free iterative improvement of $(y, z)$ before the gradient step, ensuring the model learns refinement rather than initial construction, while keeping memory cost bounded. This is the coarser-grained analog of HRM's detached forward passes, applied at the recursion process level rather than the individual function evaluation level.

• Binary cross-entropy halting over Q-learning with continue loss: eliminates the second forward pass required by HRM's ACT without sacrificing halting accuracy. The simplification rests on the observation that "is the current answer correct?" is a sufficient halting signal — no counterfactual "would I have done better by continuing?" is needed, because the deep supervision design already ensures that continuing would only be beneficial if the answer is currently wrong.

• Exponential Moving Average of weights (decay 0.999): stabilizes training on small datasets by smoothing parameter trajectories and preventing the sharp divergence that HRM experiences without it. The 7.5 percentage point gain (79.9% → 87.4%) on Sudoku-Extreme suggests that training dynamics, not just architecture, are a critical factor in small-data generalization.

• Attention-free MLP-Mixer for small $L$ over self-attention: exploits the fixed, small context length of Sudoku to replace the $O(L^2)$ attention mechanism with a simpler $[L, L]$ weight matrix, reducing parameters (5M vs. 7M) AND improving accuracy (87.4% vs. 74.7%). Fails on large-$L$ tasks due to parameter explosion and lack of spatial inductive bias.

• Heavy task-specific data augmentation over collecting more data: Sudoku shuffling, dihedral transforms, and color permutations artificially expand small training sets (1000→1M effective examples for Sudoku), teaching the model the underlying algorithmic structure rather than surface-level patterns. The 423K test examples for Sudoku (423× the training set) provide a genuine test of algorithmic generalization.

• Minority of failed experiments documented (MoE, weight tying, TorchDEQ fixed-point iteration, partial backpropagation): the paper's transparency about what DIDN'T work provides a roadmap for practitioners and strengthens the credibility of the positive results by showing they were not cherry-picked from a larger search over variations.

4. Key Insights and Innovations

Innovation 1: Backpropagation-Through-the-Full-Recursion as an Empirical Refutation of the Implicit Function Theorem Shortcut

The most consequential intellectual move in this paper is not architectural — it's diagnostic. The paper identifies that HRM's 1-step gradient approximation, justified by the Implicit Function Theorem (IFT) with an appeal to fixed-point convergence (Bai et al., 2019), is not merely a memory-saving approximation but a training signal bottleneck that fundamentally limits what recursive models can learn. The key diagnostic evidence: replacing HRM's partial backpropagation (2 of 6 steps) with TRM's full backpropagation (all 7 of 7 steps) takes Sudoku-Extreme accuracy from 56.5% to 87.4% (Table 1).

What makes this a genuine innovation rather than an obvious "more gradients = better" observation is that the IFT shortcut was motivated by a theoretical framework — deep equilibrium models and fixed-point reasoning — that the field had accepted as a principled way to train recurrent architectures with bounded memory. The paper's Section 3.1 provides a careful empirical argument that this theoretical framework does not apply to HRM: the forward residuals do not converge to zero (citing Wang et al.'s own Figure 3), the recursion pattern does not correspond to fixed-point iteration, and stopping after 4 of 6 evaluations before applying the 1-step approximation is arbitrary. The critical insight is that the IFT shortcut was solving a problem (memory cost of BPTT) that was self-inflicted by using 4-layer transformers for the recursive networks. By reducing the network to 2 layers, TRM can afford backpropagation through the full recursion cycle without memory explosion, trading per-layer capacity for accurate credit assignment.

This is a fundamental reframing, not an incremental refinement. Prior work (HRM, deep equilibrium models) treated the 1-step gradient approximation as a necessary compromise — you want deep effective computation, you can't backpropagate through all of it, so you approximate. TRM's insight is that the compromise was solving the wrong problem: the bottleneck is not the total effective depth but the parameter count per application of the network. Push depth into recursion count (cheap, reuses parameters) rather than layer count (expensive, adds parameters), and the memory cost of full backpropagation becomes manageable. The significance extends beyond this paper — it suggests that any recurrent architecture facing a BPTT-vs-memory tradeoff should consider whether reducing per-step parameter count (rather than approximating gradients) is the more productive axis to optimize.

The TorchDEQ experiment in Appendix "Ideas that failed" strengthens this interpretation: attempting to use proper fixed-point iteration with the correct IFT justification slowed down training and led to worse generalization. The fixed point wasn't the point — the full gradient signal was.

Innovation 2: Task Disambiguation Through Input Presence/Absence as a Unification of Multi-Network Architectures

HRM uses two separate networks ($f_L$ for latent reasoning, $f_H$ for answer production) motivated by biological arguments about hierarchical processing at different temporal frequencies. The field's default assumption — reinforced by Mixture-of-Experts, multi-agent debates, and modular reasoning architectures — is that different cognitive operations require different learned modules.

TRM proposes a radically simpler alternative: a single network can serve both roles if the input signature encodes which operation to perform. When the input includes the embedded question $x$, the network's task is latent reasoning (update $z$); when $x$ is excluded, the task is answer production (update $y$). No separate parameters, no architectural hierarchy, no biological metaphor — the distinction between "think about the problem" and "produce an answer" is communicated through the presence or absence of the problem representation in the input.

This is conceptually distinctive because it inverts the usual relationship between architecture and task decomposition. The standard approach is: identify sub-tasks → design a module per sub-task → route information between modules. TRM's approach is: identify what information distinguishes sub-tasks → encode that information in the input → let a shared network learn the routing implicitly through gradient descent. The evidence that this works better than separate networks (87.4% vs. 82.4% for the two-network variant, Table 1) suggests that parameter sharing between reasoning and answer production acts as a regularizer — the network cannot learn reasoning strategies that don't translate into better answers, because the same parameters must serve both functions.

This is a fundamental insight about inductive bias in small-data regimes. When data is abundant, separate modules can specialize without overfitting. When data is scarce (1000 training examples), forcing a single module to serve multiple roles acts as a bottleneck that prevents the model from learning spurious correlations in any one role — a form of architectural regularization. The paper's additional experiments with multi-scale $z$ (splitting the latent into $n+1=7$ separate features, 77.6% accuracy) and single $z$ (carrying only one latent, 71.9% accuracy) confirm that exactly 2 latents — $y$ and $z$ — is the optimal representational decomposition (Table 2). Not 1 (which forces the answer to be stored in the reasoning trace), not 7 (which fragments the reasoning unnecessarily), but 2, corresponding cleanly to "what I think the answer is" and "how I arrived at that answer."

The reinterpretation of $z_H$ and $z_L$ as $y$ and $z$ is more than renaming. It replaces a biological metaphor (hierarchical frequencies in the brain) with a functional one (answer vs. reasoning trace), making the architecture's design choices falsifiable — the paper provides Figure 6 as direct evidence that $y$ decodes to a near-correct Sudoku grid while $z$ decodes to noise — rather than appealing to analogies with mouse brains that cannot be empirically verified in the model.

Innovation 3: The Overfitting-Depth Tradeoff and the "Tiny is Better" Principle

The paper's most counterintuitive empirical finding — that reducing the network from 4 layers to 2 improves generalization — crystallizes into a principle that challenges standard scaling intuitions. The conventional wisdom, from the deep learning scaling literature (Kaplan et al., 2020) to the LLM era, is that larger models generalize better, with overfitting treated as a problem to be solved by more data or stronger regularization. TRM's results on Sudoku-Extreme (4-layer: 79.5%, 2-layer: 87.4%) invert this: on small-data reasoning tasks, smaller models generalize better even when total effective depth is held constant.

What makes this an insight rather than a trivial observation about overfitting is the mechanism the paper identifies: the ratio of effective depth (which provides reasoning capacity) to parameter count (which drives overfitting) can be optimized by pushing depth into recursion count rather than layer count. One recursion step through a 2-layer network adds 2 layers of effective depth at the cost of 0 additional parameters. One additional layer in the network architecture adds 1 layer of effective depth (per application) at the cost of an additional transformer block's worth of parameters. The 2-layer TRM with $n=6$ achieves 42 effective layers per supervision step with 5M parameters; the 4-layer TRM with $n=3$ achieves 48 effective layers with 10M parameters — more effective depth, more parameters, worse generalization.

This directly connects to the pretraining-vs-inference compute tradeoff studied in the LLM scaling literature (Snell et al., 2024), but through a different mechanism. Where that work asks "should we spend FLOPs on training a larger model or on more inference-time reasoning?", TRM asks "should we spend parameters on deeper layers or on more recursive applications of shallow layers?" Both trade off per-application cost (parameters in TRM, pretraining FLOPs in LLMs) against total effective computation. The paper's answer — that shallow-but-recursive dominates deep-but-unrolled on small data — provides a new point on the design spectrum that was not obvious a priori.

The failure of Mixture-of-Experts (MoE) reported in the appendix reinforces this principle from the opposite direction. MoEs increase capacity without increasing layer count (Shazeer et al., 2017), which should theoretically provide more representational power at similar depth. Instead, they "decrease[d] massively" — the extra capacity, even when gated sparsely, was still harmful because the total number of trainable parameters increased. The lesson is not "smaller is always better" but rather parameters dedicated to depth (via recursion count) are more generalization-efficient than parameters dedicated to width or per-layer capacity when training data is limited.

Innovation 4: The Latent Answer + Reasoning Trace Decomposition as a Minimal Sufficient State for Iterative Refinement

The paper's analysis of why exactly 2 latent features ($y$ and $z$) are needed — and what each represents — provides a clean functional answer to a design question that HRM obscured with biological metaphor. The argument is: $y$ stores the current answer estimate (decodable to a valid output), $z$ stores the reasoning trace (how the model arrived at $y$, not decodable to a valid output), and both must be carried across refinement steps because removing either degrades performance (Table 2).

This is conceptually significant because it identifies the minimal sufficient state for iterative refinement on deterministic reasoning tasks. You need to remember what you currently think the answer is (so you don't have to reconstruct it from scratch each step, and so you can detect when you've converged). You need to remember how you arrived at that answer (so you can refine the reasoning rather than starting over, and so you can backtrack from wrong turns). And you need exactly these two — splitting $z$ into sub-latents ($z_1, z_2, ..., z_n$) fragments the reasoning trace without benefit (77.6% vs. 87.4%); collapsing $y$ into $z$ forces the reasoning trace to serve double duty as answer storage (71.9% vs. 87.4%).

This is a diagnostic contribution, not an architectural one. The paper isn't claiming that every iterative reasoning system must use exactly 2 latents — that would be architecture-specific. Rather, it's identifying that the latent state decomposition in recursive refinement systems should be analyzable in terms of what information persists across iterations and why, with functional justifications rather than biological or hierarchical ones. The Figure 6 visualization — showing $y$ decodes to a meaningful (nearly correct) grid while $z$ decodes to noise — makes this analysis concrete: $y$ is transparent (you can read off the model's current answer), $z$ is opaque (it only becomes meaningful when transformed into a new $y$). This transparency/opacity distinction is testable and falsifiable, unlike HRM's hierarchical frequencies framing.

The broader implication: when designing iterative refinement systems, the persistent state should be decomposed into components with clearly distinct functional roles, and the number of components should be the minimum needed to support those roles — "less is more" applied to state representation rather than just architecture. This is a design principle the paper extracts from its ablation results rather than asserting a priori, which gives it empirical grounding that HRM's biological motivation lacked.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on four benchmarks: Sudoku-Extreme (Wang et al., 2025) — 1,000 training puzzles, 423,000 test puzzles, 9×9 grids; Maze-Hard (Wang et al., 2025) — 1,000 training, 1,000 test mazes on 30×30 grids where shortest paths exceed length 110, generated by the procedure of Lehnert et al. (2024); ARC-AGI-1 (Chollet, 2019) — 800 tasks (400 training, 400 evaluation, though the exact train/test split used is not specified beyond "public evaluation set"), each with 2–3 input-output demonstration pairs and 1–2 test inputs, with 2 attempts allowed per test input following the official evaluation protocol; ARC-AGI-2 (Chollet et al., 2025) — 1,120 tasks, harder successor to ARC-AGI-1. The paper additionally augments ARC-AGI data with 160 tasks from ConceptARC (Moskvichev et al., 2023). All datasets are explicitly chosen as hard puzzle tasks where LLMs with chain-of-thought achieve near-zero performance (Tables 4–5), providing a strong signal for whether recursive reasoning can close the gap.

• Base model(s). The primary model is TRM with 7M parameters (TRM-Att, using self-attention) or 5M parameters (TRM-MLP, on Sudoku only), consisting of a single 2-layer transformer (or MLP-Mixer variant) with embedding dimension $D = 512$, RMSNorm, no bias, rotary positional embeddings, and SwiGLU activations. For comparison, the paper reports results for HRM (Wang et al., 2025) at 27M parameters — two 4-layer transformers — and Direct prediction at 27M — a single-forward-pass 4-layer transformer without recursion or deep supervision, trained on the identical data and augmentations. For LLM baselines, the paper reports numbers for DeepSeek R1 (671B), Claude 3.7, o3-mini-high, Gemini 2.5 Pro, and Grok-4 (1.7T), all taken from Wang et al. (2025) or the ARC-AGI leaderboard (ARC Prize Foundation, 2025b). The choice of a 2-layer, 7M-parameter architecture is motivated by the paper's central finding that deeper or wider networks overfit severely on small training sets (Section 4.4); the 27M Direct prediction baseline isolates the effect of recursive refinement from model scale.

• Metrics. The primary metric is exact-match test accuracy (%) — the fraction of test examples for which every token of the predicted output matches the ground truth. For Sudoku, this means all 81 cells correct; for Maze, all 900 cells; for ARC-AGI, the full output grid must match exactly, with 2 attempts allowed per test input per the official protocol, and the final score computed as the accuracy over all test inputs across both attempts. There is no partial credit — a Sudoku with 80 correct cells and 1 error counts as a failure. The paper also reports pass@1-style accuracy for the revision model experiments (though this term is not used), where accuracy at each supervision step represents the fraction of examples correctly solved at that step without further refinement.

• Baselines. The paper compares against five categories: (1) Direct prediction — a 27M-parameter 4-layer transformer with a single forward pass, trained with identical data augmentations and hyperparameters, establishing that the tasks cannot be solved by single-pass architectures of comparable scale; (2) HRM (Wang et al., 2025) — the immediate predecessor, with $n = 2$, $T = 2$, two 4-layer transformers, 1-step gradient approximation, and Q-learning ACT, representing the prior state-of-the-art on these benchmarks; (3) LLMs with chain-of-thought — DeepSeek R1, Claude 3.7, o3-mini-high, Gemini 2.5 Pro, and Grok-4, all pretrained on massive corpora and evaluated with CoT prompting, representing the dominant scaling paradigm; (4) LLMs with bespoke test-time compute — Grok-4 with bespoke scaffolding (66.7% on ARC-AGI-1, 16.0% on ARC-AGI-2), representing the upper bound of what massive models with extensive inference-time computation can achieve; (5) Majority voting baselines — for ARC-AGI, TRM's test-time accuracy is computed via majority voting over 1,000 data augmentations per test input, which is the model's own inference protocol rather than a separate baseline.

• Generation budget / compute accounting. The paper does not use a unified "generation budget" metric in the style of LLM test-time compute scaling work. Instead, compute is measured implicitly through three axes: (a) effective depth — the product $T(n + 1) \cdot n_{\text{layers}}$ representing how many transformer layers of computation are applied per supervision step (HRM: $2 \cdot 3 \cdot 4 = 24$; TRM: $3 \cdot 7 \cdot 2 = 42$), which the paper uses to argue TRM performs more computation per step despite having fewer parameters; (b) number of parameters — 27M for HRM vs. 7M (TRM-Att) or 5M (TRM-MLP) for TRM, used to compare model capacity; (c) training time — reported in GPU-hours (Sudoku-Extreme: <36 hours on 1× L40S; Maze-Hard: <24 hours on 4× L40S; ARC-AGI: ~3 days on 4× H100), though this is provided for reproducibility context rather than as a formal compute-matched comparison. The paper does not perform a FLOPs-matched comparison analogous to Section 7 of the Snell et al. (2024) paper — it does not ask "given equal training FLOPs, should we train a larger model or add recursion to a smaller one?" — but the effective depth and parameter counts together provide a rough efficiency comparison (HRM uses 4× the parameters but achieves only 63% of TRM's Sudoku accuracy).

• Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing. The evaluation is on fixed public test sets (Sudoku-Extreme: 423K examples; Maze-Hard: 1K examples; ARC-AGI public evaluation sets). For ARC-AGI, the 2-attempt protocol follows the official evaluation, and the paper reports accuracy aggregated over all test inputs with majority voting across 1,000 augmentations. The paper does not report error bars, confidence intervals, or standard deviations for any result. The ablation experiments on Sudoku-Extreme (Tables 1–3) appear to be single runs — the paper does not mention averaging over random seeds or reporting variance. This is a notable weakness: with a 423K-example test set for Sudoku, even a 1–2 percentage point difference might be statistically significant, but the paper provides no quantification. For ARC-AGI-2, where TRM achieves 7.8% on ~1,120 tasks, the effective sample size is small enough that random variation could meaningfully affect the ranking against HRM's 5.0%.

Main Quantitative Results

Sudoku-Extreme: TRM Achieves 87.4% vs. HRM's 55.0% — A 32.4 Percentage Point Absolute Improvement

The paper's headline result on Sudoku-Extreme is a direct comparison between TRM-MLP and HRM at matched effective depth per supervision step. Table 1 and Table 4 report:

• TRM-MLP (5M parameters, 2-layer, $T = 3$, $n = 6$, no self-attention): 87.4% test accuracy on 423K test puzzles.
• HRM (27M parameters, two 4-layer networks, $n = 2$, $T = 2$): 55.0% test accuracy.
• Direct prediction (27M, single forward pass): 0.0%.
• DeepSeek R1, Claude 3.7, o3-mini-high (chain-of-thought): 0.0%.

The absolute gap between TRM and HRM (32.4 percentage points, a 59% relative improvement over HRM's baseline) is the paper's strongest single result. It represents roughly 1.6× HRM's accuracy, using 0.19× the parameters (5M vs. 27M). The effective depth calculation frames this as an efficiency gain: TRM uses 42 effective layers per supervision step vs. HRM's 24, but with 0.5× the layers per network (2 vs. 4) and 0.5× the number of networks (1 vs. 2). The paper does not provide a direct ablation that holds effective depth equal while varying parameter count, so the relative contributions of depth, parameter count, and gradient accuracy cannot be cleanly separated — but the cumulative effect is unambiguous.

The direct prediction baseline at 0.0% establishes that recursion is necessary — no single-pass model trained on 1,000 Sudoku puzzles with 1,000 augmentations per puzzle generalizes to unseen puzzles, regardless of capacity (the 27M direct prediction model has more parameters than TRM-MLP's 5M). This rules out the hypothesis that the training data or augmentation strategy alone explains the performance.

The LLM baselines at 0.0% establish that scale is insufficient — models with 671B parameters (DeepSeek R1), trained on trillions of tokens, with chain-of-thought prompting, cannot solve a single Sudoku from the test set. This is consistent with the paper's framing that auto-regressive generation without iterative state refinement is fundamentally mismatched to constraint-satisfaction problems where a single incorrect token cascades to an invalid grid.

TRM-Att Components: Ablation-Driven Gains Over HRM

Table 1 reports a series of ablations that decompose the 87.4% TRM-MLP result into the contributions of individual design choices, using the 7M-parameter TRM-Att (with self-attention) as the base for most ablations to maintain architectural comparability:

• TRM base ($T = 3$, $n = 6$, 2-layer, single network, no EMA, no ACT, with self-attention): This configuration, which uses full backpropagation through all 7 recursion steps (unlike HRM's 2-step gradient approximation), achieves 87.4% (this is the TRM-MLP entry, but the paper implies the same configuration with self-attention achieves 74.7% — the 87.4% figure is specifically for the MLP variant on Sudoku).
• w/ 1-step gradient (replacing full backpropagation with HRM's approach of detaching the first 5 of 7 evaluations): 56.5%. This is the single most impactful ablation — removing full backpropagation drops accuracy by 30.9 percentage points (from 87.4% to 56.5%). The paper interprets this as evidence that the IFT-based approximation was the primary bottleneck in HRM, not the two-network design or the biological framing.
• w/ separate $f_H$, $f_L$ (splitting the single network into two 2-layer networks, mimicking HRM's two-network design but with TRM's full backpropagation and recursion schedule): 82.4%. This confirms that the single-network design contributes 5.0 percentage points of improvement over the two-network alternative, even when both use full backpropagation.
• no EMA: 79.9%. Removing Exponential Moving Average of weights costs 7.5 percentage points. The paper attributes this to training instability on small datasets — "HRM tends to overfit quickly and then diverge" (Section 4.7).
• w/ 4-layers, $n = 3$ (increasing per-network depth while reducing recursion count to keep effective depth roughly constant at 48 layers per supervision step vs. TRM's 42, with 10M parameters vs. TRM's 5M): 79.5%. This is the critical evidence for the "less is more" principle — a deeper network with more parameters achieves worse generalization despite similar effective depth, consistent with overfitting on the 1,000-example training set.
• w/ self-attention (the TRM-Att variant on Sudoku, using standard transformer self-attention instead of the MLP-Mixer): 74.7%. This isolates the contribution of the attention-free architecture on small-context tasks — replacing self-attention with an MLP-Mixer improves accuracy by 12.7 percentage points (87.4% vs. 74.7%).
• w/ T = 2, n = 2 (reducing recursion depth to match HRM's effective depth of 12 layers per supervision step): 73.7%. This shows that recursion depth matters — keeping the network architecture fixed (2-layer, single network, full backprop, EMA) but reducing from $n = 6$, $T = 3$ to $n = 2$, $T = 2$ drops accuracy by 13.7 percentage points.
• w/ ACT (adding HRM-style Q-learning ACT with the second forward pass, but keeping all other TRM design choices): 86.1%. The 1.3 percentage point drop relative to TRM's simplified BCE halting (86.1% vs. 87.4%) is small, confirming that the second forward pass can be eliminated without significant performance cost, while reducing per-iteration training time by roughly half (since each optimization step now requires 1 rather than 2 forward passes).

The cumulative interpretation: HRM's 55.0% → TRM's 87.4% is explained by (a) full backpropagation through recursion (+30.9 pp, from the 1-step gradient ablation), (b) single network rather than two (+5.0 pp, from the separate-networks ablation), (c) EMA stabilization (+7.5 pp, from the no-EMA ablation), (d) 2-layer rather than 4-layer architecture (+7.9 pp, from the 4-layer ablation, comparing 87.4% to 79.5%), and (e) attention-free architecture for Sudoku (+12.7 pp, comparing TRM-MLP 87.4% to TRM-Att 74.7%). The sum of estimated individual contributions exceeds the total gain (suggesting interactions between design choices), but the rough breakdown makes clear that no single change accounts for the majority of the improvement — it is the combination that matters.

Number of Recursions: $T = 3$, $n = 6$ Is Optimal; Memory Limits Prevent Further Scaling

Table 3 reports a sweep over $n$ and $T$ for both HRM and TRM on Sudoku-Extreme, matching effective depth approximately across the two models (HRM uses $n = k$, 4 layers; TRM uses $n = 2k$, 2 layers, so effective depth is roughly $T(k + 1) \cdot 4$ for HRM and $T(2k + 1) \cdot 2$ for TRM):

• HRM at $T = 2$, $n = 2$ (the configuration used in all HRM experiments in Wang et al., 2025): 55.0%.
• HRM at $T = 3$, $n = 3$: 61.6% — HRM can benefit from more recursion, but it plateaus and then declines at higher settings ($T = 4$, $n = 4$: 59.5%; $T = 6$, $n = 6$: 57.5%).
• TRM at $T = 3$, $n = 6$: 87.4% — the optimum found.
• TRM at $T = 2$, $n = 4$ (similar effective depth, $2 \cdot 9 \cdot 2 = 36$ layers vs. HRM $T = 3$, $n = 3$ at $3 \cdot 4 \cdot 4 = 48$): 81.9% — substantially better than HRM at similar effective depth.
• TRM at $T = 4$, $n = 4$: 84.2% — decreasing from the optimum, suggesting overfitting or optimization difficulty at higher recursion counts.
• TRM at $T = 3$, $n = 8$ (attempting to scale $n$ further while keeping $T$ at the optimal value): Out-Of-Memory (OOM) — the memory cost of backpropagating through $n + 1 = 9$ steps exceeds the 40GB GPU memory. This is a hard constraint: TRM's full backpropagation means memory scales linearly with $n$, unlike HRM which can scale $n$ arbitrarily since it only backpropagates through the last 2 steps.
• TRM at $T = 6$, $n = 6$: OOM — similarly, increasing $T$ (which increases the number of gradient-tracked recursion processes per supervision step from 1 to 6, since the final process is the one with gradients) exceeds memory.

The key finding is not just the optimal values but the shape of the tradeoff: HRM peaks at lower recursion counts and declines, while TRM peaks higher and is bounded by memory rather than generalization degradation. This suggests that TRM's full backpropagation enables more productive use of recursion depth — each additional recursion step provides useful gradient information — while HRM's gradient approximation plateaus because the model cannot effectively learn from deeper recursion when the gradient signal is truncated.

Maze-Hard: TRM-Att Achieves 85.3% vs. HRM's 74.5%

Table 4 reports results on Maze-Hard (30×30 grids, 1,000 training, 1,000 test):

• TRM-Att (7M parameters, with self-attention): 85.3% test accuracy.
• HRM (27M parameters): 74.5%.
• TRM-MLP (19M parameters on Maze — the paper notes 19M vs. 5M on Sudoku because the 30×30 grid has $L = 900$ tokens, making the $[L, L]$ MLP weight matrix $900 \times 900$ rather than $81 \times 81$): 0.0%.
• Direct prediction (27M): 0.0%.
• DeepSeek R1, Claude 3.7, o3-mini-high: 0.0%.

The 10.8 percentage point improvement over HRM (85.3% vs. 74.5%) is substantial but notably smaller than the 32.4 point gain on Sudoku. The paper does not provide a full ablation table for Maze-Hard, so it is unclear which design choices contribute most to the improvement. The LLM baselines at 0.0% confirm that maze pathfinding remains unsolved by auto-regressive generation — even models with chain-of-thought prompting cannot reliably produce a correct 30×30 shortest path.

The TRM-MLP result at 0.0% is instructive: the attention-free architecture that excelled on Sudoku ($L = 81$) completely fails on Maze ($L = 900$). The paper attributes this to the $[L, L]$ MLP weight matrix being too large and lacking spatial inductive bias — on a 30×30 grid, knowing that two cells are neighbors in the input sequence does not capture their 2D spatial relationship, and the MLP must learn this from scratch with a 900×900 weight matrix. Self-attention with rotary positional embeddings captures pairwise relationships naturally regardless of sequence length, and the attention weights are computed from content rather than stored as fixed parameters, providing both parameter efficiency and a built-in spatial prior.

ARC-AGI-1: TRM-Att Achieves 44.6% vs. HRM's 40.3% — Surpassing Most Frontier LLMs

Table 5 reports results on ARC-AGI-1 (public evaluation set, 2 attempts per test input):

• TRM-Att (7M parameters): 44.6% test accuracy.
• HRM (27M parameters): 40.3%.
• TRM-MLP (19M parameters): 29.6%.
• Direct prediction (27M): 21.0%.
• DeepSeek R1 (671B, CoT): 15.8%.
• Claude 3.7 16K (CoT): 28.6%.
• o3-mini-high (CoT): 34.5%.
• Gemini 2.5 Pro 32K (CoT): 37.0%.
• Grok-4-thinking (1.7T, CoT): 66.7%.
• Bespoke Grok-4 (1.7T, with TTC): 79.6%.

The 4.3 percentage point improvement over HRM (44.6% vs. 40.3%) is the smallest relative gain across benchmarks — TRM provides a 10.7% relative improvement over HRM's baseline, compared to 59% on Sudoku-Extreme and 14.5% on Maze-Hard. The paper does not ablate this result to identify which TRM components contribute on ARC-AGI, but the finding that TRM-MLP underperforms self-attention (29.6% vs. 44.6%) is consistent with the Maze results: ARC-AGI grids can be up to 30×30, and the variable grid sizes across tasks make the $[L, L]$ MLP approach impractical. The embedding augmentation strategy described in Section 6 — "each puzzle ... at each data-augmentation is given a specific embedding of shape $[0, 1, D]$" — suggests the model learns a per-task embedding that helps distinguish different puzzle types in the batch.

The comparison to frontier LLMs is the paper's most striking claim: TRM with 7M parameters (roughly 0.001% of GPT-4's estimated parameter count) outperforms o3-mini-high (34.5%), Gemini 2.5 Pro (37.0%), and Claude 3.7 (28.6%), and approaches within 22 points of Grok-4 with chain-of-thought (66.7%). This is the core evidence for the paper's implicit argument that architectural innovation (iterative state refinement) can compensate for massive scale on structured reasoning tasks. However, the comparison is not entirely fair: Grok-4-thinking and the bespoke Grok-4 variant achieve substantially higher scores (66.7% and 79.6%), and the paper acknowledges that "every choice made is not guaranteed to be optimal on every dataset" (Section 6). The LLMs are evaluated with their publicly reported scores, not with a dedicated prompt engineering or fine-tuning effort for ARC-AGI specifically, while TRM and HRM are trained and tuned specifically for these benchmarks.

The direct prediction baseline at 21.0% (compared to 0.0% on Sudoku and Maze) indicates that some ARC-AGI-1 puzzles are solvable with a single forward pass — likely the simpler tasks involving pattern completion, color matching, or object copying that do not require multi-step reasoning. The gap between direct prediction (21.0%) and TRM (44.6%) represents the puzzles that require iterative refinement and are solvable by a 7M-parameter model with recursion — more than doubling the fraction of the benchmark that the small-model approach can handle.

ARC-AGI-2: TRM-Att Achieves 7.8% vs. HRM's 5.0% — Small Absolute Gains on a Harder Benchmark

Table 5 reports results on ARC-AGI-2 (public evaluation set, 2 attempts):

• TRM-Att (7M parameters): 7.8% test accuracy.
• HRM (27M parameters): 5.0%.
• TRM-MLP (19M parameters): 2.4%.
• Direct prediction (27M): 0.0%.
• DeepSeek R1 (671B): 1.3%.
• Claude 3.7 16K: 0.7%.
• o3-mini-high: 3.0%.
• Gemini 2.5 Pro 32K: 4.9%.
• Grok-4-thinking (1.7T): 16.0%.
• Bespoke Grok-4 (1.7T): 29.4%.

The 2.8 percentage point absolute improvement over HRM (7.8% vs. 5.0%) represents a 56% relative improvement — proportionally large, but small in absolute terms on a benchmark where even the best systems struggle. The key observation is that ARC-AGI-2 is hard for everyone: the best LLM without bespoke test-time compute (Grok-4-thinking) achieves only 16.0%, and the bespoke variant reaches 29.4%. TRM's 7.8% places it above all non-Grok LLMs, consistent with the ARC-AGI-1 pattern but at a much lower absolute accuracy level.

The gap between ARC-AGI-1 (44.6%) and ARC-AGI-2 (7.8%) is 36.8 percentage points for TRM — a much larger drop than for the LLM baselines. For comparison, Gemini 2.5 Pro drops from 37.0% to 4.9% (32.1 points), and o3-mini-high drops from 34.5% to 3.0% (31.5 points). This suggests that ARC-AGI-2's increased difficulty — designed to be more resistant to memorization and pattern matching (Chollet et al., 2025) — affects TRM roughly proportionally to how it affects LLMs, with no evidence that TRM's architectural advantages are more robust to the specific changes in ARC-AGI-2. However, the low absolute numbers make precise comparison unreliable without error bars.

The direct prediction baseline at 0.0% on ARC-AGI-2 (vs. 21.0% on ARC-AGI-1) confirms that ARC-AGI-2 eliminates the simpler puzzles that a single forward pass can solve. TRM's entire 7.8% performance derives from iterative refinement — without recursion, the model cannot solve a single ARC-AGI-2 test input. This is strong evidence that the benchmark is achieving its design goal of requiring multi-step reasoning rather than pattern matching.

Number of Latent Features: Exactly 2 ($y$ and $z$) Is Optimal

Table 2 reports an ablation on Sudoku-Extreme varying the number of latent features carried across supervision steps:

• TRM with $y$, $z$ (the default, 2 features): 87.4%.
• TRM with multi-scale $z$ ($n + 1 = 7$ features, treating each of the $n$ recursion steps as producing a separate $z_i$ that is carried forward, described in Algorithm 5): 77.6%.
• TRM with single $z$ (only $z$ is carried; $y$ is not explicitly maintained, described in Algorithm 4): 71.9%.

This ablation directly tests the paper's functional interpretation of the latent state decomposition. The single-$z$ variant forces the network to store the current answer estimate within the reasoning trace $z$ — since there is no separate $y$, the output head must decode from $z$ directly. The 15.5 percentage point drop (71.9% vs. 87.4%) suggests that conflating answer storage with reasoning trace degrades the model's ability to refine its output: the network must simultaneously represent "what I think the answer is" and "how I arrived at that answer" in a single embedding, and gradient signals for improving the answer may interfere with signals for improving the reasoning.

The multi-scale-$z$ variant splits the reasoning trace into $n = 6$ separate latents, each updated independently and carried across supervision steps. The 9.8 percentage point drop (77.6% vs. 87.4%) indicates that fragmenting the reasoning trace hurts generalization — the network has more parameters dedicated to latent state (since each $z_i$ is a separate embedding of size $D = 512$) but learns less effectively, likely because the fragmented representation makes it harder for the network to integrate information across reasoning steps or because the additional capacity leads to overfitting.

Both alternative configurations (single-$z$ and multi-scale-$z$) outperform the TRM with 1-step gradient (56.5%) and the HRM baseline (55.0%), confirming that while 2 features is optimal, the core value of TRM lies in the full backpropagation and recursion schedule, not just the latent decomposition. The paper interprets the 2-feature optimum as evidence for the functional interpretation: $y$ is the answer, $z$ is the reasoning trace, and precisely these two roles need separate persistent state.

Ablation Studies and Robustness Checks

1-step gradient approximation: Replacing TRM's full backpropagation through all $n + 1 = 7$ recursion steps with HRM's approach — detaching the first 5 steps and backpropagating only the last 2 — reduces Sudoku-Extreme accuracy from 87.4% to 56.5% (Table 1). This is a 30.9 percentage point drop, making it the single most impactful design choice in the paper. The ablation confirms that the IFT-based gradient shortcut was the primary bottleneck in HRM, not the two-network design or the biological framing. However, 56.5% is still slightly above HRM's 55.0% (which also uses two networks and 4 layers), suggesting that the other TRM choices (2 layers, EMA, BCE halting) provide a modest 1.5 point improvement even with the truncated gradient — the full 32.4 point gain requires the full combination.

Single network vs. two networks: Using two separate 2-layer networks ($f_L$ for $z$ updates, $f_H$ for $y$ updates) instead of a single shared network reduces accuracy from 87.4% to 82.4% (Table 1). This 5.0 percentage point drop supports the paper's claim that a single network is sufficient (the task distinction is carried by input presence/absence of $x$) and beneficial (parameter sharing acts as a regularizer). However, the two-network TRM variant at 82.4% still substantially outperforms HRM at 55.0%, reinforcing that full backpropagation, not network unification, is the dominant factor.

EMA for training stability: Removing the Exponential Moving Average of weights (decay 0.999) reduces accuracy from 87.4% to 79.9% (Table 1). The 7.5 percentage point drop is attributed to training instability on small datasets: the paper notes that HRM "tends to overfit quickly and then diverge" (Section 4.7) and that EMA "prevents sharp collapse." This is a training dynamics contribution, not an architectural one — EMA could be applied to HRM and would presumably improve its stability as well, though the paper does not test this.

Number of layers (2 vs. 4): Increasing the network depth from 2 layers to 4 layers, while reducing recursion from $n = 6$ to $n = 3$ to keep effective depth approximately constant (48 layers per supervision step for the 4-layer variant vs. 42 for the 2-layer variant), reduces accuracy from 87.4% to 79.5% (Table 1). Parameter count increases from 5M to 10M. The 7.9 percentage point drop is the key evidence for the "less is more" principle: deeper per-application networks overfit on the 1,000-example training set, even when total effective depth is held approximately constant. The paper attributes this to Kaplan et al. (2020)'s observation that "when data is too scarce and model size is large, there can be an overfitting penalty." However, the comparison is not perfectly controlled: effective depth differs slightly (48 vs. 42 layers), and the number of gradient-tracked recursion steps per supervision step is lower for the 4-layer variant (4 steps of 4-layer computation = 16 effective layers with gradients vs. 7 steps of 2-layer computation = 14 effective layers with gradients for TRM), which may interact with the generalization result.

Self-attention vs. MLP-Mixer (TRM-Att vs. TRM-MLP): On Sudoku-Extreme ($L = 81$, small fixed context), replacing self-attention with an MLP applied along the sequence dimension improves accuracy from 74.7% to 87.4% while reducing parameters from 7M to 5M (Table 1). On Maze-Hard ($L = 900$, 30×30 grid), the same substitution causes accuracy to collapse from 85.3% to 0.0% (Table 4). On ARC-AGI-1 (variable grid sizes up to 30×30), the MLP variant drops to 29.6% vs. 44.6% for the attention variant (Table 5). This pattern is consistent with the paper's explanation: the MLP-Mixer's $[L, L]$ weight matrix is parameter-efficient and well-suited when $L \leq D$ (81 < 512), but becomes parameter-intensive and lacks spatial inductive bias when $L \gg D$ (900 > 512). The paper does not report an MLP variant for ARC-AGI-2 separately, but the 2.4% result (Table 5) presumably follows the same pattern. This ablation establishes that the attention-free design is task-dependent, not a universal improvement.

Recursion schedule ($T$ and $n$): Table 3 sweeps both parameters for HRM and TRM on Sudoku-Extreme. For TRM, the optimum is $T = 3$, $n = 6$ (87.4%). Lower values: $T = 1$, $n = 2$ (effective depth ~7 layers, using 2× the $n$ for TRM) at 63.2%; $T = 2$, $n = 4$ (depth ~20 layers) at 81.9%. Higher values: $T = 3$, $n = 8$ (depth ~54 layers) at OOM; $T = 6$, $n = 6$ at OOM; $T = 3$, $n = 6$ is the empirical optimum under the memory constraint. For HRM, increasing $n$ and $T$ from the default $n = 2$, $T = 2$ (24 effective layers) to $n = 3$, $T = 3$ (48 effective layers) improves accuracy from 55.0% to 61.6%, but further increases lead to decline ($n = 4$, $T = 4$: 59.5%; $n = 6$, $T = 6$: 57.5%). The paper's interpretation: HRM's fixed-capacity networks (4 layers, not 2) benefit modestly from more recursion but are limited by the IFT gradient approximation — deeper recursion without improved gradient signal eventually hurts. TRM's 2-layer networks benefit more from deeper recursion because full backpropagation provides useful gradient information at each additional step, and the bottleneck is memory rather than diminishing returns.

ACT (Q-learning with second forward pass vs. BCE halting): Adding HRM-style Q-learning ACT (requiring two forward passes per supervision step) to TRM achieves 86.1% on Sudoku-Extreme, compared to 87.4% for TRM's simplified BCE halting (Table 1). The 1.3 percentage point difference is small, confirming that the simpler halting mechanism (a) does not sacrifice accuracy and (b) eliminates the computational cost of the second forward pass. The paper does not report the average number of supervision steps with BCE halting vs. Q-learning halting, so it is unclear whether the training efficiency (time spent per example) differs between the two methods.

Number of latent features: As described above, 2 features ($y$, $z$) achieves 87.4%; 1 feature (single $z$) achieves 71.9%; 7 features (multi-scale $z$) achieves 77.6% (Table 2). The robustness check confirms that the exactly-2-latent design is not arbitrary — it is the optimum on Sudoku-Extreme.

Negative results documented in the appendix (Section "Ideas that failed"):

• Mixture-of-Experts (MoE): Replacing the SwiGLU MLP with SwiGLU MoE "decrease[d] massively" — the paper does not provide a specific accuracy number but the qualitative language suggests a catastrophic drop.
• Weight tying input embedding and output head: "Too constraining and led to a massive generalization drop" — again no number, but clearly incompatible with the architecture.
• TorchDEQ fixed-point iteration: Using the TorchDEQ library (Geng & Kolter, 2023) to replace the recursion with proper fixed-point iteration and leverage the IFT with correct justification "slowed down training due to the fixed-point iteration and led to worse generalization" — providing evidence that converging to an actual fixed point is not helpful for this task, and that the value of TRM's recursion is in the computation performed, not in reaching equilibrium.
• Partial backpropagation compromise: Backpropagating through only the last $k = 4$ of $n = 6$ steps (a compromise between HRM's 2 and TRM's 7) "did not help generalization in any way, and it made the approach more complicated."
• Removing ACT entirely: Training without any halting mechanism (i.e., always running all 16 supervision steps on every example) "generalization dropped significantly" — the model spends too much time on already-solved examples rather than seeing diverse training data, consistent with the motivation for ACT in the first place.

Critical Assessment

Does the paper demonstrate that TRM achieves "significantly higher generalization than HRM" (Section 1)?

Yes, strongly. On Sudoku-Extreme, the improvement is 87.4% vs. 55.0% — a 32.4 percentage point absolute gain (Table 4). On Maze-Hard: 85.3% vs. 74.5% (Table 4). On ARC-AGI-1: 44.6% vs. 40.3% (Table 5). On ARC-AGI-2: 7.8% vs. 5.0% (Table 5). All four benchmarks show TRM outperforming HRM, with gains ranging from 2.8 to 32.4 percentage points. The results are consistent in direction, and the larger gains on Sudoku and Maze are backed by ablations (Table 1 and Table 3) that decompose the improvement into specific design choices.

However, the paper does not report any measure of statistical uncertainty — no error bars, standard deviations, or confidence intervals for any result. On ARC-AGI-2, the test set is 1,120 tasks, and a 2.8 percentage point difference represents approximately 31 additional correctly solved test inputs. Without knowing the variance of this estimate (which depends on the difficulty distribution of the test set and any stochasticity in training), we cannot assess whether 7.8% vs. 5.0% is a reliable difference or within the noise floor of a single training run. This is a significant omission, especially for the ARC benchmarks where the absolute numbers are low and small absolute differences carry substantial proportional weight.

Additionally, the paper does not report HRM's performance under TRM's training protocol (EMA, BCE halting, data augmentations). The direct comparison is TRM with its full suite of improvements vs. HRM as reported by Wang et al. (2025). A fairer comparison would give HRM the same training stabilizations (EMA, stable-max loss) and see how much of the gap closes. The 1-step gradient ablation (Table 1: TRM with 1-step gradient at 56.5% vs. HRM at 55.0%) suggests HRM's gradient approximation is the main bottleneck and that EMA, BCE halting, and 2-layer architecture collectively add only 1.5 points at the same gradient regime — but this inference assumes the ablation transfers to HRM's exact architecture, which has not been verified.

Does the paper demonstrate that TRM "achieves significantly higher generalization than HRM, while using a single tiny network with only 2 layers" (Abstract)?

Supported, with caveats about the architectural comparison. TRM uses 2 layers; HRM uses two 4-layer networks (8 layers total across two modules). TRM uses 5–7M parameters; HRM uses 27M. TRM achieves higher accuracy on all four benchmarks. The ablation (Table 1) confirms that a 4-layer TRM variant underperforms the 2-layer variant (79.5% vs. 87.4%), and that a two-network TRM variant underperforms the single-network variant (82.4% vs. 87.4%).

However, the comparison of "tiny" vs. "small" is confounded by the recursion schedule. TRM uses $n = 6$, $T = 3$ (21 network evaluations per supervision step, 7 with gradients); HRM uses $n = 2$, $T = 2$ (6 network evaluations, 2 with gradients). TRM is "tiny" per application but applies that tiny network many more times. The effective depth per supervision step is 42 layers for TRM vs. 24 for HRM, so TRM performs 1.75× more computation per step. The paper's claim that "less is more" refers to per-application parameters, not total computation — TRM is more parameter-efficient but potentially more compute-intensive at inference time (21 evaluations vs. 6, though the 2-layer evaluations are cheaper than HRM's 4-layer evaluations). The paper does not provide wall-clock inference time comparisons, which would clarify whether TRM's parameter efficiency translates to latency efficiency.

Does the paper demonstrate that TRM "obtains 45% test-accuracy on ARC-AGI-1 and 8% on ARC-AGI-2, higher than most LLMs" (Abstract)?

Supported, but the comparison is asymmetric in important ways. TRM at 44.6% on ARC-AGI-1 does indeed exceed DeepSeek R1 (15.8%), Claude 3.7 (28.6%), o3-mini-high (34.5%), and Gemini 2.5 Pro (37.0%) — all frontier LLMs with 4–7 orders of magnitude more parameters (Table 5). The result is striking and well-supported by the reported numbers.

However, the comparison has several asymmetries that weaken its force:

1. The LLMs are evaluated zero-shot with chain-of-thought prompting; TRM is trained specifically for ARC-AGI. The LLM baselines come from public leaderboards — they represent what the models achieve out-of-the-box with CoT, not what they could achieve with ARC-specific fine-tuning. The paper does not compare against fine-tuned LLMs (e.g., a LoRA-fine-tuned Llama-3 on ARC-AGI training data), which would be the more direct comparison to TRM's supervised training pipeline. A model trained for 100,000 epochs on ARC-specific data augmentations is solving a different task than a model prompted with general reasoning instructions.

2. TRM uses 1,000 data augmentations per puzzle and majority voting at test time, which is a form of test-time compute not accounted for in the LLM comparison. While LLMs also use test-time compute (CoT generation, potentially multiple samples), the paper compares TRM's augmented+ensembled score against LLMs' single-prediction scores in most cases. Grok-4-thinking (66.7%) and Bespoke Grok-4 (79.6%) demonstrate that LLMs with extensive test-time compute can far exceed TRM's performance. A fair comparison would match test-time compute budgets: how many FLOPs does TRM spend per puzzle (1,000 forward passes through a 7M-parameter network, 16 supervision steps each, 21 recursion evaluations per step = 1,000 × 16 × 21 × 2 layers × ~512² operations ≈ a large but computable number) vs. the LLM's CoT generation budget? The paper does not attempt this accounting.

3. The LLM baselines are cherry-picked post-hoc from leaderboards, not run by the authors under controlled conditions. The paper itself acknowledges (Table 4–5 notes) that "The numbers for Deepseek R1, Claude 3.7 8K, O3-mini-high, Direct prediction, and HRM from the Table 4 and 5 are taken from Wang et al. (2025)." There is no guarantee that these represent each model's best possible performance — the LLM community has developed sophisticated prompting techniques (self-consistency, debate, reflexion) that could improve these numbers, and the paper does not control for prompt quality.

4. Grok-4 with bespoke test-time compute achieves 79.6% on ARC-AGI-1 and 29.4% on ARC-AGI-2 — 1.78× and 3.77× TRM's scores. The paper's narrative emphasizes beating "most LLMs," but the most capable system on the leaderboard dramatically outperforms TRM. TRM's contribution is most accurately characterized as closing the gap between small trained models and zero-shot LLMs, not surpassing the LLM approach in absolute capability.

Does the ablation in Table 1 convincingly attribute the HRM→TRM improvement to specific design choices?

Partially. The ablation shows that replacing the 1-step gradient with full backpropagation causes the largest single change (56.5% → 87.4%, a +30.9 pp gain), and that additional changes (EMA, single network, attention-free, 2 layers) each contribute smaller but meaningful gains. This supports the paper's central narrative that the IFT approximation was the primary bottleneck.

However, the ablation is conducted entirely on TRM variants, not on HRM itself. We see what happens when we degrade TRM toward HRM (add 1-step gradient: 56.5%; add two networks: 82.4% etc.), not what happens when we improve HRM toward TRM (give HRM EMA, BCE halting, full backprop). The inference that HRM's poor performance is primarily due to the gradient approximation would be stronger if the paper had shown HRM + full backpropagation (keeping everything else HRM-like) approaching TRM's performance. Instead, the 1-step gradient ablation is on a TRM variant that already has 2 layers, EMA, and a single network — so we are seeing the effect of the gradient approximation in a regime where the other TRM design choices are already present. It is possible that HRM with its 4-layer, two-network architecture would benefit less from full backpropagation (e.g., due to overfitting or optimization difficulties in the larger parameter space), and the paper does not test this.

The ablation also has a missing baseline: TRM with HRM's exact recursion schedule ($n = 2$, $T = 2$, but with 2-layer single network, EMA, full backprop). This would isolate the effect of recursion depth from the other design choices. Table 3 provides some evidence: TRM with $T = 2$, $n = 2$ achieves 73.7%, and HRM with $T = 2$, $n = 2$ achieves 55.0%. The 18.7 point gap at matched depth is attributable to the other TRM design choices (2-layer, single network, EMA, full backprop). But HRM with $T = 3$, $n = 3$ achieves 61.6% — still 25.8 points below TRM at $T = 3$, $n = 6$ — so depth alone explains some but not all of the gap.

Does the paper establish that "recursion helps so much compared to using a larger and deeper network"?

The paper provides strong evidence that recursion helps more than depth, but the exact mechanism is not fully isolated. The key comparison is the 2-layer TRM ($T = 3$, $n = 6$, 42 effective layers per step, 5M parameters, 87.4%) vs. the 4-layer TRM ($T = 3$, $n = 3$, 48 effective layers per step, 10M parameters, 79.5%). The 2-layer variant has 7.9 points better accuracy despite having fewer effective layers and fewer parameters. This supports the claim that recursion count > layer count for generalization on small data.

However, the 4-layer variant has a different gradient structure: it backpropagates through 4 recursion steps of a 4-layer network (16 effective layers with gradients), while the 2-layer variant backpropagates through 7 recursion steps of a 2-layer network (14 effective layers with gradients). The gradient signal in the 2-layer variant is spread across more recursion steps, which may provide a finer-grained learning signal that is not available in the 4-layer variant. The paper's claim that "overfitting" explains the difference is plausible (more parameters → more overfitting on small data) but not proven — alternative explanations (optimization dynamics, gradient flow differences, implicit regularization from recursion) are not ruled out.

The paper itself acknowledges this gap in the conclusion: "the question of why recursion helps so much compared to using a larger and deeper network remains to be explained; we suspect it has to do with overfitting, but we have no theory to back this explanation" (Section 6). This is commendably honest, but it means the central theoretical claim of the paper — that recursive depth is fundamentally better than architectural depth for small-data reasoning — remains an empirical observation without a mechanistic explanation.

Does the paper's difficulty-based analysis (or its equivalent) demonstrate robustness?

The paper does not perform a difficulty-based analysis. Unlike the Snell et al. (2024) paper on test-time compute scaling, TRM does not bin problems by difficulty and report per-bin performance. This is a missed opportunity: understanding whether TRM's improvements over HRM are concentrated on easy, medium, or hard puzzles (within each benchmark) would illuminate why the architectural changes help. For example, if TRM's gains on Sudoku-Extreme are largest on the hardest puzzles (which require the deepest reasoning), that would support the paper's narrative that full backpropagation enables more effective use of recursion depth. Conversely, if gains are uniform across difficulty, the benefit might stem from something simpler (e.g., better optimization stability). The Sudoku-Extreme dataset's 423K test examples would support such an analysis, but the paper does not conduct it.

Missing experiments and baselines that would strengthen the paper:

1. Multiple random seeds with error bars. The paper reports single numbers for all results. A model trained on 1,000 Sudoku puzzles with heavy data augmentation may have non-trivial run-to-run variance. Reporting mean ± standard deviation over 3–5 seeds would substantially increase confidence in the reported differences, especially for the smaller ARC-AGI-2 gains.

2. HRM with TRM's training stabilizations (EMA, stable-max loss). Would HRM improve if given the same training-dynamics improvements? The paper attributes HRM's instability to small data and shows EMA helps TRM (79.9% → 87.4%), but does not test whether EMA helps HRM similarly.

3. TRM with HRM's recursion schedule ($n = 2$, $T = 2$) reported in the main ablation table. While Table 3 provides this at 73.7%, the main ablation (Table 1) uses $T = 3$, $n = 6$ as the base and does not show the effect of reducing recursion to HRM's level at full TRM architectural parity. This would isolate the contribution of recursion depth from other design choices.

4. Wall-clock inference time comparison. TRM runs 21 network evaluations per supervision step × 16 steps = 336 evaluations at test time. HRM runs 6 evaluations × 16 steps = 96 evaluations. TRM's evaluations are cheaper (2-layer vs. 4-layer), but the factor of 3.5× more evaluations may mean TRM is slower at inference despite having fewer parameters. The paper's claim of efficiency rests entirely on parameter count and training time — inference latency is unreported.

5. Evaluation on held-out puzzle difficulties. Sudoku puzzles have known difficulty metrics (e.g., number of given clues, techniques required to solve). Stratifying the 423K test set by difficulty would reveal whether TRM's advantage over HRM is uniform or concentrated on the hardest puzzles (which benefit most from deeper recursion).

6. Direct comparison to LLMs fine-tuned on ARC-AGI data. A fine-tuned small LLM (e.g., Llama-3-8B with LoRA on ARC training tasks) would be a more appropriate baseline than zero-shot frontier models, since TRM itself is trained on the ARC training data.

7. FLOPs-matched comparison between TRM and larger non-recursive models. The paper argues that recursion is more parameter-efficient than depth, but does it save total FLOPs? A comparison where total training FLOPs are held constant — TRM vs. a proportionally larger direct-prediction model — would test whether the efficiency claim holds in compute terms, not just parameter-count terms.

Summary of the paper's experimental contributions and limitations:

What the experiments convincingly show:

• TRM substantially outperforms HRM on all four benchmarks, with gains ranging from 2.8 to 32.4 percentage points (Tables 4–5).
• Full backpropagation through the recursion process is the single most impactful design choice, contributing approximately 30.9 percentage points of the Sudoku gain (Table 1).
• The 2-layer, single-network, attention-free (for small $L$) design is strictly better than the 4-layer, two-network, self-attention baseline on the tested benchmarks under the tested training conditions.
• Recursion with full backpropagation enables small models to solve hard reasoning tasks that zero-shot LLMs with 5–7 orders of magnitude more parameters struggle with, when those small models are specifically trained for the task.

What the experiments do not convincingly show (or leave ambiguous):

• Whether the gains come primarily from the gradient approximation fix or from the reduced parameter count (both change simultaneously; the interaction is not fully isolated).
• Whether TRM's parameter efficiency translates to compute or latency efficiency at inference time.
• Whether the results are statistically reliable (no error bars, no multi-seed reporting).
• Whether the improvements would persist if HRM were given the same training stabilizations (EMA, stable-max).
• How TRM would compare to an LLM that is fine-tuned on the target benchmark data rather than evaluated zero-shot.
• Whether the "less is more" principle generalizes beyond the specific small-data puzzle benchmarks tested.

6. Limitations and Trade-offs

The Approach Is Restricted to Closed-Form, Deterministic Output Tasks

TRM is fundamentally a supervised learning method that learns a deterministic mapping from input to output: given a puzzle state, produce the single correct answer grid. The paper states this explicitly in the conclusion:

"Currently, recursive reasoning models such as HRM and TRM are supervised learning methods rather than generative models. This means that given an input question, they can only provide a single deterministic answer."

The consequence is a hard capability bound: TRM cannot handle tasks where multiple valid answers exist, where the output is open-ended text rather than a structured grid, or where the reasoning process itself should be communicated to a user. A Sudoku puzzle has exactly one valid solution — the model learns to produce that grid directly. An ARC task has a single correct output grid. A maze has a single shortest path (or the model is trained to produce one specific optimal path). This deterministic, single-output design is well-matched to the benchmarks studied but excludes entire categories of reasoning tasks: mathematical proofs (where the reasoning matters as much as the final theorem), code generation (where multiple implementations can be correct), dialogue and explanation tasks, and any setting where the user needs to understand how the model arrived at its answer rather than just receiving the answer.

The paper's evidence for this limitation is structural rather than empirical: the architecture itself — an embedding-to-embedding mapping with argmax decoding — offers no mechanism for autoregressive generation, sampling, or confidence calibration beyond the single scalar Q-head halting probability. The paper does not evaluate TRM on any task requiring open-ended generation, and the Q-head provides only a binary "correct/incorrect" estimate, not calibrated probabilities that could be used for uncertainty quantification or selective prediction. The paper acknowledges this gap (Section 6: "it would be interesting to extend TRM to generative tasks") but does not propose a specific extension, leaving the path from supervised puzzle-solver to general reasoning system unspecified.

The Optimal Architecture Is Task-Dependent and Requires Per-Task Tuning

The paper's central finding — that an attention-free MLP-Mixer architecture dramatically outperforms self-attention on Sudoku — does not generalize across tasks, and the paper does not provide a principled way to choose architectures for new problems. On Sudoku-Extreme ($L = 81$ grid cells), TRM-MLP achieves 87.4% while TRM-Att achieves 74.7%, a 12.7 percentage point gap (Table 1). On Maze-Hard ($L = 900$), the same MLP architecture collapses to 0.0% (Table 4). On ARC-AGI-1 (variable grid sizes, up to 30×30), TRM-MLP drops to 29.6% vs. 44.6% for the attention variant (Table 5).

The paper's explanation — that the MLP requires an $[L, L]$ weight matrix that is cheap when $L \leq D$ but expensive and lacking spatial inductive bias when $L \gg D$ — is post-hoc rather than predictive. A practitioner approaching a new task would need to determine: (a) whether the context length is small enough for the MLP variant, (b) whether the task's spatial structure is captured by an MLP's dense pairwise interactions or requires the content-based attention patterns of self-attention, and (c) whether the optimal number of layers, recursions, and latent features found on the paper's benchmarks transfer. The paper sweeps $T$ and $n$ for Sudoku-Extreme (Table 3) but does not report equivalent sweeps for Maze-Hard or ARC-AGI, leaving open the question of whether $T = 3, n = 6$ is universally optimal or Sudoku-specific.

The consequence is that TRM as presented is a family of architectures whose best instantiation varies by task, but the paper provides only empirical guidance (try both, pick the better one) rather than a decision rule or scaling law. This matters because training TRM on ARC-AGI takes approximately 3 days on 4× H100 GPUs (Section 6), making architectural trial-and-error infeasibly expensive for many practitioners. The paper also does not report the sensitivity of results to the many hyperparameter choices that differ across tasks — learning rate (differing by 10× for embeddings on ARC-AGI), weight decay (1.0 vs. 0.1), training duration (60K vs. 100K epochs) — making it unclear whether the architecture or the hyperparameter tuning is the primary source of cross-task performance variation.

Training Requires Massive Data Augmentation That May Not Be Available for All Domains

TRM's ability to generalize from ~1,000 training examples depends critically on task-specific data augmentation that multiplies the effective training set size by 8–1000×. Sudoku-Extreme uses 1,000 Sudoku-preserving shuffling augmentations per puzzle; Maze-Hard uses 8 dihedral transformations; ARC-AGI uses 1,000 augmentations combining color permutations, dihedral transformations, and translations. The paper acknowledges the centrality of augmentation in its hyperparameter section but does not address the limitation this imposes on applicability.

The consequence is that TRM is not a general "learn from few examples" method — it is a method that learns from massively augmented few examples, where the augmentations are carefully designed to preserve the task's semantic structure. Designing such augmentations requires domain expertise: Sudoku shuffling exploits the puzzle's symmetry group; ARC color permutation exploits the fact that ARC tasks are defined by abstract patterns independent of specific colors; dihedral transformations exploit geometric symmetries. For many real-world reasoning tasks — medical diagnosis from patient records, legal reasoning from case texts, financial forecasting from time series — equivalent label-preserving augmentations either do not exist or would be extremely difficult to construct. A model trained on 1,000 patient records with "disease-preserving augmentations" would need to know what transformations of a patient record leave the diagnosis unchanged — a fundamentally harder problem than rotating a maze grid.

The paper provides no ablation showing how performance degrades as augmentation is reduced. We do not know whether TRM trained on 1,000 Sudoku puzzles with only 100 augmentations per puzzle would achieve, say, 70% accuracy (moderate drop) or 20% accuracy (catastrophic drop). This makes it impossible to assess how much of TRM's performance is attributable to the recursive architecture versus the augmentation strategy itself. The Direct Prediction baseline at 0.0% on Sudoku (Table 4) uses the same 1,000 augmentations per puzzle, so augmentation alone is not sufficient — but the interaction between augmentation and recursion (does recursion make more effective use of augmentations than single-pass architectures?) is not isolated.

Difficulty Estimation and Dynamic Budget Allocation Are Entirely Absent

The Snell et al. (2024) paper on compute-optimal test-time scaling — which this paper does not cite but which provides a relevant framework — demonstrated that the optimal allocation of inference compute varies dramatically with problem difficulty, and that difficulty-conditioned strategies can yield 4× efficiency improvements. TRM applies the same recursion schedule ($T = 3, n = 6, N_{\text{sup}} = 16$) uniformly to all problems regardless of difficulty, despite the benchmarks containing substantial difficulty variation (Sudoku puzzles range from trivially solvable to requiring advanced techniques; ARC-AGI tasks span a wide difficulty spectrum).

The consequence is wasted computation on easy problems and insufficient computation on hard ones. During test time, TRM always runs 3 recursion processes per supervision step × 16 supervision steps = 48 recursion processes, each comprising 7 network evaluations, for a total of 336 forward passes through the 2-layer network per problem. An easy Sudoku (solvable with basic elimination in a few reasoning steps) might converge to the correct answer after 2–3 supervision steps; the remaining 13–14 steps are redundant computation. Conversely, an extremely hard Sudoku requiring deep lookahead might benefit from more recursion depth ($n > 6$) or more supervision steps ($N_{\text{sup}} > 16$), but the fixed schedule provides no mechanism to dynamically allocate budget. The ACT mechanism addresses this during training by halting early on solved examples, but at test time, the paper explicitly states that ACT is disabled and "all $N_{\text{sup}} = 16$ supervision steps are executed" (Section 4.6).

The paper provides no stratification of results by problem difficulty. We do not know whether TRM's 87.4% Sudoku accuracy is 99% on easy puzzles and 60% on hard ones (suggesting the fixed schedule is adequate for easy puzzles but insufficient for hard ones), or 87% uniform across difficulties (suggesting the schedule is well-matched to the overall distribution). The Q-head, which learns to predict answer correctness, could in principle serve as an online difficulty estimator — if $\hat{q}$ is low after step $k$, the problem is likely hard and more compute should be allocated — but the paper does not explore this possibility. The contrast with the compute-optimal scaling framework is instructive: TRM has a natural mechanism for dynamic allocation (the Q-head + deep supervision loop) but uses it only for training-time early stopping, not test-time compute allocation. This represents a missed opportunity that could substantially improve TRM's test-time efficiency or accuracy.

Training Is Brittle and Sensitive to Optimization Details That the Paper Does Not Fully Explain

Several of the paper's reported failures — documented in the Appendix's "Ideas that failed" section — involve attempts to modify the architecture that resulted in "massive" or "significant" generalization drops without specific performance numbers. Mixture-of-Experts "decrease[d] massively"; weight tying between input embeddings and output head led to a "massive generalization drop"; removing ACT caused an unspecified but "significant" drop; TorchDEQ fixed-point iteration achieved "worse generalization." The successful configuration required a specific combination of EMA stabilization (0.999 decay), stable-max loss, 2 layers (not 4, not 1), exactly 2 latent features (not 1, not 7), and the BCE halting variant rather than Q-learning or no ACT at all.

The consequence is that practitioners attempting to adapt TRM to new tasks face a fragile optimization landscape where small changes to architecture or training procedure can cause catastrophic failure. The paper's ablations (Table 1) document this fragility: removing EMA costs 7.5 percentage points; moving from 2 to 4 layers costs 7.9 points; using the 1-step gradient approximation costs 30.9 points. These are not small differences — they represent the difference between state-of-the-art and near-zero performance. Yet the paper provides limited theoretical understanding of why these specific choices work. The explanation for 2 layers over 4 layers is "overfitting," but this does not explain why 1 layer (not tested) might fail, why the overfitting manifests as poor generalization rather than poor training accuracy, or why EMA, specifically at 0.999, prevents the divergence. The paper's honest acknowledgment — "the question of why recursion helps so much compared to using a larger and deeper network remains to be explained; we suspect it has to do with overfitting, but we have no theory to back this explanation" (Section 6) — highlights that the empirical success is ahead of the theoretical understanding.

This brittleness has a second consequence: the paper's headline results may not replicate robustly across different random seeds, hardware configurations, or minor hyperparameter variations. The paper does not report results over multiple seeds, and the single-reported numbers provide no information about variance. Given the demonstrated sensitivity to design choices, it is plausible that two independent implementations of TRM following the paper's description — but making slightly different choices about initialization, batch ordering, or numerical precision — could produce meaningfully different accuracy numbers. The EMA decay of 0.999 is specified, but the paper does not report whether results are sensitive to this value (e.g., 0.99 vs. 0.999 vs. 0.9999). The example in Figure 6 — a pretrained model producing a Sudoku grid with one cell wrong — suggests that even near-convergence leaves the model vulnerable to single-cell errors, which an exact-match metric counts as complete failure.

Inference Cost and Latency Are Unaccounted For, and the Parameter Efficiency Claim Is Asymmetric

The paper's central narrative — that TRM achieves strong performance with "only 7M parameters, less than 0.01% of the parameters" of frontier LLMs — emphasizes parameter count as the primary efficiency metric. However, parameter count is a poor proxy for inference cost in a recursive architecture. At test time, TRM executes 16 supervision steps × 3 recursion processes per step × 7 network evaluations per process = 336 forward passes through the 2-layer network. Each forward pass involves the full transformer computation (attention or MLP-Mixer, SwiGLU feed-forward, RMSNorm), meaning the total FLOPs per inference are 336× the cost of a single forward pass. A 7M-parameter transformer with $D = 512$ and $L \leq 900$ requires roughly $2 \cdot 7 \times 10^6$ FLOPs per forward pass (the standard 2× multiplier for the backward pass does not apply at inference), so total inference FLOPs are approximately $336 \cdot 2 \cdot 7 \times 10^6 \approx 4.7 \times 10^9$ FLOPs for Sudoku ($L = 81$) and proportionally more for Maze ($L = 900$).

Compare this to a single forward pass through a 27M-parameter Direct Prediction baseline: $2 \cdot 27 \times 10^6 = 5.4 \times 10^7$ FLOPs for the same sequence length. TRM uses approximately 87× more inference FLOPs than the Direct Prediction baseline of comparable parameter scale. The comparison to LLMs is even more extreme: an LLM with, say, 7B parameters (1000× TRM's count) using a single forward pass for a short grid output requires $2 \cdot 7 \times 10^9 = 1.4 \times 10^{10}$ FLOPs, which is only about 3× TRM's inference cost. The "less than 0.01% of the parameters" framing obscures the fact that TRM's parameter efficiency comes at the cost of repeated computation — TRM trades parameters for sequential operations, which is a legitimate design choice but one that carries latency implications the paper does not discuss.

The latency consequence is particularly acute because the recursion is inherently sequential: each supervision step depends on the $(y, z)$ output of the previous step; within each recursion process, each $z$ update depends on the previous $z$; the $y$ update depends on the final $z$. None of these 336 operations can be parallelized. A standard transformer inference on a 7M-parameter model with $L = 81$ might take ~1 millisecond on modern hardware; TRM's 336 sequential forward passes would take ~336 milliseconds — a 336× wall-clock slowdown regardless of total FLOPs. For latency-sensitive applications, this may make TRM impractical even if its total FLOP budget is competitive.

The paper provides GPU training times (Sudoku: <36 hours on 1× L40S; ARC-AGI: ~3 days on 4× H100) but does not report any inference latency figures. The test-time cost is further amplified by the 1,000-augmentation majority voting used on ARC-AGI, which multiplies the already-sequential 336 forward passes by 1,000, for 336,000 forward passes per test input. While this can be embarrassingly parallelized across augmentations (1,000 independent TRM runs), the total compute is substantial and is not compared to the test-time compute budgets used by LLM baselines (e.g., how many tokens of chain-of-thought does o3-mini-high generate per ARC puzzle?). The paper's comparison to LLMs therefore compares TRM at high test-time compute (implicitly) against LLMs at low test-time compute (zero-shot CoT), which favors TRM on accuracy but obscures the compute parity.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around small-model reasoning from "can we match LLMs by scaling up?" to "can we match LLMs by iterating more intelligently?" The core reframing is not that small models are competitive with large ones on reasoning — that claim has been made before, and the paper's own results show Grok-4 with bespoke test-time compute achieving 79.6% on ARC-AGI-1 versus TRM's 44.6%. Rather, the paper demonstrates that architectural choices about how computation is structured — not just how much computation is performed — can close a disproportionate fraction of the scale gap. A 7M-parameter model surpasses a 671B-parameter model on four hard reasoning benchmarks not by being trained on more data or by having more parameters, but by arranging its limited capacity into a form that supports iterative refinement with full gradient feedback.

This is a genuine diagnostic contribution to the field's understanding of why recursive architectures work, not merely an incremental improvement. Prior to this paper, the Hierarchical Reasoning Model (HRM) had demonstrated that recursion with deep supervision could achieve strong results on puzzle benchmarks, but the mechanism was obscured by layered justifications: biological arguments about temporal frequencies, the Implicit Function Theorem with fixed-point convergence assumptions, and a two-network design that was presented as hierarchical rather than functional. The field's takeaway from HRM could have been "recursion helps, probably for biological reasons, and you need two networks at different frequencies." The paper's careful ablation analysis (Table 1) refutes this interpretation almost point by point. The IFT-based 1-step gradient approximation — presented in HRM as a principled memory-saving technique — turns out to be the primary performance bottleneck, costing 30.9 percentage points on Sudoku-Extreme. The two-network design is not only unnecessary but actively harmful, with a single shared network outperforming separate networks by 5.0 points. The biological justification — hierarchical temporal frequencies — is replaced by a simpler functional interpretation: z stores the reasoning trace, y stores the current answer, and the presence or absence of the question embedding x in the input signals which operation to perform.

This diagnostic reframing has direct consequences for which research directions become more attractive:

More attractive: understanding gradient flow in iterative architectures. The paper's central empirical result — that replacing the 1-step gradient approximation with full backpropagation through the recursion process accounts for the majority of the performance gain — suggests that the quality of the training signal in iterative refinement systems is a first-order concern. Research on truncated backpropagation through time, synthetic gradients, and equilibrium-model training should take note: the paper provides evidence that for small-data reasoning tasks, the approximation matters enormously, and the cost of full backpropagation can be managed by making the per-step network tiny. This opens a line of inquiry into where gradient approximations are safe (the paper shows they are not safe here, but they may be safe for other types of recursion) and how to design architectures that maximize the ratio of effective depth to gradient-truncation error.

Less attractive: scaling for scaling's sake on closed-form reasoning. The paper's LLM baselines at 0.0% on Sudoku-Extreme and Maze-Hard, despite 4–7 orders of magnitude more parameters and training data, are a striking negative result for the view that scale alone — even with chain-of-thought prompting — suffices for structured reasoning. If a 671B-parameter model trained on trillions of tokens cannot solve a single Sudoku puzzle while a 5M-parameter model achieves 87.4%, then the bottleneck on these tasks is not capacity or knowledge but the architectural support for iterative state refinement. This does not mean scaling is irrelevant — Grok-4 with test-time compute achieves 79.6% on ARC-AGI-1, far exceeding TRM — but it means that scaling without structural support for iteration yields dramatically suboptimal returns on investment for constraint-satisfaction and multi-step deduction tasks. Researchers primarily interested in pushing LLM benchmark scores should redirect attention from pretraining scale to inference-time architectures that maintain and refine latent state.

Less attractive: complex multi-module reasoning systems justified by biological analogy. The paper's demonstration that a single 2-layer network outperforms HRM's two-network design undermines the default assumption that different cognitive operations require separate learned modules. The functional reinterpretation — that the task is disambiguated by what information is present in the input, not by which network processes it — provides a simpler design principle: rather than building separate modules for reasoning, verification, and answer generation, design the input representation to encode which operation is needed, and let a shared network learn the routing. This is a refutation of HRM's specific biological framing, but it also challenges a broader research trend toward modular reasoning architectures where separate components debate, critique, or refine each other's outputs. If a single tiny network can achieve strong results by iterating on its own latent state with full gradient feedback, the burden of proof shifts to modular designs to demonstrate that their additional complexity buys something that a unified iterative architecture cannot.

The paper also provides a reconciliation of a tension in the small-data learning literature. The scaling laws literature (Kaplan et al., 2020) established that larger models tend to generalize better given sufficient data, but the practical reality of many reasoning benchmarks (ARC-AGI, Sudoku variations) is that training data is inherently scarce — the whole point is to test generalization from few examples. The paper shows that this tension can be productively resolved not by scaling data or model size, but by reorganizing computation into a recursive form that extracts more learning signal per parameter. The 2-layer TRM at 5M parameters outperforms the 4-layer TRM at 10M parameters (87.4% vs. 79.5%, Table 1) because the 2-layer variant can afford more recursion steps with full backpropagation, providing a finer-grained gradient signal. This is a concrete design principle: when data is limited, push capacity into operations-per-parameter (recursion count) rather than parameters-per-operation (layer count).

Follow-Up Research This Work Enables

Difficulty-conditioned recursion scheduling using the Q-head as an online difficulty estimator. The paper's Q-head already learns to predict answer correctness (q_hat) at each supervision step, and during training it successfully identifies when to halt early. At test time, however, the Q-head is disabled and all 16 supervision steps are run uniformly. A natural extension would be to use q_hat at test time to dynamically allocate recursion depth: if q_hat exceeds some threshold (say, 0.95) at step k, halt early and return the answer; if q_hat remains low after 16 steps, extend the recursion (increase N_sup or increase n or T for the remaining steps). This would turn TRM from a fixed-compute system into an adaptive-compute system along the lines of Snell et al. (2024)'s compute-optimal scaling — but with the crucial advantage that the difficulty signal is generated internally by the model rather than requiring a separate verifier network or expensive pre-computation. A strong follow-up would evaluate the accuracy-vs-compute Pareto frontier on Sudoku-Extreme (with its 423K test examples and naturally varying puzzle difficulty), measuring whether adaptive scheduling achieves equivalent accuracy to fixed scheduling at lower average compute, or higher accuracy at equivalent compute. The key metric would be the area under the accuracy-vs-FLOPs curve, not just the endpoint accuracy.

Combining TRM-style recursion with autoregressive generation for open-ended reasoning. The paper explicitly identifies TRM's limitation to deterministic, closed-form outputs and suggests extending it to generative tasks. The most direct path is to replace the argmax decoding of y with autoregressive token generation conditioned on y, where y serves as a latent "thought vector" that is iteratively refined before each token is emitted. This would combine the benefits of iterative latent refinement (which TRM shows is critical for multi-step reasoning) with the flexibility of autoregressive generation (which allows open-ended text, multiple valid answers, and chain-of-thought). A concrete experiment: fine-tune a small decoder-only transformer (e.g., GPT-2-small, ~124M parameters) such that before generating each reasoning step token, the model first runs k recursive latent updates in embedding space (the TRM recursion block), then emits the next token conditioned on the refined latent. Test this on the MATH benchmark (Hendrycks et al., 2021) or GSM8K (Cobbe et al., 2021) to see whether the iterative latent refinement compensates for the model's small size on mathematical reasoning that requires multiple deduction steps. The baseline would be the same small transformer without latent recursion, and the comparison would quantify how much of TRM's reasoning benefit transfers to the autoregressive setting.

Systematic investigation of the overfitting-depth tradeoff across task difficulties and dataset sizes. The paper hypothesizes that 2-layer networks outperform 4-layer networks because "when data is too scarce and model size is large, there can be an overfitting penalty" (citing Kaplan et al., 2020), but this is an observation, not a controlled experiment. A systematic follow-up would vary training set size (100, 500, 1000, 5000, 10000 Sudoku puzzles), network depth (1, 2, 4, 8 layers), and recursion depth (n from 1 to the memory limit) independently, measuring both training accuracy and test accuracy at each combination. This would produce a phase diagram showing: (a) at what dataset size the "larger network = better" trend re-emerges (i.e., where does overfitting stop being the bottleneck?), (b) whether the optimal recursion depth scales with dataset size (do larger datasets allow deeper per-application networks, or do they primarily allow more recursion at fixed per-application depth?), and (c) whether the 2-layer optimum the paper found is specific to the ~1000-example regime or holds across a wider range. The prediction from the paper's narrative is that the crossover point — where deeper networks become beneficial — moves rightward (larger dataset required) as the task becomes harder, because harder tasks require deeper reasoning which increases overfitting risk at any given dataset size. This experiment would directly address the paper's acknowledged gap of having "no theory to back this explanation."

FLOPs-matched comparison between recursive small models and non-recursive larger models. The paper claims parameter efficiency (7M vs. 27M vs. 671B) but not compute efficiency, because TRM applies its parameters many more times than a single-pass model. A controlled FLOPs-matched experiment would ask: given a fixed training compute budget, should we train a TRM with P parameters and R recursions, or a Direct Prediction model with P' parameters (where P' > P to account for TRM's recursive FLOPs overhead), all other factors held constant? For example, if TRM-2L-T3-n6 at 5M parameters uses approximately 336 × 2 × 5 × 10^6 = 3.36 × 10^9 FLOPs per training example, what accuracy would a Direct Prediction model achieve if trained with the same total FLOPs (which would allow roughly 62× more parameters, or 310M, assuming similar single-pass FLOPs-per-parameter)? This experiment would reveal whether recursion is genuinely more compute-efficient than depth, or merely more parameter-efficient — two claims the paper conflates. The prediction from the paper's results (Table 1: 4-layer TRM at 10M gets 79.5% vs. 2-layer at 5M gets 87.4%) is that recursion dominates depth even at FLOPs parity, but this has not been demonstrated.

Stress-testing the "exactly 2 latents" principle on tasks requiring external memory or multi-step lookahead. The paper's functional interpretation — y stores the answer, z stores the reasoning trace — and the ablation showing that exactly 2 latent features is optimal (Table 2) are compelling for Sudoku, where the answer is a completed grid and the reasoning trace captures constraint propagation. But what happens on tasks that require remembering intermediate results that cannot be encoded in a single fixed-size embedding? Consider the game of Sokoban (push boxes to targets), where planning requires tracking the positions of multiple movable objects and simulating "what if I push box A before box B?" scenarios. A single z of dimension 512 may not have sufficient capacity to store the branching search tree. A strong follow-up would test TRM on planning tasks of increasing state complexity, measuring whether the optimal number of latent features increases with the number of independent objects or subgoals that must be tracked. If 2 latents remains optimal even on complex planning, that suggests the 2-latent decomposition captures a deep property of iterative refinement — distinct "what" vs. "how" representations — rather than being a capacity artifact of the Sudoku task. If more latents become necessary, that maps out the boundary conditions of the paper's "less is more" principle.

Application to program synthesis and algorithmic reasoning with execution feedback. The paper's benchmarks (Sudoku, Maze, ARC-AGI) all involve producing a single correct output from constraints, but they lack an interactive component: the model cannot execute its partial answer, observe the result, and use that as additional input for the next recursion step. A natural extension would integrate an execution engine into the recursive loop: after the model produces a candidate answer y, execute it (e.g., run the generated code, check the Sudoku constraints violated, measure the maze path length) and feed the execution result back as an additional signal concatenated with x for the next recursion process's z updates. This transforms TRM from a purely latent reasoning system into one that can use external verification to correct errors, similar to how AlphaGo uses rollouts to evaluate board positions. A concrete experiment: on the APPS or MBPP code generation benchmarks, have TRM generate a program, execute it against test cases, embed the test results (pass/fail per test) as a vector added to x, and recursively refine the program embedding y based on which tests failed. The prediction: execution feedback should disproportionately improve performance on hard problems (where latent reasoning alone is insufficient) and provide a natural halting signal (halt when all tests pass, rather than when the Q-head estimates correctness). This would test whether TRM's architecture is specifically suited to tasks with verifiable outputs, or whether it can also incorporate the kind of environmental feedback that makes search-based approaches like AlphaZero powerful.

Practical Applications and Downstream Use Cases

On-device puzzle and constraint-satisfaction systems for mobile applications. The paper's 5M-parameter TRM-MLP achieves 87.4% on Sudoku-Extreme with a model that fits comfortably in ~20MB of memory (5M parameters × 4 bytes/parameter in float32). This makes it feasible to deploy a state-of-the-art Sudoku solver entirely on-device — a mobile phone, tablet, or embedded system — with no network connectivity and no API calls to cloud-based LLM services. The inference cost, while higher than a single-pass model (336 forward passes per puzzle), is still well within the capabilities of modern mobile processors: a 5M-parameter model with 2 layers requires roughly 3.4 × 10^9 FLOPs per puzzle, which a modern smartphone NPU can execute in under 100ms. The broader application class includes any constraint-satisfaction task with a fixed input-output grid format: logic puzzles (Kakuro, Nonograms, KenKen), scheduling problems with small state spaces, or automated form-filling from incomplete information. The key requirement — a deterministic mapping from constrained input to unique output — is restrictive, but when it holds, the combination of tiny model size and high accuracy makes TRM a practical alternative to both cloud-based LLMs (expensive, slow, privacy-compromising) and hand-coded solvers (brittle, domain-specific, must be rewritten for each puzzle type).

Training data generation for self-improvement loops on reasoning tasks. TRM's ability to solve hard puzzles from small training sets suggests a bootstrapping application: use TRM to generate high-quality solutions for additional unlabeled puzzle instances, then use those solutions as training data to train a larger or different model. This is the self-improvement pipeline that the Snell et al. (2024) paper proposed for LLMs, but executed here with a supervised model on structured tasks. A concrete workflow: train TRM on the 1,000 labeled Sudoku-Extreme examples, achieving 87.4% test accuracy. Use the trained TRM to generate solutions for 100,000 new, unlabeled, challenging Sudoku puzzles (which are cheap to generate automatically using puzzle generators like tdoku). Filter for solutions that violate no Sudoku constraints (an automatic check, no human labeling needed). Use these ~87,400 correct solutions as additional training data to train a larger model (e.g., a 4-layer TRM variant, or a standard transformer with a different architecture) that might benefit from the extra data. The key advantage over using LLMs for this pipeline: TRM's per-puzzle inference cost is predictable and moderate (336 fixed forward passes, no token sampling), and the accuracy is high enough that the generated data is overwhelmingly correct, minimizing label noise. The paper's 87.4% accuracy translates to roughly 7 in 8 generated puzzles being perfectly correct — a higher yield than most LLM-based data generation pipelines for reasoning tasks.

Lightweight verifier and refinement module for larger generative models. TRM's architecture — a tiny network that iteratively refines an answer embedding y while maintaining a latent reasoning trace z — can be deployed as a post-processing module for a larger autoregressive model. A large LLM generates a candidate answer (as an embedding or token sequence); TRM takes that answer as its initial y, runs 16 supervision steps of recursive refinement, and produces a (potentially corrected) answer. This decouples the knowledge-intensive part of the task (where LLMs excel, due to their massive pretraining corpora) from the algorithmic refinement part (where small recursive models excel, as demonstrated on Sudoku and Maze). On ARC-AGI-1, an LLM might generate a plausible but incorrect output grid by pattern matching; TRM, trained on ARC-AGI data, could take that grid as a starting point and iteratively refine it using its learned transformation rules, potentially correcting errors the LLM could not detect autoregressively. The key implementation question is whether TRM's supervised training on puzzle tasks transfers to refining LLM-generated outputs that may have different error patterns than TRM's own training trajectories. The paper's observation that HRM's PRM struggled with distribution shift on revision model outputs (the Snell et al. paper's Figure 15a finding) suggests this transfer may not be trivial, but the approach is testable by fine-tuning TRM specifically on (LLM-generated-candidate, ground-truth) pairs for a target benchmark.