Universal YOCO for Efficient Depth Scaling

where `$\circ$` denotes function composition (the output of `$LS_1$` becomes the input to `$LS_2$`, and so on), and each `$LS_i$` is a single layer consisting of efficient self-attention followed by a SwiGLU feed-forward network:

```$$ Y^l = \text{ESA}(\text{LN}(X^l)) + X^l $$
```$$ X^{l+1} = \text{SwiGLU}(\text{LN}(Y^l)) + Y^l $$

where `$\text{LN}(\cdot)$` is RMSNorm (Root Mean Square Layer Normalization, from Zhang and Sennrich, 2019), `$\text{ESA}(\cdot)$` denotes efficient self-attention, and `$\text{SwiGLU}(X) = (\text{swish}(X W_G) \odot X W_1) W_2$` is the gated feed-forward block.

**What it computes:** Each Self-Decoder layer takes a sequence representation `$X^l$`, normalizes it, applies efficient self-attention (details below) with a residual connection to produce `$Y^l$`, then normalizes again, applies a gated feed-forward transformation, and adds another residual connection to produce the next layer's input `$X^{l+1}$`. After passing through all `$L/2$` layers, the output is a transformed representation of the input sequence.

**Why this form:** The Self-Decoder uses efficient attention (not full attention) because its role is local representation refinement — each token only needs to integrate information from nearby tokens, not the full sequence. The SwiGLU activation (a gated variant of Swish) is used because it has become standard in LLMs, empirically outperforming ReLU-based FFNs. The residual connections prevent vanishing gradients through depth. RMSNorm is used instead of LayerNorm because it is faster (no mean subtraction) and works equally well in practice.

##### Universal Self-Decoder (Recursive Computation)

The Universal Self-Decoder takes the single-pass Self-Decoder and executes it `$T$` times successively with shared parameters:

```$$ \text{USD}(X) = \underbrace{\text{Self-Decoder}^{L/2} \circ \cdots \circ \text{Self-Decoder}^{L/2}}_{T \text{ iterations}} (X) $$

**What it computes:** Given initialized embeddings `$X_0$`, the first pass through the Self-Decoder (iteration 1) produces `$X^{(1)}$`. This output becomes the input to the **same** Self-Decoder for iteration 2, producing `$X^{(2)}$`, and so on for `$T$` total iterations. The final output `$\text{USD}(X) = X^{(T)}$` is used to generate the global KV cache. No additional parameters are introduced; the `$T$` iterations share the exact same weight matrices and do not increase the model's parameter count.

**Why this form:** Recursive computation with shared parameters follows the Universal Transformer pattern (Dehghani et al., 2018). The key insight is that each iteration can be viewed as a step of iterative refinement — the model can revisit and improve its intermediate representations given the same context, analogous to how a human might re-read a sentence to extract deeper meaning. The shared weights force the model to learn a transformation function that is useful at multiple stages of refinement, acting as a form of implicit regularization. The alternative — adding `$T$` separate non-recursive layers — would multiply the parameter count by `$T$`, dramatically increasing training cost and memory requirements.

The paper sets `$T=3$` as the default, which results in approximately `$2\times$` the total FLOPs of the non-recursive baseline (the paper states this explicitly: "we default to looping the Self-Decoder 3 times, resulting in 2× the total FLOPs of the non-recursive baseline" in Section 4 preamble). This is because the Self-Decoder represents roughly half the model's total layers, so looping it 3 times means the Self-Decoder contributes 3 times its single-pass FLOPs while the Cross-Decoder contributes its single-pass FLOPs, netting approximately `$(3 + 1) / (1 + 1) = 2\times$` total FLOPs.

##### Global KV Cache Generation

After the Universal Self-Decoder completes its `$T$` iterations, the final output `$\text{USD}(X)$` is normalized and linearly projected to produce two matrices — the keys `$\hat{K}$` and values `$\hat{V}$` for cross-attention:

```$$ \hat{K} = \text{LN}(\text{USD}(X)) W_K $$
```$$ \hat{V} = \text{LN}(\text{USD}(X)) W_V $$

where `$W_K, W_V \in \mathbb{R}^{d_{\text{model}} \times d_{\text{model}}}$` are learnable parameter matrices.

**What it computes:** The normalized output of the Universal Self-Decoder (a `$|x| \times d_{\text{model}}$` matrix representing every input token's refined representation) is multiplied by two weight matrices to produce two `$|x| \times d_{\text{model}}$` matrices — `$\hat{K}$` and `$\hat{V}$`. These are **global** in the sense that they encode information about every token in the sequence, and they are **shared** across all Cross-Decoder layers.

**Why this form:** The critical design choice is that `$\hat{K}$` and `$\hat{V}$` are produced **exactly once**, after the final recursive iteration, rather than being regenerated at each iteration. This is what decouples recursion depth from KV cache size — the global cache's size is `$O(N d_{\text{model}})$`, independent of `$T$`. If instead the Self-Decoder produced new global KV caches at each iteration (or if, as in standard Transformer looping, each recursive iteration regenerated all per-layer caches), the memory cost would scale with `$T$`. The shared KV cache design is inherited from the original YOCO architecture and is not novel to YOCO-U, but it is the property that makes recursive Self-Decoder computation viable.

##### Cross-Decoder

The Cross-Decoder is a stack of `$L/2$` layer modules that take the Universal Self-Decoder's output (before the KV projection) and the global KV caches as inputs:

```$$ \text{Cross-Decoder}^{L/2}(X, \hat{K}, \hat{V}) = LC_{L/2} \circ \cdots \circ LC_1 (X, \hat{K}, \hat{V}) $$

Each Cross-Decoder layer computes:

```$$ Y^l = \text{Attention}(\hat{Q}^l, \hat{K}, \hat{V}) + X^l $$
```$$ X^{l+1} = \text{SwiGLU}(\text{LN}(Y^l)) + Y^l $$

where `$\text{Attention}(\cdot)$` is standard multi-head attention (Vaswani et al., 2017), and the per-layer queries are computed as `$\hat{Q}^l = \text{LN}(X^l) W^l_Q$` with `$W^l_Q \in \mathbb{R}^{d_{\text{model}} \times d_{\text{model}}}$` being a **layer-specific** learnable matrix.

**What it computes:** Each Cross-Decoder layer takes its input `$X^l$`, normalizes it, projects it into a query `$\hat{Q}^l$`, then computes standard scaled dot-product attention using `$\hat{Q}^l$` as queries and the **same** `$\hat{K}, \hat{V}$` (from the global cache) as keys and values. The attention output is added back via a residual connection, normalized, passed through a SwiGLU feed-forward block, and residually added to produce the next layer's input `$X^{l+1}$`. This means every Cross-Decoder layer attends to the full input sequence through the shared global cache, but each layer can ask different questions (via its unique query projection `$W^l_Q$`) about that representation.

**Why this form:** Standard Transformer decoders compute per-layer self-attention, meaning each layer has its own KV cache and must attend to the full sequence. YOCO-U replaces this with cross-attention to a single shared cache, reducing memory from `$O(L d_{\text{model}} N)$` to `$O(d_{\text{model}} N)$`. The per-layer query projections `$W^l_Q$` preserve some layer-specific expressiveness — even though keys and values are shared, each layer can extract different information by querying differently. The Cross-Decoder uses **NoPE** (no position encoding, from Yang et al., 2025) rather than RoPE or learned positions. The paper states (Section 3.3): "We use NoPE position embedding in Cross-Decoder to enhance global retrieval capability." The rationale (implied, not elaborated) is that NoPE may improve the model's ability to attend to tokens based purely on content rather than position, which is beneficial for retrieval-style cross-attention over a fixed global cache. Meanwhile, the Self-Decoder uses **RoPE** (rotary position embedding, from Su et al., 2021), which encodes relative position information into the attention computation itself — useful for local, order-sensitive processing.

After the final Cross-Decoder layer produces the output `$Y$`, a softmax classifier over the vocabulary performs next-token prediction.

---

#### Efficient Self-Attention in the Universal Self-Decoder

The Universal Self-Decoder's layers use **efficient self-attention (ESA)** rather than standard quadratic self-attention. The paper is flexible about the specific efficient attention mechanism but defaults to **sliding-window attention (SWA)** from Child et al. (2019) — what they call their "default for its simplicity and engineering stability."

**Sliding-window attention** restricts each token's attention to a fixed window of `$W$` preceding tokens. Specifically, for token at position `$i$`, its query attends only to keys and values from tokens at positions `$i-W+1$` through `$i$` (with appropriate handling for the initial tokens where `$i < W$`). This means:

- **Computational cost per token:** `$O(W)$` rather than `$O(N)$`, where `$W$` (e.g., 512) is typically much smaller than `$N$` (e.g., 128K for long contexts).
- **KV cache per self-decoder layer per iteration:** `$O(W d_{\text{model}})$` rather than `$O(N d_{\text{model}})$`, since only the most recent `$W$` tokens' KV pairs need to be stored.
- **Prefilling cost:** `$O(N W)$` total for the full sequence rather than `$O(N^2)$`, because each of `$N$` tokens attends to at most `$W$` others.

In the recursive setting, each of the `$T$` iterations through the Self-Decoder maintains its own sliding-window KV cache — this is the `$WTL$` term in the complexity analysis — but since `$W \ll N$`, this overhead is negligible compared to the global cross-attention cache's `$O(N d_{\text{model}})$`.

**Alternative efficient attention mechanisms.** The paper explicitly notes compatibility with "Linear attention variants from three generations of subquadratic modeling, such as RetNet (Sun et al., 2023), Mamba (Gu and Dao, 2023), and gated DeltaNet (Yang et al., 2024)." These are state-space or linear-attention models that achieve `$O(1)$` or `$O(\log N)$` per-token complexity through different mathematical formulations. However, the paper claims they "perform similarly to SWA within hybrid architectures" — that is, when already paired with global cross-attention in the Cross-Decoder, the choice among efficient attention variants in the Self-Decoder matters less because the Cross-Decoder provides the global context. This justifies the choice of SWA on engineering grounds (simplicity, stability, well-optimized implementations like FlashAttention) over more exotic alternatives.

**Why sliding-window attention works here but not as a full replacement:** Sliding-window attention alone would prevent the model from attending to distant tokens, limiting long-range reasoning. However, in YOCO-U, this limitation is compensated for by the Cross-Decoder — the Self-Decoder only needs to produce representations that the Cross-Decoder can later retrieve globally from. The Cross-Decoder's cross-attention over the full sequence provides the global information integration. The Self-Decoder's job is more local: refining token representations based on immediate context, resolving local ambiguities, and building progressively more abstract representations through its recursive iterations.

---

#### Recursive Computation: Iterative Refinement Mechanics

The paper's recursive design can be understood as analogous to running a fixed set of transformations multiple times, where each pass refines on the previous pass's output. The key aspects:

**Parameter sharing details.** All `$T$` iterations use the **exact same** weight matrices. This includes the attention projection matrices (`$W_Q, W_K, W_V, W_O$` for each efficient-attention layer), the feed-forward network weights (`$W_G, W_1, W_2$` for each SwiGLU layer), and the normalization parameters. There is no per-iteration conditioning or learned gating — the model learns a transformation function `$\text{Self-Decoder}^{L/2}$` that is equally useful at multiple stages of refinement.

**Input-output flow across iterations.** The first iteration processes the raw embeddings `$X_0$`. Each subsequent iteration processes the output of the previous iteration. Critically, the input to the Self-Decoder at iteration `$t+1$` is the full output representation from iteration `$t$` — there is no explicit mechanism for passing information about which iteration we're on (no iteration embeddings, no position encoding updates). The model must implicitly learn to produce representations that are meaningful as both outputs (to be read by the Cross-Decoder) and inputs (to be refined by the next iteration).

**Training behavior.** The paper reports that "the training process of YOCO-U exhibits high stability with a smooth loss trajectory and no significant spikes across the entire training regime" (Section 4.1). This is notable because recursive architectures with shared weights can sometimes suffer from training instability — gradients flowing through multiple iterations of the same parameters can explode or vanish, similar to recurrent neural network training challenges. The paper attributes stability partially to the **partial** nature of the recursion (only half the layers are looped, reducing the effective recurrence depth) and partially to architectural choices like RMSNorm and residual connections.

**Why this particular recursion pattern was chosen over alternatives:**

The paper's ablation studies (Section 4.3, Table 5) systematically rule out other recursion configurations:

1.  **"Upper Loop" (Cross-Decoder recursion):** Average accuracy drops from 48.3 (YOCO-U) to 47.3. This confirms that the Cross-Decoder's role — global information retrieval — does not benefit from iterative refinement in the same way local representation processing does. The authors connect this to findings from Koishekenov et al. (2025, "Encode-Think-Decode"): "the final layers behave like a final decoder" — that is, the deepest layers are specialized for output generation and gain little from iteration. Recursion is beneficial for "thinking" (refining intermediate representations) but not for "decoding" (producing output-ready states).

2.  **"Upper Loop w/o Shared KV":** Dropping from 47.3 to 46.4 when the Cross-Decoder's recursion also regenerates its own KV cache (using self-attention instead of the shared cache). This confirms that the shared global cache is essential for memory efficiency and that breaking the "cache once" property during recursion is harmful.

3.  **"Deep (Instead of Wide)":** A non-recursive model with double the depth but the same parameter count (1792 hidden dimension, 40 layers instead of 2560 hidden dimension, 20 layers) achieves 46.9 — comparable to the 20-layer non-recursive YOCO at 47.0, showing that depth alone (without recursion) at fixed parameters doesn't improve performance significantly; the model width matters. YOCO-U's recursive depth provides a "third axis" — it increases effective depth without trading off width or parameters.

4.  **"Deeper (Instead of Wide)" with YOCO-U:** When YOCO-U is applied to the deeper (40-layer) layout, it achieves 48.6 — a slight improvement over the standard YOCO-U (48.3), showing that recursive computation benefits are largely orthogonal to the base depth-width tradeoff.

---

#### Inference Complexity: Why YOCO-U Works Where Standard Looping Fails

Table 1 in the paper provides the formal complexity analysis. Let's break this down term by term and explain what each cost component physically corresponds to in a deployment scenario.

The variables: `$N$` is sequence length, `$L$` is total layers, `$D$` is hidden dimension, `$T$` is loop iterations, and `$W$` is the sliding-window size.

##### KV Cache Memory

- **Standard Transformer:** `$O(LND)$`. Every one of the `$L$` self-attention layers stores key-value pairs for every token position (in practice, per head, so multiplied by `$d_{\text{head}}$` but the `$O$` captures the scaling). For a 20-layer model with 128K context and 2560 hidden dimension, this is `$20 \times 128K \times 2560 \approx 6.55$` billion values, stored typically in float16 (≈13 GB for a single sequence).

- **Loop / Universal Transformer:** `$O(LTND)$`. Each of the `$T$` recursive iterations generates a new set of KV caches for all `$L$` layers, then keeps them in memory (since autoregressive decoding at the next token position needs to attend to all previous token representations across all iterations). For `$T=2$`, this doubles the already-large memory footprint.

- **YOCO:** `$O((N + WL)D)$`. The `$N$` term is the single global KV cache produced by the Self-Decoder and reused by the Cross-Decoder — it scales with full sequence length but there's only one copy. The `$WL$` term is the local sliding-window caches maintained by each of the `$L/2$` (or more generally `$L$`, though YOCO splits to half) Self-Decoder layers — each stores only the last `$W$` tokens' KV pairs. Since `$W \ll N$`, the `$WL$` term (e.g., `$512 \times 20 = 10{,}240$`) is dwarfed by the `$N$` term (e.g., `$128{,}000$`).

- **YOCO-U:** `$O((N + WTL)D)$`. The `$N$` term remains identical — the global cache is produced once regardless of `$T$`. The `$WTL$` term accounts for the fact that each iteration through the Self-Decoder maintains its own local sliding-window cache. For `$T=3$` with `$W=512$`, this is `$3 \times 512 \times 20 = 30{,}720$` — still minuscule compared to the global cache's `$128{,}000$`. The key insight: recursion only multiplies the already-small `$WL$` term, not the dominant `$N$` term. This is why the paper calls the additional overhead "negligible for long sequences" and why Figure 7c shows YOCO-U's KV cache curve essentially overlapping with non-recursive YOCO.

##### Prefilling Time

Prefilling is the initial pass where the model processes the entire input sequence in parallel to populate KV caches and produce the first output token.

- **Standard Transformer:** `$O(LN^2 D)$`. Each of `$L$` layers performs full self-attention, which is quadratic in sequence length.

- **YOCO:** `$O(\frac{L}{2} ND)$`. The Self-Decoder's efficient attention is linear in `$N$` (specifically `$O(NWD)$`, but treated as linear for scaling analysis), and the Cross-Decoder's cross-attention is also `$O(\frac{L}{2} ND)$` because attention with a shared KV cache is `$O(N)$` per query — each of the `$N$` token queries attends to `$N$` keys, but this is parallelized. The paper's pre-filling analysis focuses on the dominant term.

- **YOCO-U:** `$O(\frac{L}{2} TND)$`. Each Self-Decoder iteration adds `$O(\frac{L}{2} N W D)$` pre-filling cost, and the Cross-Decoder still costs `$O(\frac{L}{2} ND)$`. The combined scaling is linear in `$N$` and `$T$`. The practical impact (Figure 7a): at 256K context, YOCO-U achieves ~76K tokens/second pre-filling throughput versus ~7.5K for the standard Transformer — a 10× improvement — and versus ~3.7K for RINS — a 20× improvement. The gap widens with context length because YOCO-U's cost is roughly linear while the Transformer's is roughly quadratic.

##### Decoding Time

Decoding is the autoregressive phase where the model generates one token at a time, attending to all previous tokens.

- **Standard Transformer:** `$O(LND)$`. Each layer's self-attention attends to `$N$` keys per generated token.

- **YOCO-U:** `$O(\frac{L}{2}(N + WT)D)$`. The Cross-Decoder's cross-attention costs `$O(\frac{L}{2} N D)$` — attending to `$N$` keys per layer. The Self-Decoder's efficient attention costs `$O(\frac{L}{2} W T D)$` per generated token — attending to at most `$W$` tokens per layer per iteration. For long sequences where `$N \gg WT$`, the Cross-Decoder dominates the cost, and this cost is the same as non-recursive YOCO (since `$T$` does not appear in the Cross-Decoder term). The practical impact (Figure 7b): YOCO-U achieves 303 tokens/second at 256K context versus 137 for standard Transformer (2.21× improvement) and 56 for RINS (5.4× improvement). The gap over non-recursive YOCO (318 tokens/second) is only about 5%, confirming that the Self-Decoder recursion's overhead is minimal in the decoding phase.

The paper provides the raw throughput numbers in Appendix D (Tables 8, 9, 10), which are worth internalizing:

- **Prefilling at 256K:** Transformer: 7,475; YOCO: 220,407; RINS: 3,739; YOCO-U: 76,301 tokens/second.
- **Decoding at 256K:** Transformer: 137; YOCO: 318; RINS: 56; YOCO-U: 303 tokens/second.
- **KV cache at 256K:** Transformer: 10,240 MB; RINS: 20,480 MB; YOCO: 522 MB; YOCO-U: 542 MB.

The RINS numbers are particularly instructive: RINS has the same full-attention quadratic pre-filling as the Transformer but suffers additional degradation from the recursive depth (each looped layer does full attention, and there are effectively 40 layers of computation in the tested configuration). YOCO-U avoids this entirely by only recursing the efficient-attention layers.

---

#### Design Choice Deep Dive: Why Restrict Recursion to Efficient-Attention Shallow Blocks?

The paper makes a specific, non-obvious design choice: recursion is applied to:
1.  **Shallow layers** (the first `$L/2$` layers, the Self-Decoder) rather than deep layers.
2.  **Efficient-attention layers** (sliding-window attention) rather than full-attention layers.

Each of these choices has a justification grounded in both prior work and the paper's own ablations.

**Why shallow layers?** The paper cites two lines of evidence:

First, RINS (Alabdulmohsin and Zhai, 2025) showed that "looping only the shallow modules of a Transformer has been shown to provide improved computational efficiency." This is an empirical finding from prior work: early layers benefit more from recursive refinement than later layers, possibly because early layers are doing more general representation-building while later layers are more specialized for the specific output task.

Second, the paper's own ablation (Table 5, "Upper Loop" vs. YOCO-U) directly tests this: moving recursion from the Self-Decoder (bottom) to the Cross-Decoder (top) drops average accuracy from 48.3 to 47.3. The authors connect this to the Encode-Think-Decode framework (Koishekenov et al., 2025), interpreting the Self-Decoder as an "encoder" that benefits from iterative refinement and the Cross-Decoder as a "decoder" whose role is retrieval and output generation — a fundamentally different operation that gains less from iteration.

**Why efficient-attention layers?** This is the paper's central architectural insight and distinguishes YOCO-U from RINS. The theoretical justification comes from the division of labor between the Self-Decoder and Cross-Decoder:

- The Cross-Decoder handles **global information retrieval** — it can attend to any token in the sequence through the shared KV cache. This is the operation that requires the quadratic full attention (or its cross-attention equivalent).
- The Self-Decoder handles **local representation refinement** — building token representations based on local context, resolving ambiguities, and iteratively improving through multiple passes. This does not require global attention, because the Cross-Decoder will later provide global context.

The empirical justification is comparative: RINS and YOCO-U achieve nearly identical downstream task performance (both 48.3 average in Table 3), but YOCO-U does so with dramatically lower KV cache memory (542 MB vs. 20,480 MB at 256K context) and much higher decoding throughput (303 vs. 56 tokens/second). The representational benefit of recursive computation is obtained regardless of whether the looped block uses full attention or efficient attention — but the **cost** is radically different. By choosing efficient attention for the recursive block, YOCO-U captures the performance benefit while avoiding the cost explosion.

The paper also notes an alternative path not pursued: the Universal Transformer could theoretically use efficient attention in all its layers, but this would degrade global reasoning (no full-attention mechanism). YOCO-U's hybrid design — efficient attention in the recursive part, full cross-attention in the non-recursive part — ensures global reasoning is preserved while recursion is made efficient.

---

#### Training Configuration and Recipe

The paper provides detailed training hyperparameters in Appendix A (Table 6) for the main experiments (Section 4.1). These are quoted verbatim:

Key points to highlight:

**Mixture of Experts (MoE).** The model uses fine-grained MoE with shared experts, following DeepSeekMoE (Dai et al., 2024). With 64 total experts and 8 activated per token, plus one additional shared expert, the total parameter count is 10B but the activated parameter count is 1.3B. This means the model is sparse: during any forward pass, only a fraction of the feed-forward network parameters are used, reducing computational cost per token while increasing total representational capacity. The expert dimension is 1024 (the hidden size of each expert's FFN).

**Training length and data.** Models are trained for 75,000 steps with a batch size of 4 million tokens, totaling 300B training tokens. The training sequence length is 8192 tokens. The optimizer is AdamW (Loshchilov and Hutter, 2019) with `$\beta = (0.9, 0.95)$`, learning rate `$1 \times 10^{-3}$`, and zero warmup steps. Weight decay is set to 0.1, and dropout is 0.0 (no dropout). Training runs on AMD MI300X GPUs.

**Training stability.** The paper emphasizes that "the training process of YOCO-U exhibits high stability with a smooth loss trajectory and no significant spikes across the entire training regime." This is noteworthy because recursive architectures can be harder to train — the repeated application of the same layers amplifies any gradient instabilities. The authors do not provide a loss curve figure, but the qualitative description suggests the architecture is not brittle.

**Positional encoding.** The Self-Decoder uses **RoPE** (rotary position embedding) for its efficient self-attention. The Cross-Decoder uses **NoPE** (no explicit position encoding), relying on the model to infer position information from the content of the global KV cache representations. The paper states that NoPE "enhances global retrieval capability" (Section 3.3), consistent with findings in Yang et al. (2025) that position-less attention can improve retrieval when the positions are already implicitly encoded in the sequence representation.

**Thinking SFT recipe (Appendix B).** For the math reasoning experiments (Section 4.1, Figure 3), models are initialized from the 280B-token checkpoint and trained for an additional 20B tokens with a maximum sequence length of 32,768 tokens. The prompt template uses a system message ("You are a helpful and friendly AI assistant.") and a user message instructing step-by-step reasoning with the final answer in `\boxed{}`. Decoding uses greedy search with maximum generation length of 32,768 tokens.

**Why this training configuration?** The paper's goal is a fair comparison between recursive and non-recursive architectures. The models are identical in hidden size, number of layers, and total steps except that YOCO-U has `$T=3$` Self-Decoder iterations while non-recursive YOCO has `$T=1$`. Since the Self-Decoder represents half the layers, YOCO-U uses approximately 2× the training FLOPs of non-recursive YOCO at the same number of steps. This is the basis for the "Equal FLOPs" comparison: YOCO-U with fewer training tokens can be compared against non-recursive YOCO with more training tokens, holding total computational budget constant. The equal-FLOPs result — YOCO-U with roughly half the tokens achieves lower loss — is the paper's primary evidence that recursive computation improves pretraining efficiency.

---

#### Design Choices: Summary and Justifications

**Recursion on shallow, efficient-attention layers only.** Justified by ablation (Table 5): upper loop degrades performance; efficient attention in recursive blocks matches full attention's representational benefit (Table 3: YOCO-U vs. RINS) at a fraction of the memory and latency cost (Figure 7).

**Sliding-window attention over other efficient attention variants.** Justified by engineering simplicity and the observation that within hybrid architectures (efficient self-attention + global cross-attention), the choice of efficient attention variant has limited impact because the Cross-Decoder provides global context.

**NoPE in Cross-Decoder, RoPE in Self-Decoder.** The Self-Decoder's local processing benefits from explicit relative position encoding (RoPE). The Cross-Decoder's retrieval benefits from content-based attention without position bias (NoPE).

**Fixed $T=3$ iterations.** The default provides a practical balance between additional representational depth and training cost. The paper shows in Figure 6 that more iterations ($T=5$) continue to improve performance, but the default is set to a moderate value for efficiency. The paper does not explore adaptive or per-token iteration counts — this is left to future work.

**Global KV cache produced once after all iterations, not during.** This is the linchpin that makes recursive computation memory-efficient. The alternative (producing new global caches at each iteration) would multiply the dominant memory term `$O(ND)$` by `$T$`, eliminating the advantage over standard Transformer looping.

**Mixture of Experts with shared experts.** This is not novel to YOCO-U (it follows DeepSeekMoE) but enables the paper to demonstrate the approach at a 10B total / 1.3B activated parameter scale, showing the architecture is compatible with sparsity techniques.

## 4. Key Insights and Innovations

### Innovation 1: Recursive Depth Scaling Is Not Inherently Memory-Expensive — The Bottleneck Is *What* You Recurse, Not *That* You Recurse

The dominant assumption in prior work on recursive Transformers — from the Universal Transformer (Dehghani et al., 2018) through RINS (Alabdulmohsin and Zhai, 2025) — was that depth scaling via parameter sharing inevitably comes with a proportional memory tax. Each recursive iteration regenerates KV caches, and since autoregressive decoding must attend to all previous token representations, those caches must be retained in memory. The standard framing was: depth scaling trades memory for representational power, and the tradeoff is roughly linear in the number of iterations. This is the assumption encoded in Table 1's `$O(LTND)$` entry for "Loop / Universal Transformer" — multiply iterations, multiply KV cache memory.

YOCO-U's fundamental diagnostic move is to **question which part of the KV cache is actually scaling with `$T$`.** The insight is disarmingly simple but non-obvious: in a decoder-decoder architecture like YOCO, the KV cache has two components with radically different scaling properties — a global component (cross-attention cache, scales with full sequence length `$N$`) and a local component (self-attention cache, scales with window size `$W$`). Standard Transformers only have the former kind (per-layer self-attention caches, all scaling with `$N$`). Prior recursive approaches — UT, RINS — loop full-attention blocks whose caches all scale with `$N$`, so the memory cost indeed explodes as `$O(LTND)$`.

But YOCO-U's architecture separates these components and applies recursion **only to the efficient-attention half**, whose cache scales with `$W$` (e.g., 512 tokens) rather than `$N$` (e.g., 128K tokens). The global cache — produced once after all recursive iterations complete — is held constant, its size `$O(ND)$` independent of `$T$`. This means the recursion's memory overhead is `$O(WTLD)$`, where `$W \ll N$`, rather than `$O(LTND)$`. The practical consequence is visible in Figure 7c: YOCO-U's KV cache curve (542 MB at 256K context) is essentially indistinguishable from non-recursive YOCO (522 MB), while RINS — a full-attention recursive architecture with comparable downstream performance (48.3 average for both in Table 3) — requires 20,480 MB, a 38× penalty.

This reframing is significant beyond YOCO-U itself because it identifies a **general design principle** for recursive architectures: the memory cost of recursion is not determined by whether you recurse, but by what kind of attention the recursive block uses. If you can structure the architecture so that the memory-intensive components (full-attention, global context) are non-recursive and the recursive components use efficient (subquadratic, limited-context) attention, depth scaling becomes memory-cheap. This principle could generalize to other architectural frameworks — any model that can factor into "global retrieval" and "local refinement" components is a candidate for memory-efficient recursion.

The paper doesn't present this as a formal theorem — it's an architectural design insight supported by the complexity analysis in Table 1 and validated empirically by the identical accuracy of YOCO-U and RINS (Table 3) with radically different memory costs (Figure 7c). It represents a **fundamental reframing** of the depth-scaling-memory tradeoff, not an incremental improvement over prior recursive designs.

---

### Innovation 2: Recursive Computation and Efficient Attention Are Synergistic, Not Competing, Approaches to Inference Efficiency

Before YOCO-U, the landscape of inference-efficient architectures was largely two parallel tracks: **efficient attention mechanisms** (sliding-window, linear attention, state-space models) that reduce the per-token cost of attention, and **recursive/depth-sharing mechanisms** (UT, RINS, Mixture-of-Recursions) that reduce the parameter-to-depth ratio. These were largely pursued independently — papers on linear attention variants rarely discussed depth recursion, and papers on recursive depth rarely incorporated efficient attention (indeed, UT applies recursion to standard full-attention blocks). The implicit assumption was that each approach addressed a different bottleneck: efficient attention tackled the `$O(N^2)$` attention complexity, while recursion tackled the parameter cost of depth.

YOCO-U's central empirical finding — that recursive computation confined to efficient-attention blocks achieves **comparable representational benefit to recursive full-attention blocks** (Table 3: YOCO-U at 48.3 average vs. RINS at 48.3) — demonstrates that these two approaches are not merely additive but **synergistic**. The representational gains from recursive depth (increased effective depth, iterative refinement) do not require full attention in the recursive block; efficient attention suffices. Conversely, efficient attention alone (as in non-recursive YOCO, which already uses sliding-window attention in its Self-Decoder) leaves representational capacity on the table — YOCO-U achieves 4.45 points higher average accuracy than non-recursive YOCO under equal FLOPs (Table 2) by adding recursion to the efficient-attention block, with minimal additional memory.

This synergy is non-obvious because one might reasonably expect that restricting recursion to efficient attention would **limit** the depth-scaling benefit — after all, efficient attention has limited receptive field, and iterating within that limited field might not provide the same kind of representational refinement as iterating over full context. The paper's results suggest this concern is misplaced because of the architectural division of labor: the Cross-Decoder handles global context via full cross-attention over a shared cache, so the Self-Decoder's iterations don't need global receptive field — they refine local representations that will later be globally accessed. Each iteration can focus on resolving local ambiguities and building progressively more abstract token representations without needing to re-establish long-range dependencies (that's the Cross-Decoder's job).

This is a **significant conceptual advance** because it points toward a unified design philosophy for efficient LLMs: **factor the model into a global-retrieval component (full attention, non-recursive) and a local-refinement component (efficient attention, recursive)**. Neither component alone achieves the full efficiency gains; together they unlock a capability-efficiency operating point inaccessible to either approach individually. The paper's language — "a synergistic effect greater than either alone" (Abstract) — is precisely measured, even if not quantified as an interaction term.

---

### Innovation 3: Diagnostic Evidence That Deep and Shallow Layers Play Fundamentally Different Roles, and Recursion Benefits Only One

The paper's ablation study on loop position (Table 5: "Upper Loop" at 47.3 average vs. YOCO-U at 48.3) provides crisp empirical evidence for a claim that has been speculated about but rarely tested so directly: **deep and shallow layers have qualitatively different functions in language models, and recursion is beneficial for the shallow layer's function but not the deep layer's function.** Specifically, recursing the Self-Decoder (bottom `$L/2$` layers) improves performance by ~1.3 points over the non-recursive baseline; recursing the Cross-Decoder (top `$L/2$` layers) improves by only ~0.4 points, essentially noise-level.

This finding connects to the "Encode-Think-Decode" framework (Koishekenov et al., 2025) cited by the authors, where the model's layers are conceptualized as progressing from encoding (building representations) through thinking (refining representations) to decoding (producing output-ready states). Under this framing, recursion is beneficial for the "encode/think" stages but not the "decode" stage — and YOCO-U's architecture aligns with this by making the Self-Decoder recursive and the Cross-Decoder non-recursive. The angular distance analysis in Figure 8 provides representational evidence: the inter-layer similarity patterns are consistent within the Self-Decoder across iterations (suggesting stable refinement), but there is a sharp spike at the interface between Self-Decoder and Cross-Decoder, suggesting a functional transition. The decreasing mean angular distance within the Self-Decoder across iterations hints at diminishing marginal returns — representations approach a fixed point, suggesting the model naturally converges rather than destabilizing.

What makes this insight **distinctive** rather than merely confirmatory is that prior work largely studied layer roles through post-hoc analysis of trained models (probing classifiers, representational similarity metrics) without architectural manipulation. By building an architecture that deliberately applies different computational treatments to different layer groups and showing that the treatment matters — recursing the wrong group erases the benefit — the paper provides causal, not merely correlational, evidence for functional specialization. This has practical implications for architecture design beyond YOCO: if you're going to add recursion, add it to the early-to-middle layers, not the final layers. If you're going to add width via parallel scaling, perhaps add it to the later layers. The paper doesn't explore this second implication, but the diagnostic framework enables it.

---

### Innovation 4: The "Cache Once, Then Recurse" Pattern as a Generalizable Template

The paper's most architecturally innovative contribution — distinct from the specific YOCO-U instantiation — is the **design pattern** of decoupling the generation of a shared, memory-intensive global representation from an iterative, memory-cheap local refinement process. The pattern is: (1) produce a single global KV cache via a pass through efficient-attention layers, (2) iterate over those same efficient-attention layers `$T$` times for representational refinement, (3) use the refined output to generate the final global KV cache, (4) process via non-recursive cross-attention layers that reuse this cache. The critical property is that steps (1-3) can be executed `$T$` times at cost `$O(WT)$` (window-scale), while step (4)'s cost `$O(N)$` (sequence-scale) is independent of `$T$`.

This pattern is not simply "YOCO meets UT" — it's a **structural recombination** that transforms their relationship from independent (or competing) into complementary. In the original YOCO, the Self-Decoder is a single-pass efficient-attention encoder whose output is used once. In the original UT, recursion applies uniformly to all layers with full attention. YOCO-U keeps YOCO's memory advantages and UT's depth-scaling benefits while discarding UT's memory cost and (through ablation) confirming YOCO's single-pass limitation. The recombination is non-trivial because naive recombination — recursing the Cross-Decoder too (Table 5, "Upper Loop") or recursing without shared KV cache ("Upper Loop w/o Shared KV") — destroys the benefit.

The generality of this pattern is suggested by the paper's compatibility with alternatives to sliding-window attention: "Linear attention variants... such as RetNet, Mamba, and gated DeltaNet, are also compatible, though they perform similarly to SWA within hybrid architectures" (Section 3.2). If the global/local factorization holds, any architecture that can be split into a "representation refinement" component (recursive) and a "global retrieval" component (non-recursive) could adopt the "cache once, then recurse" pattern, regardless of the specific efficient attention mechanism used. The paper's recognition of this generality — despite testing only sliding-window attention — elevates the contribution from a specific model design to a **design template** for future architectures.

The significance of this template extends beyond memory efficiency. Because the global KV cache is produced once and never recomputed, the Cross-Decoder's computation is completely decoupled from the recursive depth `$T$`. This means future work can explore **adaptive recursion** — varying `$T$` per token or per sequence — without affecting the Cross-Decoder's cost or cache structure. In a UT or RINS, adaptive depth would dynamically change KV cache memory requirements because each additional iteration would generate new full-attention caches. In YOCO-U, adaptive depth would only change the already-negligible local window caches, making dynamic depth scaling deployment-feasible in a way it isn't for full-attention recursive architectures. The paper doesn't implement adaptive depth, but the template is designed to enable it — a forward-looking contribution.

## 5. Experimental Analysis

### Evaluation Methodology

- **Dataset.** The primary training data is not sourced from a public benchmark but uses a standard large-scale text corpus for language modeling (details unspecified beyond "training tokens"). For downstream evaluation, the paper uses WikiText-103 (language modeling perplexity), LAMBADA (perplexity and accuracy), and the LM Eval-Harness suite covering PIQA, OpenBookQA, HellaSwag, Winogrande, ARC-Easy, ARC-Challenge, MMLU, BBH, GSM8K, HumanEval, and DROP (Section 4.1, Table 2). For math-specific evaluation after thinking SFT, 11 benchmarks are used: GSM8K, MATH, SVAMP, ASDiv, MAWPS, CARP, TABMWP, Gaokao 2023 En, OlympiadBench, CollegeMath, and AMC23 (Section 4.1, Figure 3). Long-context evaluation uses book-level and repository-level code data (Section 4.2, Figure 4), plus the Needle In-A-Haystack retrieval test (Section 4.2, Table 4).

- **Base model(s).** The main experiments (Sections 4.1, 4.3) use a 1.3B activated parameter MoE model (10B total parameters, 64 experts with top-8 activation plus one shared expert, hidden dimension 2560, 20 layers split evenly into 10 Self-Decoder and 10 Cross-Decoder layers). The thinking SFT experiments (Section 4.1, Figure 3) train from a 280B-token checkpoint of this model. The architecture comparison experiments (Section 4.2) use dense 1.3B models (20 layers, hidden dimension 2560, no MoE) to ensure fair comparison across non-recursive Transformer, YOCO, Universal Transformer, RINS, ParScale, and YOCO-U variants. The parameter scaling experiments (Section 4.4) sweep models from 300M to 10.8B parameters with configurations detailed in Appendix C (Table 7). The choice of the 1.3B scale for primary experiments reflects a practical research budget — large enough to exhibit meaningful scaling trends, small enough to train multiple variants and perform ablations.

- **Metrics.** Language modeling performance is measured by **validation perplexity** (lower is better), reported on WikiText-103 and LAMBADA. Downstream task performance uses **accuracy** (fraction correct) for classification/QA tasks and **normalized accuracy** (`acc_n`) for multiple-choice tasks where answer lengths vary (e.g., ARC, HellaSwag). Math reasoning after thinking SFT uses exact-match accuracy with the `\boxed{}` extraction protocol (Appendix B). Long-context modeling uses **perplexity on the last 512 tokens** given varying prefix lengths (Section 4.2, Figure 4). Inference efficiency uses **throughput** (tokens/second) for both prefilling and decoding phases, and **KV cache memory** (MB per sequence) at various context lengths (Section 4.5, Figure 7; Appendix D, Tables 8–10). Training FLOPs are approximated as `tokens × layers` (Section 4.1, Figure 2 left), and parameter scaling plots use total and activated parameter counts (Section 4.4, Figure 5).

- **Baselines.** The paper compares against the following, each with distinct architectural properties:
  1. **Non-recursive YOCO** (Sun et al., 2024): The direct predecessor — identical architecture but with a single-pass Self-Decoder (`T=1`). This is the primary baseline for measuring recursive computation's benefit.
  2. **Standard Transformer** (Vaswani et al., 2017): A dense decoder-only Transformer with RoPE position encoding, 20 layers, full self-attention throughout. Represents the conventional architecture.
  3. **Universal Transformer (UT)** (Dehghani et al., 2018): Standard Transformer with the entire 20-layer block looped 2 times (40 effective layers, same parameter count). Tests full-model recursion.
  4. **RINS** (Alabdulmohsin and Zhai, 2025): Standard Transformer split into a 10-layer non-recursive block and a 10-layer recurrent block looped 3 times (40 effective layers). Tests early-layer recursion in full-attention Transformers.
  5. **ParScale** (Chen et al., 2025): Parallel scaling method using different KV cache prefixes across parallel branches, achieving 2× compute scaling factor. Tests width-based rather than depth-based compute scaling.

- **Generation budget / compute accounting.** The paper uses multiple forms of compute accounting depending on the comparison. For training comparisons (Section 4.1, Figure 2), total FLOPs are estimated as `tokens × layers` — this accounts for the fact that YOCO-U with `T=3` Self-Decoder iterations processes each token through more effective layers than non-recursive YOCO at the same step count. The "Equal FLOPs" comparison in Table 2 trains YOCO-U with fewer tokens to match the total computational budget of non-recursive YOCO. The "Equal Steps" comparison trains both for the same number of optimization steps (75k steps, 300B tokens). For the architecture comparison (Section 4.2), all recursive/parallel models are configured to have approximately 2× the FLOPs of the standard 20-layer Transformer — UT achieves this by 2× looping of all 20 layers (40 effective), RINS by 10 static + (3 × 10 looped) = 40 effective layers, ParScale by 2× parallel branching, and YOCO-U by `T=3` Self-Decoder iterations. For parameter scaling (Section 4.4), models are trained for a fixed 20k steps (20B tokens), and performance is plotted against model parameter count — the fair comparison is at equal training steps rather than equal FLOPs, since the goal is to see how performance scales with parameters given a fixed training recipe.

- **Cross-validation / statistical protocol.** The paper does not report formal cross-validation or statistical significance testing. The token scaling experiments (Figure 2) measure validation loss every 20B tokens and remove one outlier point per model, but no confidence intervals are shown. The downstream evaluations (Table 2, Table 3, Figure 3) use standard benchmark test sets with single evaluation runs. The long-context perplexity measurements (Figure 4) and inference efficiency benchmarks (Figure 7, Appendix D) appear to be single-run measurements on fixed test data or synthetic workloads. The parameter scaling experiments (Figure 5) use a consistent training recipe across model sizes with a single training run per configuration. This is a limitation — without multiple training runs or bootstrap confidence intervals, the reported performance gaps (e.g., +4.45 average for YOCO-U over YOCO in Table 2) cannot be assessed for statistical reliability, particularly given the relatively small test sets for some benchmarks.

### Main Quantitative Results

#### Language Modeling and Downstream Tasks Under Recursive vs. Non-Recursive YOCO

The primary result comparing recursive and non-recursive YOCO at the 1.3B activated parameter scale (10B total MoE parameters) is a validation loss reduction and downstream accuracy improvement that holds under both equal-FLOPs and equal-training-steps accounting.

**Token scaling and loss.** Figure 2 (left) plots validation loss against training FLOPs (estimated as `tokens × layers`). YOCO-U achieves a lower loss at the same FLOPs budget, quantified as `ΔL = 0.033` — the gap between the non-recursive YOCO curve and the YOCO-U curve at equal FLOPs. Since the non-recursive baseline's final loss is approximately 2.25 (reading from the figure), this ~0.033 reduction represents a meaningful but modest improvement. Figure 2 (right) plots loss against training tokens, showing that YOCO-U requires approximately 62% fewer tokens to reach comparable performance — specifically, "YOCO-U trained with 80B tokens is comparable with non-recursive YOCO with 210B tokens." This is a substantial efficiency gain: the recursive model achieves in 80B tokens what the non-recursive model needs 210B tokens to reach, suggesting that recursive computation makes each training token more informative for learning.

**Downstream task performance.** Table 2 reports accuracy across 8 benchmarks comparing YOCO (non-recursive, `T=1`) and YOCO-U (`T=3`) trained for 300B tokens. Two comparisons are presented: "Equal FLOPs" (where YOCO-U is trained with fewer steps to match total compute) and "Equal Steps" (both trained for 75k steps). The average across all benchmarks is:
- Non-recursive YOCO: 41.78
- YOCO-U (Equal FLOPs): 46.23 (+4.45)
- YOCO-U (Equal Steps): 47.08 (+5.30)

Several per-task details are notable. GSM8K (math word problems) shows the largest absolute gain: from 38.06 (non-recursive) to 50.49 (YOCO-U Equal FLOPs) and 50.57 (Equal Steps) — a 12+ point jump. This aligns with the intuition that recursive computation benefits multi-step reasoning tasks more than simple pattern recognition. DROP (reading comprehension with discrete reasoning) also shows a substantial gain: from 32.62 to 34.94 (Equal FLOPs) and 38.07 (Equal Steps). The gains on knowledge-heavy tasks (MMLU: 49.59 → 54.63/55.63; ARC-C: 46.50 → 47.87/48.72) are more modest, consistent with the idea that recursion primarily enhances reasoning depth rather than knowledge storage. HumanEval (code generation) shows only modest improvement (9.15 → 10.98/10.37), possibly because the benchmark's relatively short code generation tasks don't tax representational depth heavily.

**Thinking SFT (math reasoning after specialized training).** Figure 3 compares YOCO and YOCO-U across 11 math benchmarks after training an additional 20B tokens on math thinking data. YOCO-U outperforms YOCO on **all 11 benchmarks** with an average accuracy gain of 24.4% (from 51.6% to 67.1% average). The gains are particularly dramatic on harder benchmarks: OlympiadBench (5.0% → 29.5%), CollegeMath (9.7% → 34.4%), and AMC23 (32.9% → 42.7%). Even on benchmarks with already-high performance (MAWPS: 95.7% → 95.7% — effectively saturated), YOCO-U is not worse. This result supports the paper's claim that "the improvement on explicit and latent reasoning is orthogonal" — the architectural benefit from recursive computation observed in general pretraining also manifests after specialized math training, and the effects appear to compound. The paper explicitly notes this orthogonality: "recursive computations improve the accuracy in next-token prediction, explicit test-time scaling solves the difficult problem from the intrinsic long-reasoning capability of training data."

#### Architecture Comparison: Recursive YOCO vs. Recursive and Parallel Transformer Variants

Table 3 compares 1.3B dense models (no MoE) across non-recursive baselines (Standard Transformer, YOCO) and recursive/parallel variants (UT, ParScale, RINS, YOCO-U), all trained for 20B tokens with approximately 2× FLOPs for the scaled variants. The key comparisons:

**YOCO-U vs. non-recursive YOCO.** YOCO-U achieves 48.25 average vs. 46.95 for YOCO (+1.3 points). This is a smaller gap than in the MoE-based main experiments (+4.45 in Table 2), likely because these models are dense (no expert sparsity) and trained for only 20B tokens rather than 300B, so the benefits of recursion may not fully materialize with limited training. On language modeling, YOCO-U's WikiText-103 perplexity (21.01) improves over YOCO (22.25), and LAMBADA perplexity (18.32 vs. 18.30) is essentially tied.

**YOCO-U vs. RINS (recursive full-attention Transformer).** Both achieve 48.3 average — identical aggregate performance. Per-task differences are small and likely within noise: RINS does slightly better on ARC-C (39.9 vs. 37.0), YOCO-U slightly better on PIQA (68.7 vs. 69.4) and LAMBADA accuracy (41.2 vs. 39.4). This is the paper's central comparative claim: recursive computation with efficient attention (YOCO-U) provides **equivalent representational benefit** to recursive computation with full attention (RINS), but with dramatically lower inference costs (as shown in Figure 7). The parity in downstream performance is the necessary precondition for the efficiency argument — if YOCO-U underperformed RINS, the memory/latency advantages would be a tradeoff rather than a pure win.

**YOCO-U vs. Universal Transformer (full-model recursion).** YOCO-U (48.3 average) slightly edges out UT (47.8). This is consistent with the paper's ablation finding that recursion on later layers provides diminishing returns — UT loops all layers, while YOCO-U (and RINS) loop only the early half, and the early-layer-focused approaches perform better. However, the gap (0.5 points) is modest given the benchmark suite used.

**Recursive vs. parallel scaling.** ParScale (46.8 average) underperforms all recursive approaches — UT (47.8), RINS (48.3), YOCO-U (48.3). The paper's interpretation is that parallel scaling "does not increase modeling depth, and usually achieves less improvements than recursive scaling under same FLOPs" (Section 4.2). This is consistent with the idea that additional depth — even through shared parameters — provides qualitatively different representational capacity than additional width. ParScale's relatively stronger performance on ARC-C (38.4) and Winogrande (55.4) compared to its average suggests that width-scaling might be more effective for knowledge-intensive tasks than for reasoning-intensive ones, though the paper does not explore this hypothesis.

#### Long-Context Modeling and Retrieval

Figure 4 shows long-sequence perplexity (last 512 tokens) as a function of prefix length (2048 to 8192 tokens) on book and code data. Four models are compared: Transformer, YOCO (non-recursive), RINS, and YOCO-U. Two patterns stand out:

**YOCO-U maintains or exceeds non-recursive baselines.** On book data at 8192 prefix length, YOCO-U's perplexity (~11.5) is lower than Transformer (~12.5) and non-recursive YOCO (~12.0). On code data, YOCO-U's perplexity (~2.28) is essentially tied with non-recursive YOCO (~2.28) and lower than Transformer (~2.35). The key finding is that **adding recursion to the efficient-attention Self-Decoder does not degrade long-context capability** — if anything, the additional representational depth may help the model better utilize long-range context by refining local representations that the Cross-Decoder later retrieves globally.

**YOCO-U maintains parity with RINS.** On both book and code data, YOCO-U's perplexity curve is comparable to RINS, consistent with the aggregate downstream parity in Table 3. This is important because one might worry that restricting recursion to efficient-attention layers (limited receptive field) would harm long-context modeling compared to full-attention recursive approaches — the Cross-Decoder's global attention could theoretically compensate, but this needed empirical verification. The result confirms that the global cross-attention mechanism is sufficient for long-range retrieval even when the recursive Self-Decoder only sees local windows.

**Needle In-A-Haystack retrieval.** Table 4 reports accuracy on the NIAH test with 1 and 2 needles. YOCO-U achieves perfect retrieval (1.00) for single-needle and 0.95 for dual-needle — competitive with YOCO (1.00, 0.86) and RINS (0.99, 0.91), and superior to the standard Transformer (0.87, 0.82). The strong NIAH performance supports the architectural premise: the Cross-Decoder's global cross-attention effectively retrieves information across the full context regardless of the Self-Decoder's local window constraint.

#### Inference Efficiency

Figure 7 and Appendix D (Tables 8–10) provide throughput and memory measurements for 1.3B dense models at context lengths from 8K to 256K tokens. The headline results at 256K context (the longest tested):

**Prefilling throughput (Figure 7a, Table 8).** YOCO-U: 76,301 tokens/second. Transformer: 7,475 tokens/second (10.2× slower). RINS: 3,739 tokens/second (20.4× slower). Non-recursive YOCO: 220,407 tokens/second (2.9× faster than YOCO-U). The gap between YOCO-U and YOCO reflects the `T=3` Self-Decoder iterations adding prefilling work — but because this work scales with window size `W` (512) rather than sequence length `N`, the relative gap narrows at longer contexts (at 256K, YOCO-U is only ~3× slower than YOCO, not `T=3` times slower). The absolute prefilling throughput of 76K tokens/second at 256K context means YOCO-U can process a 256K-token document in approximately 3.4 seconds — fast enough for practical long-context applications.

**Decoding throughput (Figure 7b, Table 9).** YOCO-U: 303 tokens/second. Transformer: 137 tokens/second (2.2× slower). RINS: 56 tokens/second (5.4× slower). Non-recursive YOCO: 318 tokens/second (1.05× faster). The key result is that **YOCO-U incurs only ~5% decoding throughput reduction compared to non-recursive YOCO**, while providing substantial representational benefits (Table 2: +4.45 average). In contrast, RINS — which achieves the same downstream performance (Table 3: 48.3) — incurs a ~5.4× decoding slowdown compared to non-recursive YOCO. The minimal decoding overhead is because the Cross-Decoder dominates decoding cost at long context lengths (its cross-attention over `N` keys), and the Cross-Decoder is unchanged between YOCO and YOCO-U.

**KV cache memory (Figure 7c, Table 10).** YOCO-U: 542 MB. Transformer: 10,240 MB (18.9× more). RINS: 20,480 MB (37.8× more). Non-recursive YOCO: 522 MB (essentially identical — the curves overlap in Figure 7c). The 20 MB difference between YOCO-U and YOCO (542 vs. 522) is the additional local sliding-window caches from the `T=3` Self-Decoder iterations: approximately `3 × 10 layers × 512 window × 2560 hidden × 2 bytes (FP16)` ≈ 79 MB, which aligns with the measured difference. This confirms the paper's complexity analysis in Table 1: the KV cache overhead from recursion scales with `WTL` (window × iterations × layers) rather than `LTN` (layers × iterations × full sequence), and for long sequences where `N ≫ WT`, the overhead is negligible. At 256K context, RINS's 20,480 MB cache (38× YOCO-U's) would severely limit batch size — with a typical 80GB GPU, YOCO-U could serve approximately 148 sequences in parallel (80GB / 542MB), while RINS could serve only 4 (80GB / 20,480MB). This is the practical deployment consequence of the architectural design choice.

**Context length scaling behavior.** The advantages compound with length. At 16K context (a more common deployment scenario), YOCO-U already shows 1.1× prefill throughput and 1.1× decode throughput over Transformer, and the KV cache is 10.3× smaller (62 MB vs. 640 MB). The linear scaling of YOCO-U's prefilling cost is visible in Table 8: throughput drops only modestly from 75,637 at 8K to 76,301 at 256K (essentially flat, since prefilling throughput is determined by the efficient attention's `O(N)` scaling and is bandwidth-bound at these lengths). In contrast, Transformer throughput drops from 85,707 at 8K to 7,475 at 256K — roughly following the expected `O(N)` degradation for quadratic attention with Flash-Decoding optimizations.

#### Scaling Properties: Parameters, Data, and Loop Iterations

**Parameter scaling (Figure 5).** The left panel plots validation loss against total parameters (300M to 10.8B) for non-recursive YOCO and YOCO-U, trained for 20k steps (20B tokens). YOCO-U achieves comparable performance with approximately 50% fewer parameters — the YOCO-U curve is shifted leftward, meaning a 3.4B YOCO-U matches a ~6.8B non-recursive YOCO at approximately the same loss level. The right panel plots loss against **activated** parameters, accounting for MoE sparsity. Here, YOCO-U's advantage narrows significantly — the curves are closer together — and at activated parameter counts above ~10B, they are "near-comparable." The paper's interpretation is that "YOCO-U eliminates parameter redundancy" — the recursive computation allows each parameter to be used more effectively across multiple representational refinement steps, so the total parameter count can be reduced without sacrificing performance. However, the total-parameter comparison (left) is arguably the more relevant metric for deployment since memory cost depends on total parameters (you must store all expert weights), while inference FLOPs depend on activated parameters. The fact that YOCO-U matches performance with 50% fewer total parameters directly translates to lower GPU memory requirements for model weights.

**Loop scaling (Figure 6).** Training 1.3B dense models with loop counts from `T=1` (non-recursive) to `T=5` at fixed training tokens (10B and 20B) shows consistent improvements: validation loss decreases monotonically as iteration count increases. The improvement from `T=1` to `T=2` is substantial (loss drops from ~3.2 to ~3.0 at 10B tokens), with diminishing returns at higher `T` — the gap from `T=3` to `T=5` is smaller than from `T=1` to `T=3`. This diminishing-return pattern is expected: additional iterations provide progressively smaller representational refinement as the representations approach a fixed point (consistent with the angular distance analysis in Figure 8). The practical implication is that `T=3` (the paper's default) captures most of the benefit while keeping training and inference overhead modest — pushing to `T=5` adds 33% more Self-Decoder FLOPs for a relatively small further loss reduction.

### Ablation Studies and Robustness Checks

The paper's ablation experiments (Section 4.3, Table 5) systematically vary three aspects of the recursive design: loop position, KV cache sharing strategy, and depth-vs-width tradeoff. All experiments use dense 1.3B models trained for 20B tokens.

**Loop position: Self-Decoder vs. Cross-Decoder recursion.** Two variants are tested: "Upper Loop" (looping the Cross-Decoder instead of the Self-Decoder, with `T=3` iterations) and "Upper Loop w/o Shared KV" (the same but without reusing the Self-Decoder's global KV cache — each Cross-Decoder iteration uses self-attention rather than cross-attention to the fixed cache). Results:
- Non-recursive YOCO baseline: 46.95 average
- YOCO-U (Self-Decoder loop): 48.25 (+1.30)
- Upper Loop (Cross-Decoder loop with shared KV): 47.34 (+0.39)
- Upper Loop w/o Shared KV (Cross-Decoder loop with self-attention): 46.41 (−0.54, below non-recursive baseline)

The pattern is clear: recursion on the Cross-Decoder provides minimal benefit even with the shared KV cache, and **hurts** performance when the shared cache is removed. This confirms that recursion's benefits are specific to the Self-Decoder's representation-refinement role, not a generic property of "more computation." The paper connects this to the ETD framework (Koishekenov et al., 2025): "the final layers behave like a final decoder." The negative result for "Upper Loop w/o Shared KV" also validates the shared-cache design — breaking the "cache once" property degrades performance, suggesting the global cache provides stable guidance that self-attention variants disrupt.

**Model layout: depth-width tradeoff.** Two variants probe whether the benefits of YOCO-U are simply due to increased FLOPs or effective depth, independent of the recursive mechanism:
- "Deep (Instead of Wide)": Non-recursive at 46.87 average — essentially identical to the non-recursive baseline (46.95). This is a model with double the layers (40) but reduced hidden dimension (1792) to keep total parameters at 1.3B. The result shows that deeper-but-narrower architecture at fixed parameters does not improve performance — model width matters, and depth alone isn't sufficient. YOCO-U's recursion provides depth without the width tradeoff.
- "Deeper (Instead of Wide)" with YOCO-U: 48.59 — the highest average in the ablation table, slightly above standard YOCO-U (48.25). Applying recursion to the deeper (40-layer) layout provides a further small gain, suggesting that the benefits of recursive computation are largely orthogonal to the base depth-width configuration. The improvement is modest (+0.34 over YOCO-U with the standard layout), suggesting that the standard 20-layer configuration already captures most of the benefit.

**Efficient attention variants.** Not presented as a formal ablation table, but discussed in Section 3.2: "Linear attention variants... such as RetNet, Mamba, and gated DeltaNet, are also compatible, though they perform similarly to SWA within hybrid architectures." This implies (but does not empirically demonstrate) that the specific choice of efficient attention in the Self-Decoder has limited impact on downstream performance when paired with global cross-attention in the Cross-Decoder. The paper does not provide comparison numbers, so this claim remains qualitative. A proper ablation comparing sliding-window, RetNet, Mamba-based, and DeltaNet-based Self-Decoders within YOCO-U at equal training budget would strengthen this claim.

**Iteration count scaling (Figure 6).** As described in the main results, this is effectively an ablation on `T`, showing consistent but diminishing improvements from `T=1` through `T=5`. The non-obvious finding is that training is stable and beneficial even at `T=5` — there is no sign of degradation or training instability that might be expected from deep recursive computation with shared parameters.

**Representation analysis (Figure 8).** The angular distance between consecutive layers is measured across the Self-Decoder and Cross-Decoder at different loop iterations. Three patterns emerge as ablation-like diagnostics: (1) angular distance patterns within the Self-Decoder are remarkably consistent across iterations, suggesting the model learns a stable refinement function; (2) mean distance gradually decreases with iteration count, consistent with representations approaching a fixed point; (3) a sharp spike occurs at the Self-Decoder/Cross-Decoder boundary, indicating a functional transition. This analysis doesn't test a causal claim but provides representational evidence consistent with the architectural premise — the two decoder halves perform qualitatively different operations.

### Critical Assessment

#### Does YOCO-U actually demonstrate that "recursive computation confined to efficient-attention shallow blocks can substitute for parameter count and training tokens"?

The token scaling result (Figure 2, right) provides the strongest evidence: YOCO-U reaches comparable loss with ~62% fewer training tokens. This is a genuine efficiency gain — fewer tokens means less training data needed, which could translate to faster iteration cycles and lower training costs. The parameter scaling result (Figure 5, left) supports the complementary claim: YOCO-U achieves comparable loss with ~50% fewer total parameters. Together, these show that recursion substitutes for both data quantity and parameter count.

However, the experiments have important limitations:

1. **Scale ceiling.** The largest model is 10.8B total / ~1.3B activated parameters, trained on 20B tokens for the scaling experiments and 300B tokens for the main results. Modern production LLMs are 1–3 orders of magnitude larger in both parameters and tokens. The paper doesn't demonstrate whether the recursive benefits persist at larger scale — it's possible that very large models saturate the gains from recursion, or (conversely) that benefits compound at scale. The scaling curves in Figure 5 show no sign of plateau, but the range is limited.

2. **Single architecture family.** All experiments use YOCO-derived architectures. The claim about efficient-attention recursion being broadly beneficial is not tested against other decoder-decoder designs or against standard Transformers with efficient-attention self-decoder components added separately. The RINS comparison (Table 3) provides some cross-architecture validation — recursion helps in both YOCO and standard Transformer contexts — but the specific synergy of recursion + efficient attention is demonstrated only within YOCO.

3. **Training token parity.** The "Equal FLOPs" comparison in Table 2 is favorable to YOCO-U because it gives non-recursive YOCO more training tokens to equalize FLOPs — but this comparison doesn't account for potential differences in how models benefit from additional tokens. If non-recursive YOCO benefits more per added token than YOCO-U does, the equal-FLOPs comparison would overestimate YOCO-U's advantage. The Figure 2 curves suggest this isn't the case (YOCO-U's curve is consistently below YOCO's), but with only 300B tokens total, the training horizon is limited.

4. **Thinking SFT gains require careful interpretation.** The 24.4% average improvement on math benchmarks (Figure 3) is impressive but confounds two effects: YOCO-U's architectural advantage and the specialized thinking SFT training. The paper frames this as showing "orthogonality" between architectural and post-training improvements, which is plausible, but without an ablation showing YOCO-U's math performance *without* thinking SFT, we can't quantify how much of the 24.4% gain is architectural vs. training-data-driven.

#### Does YOCO-U demonstrate that recursion avoids prohibitive memory costs only when restricted to efficient-attention layers?

This claim is strongly supported by the inference efficiency measurements in Figure 7 and Appendix D. The key evidence is:

- YOCO-U KV cache at 256K: 542 MB (Figure 7c). Non-recursive YOCO: 522 MB. The difference (~20 MB) matches theoretical predictions from the local window caches.
- RINS KV cache at 256K: 20,480 MB — 38 times YOCO-U's. This is the counterfactual: recursion in full-attention blocks explodes cache size.
- YOCO-U decoding throughput at 256K: 303 tokens/second (Figure 7b). Non-recursive YOCO: 318. Only 5% degradation. RINS: 56 tokens/second — 5.4× degradation.

These are measured on real hardware (H100-80GB) with a production-grade inference stack (Nano-vLLM with Flash-Decoding, Paged Attention, kernel fusion), making them practically meaningful rather than purely theoretical.

A limitation: the throughput measurements use a batch size of 32 and generation length of 128 (Appendix D). For deployment scenarios with larger batches or different generation lengths, the relative advantages might shift. The paper's batch size choice is reasonable for interactive serving but doesn't cover the full range of deployment configurations (e.g., high-throughput batch processing with large batch sizes).

#### Does YOCO-U demonstrate that recursive and non-recursive YOCO have comparable long-context capability?

Figure 4 shows that YOCO-U's long-sequence perplexity is comparable to or better than non-recursive YOCO and RINS on book and code data up to 8192 tokens. Table 4 shows strong NIAH retrieval. However:

1. **Maximum context length is 8192 tokens.** This is modest by modern long-context standards (128K–1M tokens). The inference efficiency experiments test up to 256K tokens but do not measure perplexity or retrieval at those lengths. The claim about long-context capability is therefore validated only up to 8192 tokens for quality metrics.

2. **Single data source per domain.** One book dataset and one code repository dataset are used. Long-context performance may vary significantly across domains (legal documents, scientific papers, multi-turn conversations), and two data points don't establish robustness.

3. **No comparison to non-recursive Transformers with sliding-window attention.** The fair baseline for assessing whether YOCO-U's recursive design specifically helps or hurts long-context performance would include a standard Transformer with sliding-window attention (not full attention) — this would isolate the effect of recursion from the effect of using efficient attention in the first place. The paper compares against full-attention Transformers (which are disadvantaged at long context) and non-recursive YOCO (which already has efficient attention), but not against a Transformer with the same efficient-attention mechanism and no recursion.

#### Do the architecture comparison results (Table 3) genuinely support the claim that YOCO-U matches RINS's performance with dramatically lower cost?

The claim is supported: both achieve 48.3 average, and the per-task differences are small. However:

1. **Training budget is limited (20B tokens).** At this scale, models are far from convergence. It's possible that RINS and YOCO-U would diverge with more training — if full-attention recursion provides representational benefits that efficient-attention recursion cannot match over longer training horizons, the parity might not hold. The paper's main experiments use 300B tokens for YOCO-U but don't extend RINS to this scale for comparison.

2. **Benchmark suite is general-domain.** The 9 benchmarks in Table 3 are standard but relatively easy — accuracy scores are in the 20–70% range, with plenty of headroom. It's possible that RINS and YOCO-U are distinguished on harder, more reasoning-intensive tasks. The math benchmarks after thinking SFT (Figure 3) show large YOCO-U gains but don't include RINS as a baseline, so we can't assess whether RINS would show similar or larger math improvements.

3. **Dense vs. MoE models.** The architecture comparison uses dense models, while the main results use MoE models. The paper doesn't explain this choice, but it's likely because the compared architectures (UT, RINS, ParScale) aren't defined for MoE, and retrofitting them would introduce confounding variables. However, this means the RINS-YOCO-U parity is demonstrated only for dense models, not the MoE configuration that achieves the best absolute results.

#### Missing experiments that would strengthen the paper

1. **Direct comparison of YOCO-U against RINS at 300B training tokens.** This would test whether the representational parity observed at 20B tokens holds at larger training scales.

2. **Ablation on window size `W`.** The sliding-window size is fixed at 512 throughout. How does YOCO-U's performance and efficiency vary with window size? A smaller window would further reduce memory but might hurt the recursive Self-Decoder's ability to refine representations. A larger window would increase the local cache overhead but might provide more context per iteration. Understanding this tradeoff is important for practitioners choosing deployment configurations.

3. **Adaptive recursion (variable `T` per token/sequence).** The paper positions YOCO-U as enabling future adaptive-depth work, but doesn't explore it. A simple experiment — using fewer iterations for "easy" tokens or sequences based on a confidence measure — would validate the practical feasibility of adaptive depth in this architecture.

4. **Ablation on NoPE vs. RoPE in Cross-Decoder.** The choice of NoPE in the Cross-Decoder is justified by "enhanced global retrieval capability" and a citation to Yang et al. (2025), but no ablation is provided. A direct comparison of NoPE vs. RoPE in the Cross-Decoder's cross-attention would quantify this design choice's contribution.

5. **Latency-aware batch sizing analysis.** The throughput measurements fix batch size at 32. A more deployment-relevant comparison would show maximum throughput as a function of latency budget — given a per-token latency SLO, how many requests can each architecture serve concurrently? This would more directly capture the KV cache memory advantage's impact on serving economics.

6. **Failure mode analysis for recursive computation.** The paper reports stable training and consistent improvements, but doesn't characterize *where* recursion fails. Are there types of sequences or tasks where YOCO-U performs worse than non-recursive YOCO? The angular distance analysis (Figure 8) hints at diminishing returns, but doesn't identify specific failure cases. Qualitative analysis of recursive refinement — showing what the Self-Decoder's iterations actually change in token representations — would strengthen understanding.

In summary, the experiments credibly support the paper's core claims about the efficiency-representation tradeoff, with the strongest evidence being the inference efficiency measurements (Figure 7) that quantify the memory/latency advantages, and the token/parameter scaling results (Figures 2, 5) that demonstrate recursive computation's efficiency benefits. The primary limitations are scale (model size, training tokens, context length for quality metrics), the absence of RINS comparisons at larger training budgets, and the reliance on a single architectural framework (YOCO) for the central synergy claim. The paper's contribution is best characterized as establishing a promising design principle — restrict recursion to efficient-attention modules — with strong initial evidence, rather than a definitive demonstration that this principle universally dominates alternatives.

## 6. Limitations and Trade-offs

### 6.1 Training Scale and Model Size Are Orders of Magnitude Below Production Deployments

**The assumption or constraint.** All primary experiments train models at the ~1.3B activated parameter scale (10B total with MoE, or 1.3B dense for architecture comparisons) on 20–300B tokens. The parameter scaling curve (Section 4.4, Figure 5) extends to 10.8B total parameters but only for 20B training tokens — a small fraction of what models of that size would receive in practice. The paper does not train or evaluate any model at the scale of current production LLMs (70B+, trained on trillions of tokens).

The paper acknowledges this implicitly through experimental scope rather than explicit caveat — Section 4.1 reports training "75k steps (i.e., 300B tokens) given the resource budget," signaling a resource constraint, not a claim that this scale is representative of production deployments.

**The consequence.** The core finding — that recursive computation in efficient-attention blocks substitutes for ~62% of training tokens and ~50% of parameters — may not hold at larger scales. Recursive depth scaling could exhibit diminishing, constant, or even increasing returns as model size grows. More critically, training stability — which the paper reports is "high" at 1.3B scale — could degrade at larger model sizes. Recursive architectures with shared parameters effectively apply the same transformation function multiple times, which can amplify gradient variance or cause representational collapse in deeper or wider networks. Without evidence at production scale, a practitioner cannot confidently project the efficiency gains to a 70B or 405B model.

Additionally, the MoE configuration (64 experts, top-8 activation) interacts with recursion in ways that may not scale linearly. The paper's parameter scaling experiment (Figure 5, right) shows that YOCO-U's advantage narrows when plotted against activated parameters rather than total parameters — at ~10B activated parameters, the curves are "near-comparable." This suggests that the recursive benefit may shrink as activated parameter count increases, which would directly affect projections to larger sparse models.

**What evidence exists in the paper.** The parameter scaling curves in Figure 5 extend to 10.8B total / ~1.4B activated parameters (based on the dense model configurations in Appendix C, Table 7). The curves show no obvious saturation — YOCO-U's advantage persists across the tested range — but the range spans less than 2 orders of magnitude (300M to 10.8B). The loop scaling experiment (Figure 6) tests up to `T=5` iterations at 1.3B scale with 20B tokens, showing consistent but diminishing improvements. Neither experiment addresses scale interaction: does the optimal `T` change with model size? Does the recursive benefit increase or decrease with training tokens? Without these interactions characterized, the headline efficiency numbers (62% fewer tokens, 50% fewer parameters) must be treated as scale-specific estimates rather than general scaling laws.

**Mitigation status.** Not addressed. The paper does not discuss scale limitations as a caveat, propose larger-scale experiments for future work, or provide a theoretical argument for why the benefits should persist. The parameter scaling section treats the observed trend as evidence of a general property ("YOCO-U eliminates parameter redundancy") without acknowledging the limited range. This is the most significant omission in the paper's experimental design, given that the entire contribution targets deployment-efficient LLM architectures.

---

### 6.2 Difficulty Estimation Is Static and Uniform — There Is No Adaptive or Per-Token Recursion

**The assumption or constraint.** YOCO-U applies a **fixed number of recursive iterations `T`** (default: 3) to every token in every sequence, regardless of token difficulty, sequence position, or task characteristics. The Universal Self-Decoder executes exactly `T` passes through the same layers for every input — there is no learned halting mechanism, per-token iteration count, or dynamic computation budget.

The paper is explicit about this deliberate scope limitation. Section 1 states that YOCO-U "iterates computation for multiple steps using shared parameters" with a fixed `T`, and the paper does not claim to provide adaptive depth scaling. Section 5 mentions dynamic recursive depth approaches (Mixture-of-Recursions, Bae et al., 2025) only as related work, not as integrated capability.

**The consequence.** This is a significant missed opportunity because the YOCO-U architecture is **uniquely well-suited to adaptive recursion** in a way that full-attention recursive architectures (UT, RINS) are not. In YOCO-U, varying `T` per token would only affect the local window-based KV caches (the `O(WTLD)` term in Table 1), not the global cross-attention cache. This means dynamic depth is deployment-feasible in YOCO-U — the memory cost of an additional iteration on a single token is negligible — but the paper does not explore it.

The practical consequence is that YOCO-U spends the same computation on an unambiguous, easily-predicted token (e.g., a punctuation mark in a straightforward sentence) as on an ambiguous, reasoning-intensive token (e.g., the final answer token in a math problem). This is computationally wasteful — the model's fixed `T=3` iteration budget is a blunt instrument, and there is likely a more efficient allocation where "easy" tokens get `T=1` or `T=2` iterations while "hard" tokens get `T=4` or `T=5`.

The representational analysis (Section 4.6, Figure 8) provides indirect evidence that this matters: angular distances decrease with iteration count, suggesting representations approach a fixed point. For tokens that reach their fixed point after 2 iterations, the third iteration is redundant computation. For tokens still evolving rapidly, additional iterations might help. The paper observes this phenomenon but does not act on it architecturally.

**What evidence exists in the paper.** The loop scaling experiment (Figure 6) shows that more iterations (`T=5`) improve aggregate loss monotonically but with diminishing returns, but this is a uniform increase — it doesn't tell us which tokens benefit most. No per-token analysis of iteration utility is provided. The angular distance analysis (Figure 8) shows mean distance decreasing with iterations but provides no token-level variance — it cannot distinguish between a token where all three iterations are useful and one where only the first matters. The paper contains no experiments with variable `T`, learned halting, or confidence-based early exit.

**Mitigation status.** Not attempted. The paper positions YOCO-U as "enabling" adaptive depth by making it memory-efficient (Section 1: "a much more scalable and efficient path for recursive computation compared to its predecessors"), and Section 2 acknowledges work on dynamic recursive depth (Mixture-of-Recursions), but the integration is explicitly left to future work. This is a reasonable scope limitation for an architecture paper introducing the design pattern, but it means the paper does not demonstrate one of the most compelling use cases for its own architectural innovation.

---

### 6.3 Long-Context Quality Is Validated Only Up to 8K Tokens, While Efficiency Claims Extend to 256K

**The assumption or constraint.** The paper claims YOCO-U "maintains robust long-context modeling capabilities" (Section 1) and demonstrates inference efficiency at up to 256K context length (Section 4.5, Figure 7). However, **long-context quality metrics — perplexity and retrieval accuracy — are measured only up to 8,192 tokens** (Section 4.2, Figure 4; Table 4). There is a 32× gap between the maximum context length where quality is assessed (8K) and the maximum context length where efficiency is claimed (256K).

The paper does not explicitly acknowledge this gap as a limitation. Figure 4's x-axis extends to 8,192, and Table 4 does not specify context length (standard NIAH tests typically use configurable lengths, but no lengths are reported in the paper). Section 4.5's efficiency measurements span 8K to 256K but measure only throughput and memory, not model quality.

**The consequence.** A deployment architect reading this paper would reasonably assume that YOCO-U's strong long-context efficiency (10× faster pre-filling than Transformer at 256K, negligible KV cache overhead) applies to contexts where the model actually works well. But the quality evidence stops at 8K. Beyond 8K tokens, it is unknown whether:
- Perplexity on long-range dependencies degrades (the sliding-window Self-Decoder's limited receptive field might cause information loss that compounds over very long contexts, and the Cross-Decoder's single global cache might not fully compensate).
- Retrieval accuracy (NIAH) degrades as the distance between the query and the needle increases beyond 8K.
- The recursive Self-Decoder's iterative refinement introduces artifacts or instabilities that only manifest with very long inputs (e.g., error accumulation over `T` iterations when the context is noisy or contains contradictory information).

This is not merely a benchmarking gap — it is a **capability gap in the evidence**. The architectural premise is that the Cross-Decoder's global cross-attention compensates for the Self-Decoder's local window. For contexts of 8K tokens with a 512-token window, the Self-Decoder processes approximately 16 windows; at 256K, it processes 512 windows. Whether the Cross-Decoder can reliably retrieve information that was last "seen" by the Self-Decoder 512 windows ago (and refined through `T=3` recursive iterations in each window) is not tested.

**What evidence exists in the paper.** Figure 4 shows long-sequence perplexity on book and code data at 2K, 4K, 6K, and 8K prefix lengths. YOCO-U's perplexity is comparable to non-recursive YOCO and RINS at all measured lengths, with no degradation trend at the 8K endpoint. Table 4 shows NIAH accuracy but does not report the tested context length. The NIAH results (1.00 single-needle, 0.95 dual-needle) are strong, but if the tested context is ≤8K, the results do not validate the 256K efficiency claims. Appendix D's inference efficiency tables provide throughput and memory at 256K but no quality measurements.

**Mitigation status.** Not addressed. The paper does not discuss the quality-efficiency context length gap, propose extending quality benchmarks to 128K or 256K, or provide a theoretical argument for why quality should be preserved at arbitrary lengths. The NIAH results are presented in Section 4.2 alongside the long-context perplexity without specifying test length, obscuring the limitation. A simple NIAH evaluation at 128K tokens would partially close this gap and is computationally feasible for a 1.3B model — its absence is a notable omission.

---

### 6.4 The "Synergy" of Recursion and Efficient Attention Is Demonstrated Only Within the YOCO Architecture

**The assumption or constraint.** The paper's central conceptual claim is that "recursive computation confined to efficient-attention shallow blocks" provides a general design principle for memory-efficient depth scaling. However, all experiments demonstrating this claim are conducted within the YOCO decoder-decoder framework. The architecture comparison in Section 4.2 (Table 3) compares YOCO-U against recursive full-attention Transformers (UT, RINS), but does not test the reverse combination: a standard Transformer modified to use efficient attention in its recursive blocks.

**The consequence.** The strong interpretation of the paper's claim — that the recursive-efficient-attention synergy is a general architectural principle — is not directly supported. The paper demonstrates that YOCO-U (efficient-attention recursion) matches RINS (full-attention recursion) in quality while dramatically reducing memory, but this comparison confounds two variables: the base architecture (YOCO vs. standard Transformer) and the attention mechanism in the recursive block (efficient vs. full).

To isolate the synergy claim, one would need an additional baseline: a standard Transformer where the bottom `L/2` layers are replaced with efficient attention and recursed `T` times, while the top `L/2` layers use standard full self-attention and are non-recursive. This is essentially "RINS but with efficient attention in the recursive layers." If this hypothetical architecture matched YOCO-U's efficiency while retaining some Transformer-specific advantages, the synergy claim would be general. If it underperformed YOCO-U, the synergy might be specific to the decoder-decoder + shared-cache framework, not just to efficient-attention recursion.

The paper provides one piece of indirect evidence in the ablation: "Upper Loop w/o Shared KV" (Table 5) tests Cross-Decoder recursion without the shared cache, and it degrades performance below the non-recursive baseline. This suggests the shared global cache — the YOCO-specific element — is critical, but it doesn't disentangle whether the shared cache is necessary for *any* efficient-attention recursion to work well, or only for the specific YOCO-U configuration.

**What evidence exists in the paper.** The architecture comparison (Table 3) shows: RINS (full-attention recursive Transformer) = 48.3 average; YOCO-U (efficient-attention recursive YOCO) = 48.3 average; YOCO-U's KV cache at 256K = 542 MB vs. RINS = 20,480 MB. This supports the claim that the YOCO-U combination achieves RINS-level quality with dramatically lower cost. However, it does not test whether adding efficient attention to a recursive Transformer (without the YOCO decoder-decoder split) would achieve similar quality at similar low cost, or whether the quality-efficiency win requires the YOCO framework specifically.

**Mitigation status.** Not addressed. The paper treats the YOCO decoder-decoder framework as the natural host for efficient-attention recursion, and the ablation studies (Section 4.3) tune within that framework without testing alternative base architectures. The claim that the recursive-efficient-attention synergy is "greater than either alone" is well-supported for the YOCO family but not demonstrated to generalize. A careful reading of the paper reveals that the design principle proposed is actually narrower than the abstract suggests: it is "recursion in YOCO's Self-Decoder is efficient," not "recursion in efficient-attention blocks is efficient regardless of surrounding architecture." This does not invalidate the contribution but tempers its generality.

---

### 6.5 No Comparison Against Strong Test-Time Compute Baselines or Combined Inference-Time Scaling

**The assumption or constraint.** YOCO-U is positioned as addressing the tension between depth scaling and inference efficiency, motivated by the rise of test-time compute scaling (Section 1: "standard Transformers struggle to scale inference-time compute efficiently"). However, the paper's experimental comparisons are exclusively against **architectural alternatives** (UT, RINS, ParScale) and **non-recursive baselines** (standard Transformer, non-recursive YOCO). There is no comparison against models that use test-time compute strategies — chain-of-thought, self-consistency, best-of-N sampling, or tree search — to achieve improved reasoning without architectural recursion.

**The consequence.** The paper cannot quantify how much of the reasoning improvement YOCO-U provides is **architecturally necessary** versus achievable through inference-time strategies applied to a non-recursive model. Consider the math reasoning results after thinking SFT (Figure 3): YOCO-U achieves 67.1% average across 11 benchmarks versus 51.6% for non-recursive YOCO. But how much of this 24.4% gain comes from the explicit thinking SFT training (which both models receive) versus the recursive architecture? And would non-recursive YOCO with a test-time compute strategy — say, self-consistency with 5 samples, or a longer chain-of-thought budget — close the gap? The paper argues that "computation scaling strategies applied during pre-training are orthogonal to these inference scaling techniques" (Section 2), and that the results show "improvement on explicit and latent reasoning is orthogonal" (Section 4.1). But orthogonality is not demonstrated — it is asserted. To demonstrate orthogonality, the paper would need to show that YOCO-U with test-time compute outperforms non-recursive YOCO with equivalent test-time compute, and that the gaps are additive.

This is a practical concern: if a non-recursive architecture can achieve most of YOCO-U's gains by spending 2× more inference tokens (e.g., chain-of-thought or self-consistency), the architectural complexity of recursive depth scaling may not be worth the engineering cost for practitioners who can simply increase the inference budget. The paper's efficiency claims (4.45 average improvement under equal FLOPs) are about training efficiency, not inference-time capability per dollar of serving cost. A deployment-centric comparison would be: given a fixed per-request latency and cost budget, does YOCO-U with `T=3` outperform non-recursive YOCO with `T=1` but double the output tokens (via chain-of-thought or self-consistency)?

**What evidence exists in the paper.** None. The paper contains no experiments comparing YOCO-U against non-recursive YOCO or standard Transformers augmented with inference-time compute strategies (chain-of-thought prompting, self-consistency, best-of-N, verifier-guided search). The thinking SFT experiments (Section 4.1, Figure 3) train both YOCO and YOCO-U on math reasoning data with explicit chain-of-thought, but this is a training intervention, not an inference-time budget comparison. The models are compared with identical decoding strategies (greedy decoding, Appendix B), not with matched inference compute.

**Mitigation status.** The paper acknowledges the orthogonality conceptually — Section 2 explicitly states that "computation scaling strategies applied during pre-training are orthogonal to these inference scaling techniques" — but provides no empirical test of this claim. The "orthogonal" framing is used to carve out a distinct contribution space rather than to motivate an experiment. This is a reasonable scope choice for an architecture paper, but it means the paper cannot advise a practitioner on whether to invest in YOCO-U's recursive pretraining versus simply allocating more inference tokens to a standard model.

---

### 6.6 Difficulty Bin Construction Cost and Static Allocation Strategy

**The assumption or constraint.** YOCO-U uses a fixed number of recursive iterations `T` for all inputs, with no mechanism for estimating input difficulty or allocating computation adaptively. While the paper does not explicitly study difficulty-based allocation (this is acknowledged as a deliberate scope choice — see Limitation 6.2), the architectural design implicitly assumes that uniform `T` is a reasonable default. The paper reports that `T=3` is the default because it "results in 2× the total FLOPs of the non-recursive baseline" (Section 4 preamble) and Figure 6 shows diminishing returns at higher `T`, but there is no analysis of whether some inputs (or tokens) would benefit from `T=1` or `T=5` more than others.

**The consequence.** The uniform `T=3` allocation means YOCO-U spends approximately 2× the compute of non-recursive YOCO on every input, regardless of whether that input benefits from additional depth. If a substantial fraction of tokens or sequences are "easy" — where `T=1` is sufficient — the uniform allocation wastes approximately 33% of the Self-Decoder compute (two extra iterations). If some tokens are "hard" and could benefit from `T=5`, the uniform allocation leaves capability on the table. The paper's own representational analysis (Figure 8) shows that angular distances decrease with iterations (suggesting diminishing returns) and vary across layers (suggesting non-uniform benefit), but this variation is not exploited.

This connects to a broader limitation: the paper does not characterize **which types of inputs** benefit most from recursion. The downstream task results (Table 2) show larger gains on GSM8K (+12.4 points) and BBH (+2.4 points) than on ARC-C (+1.4 points) or HumanEval (+1.8 points), suggesting reasoning-intensive tasks benefit more. But this is a per-benchmark, not per-instance, observation. Without instance-level difficulty analysis, a practitioner deploying YOCO-U cannot determine whether the uniform `T=3` allocation is cost-effective for their specific workload distribution.

**What evidence exists in the paper.** Figure 6 shows aggregate loss improvement from `T=1` to `T=5`, with diminishing returns. The downstream task table (Table 2) shows per-benchmark gains from recursion, with significant variance (GSM8K: +12.4; Winogrande: +7.0; ARC-C: +1.4; HumanEval: +1.8). Figure 8 shows inter-layer angular distances decreasing with iteration count and varying across the Self-Decoder vs. Cross-Decoder boundary, but provides no instance-level or token-level analysis. The paper does not cluster test examples by difficulty, analyze per-position iteration utility, or measure the correlation between base model confidence and recursive benefit.

**Mitigation status.** Not addressed as a limitation. The paper treats uniform `T=3` as a design parameter with reasonable defaults, justified by the aggregate scaling curve in Figure 6. The paper acknowledges the possibility of adaptive depth by citing Mixture-of-Recursions (Bae et al., 2025) and Encode-Think-Decode (Koishekenov et al., 2025) as related work, but does not discuss the opportunity to integrate adaptive depth into YOCO-U's memory-efficient framework. As with Limitation 6.2, this is a scope choice rather than an oversight, but it means the paper provides architectural infrastructure for adaptive recursion without demonstrating that the infrastructure can be effectively used. A practitioner inspired by the efficiency results would face the unanswered question: "I have this memory-efficient recursive architecture — how do I decide how many iterations to use on each input?"

## 7. Implications and Future Directions

### How This Work Changes the Landscape

This paper makes a specific, architectural intervention rather than a sweeping paradigm shift: it demonstrates that **the memory cost of recursive depth scaling is not inherent to recursion itself, but to the attention mechanism used in the recursive block.** This reframes the conversation about test-time and training-time compute scaling in a practically important way — the design question is not "should we use recursion?" but "in which attention modules should we recurse?"

The conceptual reframing has two concrete consequences. First, it **unblocks recursive architectures for long-context deployment**. Before YOCO-U, the dominant assumption — encoded in the Universal Transformer literature and reinforced by RINS's full-attention recursion — was that depth scaling via parameter sharing inevitably multiplies KV cache memory. The complexity analysis in Table 1 formalized this: loop standard Transformer layers, and your cache grows from `O(LND)` to `O(LTND)`. YOCO-U's diagnostic move is to observe that this multiplicative relationship applies only to full-attention caches (which scale with sequence length `N`), not to efficient-attention caches (which scale with window size `W`, where `W ≪ N`). By confining recursion to the latter, YOCO-U achieves `O(N + WTL)` memory rather than `O(LTN)`. For the 256K-context, 1.3B-parameter configuration tested in Figure 7c, this means 542 MB versus 20,480 MB for RINS — a 38× difference that transforms recursion from a laboratory curiosity into a deployment-feasible technique.

Second, it **reconciles a tension between two previously separate lines of work**: efficient attention (sliding-window, linear attention, state-space models) and recursive depth scaling (Universal Transformer, RINS). These were pursued independently — efficient attention tackled the `O(N^2)` attention bottleneck, recursion tackled the parameter-cost-of-depth bottleneck — under the implicit assumption that they addressed orthogonal efficiency dimensions. YOCO-U's central empirical finding is that they are not merely additive but **synergistic**: recursive efficient-attention blocks provide comparable representational benefit to recursive full-attention blocks (Table 3: YOCO-U and RINS both achieve 48.3 average), but at a fraction of the memory and latency cost. This synergy is non-obvious because one might expect that recursion's representational benefits require global receptive field — that iterating within a local window cannot substitute for iterating over full context. The paper's architectural argument — that the Cross-Decoder's global cross-attention compensates for the Self-Decoder's local windows — explains why the synergy holds, and the empirical parity with RINS validates it.

The paper also provides **diagnostic clarity on which layers benefit from recursion**. Prior work on Universal Transformers and RINS applied recursion to early layers (RINS) or all layers (UT) based on intuition or empirical sweep, without systematic ablation of loop position. YOCO-U's ablation (Table 5) provides causal evidence: recursing the Cross-Decoder (top `L/2` layers) yields only +0.4 points over the non-recursive baseline, compared to +1.3 points for recursing the Self-Decoder (bottom `L/2` layers). Even more telling, recursing the Cross-Decoder without the shared KV cache ("Upper Loop w/o Shared KV") **degrades** performance below the non-recursive baseline. This establishes a functional division: early layers are "encoders" that benefit from iterative refinement; later layers are "decoders" whose retrieval-and-output role gains little from recursion. This finding is consistent with the Encode-Think-Decode framework (Koishekenov et al., 2025) but provides the first direct architectural ablation to support it, rather than post-hoc representational analysis.

The practical consequence of this diagnostic is that future architectures incorporating recursion now have a clear design rule: **recurse the encoder, not the decoder**. This is transferable beyond YOCO — any architecture with a factorization into representation-building and retrieval-output components can apply the principle.

The paper also **partially resolves the apparent conflict** between results showing that test-time compute helps (chain-of-thought, self-consistency) and the observation that simply looping standard Transformers is prohibitively expensive. YOCO-U demonstrates that architectural recursion during pretraining and explicit test-time reasoning during inference are orthogonal improvement axes: YOCO-U's architectural gains (Table 2: +4.45 average over non-recursive YOCO under equal FLOPs) compound with thinking SFT gains (Figure 3: +24.4% on math benchmarks). The architecture makes pretraining more efficient; test-time strategies extract additional capability. The implication is that organizations should invest in both, rather than viewing them as competing approaches to the same goal.

However, the paper does **not** resolve whether recursive architectures can substitute for inference-time compute strategies — that is, whether YOCO-U with `T=3` and greedy decoding matches non-recursive YOCO with self-consistency or extended chain-of-thought. This comparison is absent from the paper and remains an open question. The paper's contribution is establishing the architectural feasibility of memory-efficient recursion, not benchmarking it against inference-time alternatives.

### Follow-Up Research This Work Enables

**Adaptive per-token recursion depth with learned halting in YOCO-U.** The paper establishes that YOCO-U's recursion is memory-efficient (the local window caches scale with `WTL` rather than `LTN`), removing the primary barrier to dynamic depth. A natural next step is to train a halting mechanism — a lightweight classifier attached to the Self-Decoder output at each iteration that predicts whether further iterations would reduce the final loss. The training signal could come from an oracle: for each token position, run all `T` iterations, measure the contribution of each iteration to the final cross-entropy, and train the halting classifier to predict when the marginal gain drops below a threshold. A strong follow-up would measure: (1) the average number of iterations used per token on standard benchmarks (expectation: easy tokens use `T=1`, hard tokens use `T=4` or `T=5`); (2) whether adaptive depth achieves the same downstream accuracy as fixed `T=3` with fewer average FLOPs; (3) whether the halting mechanism generalizes across domains (does a halting classifier trained on language modeling assign more iterations to math reasoning tokens?). The paper's Figure 6 already shows diminishing returns from `T=1` to `T=5`, providing the aggregate signal that adaptive depth could exploit. The key risk is that the halting classifier might learn degenerate behaviors (always halting at `T=1` to minimize average iterations while sacrificing accuracy), which would require careful regularization or a budget-aware training objective.

**Scaling laws for recursive depth: how does the optimal `T` vary with model size, training tokens, and task?** The paper's parameter scaling experiments (Section 4.4, Figure 5) and loop scaling experiments (Figure 6) are conducted at modest scale (up to 10.8B total parameters, 20B training tokens) and treat `T` and model size as independent variables. A compute-optimal scaling analysis — analogous to the Chinchilla laws (Hoffmann et al., 2022) but for the `(parameters, training tokens, recursive depth T)` triplet — would characterize how these axes interact. The specific question: given a fixed FLOPs budget, what is the optimal allocation between adding static layers, adding recursive iterations, and increasing training tokens? This paper provides the architectural substrate (memory-efficient recursion) and initial evidence (`T=3` improves token efficiency by ~62% at 1.3B scale). A follow-up would train models at multiple scales (e.g., 100M, 300M, 1B, 3B, 10B parameters) with varying `T` (1 through 8) and training budgets (10B, 30B, 100B, 300B tokens), then fit a parametric scaling law. The key prediction to test: does the optimal `T` increase, decrease, or stay constant as model size grows? If recursion's benefit diminishes at larger scales (as Figure 5 right hints — YOCO-U's advantage narrows against activated parameters), then recursive architectures might be most valuable for smaller, deployment-efficient models rather than frontier-scale training.

**A direct, large-scale comparison of YOCO-U against RINS at matched quality with full cost accounting.** The paper's architecture comparison (Table 3) demonstrates quality parity (both 48.3 average) at 1.3B dense scale with 20B training tokens, and the inference benchmarks (Figure 7) show YOCO-U's dramatic cost advantage. However, this comparison is at a single (small) scale with limited training. A definitive study would train YOCO-U and RINS at a larger scale (e.g., 7B parameters, 1T tokens) to equal downstream accuracy on a comprehensive benchmark suite, then measure total cost of ownership: training FLOPs, inference throughput at various batch sizes, KV cache memory limiting maximum batch size, and prefilling latency at long contexts (128K+). The hypothesis is that YOCO-U achieves the same accuracy with lower inference cost but potentially higher training cost (since the recursive Self-Decoder increases training FLOPs per token). The study would characterize the break-even point: for a model that will serve `K` inference queries over its lifetime, how many queries are needed before YOCO-U's inference savings outweigh its training premium? This is precisely the analysis that practitioners need for deployment decisions. The paper's existing data (Figure 7, Table 2) provides the inference-side numbers; the missing piece is a larger-scale training comparison and a formalized total-cost model.

**Long-context quality evaluation of YOCO-U at 128K+ tokens to stress-test the global-local factorization.** The paper demonstrates long-context quality only up to 8,192 tokens (Figure 4, Table 4) but claims efficiency benefits at 256K tokens (Figure 7). A critical follow-up would extend quality benchmarks — perplexity on long-context datasets (PG-19, BookSum, narrative QA), retrieval (Needle In-A-Haystack at 128K and 256K), and multi-hop reasoning across long documents — to context lengths matching the efficiency claims. The specific stress test: does YOCO-U's quality degrade as context length grows beyond the 16 windows covered at 8K, and if so, at what length does degradation begin? The architectural concern is that the Self-Decoder's 512-token sliding window means tokens at position 256K were last directly processed by the Self-Decoder 500 windows ago. The Cross-Decoder's global cross-attention should theoretically retrieve these tokens, but the quality of the global KV cache — produced after `T=3` Self-Decoder iterations — might degrade for very distant tokens if the iterative refinement introduces subtle distortions. A negative result (quality degradation at 128K) would identify a fundamental limitation of the local-global factorization and motivate architectures with hybrid attention patterns (e.g., global tokens interleaved with sliding windows).

**Combining YOCO-U's recursive pretraining with inference-time test-time compute strategies in a controlled comparison.** The paper argues that recursive pretraining and test-time inference scaling are orthogonal, but provides no empirical test. A clean experiment: train non-recursive YOCO and YOCO-U to comparable downstream accuracy (using the token efficiency result: 80B tokens for YOCO-U, 210B for non-recursive YOCO), then evaluate both with increasing inference-time compute budgets — majority voting with 1, 4, 16, and 64 samples, chain-of-thought prompting, or best-of-N with an outcome verifier. The hypothesis is that YOCO-U's gains are partially redundant with test-time compute: if recursion already provides iterative refinement during pretraining, the marginal benefit of additional inference-time reasoning might be smaller than for non-recursive models. If confirmed, this would suggest that recursive pretraining and test-time scaling are **substitutive** rather than additive — and the optimal strategy depends on whether you want to spend compute during training (recursion) or inference (test-time strategies). If disconfirmed (the improvements are additive and independent), it strengthens the paper's orthogonality claim and motivates joint optimization of pretraining depth and inference budget.

**Ablation on efficient attention variants within YOCO-U with matched training budget.** The paper claims compatibility with RetNet, Mamba, and gated DeltaNet as alternatives to sliding-window attention, and asserts they "perform similarly to SWA within hybrid architectures" (Section 3.2), but provides no data. A rigorous ablation would train YOCO-U models with sliding-window, Mamba-2, RetNet, and gated DeltaNet Self-Decoders, all at the same parameter count and training budget (e.g., 1.3B parameters, 100B tokens), and evaluate on: (1) downstream task accuracy (Tables 2-3 benchmarks); (2) long-context perplexity at 8K-128K tokens (Figure 4 benchmarks); (3) inference throughput and KV cache memory at 8K-256K (Figure 7 benchmarks). The key question: does the choice of efficient attention affect quality, and is there a Pareto frontier between quality and efficiency? Sliding-window attention is simple and well-optimized (FlashAttention-compatible) but has limited receptive field; linear attention variants offer theoretically unbounded context but can struggle with recall-intensive tasks. The YOCO-U framework — where global retrieval is handled by cross-attention — might reduce the recall burden on the Self-Decoder, making linear attention more competitive. This ablation would guide practitioners choosing among efficient attention backends for their recursive architectures.

### Practical Applications and Downstream Use Cases

**Long-context document processing and retrieval-augmented generation (RAG) serving.** YOCO-U's pre-filling throughput advantage — 76K tokens/second at 256K context versus 7.5K for standard Transformers (Figure 7a, ~10× faster) — directly benefits applications that need to ingest and process very long documents: legal document review, scientific literature synthesis, codebase-wide analysis. The linear pre-filling scaling means that doubling the context length from 128K to 256K increases pre-filling time by only ~0.2% (Table 8: 75,534 to 76,301 tokens/second), while a standard Transformer's throughput drops by ~50% (14,987 to 7,475). For a RAG system that retrieves 50 documents of ~5K tokens each (250K total context) and generates a 500-token summary, YOCO-U would pre-fill in ~3.3 seconds and decode in ~1.7 seconds (at 303 tokens/second), totaling ~5 seconds per query. A standard Transformer would take ~33 seconds for pre-filling alone. The 542 MB KV cache per sequence (at 256K) means a single H100-80GB GPU can hold ~148 concurrent sequences, enabling batch processing of long-context queries. The practical deployment scenario is a legal-tech company processing discovery documents: YOCO-U enables real-time interactive querying over full document sets that would be batch-processed with minutes of latency using standard Transformers.

**On-device or edge deployment of reasoning-capable models with limited memory.** The parameter scaling result — YOCO-U achieves comparable performance with approximately 50% fewer total parameters (Figure 5, left) — translates directly to smaller model weights on disk and in GPU memory. Combined with the negligible KV cache overhead (542 MB at 256K context vs. 10.2 GB for standard Transformers), YOCO-U enables a deployment profile where a 7B-total-parameter YOCO-U model (approximately matching a 14B standard Transformer in quality, per the parameter scaling trend) fits within the memory budget of a consumer GPU (e.g., 24 GB RTX 4090) while still serving long-context queries. The 62% token efficiency result (Figure 2, right) further reduces the training cost to achieve that quality. For edge AI applications — code assistants running locally on developer laptops, privacy-sensitive document analysis, offline language models for mobile devices — YOCO-U's combination of parameter efficiency and long-context memory efficiency is directly applicable. The specific deployment: a 7B YOCO-U model with `T=3` iterations, trained on 500B tokens (which the token efficiency result suggests would match a 14B non-recursive model trained on ~1.3T tokens), running on a single laptop GPU with a 128K context window, achieving sub-second pre-filling for typical documents and interactive decoding latency.

**Self-improving training pipelines with iterative data generation.** The paper's thinking SFT results (Figure 3) — YOCO-U achieves 67.1% average on 11 math benchmarks after specialized training versus 51.6% for non-recursive YOCO — suggest that recursive architectures are particularly effective at leveraging specialized training data. In a self-improvement pipeline (e.g., STaR, ReSTᴱᴹ, rejection sampling fine-tuning), a base model generates solutions to training problems, correct solutions are filtered and used for further fine-tuning, and the cycle repeats. YOCO-U's recursive depth provides more representational capacity to refine solutions during generation (the `T=3` Self-Decoder iterations allow iterative improvement of the reasoning path), potentially increasing the yield of correct solutions per generation batch. This higher yield would accelerate the self-improvement loop — fewer generation rounds needed to accumulate a training batch of correct solutions. The practical scenario: a math tutoring system that continuously improves by generating and verifying solutions to new problems, using YOCO-U as the base generator for its higher initial accuracy and greater capacity to refine reasoning through recursive computation during generation.

### When to Prefer This Method

The paper explicitly positions YOCO-U against specific architectural alternatives — non-recursive YOCO, standard Transformers with full-attention recursion (RINS, UT), and parallel scaling (ParScale) — each with quantified tradeoffs in Tables 2-3 and Figure 7. The following decision rules are grounded in the paper's empirical findings:

- **Prefer YOCO-U over non-recursive YOCO when** training token efficiency or parameter efficiency is the primary constraint, and the deployment will serve long-context queries where the Cross-Decoder dominates inference cost. YOCO-U achieves the same loss with ~62% fewer training tokens (Figure 2, right) and comparable performance with ~50% fewer total parameters (Figure 5, left), while the 5% decoding throughput reduction (Figure 7b: 303 vs. 318 tokens/second at 256K) is negligible. This applies to scenarios where training budget is limited but inference throughput is not the binding constraint — e.g., research labs training models on academic compute budgets.

- **Prefer YOCO-U over RINS or Universal Transformer (full-attention recursive architectures) when** inference memory and long-context throughput are critical, and the deployment must serve at least moderately long sequences (16K+ tokens). YOCO-U achieves identical downstream accuracy to RINS (Table 3: both 48.3 average) but with 38× less KV cache memory at 256K context (Figure 7c: 542 MB vs. 20,480 MB) and 5.4× higher decoding throughput (Figure 7b: 303 vs. 56 tokens/second). This applies to nearly all production serving scenarios — the memory savings translate directly to larger batch sizes, higher throughput per GPU, and lower serving cost. The only scenario where RINS might be preferred is very short contexts (≤4K tokens) where the KV cache advantage is smaller and RINS's full-attention recursion might provide quality benefits not captured in the paper's benchmarks (not empirically demonstrated here).

- **Prefer YOCO-U over ParScale or other parallel scaling methods when** representational depth matters more than latency (for a given FLOPs budget). The paper's architecture comparison shows ParScale achieving 46.8 average versus YOCO-U's 48.3 (Table 3), consistent with the claim that "recursive scaling is more effective than parallel scaling" because it increases modeling depth. ParScale or parallel branch methods may be preferred when latency is the absolute binding constraint (parallel branches add no serial depth) and the quality tradeoff is acceptable.

- **Prefer non-recursive YOCO when** the absolute minimum inference latency is required and the quality gap from recursion (+4.45 average under equal FLOPs, Table 2) is acceptable. Non-recursive YOCO achieves 2.9× faster pre-filling (Figure 7a: 220K vs. 76K tokens/second at 256K) and 5% faster decoding (Figure 7b: 318 vs. 303 tokens/second) because it avoids the `T=3` Self-Decoder iterations entirely. For applications serving very short contexts at extremely high throughput — e.g., real-time chat with brief messages — the marginal latency cost of recursion may outweigh the quality benefit. The paper provides the numbers to make this cost-benefit calculation for a specific deployment context.