ArXiv: 2601.22156
🎯 Pitch
You can convert a 7B Transformer into a hybrid RNN model that runs 3× faster at 512K context using only 2.3B tokens of distillation—less than 0.01% of the original pre-training data—yet prior methods needed 10–400B tokens and still failed at long-context recall. The key is a new position encoding scheme (HyPE) that applies RoPE in the RNN layers and NoPE in the attention layers, combined with the insight that data-independent forget gates are essential for length generalization beyond the training horizon.
1. Executive Summary
This paper introduces HALO (Hybrid Attention via Layer Optimization), a cross-architecture distillation pipeline that converts pre-trained Transformer models into RNN-attention hybrid architectures using fewer than 3B tokens, and HypeNet, a novel hybrid architecture featuring HyPE (Hybrid Position Encoding) — a position encoding scheme that applies RoPE in RNN layers and NoPE in attention layers, combined with dynamic attention logits scaling — along with complementary architectural modifications (QK-normalization, GQA-to-MHA decoupling, and output gating). Applied to the Qwen3 series (1.7B, 4B, 8B), HypeNet achieves up to 3.0× decoding speedup and 3.4× prefilling speedup at 512K context length while maintaining competitive commonsense reasoning performance, and delivers superior long-context recall on NIAH benchmarks compared to prior distilled hybrid models, despite using an order of magnitude fewer training tokens (2.3B vs. 20B–400B). The paper establishes that strong length generalization to contexts far beyond training length (up to 256K tokens from 16K training) is achievable through the HyPE design, but only when paired with Lightning Attention as the RNN mixer — a finding grounded in the observation that data-independent forget gates avoid the poor length generalization that data-dependent forget gates introduce.
2. Context and Motivation
The Core Problem: Transformer Inference Is Quadratically Expensive, and Hybrid Models Are Hard to Build
The fundamental challenge this paper addresses is the tension between inference efficiency and model quality in long-context language modeling. Transformer-based models, which dominate the current LLM landscape, rely on softmax attention — a mechanism whose computational and memory cost scales quadratically with sequence length. For a context of tokens, the attention matrix requires operations (where is the head dimension), and the key-value (KV) cache grows linearly in , consuming memory that becomes prohibitive at modern context lengths of 128K, 256K, or 1M tokens. The paper makes this concrete in Figure 1 (right): at 1M context length, Qwen3-1.7B simply runs out of GPU memory on a single A800, making it unusable for ultra-long-context applications regardless of how much patience one has for latency.
This is not merely an academic concern — it has direct practical implications. Long-horizon reasoning tasks (multi-step mathematical proofs, document-grounded question answering over book-length texts), agentic behaviors requiring extended interaction histories, and code generation over entire codebases all push context windows to their limits. The quadratic bottleneck forces practitioners into uncomfortable tradeoffs: use a smaller context window and truncate information, deploy larger or more GPUs to handle the KV cache, or accept unacceptably slow generation. None of these are satisfactory.
Recurrent neural networks (RNNs) offer a principled escape from this quadratic trap. RNN layers — including linear attention variants and state space models — process tokens sequentially with a fixed-size recurrent state , giving them complexity that grows only linearly with sequence length, and eliminating the KV cache entirely. The recurrent state serves as a compressed summary of all past context, updated token by token. For long sequences, the theoretical speedup over softmax attention is dramatic: as context length grows, the quadratic term in attention eventually dominates any constant factors, while the RNN's cost remains proportional to .
However, RNNs have a well-documented weakness: their fixed-size state imposes an information bottleneck that limits performance on recall-intensive tasks — tasks requiring the model to retrieve or copy specific information from earlier in the sequence. This is particularly acute for needle-in-a-haystack (NIAH) style evaluations, where the model must locate and reproduce a fact buried deep in a long context. Pure RNN models consistently underperform softmax attention Transformers on such tasks, even when their commonsense reasoning scores are competitive. The reason is intuitive: an RNN's state vector, typically of size (e.g., elements for a single head), must compress an arbitrarily long context into this fixed representation. Information that appeared thousands of tokens ago competes for representation capacity with recent information, and recall fidelity degrades.
This creates a capability-efficiency tradeoff:
- Pure Transformers excel at recall (thanks to exact attention over all past tokens) but are slow and memory-hungry on long contexts.
- Pure RNNs are fast and memory-efficient but lose recall performance.
The Hybrid Compromise and Why It's Inaccessible
The natural resolution — and the one the field has converged toward — is hybrid architectures that interleave softmax attention layers with RNN layers. The attention layers provide a small number of "recall checkpoints" where the model can access the full context with quadratic attention, while the RNN layers handle the bulk of computation efficiently. Models like Jamba, MiniMax-01, Qwen3-Next, Kimi-Linear, and NVIDIA Nemotron 3 have demonstrated that this architecture achieves a favorable performance-throughput tradeoff, matching Transformer-level commonsense reasoning while offering substantial speedups on long sequences.
But here is the problem that motivates HALO: building these hybrid models requires large-scale pre-training from scratch, which is prohibitively expensive for most research teams. Training a frontier hybrid model from scratch — even a modestly sized one — requires hundreds of billions of tokens, thousands of GPU-hours, and significant engineering infrastructure. The Qwen3-Next hybrid, for example, was trained from scratch at a scale beyond what any academic lab can afford. This creates a barrier to entry that excludes the vast majority of the research community from innovating on hybrid architectures, even though hybrid architectures are arguably the most fertile ground for long-context efficiency research.
This is where distillation from pre-trained Transformers enters as an attractive alternative. Rather than training a hybrid model from scratch, one can take an existing, publicly available Transformer checkpoint and convert some of its attention layers into RNN layers through a training process that requires far fewer tokens than full pre-training. This approach has two compelling advantages: (1) it leverages the massive investment in Transformer pre-training, and (2) it allows the research community to validate ideas about hybrid architectures at a fraction of the cost, accelerating the research cycle.
Where Existing Distillation Methods Fall Short
Several prior works have attempted Transformer-to-hybrid distillation, and the paper catalogs them in Table 1. The methods and their training token requirements are:
- Mamba-in-the-Llama (Wang et al., Aug 2024): 20B tokens
- SMART / Zebra-Llama (Yang et al., May 2025): >7B tokens
- RAD (Hoshino et al., May 2025): 20B tokens
- Jet-Nemotron (Gu et al., Aug 2025): 400B tokens
- KL-LS (Li et al., Dec 2025): 25B tokens
All of these require tens to hundreds of billions of tokens for distillation. While this is still far less than pre-training from scratch (which can use trillions of tokens), it remains out of reach for many academic teams. Training on 20B tokens is a non-trivial undertaking — requiring substantial GPU clusters for days or weeks — and the cost scales with both model size and the number of experiments one wants to run. For a research lab that wants to iterate on architecture design, layer selection strategies, or RNN mixer choices, running dozens of 20B-token distillation runs is infeasible.
This is the first gap HALO addresses: data efficiency. The paper claims to achieve competitive or superior results with just 2.3B tokens, a 10–200× reduction over prior methods. This is not a small constant-factor improvement — it changes what is practical for academic research. At 2.3B tokens, a distillation run on a 1.7B-parameter model takes approximately 91 A800 GPU-hours total (Table 8), which is feasible on a small cluster or even a single 8-GPU node running for a couple of days. This enables rapid experimentation, ablation studies, and architecture search that would be impossible with 20B+ token budgets.
The second and arguably more critical gap: prior distilled hybrid models exhibit severe performance degradation on long-context tasks. This is precisely the scenario where hybrid models are supposed to shine — if a distilled hybrid model is only good on short-context commonsense reasoning but falls apart on long-context recall, it defeats the purpose of building a hybrid in the first place. The paper provides direct evidence of this failure mode. Consider Table 2, which reports NIAH (Needle-in-a-Haystack) performance at various context lengths:
Jet-Nemotron (2B, trained on 400B tokens): At 32K context, NIAH-Single-1 accuracy is 99.8% — essentially perfect. But at 64K, it drops to 56.0%, and at 128K and beyond, it collapses to 0.0%. On NIAH-Single-3, performance at 64K is already 15.4%, and at 128K it is 0.0%. This is a catastrophic failure: the model completely loses the ability to perform the core recall task beyond 64K tokens, despite being trained on a massive 400B-token corpus.
KL-LS with GDN mixer (3B, 25B tokens): Performs better than Jet-Nemotron but still degrades substantially. On NIAH-Single-1, accuracy falls from 99.8% at 32K to 68.4% at 128K to 14.8% at 256K. The degradation is progressive, and by 256K context, recall is effectively random.
In stark contrast, the Qwen3 teacher model (1.7B, which has all softmax attention layers and no RNNs) maintains near-perfect recall at 128K (100%, 98.8%, 98.4% across the three NIAH variants) and only degrades significantly at 256K (mostly on NIAH-Single-2 and -3, where accuracy drops to ~17–19%). The teacher model, despite being smaller than KL-LS, demonstrates that softmax attention provides robust long-context recall that current distillation methods fail to preserve.
This long-context degradation is the paper's central motivating observation. It suggests that existing distillation methods, while adequate for short-context commonsense reasoning, fail to transfer the recall capabilities that make Transformers effective on long contexts. The RNN layers in these distilled hybrids are not learning to compress and retrieve information from long contexts effectively, and the remaining attention layers are either too few or positioned suboptimally to compensate.
The Position Encoding Puzzle
A more subtle challenge that the paper identifies involves position encoding (PE) — how the model represents token order and relative distances. This may seem like a minor architectural detail, but the paper argues it is central to length generalization: the ability of a model trained on sequences of length to perform well on sequences of length .
In the current Transformer landscape, Rotary Position Embedding (RoPE) has become the de facto standard. RoPE encodes positional information by applying position-dependent rotations to query and key vectors before the attention computation. This works well for in-distribution lengths but has a known weakness: standard RoPE models struggle with length generalization beyond their training context, typically requiring specialized scaling techniques (like YaRN) for context window extension.
In contrast, NoPE (no position encoding in attention layers) has been shown to exhibit superior training-free length generalization. Without explicit position encodings, the model learns to infer positional relationships from the sequential structure of the data, and this learned implicit position understanding often generalizes better to unseen lengths. However, NoPE-only models typically underperform RoPE-equipped models on standard benchmarks at training-length contexts — the explicit position signal is helpful when it is available.
The challenge for hybrid models is that attention and RNN layers have fundamentally different ways of handling positional information:
- Attention layers in most Transformers use RoPE.
- RNN layers are inherently position-aware through their state transition mechanism — the update rule naturally encodes sequential order. Most RNNs therefore use NoPE (no RoPE applied to QK).
In existing hybrid architectures (Qwen3-Next, MiniMax-01), this leads to a heterogeneous PE scheme: attention layers get RoPE, RNN layers get NoPE. The paper's key insight — motivated by the finding that RNNs have a limited "receptive field" (Chen et al., 2025b) — is that this arrangement may be suboptimal for length generalization.
The reasoning goes as follows: RNNs primarily model short-distance dependencies (within their receptive field), while attention layers model long-distance dependencies that the RNN's fixed-size state cannot capture. When the context length exceeds the RNN's receptive field, the RNN layers become agnostic to context length — their behavior is essentially the same regardless of whether the sequence is 10K or 100K tokens long. Therefore, the model's length generalization depends primarily on the attention layers, not the RNN layers.
This suggests a counterintuitive design: give NoPE to the attention layers (which handle long-distance dependencies and benefit from NoPE's superior length generalization) and give RoPE to the RNN layers (which handle short-distance dependencies and benefit from RoPE's richer positional information at training-length contexts). This is HyPE, and it flips the conventional hybrid design on its head.
The Attention Layer Selection Problem
When distilling a Transformer into a hybrid, a critical decision is which attention layers to keep unconverted and which to replace with RNNs. The number of attention layers is directly tied to efficiency: more attention layers mean a larger KV cache and more quadratic-attention operations, reducing the speedup over a pure Transformer. The goal is to retain as few attention layers as possible while preserving performance, especially recall capability.
Prior work has approached this with varying sophistication:
- Even distribution (e.g., keep every 4th layer): Simple but suboptimal, as it ignores the fact that different layers have different functional roles.
- Performance-drop-based selection (Jet-Nemotron): Measure the performance drop on certain tasks when each attention layer is individually replaced with an RNN, and keep the layers whose replacement causes the largest drop.
- KL-divergence-based selection (KL-LS): Use the KL divergence from the teacher model as a proxy for layer importance, requiring a full distillation run for each layer to estimate importance — computationally expensive.
- Output distribution shift (Zebra-Llama): Measure how much the model's output distribution changes when a layer is replaced, as a proxy for importance.
The paper identifies a specific weakness in these approaches: they often focus on a single metric (either commonsense reasoning or recall) rather than explicitly trading off between the two. Since RNN layers underperform attention on recall-intensive tasks specifically (not on all tasks equally), the optimal layer selection should prioritize keeping attention layers that are most critical for recall while being willing to sacrifice layers that are less important for recall but more important for non-recall tasks. The paper's proposed importance score in Equation 8 formalizes this tradeoff: a layer is considered important if replacing it causes a large drop in recall relative to the drop in commonsense reasoning. The ratio penalizes layers whose replacement mainly hurts CSR (commonsense reasoning) — those can potentially be addressed through better RNN training — while prioritizing layers whose replacement disproportionately hurts recall.
How This Paper Positions Itself
The paper positions HALO and HypeNet not as a single monolithic contribution but as an integrated solution to three interconnected problems that have hindered adoption of distilled hybrid architectures:
-
Data efficiency: HALO achieves competitive results with 2.3B tokens, a 10–200× reduction from prior methods, by building on and improving the RADLADS framework with careful stage design (hidden state alignment via MSE, followed by end-to-end KL distillation, followed by long-context fine-tuning) and a novel layer selection method.
-
Long-context recall: HypeNet's architectural design — particularly HyPE with its counterintuitive RoPE-for-RNN/NoPE-for-attention scheme and dynamic attention logits scaling — directly targets the length generalization problem that causes prior distilled hybrids to collapse on long contexts. The results in Table 2 show HypeNet maintaining 99.8%, 99.6%, and 86.2% NIAH accuracy at 256K context (for the three NIAH variants), compared to 0.0% for Jet-Nemotron and 10.4–14.8% for KL-LS.
-
RNN mixer compatibility and selection: By making HypeNet agnostic to the specific RNN mixer (as long as it follows the QKV-based formulation in Equations 4–6), the paper enables systematic comparison of different RNN variants and identifies Lightning Attention — with its data-independent forget gates — as the superior choice for length generalization. This finding, validated in Section 6.3, is grounded in the observation that data-dependent forget gates cause the RNN's effective receptive field to shrink over long sequences (the "inability to forget" problem documented by Chen et al., 2025b).
Importantly, the paper explicitly frames itself within the academic research context: "They are valuable since they allow teams without resources to scale up pre-training to validate research ideas." This is not a paper about building the best possible hybrid model (that would require pre-training from scratch at scale). Rather, it is about lowering the barrier to entry for hybrid architecture research by making distillation practical and effective. The 2.3B-token budget, combined with the use of only open-source data (FineWeb-edu) and publicly available base models (Qwen3), means that the entire pipeline can be replicated by academic groups with modest GPU resources. The code and checkpoints are released, further reducing the barrier.
The paper also positions itself as complementing rather than competing with methods that train hybrid models from scratch. Distilled models, as the paper acknowledges, typically underperform those trained from scratch. But distillation is the only viable path for most research teams to experiment with hybrid architectures, and improving distillation unlocks a wave of experimentation and validation that would otherwise be impossible.
3. Technical Approach
3.1 Reader Orientation
HALO is a three-stage training pipeline that converts a pre-trained Transformer language model into a hybrid model where most attention layers are replaced by recurrent (RNN) layers, and HypeNet is the specific hybrid architecture produced by this pipeline, featuring a novel position encoding scheme (HyPE) and several architectural enhancements. The system solves the problem of making Transformer models efficient for long-context inference without requiring prohibitively expensive pre-training from scratch, by distilling existing Transformer checkpoints into hybrid architectures using only 2.3B tokens while preserving — and in some cases improving — long-context recall performance that prior distillation methods catastrophically degrade.
3.2 Big-Picture Architecture (Diagram in Words)
The system has two intertwined halves: the HALO distillation pipeline (the process) and the HypeNet model architecture (the product). The pipeline takes a pre-trained Transformer as input and produces HypeNet as output through four sequential phases:
-
Initialization Stage (weight transfer): For every attention layer in the Transformer, an RNN layer is instantiated using the attention layer's projection weights (
\(\mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v, \mathbf{W}_o\)) as initialization. Any RNN-specific components not covered by these weights (e.g., output gates) are randomly initialized. -
Stage 1 (hidden state alignment): Each RNN layer is trained independently to match the output of its corresponding attention teacher layer, using mean squared error. Only the RNN layers are trained; the rest of the model is frozen. This produces, for every attention layer, a student RNN layer that can approximately replicate the teacher's behavior.
-
Attention Layer Selection: A lightweight evaluation is performed to determine which attention layers to keep as softmax attention (typically 25% of layers, or
\(\lfloor L/4 \rfloor\)) and which to replace with the trained RNN layers. The selection criterion explicitly trades off recall performance against commonsense reasoning performance, prioritizing layers critical for recall. -
Stage 2 (knowledge distillation) and Stage 3 (finetuning): The final hybrid model is constructed using the selected attention layers and trained RNN layers, then undergoes end-to-end KL-divergence distillation (Stage 2, 1B tokens) followed by long-context finetuning (Stage 3, 1B tokens at 16K context length).
The HypeNet architecture that emerges from this process incorporates four key modifications (marked ➊–➍ in Figure 3): HyPE position encoding (➊), QK-normalization in RNN layers (➋), decoupling grouped-query attention heads into multi-head attention in RNN layers (➌), and output gating in both RNN and attention layers (➍). The RNN mixer choice is architecturally decoupled from the rest of the design, though Lightning Attention is the recommended choice based on empirical length generalization results.
Information flows as follows: input tokens enter the first layer → each layer applies either softmax attention (with NoPE and dynamic logits scaling) or an RNN update (with RoPE, QK-norm, and output gating) → the outputs pass through a standard SwiGLU MLP with RMSNorm → the process repeats for all \(L\) layers → the final layer output is projected to vocabulary logits via tied embeddings.
3.3 Roadmap for the Deep Dive
- First, the formal model formulation (Equations 1–6), which establishes the shared mathematical language for both attention and RNN layers and defines the key notation (
\(\mathcal{I}_{\text{attn}}\),\(\mathbf{S}_t\),\(\mathbf{F}_t\)) used throughout the paper. - Second, the HALO pipeline in detail — weight transfer, hidden state alignment, attention layer selection, knowledge distillation, and finetuning — since this is the enabling infrastructure that makes HypeNet accessible without pre-training from scratch.
- Third, the HyPE position encoding scheme and attention logits scaling, because position encoding is the central architectural innovation that enables HypeNet's superior length generalization, and its design motivation requires understanding the receptive field argument.
- Fourth, the complementary architectural modifications (QK-norm, GQA-to-MHA decoupling, output gating), explaining what each does and why it helps specifically in the distillation setting.
- Fifth, the RNN mixer formulation and the choice of Lightning Attention, including the mathematical form of the forget gate and the empirical rationale for data-independent forgetting over data-dependent alternatives.
- Sixth, a synthesis of design choices: why the full combination (HyPE + Lightning Attention + output gating + QK-norm + 25% attention layer retention) works together, and what would break if any component were removed.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily a systems and architecture paper whose core idea is that a carefully designed position encoding scheme (HyPE) combined with a data-independent RNN mixer (Lightning Attention) and a lightweight distillation pipeline (HALO) can produce hybrid models that match or exceed the long-context recall of the original Transformer teacher while requiring dramatically fewer training tokens than prior distillation methods.
Formal Model Formulation: A Shared Language for Attention and RNN Layers
The paper begins by establishing a unified mathematical notation that can describe both softmax attention layers and RNN layers within the same framework. This is essential because HALO needs to convert one type of layer into the other — having a shared vocabulary makes the conversion operation well-defined.
Every model in this study — whether a pure Transformer or a hybrid — consists of a stack of \(L\) layers, indexed by \(l \in \{1, \dots, L\}\). The \(l\)-th layer operates on an input matrix \(\mathbf{X}^{(l-1)} \in \mathbb{R}^{T \times d}\), where \(T\) is the sequence length and \(d\) is the model's hidden dimension. The output is \(\mathbf{X}^{(l)} \in \mathbb{R}^{T \times d}\), which becomes the input to the next layer.
Each layer follows a standard residual structure with two sub-layers:
where \(\text{Mixer}^{(l)}\) is either a softmax attention operation or an RNN operation, \(\text{MLP}^{(l)}\) is a position-wise feedforward network (SwiGLU in practice), and \(\mathbf{H}^{(l)} \in \mathbb{R}^{T \times d}\) is the intermediate hidden state after the mixer but before the MLP.
What this computes: the layer takes the previous layer's output \(\mathbf{X}^{(l-1)}\), applies a token-mixing operation (either attention or recurrence) to produce \(\mathbf{H}^{(l)}\), adds the original input as a residual connection, then applies a per-token feedforward transformation and another residual connection to produce \(\mathbf{X}^{(l)}\).
Why this form: the residual structure is the standard Transformer formulation from Vaswani et al. and is preserved exactly in hybrid models. This is critical for HALO because it means the non-mixer components (MLPs, layer norms) are shared between architectures and need no modification during conversion. Only the \(\text{Mixer}^{(l)}\) changes.
For a hybrid model specifically, an attention index set \(\mathcal{I}_{\text{attn}} = \{l_{\text{attn},i} \mid i = 1, \dots, L_{\text{attn}}\}\) specifies which layer indices use softmax attention. All other layers use an RNN mixer:
where \(L_{\text{attn}}\) is the number of attention layers retained. By default, the paper uses \(L_{\text{attn}} = \lfloor L/4 \rfloor\), meaning 25% of layers remain as softmax attention. For Qwen3-1.7B with \(L = 28\), this means 7 attention layers and 21 RNN layers.
Softmax Attention Layers
For layers where \(l \in \mathcal{I}_{\text{attn}}\), the mixer is standard softmax attention (ignoring the multi-head mechanism for notational simplicity):
where \(\mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v, \mathbf{W}_o \in \mathbb{R}^{d \times d_h}\) are learnable projection matrices (with \(d_h\) being the per-head dimension), \(\mathbf{M}\) is the causal attention mask, and \(\odot\) denotes element-wise multiplication.
What this computes: for each query position, compute dot-product similarity scores with all key positions, scale by \(1/\sqrt{d_h}\) to control variance, apply causal masking to prevent attending to future tokens, normalize with softmax to produce attention weights, then compute a weighted sum of value vectors. The output is projected back to dimension \(d\) by \(\mathbf{W}_o^\top\).
Why this form: this is the standard Transformer attention, reproduced here only to establish notation. The key detail for HALO is that the weight matrices \(\mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v, \mathbf{W}_o\) are what get transferred when initializing an RNN layer, and the \(1/\sqrt{d_h}\) scaling factor interacts with the dynamic attention logits scaling introduced later.
Modern RNN Layers
For layers where \(l \notin \mathcal{I}_{\text{attn}}\), the mixer is a recurrent layer that the paper constrains to a specific outer-product-based form. This constraint is important because it ensures the RNN layer takes the same Q, K, V inputs as the attention layer it replaces, enabling direct weight transfer.
At each time step \(t\), the RNN processes input \(\mathbf{x}_t \in \mathbb{R}^{1 \times d}\) (the \(t\)-th row of \(\mathbf{X}\)) and maintains a recurrent state \(\mathbf{S}_t \in \mathbb{R}^{d_h \times d_h}\):
where \(\mathbf{F}_t \in \mathbb{R}^{d_h \times d_h}\) is the transition matrix (also called the forget gate), which is a function of the current input \(\mathbf{x}_t\).
Equation-by-equation meaning:
-
QKV projection (same as attention): The input
\(\mathbf{x}_t\)is projected into query, key, and value vectors using the exact same weight matrices\(\mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v\)as softmax attention. This is the crux of the weight transfer: the RNN reinterprets the attention weights in a recurrent context. -
State update (the "update rule"): The recurrent state
\(\mathbf{S}_{t-1}\)from the previous time step is first multiplied by the transition matrix\(\mathbf{F}_t\)(which controls how much of the past state to retain or forget), then the outer product\(\mathbf{k}_t^\top \mathbf{v}_t \in \mathbb{R}^{d_h \times d_h}\)of the current key and value vectors is added. This outer product is a rank-1 matrix that encodes the current token's information in a form that can be queried later. -
Output computation (the "query rule"): The current query
\(\mathbf{q}_t\)is multiplied by the recurrent state\(\mathbf{S}_t\), producing a vector that can be interpreted as "what information from the past is relevant to the current query." This is analogous to how softmax attention computes\(\mathbf{q}_t \mathbf{K}^\top \mathbf{V}\), but without explicitly materializing all past keys — they are compressed in\(\mathbf{S}_t\). The result is then projected back to dimension\(d\)via\(\mathbf{W}_o^\top\).
Why this form: this outer-product-based RNN formulation encompasses many modern recurrent architectures as special cases, including Mamba2 (with \(\mathbf{F}_t = \alpha_t \mathbf{I}\)), Gated Linear Attention or GLA (with \(\mathbf{F}_t = \text{diag}(\alpha_t)\)), Gated DeltaNet or GDN (with \(\mathbf{F}_t = \alpha_t(\mathbf{I} - \beta_t \mathbf{k}_t^\top \mathbf{k}_t)\)), and Lightning Attention (with \(\mathbf{F}_t = \gamma \mathbf{I}\) where \(\gamma\) is a fixed scalar). The paper's key constraint is that \(\mathbf{F}_t\) must be diagonal or rank-1 for computational efficiency — this enables fast parallel scan algorithms that compute all \(\mathbf{S}_t\) in \(O(T \log T)\) time rather than sequentially.
The explicit separation into Q, K, V projections followed by a state update is what makes HALO's weight transfer possible. An RNN architecture that does not use Q, K, V projections (e.g., the original LSTM or GRU) cannot be initialized from attention weights and would require training from scratch.
HALO Stage-by-Stage: The Distillation Pipeline in Detail
HALO (Hybrid Attention via Layer Optimization) is the pipeline that converts a pre-trained Transformer into a HypeNet hybrid model. It is directly adapted from RADLADS (Rapid Attention Distillation to Linear Attention Decoders at Scale, Goldstein et al., 2025), with modifications that improve data efficiency and enable hybrid (rather than pure-RNN) conversion. The pipeline proceeds in four phases.
Phase 0: Initialization Stage — Attention Weight Transfer
This is a training-free phase that happens once, before any gradient updates.
Given a pre-trained Transformer model consisting entirely of softmax attention layers, for each layer index \(l \in \{1, \dots, L\}\), the initialization procedure:
- Extracts the attention projection weights
\(\mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v, \mathbf{W}_o\)from layer\(l\)of the teacher model. - Instantiates a corresponding RNN layer
\(\text{RNN}^{(l)}\)that uses these weights directly. The RNN layer's Q, K, V projections are initialized to the exact same matrices as the attention layer's projections. - Randomly initializes any RNN-specific components that have no analog in softmax attention. This includes the output gate weights
\(\mathbf{W}_z\)(see Section 5.2) and any parameters specific to the chosen RNN mixer (e.g., the forget gate slope rates in Lightning Attention, which are fixed by a formula and not learned).
The key insight is that because the RNN formulation in Equations 4–6 uses Q, K, V projections with exactly the same shapes as the attention layer, the weight transfer is simply a shallow copy of the tensor values. No reshaping, no linear algebra transformations, no learned mappings. This is what makes the initialization so fast — it is essentially free.
After initialization, for every attention layer in the teacher, there exists a corresponding student RNN layer that can accept the same input \(\mathbf{X}^{(l-1)}\) and produce an output that — at initialization — will not match the teacher, but has a structurally sensible starting point for training.
Phase 1: Stage 1 — Hidden State Alignment
Stage 1 trains each instantiated RNN layer independently to match the output of its corresponding teacher attention layer. Crucially, only the RNN layers are trained; all other parameters in the model (the MLPs, the attention layers in other positions, the embedding and output layers) are frozen.
The training objective for layer \(l\) is:
where \(\mathbf{Y}^{(l)}_{\text{teacher}} \in \mathbb{R}^{T \times d}\) is the output of the \(l\)-th attention layer in the frozen teacher model (after the residual connection and before the MLP in the teacher's formulation), and \(\text{RNN}^{(l)}(\mathbf{X}^{(l-1)})\) is the output of the student RNN layer given the same input.
What this computes: for each position in each training sequence, the squared difference between the RNN's output vector and the attention layer's output vector, summed over all \(d\) dimensions and averaged over all \(T\) positions and all training examples. The loss is a single scalar measuring how well the RNN replicates the attention layer's behavior.
Why this form (MSE rather than KL divergence): Stage 1 operates at the level of hidden states (intermediate activations), not output logits. The hidden states are continuous vectors in \(\mathbb{R}^d\), so MSE is the natural regression loss. Using KL divergence here would require converting hidden states to probability distributions, which would add unnecessary complexity and potential instability. MSE directly encourages the RNN to produce the same vector output as the attention layer for every input, which is exactly the goal: make the RNN a drop-in replacement.
Why train independently per layer: training all RNN layers simultaneously would be an end-to-end optimization problem, which is much harder — errors compound across layers, and the optimization landscape has many local minima. By training each layer in isolation, with frozen inputs from the teacher, each RNN only needs to solve a single-layer regression problem. This is far more data-efficient and converges faster. The paper reports using 320M tokens for Stage 1, with a cosine learning rate schedule from \(1 \times 10^{-3}\) down to \(1 \times 10^{-5}\), batch size 32, and context length 512 (Table 5).
A notable challenge that Stage 1 does not address: the RNN layers are trained on the teacher's hidden states, which assume that previous layers are attention layers. But in the final hybrid model, the previous layers may be RNN layers with slightly different output distributions. This distribution shift is why subsequent stages (Stage 2 and Stage 3) are needed — they fine-tune the full model end-to-end to account for this compounding effect.
After Stage 1 completes, the paper has trained \(L\) independent RNN layers, each of which can approximately replace its corresponding attention layer. But the question remains: which ones should actually be replaced? This is where attention layer selection enters.
Phase 2: Attention Layer Selection
The goal of layer selection is to determine the optimal set \(\mathcal{I}_{\text{attn}}\) of size \(k = \lfloor L/4 \rfloor\) — the 25% of layers that will remain as softmax attention in the final hybrid model. The criterion must identify layers that are most critical for recall, since RNNs are known to underperform attention specifically on recall-intensive tasks.
Let \(M^{(i)}\) denote a model that is identical to the original Transformer except that the \(i\)-th attention layer has been replaced with the corresponding RNN layer trained in Stage 1. The paper evaluates \(M^{(i)}\) on two dimensions:
\(\mathcal{R}(M^{(i)})\): recall performance, specifically the average score on SQuAD (reading comprehension requiring information extraction), FDA (structured data extraction), and SWDE (semi-structured web data extraction). These tasks require the model to locate and reproduce specific facts from the input context.\(\mathcal{C}(M^{(i)})\): commonsense reasoning (CSR) performance, specifically the normalized accuracy on HellaSwag, ARC-Easy, and ARC-Challenge. These tasks require general world knowledge and reasoning but not precise recall of input tokens.
The importance score for layer \(i\) is then:
where \(\epsilon = 10^{-6}\) prevents division by zero.
What this computes: for each layer, the numerator is the gap between the best-performing single-layer replacement (in terms of recall) and the recall performance when that specific layer is replaced. A large numerator means replacing this layer causes a large recall drop — it is important for recall. The denominator is the gap between the best CSR performance and the CSR performance when this layer is replaced. A large denominator means replacing this layer causes a large CSR drop. The ratio \(s_i\) is high when a layer is disproportionately important for recall relative to CSR.
Why this form (ratio rather than a single metric): recall and CSR are not equally important for layer selection. RNN layers generally underperform attention on recall but can approach or match attention on CSR tasks with sufficient training. Therefore, layers that are important mainly for CSR (high denominator, small numerator) can potentially be replaced with RNNs without catastrophic loss — the RNN can learn the CSR-relevant computation through distillation. Layers that are important for recall specifically (high numerator, small denominator) should be kept as attention, because the RNN's architectural limitation makes it fundamentally worse at recall. The ratio captures this asymmetry: it selects for recall-critical, CSR-uncritical layers.
After computing \(s_i\) for all layers \(i \in \{1, \dots, L\}\), the final set of retained attention layers is simply the top-\(k\) scoring layers:
with \(k = \lfloor L/4 \rfloor\).
Computational cost of layer selection: the paper notes that the evaluation data for layer selection contains 8.36M tokens total, and the process requires only inference (no backward passes), making it relatively cheap compared to the training stages (Table 8 reports \(6.5 \times 10^{17}\) FLOPs for layer selection on Qwen3-1.7B).
The actual layer indices selected for each model are reported in Table 7. For Qwen3-1.7B (28 layers total), HALO selects layers \(\{3, 21, 2, 9, 25, 6, 8\}\) as the 7 attention layers (indices 0-based). The ordering by importance, from most important to least: 3, 21, 2, 9, 25, 6, 8, 19, 16, 24, ..., 5. The first layer (index 0) and last layer (index 27) are notably among the least important for recall — the critical attention layers are in the middle of the network.
Comparison with prior methods: Jet-Nemotron selects based on performance drop on specific tasks without the recall/CSR ratio, and KL-LS uses KL divergence from the teacher as the importance metric, requiring a full distillation run for every layer — far more expensive. The paper's ablation in Table 4 shows that HALO's selection method achieves better NIAH performance at long contexts (74.3% at 256K) compared to Jet-Nemotron (56.2%) and KL-LS (44.3%), while maintaining comparable CSR (55.9 for HALO vs. 55.0 for Jet-Nemotron vs. 55.3 for KL-LS, compared to 58.5 for the Qwen3-1.7B teacher).
Phase 3: Stage 2 — Knowledge Distillation
With \(\mathcal{I}_{\text{attn}}\) determined, the final hybrid model \(f_{\text{hybrid}}\) is constructed: layers \(l \in \mathcal{I}_{\text{attn}}\) use softmax attention, all other layers use the RNN mixer trained in Stage 1. Stage 2 trains this full hybrid model end-to-end to match the output distribution of the original Transformer teacher \(f_{\text{orig}}\).
The objective is standard knowledge distillation with KL divergence:
where \(f_{\text{orig}}(\mathbf{X})\) and \(f_{\text{hybrid}}(\mathbf{X})\) are the probability distributions over the vocabulary produced by the teacher and student models, respectively, given the same input sequence \(\mathbf{X}\). The teacher model weights are completely frozen; only the student (hybrid model) is updated.
What this computes: the KL divergence \(D_{\text{KL}}(P \| Q) = \sum_v P(v) \log \frac{P(v)}{Q(v)}\) measures how much information is lost when using the student's distribution \(Q\) to approximate the teacher's distribution \(P\), summed over all vocabulary items \(v\). Minimizing this encourages the student to produce the same token-level predictions as the teacher at every position.
Why this form (KL divergence rather than cross-entropy with hard labels): standard language model training uses cross-entropy with one-hot ground-truth tokens, which only provides a training signal for the correct next token. Knowledge distillation with KL divergence provides a rich, dense training signal: for every position, the teacher provides a full probability distribution over the vocabulary, indicating not just the correct token but also which incorrect tokens are plausible. This is especially valuable when converting architectures, because the student needs to learn not just the final answer but the teacher's entire reasoning process as reflected in its token-level uncertainties.
Training configuration: Stage 2 uses 1B training tokens from FineWeb-edu, with a cosine learning rate schedule that decays from \(\eta_{\text{stage2}}\) to \(1 \times 10^{-5}\). The value of \(\eta_{\text{stage2}}\) is model-dependent: \(1 \times 10^{-4}\) for the 2B model, \(5 \times 10^{-5}\) for the 5B model, and \(3 \times 10^{-5}\) for the 9B model. The paper reports (Table 6) that these values were determined by a separate hyperparameter search for each model size. Batch size is 96, context length is 512 (the same as Stage 1), and training runs for 20,000 steps. The optimizer is AdamW with \(\beta = (0.9, 0.95)\), no weight decay, and BFloat16 precision.
Why a cosine schedule with a minimum learning rate of \(1 \times 10^{-5}\): the ablation in Table 9 shows that the learning rate in Stage 2 is critically sensitive. The default constant learning rate used in RADLADS (which the authors characterize as "highly suboptimal") gives poor results. A maximum learning rate of \(1 \times 10^{-4}\) with cosine decay is selected as the sweet spot — higher rates (e.g., \(3 \times 10^{-4}\)) cause the model to collapse to near-random performance (CSR drops to 46.0), while lower rates (e.g., \(1 \times 10^{-5}\)) fail to transfer sufficient knowledge (CSR drops to 46.9). The cosine decay to \(1 \times 10^{-5}\) ensures a smooth landing and prevents catastrophic forgetting of what was learned in Stage 1.
Why this stage is necessary despite Stage 1: Stage 1 trained each RNN layer in isolation, but the hybrid model's layers interact. An RNN at layer \(l\) receives inputs from a mix of attention and RNN layers below it, which differ from the pure-attention inputs it was trained on in Stage 1. Stage 2 adapts all layers jointly to this compounding distribution shift. It is the difference between practicing each instrument section in isolation versus rehearsing the full orchestra together — the latter is necessary for the ensemble to sound coherent.
After Stage 2, the model has matched the teacher's short-context behavior (CSR scores are close, as shown in Table 3 and Table 4), but its long-context recall may still be suboptimal because all training so far has been at context length 512.
Phase 4: Stage 3 — Long-Context Finetuning
Stage 3 addresses the length generalization gap by further training the hybrid model on longer sequences. The objective is the same KL divergence as Stage 2, but with:
- Context length increased from 512 to 16,384 (16K) tokens. This directly trains the model to handle sequences 32× longer than those seen in Stages 1 and 2, pushing the model to learn effective long-range dependencies in its attention layers and stable recurrent state management in its RNN layers.
- Learning rate fixed at a constant
\(1 \times 10^{-5}\)(no decay). This small, steady learning rate allows the model to adapt to the longer-context distribution without disrupting the short-context capabilities learned in Stage 2. - 1B training tokens, the same volume as Stage 2, but with batch size 128 (larger than Stage 2's 96, likely because longer sequences increase GPU memory pressure and reduce the number of sequences per batch) and only 500 training steps (because each step processes more tokens due to the longer context).
- The teacher model is now evaluated with YaRN scaling enabled, since its default maximum context is 40,960 tokens and evaluating it at 16K may require the context extension technique recommended by the Qwen3 authors.
Why constant learning rate rather than decay: the goal of Stage 3 is not to converge to a new minimum but to shift the model's operating point to handle longer contexts without otherwise changing its behavior. A decaying learning rate would cause the model to take increasingly smaller steps, slowing adaptation. A constant small learning rate allows steady, continuous adaptation throughout the 500 steps.
Why 16K context specifically: the paper does not provide a detailed ablation of Stage 3 context lengths, but the choice of 16K — while the final model is evaluated at up to 256K — reflects an important principle: with HyPE's length generalization properties, training at 16K is sufficient to enable generalization to much longer contexts. The attention layers, which use NoPE, generalize well beyond their training length, and the RNN layers, which use RoPE, are primarily handling short-range dependencies within their receptive field anyway. The 16K training length provides enough long-range signal for the model to learn effective attention patterns without requiring the computational expense of training at 128K or 256K (which would dramatically increase memory requirements and reduce batch size).
Computational cost breakdown: Table 8 reports the FLOPs and GPU-hours for each stage when converting Qwen3-1.7B:
- Stage 1: 320M tokens,
\(4.15 \times 10^{9}\)FLOPs/token,\(2.7 \times 10^{18}\)total FLOPs, 10.0 A800 GPU-hours. - Layer selection: 8.36M tokens (inference only),
\(1.38 \times 10^{9}\)FLOPs/token,\(6.5 \times 10^{17}\)total FLOPs. - Stage 2: 1B tokens,
\(4.15 \times 10^{9}\)FLOPs/token,\(8.3 \times 10^{18}\)total FLOPs, 43.4 GPU-hours. - Stage 3: 1B tokens,
\(6.88 \times 10^{9}\)FLOPs/token (higher due to longer context),\(1.4 \times 10^{19}\)total FLOPs, 37.7 GPU-hours. - Total: 2.3B tokens,
\(2.5 \times 10^{19}\)FLOPs, 91.1 A800 GPU-hours.
This is a modest computational budget — 91 GPU-hours on a single A800 is less than 4 days on a single GPU, or about 12 hours on an 8-GPU node. The contrast with prior methods (20B–400B tokens, roughly 10–200× more) makes the point concrete: HALO makes hybrid model distillation feasible for academic labs.
HyPE: Hybrid Positional Encoding — The Core Architectural Innovation
HyPE is the position encoding scheme that enables HypeNet's superior length generalization. Its design is counterintuitive when viewed through the lens of conventional hybrid architectures, but follows logically from the paper's analysis of how attention and RNN layers handle positional information differently.
The Conventional Approach and Its Problem
In existing hybrid models (Qwen3-Next, MiniMax-01):
- Attention layers use RoPE, which applies position-dependent rotations to query and key vectors before computing attention scores. This gives each token an explicit sinusoidal positional signature.
- RNN layers use NoPE (no position encoding), because RNNs are already position-aware through their sequential state update.
The paper identifies a subtle problem with this arrangement: RNNs have a limited "receptive field" due to their fixed-size state \(\mathbf{S}_t \in \mathbb{R}^{d_h \times d_h}\). As the context grows, older information in the state gets progressively compressed and overwritten by newer information. There is an effective horizon beyond which the RNN cannot reliably distinguish or retrieve information. This means that in a hybrid model, RNN layers primarily handle short-distance dependencies (within their receptive field) while attention layers handle long-distance dependencies (anything beyond the RNN's effective memory).
This division of labor has implications for length generalization. When the context length exceeds the RNN's receptive field, the RNN layers' behavior becomes essentially invariant to context length — they process the local neighborhood identically regardless of whether the full sequence is 10K or 100K tokens. Therefore, the model's ability to generalize to longer contexts depends almost entirely on the attentin layers, since those are the only components whose behavior changes with sequence length.
If the attention layers use RoPE (the conventional choice), they are trained with sinusoidal position encodings up to the training context length and may not generalize well beyond it — standard RoPE models require specialized scaling techniques like YaRN for context extension. If instead the attention layers use NoPE, they can generalize to arbitrary lengths (as shown by Kazemnejad et al. and others), but the model loses the rich positional signal that RoPE provides at training-length contexts.
The HyPE Solution
HyPE inverts the conventional assignment:
- Attention layers use NoPE (no position encoding). These layers, which handle long-distance dependencies, benefit from NoPE's superior length generalization. Since they see the full context through exact attention, they can learn implicit position representations that generalize to unseen lengths.
- RNN layers use RoPE (rotary position encoding applied to
\(\mathbf{q}_t\)and\(\mathbf{k}_t\)). These layers, which handle short-distance dependencies, benefit from RoPE's explicit positional signal at training-length contexts without suffering length generalization problems, because their effective behavior is dominated by recent tokens regardless of total context length. The RoPE rotations give the RNN layers a rich local positional representation that aids in modeling grammatical structure, local coherence, and other short-range patterns.
In the complete formulation (Appendix A, Equation 18), the RNN layer's query and key computations become:
where \(\text{RoPE}_t\) is the rotary position embedding matrix for position \(t\), applying a rotation to the query and key vectors before they interact via the recurrent state.
What this computes: the query and key vectors are first projected from the input, then normalized (QK-norm, discussed later), then rotated by position-dependent angles. The rotation encodes the absolute position \(t\) into the vector's phase while preserving its magnitude. The key is also scaled by \(1/\sqrt{d_h}\), analogous to the scaling in softmax attention, to control the variance of the outer product \(\tilde{\mathbf{k}}_t^\top \mathbf{v}_t\) added to the recurrent state. The state update then proceeds as \(\mathbf{S}_t = \mathbf{S}_{t-1} \gamma + \tilde{\mathbf{k}}_t^\top \mathbf{v}_t\), and the output is \(\mathbf{o}_t = \tilde{\mathbf{q}}_t \mathbf{S}_t\).
Why this form (RoPE in RNN, not in attention): the key insight is that length generalization depends on the attention layers, not the RNN layers. By moving RoPE to the RNN layers, the model gets the benefits of explicit positional encoding (better training-length performance) without sacrificing length generalization (because the RNN layers don't need to generalize — they operate locally). The attention layers, freed from RoPE's length constraints, can generalize to arbitrary context lengths using only learned implicit position representations. This is the paper's central architectural innovation.
Attention Logits Scaling
NoPE attention layers can suffer from degraded performance at very long contexts because the entropy of the attention distribution increases with sequence length — as the model sees more tokens, its attention becomes more diffuse unless explicitly controlled. To address this, HyPE incorporates a dynamic attention logits scaling mechanism adopted from SWAN-GPT (Puvvada et al., 2025):
where \(a\) is a hyperparameter chosen after training by minimizing loss on a set of pre-training documents.
What this computes: the standard attention logits \(\mathbf{q}_t \mathbf{K} / \sqrt{d_h}\) are multiplied by a position-dependent scaling factor \(s_t\) before the softmax. The scaling factor \(s_t = \log_a(t + a)\) grows logarithmically with the query position \(t\). For example, with \(a = 500\) (the value used for HypeNet-2B), at position \(t = 0\), \(s_0 = \log_{500}(500) = 1\). At position \(t = 128\text{K}\), \(s_{128\text{K}} = \log_{500}(128000 + 500) \approx \log_{500}(128500) \approx 1.87\). The scaling factor gradually increases, sharpening the attention distribution at later positions.
Why this form (logarithmic scaling): consider what happens without scaling (i.e., \(s_t = 1\) for all \(t\)). At long contexts, the softmax denominator \(\sum_{j=1}^t \exp(\mathbf{q}_t \mathbf{k}_j^\top / \sqrt{d_h})\) sums over exponentially many terms, driving the attention weights toward a uniform distribution. This is sometimes called "attention dilution" — the model cannot focus sharply on relevant tokens because the softmax normalizes over the entire history. The logarithmic scaling counteracts this by increasing the effective temperature of the softmax as \(t\) grows, making the attention distribution more peaked. The specific choice of \(\log_a(t + a)\) ensures that \(s_0 = 1\) at the beginning (preserving the standard attention scale) and grows slowly enough not to cause numerical issues.
Why \(a\) is chosen post-training: the optimal rate of scaling growth depends on the specific model and data distribution. Rather than trying to learn it during training (which would add complexity), the paper selects \(a\) by grid search after all training is complete, minimizing perplexity on a held-out validation set of pre-training documents. The values are model-size-dependent: 500 for HypeNet-2B, 600 for HypeNet-5B, and 900 for HypeNet-9B (Table 6). The trend (larger models using larger \(a\), meaning slower growth of \(s_t\)) suggests that larger models have inherently better length generalization and require less aggressive attention sharpening.
Effectiveness validation (Appendix F.1, Figure 8): without logits scaling (\(s_t = 1\)), HyPE exhibits limited length generalization — NIAH accuracy at 256K context is near 0%. With constant scaling (\(s_t = 1.5\) independently of position), length generalization improves but still degrades substantially. With the position-dependent logarithmic scaling \(s_t = \log_a(t + a)\), NIAH accuracy remains high (around 75–80%) at 256K. This validates that the position-dependence is essential — the model needs different amounts of sharpening at different positions, with later positions requiring more to counteract the larger denominator.
Implementation note: the scaling factor \(s_t\) is applied before the softmax operator and does not affect the attention mechanism's computational complexity. It is essentially a scalar multiplication that adds negligible overhead to the attention computation.
Complementary Architectural Modifications
In addition to HyPE, HypeNet incorporates three modifications that improve performance with minimal added cost. These are validated by ablation in Table 3, which shows that removing any of them degrades either CSR or long-context NIAH performance.
QK-Normalization (➋)
QK-normalization, originally proposed by Henry et al. (2020), applies a normalization operation to the query and key vectors after projection:
where \(\text{Norm}(\cdot)\) is typically RMSNorm (the same normalization used elsewhere in the model). This has been adopted by recent open-source Transformer LLMs like Qwen3 and Gemma3, but is not commonly used in RNN layers.
What this computes: instead of using the raw linear projections \(\mathbf{x}_t \mathbf{W}_q\) and \(\mathbf{x}_t \mathbf{W}_k\) directly, the vectors are first normalized to have unit root-mean-square norm. The normalization is applied per-head, independently for each query and key vector.
Why it helps specifically in RNN layers: in softmax attention, the dot product \(\mathbf{q}_t \mathbf{k}_i^\top\) is divided by \(\sqrt{d_h}\) and passed through softmax, which is scale-invariant (multiplying all logits by a constant does not change the softmax output, only the temperature). This means that large norms in Q or K projections do not cause instability. In RNN layers, however, the query and key interact through the recurrent state \(\mathbf{S}_t\), where the key's magnitude directly affects the magnitude of the outer product \(\mathbf{k}_t^\top \mathbf{v}_t\) added to the state. Uncontrolled key magnitudes can cause the state to grow or shrink over time, degrading the RNN's ability to maintain stable long-term memories. QK-norm prevents this by ensuring all queries and keys have consistent norm before any further computation.
Empirical evidence: Table 3 shows that removing QK-norm from RNN layers causes NIAH accuracy at 128K to drop from 79.9% to 17.3% — a catastrophic 62.6-percentage-point decline. This is the largest single ablation effect in the table and highlights that stable key norms are critical for the RNN's long-context behavior.
Why not add QK-norm to attention layers as well: the base Qwen3 model already has QK-norm in its attention layers. The modification ➋ specifically adds QK-norm to RNN layers where it was absent. When converting from a model without QK-norm (e.g., LLaMA), HALO would add QK-norm to the RNN layers while leaving the attention layers as they were in the teacher.
GQA to MHA: Decoupling KV Heads (➌)
Most modern Transformer models, including Qwen3, use Grouped-Query Attention (GQA) to reduce the KV cache size. In GQA, the model has \(n_h\) query heads but only \(n_h / g\) key and value heads, where \(g\) is the group size. For Qwen3-1.7B, \(n_h = 16\) attention heads and 8 KV heads, meaning \(g = 2\) — every two query heads share the same KV projections. This reduces the KV cache by a factor of \(g\), saving significant memory during inference.
However, in an RNN layer, there is no KV cache. The RNN's state \(\mathbf{S}_t\) is per-head and is updated and queried at every time step, but it is not stored per-position. The memory savings from GQA are therefore irrelevant for RNN layers. Moreover, sharing KV projections across heads reduces the RNN layer's expressivity — each head processes a distinct representation of the input, and constraining multiple heads to use the same keys and values limits their ability to capture different aspects of the sequence.
The solution is straightforward: when initializing an RNN layer from an attention layer with GQA, clone the KV projection weights so that every head has its own independent \(\mathbf{W}_k\) and \(\mathbf{W}_v\):
where \(\mathbf{W}_{\square}^{(i)}\) is the KV projection weight for the \(i\)-th head, \(g\) is the group size, and \(\lfloor i/g \rfloor\) identifies the group that head \(i\) belongs to.
What this computes: for each head \(i\), its key projection matrix is initialized to a copy of the key projection matrix that was shared by head \(i\)'s group in the attention layer. Same for the value projection. This effectively converts GQA to standard MHA (Multi-Head Attention) in the RNN layers, increasing the number of distinct KV projections from \(n_h / g\) to \(n_h\).
Why this increases model size (the 10% parameter increase): cloning means the total number of KV parameters increases from \(2 \times (n_h / g) \times d \times d_h\) to \(2 \times n_h \times d \times d_h\). For Qwen3-1.7B, this adds roughly \(2 \times 8 \times 2048 \times 128 \approx 4.2\) million parameters per RNN layer. With 21 RNN layers, the total increase is approximately 88M parameters on a base of about 1.7B — roughly 5%. Combined with the output gate parameters (discussed next), the total increase reaches approximately 10%.
Empirical evidence: Table 3 shows that removing GQA-to-MHA decoupling (keeping the RNN layers with GQA) reduces NIAH accuracy at 128K from 79.9% to 83.5% — a smaller effect than the other ablations. The impact at 128K is actually minimal; the more noticeable effect is at shorter contexts (4K: 89.7% vs. 95.9%, a 6.2-point drop). This suggests that the increased expressivity from independent KV heads helps the RNN layers model fine-grained patterns that matter more for short-context tasks, while long-context NIAH is dominated by other factors (HyPE, QK-norm, the RNN mixer choice).
A subtle design choice: the attention layers retain GQA after conversion. Only the RNN layers are converted to MHA. This means the attention layers still benefit from the reduced KV cache, keeping the hybrid model's overall KV cache size smaller than it would be with full MHA attention layers.
Output Gate (➍)
Many modern recurrent architectures incorporate an output gate: a data-dependent gating mechanism that modulates the mixer's output before the output projection. The paper adopts this for both RNN and attention layers in HypeNet.
For RNN layers, the output gate is added as:
where \(\text{Mixer}(\mathbf{x}_t)\) is the RNN operation producing the raw output \(\mathbf{o}_t \in \mathbb{R}^{1 \times d_h}\), \(\mathbf{W}_z \in \mathbb{R}^{d \times d_h}\) is a new learnable parameter (randomly initialized during HALO initialization), \(\sigma\) is the sigmoid activation function, \(\text{Norm}\) is RMSNorm, and \(\odot\) denotes element-wise multiplication.
What this computes: the raw mixer output \(\mathbf{o}_t\) is first normalized to unit norm, then multiplied element-wise by a learned gate vector \(\mathbf{z}_t \in [0, 1]^{d_h}\). The gate selectively attenuates or amplifies each dimension of the output based on the current input \(\mathbf{x}_t\). The gated result is then projected to dimension \(d\) via \(\mathbf{W}_o^\top\).
Why this form (gating after the mixer, before output projection): the output gate provides the model with a learned mechanism to control how much of the mixer's computation to pass through to the output. For an RNN layer, this means the model can learn to suppress dimensions of the state-derived output \(\mathbf{o}_t\) when the retrieved information is irrelevant or noisy. The sigmoid activation ensures the gate values are in \((0, 1)\), so each dimension is attenuated independently. The normalization before gating prevents the gate from interacting with the output's scale in unpredictable ways.
Why this helps specifically in the distillation setting: when an RNN is replacing an attention layer, the RNN's output distribution may differ from the attention layer's in subtle ways — some dimensions may be overactive, others underactive. The output gate provides a learned adjustment mechanism that can compensate for these differences, making the RNN's output more closely match the teacher's distribution at each dimension. The fact that \(\mathbf{W}_z\) is randomly initialized and learned during Stage 1 means it is optimized specifically for the teacher-matching objective.
The same mechanism for attention layers: the paper also adds output gating to the attention layers, but at the start of Stage 2 rather than Stage 1 (since attention layers are not trained in Stage 1). For attention layers, the output gate takes the form:
The rationale is different from RNNs: Qiu et al. (2025) have shown that adding output gating to softmax attention improves model quality and length generalization. The gate provides a non-linear modulation that helps attention layers focus their output and avoid "attention sink" phenomena where certain tokens absorb disproportionate attention.
Empirical evidence: Table 3 shows that removing the output gate from RNN layers reduces NIAH accuracy at 128K from 79.9% to 74.5% (a 5.4-point drop) and slightly reduces CSR from 55.9 to 55.6. Removing the output gate from attention layers reduces CSR from 55.9 to 55.4 and NIAH at 128K from 79.9% to 80.9% — the long-context effect is minimal, consistent with the output gate's primary role being quality improvement rather than length generalization for attention layers.
Implementation note on parameter increase: the output gate adds \(d \times d_h\) parameters per layer (the \(\mathbf{W}_z\) matrix). For Qwen3-1.7B with \(d = 2048\) and \(d_h = 128\), this is approximately 262K parameters per layer. Across 28 layers, this adds roughly 7.3M parameters — a fraction of the GQA-to-MHA increase and well within the "roughly 10% larger" that the paper claims for HypeNet overall.
RNN Mixer: Lightning Attention and the Choice of Forget Gate
HypeNet's architectural design is agnostic to the specific RNN mixer — as long as the mixer follows the outer-product formulation in Equations 4–6, it can be plugged into the framework. However, the paper identifies a critical property that determines long-context performance: whether the forget gate is data-independent or data-dependent.
Lightning Attention
The recommended RNN mixer for HypeNet is Lightning Attention (Qin et al., 2024a), which has the simplest possible forget gate: a fixed, data-independent scalar \(\gamma\) per head:
where \(\gamma \in (0, 1)\) is a constant decay factor. For HypeNet, the decay factors are fixed by a geometric formula:
where \(h \in \{1, \dots, H\}\) is the head index and \(H\) is the number of heads (e.g., \(H = 32\) for HypeNet-5B and HypeNet-9B).
What this computes: the decay factor \(\gamma_h\) is a number between 0 and 1 that controls how much of the previous state \(\mathbf{S}_{t-1}\) is retained at each time step. A value close to 1 means the state changes very slowly and retains information for a long time; a value closer to 0 means the state resets quickly and only captures very recent information. The geometric formula \(\exp(-2^{-8h/H})\) produces a range of decay rates across heads: for \(H = 32\), the values are \(\gamma_1 \approx 0.431\), \(\gamma_8 \approx 0.779\), \(\gamma_{16} \approx 0.939\), \(\gamma_{32} \approx 0.996\). Early heads have fast decay (short memory), late heads have slow decay (long memory). This provides a spectrum of temporal scales, allowing the model to capture dependencies at multiple ranges.
Why data-independent: because \(\gamma_h\) does not depend on the input \(\mathbf{x}_t\), the forget behavior is consistent across all inputs and all positions. A head with \(\gamma_h = 0.996\) retains \(0.996^T\) of the initial state after \(T\) steps regardless of what tokens it processes. This stability is what enables good length generalization — the RNN's effective memory span is a known, fixed quantity, not something that can shrink unexpectedly for certain inputs.
Why not rescale with a layer-specific factor: the original Lightning Attention implementation includes a layer-specific scaling factor that multiplies \(\gamma_h\). The paper reports that "our preliminary results show that it does not yield performance gains in a hybrid model," and removes it from HypeNet. This simplification reduces hyperparameter count without affecting performance.
The Data-Dependent Problem
Other modern RNN mixers — Mamba2, GLA, GDN, RWKV-7 — use data-dependent forget gates, where \(\mathbf{F}_t\) is a function of the input \(\mathbf{x}_t\). For example:
- Mamba2:
\(\mathbf{F}_t = \alpha_t \mathbf{I}\)where\(\alpha_t\)is computed from\(\mathbf{x}_t\). - GLA:
\(\mathbf{F}_t = \text{diag}(\alpha_t)\)where\(\alpha_t\)is a per-dimension forget vector from\(\mathbf{x}_t\). - GDN:
\(\mathbf{F}_t = \alpha_t(\mathbf{I} - \beta_t \mathbf{k}_t^\top \mathbf{k}_t)\), combining a scalar decay with a rank-1 correction.
The intuition behind data-dependent forgetting is appealing: the model should decide what to forget based on what it is currently reading. When encountering a section break or topic shift, it should forget old information; when continuing the same thought, it should retain.
However, Chen et al. (2025b) showed that data-dependent forget gates can cause a failure mode they call the "inability to forget" — for certain inputs, the forget gate parameters can lead to the state retaining too much information, effectively making the state "oversized" relative to its capacity. Over long sequences, this causes the state to become saturated with old information that cannot be cleared, degrading the RNN's ability to incorporate new information. The result is poor length generalization: the model performs well at training length but degrades rapidly when the context is extended.
Empirical verification (Figure 5): the paper trained 500M-parameter HypeNet variants from scratch with different RNN mixers, using 20B tokens at 4K context length, and evaluated NIAH accuracy at context lengths from 4K to 256K (64× the training length). The results are striking:
- Lightning Attention: NIAH accuracy remains above 90% at 256K context, showing almost no degradation from the training length to 64× training length. This is the best length generalization of any tested mixer.
- GDN, GLA, RWKV-7, Mamba2: all show substantial degradation. Accuracy at 256K drops to below 40% for most, with some (like GDN) falling below 20%.
The paper suggests one possible explanation: "Lightning Attention employs data-independent forget gates. In contrast, the other RNN mixers have data-dependent forget gates, which may result in poor length generalization." While this single factor explanation is a simplification (there are other architectural differences between these mixers), the empirical pattern is consistent and convincing.
Implications for the full HypeNet: The main results (Table 2) use Lightning Attention as the RNN mixer for the converted Qwen3 models, achieving 99.8%, 99.6%, 86.2% NIAH accuracy at 256K (for the three NIAH variants). The from-scratch results in Figure 5 suggest that choosing a data-dependent RNN mixer would substantially degrade these numbers, possibly matching the poor long-context performance of Jet-Nemotron or KL-LS.
Synthesis: Why the Full Design Works Together
The HypeNet architecture is more than the sum of its parts. Each component addresses a specific failure mode that would arise without it:
-
Without HyPE (attention RoPE + RNN NoPE, the conventional choice): the attention layers would have RoPE, limiting their length generalization. As shown in the HyPE ablation (Table 3, "w/ attention RoPE"), NIAH accuracy at 128K drops from 79.9% to 19.7% when RoPE is restored to attention layers. The model essentially loses its ability to handle contexts longer than training because the attention layers — which are the components responsible for long-range retrieval — cannot generalize.
-
Without HyPE (all NoPE): the model would lose the rich positional signal in the RNN layers. Table 3 ("w/o RNN RoPE") shows CSR drops from 55.9 to 53.8 and NIAH at 128K drops from 79.9% to 47.9%. The RNN layers perform worse at all context lengths without explicit position encodings.
-
Without QK-norm in RNN layers: the key and query magnitudes would be unconstrained, causing instability in the recurrent state. Table 3 shows NIAH at 128K dropping to 17.3% — a catastrophic failure. This is the most important architectural modification after HyPE itself.
-
Without GQA-to-MHA decoupling: the RNN layers would be less expressive, with multiple heads forced to share the same KV projections. Table 3 shows a modest but consistent drop, particularly at shorter contexts.
-
Without output gating: the model loses the data-dependent modulation of mixer outputs. Table 3 shows drops of 1–5 points across both CSR and NIAH.
-
With a data-dependent RNN mixer instead of Lightning Attention: the RNN's length generalization would degrade (as shown in Figure 5), likely causing the hybrid model to exhibit the same long-context collapse seen in Jet-Nemotron and KL-LS.
-
Without the three-stage distillation pipeline (HALO): the model would need to be trained from scratch at massive scale, placing it out of reach for academic research.
The combination of HyPE + Lightning Attention + the complementary modifications + HALO's efficient distillation creates a system where each component reinforces the others. HyPE ensures the attention layers can generalize to long contexts; Lightning Attention ensures the RNN layers don't interfere with this generalization; QK-norm stabilizes the RNN computations; and HALO makes the whole pipeline practical at 2.3B tokens of training data, roughly 2–3 days on a single 8-GPU node.
4. Key Insights and Innovations
Innovation 1: Length Generalization as a Test-Time Property Determined by Which Layers Carry Positional Information
The dominant assumption in the hybrid model literature before this paper was that position encoding follows a natural division of labor: attention layers, which process global context, should use explicit position encodings (RoPE) to know where things are, while RNN layers, which are inherently sequential, don't need them and use NoPE. This is the scheme adopted by Qwen3-Next, MiniMax-01, and essentially every existing hybrid architecture trained from scratch or distilled. It is intuitive, it mirrors the historical design of Transformers and RNNs independently, and until this paper, there was no empirical reason to question it.
HypeNet's HyPE scheme — RoPE in RNN layers, NoPE in attention layers — inverts this consensus. The paper does not present this as an arbitrary alternative but as a deductive consequence of a diagnostic insight about where length generalization lives in a hybrid model. The reasoning chain is: (1) RNNs have a limited effective receptive field due to their fixed-size state, meaning they primarily model short-distance dependencies; (2) when context length exceeds this receptive field, RNN behavior becomes invariant to total sequence length; (3) therefore, the model's ability to generalize to unseen lengths depends almost entirely on the attention layers; (4) attention layers with NoPE are known to generalize far better to unseen lengths than attention layers with RoPE; (5) ergo, give NoPE to attention layers and give RoPE to RNN layers, where it provides useful local positional signal without creating a length generalization liability.
This is a reframing of the position encoding problem, not just a new PE scheme. Before this paper, length generalization was treated as a property of the model as a whole — you applied YaRN or some other context extension technique uniformly, and the model either generalized or it didn't. HyPE isolates length generalization as a property of specific architectural components in a specific functional role. It says: in a hybrid model, long-context generalization is determined by the attention layers' PE, not by any global property. This is a diagnostic move — it tells you where to look when length generalization fails, and it tells you what to change (the attention layers' PE) without touching other components.
The evidence for this framing is not just the headline NIAH numbers in Table 2 (99.8% at 256K for HypeNet vs. 0.0% for Jet-Nemotron), but the ablation in Table 3 that cleanly isolates the mechanism. Adding RoPE back to attention layers ("w/ attention RoPE") drops NIAH-128K from 79.9% to 19.7% — a 60-point collapse — while removing RoPE from RNN layers ("w/o RNN RoPE") drops it to 47.9%. The first ablation shows that attention-layer RoPE is actively harmful to length generalization; the second shows that RNN-layer RoPE is actively helpful. Both directions matter, and together they demonstrate that the conventional assignment was exactly backwards for length generalization.
This is a fundamental conceptual contribution, not an incremental refinement, because it changes what question you ask when designing a hybrid architecture's PE. The old question was: "What PE should each layer type use, based on its inherent properties?" The new question is: "Which layers determine length generalization, and what PE maximizes their length generalization?" The answer leads to the opposite design. This is the kind of insight that, once stated, feels obvious — but the fact that no prior hybrid model adopted it (despite the NoPE length generalization literature being well-known since Kazemnejad et al., 2023) suggests it was not.
The attention logits scaling mechanism (s_t = log_a(t + a)) is a necessary complement to this insight, not a separate innovation. NoPE attention layers suffer from attention dilution at extreme lengths without it (Figure 8 shows NIAH collapsing without scaling). The scaling itself is adopted from SWAN-GPT, so the novelty is not in the formula but in recognizing that it is required for NoPE attention to fulfill its length generalization role in a hybrid model. The combination — NoPE attention + logarithmic scaling — is what makes the inverted PE assignment work in practice.
Innovation 2: Distillation Is a Layer-Level Problem, and Data Efficiency Comes from Isolating It
Prior Transformer-to-hybrid distillation methods (Mamba-in-the-Llama, Jet-Nemotron, KL-LS, RAD, Zebra-Llama) all operate on a similar assumption: converting a Transformer into a hybrid requires end-to-end training of significant scale — 7B to 400B tokens — because the task is to teach the full model a new computation pattern. This assumption is not unreasonable: replacing multiple attention layers with RNN layers changes the model's internal computation fundamentally, and it seems natural that recovering performance would require substantial re-training.
HALO's three-stage design (MSE hidden-state alignment → end-to-end KL distillation → long-context finetuning) achieves competitive results with 2.3B tokens — a 10–200× reduction from prior methods. The intellectual move here is not the individual stages (hidden-state alignment via MSE is from RADLADS, KL distillation is standard), but the decomposition of the distillation problem into independent sub-problems that can be solved with dramatically different data budgets.
The key insight is that matching a teacher layer's output distribution at the hidden-state level is a much simpler optimization problem than matching the teacher's token-level predictions end-to-end. When an RNN is trained to minimize MSE against a specific attention layer's output in Stage 1, it is solving a single-layer regression problem with a fixed input distribution (the frozen teacher's hidden states). This problem requires relatively little data — the paper finds 320M tokens is sufficient, and interestingly, increasing to 1.3B tokens actually harms final performance (Table 9), suggesting that overfitting to the frozen teacher's distribution can hurt the RNN's ability to adapt in later stages. The end-to-end KL distillation in Stage 2 then only needs to correct for the compounding distribution shift that occurs when multiple RNN layers are chained together — a correction that requires far less data than learning the RNN computation from scratch end-to-end.
This decomposition is a fundamental conceptual contribution to the distillation methodology, not an incremental engineering improvement. It says: the difficulty of cross-architecture distillation is not in the architecture gap per se, but in the compounding of per-layer errors across the depth of the model. By isolating and solving the per-layer problem first (Stage 1), the end-to-end problem becomes dramatically easier (Stage 2 needs only 1B tokens instead of 20B+). This is analogous to how pretraining + finetuning decomposes the language modeling problem: learn general representations first on massive data, then adapt to specific tasks on minimal data. HALO's innovation is recognizing that the same decomposition applies within the distillation process itself.
The evidence for this decomposition is not a single ablation but the structural design of the paper's training regimen. Stage 1 trains each RNN layer independently with no interaction between layers — this would be impossible if the layers' errors were inherently entangled. Stage 2 uses only 1B tokens of end-to-end training and achieves CSR scores close to the teacher (55.9 vs. 58.5 for Qwen3-1.7B, Table 4). If the per-layer problem and the end-to-end problem required comparable data budgets, the three-stage design would offer no advantage over training end-to-end from the start. The fact that it works with 2.3B tokens when prior methods need 10–200× more is direct evidence that the decomposition is real and exploitable.
Innovation 3: The Recall-CSR Tradeoff as a Layer Selection Criterion — Not a Unified Metric, but Their Ratio
When deciding which attention layers to retain in a hybrid model, prior work used metrics that treat layer importance as a one-dimensional quantity. Jet-Nemotron uses the performance drop on specific tasks when a layer is replaced — if replacing layer i causes a large accuracy drop, layer i is important. KL-LS uses the KL divergence between the teacher model and a model with layer i replaced — if the distributions diverge significantly, the layer is important. Zebra-Llama uses the shift in the output distribution similarly. Evenly distributing attention layers (every k-th layer) ignores importance entirely.
All of these approaches assume that "importance" means the same thing for all tasks: a layer that matters for one task matters for all tasks, and the magnitude of the performance drop is the sole signal. This assumption is contradicted by a well-known empirical fact that the paper explicitly cites: RNN layers specifically underperform attention layers on recall-intensive tasks (Jelassi et al., 2024; Shen et al., 2025), not on all tasks equally. Attention layers do things that RNNs fundamentally cannot replicate with a fixed-size state — particularly, they can attend to arbitrary past tokens with perfect fidelity, which is essential for recall. But for many other computations (local pattern recognition, grammatical processing, semantic composition), RNNs can be trained to match attention layers well.
The paper's layer selection criterion (Equation 8) implicitly encodes this asymmetry. The importance score s_i is the ratio of the recall performance drop to the CSR performance drop when layer i is replaced:
s_i ∝ (worst-case recall drop − recall drop for layer i) / (worst-case CSR drop − CSR drop for layer i)
This is conceptually novel in a way that is easy to miss because the formula is so simple. It treats recall-critical and CSR-critical layers as different categories that should be selected for differently. A layer whose replacement causes a large CSR drop but a small recall drop gets a low score — the RNN can potentially learn the CSR-relevant computation through distillation, so keeping this layer as attention is not critical. A layer whose replacement causes a large recall drop but a small CSR drop gets a high score — the RNN's architectural limitation means it cannot learn this computation, so the attention layer must be retained.
This is a reframing of layer selection from a one-dimensional importance ranking to a two-dimensional tradeoff space where the axes have different implications for what can be recovered through training. The ratio formulation is a specific operationalization of this reframing, but the conceptual move is recognizing that "important" is not a scalar property when the replacement mechanism (RNN) has known, asymmetric limitations.
The significance goes beyond the specific formula. It implies that layer selection methods that use a single metric (performance drop on one task, KL divergence, output distribution shift) are systematically suboptimal because they cannot distinguish between "important because RNNs can't do this" and "important but RNNs can learn this." The ablation in Table 4 provides evidence: HALO's ratio-based selection achieves 74.3% NIAH at 256K, compared to 56.2% for Jet-Nemotron (task-performance-drop) and 44.3% for KL-LS (KL-divergence-based). The CSR scores are comparable (55.9 vs. 55.0 vs. 55.3), suggesting that the ratio-based method specifically preserves recall-critical layers without sacrificing CSR. This is not an incremental improvement — it is a qualitative difference in what the selection criterion targets.
A subtle point: the evaluation data for the recall metric uses SQuAD, FDA, and SWDE — relatively short-context tasks. Yet the resulting layer selection improves NIAH at 256K. This suggests that the layers critical for short-context recall are also critical for long-context recall, and that selecting based on short-context recall tasks (which are cheap to evaluate) transfers to long-context performance. This is a practical insight: you don't need long-context evaluation data to select layers for long-context performance.
Innovation 4: Data-Independent Forgetting as the Critical Architectural Property for RNN Length Generalization in Hybrid Models
The choice of RNN mixer might seem like a hyperparameter — something to be optimized through benchmarking and swept under the rug. The paper treats it as a scientific question: why does Lightning Attention generalize to 64× training length while more sophisticated RNNs (Mamba2, GLA, GDN, RWKV-7) collapse?
The answer the paper proposes is specific and falsifiable: data-independent forget gates generalize; data-dependent forget gates don't. The mechanism, drawing on Chen et al. (2025b), is that data-dependent forget gates can fail to clear old information for certain input patterns, causing the recurrent state to saturate over long sequences and lose the ability to incorporate new information — the "inability to forget" problem. Data-independent gates like Lightning Attention's fixed γ_h decay at a constant, predictable rate regardless of input, ensuring that the state's effective memory span is a known, stable quantity.
This is a fundamental diagnostic contribution, not merely a benchmark result, because it identifies a causal mechanism rather than a correlation. Prior work comparing RNN mixers (e.g., the benchmarking literature) typically reports which mixer performs best on which tasks without explaining why. The paper's contribution is isolating one architectural property — data-dependence of the forget gate — and providing evidence that it is the dominant factor for length generalization. The evidence is Figure 5: across five different RNN mixers with varying complexity and design philosophies, the single mixer with data-independent forgetting (Lightning Attention) is the only one that generalizes to 64× training length. The other four, despite being more recent and more sophisticated in other respects, all degrade substantially.
This finding has a negative implication that is as important as the positive one. The RNN literature has trended toward increasing sophistication: data-dependent gating, low-rank parameterizations, learned decay rates, delta-rule updates. These innovations improve training-length performance and expressivity. But the paper's results suggest that this sophistication comes at a cost — specifically, a loss of the predictable forgetting behavior that enables length generalization. If this hypothesis is correct, it implies that the RNN mixer design space contains a fundamental tradeoff between expressivity (enabled by data-dependence) and length generalization (enabled by data-independence), and that for hybrid models where the attention layers handle long-range retrieval anyway, expressivity is less critical than stability — tipping the balance toward data-independent designs.
The paper does not fully prove this causal mechanism — it acknowledges that the explanation is "one possible" interpretation and that there are other architectural differences between the mixers. But the empirical pattern is stark enough to elevate this from an observation to a contribution: it tells the field that if you care about length generalization in hybrid models, you should be skeptical of data-dependent RNN mixers until proven otherwise. This is actionable guidance that changes how researchers should think about RNN mixer selection, going beyond "Lightning Attention works well" to "here is the property that likely explains why it works well, and here is what to look for in future RNN designs."
The practical significance is reinforced by the fact that this insight emerged from the distillation setting but was validated independently through training from scratch (Section 6.3). The from-scratch experiments use 500M-parameter models trained on 20B tokens at 4K context length — a completely different training regime from the HALO pipeline. The fact that the same pattern (Lightning Attention generalizes, data-dependent mixers don't) appears in both settings strengthens the claim that this is an architectural property, not an artifact of the distillation process.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. All experiments use FineWeb-edu (Penedo et al., 2024) for training data, specifically randomly sampled from the 10B subset. This is a popular open-source, high-quality Internet-scale pre-training corpus, chosen deliberately to keep the entire pipeline reproducible with public data. For evaluation, the paper uses two distinct suites: commonsense reasoning (CSR) tasks and long-context recall tasks. CSR evaluation covers ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, PIQA, LAMBADA, and MMLU, all evaluated zero-shot with normalized accuracy via LM-Evaluation-Harness v0.4.10.dev0. Long-context recall is measured using NIAH (Needle-in-a-Haystack) from the RULER benchmark, specifically the average of NIAH-Single-1, NIAH-Single-2, and NIAH-Single-3, reported at context lengths from 4K to 256K tokens.
-
Base model(s). The paper applies HALO to the Qwen3 series (Yang et al., 2025a) at three scales: 1.7B, 4B, and 8B parameters. Qwen3 is chosen because it is "one of the most widely-used open-source language model series," its architecture (GQA with QK-norm, RoPE, SwiGLU MLPs) is representative of modern Transformers, and its performance is strong enough that the distilled hybrid model can be meaningfully compared. The teacher model for the FLOPs-matched comparison and efficiency benchmarks is the original Qwen3 at the corresponding size. For from-scratch validation experiments (Section 6.3), the paper trains 500M-parameter models on 20B tokens.
-
Metrics. Two primary metrics are reported. CSR performance is the average normalized accuracy across the seven downstream tasks listed above, expressed as a percentage. Long-context recall is NIAH accuracy at various context lengths (4K, 8K, 16K, 32K, 64K, 128K, 256K), with NIAH-Single-1, -2, and -3 reported separately in Table 2 and averaged in most other tables and figures. For efficiency measurements, the paper reports throughput (tokens per second), time per output token (TPOT), and prefilling time at various context lengths, all measured on a single NVIDIA A800-80GB GPU with PyTorch 2.9.1, CUDA 12.4, Flash-Attention-2 v2.8.3 for softmax attention, and Triton kernels from Flash-Linear-Attention v0.4.1 for RNN mixers. Batch size is 1 for all efficiency measurements.
-
Baselines. The paper compares against multiple categories of baselines. Teacher model: Qwen3 at the same parameter scale, evaluated with greedy decoding and YaRN scaling (as recommended by the Qwen3 authors) for contexts beyond its default 40,960-token maximum. Prior distilled hybrid models: Jet-Nemotron (Gu et al., 2025, 2B parameters, 400B training tokens), KL-LS with GDN mixer (Li et al., 2025, 3B parameters, 25B training tokens), and RADLADS (Goldstein et al., 2025, pure RNN with no attention layers retained). Alternative layer selection methods: Jet-Nemotron's performance-drop-based selection, KL-LS's KL-divergence-based selection, evenly distributing attention layers (every
k-th layer), and evenly distributing attention layers in the latter half of the network. Training from scratch baselines (Section 6.3): a standard Transformer with RoPE, SWAN-GPT (Puvvada et al., 2025), and HypeNet variants with different PE schemes (all RoPE, all NoPE, attention RoPE + RNN NoPE) and different RNN mixers (Lightning Attention, GDN, GLA, RWKV-7, Mamba2). Within-model ablations: HypeNet with individual components removed (no RNN RoPE, with attention RoPE, no RNN QK-norm, no GQA-to-MHA decoupling, no RNN output gate, no attention output gate). -
Generation budget / compute accounting. For the main distillation results, the "budget" is measured in training tokens consumed, not inference-time generations. HALO uses exactly 2.32B tokens total across its three stages (320M in Stage 1, 1B in Stage 2, 1B in Stage 3), plus 8.36M inference-only tokens for layer selection. The total computational cost is reported in FLOPs and GPU-hours: for Qwen3-1.7B, the conversion requires 2.5 × 10¹⁹ FLOPs and 91.1 A800 GPU-hours. For efficiency measurements, the paper uses wall-clock time and GPU memory consumption at specific context lengths, with batch size 1 for fair comparison. For the from-scratch experiments, all models are trained on exactly 20B tokens at 4K context length, making comparisons across architectures directly comparable in terms of training compute.
-
Cross-validation / statistical protocol. The paper does not employ cross-validation for the main distillation results — the layer selection is performed once using the evaluation tasks described in Section 6.1, and the selected layers are used directly for all subsequent training and evaluation. The attention logits scaling hyperparameter
ais chosen post-training "by minimizing loss on a set of pre-training documents" (a held-out validation set from FineWeb-edu), which is standard. For the from-scratch experiments, no cross-validation is reported. The paper does not report confidence intervals, standard deviations, or statistical significance tests for any of its results. The test sets for CSR evaluation have fixed sizes (e.g., ARC-Challenge has 1,172 questions, MMLU has ~14,000 questions), and NIAH is evaluated with a fixed protocol (the number of test examples per context length is not specified in the main text).
Main Quantitative Results
Distillation Performance: CSR and Long-Context Recall
The headline result from the distillation experiments is that HypeNet + HALO achieves CSR scores close to the original Qwen3 teacher while dramatically outperforming prior distilled hybrid models on long-context NIAH, despite using an order of magnitude fewer training tokens.
CSR performance (Figure 1, left; Table 4): HypeNet-2B (converted from Qwen3-1.7B) achieves a CSR score of 55.9, compared to 58.5 for the Qwen3-1.7B teacher — a drop of only 2.6 percentage points. This is competitive with prior distillation methods: Jet-Nemotron-2B achieves 55.0 (0.9 points below HypeNet), KL-LS achieves 55.3 (0.6 points below), and a naive even-distribution baseline achieves 54.0 (1.9 points below). The RADLADS pure-RNN baseline actually achieves a slightly higher CSR (56.0), but at catastrophic cost to long-context recall (see below). The CSR comparison is reported at the default evaluation context lengths for each task, which are typically short (most CSR tasks involve passages under 1K tokens).
Long-context NIAH performance (Table 2, Table 4, Figure 1, center and right): This is where the results are starkest and where HypeNet's architectural innovations pay off.
In Table 2, HypeNet-2B + HALO is compared against the Qwen3 teacher and two prior distilled hybrid models across NIAH-Single-1, -2, and -3 at context lengths from 32K to 256K:
-
NIAH-Single-1 at 256K: HypeNet achieves 99.8%, essentially perfect. The Qwen3 teacher (with YaRN) achieves 17.0%. Jet-Nemotron achieves 0.0%. KL-LS achieves 14.8%. HypeNet is not just better — it maintains near-perfect recall at a context length where the teacher model (with its full set of attention layers) has already collapsed, and where competing distilled models have completely failed.
-
NIAH-Single-2 at 256K: HypeNet achieves 86.2%. Qwen3 teacher: 19.2%. Jet-Nemotron: 0.0%. KL-LS: 10.4%. The gap narrows slightly (HypeNet is not perfect on this harder variant) but the qualitative pattern is the same: HypeNet is the only model that remains functional.
-
NIAH-Single-3 at 256K: HypeNet achieves 48.8%. Qwen3 teacher: 19.0%. Jet-Nemotron: 0.0%. KL-LS: 11.0%. On the hardest NIAH variant (three needles), HypeNet's performance degrades but still substantially exceeds all baselines.
The striking observation is that HypeNet at 256K context outperforms the Qwen3 teacher with YaRN on all three NIAH variants. The teacher model, despite having 4× more attention layers (28 vs. 7) and being the source of HypeNet's weights, collapses on long contexts when YaRN scaling is applied. HypeNet, with only 7 attention layers using NoPE + dynamic logits scaling, generalizes far better. This is direct evidence that HyPE's design — giving NoPE to attention layers and letting them handle length generalization — works as intended, and that YaRN scaling (the recommended approach for extending Qwen3's context window) is substantially less effective for maintaining recall at extreme lengths.
Efficiency results (Figure 1, center and right; Figure 6): At 128K context length, HypeNet-2B achieves higher throughput and lower time per output token than Qwen3-1.7B (exact numbers depend on the specific throughput measurement, but Figure 1 center shows HypeNet-2B positioned to the upper-right of Qwen3-1.7B on the throughput-vs-CSR plot, indicating both higher throughput and similar CSR). At 512K context length, the paper reports up to 3.0× decoding speedup and 3.4× prefilling speedup over Qwen3-1.7B (Section 6.6). At 1M context length, Qwen3-1.7B runs out of GPU memory on a single A800, while HypeNet continues to operate. Figure 6 shows that HypeNet's prefilling time scales roughly linearly with context length (consistent with RNN dominance), while Qwen3's prefilling time scales superlinearly (consistent with quadratic attention). The curves diverge increasingly as context length grows.
Layer Selection Comparison (Table 4)
Table 4 compares HALO's layer selection method against Jet-Nemotron's, KL-LS's, and two naive baselines, all evaluated within the same HALO pipeline (only the layer selection criterion differs) on converting Qwen3-1.7B:
- HALO (ratio-based): CSR 55.9, NIAH at 256K: 74.3%.
- Jet-Nemotron (performance-drop): CSR 55.0, NIAH at 256K: 56.2%. CSR is close to HALO (0.9 points lower), but NIAH is 18.1 points lower.
- KL-LS (KL-divergence): CSR 55.3, NIAH at 256K: 44.3%. Similar CSR, dramatically worse NIAH (30 points lower).
- Evenly distribute attention layers: CSR 54.0, NIAH at 256K: 50.9%.
- Evenly distribute in latter half: CSR 55.8 (surprisingly high), NIAH at 256K: 40.4%.
- RADLADS (RNN-only, no attention): CSR 56.0 (highest of all), NIAH at 256K: 0.0%.
The clear pattern is that the ratio-based selection achieves the best tradeoff: it maintains competitive CSR (within 0.4 points of the best CSR, which is RADLADS) while delivering substantially better long-context recall than any alternative. The evenly-distribute-in-latter-half baseline is particularly revealing: it achieves good CSR (55.8) but terrible NIAH (40.4%), suggesting that putting attention layers in the latter half alone is not sufficient — which specific layers are kept matters greatly. The RADLADS result (CSR 56.0, NIAH 0.0%) is the extreme case: removing all attention layers actually slightly improves CSR (perhaps because the RNN layers, when trained to match the teacher's hidden states, produce cleaner representations for CSR tasks), but completely destroys recall. This validates the paper's core premise that a small number of attention layers, strategically selected, are essential for preserving recall.
The layer indices selected by HALO for Qwen3-1.7B (Table 7) are revealing: the most important layers are 3, 21, 2, 9, 25, 6, 8 (0-based indexing, out of 28 layers). The first layer (index 0) and last layer (index 27) are among the least important — they are ranked 26th and 24th out of 28, respectively. This runs counter to the intuition that early layers (which process raw input) and late layers (which produce the final representation) would be most critical. Instead, the critical attention layers are distributed throughout the middle of the network. HALO's ratio-based selection captures this, while Jet-Nemotron and KL-LS select different sets (also reported in Table 7), with some overlap (e.g., layer 21 is selected by all methods for Qwen3-1.7B) but notable differences.
Training From Scratch: HyPE and RNN Mixer Validation (Section 6.3)
The from-scratch experiments serve two purposes: validating that HyPE's benefits are not artifacts of the distillation process, and identifying which RNN mixer is optimal for length generalization.
Position encoding comparison (Figure 4): Models with 500M parameters are trained from scratch on 20B tokens at 4K context length and evaluated on NIAH at lengths from 4K to 256K (up to 64× the training length):
-
HypeNet with HyPE (attention NoPE + RNN RoPE + dynamic scaling): NIAH accuracy remains above 90% at 64K, around 85% at 128K, and approximately 75% at 256K. This is the best length generalization of any tested configuration.
-
Standard Transformer with RoPE: Accuracy drops sharply beyond the training length — at 64K (16× training length), accuracy is near 0%. This is the well-known RoPE length generalization failure.
-
SWAN-GPT (sliding window attention with RoPE + full attention with NoPE): Better than the vanilla Transformer but worse than HypeNet. Accuracy at 256K is around 50%, compared to HypeNet's ~75%. SWAN-GPT has full attention layers with NoPE (similar to HypeNet's attention layers) but does not have RNN layers with RoPE, suggesting that the combination of NoPE attention + RoPE RNN is specifically beneficial.
-
HypeNet with all NoPE (no RoPE anywhere): Accuracy at 256K drops to around 25%, showing that removing RoPE from RNN layers significantly degrades length generalization. This is consistent with the distillation ablation in Table 3 ("w/o RNN RoPE": NIAH-128K drops from 79.9% to 47.9%), but the effect is even more pronounced in the from-scratch setting.
-
HypeNet with all RoPE (RoPE in both attention and RNN): Similar to vanilla Transformer — collapses beyond training length.
-
HypeNet with attention RoPE + RNN NoPE (the conventional hybrid PE): Accuracy at 256K is near 0%. This is the scheme used by existing hybrid models, and it performs worst in terms of length generalization. Combined with the distillation ablation ("w/ attention RoPE": NIAH-128K from 79.9% to 19.7%), this is the paper's strongest evidence that the conventional PE assignment is specifically harmful for length generalization.
A notable detail: the from-scratch training results actually show better length generalization than the distillation results. The from-scratch HypeNet achieves 93.5% NIAH accuracy at 64× training context length (256K from 4K training), while the distilled HypeNet achieves 79.9% at 128K from 16K training (Table 3, 8× training length). The paper attributes this to the from-scratch training being more amenable to learning HyPE's position patterns than the distillation process, which is constrained by the teacher's representations. This is acknowledged as a limitation — distilled models, while practical, do not reach the full potential of the architecture.
RNN mixer comparison (Figure 5): Same 500M-parameter HypeNet variants, all with HyPE, trained on 20B tokens at 4K context length, with different RNN mixers:
-
Lightning Attention: NIAH accuracy remains above 90% at 256K (64× training length). Essentially flat generalization curve.
-
GDN (Gated DeltaNet): NIAH drops to approximately 15% at 256K. Severe degradation.
-
GLA (Gated Linear Attention): NIAH drops to approximately 35% at 256K. Better than GDN but substantially worse than Lightning Attention.
-
RWKV-7: NIAH drops to approximately 25% at 256K.
-
Mamba2: NIAH drops to approximately 30% at 256K.
The pattern is consistent: all mixers with data-dependent forget gates degrade substantially at lengths beyond training, while the one mixer with data-independent forgetting (Lightning Attention) generalizes almost perfectly. The paper does not provide a mechanistic proof that data-dependence causes the degradation, but the correlation is strong enough that it forms the basis for the recommendation to use Lightning Attention in HypeNet. The from-scratch results are critical here because they show that this pattern persists even without the confounding factors of distillation (distribution shift, teacher constraints, limited training budget), strengthening the case that this is an architectural property rather than a training artifact.
The paper's takeaway from Figure 5 is explicit: "The incorporation of HyPE and Lightning Attention is both essential for achieving the exceptional length generalization of HypeNet." The "both" is important — HyPE provides the architectural framework for length generalization (NoPE attention), but that framework only works if the RNN layers don't interfere, and Lightning Attention's data-independent forgetting ensures they don't.
Ablation of Architectural Modifications in Distillation (Table 3)
Table 3 reports the ablation of HypeNet-2B (converted from Qwen3-1.7B) with individual components removed. The full HypeNet achieves CSR 55.9 and NIAH at 128K of 79.9%. The ablation results are:
-
Without RNN RoPE (➊, i.e., RNN NoPE): CSR drops from 55.9 to 53.8 (-2.1), NIAH-128K drops from 79.9% to 47.9% (-32.0). Removing RoPE from RNN layers substantially harms both CSR and length generalization. This is the second-largest NIAH drop in the table.
-
With attention RoPE (➊, i.e., conventional hybrid PE): CSR 55.8 (essentially unchanged), NIAH-128K drops from 79.9% to 19.7% (-60.2). This is the single largest ablation effect — restoring RoPE to attention layers almost completely destroys length generalization, even though CSR is unaffected.
-
Without RNN QK-norm (➋): CSR drops from 55.9 to 55.3 (-0.6), NIAH-128K drops from 79.9% to 17.3% (-62.6). This is the largest NIAH drop overall and highlights the critical role of QK-norm in stabilizing RNN computation for long contexts.
-
Without GQA to MHA decoupling (➌): CSR drops from 55.9 to 55.8 (-0.1), NIAH-128K drops from 79.9% to 83.5% (actually a slight increase at 128K, though performance at 4K–16K is lower: 89.7% vs. 95.9% at 4K). The effect on long-context NIAH is negligible; the main benefit of MHA decoupling appears to be at shorter contexts, consistent with increased expressivity helping local pattern recognition.
-
Without RNN output gate (➍): CSR drops from 55.9 to 55.6 (-0.3), NIAH-128K drops from 79.9% to 74.5% (-5.4). A modest but consistent effect.
-
Without attention output gate (➍): CSR drops from 55.9 to 55.4 (-0.5), NIAH-128K drops from 79.9% to 80.9% (again a slight increase at 128K). The attention output gate primarily affects CSR, not length generalization.
The salience ordering of the ablations is clear: the components most critical for long-context NIAH are, in order, QK-norm in RNN layers (62.6-point drop without it), attention NoPE (60.2-point drop if RoPE is restored to attention), RNN RoPE (32.0-point drop without it), RNN output gate (5.4-point drop), with GQA-to-MHA and attention output gate having negligible or slightly negative effects on NIAH-128K. For CSR, the effects are smaller and more evenly distributed, with RNN RoPE removal being the largest factor (2.1-point CSR drop).
HALO Training Configuration Ablations (Table 9)
Table 9 in Appendix F.1.1 reports an important finding about training data volume and learning rates:
-
Stage 1 data volume: Increasing Stage 1 training tokens from 320M to 625M or 1.3B does not improve final performance. In fact, with 1.3B tokens, NIAH-128K drops to 61.2% (from 80.9% with 320M tokens), despite slightly higher CSR at intermediate lengths. This is a non-obvious result: more data in the isolated layer alignment stage actually harms end-to-end performance. The paper does not fully explain this, but the likely mechanism is overfitting — the RNN layers become too specialized to mimicking the frozen teacher's hidden states and lose the flexibility needed to adapt in Stage 2 when the full model is trained end-to-end. This is practically important: it means the 320M-token budget for Stage 1 is not just a minimum but close to an optimum, and blindly increasing it would be counterproductive.
-
Stage 2 maximum learning rate: The default constant LR used in RADLADS (which the paper notes is
1e-5with a constant schedule) gives CSR of 46.9 and NIAH-128K of 60.7%. With HALO's cosine schedule, a max LR of1e-4gives CSR of 56.4 and NIAH-128K of 68.7%. A max LR of3e-4causes CSR to collapse to 46.0, while a max LR of1e-3gives CSR 36.8 but surprisingly good NIAH-128K (75.1%). However, the CSR collapse at3e-4and1e-3makes these unusable — the model essentially forgets its language modeling capabilities while somehow maintaining or improving recall. This extreme sensitivity to the Stage 2 learning rate (usable range appears to be roughly3e-5to1e-4) is a practical challenge for anyone replicating HALO, and the paper's reporting of model-specificη_stage2values (Table 6) is essential for reproducibility.
Ablation Studies and Robustness Checks
Attention logits scaling (Appendix F.1, Figure 8): Without logits scaling (s_t = 1), HyPE exhibits limited length generalization — NIAH accuracy at 256K is near 0%. With constant scaling (s_t = 1.5 for all positions), length generalization improves but the NIAH curve still drops substantially by 256K (to roughly 40–50%). With the position-dependent logarithmic scaling s_t = log_a(t + a), NIAH accuracy remains high (roughly 75–80%) at 256K. This validates that position-dependence is essential — later positions need more aggressive sharpening to counteract attention dilution from the larger softmax denominator. The model-dependent choice of a (500 for 2B, 600 for 5B, 900 for 9B, Table 6; and 300 for the from-scratch 500M Lightning model, Table 14) is determined post-training, which means it does not affect training and can be tuned cheaply.
Stage 3 context length: The paper does not provide a direct ablation of Stage 3's 16K context length against alternatives (e.g., 8K or 32K). The choice of 16K is justified implicitly by the results: HypeNet generalizes to 256K (16× the Stage 3 training length) with high accuracy, suggesting 16K is sufficient to provide the long-range signal needed. A direct ablation showing the sensitivity to this hyperparameter would strengthen the paper's claims about data efficiency, but it is absent.
Effect of increased model size from architectural modifications: HypeNet is "roughly 10% larger" than the base Qwen3 model due to GQA-to-MHA decoupling and output gate additions. The paper does not provide an ablation that controls for parameter count — i.e., a HypeNet variant where these components are added but the parameter budget is reduced elsewhere to match the original model size. This leaves open the possibility that some of HypeNet's performance advantage comes from the increased parameter count rather than the architectural design. The paper argues (citing Chen et al., 2025a) that "increasing model size while reducing the KV size is more cost-effective in long-context scenarios," but this is a claim about the efficiency tradeoff, not about the source of HypeNet's recall improvements. The ablation in Table 3 removes individual components but does not separately control for the resulting parameter reduction — removing GQA-to-MHA, for instance, reduces the model's parameter count by roughly 5%, so the small CSR decrease could be due to fewer parameters rather than the architectural change. This is a genuine limitation.
Compatibility with different base models: All distillation results use Qwen3 as the base model. HALO's design is architecturally general (any Transformer with QKV attention can be converted), but the paper does not replicate the results with other model families (LLaMA, Mistral, Gemma). The Qwen3-specific features (already having QK-norm in attention layers, specific RoPE theta of 1M) may interact favorably with HALO in ways that other architectures would not. This limits the strength of the claim that HALO is a general-purpose pipeline.
Kimi Delta Attention failure (Appendix G.2): The paper transparently reports a failed attempt to convert Qwen3's attention layers into Kimi Delta Attention (KDA) using the same HALO configuration. "The training process could not converge with the gradient norm becoming inf after a few steps in stage 2. We tried reducing the learning rate but it did not help." This is a valuable negative result that highlights the sensitivity of the distillation process to the RNN mixer's formulation. Not all QKV-based RNNs that fit the outer-product framework (Equation 4–6) are equally amenable to HALO's weight transfer and training procedure. The paper does not diagnose why KDA specifically fails, but the transparency about this failure strengthens credibility.
Short convolution removal (Appendix D): Many modern RNNs (Mamba2, GDN, Jet-Nemotron) include a short convolution layer (a per-channel 1D convolution with small kernel size) before the recurrent computation. The paper explicitly found that adding this through post-training "does not provide performance gains for the 8B model and even failed to converge when applied to the 1.7B model." This is a practical simplification that reduces implementation complexity (one less custom CUDA kernel), but it means HypeNet's RNN layers are simpler than those in some competing architectures. The failure to converge on smaller models suggests a stability issue that the paper leaves unresolved.
Oracle difficulty of layer selection: The layer selection method requires evaluating 28 single-layer-replacement models (for Qwen3-1.7B) on SQuAD, FDA, SWDE, HellaSwag, ARC-Easy, and ARC-Challenge. At 8.36M tokens of evaluation data total (Table 8), this is computationally cheap, but it requires ground-truth labels for all evaluation tasks. This is reasonable for a research setting but would not be available in all deployment scenarios. The paper does not explore whether the layer selection transfers across model families (e.g., using the layers selected for Qwen3-1.7B as a template for other 28-layer models) — this would be a practical simplification but is not tested.
Critical Assessment
Claim 1: "HALO converts Transformer models into hybrid models using fewer than 3B tokens, a 10–200× reduction over prior methods." This claim is well-supported by direct comparison with prior work in Table 1 and the explicit reporting of training token counts in Table 5 (320M + 1B + 1B = 2.32B tokens). However, the claim's significance depends on whether the resulting models are genuinely comparable. The paper's comparison in Table 2 is against Jet-Nemotron (2B, 400B tokens) and KL-LS (3B, 25B tokens) — HypeNet achieves better long-context performance, but the baselines use different base models, different architectures, and different training data. A controlled experiment where the same base model is distilled with HALO vs. a prior method at the same token budget would directly test the claim that HALO's method is more data-efficient rather than that its output happens to be better. This experiment is not run. The claim about data efficiency is therefore supported in the sense that HALO achieves strong results with few tokens, but the causal attribution to HALO's design (vs. the base model quality, the specific RNN mixer, or HyPE) is not isolated.
Claim 2: "HypeNet achieves superior long-context performance compared to prior distilled hybrid models." Strongly supported by Table 2. HypeNet's NIAH scores at 256K (99.8%, 86.2%, 48.8% for the three single-NIAH variants) are dramatically higher than Jet-Nemotron (0.0% across all three) and KL-LS (14.8%, 10.4%, 11.0%). Even at 128K, where the gap is smaller, HypeNet (99.8%, 97.8%, 44.8%) substantially exceeds KL-LS (68.4%, 28.2%, 24.8%). The comparison is somewhat unfair in that Jet-Nemotron and KL-LS use different base models and different RNN architectures, but the gap is large enough that it is unlikely to be explained by these differences alone. The HypeNet vs. Qwen3 teacher comparison at 256K is genuinely surprising — HypeNet outperforms the teacher on NIAH — and this is the strongest evidence that HyPE's length generalization is not just "less degradation" but actual improvement over the full-attention baseline at extreme lengths.
Claim 3: "HyPE achieves superior length generalization by combining NoPE in attention layers and RoPE in RNN layers." The evidence for the superiority of HyPE over alternatives is strong: Figure 4 shows HyPE outperforming all other PE schemes (all RoPE, all NoPE, conventional hybrid PE, SWAN-GPT) at context lengths beyond training. Table 3 shows that adding RoPE to attention layers collapses NIAH-128K by 60.2 percentage points, and removing RoPE from RNN layers drops NIAH-128K by 32.0 points. Both directions of the ablation are tested, and both support the claim.
However, the mechanism the paper proposes for why HyPE works — that RNNs have limited receptive fields and thus length generalization depends on attention layers — is not directly tested. The paper does not measure the RNN's effective receptive field, does not show that RNN layers attend primarily to short-range dependencies, and does not demonstrate that attention layers handle the long-range retrieval in the hybrid model. The mechanism is a plausible interpretation that explains the empirical results, but it is not an empirically validated causal model. The paper presents an effective design and a coherent story for why it works, but the story has more the character of post-hoc rationalization than hypothesis-driven experimentation.
Claim 4: "Lightning Attention's data-independent forget gates are essential for length generalization." The correlation is strong (Figure 5), but correlation is not causation. The five RNN mixers tested differ in many ways beyond data-dependence of the forget gate — they have different parameterizations, different numbers of parameters, different numerical stability properties, and different compatibility with the QKV weight transfer from HALO. The paper's causal claim is plausible and supported by prior work (Chen et al., 2025b), but the experiment that would directly test it — taking a data-dependent mixer and replacing its forget gate with a data-independent one while keeping everything else the same — is not run. The paper acknowledges this limitation in its language ("One possible explanation is..."), which is appropriately cautious, but the strength of the empirical recommendation (Lightning Attention as the clear choice for HypeNet) somewhat exceeds the strength of the mechanistic evidence.
Genuine weaknesses that are not fully addressed:
-
No direct replication of prior methods under controlled conditions. The comparison with Jet-Nemotron and KL-LS uses their reported results, not re-implementations within the HALO framework. This means the performance differences could be due to base model quality, training data composition, or implementation details rather than the layer selection or distillation method. The one controlled comparison in Table 4 (replacing only the layer selection method within HALO) shows HALO's selection outperforms Jet-Nemotron's and KL-LS's selection criteria, which is the strongest apples-to-apples evidence in the paper. But the full distillation pipelines are not compared head-to-head.
-
No parameter-count-matched ablation. HypeNet is ~10% larger than the base Qwen3 due to GQA-to-MHA decoupling and output gates. The paper does not compare against a Qwen3 variant that is 10% larger or a HypeNet variant that is parameter-matched. This clouds the interpretation of the CSR results — is HypeNet matching Qwen3's CSR because of the architecture, or because it has more parameters?
-
Single base model family (Qwen3). The paper's claims about HALO's generality are not empirically validated on other Transformer architectures. Qwen3 has specific architectural features (QK-norm, GQA, RoPE theta of 1M) that may interact with HALO's design in non-obvious ways. The failed KDA conversion (Appendix G.2) demonstrates that not everything works, which actually strengthens the need for cross-architecture validation.
-
No standard deviations or significance tests. The paper reports point estimates for all metrics without any measure of uncertainty. For CSR evaluation over 7 tasks, the aggregate score is an average of averages, and its stability is unknown. For NIAH, the number of test examples per context length is not reported, making it impossible to assess whether differences of a few percentage points are meaningful. The 500-question test set for MATH-based evaluation is common in this literature, but NIAH typically uses a small number of needle placements (often 500 per configuration), and variance can be substantial.
-
The attention logits scaling hyperparameter
ais chosen post-training on a validation set, but its sensitivity is not explored. How much does the optimaladepend on the target context length? If a model is tuned for 128K, does it still work at 256K? The paper's from-scratch experiments (Table 14) use differentavalues for different RNN mixers (ranging from 200 to 5000), suggesting significant mixer-dependence, but no systematic sweep is reported. -
No comparison with training a hybrid from scratch at the same total FLOPs. HALO uses 91 GPU-hours for conversion plus the cost of pre-training Qwen3-1.7B (which is not reported, but is presumably trillions of tokens). A comparison with training a small hybrid model from scratch using only the 91 GPU-hours would not be fair (the pre-training is the dominant cost), but a comparison allocating the total FLOPs (pre-training + distillation) between pure Transformer training and hybrid training would be informative about whether distillation recovers a significant fraction of what training from scratch would achieve.
Missing experiments that would strengthen the paper:
- An ablation testing Stage 3 with different context lengths (8K, 16K, 32K) to characterize how much long-context training is actually needed with HyPE.
- A controlled comparison where the same Qwen3 base model is distilled using Mamba-in-the-Llama, Jet-Nemotron's full pipeline, and HALO, each at their default token budgets and also at a fixed budget (e.g., 2.3B tokens), to isolate method from budget.
- A "HypeNet but with parameters matched" variant that removes the parameter increase from GQA-to-MHA and output gates (e.g., by reducing hidden dimension slightly) to test whether the architectural changes matter beyond the parameter count.
- Replication on a non-Qwen base model (LLaMA-3, Mistral) to test generality.
- A direct measurement of the RNN's effective receptive field in HypeNet (e.g., via probing or mutual information analysis) to support the mechanistic claim behind HyPE.
6. Limitations and Trade-offs
Limitation 1: The Difficulty Estimation for Layer Selection Is Computationally Cheap but Requires Task-Specific Ground-Truth Labels
The assumption or constraint. HALO's attention layer selection method requires evaluating L models (one per layer, each with that layer replaced by an RNN) on curated evaluation suites: SQuAD, FDA, and SWDE for recall performance, and HellaSwag, ARC-Easy, and ARC-Challenge for CSR performance. The paper reports that this requires 8.36M inference-only tokens and 6.5 × 10¹⁷ FLOPs for Qwen3-1.7B (Table 8) — computationally negligible compared to the training stages. However, all of these evaluation tasks require ground-truth labels (answer keys for reading comprehension, structured extraction references, multiple-choice answer keys). This is acknowledged implicitly by the choice of benchmarks — all are standard labeled evaluation datasets — but the paper does not discuss the label-availability assumption.
The consequence. In a deployment setting where the goal is to distill a Transformer into a hybrid for a specific downstream application, the practitioner may not have access to labeled recall and CSR benchmarks that are representative of their target distribution. The layer selection method would then require either (a) acquiring or creating labeled evaluation data for the recall-vs-CSR tradeoff, which may be expensive or infeasible, or (b) falling back to a heuristic layer selection strategy (e.g., evenly distributing attention layers), which Table 4 shows produces substantially worse long-context NIAH performance (50.9% at 256K vs. 74.3% for HALO's ratio-based selection on Qwen3-1.7B). The layer selection is also specific to the base model — the paper does not test whether the selected layer indices transfer across model families or even across model sizes within the same family, though Table 7 shows that for Qwen3-4B and Qwen3-8B, the selected layers differ from the 1.7B model.
What evidence exists in the paper. Table 4 shows that alternative layer selection methods (Jet-Nemotron, KL-LS, even distribution) all produce meaningfully worse NIAH at long contexts, establishing that layer selection matters — HALO's ratio-based method is not a minor refinement but a significant contributor to long-context recall. Table 7 reports the selected layer indices for each model size, showing they vary across model scales, which suggests the selection is not trivially transferable. The paper does not report the performance of using the 1.7B model's selected layers on the 4B or 8B conversion, nor does it test whether a purely unsupervised selection criterion (e.g., using the PRM's internal metrics rather than downstream task accuracy) would produce comparable results.
Mitigation status. The paper does not address this limitation or suggest alternatives to labeled data for layer selection. The layer selection process is treated as a one-time cost during model conversion that assumes access to standard evaluation benchmarks. In the academic research setting the paper targets, this is reasonable — these benchmarks are publicly available and widely used. In a production setting distilling for a custom domain, it is a non-trivial requirement. The paper's silence on unsupervised or self-supervised alternatives leaves a gap for practitioners who do not have labeled recall data.
Limitation 2: The "Roughly 10% Larger" Parameter Count Is Not Controlled For, Confounding Architecture and Capacity Effects
The constraint. HypeNet is "roughly 10% larger" than the Qwen3 model it is distilled from (Section 5.2, "Increased Model Size"), due to two architectural additions: GQA-to-MHA decoupling (➌, cloning KV projection heads so every attention head has independent keys and values in RNN layers) and output gates (➍, adding W_z matrices in both RNN and attention layers). The paper estimates this increase at approximately 10% but does not provide exact parameter counts for HypeNet vs. Qwen3 at each scale, nor does it control for parameter count in any ablation.
The consequence. The CSR and NIAH comparisons between HypeNet and Qwen3 in Figure 1, Table 2, and Table 4 are not parameter-matched. When HypeNet-2B achieves CSR 55.9 vs. Qwen3-1.7B's CSR 58.5 (a 2.6-point drop), it is unclear how much of that drop to attribute to the architectural change (replacing attention layers with RNNs) versus how much is masked by the parameter increase (more parameters can partially compensate for less expressive computation). Conversely, when HypeNet outperforms Qwen3 on NIAH at 256K (99.8% vs. 17.0% on NIAH-Single-1, Table 2), it is unclear whether the improvement is due to HyPE's length generalization or partially due to having ~10% more parameters absorbing and processing the long context. The paper's claim that "increasing model size while reducing the KV size is more cost-effective in long-context scenarios" (citing Chen et al., 2025a) is a statement about the efficiency tradeoff (the combined effect of more parameters but less KV cache memory), not about the source of performance improvements. A parameter-matched comparison would isolate the architectural contribution.
What evidence exists in the paper. The ablation in Table 3 removes individual architectural components (GQA-to-MHA, output gates) and shows that performance degrades. However, removing these components also removes parameters — the "w/o RNN GQA to MHA" ablation reduces the model's KV projection parameters by roughly half in the RNN layers, and "w/o RNN output gate" removes d × d_h parameters per RNN layer. These ablations conflate the effect of the architectural mechanism with the effect of the parameter reduction. The paper does not include a "parameter-matched HypeNet" baseline where parameters are reduced elsewhere (e.g., narrower FFN, fewer heads) to keep the total count equal to the base Qwen3.
Mitigation status. The paper does not address this confound. The 10% parameter increase is mentioned as a feature (more cost-effective in long contexts) rather than a limitation to be controlled for. In the from-scratch experiments (Section 6.3, Table 13), the baseline Transformer, SWAN-GPT, and HypeNet variants are all controlled to ~500M parameters, which makes those comparisons clean. But the distillation results — which are the paper's main contribution — are not parameter-controlled, leaving the architectural claim partially supported. This is a genuine methodological gap that weakens the strength of the conclusion that HypeNet's architecture (as opposed to HypeNet's size) drives the improvements.
Limitation 3: All Distillation Results Use a Single Base Model Family (Qwen3), Limiting Generality Claims
The constraint. Every distillation experiment in the paper converts Qwen3 models (1.7B, 4B, 8B) using HALO. The paper states that HALO is designed to convert "pre-trained Transformer models into hybrid models" (Section 1) and that "HypeNet is agnostic to the RNN mixer" (Section 5.3), implying generality. However, no results are reported for other widely-used Transformer families (LLaMA, Mistral, Gemma, Phi). Qwen3 has specific architectural features — notably, QK-normalization already present in its attention layers, GQA with 8 KV heads, a RoPE theta of 1M, and SwiGLU MLPs — that may interact favorably or unfavorably with HALO's design choices.
The consequence. A practitioner attempting to distill a LLaMA-3 model (which, unlike Qwen3, does not have QK-norm by default) would need to add QK-norm to the RNN layers during initialization. The paper states that when converting models without QK-norm, "we add QK-norm to the RNN layers" (Section 5.2, ➋). But this means the RNN layers receive an architectural component absent in the teacher's attention layers, potentially changing the distribution of hidden states in ways that affect the teacher-student alignment in Stage 1. The Qwen3 base model was trained with QK-norm throughout, meaning the hidden state distributions entering each layer are already conditioned on normalized queries and keys; an RNN layer receiving these normalized inputs in Stage 1 is operating in a regime consistent with the teacher's pre-training. For a LLaMA model without QK-norm, the RNN would receive un-normalized hidden states but apply normalization internally — a different regime. Whether HALO's data efficiency holds under these conditions is unknown.
Additionally, the failed conversion of Kimi Delta Attention (Appendix G.2) — where "the training process could not converge with the gradient norm becoming inf after a few steps in stage 2" — demonstrates that not all RNN architectures that fit the outer-product formulation are compatible with HALO even when using Qwen3 as the base model. This fragility to mixer choice, combined with the single-family base model limitation, means the space of (base model, RNN mixer) pairs where HALO works is not well-characterized.
What evidence exists in the paper. The paper provides evidence only for Qwen3 conversions at three scales (1.7B, 4B, 8B) using Lightning Attention as the RNN mixer. The from-scratch experiments (Section 6.3) test multiple RNN mixers but use a custom 500M-parameter architecture, not conversions from pre-trained models. The KDA failure is reported transparently but not diagnosed. There is no experiment converting a non-Qwen3 model using HALO.
Mitigation status. The limitation is not acknowledged or discussed. The paper's claim that HALO is a general pipeline for "converting pre-trained Transformer models into RNN-attention hybrid architectures" (abstract) is stated without qualification. The choice to use only Qwen3 is not defended, nor is the potential interaction between base model architecture and HALO's effectiveness examined. The KDA failure is reported in an appendix and treated as a mixer-specific issue, but it could equally indicate a broader fragility in HALO's assumption that any QKV-based RNN can be initialized from attention weights and fine-tuned with the same recipe.
Limitation 4: Post-Training Capabilities (Instruction Following, Alignment) Are Destroyed by the Distillation Process
The constraint. HALO trains exclusively on FineWeb-edu, an Internet-scale pre-training corpus consisting of raw web text. The Qwen3 base models have undergone significant post-training (instruction tuning, RLHF, or other alignment procedures) that instills instruction-following behavior, safety properties, and conversational capabilities. The distillation process — particularly Stage 2 (KL divergence on pre-training data) and Stage 3 (long-context pre-training data) — uses only pre-training-style text. The paper acknowledges this in the Limitations section:
"Instruction-following and alignment behaviors of the pre-training model introduced by post-training may be diminished by our conversion process. However, this is a common shortcoming of all existing distillation methods for converting into hybrid architectures. How to efficiently recover the base models' capabilities remains an open question."
The consequence. HypeNet, as produced by HALO, is a base model (pre-trained-style), not an instruction-tuned or chat model. It cannot be used as a drop-in replacement for Qwen3-Chat or Qwen3-Instruct in applications that require instruction following, safety guardrails, or conversational ability. The efficiency gains (3.0× decoding speedup, 3.4× prefilling speedup at 512K) are measured on a model that lacks the capabilities that make Qwen3 useful in practice. A practitioner would need to apply post-training (instruction tuning, preference optimization) to HypeNet after HALO conversion, which would require additional data, compute, and engineering effort not accounted for in the paper's 91 GPU-hour budget. Moreover, applying post-training to a hybrid model with RNN layers raises open questions about whether standard instruction-tuning recipes (designed for pure Transformers) transfer without modification, and whether the efficiency advantages (smaller KV cache, faster inference) persist after post-training adds new behavioral requirements.
What evidence exists in the paper. The paper evaluates HypeNet exclusively on pre-training-style benchmarks: CSR tasks (zero-shot multiple choice) and NIAH (needle retrieval). These do not require instruction following. The paper provides no evaluation of HypeNet on instruction-following benchmarks (IFEval, MT-Bench, AlpacaEval), safety benchmarks, or conversational quality metrics. The Qwen3 teacher model evaluated in the paper is the base model (not Qwen3-Instruct), which means the CSR and NIAH comparisons are fair in a pre-training sense. But for any deployment scenario where users interact with the model through instructions rather than zero-shot prompts, HypeNet's performance is unknown.
Mitigation status. The paper explicitly acknowledges this limitation in the dedicated Limitations section (quoted above), which is transparent and appropriate. However, it does not estimate the cost or complexity of recovering post-training capabilities, nor does it provide any partial mitigation (e.g., a lightweight instruction-tuning stage as Stage 4 of HALO). The framing — "this is a common shortcoming of all existing distillation methods" — is accurate but does not reduce the practical barrier. For a practitioner, the choice is not just between HypeNet and Qwen3-base, but between HypeNet-plus-post-training and Qwen3-Instruct, and the paper provides no data on the former.
Limitation 5: The Mechanism Behind HyPE's Length Generalization Is Not Empirically Validated, Only Hypothesized
The constraint. The paper proposes a specific mechanism for why HyPE works: RNN layers have a limited effective receptive field due to their fixed-size state, so they primarily handle short-distance dependencies, while attention layers handle long-distance dependencies. When context length exceeds the RNN's receptive field, RNN behavior becomes agnostic to total sequence length, meaning length generalization depends only on the attention layers. Therefore, giving NoPE to attention layers (which generalize well to unseen lengths) and RoPE to RNN layers (which don't need to generalize) produces the optimal length generalization. This mechanistic story is stated in Section 5.1 (Motivation) and is used to justify the counterintuitive inversion of the conventional PE assignment.
The consequence. If the mechanism is incorrect, HyPE's effectiveness may be contingent on factors not captured by the receptive-field hypothesis (e.g., specific properties of Lightning Attention, training data distribution, or interaction with dynamic logits scaling). This matters for practitioners who want to extend or modify HypeNet: without knowing why HyPE works, they cannot predict whether changes to the architecture (different RNN mixers, different layer ratios, different training recipes) will preserve or break the length generalization property. For example, if a different RNN mixer is shown to have a larger receptive field than Lightning Attention, does that mean RoPE should be moved back to attention layers? The paper's hypothesis would predict that, but it is untested. The from-scratch results in Figure 5 show that Lightning Attention is the only mixer that preserves length generalization, but the paper attributes this to data-independent forgetting, not to receptive field properties. The two hypotheses (receptive field mechanism for HyPE, data-independence mechanism for mixer choice) are presented as complementary but are not connected or tested jointly.
What evidence exists in the paper. The paper provides extensive evidence that HyPE works (Figure 4, Table 3, Table 2) but no direct evidence for the mechanism. Specifically, the paper does not:
- Measure the effective receptive field of RNN layers in HypeNet (e.g., via probing classifiers, gradient-based attribution, or perturbation analysis).
- Demonstrate that attention layers in HypeNet are primarily responsible for long-range retrieval (e.g., by ablating attention layers at different positions and measuring the impact on NIAH at different context lengths).
- Show that the RNN layers' behavior is invariant to context length beyond some threshold (e.g., by analyzing the recurrent state
S_tat different context lengths).
The mechanistic claims cite prior work (Chen et al., 2025b for the receptive field of RNNs; Kazemnejad et al., 2023 for NoPE length generalization), but these are general findings about RNNs and Transformers respectively — they are not validated within the specific context of HypeNet's hybrid architecture where attention and RNN layers interact.
Mitigation status. The paper does not acknowledge this as a limitation or present the mechanistic story as a hypothesis rather than a validated explanation. The language in Section 5.1 is assertive ("HyPE is motivated by the finding that RNNs have a limited 'receptive field'... This implies that in hybrid models, RNN layers primarily model short-distance dependencies"), not hedged. The paper could strengthen its contribution by either (a) providing direct evidence for the mechanism within HypeNet, or (b) explicitly framing the mechanism as a plausible interpretation that remains to be validated. The current presentation conflates "this design works" with "this is why the design works," which is a common pattern in systems papers but leaves the theoretical understanding incomplete.
Limitation 6: The Attention Logits Scaling Hyperparameter a Is Chosen Post-Training on a Validation Set, and Its Sensitivity and Transfer Properties Are Unexplored
The constraint. HyPE's attention logits scaling mechanism uses a position-dependent scaling factor s_t = log_a(t + a), where a is a hyperparameter chosen after training by minimizing perplexity on a set of pre-training documents (Section 5.1). The paper reports the selected a values: 500 for HypeNet-2B, 600 for HypeNet-5B, 900 for HypeNet-9B (Table 6, from distillation), and values ranging from 200 to 5000 for the from-scratch experiments depending on the RNN mixer (Table 14). The selection of a is not part of the training process — it is a post-hoc calibration step.
The consequence. This introduces a practical complication for deployment. The optimal a depends on at least three factors: model size (larger models use larger a, meaning slower scaling growth), the RNN mixer (Lightning Attention uses a=300 from scratch vs. GDN at a=200 vs. RWKV-7 at a=5000), and the target context length (implicitly, since the validation perplexity is evaluated at some context length). A practitioner deploying HypeNet at a specific context length needs to determine a for their specific model size, RNN mixer, and target length. The paper provides no guidance for how to select a efficiently — the reported values are specific to the models and evaluations in the paper.
More critically, the paper does not investigate whether a optimized for one context length transfers to other lengths. Figure 8 shows that without scaling (s_t = 1), NIAH accuracy at 256K collapses to near 0%, suggesting a is essential for extreme lengths. But if a=500 works well for HypeNet-2B at 256K, would a=300 or a=700 work better? Would the same a be optimal at 128K and 1M? The paper's evaluation of attention logits scaling (Appendix F.1, Figure 8) only compares no scaling, constant scaling (s_t = 1.5), and the proposed logarithmic scaling with a single a value — it does not sweep a or report sensitivity.
What evidence exists in the paper. Table 6 reports model-specific a values; Table 14 reports mixer-specific a values for the from-scratch experiments. The variation across mixers (200–5000) is large, indicating that a is not a minor tuning knob but a significant hyperparameter whose optimal value varies by an order of magnitude depending on the architecture. Figure 8 shows that the logarithmic scaling dramatically outperforms constant scaling and no scaling, but only for a single a value. The paper does not report how a was swept (grid search over what range? at what granularity?), what the validation perplexity curve looks like as a function of a, or how sensitive NIAH performance is to suboptimal a choices.
Mitigation status. The paper treats a as a model-specific hyperparameter analogous to learning rate or batch size — something to be tuned per model — but does not provide the tuning methodology, the computational cost of tuning, or a sensitivity analysis. In the from-scratch experiments, a is reported alongside other hyperparameters in Table 14 without comment on its selection. A practitioner without the resources to run a full grid search over a at deployment context lengths has no principled way to set this hyperparameter beyond copying the paper's values and hoping they transfer — which, given the mixer-dependence, they may not.
7. Implications and Future Directions
How This Work Changes the Landscape
This paper does not introduce a fundamentally new class of models — RNN-attention hybrids existed before, and distillation from pre-trained Transformers into hybrids existed before. What it changes is accessibility and diagnostic precision. The shift is from "hybrid models are promising but only large industrial labs can build and study them" to "hybrid models are an architecture that academic groups can now empirically investigate and iterate on, with a 2.3B-token, 91-GPU-hour budget." This is a shift in who can participate in hybrid architecture research, and by extension, in how fast the field can explore the design space.
The paper's second contribution — identifying HyPE as a counterintuitive but superior position encoding scheme for length generalization — is a diagnostic reframing rather than a paradigm shift. It does not change the fundamental computational model (attention plus recurrence), but it does change what question you ask when designing a hybrid model's position encoding. Before this work, the question was: "What PE is natural for each layer type?" The answer was RoPE for attention, NoPE for RNNs — as adopted by every prior hybrid model. After this work, the question becomes: "Which layers determine length generalization, and what PE maximizes that property?" The answer inverts the conventional assignment. This is the kind of insight that, once demonstrated, seems obvious in retrospect, but the fact that no prior hybrid model adopted it despite the NoPE length generalization literature being well-known since 2023 indicates it was a non-obvious connection.
The paper reconciles a latent tension in the literature between two empirical findings. Finding 1, from Kazemnejad et al. (2023) and others: NoPE attention generalizes to unseen lengths better than RoPE attention. Finding 2, from the hybrid model literature (Qwen3-Next, MiniMax-01, Jamba): hybrid models adopting RoPE in attention layers achieve strong performance within their training context. The tension is: if NoPE is better for length generalization, why don't hybrid models use it? The paper's resolution is that the RNN layers in these hybrids were evaluated at training-length contexts where explicit position encoding helps, and the length generalization question was simply not systematically tested at the extreme lengths (128K–256K) where NoPE's advantage materializes. HyPE breaks this tension by assigning the two PEs to different layer types based on their functional roles in length generalization, resolving the apparent contradiction: both RoPE and NoPE are useful, but in different places.
A more subtle landscape shift concerns which research directions become more or less attractive:
Directions that become more attractive:
- Data-efficient cross-architecture distillation as a research methodology, not just an engineering shortcut. HALO's three-stage decomposition (hidden state alignment → end-to-end distillation → long-context finetuning) demonstrates that isolating per-layer problems before solving the joint problem dramatically reduces data requirements. This decomposition principle could apply beyond hybrid models — for instance, converting Transformers to other subquadratic architectures (sparse attention, linear attention variants with different state representations) by decomposing the conversion into layer-local and global stages.
- Position encoding design as a length-generalization lever specifically tuned to which layers handle long-range dependencies. HyPE's counterintuitive inversion suggests that PE design should not be based on layer type (attention vs. RNN) but on functional role (long-range retrieval vs. local processing). Future architectures with more exotic layer mixtures could apply this diagnostic: identify which components handle long-range dependencies and optimize their length generalization properties independently.
- Data-independent recurrence for long-context stability. Lightning Attention's empirical dominance in Figure 5 provides a concrete tradeoff: data-independent forget gates sacrifice expressivity (less flexible state updates) in exchange for predictable and stable forgetting behavior that generalizes to 64× training length. This suggests a research program on designing RNN state updates that achieve the stability of data-independent forgetting without fully sacrificing adaptivity — perhaps through bounded data-dependence or hybrid forgetting schemes.
Directions that become less attractive:
- Training hybrid models from scratch at small to medium scale for research purposes. If HALO can produce competitive hybrid models from existing Transformer checkpoints at 2.3B tokens, the marginal value of training a hybrid from scratch on 20B+ tokens for architecture exploration decreases sharply. The paper makes this point explicitly: distillation enables "teams without resources to scale up pre-training to validate research ideas." This does not obviate pre-training from scratch for frontier models, but it does mean that the research community can decouple architecture validation from the expense of large-scale pre-training.
- Sophisticated data-dependent RNN mixers for long-context modeling. Figure 5 shows that GDN, GLA, RWKV-7, and Mamba2 all degrade substantially at lengths beyond training despite being more recent and more expressive than Lightning Attention. If the mechanism is indeed data-dependent forgetting (as the paper hypothesizes based on Chen et al., 2025b), then further increasing the sophistication of data-dependent forget gates without addressing the stability problem is unlikely to improve long-context performance. The direction shifts from "more expressive gating" to "stable, predictable, or bounded forgetting."
- Uniform attention layer distribution as a default. Table 4 shows that evenly distributing attention layers throughout the network (the "every
k-th layer" approach) produces substantially worse long-context recall (50.9% NIAH at 256K) than HALO's ratio-based selection (74.3%). This is not a small difference — it means the simple heuristic that has been a common default in hybrid architecture design is significantly suboptimal, and the field should move toward principled layer selection methods that account for the recall-vs-reasoning tradeoff.
Follow-Up Research This Work Enables
Characterizing the effective receptive field of RNN layers in hybrid models. The paper's central mechanistic hypothesis — that RNN layers primarily handle short-distance dependencies while attention layers handle long-distance dependencies — is asserted based on prior work (Chen et al., 2025b) but never directly measured within HypeNet. A strong follow-up would quantify the effective receptive field of each RNN layer in a trained HypeNet model. This could be done through causal tracing: perturb a token at position j in the input and measure the effect on the RNN layer's output at position t, as a function of the distance t − j. The prediction from the paper's hypothesis is that RNN layers' influence decays rapidly with distance (effective receptive field of perhaps a few hundred to a few thousand tokens), while attention layers' influence remains non-zero at arbitrary distances. The Lightning Attention decay rates γ_h (ranging from 0.431 to 0.996 per head) provide an upper bound: a head with γ = 0.996 retains 0.996^T ≈ 0.018 of information after T = 1000 tokens, suggesting the RNN's effective memory is bounded by these decay rates regardless of data-dependence. Measuring this directly would validate the paper's motivation for HyPE and also provide a diagnostic for when the RNN's receptive field is insufficient — informing decisions about how many attention layers are needed and where to place them.
Transferability of HALO's layer selection across model families and the sensitivity to evaluation tasks. The paper's layer selection method is evaluated only on Qwen3 at three sizes. A study that applies HALO's ratio-based selection to LLaMA-3, Mistral, and Gemma base models, and compares the selected layer indices, would establish whether there is a universal pattern (e.g., early-to-mid layers are consistently more recall-critical) or whether the optimal set is highly model-specific. If a universal pattern emerges, practitioners could adopt a default template for layer selection without running the full evaluation — dramatically lowering the barrier to entry further. If the pattern is model-specific, it would indicate that the recall-vs-CSR tradeoff is a function of the model's pre-training dynamics, and layer selection must remain model-specific. The experiment would additionally test whether HALO's overall distillation pipeline (not just layer selection) transfers: convert a LLaMA-3-8B into a HypeNet variant using HALO with 2.3B tokens, and compare against a Qwen3-8B HypeNet conversion. Does the same 2.3B-token budget suffice for a model with different pre-training characteristics (no QK-norm, different RoPE theta, different training data distribution)? This would directly address Limitation 3.
A parameter-matched HypeNet baseline to isolate architectural effects from capacity effects. The paper's main distillation results are confounded by HypeNet being ~10% larger than the base Qwen3. A controlled experiment would create a "parameter-matched HypeNet" where the hidden dimension or number of heads is reduced to compensate for the parameters added by GQA-to-MHA decoupling and output gates, keeping the total parameter count equal to the original Qwen3. This would isolate whether HypeNet's CSR and NIAH performance relative to Qwen3 (e.g., CSR 55.9 vs. 58.5, NIAH-256K 99.8% vs. 17.0%) is due to the architecture or the larger capacity. If the parameter-matched HypeNet still substantially outperforms Qwen3 on long-context NIAH, the architectural claim is strengthened. If most of the NIAH advantage disappears, the contribution shifts from "HyPE enables length generalization" to "HyPE plus extra parameters enables length generalization," which would be a substantially weaker claim but still practically useful if the parameter increase is cost-effective.
A "HypeNet-Instruct" variant with post-training recovery to test deployability. The paper acknowledges that HALO destroys post-training capabilities. A natural follow-up would add a Stage 4 to HALO: instruction tuning on a modest budget (e.g., 100M–500M tokens of instruction data) after Stage 3, and evaluate on standard instruction-following benchmarks (IFEval, MT-Bench) and long-context instruction tasks (LongBench, ∞Bench). This would answer the practical question: can HypeNet be made usable as a chat/instruct model, and if so, at what additional training cost? It would also test whether the RNN layers in HypeNet, which were trained only on pre-training data, can adapt to instruction-following via standard fine-tuning without losing their length generalization properties. A negative result here — instruction tuning degrades long-context NIAH — would indicate that HypeNet's length generalization is fragile to distribution shift and may not survive standard post-training, a critical practical concern.
Stress-testing HyPE at extreme context lengths (1M–4M tokens) and characterizing the failure mode. The paper evaluates up to 256K tokens and shows HypeNet maintains high NIAH accuracy at that length. But Figure 4 suggests a gradual decline from 128K to 256K (from ~85% to ~75% in the from-scratch setting). Extending the evaluation to 1M and 4M tokens would characterize the failure mode: does NIAH collapse suddenly at some threshold, or decay gradually? Does the logarithmic attention scaling s_t = log_a(t + a) remain effective at these lengths, or does it require re-tuning? Does the RNN state S_t exhibit numerical instability (overflow, underflow, or saturation) at million-token scales? The paper's HyPE design predicts that length generalization is primarily determined by the attention layers (which use NoPE and should generalize indefinitely) and the RNN's stability (which depends on the data-independent decay rates). If NIAH collapses at 1M despite NoPE attention, it would indicate that the RNN layers do contribute to long-range retrieval in ways the paper's model does not capture, or that attention dilution overwhelms the logarithmic scaling at extreme lengths. This experiment would directly test the limits of the paper's mechanistic hypothesis.
Joint optimization of layer selection and RNN mixer choice within the same distillation budget. The paper treats layer selection and RNN mixer choice as independent: Section 4.3 selects attention layers assuming Lightning Attention as the RNN mixer, and Section 5.3 selects Lightning Attention based on from-scratch experiments. A joint optimization study would ask: if we change the RNN mixer (e.g., to GDN or Mamba2), does the optimal set of attention layers change? The hypothesis is that data-dependent mixers, which have worse length generalization, might require more attention layers or different attention layer positions to compensate for the RNN layers' shorter effective memory span. This would reveal whether the layer selection recipe (25% attention layers, ratio-based criterion) is specific to the Lightning Attention + HyPE combination or is a general principle. A negative result — where Mamba2-based HypeNet with optimized layer selection still collapses on long contexts — would further strengthen the paper's claim that data-independent forgetting is the critical architectural property, not the layer selection method itself.
Practical Applications and Downstream Use Cases
Academic research on hybrid architectures. This is the use case the paper explicitly targets, and the numbers support it. A research group with access to a single 8-GPU A800 node can convert Qwen3-1.7B into HypeNet-2B in approximately 91 GPU-hours (Table 8), or roughly 12 hours of wall-clock time if using all 8 GPUs. The 2.3B-token training dataset (FineWeb-edu) is publicly available. The code and model checkpoints are released. This means a PhD student can, in a single day, produce a functional hybrid model and then spend the rest of their research budget on architecture ablations, new PE schemes, novel RNN mixers, or layer selection criteria — rather than spending weeks or months on the distillation itself. The comparison with prior methods makes this concrete: running Jet-Nemotron's pipeline (400B tokens, 200× more) on the same hardware would take months and is not feasible for most academic groups. HALO's 2.3B-token budget makes the research cycle fast enough for iterative experimentation.
Cost-efficient long-context inference for batch processing and retrieval pipelines. For applications where many long documents need to be processed (e.g., embedding computation for retrieval over legal documents, summarization of financial reports, or NIAH-style extraction from large corpora), HypeNet's efficiency advantages translate directly to cost savings. At 512K context length, HypeNet achieves up to 3.0× decoding speedup and 3.4× prefilling speedup over Qwen3-1.7B, and at 1M context length, Qwen3 runs out of GPU memory entirely on a single A800 while HypeNet continues to operate (Section 6.6, Figure 1 right). For a batch processing pipeline running inference on thousands of 128K-context documents, a 2–3× speedup means the pipeline completes in less than half the time or requires half the GPU instances. The reduced KV cache size (only 7 attention layers instead of 28) means the model can fit longer contexts or larger batch sizes in the same GPU memory, improving throughput further.
Deployment of long-context models on memory-constrained hardware. HypeNet's small KV cache (25% of the original attention layers) makes it viable to run long-context inference on GPUs with limited memory. The paper reports that at 1M context length, Qwen3-1.7B runs out of GPU memory on an 80GB A800, while HypeNet-2B does not. This enables edge deployment scenarios or single-GPU serving of ultra-long-context models that would otherwise require multi-GPU setups or aggressive KV cache quantization. For applications like document-grounded question answering over book-length texts, or code understanding over entire repositories, being able to fit the full context on a single consumer or edge GPU is the difference between feasible and infeasible deployment.
Data generation for self-improvement or distillation pipelines. When using an LLM to generate training data for another model (e.g., teacher-student distillation, synthetic data generation for domain-specific fine-tuning), inference cost is the dominant factor if the generation budget is large. HypeNet's 2–3× speedup at long contexts means a data generation pipeline processing long documents can produce 2–3× more training examples in the same compute budget. This is particularly relevant for long-context fine-tuning data generation (e.g., creating training examples that require reasoning over full-length documents), where the documents themselves are long and the inference cost per example is high. The tradeoff would need to account for HypeNet's slightly lower CSR (55.9 vs. 58.5, a 2.6-point drop) — if the target task can tolerate this quality reduction, the efficiency gain is pure profit.
