mHC: Manifold-Constrained Hyper-Connections

🎯 Pitch

Hyper-Connections extend residual streams but lose the identity mapping, causing training to explode—some gain metrics spike to 3000. By projecting their connection matrices onto the Birkhoff polytope of doubly stochastic matrices, mHC caps this instability at ~1.6, enabling stable scaling that beats standard residuals by 0.021 in loss with only minor overhead.

1. Executive Summary

This paper proposes Manifold-Constrained Hyper-Connections (mHC), a framework that projects the learnable residual connection matrices of Hyper-Connections (HC) onto the Birkhoff polytope of doubly stochastic matrices using the Sinkhorn-Knopp algorithm (iteratively normalizing rows and columns to sum to 1), thereby restoring the identity mapping property that HC sacrifices for its multi-stream expressivity. Evaluated on language model pretraining with MoE architectures up to 27B parameters on standard benchmarks, mHC eliminates the catastrophic training instability of HC—dropping the composite mapping’s Amax Gain Magnitude from peaks of ~3000 down to a bounded ~1.6—while delivering a final loss reduction of 0.021 over the standard residual baseline and consistently outperforming HC on downstream tasks by up to 2.1% on BBH and 2.3% on DROP. Critically, these stability and performance gains come at only a 6.7% additional training time overhead under infrastructure optimization, establishing that diversified residual connections can scale to large models only when the residual mapping is constrained to a norm-preserving manifold that prevents unbounded signal amplification across layers.

2. Context and Motivation

The Core Problem: Scaling Model Capacity Without Losing Training Stability

The fundamental tension this paper addresses is architectural: how do we increase the information capacity of a deep network's residual connections without breaking the identity mapping property that makes those connections trainable at scale? This is not merely a theoretical curiosity — it sits at the heart of why we can train thousand-layer vision models and hundred-layer language models with standard residual connections, yet struggle the moment we try to enrich those connections beyond the simple additive formulation $x_{l+1} = x_l + \mathcal{F}(x_l, W_l)$ .

The standard residual connection (He et al., 2016a,b), illustrated in Figure 1(a), has remained essentially unchanged since ResNet swept the field a decade ago. Its power comes not from any clever transformation of the residual stream, but from what it doesn't do: it passes the signal $x_l$ through completely unmodified. When you unroll a residual network across $L$ layers, you get:

$x_L = x_l + \sum_{i=l}^{L-1} \mathcal{F}(x_i, W_i)$

The term $x_l$ — the identity mapping — acts as a highway. During backpropagation, gradients flow through this highway unimpeded, regardless of how many layers intervene. This is why residual networks don't suffer from the vanishing gradient problem that plagued pre-ResNet deep architectures: the gradient of the loss with respect to $x_l$ always contains an additive term directly from $x_L$ , ensuring that even if every $\mathcal{F}(x_i, W_i)$ contributes near-zero gradient, the identity path still provides a stable signal. This property is so fundamental that it's easy to take for granted — until you try to change the residual connection and suddenly your model spirals into NaN territory at step 12,000.

Why This Gap Matters Now: The Architecture Scaling Bottleneck

The past several years have seen massive scaling of language models along two well-understood axes: model size (parameters, FLOPs) and training data. The scaling laws literature (Hoffmann et al., 2022) tells us how to allocate compute between these axes optimally. But scaling model FLOPs — by adding more layers or widening existing ones — eventually hits diminishing returns and hardware constraints. This has motivated a third, less-explored axis: architectural expressivity through enriched connectivity patterns.

The key insight driving this direction is that the information capacity of the residual stream — the "working memory" that carries representations from layer to layer — is currently bottlenecked by the input dimension $C$ of each layer. In a standard Transformer, each layer $l$ receives a single $C$ -dimensional vector $x_l$ and produces a single $C$ -dimensional vector $x_{l+1}$ . The residual stream is essentially a one-lane highway: there's no mechanism for different "types" of information to flow through separate, interacting channels, and no learned gating of how information from different depths combines.

This bottleneck matters practically. In mixture-of-experts architectures like DeepSeek-V3 (Liu et al., 2024b), the computational cost of each layer is dominated by the expert FFN operations, which scale with the input dimension $C$ . To increase model capacity without proportionally increasing FLOPs, you want to decouple the information capacity of the residual stream from the computational dimension of individual layers. This is exactly what Hyper-Connections (HC) (Zhu et al., 2024) attempted: expand the residual stream to $n \times C$ dimensions while keeping layer computations at $C$ dimensions, using learned matrices $H^{\text{pre}}_l$ , $H^{\text{post}}_l$ , and $H^{\text{res}}_l$ to read from, write to, and mix within this widened stream. The idea is compelling because it adds negligible FLOPs — the additional matrix operations are tiny ( $O(n^2)$ or $O(nC)$ ) compared to the attention and FFN computations — while theoretically multiplying the information that can persist across layers.

But this is where the tension crystallizes. Standard residual connections give you stability for free (the identity map) but zero intra-stream mixing. HC gives you rich intra-stream mixing but — as the paper shows through systematic analysis — sacrifices stability in a way that becomes catastrophically worse as depth increases. The paper's Figure 3 visualizes this: the composite mapping $\prod_{i=1}^{L-l} H^{\text{res}}_{L-i}$ in HC reaches Amax Gain Magnitude peaks of approximately 3000, meaning a signal from a shallow layer can be amplified by three orders of magnitude by the time it reaches a deeper layer. The corresponding backward gradient gains are similarly explosive. This isn't a theoretical edge case — it manifests as actual training collapse. Figure 2 shows a 27B HC model experiencing a sudden loss spike at around step 12,000, correlated with a gradient norm spike that the mHC model doesn't exhibit.

Where Prior Approaches Fall Short

The paper identifies specific, well-defined failures in HC and its conceptual relatives, establishing that the problem isn't HC's specific parameterization but rather a fundamental property of unconstrained multi-stream residual connections.

HC compromises identity mapping without realizing it. The core issue is visible when you recursively expand HC across multiple layers (Equation 4):

$x_L = \left( \prod_{i=1}^{L-l} H^{\text{res}}_{L-i} \right) x_l + \sum_{i=l}^{L-1} \left( \prod_{j=1}^{L-1-i} H^{\text{res}}_{L-j} \right) H^{\text{post} \top}_i \mathcal{F}(H^{\text{pre}}_i x_i, W_i)$

Compare this to the standard residual unrolling (Equation 2). In the standard case, the identity mapping $x_l$ appears with coefficient 1 — it's preserved exactly. In HC, it appears multiplied by $\prod_{i=1}^{L-l} H^{\text{res}}_{L-i}$ , a product of $L-l$ unconstrained learned matrices. Since each $H^{\text{res}}_l$ can have arbitrary eigenvalues (potentially much larger or smaller than 1), their product can exponentially amplify or attenuate the signal. This is the mathematical root of the instability: the norm-preservation property of identity mappings is completely lost.

The paper quantifies this through two metrics introduced in Section 3.1: the Amax Gain Magnitude for forward signals (maximum absolute row sum of the composite mapping) and backward gradients (maximum absolute column sum). For an identity mapping, both should equal exactly 1. HC's single-layer mappings already deviate significantly (Figure 3a), but the composite mappings across 30+ layers (Figure 3b) explode to values of 3000 or more. This means that during training, the effective learning rate for parameters in shallow layers can vary by orders of magnitude as the composite mapping fluctuates — a recipe for instability.

The broader family of multi-stream architectures shares this vulnerability. While this paper's experiments focus specifically on HC, the instability is a general property of any architecture that replaces the identity residual connection with learned mixing across a widened stream. DenseNet (Huang et al., 2017) concatenated features but kept each path unmodified; FractalNet (Larsson et al., 2016) used multiple paths with fixed aggregation; Deep Layer Aggregation (Yu et al., 2018) merged features recursively but without learnable per-path weights. These approaches avoided the instability problem but at the cost of expressivity — they couldn't learn which features to route where based on the input.

More recent proposals like the Residual Matrix Transformer (Mak and Flanigan, 2025), which replaces the residual stream with an outer-product memory matrix, and MUDDFormer (Xiao et al., 2025), which uses multiway dynamic dense connections, face the same fundamental challenge: learnable cross-stream mixing inevitably introduces matrices whose products can diverge. The paper's contribution is not just that HC is unstable (which practitioners might have discovered empirically), but that there exists a specific, mathematically principled constraint — double stochasticity — that restores stability while preserving the ability to learn meaningful inter-stream mixing. This transforms the problem from "how do we make multi-stream connections stable?" (an open-ended optimization challenge) to "how do we project unconstrained matrices onto the Birkhoff polytope efficiently?" (a well-defined algorithmic problem).

Infrastructure overhead is an unrecognized barrier to adoption. Beyond instability, the paper identifies a practical obstacle: the widened residual stream introduces substantial memory access overhead that the original HC paper didn't address. Table 2's analysis shows that HC multiplies per-token memory I/O by a factor proportional to $n$ — approximately $(5n+1)C$ elements read and $(3n+1)C$ elements written, compared to $2C$ read and $C$ written for standard residuals. For $n=4$ , this means roughly an order of magnitude more memory traffic for the residual connection alone.

This is significant because modern Transformer training is typically memory-bandwidth-bound rather than compute-bound (the "memory wall" problem discussed in Dao et al., 2022). Adding FLOPs-efficient operations that saturate memory bandwidth can substantially increase training time even if they appear "free" on a FLOPs basis. The paper notes that HC's original design "remains unaddressed in terms of hardware efficiency" (Section 1), implying that without the kernel fusion, recomputing, and communication overlapping strategies described in Section 4.3, HC's practical training overhead would be substantially larger than the 6.7% achieved by mHC under optimization.

Additionally, the widened stream creates a communication bottleneck in pipeline parallelism. When a pipeline stage boundary cuts between layers, the $n \times C$ hidden matrix must be transmitted between devices — $n$ times more data than the standard $C$ -dimensional vector. Under standard pipeline schedules, this increased communication would expand pipeline bubbles (idle time while devices wait for data), further degrading throughput. The paper addresses this through an extended DualPipe schedule (Section 4.3.3, Figure 4), but the fact that this optimization is necessary underscores that macro-architecture designs cannot be evaluated purely on FLOPs or accuracy — they must account for systems-level implications.

How This Paper Positions Itself

The paper positions mHC as a generalization and normalization of HC, not a competitor. Several design choices signal this positioning:

First, mHC preserves HC's core mechanism — learnable mappings $H^{\text{pre}}_l$ , $H^{\text{post}}_l$ , and $H^{\text{res}}_l$ — intact, only adding a projection step after the raw mappings are computed (Equation 8). The projection is a post-hoc constraint applied to the same parameterization HC uses. This means any insight or technique developed for HC transfers to mHC; the only difference is that mHC guarantees stability where HC doesn't.

Second, the manifold constraint is explicitly chosen to be the minimal restriction that restores stability. The Birkhoff polytope of doubly stochastic matrices is the set of matrices with non-negative entries and row/column sums equal to 1. This constraint has three properties that make it "minimal" in a precise sense:

It's the smallest departure from identity. When $n=1$ , the constraint reduces to the scalar 1 — recovering exactly the standard residual connection. As $n$ grows, the constraint permits any convex combination of permutation matrices (by the Birkhoff-von Neumann theorem), which is a rich set of transformations — far more expressive than the identity matrix — but all of which are norm-non-expansive. This means mHC can still learn complex mixing patterns (as Figure 8's visualized matrices confirm) but can never amplify signals unboundedly.
Compositional closure. The product of two doubly stochastic matrices is doubly stochastic. This is not true for arbitrary matrices with bounded row sums or column sums individually. It means that the stability guarantee is depth-independent: no matter how many layers compose their $H^{\text{res}}$ matrices, the compound mapping remains doubly stochastic and therefore norm-preserving. Figure 7 confirms this empirically: the composite mapping's Amax Gain Magnitude peaks at approximately 1.6, compared to ~3000 for HC — a reduction of roughly three orders of magnitude, and the residual deviation from 1 is attributable solely to the finite Sinkhorn-Knopp iteration count ( $t_{\text{max}} = 20$ ).
Geometric interpretability. The Birkhoff polytope is the convex hull of permutation matrices. So every $H^{\text{res}}_l$ can be understood as a "soft permutation" — redistributing information across the $n$ residual streams without creating or destroying total signal energy. This connects mHC to a long line of work on permutation-equivariant architectures while operating at a different level of abstraction (mixing features across depth rather than spatial positions).

Third, by naming the approach Manifold-Constrained Hyper-Connections, the paper signals that the constraint framework is general. The Sinkhorn-Knopp projection onto the Birkhoff polytope is one instantiation; other manifolds with different properties could serve different purposes. Section 6 explicitly invites exploration of "diverse manifold constraints tailored to specific learning objectives," suggesting that mHC is intended as a proof-of-concept for constrained residual connection design rather than the final word.

The Practical Stakes

The paper's empirical context makes the stakes concrete. The 27B baseline model — a DeepSeek-V3-style MoE architecture — achieves 22.0% exact match on MATH and 59.0% accuracy on MMLU (Table 4). HC improves these to 26.4% and 63.0% respectively — substantial gains of 4.4 and 4.0 percentage points. But as Figure 2 shows, this improved model is untrainable at the full scale — it experiences loss divergence around step 12,000 that would require intervention (learning rate reduction, checkpoint rollback, or outright restart) to recover from.

mHC achieves 26.0% on MATH and 63.4% on MMLU — essentially matching or exceeding HC on every benchmark (Table 4: it beats HC on 6 of 8 tasks, ties on 1, and loses narrowly on 1 by 0.4 points on MATH) — while maintaining the training stability of the standard residual baseline. The gradient norm plot in Figure 5(b) shows mHC and the baseline tracking each other closely, while HC spikes uncontrollably. This is the paper's central empirical claim: you can have HC's performance benefits without HC's instability, and the cost is a specific, implementable projection step plus careful systems engineering.

The FLOPs-matched comparison in Figure 6 further strengthens this narrative. The compute scaling curve (Figure 6a) shows mHC's advantage over the baseline persisting from 3B through 27B parameters, with only "marginal attenuation." The token scaling curve (Figure 6b) shows the advantage emerging early in training and maintained throughout the 3B model's 1T-token run. These results suggest that the manifold constraint isn't just a training stabilization trick — it genuinely improves the model's representational capacity in a way that scales.

A Brief, Important Distinction

It's worth clarifying what mHC does not do, because the paper's framing could be misread. mHC does not "restore the identity mapping" in the sense of making $H^{\text{res}}_l = I$ . The projection onto the Birkhoff polytope produces doubly stochastic matrices, not identity matrices. What mHC restores is the stability properties of the identity mapping — norm preservation, compositional closure, bounded signal amplification — while still allowing $H^{\text{res}}_l$ to learn non-trivial mixing across streams. This is a strictly more general condition: every identity matrix is doubly stochastic, but most doubly stochastic matrices are not identity. The paper's contribution is identifying this precise middle ground between "no mixing" (standard residuals) and "unbounded mixing" (HC), and showing that it's both stable and expressive enough to be useful at scale.

3. Technical Approach

3.1 Reader Orientation

mHC is a drop-in replacement for the residual connection in any Transformer architecture that replaces the standard identity shortcut with a learnable multi-stream mixing operation constrained to lie on a specific mathematical manifold. It solves the problem of how to enrich residual connections with learned inter-stream communication (which improves performance) without sacrificing the training stability that the identity mapping provides (which HC, its unconstrained predecessor, catastrophically loses at scale). The solution takes the shape of a projection: HC's learnable residual mapping matrices are computed as usual from input-dependent and static parameters, then projected onto the Birkhoff polytope of doubly stochastic matrices via the Sinkhorn-Knopp algorithm, guaranteeing that every composite product across arbitrary depth remains norm-preserving.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, operating at each Transformer layer $l$ :

Hidden state expansion: The standard $C$ -dimensional input $x_l$ is expanded into an $n \times C$ matrix $\mathbf{x}_l$ (the "multi-stream residual"), where $n$ is the expansion rate (set to 4 in all experiments). This matrix stores $n$ separate $C$ -dimensional feature vectors that can carry different types of information across layers.
Coefficient computation: Three sets of learnable coefficients — $H^{\text{pre}}_l \in \mathbb{R}^{1 \times n}$ (input mixing), $H^{\text{post}}_l \in \mathbb{R}^{1 \times n}$ (output mixing), and $\tilde{H}^{\text{res}}_l \in \mathbb{R}^{n \times n}$ (residual mixing) — are computed from the flattened hidden state via linear projections, gating factors, and static biases, producing raw unconstrained matrices.
Manifold projection: The raw residual mapping $\tilde{H}^{\text{res}}_l$ is projected onto the Birkhoff polytope via Sinkhorn-Knopp iteration (20 steps), producing $H^{\text{res}}_l = \text{Sinkhorn-Knopp}(\tilde{H}^{\text{res}}_l)$ , a doubly stochastic matrix. The input/output mappings $H^{\text{pre}}_l, H^{\text{post}}_l$ receive sigmoid projections enforcing non-negativity.
Layer computation with stream mixing: The $n$ streams are aggregated into a single $C$ -dimensional input via $H^{\text{pre}}_l \mathbf{x}_l$ , processed by the standard layer function $\mathcal{F}$ (attention or FFN), then written back into the residual stream via $H^{\text{post}}_l$ while the existing streams are mixed via $H^{\text{res}}_l \mathbf{x}_l$ .
Infrastructure optimization layer: Kernel fusion, selective recomputing, and extended DualPipe scheduling amortize the memory access and communication overhead of the widened stream, achieving only 6.7% training time overhead at $n=4$ .

Information flows: the $n \times C$ hidden matrix enters layer $l$ → flattened and normalized → projected to produce raw coefficients → raw $H^{\text{res}}_l$ projected to doubly stochastic, $H^{\text{pre}}_l, H^{\text{post}}_l$ sigmoided → streams aggregated to $C$ -dim via $H^{\text{pre}}_l$ → layer function $\mathcal{F}$ processes it → output written back via $H^{\text{post}}_l$ while $H^{\text{res}}_l$ mixes existing streams → $n \times C$ hidden matrix exits to layer $l+1$ .

3.3 Roadmap for the Deep Dive

First, the HC baseline formulation (Equation 3 and its recursive expansion to Equation 4), because mHC inherits HC's parameterization and architecture entirely — understanding what mHC constrains requires understanding what HC does unconstrained.
Second, the coefficient computation (Equations 5 and 7), because mHC's projection step operates on the raw coefficients produced by the same gating and linear projection machinery as HC, with one small modification (flattening the hidden matrix to preserve full context).
Third, the manifold constraint and Sinkhorn-Knopp projection (Equations 6, 8, 9), because this is the core novel mechanism: what the Birkhoff polytope is, why double stochasticity provides the stability guarantees it does, and how the iterative algorithm computes the projection.
Fourth, the stability analysis framework (the Amax Gain Magnitude metric and Figures 3, 7), because this is how the paper quantifies and diagnoses the instability problem mHC solves, and understanding the metric makes the empirical comparison between HC and mHC meaningful.
Fifth, the infrastructure optimizations (kernel fusion, recomputing, DualPipe extension), because without these, mHC's theoretical elegance would be undermined by practical training slowdowns — this section explains how the 6.7% overhead figure is achieved and why each optimization is necessary.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a stabilization and infrastructure engineering paper whose core idea is that unconstrained learned residual mixing matrices cause unbounded signal amplification across depth (because their products have uncontrolled singular values), and that projecting these matrices onto the Birkhoff polytope of doubly stochastic matrices — which are closed under multiplication and have spectral norm $\leq 1$ — eliminates this instability while preserving the expressivity benefits of multi-stream residual connections.

Hyper-Connections (HC) as the Starting Point

The paper builds directly on Hyper-Connections (Zhu et al., 2024), so a precise understanding of HC's formulation is prerequisite to understanding what mHC modifies and why.

HC Single-Layer Propagation

At each layer $l$ , HC expands the standard $C$ -dimensional residual stream into an $n$ -stream residual: the input to layer $l$ is a hidden matrix $\mathbf{x}_l \in \mathbb{R}^{n \times C}$ , constructed by stacking $n$ copies of the $C$ -dimensional input (or, equivalently, by maintaining $n$ parallel feature vectors of dimension $C$ that evolve jointly across layers). The single-layer propagation is:

$\mathbf{x}_{l+1} = H^{\text{res}}_l \mathbf{x}_l + H^{\text{post} \top}_l \mathcal{F}(H^{\text{pre}}_l \mathbf{x}_l, W_l)$

where $\mathbf{x}_l, \mathbf{x}_{l+1} \in \mathbb{R}^{n \times C}$ are the input and output hidden matrices, $H^{\text{res}}_l \in \mathbb{R}^{n \times n}$ is the residual mixing matrix, $H^{\text{pre}}_l \in \mathbb{R}^{1 \times n}$ is the input aggregation vector (mapping from $n \times C$ down to $1 \times C$ ), $H^{\text{post}}_l \in \mathbb{R}^{1 \times n}$ is the output expansion vector (mapping the $1 \times C$ layer output back to $n \times C$ ), $\mathcal{F}$ is the standard layer function (attention or FFN), and $W_l$ are the layer's parameters.

What it computes: The first term, $H^{\text{res}}_l \mathbf{x}_l$ , mixes information across the $n$ residual streams before adding the layer's contribution. The second term computes the layer's output on an aggregated input: $H^{\text{pre}}_l \mathbf{x}_l$ collapses the $n$ streams into a single $C$ -dimensional vector (via a learned weighted sum), $\mathcal{F}$ processes this vector normally, and $H^{\text{post} \top}_l$ expands the scalar output back into contributions to each of the $n$ streams (via a learned distribution). The result $\mathbf{x}_{l+1}$ has the same shape $\mathbb{R}^{n \times C}$ as $\mathbf{x}_l$ , making the operation recurrent.

Why this form: The standard residual connection $x_{l+1} = x_l + \mathcal{F}(x_l, W_l)$ is a special case when $n=1$ (so $H^{\text{pre}}_l$ , $H^{\text{post}}_l$ , and $H^{\text{res}}_l$ all degenerate to the scalar 1). For $n > 1$ , this formulation decouples the information capacity of the residual stream (which now holds $n \times C$ elements) from the computational dimension of the layer function $\mathcal{F}$ (which still operates on $C$ -dimensional vectors). The additional FLOPs are $O(nC + n^2)$ , which is negligible when $n \ll C$ (typical values: $n=4$ , $C=1280$ – $2560$ ). The learnable matrices $H^{\text{pre}}_l$ , $H^{\text{post}}_l$ , and $H^{\text{res}}_l$ allow the model to learn input-dependent routing: which streams to read from, which streams to write to, and how to mix information among streams.

HC Recursive Expansion Reveals the Instability

Recursively applying the single-layer propagation from a shallow layer $l$ to a deep layer $L$ yields Equation 4:

$\mathbf{x}_L = \left( \prod_{i=1}^{L-l} H^{\text{res}}_{L-i} \right) \mathbf{x}_l + \sum_{i=l}^{L-1} \left( \prod_{j=1}^{L-1-i} H^{\text{res}}_{L-j} \right) H^{\text{post} \top}_i \mathcal{F}(H^{\text{pre}}_i \mathbf{x}_i, W_i)$

where $\prod$ denotes left-multiplication in order of increasing layer index.

What it computes: The signal $\mathbf{x}_l$ from layer $l$ reaches $\mathbf{x}_L$ multiplied by the composite matrix $\prod_{i=1}^{L-l} H^{\text{res}}_{L-i}$ , which is the product of $L-l$ unconstrained $n \times n$ matrices. Each intermediate layer's contribution is similarly multiplied by the composite of all subsequent $H^{\text{res}}$ matrices.

Why this causes instability: In the standard residual case (Equation 2), the composite coefficient of $\mathbf{x}_l$ is the identity matrix $I$ , with all eigenvalues exactly 1. The signal is preserved with zero amplification or attenuation regardless of depth. In HC, each $H^{\text{res}}_l$ is an arbitrary $n \times n$ matrix whose singular values can be anywhere in $[0, \infty)$ . The product of $L-l$ such matrices can have singular values that grow or shrink exponentially with depth — a standard result from random matrix theory: if the matrices have expected squared singular values not exactly equal to 1, the product's norm diverges or vanishes exponentially. This means that during training, small changes in shallow-layer parameters get amplified by factors of hundreds or thousands by the time they affect deep-layer representations, causing gradient norms to spike and loss to diverge (as observed in Figure 2 around step 12,000).

Coefficient Computation in HC and mHC

Both HC and mHC compute their coefficients from two components: input-dependent (dynamic) mappings and input-independent (static) mappings. The difference is subtle but important for understanding why mHC's stability guarantee holds regardless of the input.

HC Coefficient Computation (Equation 5)

HC computes the three mapping vectors/matrices as:

$\begin{cases} \tilde{\mathbf{x}}_l = \text{RMSNorm}(\mathbf{x}_l) \\ H^{\text{pre}}_l = \alpha^{\text{pre}}_l \cdot \tanh(\theta^{\text{pre}}_l \tilde{\mathbf{x}}^\top_l) + b^{\text{pre}}_l \\ H^{\text{post}}_l = \alpha^{\text{post}}_l \cdot \tanh(\theta^{\text{post}}_l \tilde{\mathbf{x}}^\top_l) + b^{\text{post}}_l \\ H^{\text{res}}_l = \alpha^{\text{res}}_l \cdot \tanh(\theta^{\text{res}}_l \tilde{\mathbf{x}}^\top_l) + b^{\text{res}}_l \end{cases}$

where $\tilde{\mathbf{x}}_l \in \mathbb{R}^{n \times C}$ is the RMSNorm-normalized hidden matrix (normalization applied along the last dimension, i.e., independently for each of the $n$ streams), $\alpha^{\text{pre}}_l, \alpha^{\text{post}}_l, \alpha^{\text{res}}_l \in \mathbb{R}$ are learnable scalar gating factors initialized to small values (0.01), $\theta^{\text{pre}}_l \in \mathbb{R}^{1 \times C}$ , $\theta^{\text{post}}_l \in \mathbb{R}^{1 \times C}$ , and $\theta^{\text{res}}_l \in \mathbb{R}^{n \times C}$ are linear projection parameters for the dynamic (input-dependent) part, $b^{\text{pre}}_l \in \mathbb{R}^{1 \times n}$ , $b^{\text{post}}_l \in \mathbb{R}^{1 \times n}$ , and $b^{\text{res}}_l \in \mathbb{R}^{n \times n}$ are the static biases, and $\tanh$ is applied element-wise.

What it computes: For each mapping, the raw value is a sum of a dynamic term (the $\tanh$ of a linear projection of the normalized input, scaled by a learnable gate) and a static term (the learnable bias). The $\tanh$ bounds the dynamic contribution to $[-1, 1]$ , providing bounded input sensitivity. The gating factors $\alpha$ are initialized to 0.01, meaning that at initialization the mappings are dominated by their static biases — the model starts close to a fixed mixing pattern and gradually learns to make it input-dependent. The static bias $b^{\text{res}}_l$ is initialized as an identity-like matrix ( $b^{\text{res}}_l \approx I$ , though the exact initialization is not specified), so at initialization the residual mapping approximates the standard identity shortcut.

Why this form: Separating dynamic and static components allows the model to learn both a default routing pattern (static bias) and input-dependent modulation (the $\tanh$ term). The small initialization of $\alpha$ ensures that training starts near the stable identity-mapping regime, but as training progresses and $\alpha$ grows, the mappings can become strongly input-dependent — which is when instability risks emerge if $H^{\text{res}}_l$ is unconstrained. The RMSNorm ensures the input to the linear projections has consistent scale regardless of the residual stream's absolute magnitude.

mHC Coefficient Computation (Equation 7): The Flattening Modification

mHC introduces a single modification to the coefficient computation: instead of applying RMSNorm to the $n \times C$ matrix $\mathbf{x}_l$ along the last dimension (which normalizes each stream independently), mHC flattens $\mathbf{x}_l$ into a vector $\vec{\mathbf{x}}_l = \text{vec}(\mathbf{x}_l) \in \mathbb{R}^{1 \times nC}$ , normalizes that vector, and uses the flattened representation for all linear projections:

$\begin{cases} \vec{\mathbf{x}}'_l = \text{RMSNorm}(\vec{\mathbf{x}}_l) \\ \tilde{H}^{\text{pre}}_l = \alpha^{\text{pre}}_l \cdot (\vec{\mathbf{x}}'_l \varphi^{\text{pre}}_l) + b^{\text{pre}}_l \\ \tilde{H}^{\text{post}}_l = \alpha^{\text{post}}_l \cdot (\vec{\mathbf{x}}'_l \varphi^{\text{post}}_l) + b^{\text{post}}_l \\ \tilde{H}^{\text{res}}_l = \alpha^{\text{res}}_l \cdot \text{mat}(\vec{\mathbf{x}}'_l \varphi^{\text{res}}_l) + b^{\text{res}}_l \end{cases}$

where $\vec{\mathbf{x}}'_l \in \mathbb{R}^{1 \times nC}$ is the flattened and normalized hidden state, $\varphi^{\text{pre}}_l \in \mathbb{R}^{nC \times n}$ and $\varphi^{\text{post}}_l \in \mathbb{R}^{nC \times n}$ are linear projection matrices for the dynamic pre/post mappings, $\varphi^{\text{res}}_l \in \mathbb{R}^{nC \times n^2}$ is the linear projection for the dynamic residual mapping, $\text{mat}(\cdot)$ reshapes the $1 \times n^2$ projection output into an $n \times n$ matrix, and $\alpha$ , $b$ are as in HC (but note: the $\tanh$ activation is removed — this is significant and discussed below).

What it computes: Identical in spirit to HC but with one crucial difference: the linear projections for computing $H^{\text{res}}_l$ now see the entire $nC$ -dimensional flattened state rather than the $C$ -dimensional per-stream states independently. This means that when computing how stream $i$ should mix into stream $j$ , the projection has access to the values of all streams simultaneously, not just stream $i$ . This enables cross-stream conditioning in the dynamic residual mapping computation.

Why flattening and why remove $\tanh$ : The flattening is motivated by the need for "full context information" (Section 4.2) when computing the residual mixing matrix — since $H^{\text{res}}_l$ governs how all streams interact, its computation should see all streams. The removal of $\tanh$ is not explicitly justified in the paper but is a design choice that follows from the Sinkhorn-Knopp projection: since the projection will enforce non-negativity (via exponentiation in the first Sinkhorn-Knopp step, Equation 9), there is no need for $\tanh$ to bound the raw coefficients — the projection itself handles boundedness. Removing $\tanh$ also eliminates the vanishing gradient problem that $\tanh$ saturation can cause, though the paper does not discuss this tradeoff.

The Manifold Constraint: Doubly Stochastic Matrices and the Birkhoff Polytope

This is the core novel mechanism. The key insight is that the instability in HC arises because the spectral norm of $H^{\text{res}}_l$ is unbounded, and that constraining $H^{\text{res}}_l$ to be doubly stochastic — a specific class of matrices with non-negative entries where every row and every column sums to exactly 1 — solves this problem while still permitting rich inter-stream mixing.

Definition and Geometric Intuition (Equation 6)

The manifold $\mathcal{M}_{\text{res}}$ of doubly stochastic matrices is defined as:

$\mathcal{M}_{\text{res}} = \left\{ H \in \mathbb{R}^{n \times n} \;|\; H \mathbf{1}_n = \mathbf{1}_n,\; \mathbf{1}^\top_n H = \mathbf{1}^\top_n,\; H \geq 0 \right\}$

where $\mathbf{1}_n$ is the $n$ -dimensional column vector of all ones.

What it computes: This is a set definition, not an operation. A matrix $H$ is in $\mathcal{M}_{\text{res}}$ if and only if: (1) every entry is non-negative, (2) the sum of each row equals 1 (so applying $H$ to a vector produces a weighted average of the vector's components), and (3) the sum of each column equals 1 (so the transpose $H^\top$ is also row-stochastic). The set $\mathcal{M}_{\text{res}}$ is geometrically a convex polytope — the Birkhoff polytope — whose vertices are exactly the $n \times n$ permutation matrices (by the Birkhoff-von Neumann theorem).

Why this form — three critical properties:

Norm Preservation (non-expansiveness): The spectral norm (largest singular value) of any doubly stochastic matrix satisfies $\|H\|_2 \leq 1$ , with equality if and only if $H$ is a permutation matrix. This follows from the Perron-Frobenius theorem: the dominant eigenvalue of a row-stochastic matrix is 1, and for a doubly stochastic matrix, the corresponding left and right eigenvectors are both $\mathbf{1}_n$ , giving singular value exactly 1 for the all-ones direction. All other singular values are $\leq 1$ . This means that applying $H^{\text{res}}_l$ to the hidden matrix $\mathbf{x}_l$ (operating on each of the $C$ feature dimensions independently) can never amplify the Frobenius norm — it performs at most a rotation/projection, never an expansion. During backpropagation, the gradient through $H^{\text{res}}_l$ similarly cannot be amplified, preventing gradient explosion.
Compositional Closure: The set of doubly stochastic matrices is closed under matrix multiplication: if $A$ and $B$ are doubly stochastic, then $AB$ is doubly stochastic. This is easily verified: $(AB)\mathbf{1}_n = A(B\mathbf{1}_n) = A\mathbf{1}_n = \mathbf{1}_n$ , and similarly for the column sum. This means that the composite mapping $\prod_{i=1}^{L-l} H^{\text{res}}_{L-i}$ across arbitrary depth remains doubly stochastic, inheriting all the norm-preservation properties of the individual matrices. This is the property that makes the stability guarantee depth-independent — unlike HC, where the composite product can diverge exponentially, mHC's composite product stays within the Birkhoff polytope.
Convex Hull of Permutations: The Birkhoff polytope is the convex hull of the $n!$ permutation matrices. This means every doubly stochastic matrix can be expressed as a convex combination $H = \sum_{k} \lambda_k P_k$ where $P_k$ are permutation matrices and $\lambda_k \geq 0$ , $\sum_k \lambda_k = 1$ . Operationally, $H^{\text{res}}_l \mathbf{x}_l$ acts as a "soft permutation" of the $n$ streams — redistributing information across streams without creating or destroying total signal mass. When a permutation matrix is applied repeatedly, it cycles through the streams; when a convex combination of permutations is applied repeatedly, the streams gradually mix toward a uniform distribution (the stationary distribution of the Markov chain defined by the doubly stochastic matrix). This provides a natural information fusion mechanism: over many layers, the streams converge toward consensus, but the rate and pattern of mixing is learned.

Why Not Just Constrain the Spectral Norm?

A natural alternative would be to directly constrain $\|H^{\text{res}}_l\|_2 \leq 1$ via spectral normalization or orthogonal parameterization. The paper does not discuss this alternative explicitly, but the properties above suggest why double stochasticity is preferable: (1) spectral norm constraint alone does not guarantee compositional closure — the product of two matrices each with spectral norm $\leq 1$ can still have products with spectral norm $> 1$ for specific input directions (the singular vectors do not necessarily align); (2) orthogonality ( $H^\top H = I$ ) would provide closure but is too restrictive — it forces the matrix to be a rigid rotation/reflection, preventing the kind of asymmetric mixing (e.g., routing more information to some streams than others) that gives HC its expressivity; (3) double stochasticity sits in a "Goldilocks zone" — strict enough to guarantee global stability (via the three properties above), loose enough to allow a rich family of learned mixing patterns (any convex combination of permutations, which at $n=4$ spans a 9-dimensional polytope).

The Sinkhorn-Knopp Projection (Equations 8 and 9)

Once the raw unconstrained matrix $\tilde{H}^{\text{res}}_l$ is computed via Equation 7, mHC projects it onto the Birkhoff polytope using the Sinkhorn-Knopp algorithm. The full projection pipeline for all three mappings is:

$\begin{cases} H^{\text{pre}}_l = \sigma(\tilde{H}^{\text{pre}}_l) \\ H^{\text{post}}_l = 2\sigma(\tilde{H}^{\text{post}}_l) \\ H^{\text{res}}_l = \text{Sinkhorn-Knopp}(\tilde{H}^{\text{res}}_l) \end{cases}$

where $\sigma(\cdot)$ is the element-wise sigmoid function.

Sigmoid for Pre and Post Mappings

For the input and output mappings $H^{\text{pre}}_l$ and $H^{\text{post}}_l$ , the paper enforces non-negativity via sigmoid: $\sigma(x) = 1/(1+e^{-x})$ maps any real value to $(0, 1)$ . The factor of 2 in $H^{\text{post}}_l = 2\sigma(\tilde{H}^{\text{post}}_l)$ expands the range to $(0, 2)$ , though the paper does not explain this asymmetry — it may reflect a design choice to allow output mappings to amplify signals (since $H^{\text{post}}_l$ controls how much each stream receives from the layer output, and the total write-in energy need not sum to 1).

Why non-negativity but not double stochasticity for pre/post: The pre and post mappings are vectors (not matrices), so the concept of double stochasticity doesn't apply. Non-negativity alone prevents signal cancellation — if $H^{\text{pre}}_l$ had both positive and negative entries, streams with opposite signs could cancel each other in the weighted sum, causing destructive interference. This is a weaker constraint than double stochasticity: it prevents cancellation but doesn't bound amplification.

Sinkhorn-Knopp for Residual Mapping

The Sinkhorn-Knopp algorithm takes a square matrix with all positive entries and iteratively normalizes its rows and columns to sum to 1, converging to a doubly stochastic matrix. Since $\tilde{H}^{\text{res}}_l$ may have negative entries (it's the raw unconstrained output of a linear projection), the first step exponentiates to ensure positivity:

$M^{(0)} = \exp(\tilde{H}^{\text{res}}_l)$

where $\exp$ is applied element-wise. Then for $t = 1, 2, \ldots, t_{\text{max}}$ :

$M^{(t)} = T_r\left(T_c(M^{(t-1)})\right)$

where $T_c$ normalizes each column to sum to 1 (dividing each element by its column sum) and $T_r$ normalizes each row to sum to 1 (dividing each element by its row sum). The paper sets $t_{\text{max}} = 20$ as "a practical value."

$H^{\text{res}}_l = M^{(t_{\text{max}})}$

What it computes sequentially: Starting from the exponentiated raw matrix $M^{(0)}$ (which is strictly positive), the algorithm alternates between making the matrix column-stochastic and row-stochastic. After one full iteration (column then row normalization), the matrix is row-stochastic but not necessarily column-stochastic (the column normalization was undone by the subsequent row normalization). As $t \to \infty$ , the process converges to a matrix that is simultaneously row- and column-stochastic — a doubly stochastic matrix. At $t_{\text{max}} = 20$ , the result is an approximation; Figure 7(a) shows that the backward gradient gain (column sums) deviates slightly from the ideal value of 1, indicating that 20 iterations leave a small residual.

Why exponentiate first: The Sinkhorn-Knopp algorithm requires a positive input matrix (all entries > 0) to guarantee convergence to a unique doubly stochastic matrix. The raw $\tilde{H}^{\text{res}}_l$ from the linear projection can have arbitrary real values. Exponentiating maps $\mathbb{R} \to (0, \infty)$ , ensuring positivity while preserving the relative ordering of entries (since $\exp$ is monotonic). This also creates an implicit "soft" non-negativity constraint: entries that were very negative in $\tilde{H}^{\text{res}}_l$ become near-zero in $M^{(0)}$ , effectively allowing the model to learn sparse mixing patterns (certain stream-to-stream connections can be nearly zero after Sinkhorn-Knopp if the corresponding raw entry is very negative).

Why this algorithm rather than direct optimization: The Sinkhorn-Knopp algorithm is differentiable with respect to its input (the backward pass can be implemented by unrolling the iterations or via implicit differentiation), making it suitable for end-to-end training. Directly parameterizing doubly stochastic matrices is challenging — they form an $(n-1)^2$ -dimensional subset of $\mathbb{R}^{n^2}$ with no simple unconstrained parameterization. The Sinkhorn-Knopp approach provides a differentiable map from unconstrained $\mathbb{R}^{n \times n}$ onto (approximately) the Birkhoff polytope, allowing the model to learn representations in the unconstrained space while operating in the constrained space. The choice of $t_{\text{max}} = 20$ balances approximation quality (closer to true doubly stochasticity) against computational cost — each iteration requires $O(n^2)$ operations for the row/column sums, so 20 iterations add $20 \times 2 \times n^2$ operations per layer.

Why 20 iterations is sufficient: Figure 7(a) shows that with 20 iterations, the single-layer backward gradient gain (column sum) reaches approximately 1.0–1.5, compared to values of 0–15 for HC (Figure 3a). The composite mapping in Figure 7(b) peaks at approximately 1.6 — a three-order-of-magnitude reduction from HC's peak of ~3000. While more iterations would bring this closer to exactly 1, the paper's results suggest 20 is sufficient for practical stability. The remaining small deviation may actually be beneficial, as it provides a slight flexibility in signal scaling that purely identity-preserving connections would not — the authors do not discuss this.

The Stability Analysis Framework: Amax Gain Magnitude

To quantify the instability and validate mHC's stabilization, the paper introduces the Amax Gain Magnitude metric (Section 3.1 and Figures 3, 7). This is not a training hyperparameter but an analytical tool for diagnosing signal propagation through the residual mappings.

Definition

For the forward pass, the Amax Gain Magnitude of a mapping matrix $M$ is the maximum absolute row sum:

$\text{Amax Gain}_{\text{forward}}(M) = \max_i \left| \sum_{j=1}^n M_{ij} \right|$

For the backward pass, it is the maximum absolute column sum:

$\text{Amax Gain}_{\text{backward}}(M) = \max_j \left| \sum_{i=1}^n M_{ij} \right|$

These are computed for both single-layer mappings ( $M = H^{\text{res}}_l$ or $\mathcal{P}_{\mathcal{M}_{\text{res}}}(H^{\text{res}}_l)$ ) and composite mappings ( $M = \prod_{i=1}^{L-l} H^{\text{res}}_{L-i}$ or its mHC equivalent), averaged over all tokens in a selected sequence.

What it computes: For a single layer's forward propagation, the row sum $\sum_j M_{ij}$ is the total gain applied to stream $i$ 's signal from all input streams. If this sum is exactly 1, the signal contribution from the residual path is a convex combination of the input streams — preserving the average magnitude. If it is 3000, the signal is amplified by three orders of magnitude. The max over rows captures the worst-case amplification across all streams. For the backward pass, the column sum captures how much gradient flows back from output stream $j$ to all input streams — the worst-case column sum governs potential gradient explosion. For the composite mapping, the same logic applies but across all $L-l$ layers: the product's row sums tell you how much a signal from layer $l$ is amplified by the time it reaches layer $L$ , summed over all intermediate mixing paths.

Why this metric rather than the spectral norm: The spectral norm $\|M\|_2$ captures the worst-case amplification over all unit-norm inputs, but it doesn't decompose into forward and backward properties separately. The row sum metric specifically measures what happens to the residual signal (the $\mathbf{x}_l$ term in Equation 4), which is the component that should benefit from identity-mapping preservation. A row sum near 1 means each output stream receives approximately one "unit" of total input signal, averaged across input streams — the key stability property. The spectral norm could be near 1 even if row sums deviate, as long as the deviations are orthogonal to the dominant singular vector, but for stability of the specific residual pathway, row sum control is more directly relevant.

Applying the Metric to Diagnose HC Instability (Figure 3)

Figure 3(a) plots the single-layer forward and backward Amax Gain Magnitude for each of the 60 layers (30 Transformer blocks, each split into attention and FFN sub-layers) in the 27B HC model. The forward gain varies between roughly 1 and 15 across layers, and the backward gain similarly. Some layers have gains near 1 (well-behaved), others have gains of 10–15 (moderately amplifying). Figure 3(b) shows the composite mapping from layer $l$ to layer 60: the forward gain (for $\prod_{i=1}^{61-l} H^{\text{res}}_{61-i}$ ) reaches values of ~3000 at intermediate depths, and the backward gain similarly explodes. This means that a unit-norm signal at layer 30 will be amplified to norm ~3000 by layer 60 — a catastrophic expansion.

Applying the Metric to Validate mHC Stabilization (Figure 7)

Figure 7(a) shows mHC's single-layer gains: the forward gain sits at approximately 1.0 for all layers (the doubly stochastic constraint ensures exact row sums of 1), while the backward gain (column sums) deviates slightly from 1.0, reaching up to about 1.6 at some layers. This deviation is the residual error from using finite Sinkhorn-Knopp iterations ( $t_{\text{max}} = 20$ rather than $\infty$ ). Figure 7(b) shows the composite mapping: the backward gain accumulates the iteration error, peaking at approximately 1.6 for the deepest composite paths. Crucially, this is bounded — it does not grow with depth beyond this small constant, because the doubly stochastic approximation error per layer is small and does not compound multiplicatively (the algorithm ensures near-stochasticity at each layer independently). The three-order-of-magnitude reduction (3000 → 1.6) is the quantitative evidence that the manifold constraint achieves its stability goal.

Visualization of Learned Mappings (Figure 8)

Figure 8 provides qualitative evidence that mHC's learned mappings are meaningfully different from HC's while being stable. The figure shows representative $H^{\text{res}}$ matrices and composite products at three depths (layers 1, 30, 60) for both methods, averaged over all tokens in one sequence.

HC matrices: The single-layer $H^{\text{res}}_1$ shows mixed positive and negative entries ranging from roughly -16 to +19, with row sums far from 1 (annotated along the y-axis) and column sums similarly erratic. By layer 60, entries reach magnitudes of 6–22, with a clear diagonal structure but substantial off-diagonal "noise." The composite $\prod_{i=1}^{30} H^{\text{res}}_{31-i}$ shows entries of $\pm 135$ to $\pm 500$ , with row sums that are wildly inconsistent — confirming the explosive amplification. The $\prod_{i=1}^{60} H^{\text{res}}_{61-i}$ composite is even more extreme, with entries of -500 to +500 and row sums ranging from -475 to +509. One row of this composite maps a unit input to an output magnitude of 509 — direct evidence of the 3000× Amax Gain Magnitude reported quantitatively.

mHC matrices: The projected single-layer $\mathcal{P}_{\mathcal{M}_{\text{res}}}(H^{\text{res}}_1)$ shows all entries between 0 and 1, with row sums exactly 1.0 (annotated) and column sums very close to 1.0. The matrix is not identity — it has learned an off-diagonal structure where certain streams route heavily to others (e.g., row 1 has entries 0.83, 0.73, 0.66, 0.75, indicating a preference for the first stream). By layer 30, the matrix shows more diffuse mixing, and by layer 60 the structure is different again — the model learns different mixing patterns at different depths. The composite matrices show the expected behavior: as more layers compose, the matrices become increasingly uniform (all entries approaching $1/n = 0.25$ ), representing progressive mixing toward consensus. The composite row sums remain exactly 1.0, confirming the closure property.

What this reveals about mHC's expressivity: Despite the strict constraint, mHC learns non-trivial, depth-dependent mixing patterns. The early-layer matrices often show concentrated routing (one stream dominates), while deeper-layer matrices show more distributed mixing. This is behavior that the identity mapping (all matrices = $I$ ) cannot produce but that HC's unconstrained matrices can — except mHC produces it without the instability. The convergence toward uniformity in the composite matrices is the expected mathematical behavior of repeated doubly stochastic multiplication (it's the convergence to the stationary distribution of the Markov chain), which acts as a form of progressive feature fusion: related information across streams is gradually merged as depth increases, similar in spirit to how attention heads pool information but operating across the residual stream dimension rather than the sequence dimension.

Why the Manifold Constraint Works: A Stability Proof Sketch

While the paper does not provide a formal proof, the theoretical argument is implicit in the properties of doubly stochastic matrices and can be reconstructed:

Claim: For any depth $d = L - l$ , the spectral norm of the composite mapping satisfies $\|\prod_{i=1}^d H^{\text{res}}_{L-i}\|_2 \leq 1$ .

Argument: Each $H^{\text{res}}_l$ is doubly stochastic, so $\|H^{\text{res}}_l\|_2 \leq 1$ (property of doubly stochastic matrices). By submultiplicativity of the spectral norm, $\|\prod_{i=1}^d H^{\text{res}}_{L-i}\|_2 \leq \prod_{i=1}^d \|H^{\text{res}}_{L-i}\|_2 \leq 1^d = 1$ . This bound is tight only for permutation matrices; for "interior" doubly stochastic matrices (convex combinations of permutations), the spectral norm is strictly less than 1, meaning the composite actually contracts toward the uniform distribution over depth — the gradual mixing observed in Figure 8.

Why this prevents gradient explosion: During backpropagation, the gradient through the residual pathway from layer $L$ back to layer $l$ passes through the transpose of the composite mapping (or the composite mapping in reverse order). Since the transpose of a doubly stochastic matrix is also doubly stochastic, the same bound applies: gradient norms cannot be amplified beyond their original magnitude. Combined with the additive term in the backward pass (analogous to the forward case), the total gradient at layer $l$ is the sum of the direct gradient (through the layer function $\mathcal{F}$ ) and the residual gradient (through the identity mapping). The residual gradient is bounded by the direct gradient, preventing the divergence seen in HC.

Why this doesn't hurt expressivity: The bound $\|H^{\text{res}}_l\|_2 \leq 1$ limits amplification but not redistribution. A doubly stochastic matrix can still arbitrarily reroute information across streams — stream 1 can send 99% of its signal to stream 3 and 1% to stream 4, as long as the total sums to 1. This is sufficient for learning complex routing patterns. What it prevents is the pathological case where a small perturbation in stream 1 gets multiplied by a factor of 50 and dominates all other signals — exactly the instability HC exhibits.

Infrastructure Optimizations

The paper's Section 4.3 describes three categories of systems-level optimization that reduce mHC's runtime overhead to 6.7% at $n=4$ . These are not architectural contributions per se but are essential for mHC to be practically deployable at scale.

Kernel Fusion (Section 4.3.1)

The naive implementation of mHC involves many small memory-bound operations (RMSNorm, linear projections, sigmoid, Sinkhorn-Knopp, and the application of the three mappings to the hidden matrix), each of which loads and stores the $n \times C$ hidden matrix from GPU memory. Since $nC$ is large (e.g., $4 \times 2560 = 10240$ for the 27B model), repeated memory access dominates latency.

RMSNorm reordering: Instead of computing $\text{RMSNorm}(\vec{\mathbf{x}}_l)$ as a standalone operation followed by the linear projection, the kernel fuses the normalization into the projection step. Specifically, the RMSNorm operation divides $\vec{\mathbf{x}}_l$ by its RMS norm $r = \|\vec{\mathbf{x}}_l\|_2 / \sqrt{nC}$ . The kernel first computes the unnormalized projection $[\tilde{\tilde{H}}^{\text{pre}}_l, \tilde{\tilde{H}}^{\text{post}}_l, \tilde{\tilde{H}}^{\text{res}}_l] = \vec{\mathbf{x}}_l \varphi_l$ (Equation 14), then separately computes the norm $r$ (Equation 15), then applies the normalization as $1/r$ times the projection plus the biases (Equation 16). This maintains mathematical equivalence while allowing the projection to use $\vec{\mathbf{x}}_l$ directly (avoiding a separate normalized buffer) and applying the normalization factor to the much smaller projection output ( $n^2 + 2n \ll nC$ elements).

Unified projection kernel: Equations 14 and 15 are fused into a single kernel that performs two scans over $\vec{\mathbf{x}}_l$ (one for the projection, one for the norm computation), using matrix multiplication units to maximize memory bandwidth utilization. The backward pass, which requires two matrix multiplications (gradient with respect to $\vec{\mathbf{x}}_l$ and with respect to $\varphi_l$ ), is similarly consolidated into a single kernel to eliminate redundant loading of $\vec{\mathbf{x}}_l$ . Both forward and backward kernels feature "a finely tuned pipeline (load, cast, compute, store) to efficiently handle mixed-precision processing" — loading data as bfloat16, casting to float32 for accumulation, and storing results as float32.

Coefficient post-processing fusion: Equations 16 through 18 (applying normalization, gating, bias, and sigmoid) operate on small tensors of size $1 \times (n^2 + 2n)$ — only 24 elements at $n=4$ . These lightweight operations are fused into a single kernel to reduce kernel launch overhead (the fixed cost of initiating a GPU kernel, which can dominate runtime for tiny operations).

Sinkhorn-Knopp kernel: The entire 20-iteration Sinkhorn-Knopp process (Equation 19) is implemented as a single kernel. For the backward pass, rather than storing all 20 intermediate matrices (which would require $20 \times n^2 \times 4$ bytes of memory), a "custom backward kernel" recomputes the intermediate results on-chip and traverses the entire iteration in reverse. This recompute-in-backward strategy trades increased computation in the backward pass for reduced memory footprint during the forward pass.

Application kernels: Two additional fused kernels apply the computed coefficients to the hidden matrix. Fpre computes $H^{\text{pre}}_l \mathbf{x}_l$ (aggregating $n$ streams into one $C$ -dimensional vector for the layer function). Fpost,res computes $H^{\text{res}}_l \mathbf{x}_l + H^{\text{post} \top}_l \mathcal{F}(\cdot, \cdot)$ , fusing the residual mixing and output write-back with the residual merge. This fusion reduces the number of elements read from $(3n+1)C$ to $(n+1)C$ and elements written from $3nC$ to $nC$ for this kernel — because the intermediate results of applying $H^{\text{res}}_l$ and $H^{\text{post}}_l$ are never stored separately but immediately summed into the output.

Implementation framework: The majority of kernels (excluding the unified projection kernel, which requires careful tuning of matrix multiplication units) are implemented using TileLang (Wang et al., 2025), a composable tiled programming model for AI systems. This choice is pragmatic: TileLang "streamlines the implementation of kernels with complex calculation process and allows us to fully utilize the memory bandwidth with minimal engineering effort" (Section 4.3.1) — a candid acknowledgment that writing highly optimized CUDA kernels for the Sinkhorn-Knopp iteration and coefficient application would be labor-intensive, and a higher-level abstraction yields sufficient performance.

Selective Recomputing (Section 4.3.2)

The $n$ -stream residual design increases the memory footprint of training because each layer's hidden matrix $\mathbf{x}_l \in \mathbb{R}^{n \times C}$ must be stored for the backward pass (to compute gradients through $H^{\text{pre}}_l$ , $H^{\text{post}}_l$ , and $H^{\text{res}}_l$ ). At $n=4$ , this is 4× more activation memory per layer for the residual stream alone.

Strategy: Instead of storing the intermediate activations of all mHC kernels, the system discards them after the forward pass and recomputes them on-the-fly in the backward pass by re-executing the mHC kernels (without the heavy layer function $\mathcal{F}$ , which has its own separate activation storage). This trades additional computation in the backward pass for reduced memory footprint in the forward pass.

Block-wise recomputation: Rather than recomputing layer-by-layer (which would require storing $\mathbf{x}_l$ for every layer), recomputation is performed in blocks of $L_r$ consecutive layers. For each block, only the input $\mathbf{x}_{l_0}$ to the first layer in the block is persistently stored. During backpropagation through the block, the mHC kernels are re-executed starting from $\mathbf{x}_{l_0}$ to regenerate all intermediate $\mathbf{x}_l$ within the block. Table 3 summarizes what is stored versus recomputed: only $\mathbf{x}_{l_0}$ (the first layer's input), $\mathcal{F}(H^{\text{pre}}_l \mathbf{x}_l, W_l)$ (the layer function's output, needed for its own backward pass), $\mathbf{x}_l$ (the hidden state, recomputed transiently), $H^{\text{pre}}_l \mathbf{x}_l$ (the aggregated input, stored transiently), and $\text{RMSNorm}(H^{\text{pre}}_l \mathbf{x}_l)$ (the normalized input, stored transiently) need to be materialized.

Optimal block size (Equation 20): The total memory footprint for the residual stream is the resident memory for the first input of each block plus the transient memory for the active block:

$\text{Memory} = nC \times \left\lceil \frac{L}{L_r} \right\rceil + (n+2)C \times L_r$

where the first term is the stored $\mathbf{x}_{l_0}$ for all $\lceil L/L_r \rceil$ blocks, and the second term is the transient memory ( $nC$ for $\mathbf{x}_l$ , $C$ for $H^{\text{pre}}_l \mathbf{x}_l$ , and $C$ for the RMSNorm output) times the block size. Minimizing this expression with respect to $L_r$ (treating it as continuous) yields:

$L^*_r = \arg \min_{L_r} \left( nC \cdot \frac{L}{L_r} + (n+2)C \cdot L_r \right) \approx \sqrt{\frac{nL}{n+2}}$

What this means in practice: For the 27B model with $n=4$ , $L=30$ Transformer blocks (60 sub-layers), $L^*_r \approx \sqrt{4 \cdot 60 / 6} \approx \sqrt{40} \approx 6.3$ sub-layers, or about 3 Transformer blocks. The paper notes that "the theoretical optimum $L^*_r$ typically aligns with the number of layers per pipeline stage," so they "choose to synchronize the recomputation boundaries with the pipeline stages." This alignment is important because pipeline stage boundaries already represent points where communication occurs — placing recomputation block boundaries at the same points avoids additional synchronization overhead.

Why not store everything (standard gradient checkpointing): Standard gradient checkpointing (also called activation recomputation) would discard and recompute the entire layer's activations, including the expensive attention and FFN intermediates. mHC's selective approach only recomputes the lightweight mHC coefficient operations, which are $O(nC)$ rather than $O(C^2)$ or $O(BC)$ (where $B$ is batch size), making the recomputation cost negligible. The heavy layer function $\mathcal{F}$ 's activations are still stored (or handled by a separate checkpointing scheme for $\mathcal{F}$ ).

Overlapping Communication in DualPipe (Section 4.3.3)

Large-scale training of MoE models requires pipeline parallelism, where different layers (or groups of layers) are placed on different devices. Data must be communicated between pipeline stages at stage boundaries — in the forward pass, sending the hidden state from the last layer of stage $s$ to the first layer of stage $s+1$ ; in the backward pass, sending gradients in the reverse direction.

The problem with mHC: The $n$ -stream hidden matrix $\mathbf{x}_l \in \mathbb{R}^{n \times C}$ is $n$ times larger than the standard $C$ -dimensional hidden vector, so communication between pipeline stages is $n$ times more expensive. Additionally, at stage boundaries, the recomputation of mHC kernels for all $L_r$ layers within the stage introduces computational overhead that could stall the pipeline if not carefully scheduled.

The DualPipe extension (Figure 4): The paper extends the DualPipe schedule (from DeepSeek-V3, Liu et al., 2024b) to overlap this communication with computation. DualPipe already overlaps the communication of expert parallelism (sending tokens to and from expert devices) with the computation of attention and MLP layers. mHC adds three modifications:

High-priority compute stream for MLP $F_{\text{post,res}}$ kernels: The $F_{\text{post,res}}$ kernel (which applies $H^{\text{res}}_l$ and $H^{\text{post}}_l$ and performs the residual merge) for MLP/FFN layers is executed on a dedicated high-priority compute stream. This prevents the communication stream from being blocked waiting for the computation to finish — the high-priority stream can preempt lower-priority work, allowing communication to proceed as soon as data is ready.
Non-persistent kernels for attention layers: Persistent kernels (which occupy the GPU's compute units for extended periods, common in optimized attention implementations) are avoided for attention layers' mHC operations, "thereby preventing extended stalls" that would delay communication. Instead, attention computation is broken into smaller chunks that can be interleaved with communication.
Decoupled recomputation from communication: The recomputation process for each pipeline stage does not depend on communication with neighboring stages because "the initial activation of each stage $\mathbf{x}_{l_0}$ is already cached locally" — it was stored during the forward pass as part of the selective recomputation strategy. This means the backward pass recomputation can proceed independently without waiting for gradient communication from deeper stages.

The resulting schedule (Figure 4): The figure shows that in each pipeline stage, the forward pass interleaves F_A (attention kernels) and F_M (MLP kernels) with DISPATCH and COMBINE (expert parallelism communication) and PP Send Recv (pipeline parallelism communication). The backward pass similarly interleaves the backward kernels with communication. The Whole Stage Recompute (B) block appears at the start of the backward pass for each stage, running on the normal compute stream while communication proceeds on the communication stream. The high-priority compute stream handles the $F_{\text{post,res}}$ backward kernels for MLP layers, allowing them to preempt attention backward computation when results are needed for communication.

Why this achieves 6.7% overhead: The combined effect of kernel fusion (reducing per-layer memory traffic by up to 70% for the $F_{\text{post,res}}$ kernel), selective recomputing (eliminating persistent storage of $nC$ elements per layer), and communication overlapping (hiding the $n$ -fold communication increase behind computation) reduces mHC's effective wall-clock overhead to only 6.7% compared to the standard residual baseline. Without these optimizations, the paper implies that HC/mHC would be substantially slower, making the architecture impractical for large-scale training despite its accuracy benefits. The 6.7% figure is the concrete claim that makes the paper's contribution actionable: it demonstrates that manifold-constrained multi-stream residuals are not just theoretically interesting but practically deployable.

Summary of Design Choices and Their Justifications

Flattening the hidden matrix before coefficient computation: Provides cross-stream context for computing $H^{\text{res}}_l$ , enabling the model to learn stream interactions that depend on the values in all streams simultaneously.
Removing $\tanh$ from mHC's coefficient computation: Obviated by the Sinkhorn-Knopp projection's exponentiation step, which already provides a smooth, monotonic mapping to positive values; removing $\tanh$ eliminates gradient saturation issues.
Doubly stochastic constraint on $H^{\text{res}}_l$ rather than spectral normalization: Guarantees depth-independent stability through compositional closure (product of doubly stochastic matrices is doubly stochastic), which spectral norm constraint alone does not provide.
Sinkhorn-Knopp with $t_{\text{max}} = 20$ rather than exact projection: Balances approximation quality against computational cost; 20 iterations reduces Amax Gain Magnitude by three orders of magnitude (3000 → 1.6), which is sufficient for practical stability.
Exponentiation in Sinkhorn-Knopp rather than ReLU/clamp for non-negativity: Provides a differentiable, monotonic mapping that preserves the relative magnitude ordering of raw coefficients, enabling gradient-based learning.
Sigmoid projection for $H^{\text{pre}}_l$ and $H^{\text{post}}_l$ rather than full doubly stochastic: These are vectors, not matrices — non-negativity prevents destructive interference between streams but does not require row/column sum constraints.
Selective recomputing of mHC kernels only (not $\mathcal{F}$ ): Minimizes memory overhead while keeping recomputation cost negligible, since mHC coefficient operations are $O(nC)$ , much cheaper than the $O(C^2)$ attention/FFN operations.
Synchronizing recomputation boundaries with pipeline stage boundaries: Eliminates the need for separate communication at recomputation block edges, since pipeline stage boundaries already handle hidden state transfer.
TileLang for most kernels: Reduces engineering effort for implementing complex operations (especially Sinkhorn-Knopp) while achieving sufficient performance; the most performance-critical kernel (unified projection) uses hand-tuned mixed-precision pipelining.

4. Key Insights and Innovations

Innovation 1: Diagnosing the Instability Problem as a Failure of Compositional Closure, Not a Failure of Individual Layers

The paper's most distinctive intellectual contribution is not the solution (the doubly stochastic constraint) but the diagnosis: the instability in Hyper-Connections is not caused by any single layer's residual matrix becoming pathological, but by the compositional accumulation of individually moderate deviations across many layers. This reframes the problem from "how do we prevent individual matrices from becoming unstable?" to "how do we ensure that the product of many matrices remains well-behaved?"

Prior work on residual network stability (He et al., 2016b) established that identity mappings matter, but the field had internalized this as a property of the single-layer formulation: if each layer is a residual block with an identity shortcut, the network is stable. HC broke this assumption by introducing learnable mixing while keeping the residual structure (a shortcut plus a learned transformation). The paper's recursive expansion of HC (Equation 4) reveals why this breaks: the $x_l$ term is no longer $I \cdot x_l$ but $(\prod H^{\text{res}}) \cdot x_l$ , and while each individual $H^{\text{res}}$ might be well-behaved at initialization (initialized near identity with small gating factors), their product is not constrained to remain near identity. This is a genuinely new failure mode — not the vanishing/exploding gradients of pre-ResNet deep networks (which residual connections solved), but an explosion introduced by the residual mixing mechanism itself.

The Amax Gain Magnitude metric introduced in Section 3.1 is the diagnostic tool that makes this insight operational. By separately measuring forward signal gain (maximum absolute row sum of the composite mapping) and backward gradient gain (maximum absolute column sum), the paper provides a quantitative decomposition of the instability that explains why it manifests as both loss spikes (forward signal explosion) and gradient spikes (backward gradient explosion). Figure 3's visualization of single-layer vs. composite gains makes the argument visually undeniable: single-layer gains of ~1–15 look manageable (and might fool a practitioner inspecting a few layers), but the composite gain of ~3000 at intermediate depths shows the exponential accumulation. This is a significant conceptual move because it shifts attention from layer-local properties (which optimizers and initializations can partially control) to path-global properties (which require architectural constraints).

The significance extends beyond HC. Any architecture that replaces the identity residual shortcut with a learned transformation (DenseFormer, MUDDFormer, Residual Matrix Transformers — Pagliardini et al., 2024; Xiao et al., 2025; Mak and Flanigan, 2025) faces the same compositional closure problem. The paper provides the conceptual framework and diagnostic toolkit for evaluating whether a proposed multi-stream architecture will be stable at scale: compute the composite mapping's norm across depth and check for exponential growth. This is a fundamental contribution, not incremental — it identifies a general principle (compositional closure is required for deep learnable residual mixing) where the field previously had only empirical heuristics.

The evidence for this diagnostic's validity is the tight correlation between the Amax Gain Magnitude explosion in Figure 3 and the actual training instability in Figure 2: the loss spike at step ~12,000 for HC aligns with the regime where composite mapping norms have grown large enough to dominate gradient signals. The mHC stabilization (Figure 7) reduces composite gains from ~3000 to ~1.6, and the training remains stable — confirming that the instability was caused by the diagnosed mechanism, not some other confounding factor.

Innovation 2: The Birkhoff Polytope as a "Minimal Sufficient Constraint" for Multi-Stream Residual Connections

The choice of doubly stochastic matrices is not arbitrary — it represents a precise characterization of the minimal constraint that simultaneously guarantees stability (through norm preservation and compositional closure) and preserves expressivity (through the rich geometry of the Birkhoff polytope). This is a conceptual contribution distinct from the projection mechanism: the paper identifies what to constrain, not just how to constrain it.

Before mHC, the field's approach to stabilizing learnable residual connections was either (a) heuristic — clip gradients, reduce learning rates, use careful initialization — which treats symptoms rather than causes, or (b) overly restrictive — enforce orthogonality ( $H^\top H = I$ ) or scalar gating ( $H = \alpha I$ ), which eliminates the cross-stream mixing that gives HC its performance advantage. The paper's insight is that the space of doubly stochastic matrices occupies a "Goldilocks zone": it is strict enough to prevent unbounded amplification (spectral norm ≤ 1), closed enough under multiplication to prevent compositional divergence (the product of doubly stochastic matrices is doubly stochastic), and flexible enough to permit rich learned mixing (any convex combination of permutation matrices, spanning an $(n-1)^2$ -dimensional polytope).

The geometric interpretation via the Birkhoff-von Neumann theorem — that every doubly stochastic matrix is a convex combination of permutations — is theoretically elegant but also practically significant. It means that each $H^{\text{res}}_l$ can be understood as a "soft permutation" of the $n$ residual streams. The model is not merely preserving signal magnitude; it is learning a redistribution policy: which streams route information to which other streams, and in what proportion. The repeated application of these soft permutations across depth produces a gradual convergence toward uniform mixing (visible in Figure 8's composite matrices), which the paper implicitly frames as a feature, not a bug — it acts as a form of progressive feature fusion, similar in spirit to how deep networks gradually abstract features, but operating on the cross-stream connectivity dimension rather than the representational dimension.

This contrasts sharply with HC's unconstrained mixing, where the matrices can have arbitrary entries (including negative values, as Figure 8 shows with entries from -16 to +22) and unbounded row sums. HC's mixing is not interpretable as any kind of redistribution — it can amplify, cancel, and invert signals arbitrarily, which gives it high expressivity (it can represent any linear transformation) but at the cost of losing all stability guarantees. mHC's manifold constraint restricts the expressivity to signal-preserving transformations only, which is a principled tradeoff: the model can still learn complex routing patterns, but it cannot learn patterns that would destabilize training.

The fact that the performance gap between mHC and HC is negligible (Table 4: mHC outperforms HC on 6 of 8 benchmarks, matches on 1, and trails narrowly on MATH by 0.4 points) is the critical evidence that this expressivity restriction is not harmful in practice. The learned mixing patterns under the doubly stochastic constraint are sufficient to capture the performance benefits of multi-stream residuals; the additional degrees of freedom that HC provides (negative entries, unbounded row sums) appear to be either unused for useful computation or actively harmful (causing instability). This is a strong empirical argument that doubly stochasticity is not just a convenient constraint but the right constraint — it preserves what matters and eliminates what doesn't.

This innovation is fundamental rather than incremental because it identifies a specific mathematical object (the Birkhoff polytope) that solves a general architectural problem (stable multi-stream residual mixing) in a way that is provably correct (compositional closure guarantees depth-independent stability) and empirically validated (training remains stable at 27B scale, Figures 5 and 7). Future architectures that explore different manifold constraints (as Section 6 invites) will build on this insight, not replace it — the conceptual framework of "identify a manifold closed under composition that preserves expressivity" is the lasting contribution.

Innovation 3: Reframing Infrastructure Overhead as an Architectural Design Constraint, Not an After-the-Fact Optimization

The paper's third distinctive insight is methodological rather than mathematical: architectural innovations that modify the residual stream must be co-designed with their systems-level implementation to be viable at scale. This is not a new idea in general (the "memory wall" problem is well-known from FlashAttention, Dao et al., 2022), but the paper applies it with unusual rigor to the macro-architecture design space, demonstrating that the choice of manifold constraint itself enables infrastructure optimizations that would be difficult under the unconstrained HC formulation.

Table 2's analysis of per-token memory access is the key diagnostic. It shows that HC multiplies memory I/O by a factor proportional to $n$ , which the paper calls "frequently overlooked in architectural design, yet it decisively impacts runtime efficiency." This is a pointed critique of how macro-architecture research is typically evaluated — on FLOPs and accuracy alone, HC looks nearly free (the additional $O(n^2 + nC)$ operations are negligible compared to attention and FFN costs), but in wall-clock time, the memory bandwidth saturation from widened residual streams can dominate. The paper's claim that mHC achieves only 6.7% overhead when optimized implicitly argues that HC's overhead would be substantially higher without similar optimization — implying that HC's original evaluation may have understated its practical cost.

What makes this more than a standard engineering contribution is that the manifold constraint enables certain optimizations. The Sinkhorn-Knopp projection operates on small $n \times n$ matrices, making it amenable to on-chip recomputation in the backward pass rather than storing intermediate results — the tradeoff of compute for memory is viable precisely because the projection is a fixed-point iteration on a compact domain. The doubly stochastic property's compositional closure means that recomputation boundaries can align with pipeline stage boundaries without introducing additional error accumulation — each stage's initial $\mathbf{x}_{l_0}$ is stored faithfully, and recomputing within-stage activations from it is exact (up to the finite iteration count). An unconstrained HC, by contrast, would have more difficulty with recomputation because the matrices' products are not bounded, making numerical error in recomputation potentially more consequential.

The selective recomputation strategy (Equation 20's derivation of optimal block size) and the DualPipe extension (Figure 4) are systems contributions, but the core insight they embody is architectural: the design of a residual connection scheme must account for how its memory and communication patterns interact with the parallelism strategy of the training system. The paper demonstrates that pipeline stage boundaries are natural recomputation boundaries, that the widened stream's communication overhead must be hidden behind computation via priority scheduling, and that kernel fusion can eliminate redundant memory traffic — but all of these optimizations are more tractable because mHC's mathematical structure (small matrix operations with bounded intermediate values) makes them so. This is a methodological innovation: it establishes a template for how future macro-architecture proposals should be evaluated and optimized, with memory I/O analysis, recomputation strategies, and communication scheduling treated as first-class design considerations rather than deferred to systems engineers.

The 6.7% overhead figure is the concrete validation: it proves that the co-design approach works. But the deeper contribution is the demonstration that architectural constraints can be chosen to facilitate systems optimization, not just to improve accuracy or stability. Future work that explores "diverse manifold constraints" (Section 6) will need to evaluate their systems implications alongside their mathematical properties, and this paper provides both the analytical framework (Table 2's I/O breakdown) and the optimization toolkit (kernel fusion, selective recomputing, communication overlapping) for doing so.

Innovation 4: A Negative Result That Redirects the Research Agenda — Empirically Demonstrating That Unconstrained Multi-Stream Residuals Cannot Scale

The paper contains a significant negative result: that HC, despite its compelling performance at smaller scales (as reported in Zhu et al., 2024), exhibits catastrophic training instability when scaled to 27B parameters with proportional data. This is not a failure of implementation or hyperparameter tuning — it is a fundamental property of the architecture, as the theoretical analysis in Section 3.1 demonstrates. The field has a tendency to publish only positive results (architectures that work), leaving practitioners to discover scaling limitations through expensive trial and error. This paper's candid documentation of HC's failure mode — the loss spike at step ~12,000 in Figure 2, the gradient norm divergence, the composite mapping explosion to ~3000× gain — is a valuable contribution in its own right.

What makes this negative result intellectually significant is that it provides a causal mechanism for the failure, not just an observation. The paper could have simply reported "HC is unstable at 27B" and proposed mHC as a fix. Instead, it traces the instability through the recursive expansion (Equation 4), quantifies it via the Amax Gain Magnitude metric (Figure 3), and verifies that the proposed fix eliminates the diagnosed mechanism (Figure 7). This transforms the negative result from an anecdote into a general lesson: any architecture that replaces the identity residual connection with unconstrained learned mixing will eventually hit an instability barrier as depth or model size increases, because the spectral norm of the composite mapping diverges exponentially with depth.

This connects to a broader pattern in deep learning architecture design: the tension between expressivity (which pushes toward unconstrained, learned operations) and trainability (which requires constraints that ensure stable gradient flow). Residual connections resolved this tension for the standard architecture by providing an unmodified information highway. HC attempted to enrich the highway at the cost of partially closing it (making it learned), and the paper shows that this tradeoff becomes untenable at scale — the instability emerges not gradually but catastrophically, with a sudden loss spike after thousands of training steps. This suggests that architectural innovations that compromise the identity mapping property have a hard scaling ceiling, not a smooth degradation.

The downstream implication is that future macro-architecture research should treat "preservation of compositional closure" as a non-negotiable design requirement for any residual mixing scheme, analogous to how "gradient flow through the identity path" is a non-negotiable requirement for residual networks. The paper's demonstration that the Birkhoff polytope satisfies this requirement while still enabling useful mixing provides a constructive proof that the requirement is not overly restrictive — it shifts the research question from "can we learn residual mixing?" (which HC answered affirmatively) to "what is the most expressive residual mixing that remains compositionally closed?" (which mHC begins to answer, but the space of possible manifolds remains largely unexplored).

The evidence for the causal diagnosis is strong but not airtight. The paper does not perform controlled experiments varying the spectral properties of $H^{\text{res}}_l$ while holding other factors constant to confirm that the composite mapping norm is the sole cause of instability. The correlation between loss spikes (Figure 2) and composite gain explosion (Figure 3) is suggestive, and the fact that constraining the matrices eliminates both simultaneously is compelling, but a rigorous ablation (e.g., comparing different constraint strengths that yield different composite norms) would strengthen the causal claim. This is a minor limitation in an otherwise well-diagnosed negative result.

Nevertheless, this innovation is fundamental: it establishes a scaling law for residual mixing architectures (instability emerges when composite mapping norms diverge) that the field did not previously recognize, and it provides a proven template (manifold constraints with compositional closure) for avoiding it. Future papers that cite this work will likely treat "is the residual mixing manifold closed under composition?" as a standard evaluation criterion, much as "does it preserve the identity mapping?" became standard after He et al. (2016b).

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use language model pretraining — the models are trained on a proprietary in-house corpus and evaluated on a diverse set of downstream benchmarks: BBH (3-shot, exact match), DROP (3-shot, F1), GSM8K (8-shot, exact match), HellaSwag (10-shot, accuracy), MATH (4-shot, exact match), MMLU (5-shot, accuracy), PIQA (0-shot, accuracy), and TriviaQA (5-shot, exact match). These benchmarks collectively span reasoning, reading comprehension, mathematical problem-solving, commonsense, factual knowledge, and multi-task language understanding — a standard evaluation suite in the LLM literature (Bisk et al., 2020; Cobbe et al., 2021; Hendrycks et al., 2020, 2021; Joshi et al., 2017; Suzgun et al., 2022; Zellers et al., 2019). The pretraining data is not a fixed benchmark corpus but scales with model size: the 3B model trains on 39.3B tokens, the 9B on 105B tokens, and the 27B on 262B tokens (Table 5), following a compute-optimal scaling regime where data is proportional to parameters (similar in spirit to Hoffmann et al., 2022). An additional 3B model trained on 1.05T tokens separately evaluates token scaling behavior.
Base model(s). All experiments use DeepSeek-V3-style Mixture-of-Experts (MoE) architectures (Liu et al., 2024b) at three scales: 3B total parameters (612M active), 9B total (1.66B active), and 27B total (4.14B active). The 27B model serves as the primary subject for system-level results (Table 4, Figures 5, 7, 8), while the 3B and 9B models populate the compute scaling curve (Figure 6a). The architecture uses Multi-Head Latent Attention (MLA; Liu et al., 2024a), Rotary Position Embedding (RoPE; Su et al., 2024), RMSNorm (Zhang and Sennrich, 2019), and loss-free load balancing for MoE (Wang et al., 2024). The choice of DeepSeek-V3-derived architectures is deliberate: these are state-of-the-art production-scale models where architectural improvements to training stability and downstream performance have immediate practical relevance. Detailed hyperparameters are in Table 5 (Appendix A.1), including layer counts (12/18/30 for 3B/9B/27B), hidden dimensions (1280/1920/2560), attention heads (16/24/32), expansion rate n=4 for both HC and mHC, Sinkhorn-Knopp t_max=20 for mHC, gating factor initialization α=0.01, sequence length 4096, and the AdamW optimizer with base learning rates decaying via a step schedule (decay at 0.8× and 0.9× of total steps, decay factors 0.316 and 0.1). Batch sizes increase with model scale: 320 for 3B, 512 for 9B, 1280 for 27B, and 2560 for the 3B 1T-token run.
Metrics. Training loss gap (absolute difference in pretraining loss relative to the baseline) is the primary training-time metric, shown across training steps (Figures 2a, 5a). Gradient norm (L2 norm of the parameter gradients) tracks training stability (Figures 2b, 5b). For downstream evaluation, each benchmark uses its standard metric as listed above — exact match for BBH, GSM8K, MATH, TriviaQA; F1 for DROP; accuracy for HellaSwag, MMLU, PIQA. For stability analysis, the Amax Gain Magnitude is computed as the maximum absolute row sum (forward signal gain) or column sum (backward gradient gain) of the residual mapping matrices, averaged over all tokens in a selected sequence (Figures 3, 7). The compute scaling curve (Figure 6a) plots relative loss ratio (mHC loss divided by baseline loss, expressed as a percentage) against total training FLOPs spanning 3B→9B→27B. The token scaling curve (Figure 6b) plots relative loss ratio against training tokens for the 3B 1T-token run. The absolute loss gap in Table 1 is the difference in final pretraining loss when specific HC components are disabled, serving as an ablation metric.
Baselines. Three configurations are compared throughout: (1) Standard Residual Connection (Baseline) — the standard pre-norm Transformer with identity residual connections as defined in Equation 1, using the same DeepSeek-V3 MoE architecture but without any multi-stream expansion. (2) Hyper-Connections (HC) — the architecture described in Equation 3 with unconstrained learnable mappings Hpre_l, Hpost_l, and Hres_l, expansion rate n=4. (3) Manifold-Constrained Hyper-Connections (mHC) — the proposed method, identical to HC in architecture but with the manifold projections applied (sigmoid for pre/post, Sinkhorn-Knopp for residual). All three share identical hyperparameters for the layer function F (attention, FFN, MoE routing), optimizer, learning rate schedule, and training data, making the residual connection architecture the only varying factor.
Generation budget / compute accounting. The paper does not use a "generation budget" metric (there is no sampling or search at inference time — all evaluations use the pretrained model's logits directly). Instead, compute is accounted for in two ways: (1) Training FLOPs for the compute scaling curve (Figure 6a), where models at 3B, 9B, and 27B are trained with data proportional to their parameters (39.3B, 105B, 262B tokens respectively), following a compute-optimal scaling paradigm. (2) Training time overhead for the infrastructure efficiency analysis (Section 4.3), where mHC with n=4 is reported to add 6.7% additional training time compared to the standard residual baseline under the optimized implementation. The paper does not report raw FLOPs numbers for the additional mHC operations, but notes they are negligible (O(nC + n²) per layer versus the O(C²) or larger costs of attention and FFN). The 6.7% wall-clock overhead accounts for all system-level factors including kernel fusion, recomputation, and communication overlap.
Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. The downstream benchmarks are standard evaluation suites with fixed test sets; the paper reports single-run results in Table 4. The training stability analyses (Figures 2, 5) show single training trajectories rather than averaged over multiple seeds. The compute scaling curves (Figure 6) show one training run per model size. The paper does not report confidence intervals, error bars, or multi-seed variance. This is a limitation of the experimental methodology — the results should be interpreted as a demonstration that mHC is stable and performant at these scales, but without statistical quantification of the reliability of the claimed gains.

Main Quantitative Results

Training Stability and Convergence

The paper's central empirical claim is that mHC eliminates HC's catastrophic training instability while maintaining improved convergence over the standard baseline. Figure 5(a) shows this directly: the 27B baseline serves as reference (loss gap of 0), HC initially shows better convergence (negative loss gap, meaning lower loss) but exhibits a sudden loss spike around step 12,000, with the gap peaking at approximately +0.012 (worse than baseline) before partially recovering. The paper's caption states the spike correlates with gradient norm instability, visible in Figure 5(b), where HC's gradient norm spikes to approximately 0.20–0.25 in the same region where mHC and the baseline remain at approximately 0.05–0.10. In contrast, mHC maintains a consistently negative loss gap throughout training, ending at a final loss reduction of 0.021 below the baseline (Figure 5a). This is the quantitative validation of the stability claim: mHC achieves the improved convergence that HC promises but cannot sustain at scale.

The gradient norm comparison in Figure 5(b) reinforces this. The baseline and mHC track each other closely throughout training, with gradient norms remaining in the 0.05–0.10 range and showing the expected gradual decline as training progresses. HC's gradient norm is initially comparable but diverges sharply at step ~12,000, reaching values of 0.20–0.25 — roughly 2–3× higher than the stable methods. This divergence is the proximal cause of the loss spike: when gradient norms surge, the optimizer takes disproportionately large steps, potentially knocking parameters out of well-conditioned regions and causing the loss to jump. The fact that mHC's gradient norm behavior is nearly indistinguishable from the standard residual baseline (which has the identity mapping guaranteeing stability) is strong evidence that the doubly stochastic constraint successfully restores the gradient flow properties of identity mappings.

Downstream Benchmark Performance

Table 4 presents the downstream performance comparison across eight benchmarks for the 27B models. The headline results:

mHC consistently outperforms the baseline. Across all eight benchmarks, mHC achieves higher scores than the standard residual connection. The gains range from +1.0 (PIQA: 78.5→80.5, a 2.0-point improvement) to +4.0 (MATH: 22.0→26.0, an 18.2% relative improvement). On average across benchmarks, mHC improves the baseline by approximately 3.6 points. This validates that multi-stream residual connections with learned mixing (even constrained mixing) provide genuine representational benefits beyond what the standard identity residual connection offers.
mHC outperforms HC on 6 of 8 benchmarks. On BBH (51.0 vs. 48.9, +2.1), DROP (53.9 vs. 51.6, +2.3), HellaSwag (74.7 vs. 74.3, +0.4), MMLU (63.4 vs. 63.0, +0.4), PIQA (80.5 vs. 79.9, +0.6), and TriviaQA (57.6 vs. 56.3, +1.3), mHC achieves higher scores than unconstrained HC. On GSM8K, the scores are nearly tied (53.8 for mHC vs. 53.2 for HC, mHC ahead by 0.6). On MATH, HC narrowly outperforms mHC (26.4 vs. 26.0, HC ahead by 0.4). This is a striking result: the constraint intended to stabilize training does not sacrifice performance — in fact, on the majority of tasks, mHC achieves better downstream performance than unconstrained HC. This suggests that HC's additional degrees of freedom (negative entries, unbounded row sums) are either not useful for these tasks or are actively harmful (perhaps the instability during training damages the final parameter quality even if the run doesn't fully diverge).
The reasoning-intensive benchmarks show the largest gains. On BBH (a collection of challenging BIG-Bench tasks requiring complex reasoning, measured 3-shot), mHC improves over the baseline by +7.2 points (43.8→51.0, a 16.4% relative gain). On DROP (reading comprehension with discrete reasoning, measured by F1), the improvement is +6.9 points (47.0→53.9). On GSM8K (grade-school math word problems), +7.1 points (46.7→53.8). On MATH (competition mathematics), +4.0 points (22.0→26.0). This pattern — larger gains on tasks requiring multi-step reasoning — aligns with the intuition that enriched residual connections help propagate and combine information across layers, which is particularly valuable when solving problems that require integrating evidence from multiple reasoning steps.
Knowledge-intensive benchmarks show notable but smaller gains. On MMLU (multi-task factual knowledge), +4.4 points (59.0→63.4). On TriviaQA (trivia), +3.3 points (54.3→57.6). On HellaSwag (commonsense reasoning), +1.0 point (73.7→74.7). On PIQA (physical commonsense), +2.0 points (78.5→80.5). The gains are consistent but generally smaller than on reasoning benchmarks, suggesting that multi-stream residuals primarily improve the model's ability to manipulate and transform information across layers rather than simply storing more factual knowledge.

Scaling Behavior

Figure 6 presents two complementary scaling analyses — one across model sizes (compute scaling, Figure 6a) and one across training tokens within a single model (token scaling, Figure 6b).

Compute scaling curve (Figure 6a): The solid lines plot the absolute loss gap (left y-axis) and relative loss ratio (right y-axis) of mHC versus the baseline across total training FLOPs spanning the 3B, 9B, and 27B models (each trained with proportional data). The absolute loss gap shows mHC achieving a loss improvement of approximately -0.030 to -0.035 at the 3B scale (leftmost point on the x-axis, approximately 10^21 FLOPs), slightly narrowing to approximately -0.020 to -0.025 at the 27B scale (rightmost point, approximately 2–4 × 10^22 FLOPs). The paper states the advantage "is robustly maintained even at higher computational budgets, showing only marginal attenuation." The relative loss ratio (right y-axis, plotted as percentage where 100% = baseline loss) shows mHC achieving approximately 98.5% of baseline loss at 3B, improving slightly to approximately 99.0% at 27B — a counterintuitive result suggesting that mHC's advantage as a fraction of total loss is roughly constant or slightly improving with scale.

Token scaling curve (Figure 6b): The 3B model trained on 1.05T tokens (the "3B 1T Tokens" column in Table 5) shows mHC's loss advantage over the baseline throughout training. The absolute loss gap starts near zero (at initialization, both models have the same loss), gradually grows to approximately -0.020 by 10^21 FLOPs (around 200B tokens), and maintains approximately -0.015 to -0.020 through the remainder of training. The relative loss ratio similarly stabilizes around 99.0% of baseline loss for most of the trajectory. Crucially, the advantage does not diminish or vanish with extended training — it is sustained across three orders of magnitude of token count. This suggests that mHC's benefits are not merely a "training speedup" that the baseline eventually catches up to, but rather a genuine improvement in the model's representational capacity that persists throughout the learning process.

What these curves do not show: The compute scaling curve only spans a factor of ~10–20× in FLOPs (3B to 27B), which is relatively narrow compared to typical scaling law studies that span 3–5 orders of magnitude. The paper cannot make strong claims about whether mHC's advantage would persist at 100B+ parameter scales. The curves also show single data points per model size rather than a continuous sweep of compute-optimal configurations (which would require training multiple model sizes at multiple data scales to find the Pareto frontier). The "marginal attenuation" noted by the authors may be genuine but the narrow range makes it difficult to distinguish from noise or hyperparameter sensitivity.

Stability Quantification via Learned Mapping Analysis

Figures 7 and 8 provide the quantitative and qualitative evidence that mHC's manifold constraint successfully controls signal propagation.

Single-layer stability (Figure 7a): For the 27B mHC model, the forward signal gain (maximum absolute row sum of the single-layer projected mapping P_M_res(Hres_l)) is exactly 1.0 for all 60 layers — this is a mathematical consequence of the doubly stochastic constraint (every row sums to 1, and with non-negative entries, the sum of absolute values equals the row sum). The backward gradient gain (maximum absolute column sum) deviates slightly from 1.0, reaching values between approximately 1.0 and 1.6 at some layers. This deviation is the residual error from using finite Sinkhorn-Knopp iterations (t_max = 20 rather than ∞). The paper notes that this is "an approximate solution" (Section 5.4). Compare this to HC's single-layer gains in Figure 3(a), which range from approximately 1 to 15 for forward and 0 to 15 for backward — mHC reduces the single-layer gain variability by roughly an order of magnitude, and the gains are bounded above by a small constant rather than growing arbitrarily.

Composite mapping stability (Figure 7b): The deeper composite mapping from layer l to layer 60 shows the backward gradient gain peaking at approximately 1.6 for the longest paths (shallowest layers composing all the way to layer 60). The forward gain similarly remains tightly clustered near 1.0. Crucially, the peak gain does not grow with depth — layers 0–60 show the same ~1.6 maximum as layers 30–60, indicating that the finite-iteration approximation error accumulates additively or sub-linearly rather than multiplicatively. This is the evidence for the compositional closure property: if the error per layer were compounding exponentially, the composite gain would grow with path length, but it doesn't. Compare to HC's composite gains in Figure 3(b), which reach approximately 3000 — mHC reduces the maximum composite gain by three orders of magnitude.

Visualization of learned mappings (Figure 8): The figure shows 4×4 matrices for HC (top row) and mHC (bottom row) at three representative depths (layers 1, 30, 60), plus composite products over 30 and 60 layers. For HC: the single-layer matrices show entries ranging from -16 to +19 at layer 1, with row sums (annotated along the y-axis) like -6.81, 18.73, -15.29, -14.79 — far from the ideal of 1. The product over 30 layers shows entries of ±135 to ±500 with row sums like -135.4, -251.4, -69.9, 74.8 — chaotic and unbounded. The product over 60 layers is even more extreme: entries of -500 to +500 with row sums of -259.2, -475.3, -132.8, 142.3. One row maps a unit signal to output magnitude 509. For mHC: all single-layer matrices have entries between 0 and 1 with row sums exactly 1.0 and column sums annotated as 0.98–1.04 (close to 1.0). The matrices are not identity — they show learned structure. At layer 1, row 1 is (0.83, 0.73, 0.66, 0.75), indicating stream 1 routes strongly to itself and streams 2–3, but the off-diagonal entries (0.00–0.22 elsewhere) show selective connectivity. At layer 30, the pattern changes to a different structure, and at layer 60, yet another pattern emerges. The composite products show progressive mixing toward uniformity: the 30-layer composite has entries of 0.17–0.34 (approaching 1/n = 0.25), and the 60-layer composite continues this trend. Row sums remain exactly 1.0 throughout. The column sums in the composites deviate slightly (0.90–1.50 range at 30 layers), again reflecting the finite-iteration approximation.

The crucial empirical takeaway from Figure 8 is that mHC learns meaningful, depth-dependent mixing patterns (the matrices at layers 1, 30, and 60 look qualitatively different) while maintaining the mathematical guarantee of bounded signal propagation. HC's matrices also show depth-dependent structure, but their unbounded entries make training at scale impossible.

Ablation Studies and Robustness Checks

Ablation of HC components (Table 1): The paper conducts an ablation study (presumably at a smaller scale, though the model size is not specified in the table or surrounding text) to determine which of the three HC mappings — Hres_l, Hpre_l, Hpost_l — contributes most to performance. The "Absolute Loss Gap" column reports the final training loss difference relative to a configuration with all three mappings disabled (which reduces to a standard residual connection). Disabling Hres_l (replacing it with the identity matrix) while keeping Hpre_l and Hpost_l yields an absolute loss gap of -0.022. Enabling Hres_l and Hpre_l (disabling Hpost_l with uniform weights of 1) yields -0.025. Enabling all three yields -0.027. The finding: the residual mapping Hres_l yields the most significant performance gain — going from Hres_l disabled to enabled (with Hpost_l also enabled) improves from -0.022 to -0.027, a 0.005 loss reduction. Going from Hres_l alone to Hres_l + Hpre_l adds another 0.003. This justifies the paper's focus on constraining Hres_l specifically, since it is the mapping that both contributes most to performance and causes the instability when unconstrained.

The paper does not report an analogous ablation for mHC to confirm that constraining Hres_l does not reduce the contribution of the other mappings. This is a minor gap: one could imagine that the doubly stochastic constraint on Hres_l changes how Hpre_l and Hpost_l are utilized, and an ablation confirming that all three mappings remain beneficial under the constraint would strengthen the completeness of the analysis.

Sinkhorn-Knopp iteration count (t_max=20, implicit ablation): The paper does not perform an explicit sweep over t_max values, but the stability analysis in Figure 7 serves as an implicit evaluation. With t_max=20, the single-layer backward gradient gain deviates from 1.0 by up to 0.6 (reaching ~1.6), and the composite gain peaks at approximately 1.6. With fewer iterations, these deviations would be larger, and the composite gain might grow more rapidly with depth (since the per-layer error would be larger, potentially compounding). With more iterations, the deviation would shrink toward zero. The choice of 20 is presented as "a practical value" without sensitivity analysis. A sweep over t_max ∈ {5, 10, 20, 40} showing the tradeoff between training stability (Amax Gain Magnitude) and computational cost (iteration overhead) would have been informative for practitioners choosing this hyperparameter. The paper's claim that 20 iterations reduces composite gains from ~3000 to ~1.6 is strong, but the reader cannot assess how much of this reduction occurs at 5 or 10 iterations, which would be cheaper to compute.

Sigmoid projection for Hpre_l and Hpost_l (Equation 8): The paper constrains Hpre_l and Hpost_l with sigmoid (ensuring non-negativity) rather than attempting a doubly stochastic-like constraint. The factor of 2 in Hpost_l = 2σ(H̃post_l) (expanding the range to (0,2) rather than (0,1)) is an asymmetry that the paper does not justify. This is a curious design choice: it means the output mapping can amplify the layer's contribution by up to 2×, which seems to partially counteract the norm-preservation philosophy applied to Hres_l. No ablation comparing sigmoid with and without the factor of 2, or comparing sigmoid to a sum-to-1 normalization (softmax), is reported. The implicit assumption is that the non-negativity constraint alone is sufficient for pre/post stability because these are vectors applied once per layer (not composed across depth), but the asymmetry of the 2× factor is unexplained.

Flattening the hidden matrix before coefficient computation (Equation 7 vs. Equation 5): The paper modifies HC's coefficient computation by flattening the n×C hidden matrix into a 1×nC vector before normalization and projection, providing cross-stream context for computing Hres_l. This modification is not ablated — the paper does not compare mHC with and without flattening to determine whether the flattening contributes to performance independently of the manifold constraint. This is a significant gap because the flattening increases the parameter count of the linear projections (φ_res_l is n²C parameters for HC vs. (nC) × n² for mHC — though this is still small compared to the main model parameters) and changes the inductive bias of how cross-stream interactions are computed. The observed performance improvements of mHC over HC (Table 4) could partially derive from the flattening rather than the manifold constraint, and the paper cannot disentangle these two factors without an ablation. If mHC without flattening (using per-stream normalization as in HC but still projecting onto the Birkhoff polytope) performed similarly to the current mHC, the flattening could be dropped to save parameters. If it performed worse, the flattening would be identified as a necessary component.

Initialization of gating factors (α=0.01): Both HC and mHC initialize the scalar gating factors α to 0.01, meaning that at the start of training, the mappings are dominated by their static biases (initialized to approximate identity for Hres_l). This is an important stabilization choice — it ensures both methods start near the stable identity-mapping regime and gradually learn input-dependent mixing as α grows. The paper does not ablate this initialization value. A larger initial α (e.g., 0.1 or 1.0) would give the model more dynamic mixing from the start, potentially accelerating learning of useful routing patterns but also increasing the risk of early instability — particularly for HC, where the composite mapping would diverge earlier in training. For mHC, the doubly stochastic constraint would still bound the mapping regardless of α, but the learning dynamics might differ. The choice of 0.01 is conventional (small random initialization is standard) but not specifically justified for this architecture.

Selective recomputation block size Lr (Equation 20, implicit validation):* The paper derives the optimal recomputation block size Lr* ≈ √(nL/(n+2)) and notes that "the theoretical optimum Lr* typically aligns with the number of layers per pipeline stage" (Section 4.3.2). They "choose to synchronize the recomputation boundaries with the pipeline stages." This is presented as a design decision rather than an empirical finding — the paper does not report ablation results comparing different Lr values or comparing aligned vs. unaligned boundaries. Since the 6.7% overhead figure depends on this and other infrastructure optimizations, the lack of a breakdown showing how much each optimization contributes (kernel fusion alone, recomputation alone, communication overlap alone, and their combination) makes it difficult to assess which optimizations are essential and which are marginal. The paper states these are implemented together, so their individual contributions cannot be extracted post hoc, but an ablation table reporting training throughput (tokens/second) for mHC with and without each optimization would significantly strengthen the infrastructure contribution.

HC with gradient clipping or learning rate adjustments (missing baseline): The paper demonstrates that HC is unstable at 27B with the same optimizer settings as the baseline and mHC. However, a practical question is whether HC could be stabilized through standard techniques — gradient clipping, reduced learning rate, or more aggressive warmup — without architectural modification. The paper does not report experiments attempting to rescue HC through hyperparameter tuning. This is a somewhat unfair omission: the fair baseline for "can HC be made to work at scale?" would be HC with training interventions that address the diagnosed instability (exploding gradients) without changing the architecture. If gradient clipping to a threshold of, say, 1.0 could prevent HC's loss spike (by capping the gradient norms visible in Figure 2b), then the practical advantage of mHC would be narrower. If gradient clipping fails (because it treats the symptom without addressing the underlying signal explosion in the forward pass), then mHC's architectural solution would be established as necessary, not just sufficient. The paper's theoretical argument — that the instability is in the forward signal, not just the backward gradient — implies that gradient clipping alone would not suffice (it would prevent the optimizer from taking destructive steps but would not prevent the forward representations from exploding, making the model's outputs nonsensical), but this is not empirically verified.

Critical Assessment

The experiments collectively demonstrate that mHC achieves stable training at 27B scale where HC exhibits catastrophic instability, and that mHC delivers consistent downstream performance improvements over the standard residual baseline. However, the experiments demonstrate a narrower set of claims than the paper's broader framing suggests, and several important validity questions are left unaddressed.

Claim: mHC eliminates HC's training instability while maintaining performance

What the experiments actually show: For the specific 27B DeepSeek-V3-style MoE architecture with n=4, trained with the specific hyperparameters in Table 5, mHC does not experience the loss spike that HC exhibits around step 12,000 (Figure 5a), and mHC's gradient norms remain stable where HC's diverge (Figure 5b). The Amax Gain Magnitude analysis confirms the mechanism: HC's composite residual mapping amplifies signals by up to ~3000× (Figure 3b), while mHC's composite mapping stays bounded at ~1.6× (Figure 7b). The downstream results (Table 4) show mHC outperforming HC on 6 of 8 benchmarks and matching or nearly matching on the other 2.

What has not been shown: The experiments use a single training run per configuration. The loss spike at step 12,000 for HC might be stochastic — with a different random seed, HC might spike at step 8,000, at step 25,000, or potentially not at all. The paper does not report whether HC's instability is reproducible across multiple seeds or whether it always occurs at approximately the same training step. If HC is unstable only some fraction of the time (e.g., in 2 of 5 seeds), then mHC's stability advantage is real but probabilistic rather than deterministic. The paper's theoretical analysis (the composite mapping norm diverging over depth) suggests the instability should be inevitable given enough training, but the empirical evidence from one run is suggestive rather than conclusive. Similarly, the downstream performance comparisons (Table 4) are from single pretraining runs — the differences between mHC and HC (e.g., 51.0 vs. 48.9 on BBH, a 2.1-point gap) could be within the variance of different random seeds, different data orderings, or different initialization. Without multiple runs or confidence intervals, the reader cannot distinguish systematic improvement from noise.

The scale range is limited — 3B to 27B parameters spans less than one order of magnitude in model size, and 10^21 to ~3×10^22 FLOPs spans about 1.5 orders of magnitude in compute. The paper's abstract and introduction frame mHC as enabling "training at scale" and "superior scalability," but the largest model tested (27B) is small by contemporary standards (DeepSeek-V3 itself is 671B total parameters). The paper mentions "in-house large-scale training" in the introduction and again in Section 5.3 ("This conclusion is further corroborated by our in-house large-scale training experiments"), suggesting that the authors have tested mHC at larger scales but do not report these results. This is a significant omission: if mHC has been tested successfully at, say, 100B+ parameters, reporting those results would substantially strengthen the scaling claim; withholding them suggests either that the results are not yet ready for publication or that the scaling behavior at very large scales introduces new challenges not addressed in the current manuscript.

Claim: mHC is "a general framework" applicable beyond the tested configuration

What the experiments actually show: mHC is tested on exactly one architecture family (DeepSeek-V3 MoE with MLA, RoPE, RMSNorm), one expansion rate (n=4), one Sinkhorn-Knopp iteration count (t_max=20), and one training paradigm (language model pretraining with the specific data mixture and token counts in Table 5). The "general framework" claim (abstract, Section 4) is aspirational — the experiments do not demonstrate generality across architecture families (e.g., dense Transformers, encoder-decoder models, vision Transformers), expansion rates (n=2, n=8), or training paradigms (fine-tuning, RLHF, multimodal pretraining).

What would strengthen this claim: At minimum, showing that mHC works with different values of n (the paper sets n=4 throughout, with no justification for this choice) would demonstrate that the framework generalizes across expansion rates. Testing on a dense (non-MoE) Transformer would demonstrate that the benefits are not tied to the MoE architecture's specific properties (e.g., the already-present routing mechanisms in MoE might interact with the residual stream routing in non-obvious ways). The paper notes that n "is much smaller than the input dimension C" (Section 3), but the relationship between n and stability is unexplored — for larger n, the Birkhoff polytope is higher-dimensional (9-dimensional for n=4, 16-dimensional for n=5), and the Sinkhorn-Knopp algorithm might require more iterations to converge to the same approximation quality, potentially increasing both computational cost and residual deviation from true doubly stochasticity.

Claim: The 6.7% training overhead is achievable

What the experiments actually show: The paper reports that mHC with n=4 introduces "only a 6.7% additional time overhead" (Section 1) under "rigorous infrastructure optimization" (Section 4.3). The detailed description of kernel fusion, selective recomputation, and DualPipe extension provides a credible account of how this number is achieved. However, the paper does not break down the overhead by component or report the overhead without these optimizations. This makes it impossible to assess: (a) which optimizations are essential (would the overhead be 15%, 30%, or 50% without kernel fusion? without recomputation? without communication overlap?), (b) whether the 6.7% figure is robust to different hardware configurations (the paper uses unspecified GPU hardware, likely NVIDIA H800 or similar based on the DeepSeek-V3 context, but does not specify), and (c) whether the optimizations are specific to the DeepSeek training infrastructure (DualPipe, TileLang) or transferable to standard frameworks (PyTorch FSDP, Megatron-LM, etc.).

The 6.7% figure is presented as a single number without context about absolute throughput (tokens per second), model FLOPs utilization (MFU), or comparison to the standard HC overhead (which the paper implies would be higher due to its unaddressed memory access patterns). A table reporting training throughput for Baseline, HC (without mHC's optimizations), HC (with mHC's optimizations retrofitted), and mHC (fully optimized) would make the infrastructure contribution evaluable. As presented, the 6.7% figure is a claim that relies on the reader trusting the authors' engineering without providing the evidence to verify it.

Claim: mHC restores the identity mapping property

What the experiments actually show: The Amax Gain Magnitude analysis (Figures 7, 8) demonstrates that mHC's composite mappings are bounded near 1.0 (specifically ~1.0 for forward, ~1.6 for backward), a three-order-of-magnitude reduction from HC's ~3000. This is strong evidence that the manifold constraint prevents the unbounded signal amplification that HC suffers from. However, the paper's language about "restoring the identity mapping property" (abstract, Section 1) is imprecise: mHC does not make Hres_l equal to the identity matrix. It constrains Hres_l to be doubly stochastic, which is a much larger set of matrices than just the identity. The property being "restored" is norm preservation and compositional closure — the practical consequences of identity mappings (stable signal propagation) rather than identity mappings themselves. The learned matrices in Figure 8 are visibly not identity — they have off-diagonal structure that the model has learned to use for cross-stream mixing. This is a terminological imprecision rather than an experimental weakness, but it matters because it might lead readers to expect that mHC forces Hres_l ≈ I, which would eliminate the very cross-stream mixing that provides the performance benefits.

Missing experiments that would strengthen the paper

Multiple training seeds for the 27B comparison (at minimum 3 seeds each for Baseline, HC, and mHC) to establish that mHC's stability and performance advantages are statistically reliable.
A sweep over expansion rate n (e.g., n=2, 4, 8) for a fixed model size (e.g., 3B) to characterize how stability, performance, and overhead scale with n. The paper's choice of n=4 is never justified, and it is the only value tested.
An explicit comparison of HC with gradient clipping (e.g., clip grad norm to 1.0) to test whether the instability can be mitigated through standard training interventions without architectural modification.
Ablation of the flattening modification (Equation 7 vs. Equation 5) to determine whether the performance gains over HC derive from the manifold constraint, the flattening, or both.
Sensitivity to Sinkhorn-Knopp iteration count t_max ∈ {5, 10, 20, 40, 100} to characterize the tradeoff between approximation quality (Amax Gain Magnitude) and computational cost.
Testing on a dense (non-MoE) Transformer to demonstrate that the benefits of mHC are not specific to the MoE architecture.
Breakdown of the 6.7% overhead into contributions from different optimizations (kernel fusion, recomputation, communication overlap), and reporting of absolute throughput (tokens/second) for all configurations.
Larger-scale results — the paper alludes to "in-house large-scale training" that corroborates the findings but does not report these results. If mHC has been tested at the 100B+ scale, those results are the strongest evidence for the scalability claim and their absence is conspicuous.

Summary evaluation

The experiments convincingly demonstrate that mHC solves the specific instability problem that HC exhibits at 27B scale on the tested architecture. The mechanism is well-diagnosed: the doubly stochastic constraint bounds the composite residual mapping's norm, preventing the exponential signal amplification that causes HC's loss and gradient spikes. The downstream performance results confirm that this constraint does not sacrifice the representational benefits of multi-stream residual connections — mHC performs comparably to or better than HC across a diverse benchmark suite. The infrastructure optimizations are described in detail and plausibly account for the claimed 6.7% overhead, though without an ablation of individual optimizations, their relative contributions cannot be assessed.

The paper's broader claims — that mHC is a "general framework," that it provides "superior scalability," and that it will "contribute to a deeper understanding of topological architecture design" — are supported in principle by the experimental results but not empirically demonstrated in their full generality. The single architecture family, single expansion rate, single training paradigm, and absence of statistical quantification or large-scale validation mean that the paper establishes feasibility and mechanism rather than comprehensive empirical validation. This is common and acceptable for an architecture paper introducing a new principle (manifold-constrained residual connections), but readers should understand that the evidence is narrower than the framing suggests. The paper's primary contribution is the conceptual framework — the diagnosis of HC's instability as a compositional closure failure and the identification of the Birkhoff polytope as the minimal sufficient constraint — and the experiments serve as a proof-of-concept rather than an exhaustive evaluation.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For — And Is Likely Prohibitive

The entire compute-optimal allocation framework depends on estimating each prompt's difficulty before deciding how to spend the inference budget. The paper's method for doing so is extraordinarily expensive and not included in the reported efficiency calculations.

The assumption: The paper estimates difficulty by generating 2048 samples per question from the base model, then either checking ground-truth correctness (oracle) or averaging the PRM's predicted final-answer scores (predicted). The 4× efficiency gains over best-of-N are computed after this difficulty is known, without amortizing the cost of learning it. The paper acknowledges this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence: The 4× figure should be understood as an upper bound on achievable efficiency in a deployment scenario. In practice, the total cost would be difficulty estimation + strategy execution. For a single question, generating 2048 samples costs roughly an order of magnitude more than the largest test-time budgets studied (256–512 generations), completely dwarfing any savings from smarter allocation. Even amortized across many questions from the same distribution, the upfront cost is enormous and cannot be ignored. The paper frames this as "an exploration-exploitation tradeoff" (Section 3.2) but provides no mechanism for balancing it.

Evidence in the paper: None — the difficulty estimation cost is never measured, reported, or compared to the test-time budget. Figures 4 and 8 show compute-optimal scaling curves where the x-axis ("generations") only accounts for the strategy execution, not difficulty estimation. The overlap between oracle and predicted difficulty curves in Figures 4 and 8 confirms that the PRM can substitute for ground-truth labels, but this does not address the cost of generating 2048 samples in the first place.

Mitigation status: The paper explicitly acknowledges this gap and frames it as future work (Section 8):

"our experiments do not account for this cost largely for simplicity, and we leave the development of more efficient difficulty estimation methods to future work"

No prototype or proposal for a cheap estimator (lightweight classifier, adaptive estimation, etc.) is provided. This is the single largest barrier to practical deployment of the compute-optimal framework as described.

No Mechanism Exists for Dynamically Adjusting Strategy Mid-Computation

The paper's compute-optimal policy is a static, pre-computed lookup: given an estimated difficulty bin and a total budget, use a predetermined strategy (e.g., beam search with M=4 for medium-difficulty problems, sequential revisions for easy problems). The policy does not adapt during the solution process itself.

The assumption: The five-quintile difficulty binning is a coarse discretization of a continuous property, computed once from the PRM's score distribution over 2048 samples. Within a single bin, there may be substantial heterogeneity — a question at the easy end of bin 3 and one at the hard end of bin 3 receive identical strategy allocations. The paper does not explore whether difficulty estimates could be refined during the solution process (e.g., generating a few initial samples, assessing the verifier's score distribution on those samples, and reallocating the remaining budget based on this live signal).

The consequence: The static policy leaves efficiency on the table. If the initial samples from a problem believed to be "medium" difficulty consistently score very poorly under the PRM (suggesting the problem is actually hard), the policy should switch from beam search (which is ineffective on hard problems per Figure 3 right) to best-of-N or abandon the problem. Conversely, if initial samples on a suspected hard problem turn out to have non-trivial verifier scores, the policy could upgrade to more aggressive search. The current approach cannot make these adjustments. Additionally, the five bins are an arbitrary discretization — the paper does not test sensitivity to the number of bins (would 10 bins perform better? would 3 bins perform nearly as well?), so the reader cannot assess whether the discretization choice is near-optimal or arbitrary.

Evidence in the paper: The difficulty-dependent results in Figures 3 (right) and 7 (right) show qualitatively different optimal strategies across bins, confirming that bin choice matters. However, no experiment varies the number of bins, no dynamic allocation scheme is tested, and the paper does not report whether the sharp boundaries between bins correspond to genuine discontinuities in optimal strategy or are artifacts of the quintile discretization. The compute-optimal curves in Figures 4 and 8 use fixed pre-computed strategies per bin with no adaptation.

Mitigation status: The paper does not address dynamic allocation at all. Section 8's future work suggestions focus on cheap difficulty estimation and combining search with revisions, not on online strategy adaptation. This is a self-contained limitation: the static nature of the policy is a design choice, not a fundamental constraint of the framework, but demonstrating that dynamic allocation improves upon static allocation would significantly strengthen the practical case for compute-optimal test-time scaling.

Hard Problems Remain Unsolvable — Test-Time Compute Cannot Compensate for Capability Gaps

Across all methods studied — search, revisions, and their compute-optimal combinations — the hardest 20% of questions (difficulty bin 5) show near-zero improvement regardless of how much test-time compute is allocated. This places a hard boundary on what test-time scaling can achieve.

The consequence: Test-time compute amplifies existing capability but does not create it. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help — there are no correct solutions in the proposal distribution to find or refine. The FLOPs-matched comparison in Section 7 makes this explicit: on hard questions (bins 4–5), the smaller model with additional test-time compute performs substantially worse than the ~14× larger model. For PRM search at R ≫ 1, the relative disadvantage on hard questions is −52.9%. The paper's own takeaway (Section 7) acknowledges: "test-time compute provides minimal gains on problems that are fundamentally outside the base model's capability range."

Evidence in the paper: Figure 3 (right, bin 5): accuracy hovers at 1–3% for all methods and all budgets up to 256 generations. Figure 7 (right, bin 5): accuracy around 2–3% regardless of sequential-to-parallel ratio. Figure 9 (bin 5, blue line): the scaling curve is essentially flat near 0–5% across all budgets, and the ~14× larger model's performance (star) is uniformly above the test-time scaling curve. The paper is transparent about this (Section 7), but the consequence for practical deployment is significant: for any problem distribution with a non-trivial fraction of "hard" problems (where the base model rarely produces correct answers), the overall performance ceiling is set by pretraining capability, not by clever test-time allocation.

Mitigation status: The paper acknowledges this boundary but does not and cannot solve it — it is a fundamental limitation of any approach that only modifies inference-time computation without changing the underlying model. The paper's contribution is precisely characterizing where this boundary lies (the pass@1 threshold below which test-time compute adds nothing) rather than moving it. For practitioners, this means the compute-optimal framework is only applicable to problems within the base model's approximate capability range — it cannot turn a weak model into a strong one on genuinely difficult tasks.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate — And No Principled Fix Is Provided

The revision model, which is central to the sequential revision strategy that dominates on easy problems, suffers from a predictable but serious failure: when it encounters a correct answer in its own revision chain, it frequently "revises" it into an incorrect answer.

The assumption: The revision model is trained exclusively on trajectories where all in-context answers are incorrect, followed by a correct target (Section 6.1). This means the model never sees examples of what to do when the current answer is already correct. At inference time, when earlier revisions happen to produce a correct answer, the model has no training signal for how to handle this situation — and its learned behavior (conditioning on previous answers and producing a revision) actively works against it.

The consequence: The paper reports that approximately 38% of correct answers get converted back to incorrect ones during a revision chain (Section 6.1). This is a massive reversion rate: for every two correct answers produced, nearly one is immediately lost in the next revision step. The paper mitigates this with within-chain selection — using majority voting or verifier-based selection to pick the best answer from any point in the chain rather than always taking the last revision. But this is a patch, not a solution. It means that the revision chain is partially self-defeating: the model generates correct answers, then destroys them, and the selection mechanism must identify the correct answer amidst a trail of self-inflicted errors. Any computational budget spent on revisions after a correct answer has been produced is effectively wasted — and worse, it might make the correct answer harder to retrieve if the verifier scores are noisy.

Evidence in the paper: Section 6.1 states the 38% reversion rate explicitly. The ReST^EM experiment in Appendix K (Figure 16) further demonstrates the fragility of revision training: attempting to optimize the revision model with reinforcement learning caused performance to degrade substantially with sequential revisions (fully sequential drops to ~33.5% vs. ~38.5% at the optimal ratio), suggesting that "on-policy data collection amplifies spurious correlations in revision data." This negative result indicates that the revision approach is highly sensitive to the training data construction procedure and may not be robust to changes in the training recipe.

Mitigation status: The within-chain selection (majority voting or verifier-based selection across the chain) partially mitigates the reversion problem by not committing to the final revision. However, this is an inference-time heuristic that does not address the root cause: the model has no notion of "stop revising when the answer is correct." The paper does not explore training the model with explicit "no revision needed" tokens, confidence estimation, or early stopping criteria. This is a significant practical limitation because it means revision chains are inherently inefficient — a substantial fraction of the chain's compute is spent undoing correct work.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained — Making the FLOPs-Matched Comparison Favorable to Test-Time Compute

The FLOPs-matched comparison in Section 7, which is central to the paper's claim that test-time compute can substitute for pretraining, compares against a larger model that is not trained under a compute-optimal regime.

The assumption: When scaling pretraining compute, the paper scales only model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The paper explicitly acknowledges (Section 7) that this departs from compute-optimal pretraining (Hoffmann et al., 2022), where both data and parameters would be scaled equally. The ~14× larger model is trained on the same amount of data as the smaller model, meaning it is undertrained relative to a Chinchilla-optimal configuration — its performance is below what a properly scaled model of that parameter count could achieve.

The consequence: The reported advantages of test-time compute over pretraining — e.g., +27.8% relative improvement on easy-to-medium questions at R ≪ 1 for revisions (Figure 1, top-right bar chart) — likely overstate the true benefit. A compute-optimal larger model, trained on proportionally more data (roughly 14× more tokens for 14× more parameters, following the Chinchilla prescription), would achieve lower loss and better downstream performance than the parameter-only-scaled baseline used here. This would shift or reverse the crossover points where test-time compute becomes preferable. The paper's own numbers show that the advantage of test-time compute narrows substantially as R increases (from +27.8% at R ≪ 1 to +5.4% at R ≫ 1 for revisions on medium questions), suggesting that even with the undertrained baseline, the margin is thin in high-inference-volume regimes. Against a properly trained baseline, test-time compute might lose in more regimes.

Additionally, the larger model uses only greedy decoding — no majority voting, no best-of-N, no search. Giving the larger model even a modest test-time compute budget (e.g., best-of-8 or best-of-16) would create a much stronger baseline that is never tested. The paper's framing of "test-time compute vs. pretraining" implicitly assumes that the larger model cannot also benefit from test-time compute, which is unrealistic — in practice, a larger model would also use some form of test-time augmentation, narrowing the gap further.

Evidence in the paper: The caveat is disclosed in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The FLOPs-matched results in Figure 9 and the bar charts in Figure 1 are entirely dependent on this choice. The paper does not report what a Chinchilla-optimal larger model's performance would be, nor does it provide the data scaling curve for the larger model that would allow readers to estimate it.

Mitigation status: The paper acknowledges the limitation and defers it to future work. This is a fair disclosure but a substantial caveat to one of the paper's headline claims. The open question — whether test-time compute can substitute for compute-optimally trained larger models — remains unaddressed, and the current results should be interpreted as an upper bound on the substitution benefit, not a precise estimate.

The 500-Question Test Set and Two-Fold Cross-Validation Provide Weak Statistical Reliability for Strategy Selection

The compute-optimal strategy — the centerpiece of the paper — is selected based on a small sample and evaluated without confidence intervals, making the reported gains difficult to assess statistically.

The assumption: The test set consists of 500 MATH questions, split into five difficulty quintiles of approximately 100 questions each. Two-fold cross-validation within each bin means the compute-optimal strategy is selected based on approximately 50 questions per fold per bin (Section 3.2). All main results (Figures 4, 8; Tables in Section 5; FLOPs-matched comparisons in Section 7) report point estimates from this single split with no error bars, confidence intervals, or multi-seed analysis.

The consequence: The selected strategies may not be robust. With only 50 questions per fold per bin, the variance in estimated accuracy for a given strategy is substantial — a ±3 percentage point swing on 50 questions changes which strategy appears optimal, and this could propagate to the compute-optimal allocation policy. The paper does not report whether the strategy selected on fold A differs from the strategy selected on fold B within the same difficulty bin, which would indicate instability in the selection. If the selected strategies vary significantly across folds, the reported compute-optimal performance (which averages across folds) may be optimistic relative to a single fixed policy deployed in practice.

More broadly, the absence of confidence intervals makes it impossible to assess whether the observed differences between methods (e.g., mHC vs. HC in Table 4) are statistically significant or consistent with random variation from a single training run. The 2.1-point gap on BBH (51.0 vs. 48.9) and 2.3-point gap on DROP (53.9 vs. 51.6) are reported without any quantification of uncertainty, and with a 500-question test set (or the specific benchmark sizes, which may be smaller), these gaps may not be statistically distinguishable.

Evidence in the paper: Section 3.2 describes the cross-validation protocol. Section 5 reports all results as point estimates. No standard deviations, standard errors, or bootstrap confidence intervals are reported anywhere. The single-run nature of the training experiments (one seed per configuration) is not discussed as a limitation.

Mitigation status: The paper does not address statistical reliability at all. The use of two-fold cross-validation is a basic safeguard against overfitting the strategy to the test set, but it does not provide variance estimates. A minimal fix — reporting ±1 standard error for the main results, or running 3–5 seeds for the key stability comparison (Figure 5) — would substantially strengthen confidence in the conclusions. The current presentation requires the reader to assume that the observed differences are robust, which is plausible given the consistent pattern across benchmarks but not empirically established.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around macro-architecture design for deep networks from one of unconstrained expressivity maximization to one of constrained optimization over manifold-structured connectivity. The field has spent the better part of a decade treating the residual connection as a solved problem — an identity shortcut that, once established by He et al. (2016a,b), required no further innovation beyond being preserved. Hyper-Connections (Zhu et al., 2024) challenged this assumption productively by demonstrating that enriching the residual stream with learnable mixing yields tangible performance gains. But HC's contribution was incomplete: it showed that diversification works at moderate scales but left unaddressed why it fails catastrophically at larger scales, and more importantly, whether that failure was fundamental or correctable.

mHC's primary conceptual contribution is answering that question definitively: the failure is fundamental — it stems from the loss of compositional closure when unconstrained matrices are multiplied across depth — but it is correctable through a specific, mathematically principled constraint that preserves exactly the degrees of freedom that matter for expressivity while eliminating those that cause instability. This reframes the research program on enriched residual connections from "can we make this work at scale?" (an empirical question) to "which manifolds with compositional closure provide the best expressivity-stability tradeoff?" (a design question with clear mathematical criteria). The paper thus transitions the field from exploration to engineering: it establishes that the Birkhoff polytope is sufficient (it works at 27B scale with performance matching or exceeding HC on most benchmarks), which immediately raises the question of what other manifolds might be better (more expressive, cheaper to project onto, providing different mixing properties).

The diagnostic framework the paper introduces — particularly the Amax Gain Magnitude metric for quantifying composite mapping stability — is likely to have lasting methodological impact beyond mHC itself. The key insight is that stability cannot be assessed by inspecting single layers in isolation; it is a path-global property that emerges from the composition of many individually-well-behaved matrices. Figure 3's juxtaposition of single-layer gains (manageable, 1–15) against composite gains (catastrophic, ~3000) makes this point visually undeniable and provides a reusable template for evaluating any future architecture that replaces identity shortcuts with learned transformations. A researcher proposing a new connectivity scheme can compute the analogous composite mapping norm across depth and check for exponential growth — if it diverges, the architecture has an instability risk regardless of how well it performs on small-scale benchmarks. This is a significant methodological upgrade for the field: prior to this work, the stability of residual mixing architectures was assessed through empirical training runs (does the loss diverge?), which is expensive and noisy. The Amax Gain Magnitude provides a cheap, quantitative, architecture-inherent stability diagnostic that can be applied before full training.

The paper also resolves a latent tension in the prior literature that likely deterred adoption of multi-stream residual architectures. HC showed promising performance gains, but practitioners attempting to scale it would have encountered instability and (reasonably) attributed it to implementation bugs, hyperparameter sensitivity, or incompatibility with their specific training infrastructure. The field had no framework for distinguishing "this architecture is fundamentally unstable at scale" from "we haven't tuned it correctly." mHC provides that framework by identifying the precise mathematical property (compositional closure) that separates stable from unstable enriched residual connections. This means that future architecture proposals can be evaluated against a clear criterion — "does your residual mixing manifold have compositional closure?" — rather than relying on expensive and inconclusive empirical scaling studies. The implicit message is that architectures without this property should be presumed unstable at scale until proven otherwise, which shifts the burden of proof.

Perhaps most significantly, mHC establishes that infrastructure co-design is not optional for macro-architecture innovation — it is a binding constraint that determines whether a mathematically elegant idea becomes practically deployable. The paper's Section 4.3 is not an afterthought; it is roughly a third of the method's description and is essential to the 6.7% overhead claim that makes mHC actionable. This sets a precedent for how architecture papers should be evaluated: a proposal that is FLOPs-efficient but memory-bandwidth-heavy, or that increases pipeline communication costs without a mitigation strategy, is incomplete. Future work on residual stream widening, or any architectural change that modifies the per-token memory footprint, will be expected to provide the equivalent of Table 2's I/O analysis and demonstrate that the overhead is manageable under realistic parallelism strategies. This is a positive development for the field — it closes the gap between architecture design (which has traditionally been evaluated on FLOPs and accuracy) and systems implementation (which is dominated by memory bandwidth and communication costs), making architecture research more directly relevant to production-scale training.

The paper also redirects attention toward manifold-constrained optimization as a design paradigm for deep networks, beyond residual connections. The Sinkhorn-Knopp projection onto the Birkhoff polytope is one instantiation of a more general principle: when a learned transformation is applied repeatedly across depth (or time, or any compositional axis), constraining it to lie on a manifold that is closed under the composition operation guarantees global stability from local constraints. This principle has natural applications beyond residual mixing — for example, in recurrent architectures where hidden-to-hidden transformations are composed across time steps, in normalizing flows where invertible transformations are composed to build complex distributions, or in iterative refinement models where the same transformation is applied multiple times. While the paper does not develop these applications, it provides the conceptual vocabulary (manifold constraint, compositional closure, Sinkhorn-Knopp projection as a differentiable mapping from unconstrained to constrained space) that makes them tractable.

Finally, the paper's negative result on HC — the careful documentation of its catastrophic instability at 27B scale, with mechanistic diagnosis via the Amax Gain Magnitude metric and visual evidence in Figure 8 of unbounded composite matrices — serves as a cautionary tale that the field would benefit from internalizing. Architectural innovations that look promising at moderate scales (HC was validated at smaller scales in Zhu et al., 2024) can harbor fundamental scaling defects that manifest only when depth or width crosses a threshold. The implication is that scaling behavior should be a first-class evaluation criterion for architecture proposals, not something deferred to future work. A paper proposing a new connectivity pattern should, at minimum, analyze the composite mapping's spectral properties as depth increases and demonstrate that no exponential divergence occurs. mHC provides both the cautionary example (HC's failure) and the methodological toolkit (Amax Gain Magnitude, compositional closure analysis) for performing this evaluation.

Follow-Up Research This Work Enables

Characterizing the tradeoff between Sinkhorn-Knopp iteration count, stability, and throughput at scale. The paper uses t_max = 20 for all experiments, produces composite Amax Gain Magnitudes peaking at ~1.6 (vs. 1.0 ideally and ~3000 for HC), and reports 6.7% overhead. A systematic sweep over t_max ∈ {1, 2, 5, 10, 20, 40, 100} for a fixed model (e.g., 3B or 9B) would characterize: (a) how quickly the composite gain converges toward 1.0 as t_max increases — does the majority of the stability benefit come from the first few iterations (suggesting t_max = 5 might suffice) or is the convergence slow (requiring t_max ≥ 20)? (b) the throughput cost per iteration — how much of the 6.7% overhead is attributable to the Sinkhorn-Knopp loop specifically versus other mHC operations? (c) whether downstream benchmark performance is sensitive to t_max — does the remaining ~0.6 deviation from ideal stochasticity at t_max=20 matter for accuracy, or would t_max=5 (with, say, composite gains of ~3–5) perform equally well? The key measurement would be a 3D plot or table showing Amax Gain Magnitude, tokens/second, and downstream accuracy as functions of t_max, identifying the Pareto frontier where stability is "good enough" at minimal cost. This would replace the current single-point choice with a principled hyperparameter guideline.

Testing mHC on dense (non-MoE) Transformers to isolate the interaction between residual stream routing and expert routing. The paper's experiments all use DeepSeek-V3-style MoE architectures, which already contain learned routing mechanisms (the expert selection gate). This creates a potential confound: the residual stream mixing in mHC might be partially redundant with expert routing (both are mechanisms for selectively routing information), and the observed benefits might be smaller or qualitatively different in dense architectures where no such routing exists. A controlled comparison on a dense Transformer (e.g., a standard LLaMA-style architecture at 1B–7B scale) with Baseline vs. HC vs. mHC would answer: (a) whether mHC's performance gains over the baseline persist in the absence of expert routing — if they shrink substantially, it suggests the benefits are partially attributable to complementarity with MoE rather than being a universal property of multi-stream residuals; (b) whether HC's instability manifests at similar scales in dense models or is exacerbated/ameliorated by the absence of expert routing (expert routing might partially regularize the residual stream by providing alternative information pathways); (c) whether the optimal expansion rate n differs for dense vs. MoE architectures. This experiment would determine the scope of mHC's applicability and identify whether the architecture is genuinely general-purpose or best suited to MoE settings.

Exploring alternative manifold constraints beyond doubly stochastic matrices — specifically, the tradeoff between expressivity and stability across different polytopes. The paper explicitly invites this in Section 6 ("the framework accommodates the exploration of diverse manifold constraints"), but provides no empirical guidance. Obvious candidates include: (a) column-stochastic only (relaxing the row sum constraint to allow some streams to receive more total signal than others, at the cost of losing backward gradient bounding), which might be useful when certain residual streams are "read-only" memory that should accumulate information but not transmit it forward equally; (b) orthogonal matrices (H^T H = I, via Cayley parameterization or Householder reflections), which provide perfect norm preservation and compositional closure but at the cost of restricting to rigid rotations — useful as a lower bound on expressivity; (c) non-negative matrices with bounded L1 norm (a weaker constraint than double stochasticity that still prevents unbounded amplification but allows asymmetric row/column sums); (d) learned projections onto the Birkhoff polytope with different entropic regularizations (varying the exponentiation temperature in Sinkhorn-Knopp, or using Gumbel-Softmax for discrete permutation sampling). A systematic comparison at a fixed scale (e.g., 3B parameters) measuring: (i) composite Amax Gain Magnitude vs. depth, (ii) downstream benchmark performance on the same suite as Table 4, (iii) training stability (loss curves, gradient norms), and (iv) computational cost of the projection operation, would produce a "stability-expressivity Pareto frontier" that guides future architecture design. The null hypothesis — that doubly stochasticity is near-optimal and alternatives provide no additional benefit — would itself be a valuable finding, confirming the paper's implicit claim.

Developing cheap, online difficulty estimation to make the compute-optimal framework deployable. The paper's largest unaddressed practical gap is that difficulty estimation via 2048 samples per question costs more than the test-time budget being optimized. However, mHC's stability guarantees — specifically, the bounded composite mapping norm and the well-conditioned signal propagation — suggest a new approach: difficulty estimation from internal representations rather than output samples. The key insight is that mHC's residual stream maintains stable, bounded activations across depth (Figure 7), which means that features extracted from intermediate layers should be better-behaved and more informative than in standard architectures where signal magnitudes can vary unpredictably. A concrete experiment: train a lightweight difficulty predictor (a small MLP or linear probe) that takes as input the concatenated hidden states from a few representative layers of an mHC-pretrained model and predicts the oracle difficulty bin (the pass@1 quintile). If this predictor achieves reasonable accuracy (e.g., >80% bin classification accuracy) using only a single forward pass (no sampling), it would eliminate the 2048-sample estimation cost entirely. The research question is: does mHC's stable signal propagation make internal representations more predictive of downstream task difficulty than standard residual connections? A comparison of predictor accuracy when trained on mHC vs. baseline hidden states, at various layer depths and predictor complexities, would determine whether architectural stability translates to better difficulty estimability — a non-obvious benefit of the manifold constraint.

Combining mHC with verifier-guided search or self-consistency decoding to test whether stable residual mixing improves test-time reasoning. The paper evaluates mHC only through standard pretraining loss and downstream benchmark accuracy (greedy or few-shot evaluation). However, the enriched residual stream — which can carry multiple "hypotheses" in parallel across the n streams — is architecturally well-suited to test-time reasoning strategies that benefit from maintaining diverse intermediate representations. A concrete experiment: take the pretrained 27B mHC and baseline models, apply best-of-N sampling with a trained process reward model (PRM) or outcome reward model (ORM) on a reasoning benchmark like MATH or GSM8K, and measure whether mHC's multi-stream architecture provides larger gains from increased sampling than the baseline's single-stream architecture. The hypothesis is that mHC's n=4 streams naturally maintain diverse solution approaches across depth (the different streams can specialize to different reasoning strategies, as Figure 8's learned matrices show depth-dependent routing), which means that a single forward pass already captures some of the diversity that would require multiple samples in a standard architecture. If mHC achieves higher pass@k for the same k, or achieves equivalent pass@1 performance with fewer samples, it would demonstrate a new benefit of multi-stream residuals: improved sample efficiency for test-time compute strategies. Conversely, a null result (mHC and baseline benefit equally from sampling) would suggest that the stream diversity is used for representation enrichment rather than hypothesis diversity, clarifying the role of the learned mixing.

Investigating whether the flattening modification (Equation 7) or the manifold constraint drives mHC's performance gains — a necessary disentanglement ablation. The paper introduces two changes relative to HC simultaneously: (1) flattening the n×C hidden matrix into a 1×nC vector before coefficient computation (providing cross-stream context), and (2) projecting Hres_l onto the Birkhoff polytope. The stability improvement is clearly attributable to (2), but the performance improvement over HC on 6 of 8 benchmarks (Table 4) could derive from (1), (2), or their interaction. A straightforward ablation: train four configurations at a moderate scale (e.g., 3B): (a) HC as originally formulated (Equation 5, no flattening, tanh activation, unconstrained Hres_l), (b) HC with flattening but no manifold constraint (Equation 7 parameterization but without the Sinkhorn-Knopp projection — i.e., use the raw H̃res_l directly, which may still be unstable but at small scales might train), (c) mHC without flattening (keep HC's per-stream normalization and tanh but project Hres_l onto the Birkhoff polytope), and (d) full mHC (flattening + manifold constraint). Comparing (b) vs. (a) isolates the flattening effect; comparing (c) vs. (a) isolates the manifold constraint effect; comparing (d) vs. (c) isolates whether flattening adds anything on top of the constraint. If (c) already matches (d) in performance and stability, the flattening can be dropped (saving parameters in φres_l and simplifying the implementation). If (b) is unstable even at 3B (suggesting the flattening exacerbates instability by making Hres_l more input-sensitive), it provides evidence that the manifold constraint is necessary to enable the flattening's expressivity benefits — a stronger claim about their relationship.

Practical Applications and Downstream Use Cases

Training large-scale MoE language models with improved stability and downstream reasoning performance at minimal overhead. The most direct application of mHC is as a drop-in replacement for standard residual connections in MoE Transformer training, where the 6.7% overhead (at n=4) is small enough to be acceptable in production training budgets. The concrete benefit: on reasoning-intensive benchmarks, mHC delivers improvements of 4.0 points on MATH (22.0→26.0, an 18% relative gain) and 7.1 points on GSM8K (46.7→53.8) over the standard residual baseline at 27B scale (Table 4), while maintaining stable training that HC cannot provide (Figure 5a). For organizations training models in the 10B–100B parameter range — a common scale for deployment-focused models where full 671B-scale training is uneconomical — these improvements represent a meaningful capability upgrade at a cost that is primarily engineering (implementing the Sinkhorn-Knopp kernel and recomputation strategy) rather than compute. The infrastructure optimizations described in Section 4.3 (kernel fusion via TileLang, selective recomputation aligned with pipeline stages, DualPipe communication overlap) provide a template that can be adapted to other training frameworks, though the specific 6.7% figure depends on the DeepSeek training stack and may vary with different infrastructure.

Architectural regularization for long-context or deep models where signal propagation stability is a known bottleneck. A deployment scenario the paper does not explicitly discuss but its results naturally support: training Transformers with unusually many layers (e.g., 100+ layers) or processing very long sequences where the effective depth of gradient propagation is high. The standard residual connection provides stable signal propagation in principle, but in practice, very deep standard Transformers can still experience optimization difficulties because the additive contributions from hundreds of layers can accumulate scale despite the identity shortcut. mHC's doubly stochastic constraint provides an additional regularization: the composite residual mapping contracts toward uniform mixing over depth (as Figure 8's composite matrices show, the entries converge toward 1/n), which acts as a form of depth-dependent feature smoothing — early layers can maintain specialized streams, while deeper layers gradually fuse information. This property is theoretically appealing for architectures where the effective depth (layers × sequence length for recurrent computations) is large, because it prevents any single stream from accumulating unbounded influence. A practitioner training a 100-layer dense Transformer for long-document modeling could adopt mHC expecting more stable optimization than either standard residuals (which provide no cross-stream mixing) or HC (which would be unstable at that depth). The paper's scaling curves (Figure 6) showing mHC's advantage persisting from 3B to 27B with only marginal attenuation suggest the benefit would continue at greater depths, though this extrapolation is speculative without direct experiments.

Improving sample efficiency in self-improvement or iterative refinement pipelines where models repeatedly process their own outputs. The paper demonstrates that mHC maintains stable signal propagation across depth (Figure 7), but an underexplored implication is that this stability should also hold across recurrent depth — when the same model is applied iteratively to its own outputs (as in self-refinement, chain-of-thought with multiple rounds, or recursive self-improvement). In such settings, the hidden state from iteration t becomes the input to iteration t+1, and the effective "depth" is the product of model layers and refinement iterations. The standard residual connection's identity mapping ensures stability across layers within one forward pass, but across iterations, the state can drift arbitrarily. mHC's doubly stochastic constraint on Hres_l, if extended to a recurrent setting (where the same Hres matrices are applied at each iteration), would guarantee that the hidden state does not diverge across iterations for the same reason it does not diverge across layers: the composite mapping across iterations remains doubly stochastic. A concrete use case: generating training data via iterative refinement, where a model produces an answer, critiques it, and revises it over multiple rounds. Using mHC for the base model would ensure that the hidden representations remain well-conditioned throughout the multi-round process, potentially improving the quality of later-round outputs compared to a standard residual model where representations may degrade. The paper does not test this, but the mathematical property (compositional closure of doubly stochastic matrices under any number of multiplications, not just within a forward pass) directly supports it.

When to Prefer This Method

The paper does not present mHC as one option among named, directly comparable alternatives with explicit tradeoffs. It positions mHC as a strict improvement over HC — same architecture, same parameterization, same computational budget (up to the 6.7% infrastructure-optimized overhead), but with guaranteed stability where HC is unstable at scale. The comparison to the standard residual connection baseline is similarly one-sided: mHC provides consistent downstream performance improvements (Table 4) with acceptable overhead, making it preferable in any setting where the 6.7% training time cost is acceptable. The paper does not identify a regime where HC outperforms mHC (HC's slight edge on MATH, 26.4 vs. 26.0, is within plausible single-run variance and does not constitute a tradeoff), nor does it identify conditions where the standard baseline is preferable (mHC outperforms it on all eight benchmarks). There is no accuracy-efficiency tradeoff to adjudicate — mHC is simply better on both axes than HC (stable where HC is unstable, more accurate on most tasks) and better on accuracy than the standard baseline at small overhead cost. Forcing a decision matrix of the form "Prefer mHC when X, prefer HC when Y" would fabricate a tradeoff the paper does not establish. The only practical decision is whether the 6.7% training overhead is acceptable for the deployment — if yes, mHC dominates both alternatives; if no, the standard residual connection is the fallback (HC is not a viable fallback because of its instability at scale). The paper does not explore configurations that would change this dominance relationship (e.g., different expansion rates, different manifolds, different training regimes), so a conditional recommendation matrix would be speculative rather than paper-grounded.