Virtual Width Networks

ArXiv: 2511.11238

🎯 Pitch

VWN expands a model's internal representation 8× while keeping compute nearly flat by using learned linear-attention-style routing over depth—delivering over 2× faster optimization and a log-linear scaling law where every doubling of width drops the loss by ~0.007.

---

1. Executive Summary

This paper introduces Virtual Width Networks (VWN), a framework that decouples representational width from backbone width to capture the benefits of wider representations without incurring the quadratic compute cost of increasing hidden size. The approach is evaluated on internal Mixture-of-Experts Transformer models trained on large-scale corpora, using Generalized Hyper-Connections (GHC) — a learned, fixed-cost routing mechanism that compresses over-width embeddings to backbone width before each attention or FFN sublayer and expands outputs back (functioning as a linear-attention-like carry/forget operator over depth) — paired with multi-token prediction objectives. On a 3.3B-activation MoE model, an 8× virtual width expansion accelerates optimization by more than 2× for next-token prediction and more than 3× for next-2-token prediction, with the efficiency advantage amplifying over training as both the loss gap and convergence-speedup ratio grow. A log-linear scaling relation between the virtual width factor and loss reduction is identified (each doubling of width yields approximately a 0.0069 loss decrease), establishing that virtual width can serve as a predictable scaling dimension complementary to depth, width, and data scaling laws — though practical deployability currently favors modest expansions in the 1.5×–4× range given hardware constraints on very wide activations and cross-device routing.

2. Context and Motivation

The Core Problem: The Hidden Dimension Bottleneck

The fundamental problem this paper addresses is a structural tension in Transformer architecture design: wider hidden dimensions produce more capable models, but scaling width incurs quadratic growth in compute and parameters. This tension creates a hard tradeoff — practitioners must choose between model quality and computational feasibility, and the quadratic cost function means that each incremental improvement in representational capacity becomes progressively more expensive.

This is not merely a theoretical concern. The hidden dimension $D$ determines the dimensionality of every intermediate representation throughout the network: token embeddings, attention queries/keys/values, feed-forward network activations, and residual stream states. In a standard $L$-layer Transformer with width $D$, the dominant computational complexity is $\mathcal{O}(D^2)$ per token per layer (Section 3.1). This means that doubling the width — which one might naively expect to roughly double representational capacity — actually quadruples the FLOPs. Training a model that is twice as wide is not twice as expensive; it is four times as expensive. At the scales where modern LLMs operate (models with hidden dimensions in the thousands to tens of thousands), this quadratic penalty makes naïve width scaling economically prohibitive.

The paper frames this concretely in Section 3.1:

"scaling the model width $D$ results in a quadratic increase in computational cost"

Yet the empirical evidence from scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) and practical experience consistently shows that wider models learn richer representations. A wider hidden state can encode more features per token, maintain more nuanced contextual information, and support more complex compositional operations. The question, then, is whether there exists a way to achieve the benefits of width scaling while breaking the quadratic cost relationship — that is, can we get "wider" representations without actually widening the backbone?

Why This Problem Matters

The significance of this problem spans both theoretical and practical dimensions.

Theoretically, the hidden dimension bottleneck represents a missing piece in our understanding of model scaling. The field has developed reasonably mature scaling frameworks along several axes: depth (more layers), width (larger hidden dimensions), data (more training tokens), and parameter count (more total parameters, e.g., via MoE). Each of these has associated scaling laws that characterize how performance improves with investment. However, all existing width-scaling approaches tie representational width directly to computational width — if you want a $D$-dimensional representation at every layer, you pay $\mathcal{O}(D^2)$ at every layer. The possibility of decoupling representational capacity from computational cost would add a fundamentally new degree of freedom to the scaling landscape, one where the quality-per-FLOP frontier could shift substantially.

Practically, the quadratic cost of width scaling imposes hard constraints on model design under resource budgets. Consider a typical scenario: a team has the compute budget to train a model with hidden dimension $D = 4096$. Doubling to $D = 8192$ would quadruple the attention and FFN costs, requiring either 4× the hardware, 4× the training time, or some combination thereof. For many organizations, this is simply infeasible. The result is that models are trained at widths below what would be representationally optimal, purely due to the quadratic FLOPs penalty. If that penalty could be circumvented — if wider representations could be achieved at near-constant backbone cost — the effective quality ceiling for a given compute budget would rise.

The paper's framing in Section 1 captures this motivation directly:

"naively increasing hidden dimensions leads to quadratic growth in parameters and compute, posing challenges in resource-constrained settings"

This is not an abstract concern. In production LLM deployments, the gap between what is representationally desirable and what is computationally feasible directly impacts downstream task performance, inference latency, and serving costs. A method that could deliver, say, 8× wider representations with minimal additional FLOPs would, if effective, represent a step-change in the efficiency-quality Pareto frontier.

Where Prior Approaches Fall Short

The paper situates its contribution against several lines of prior work, each of which addresses aspects of the width-scaling problem but leaves a specific gap unfilled.

Mixture-of-Experts (MoE) expands the wrong dimension. MoE architectures (Shazeer et al., 2017; Lepikhin et al., 2020; Fedus et al., 2022) are perhaps the most successful approach to scaling model capacity without proportionally scaling compute. By replacing the dense feed-forward network with a set of experts and routing each token to only a subset, MoE increases total parameters while keeping per-token FLOPs roughly constant. The paper explicitly acknowledges this as a valid strategy for scaling capacity:

"MoE models significantly improve throughput and enable efficient scaling to very large model sizes, without proportionally increasing per-token computational cost"

However, the paper identifies a critical limitation: MoE expands only the inner dimension of the FFN, while the backbone hidden dimension remains fixed. The attention layers, the residual stream, the layer normalization — all operate at the original width $D$. This means that even a massive MoE model with trillions of total parameters still represents each token as a $D$-dimensional vector throughout the attention and routing pathways. The paper states this limitation explicitly:

"conventional MoE architectures can be viewed as expanding only the inner dimension of the feed-forward networks (FFN), while the backbone hidden dimension remains fixed. Consequently, the model's representational capacity is still bottlenecked by the hidden dimension"

This is a precise diagnosis: MoE increases the model's ability to process information (via more FFN capacity) but does not increase its ability to represent information (the hidden state dimensionality is unchanged). The distinction between processing capacity and representational capacity is subtle but important — the former determines how many computations can be applied to a token, while the latter determines how much information can be packed into the token's vector representation and carried forward through the network. MoE solves the processing bottleneck but leaves the representational bottleneck untouched.

Direct width scaling is computationally prohibitive. The most obvious solution — just make the hidden dimension larger — works in terms of quality but fails in terms of cost. The quadratic scaling of attention and FFN operations with $D$ means that even modest width increases (e.g., 1.5×) carry substantial compute penalties. The paper uses this as the baseline strawman: naïvely scaling both embedding and backbone dimensions proportionally (Figure 2b). This approach achieves wider representations but at exactly the quadratic cost the paper seeks to avoid.

Hyper-Connections and AltUp are limited instantiations of the idea. The paper identifies two prior methods that begin to explore the space of decoupling representational width from backbone width:

• Hyper-Connections (HC) (Zhu et al., 2024): Introduces a mechanism where the hidden dimension is expanded through low-cost compositional links across layers, with each extension updated using a few scalar weights. The paper notes that while HC enhances expressiveness, it "often under-utilizes the expanded representations, since each extension is updated using only a few scalar weights, making it difficult to fully exploit the additional capacity" (Section 2). The limitation is in the routing mechanism's expressivity — scalar weights provide limited control over how the expanded capacity is allocated, leaving much of the additional representational space effectively unused.

• AltUp (Baykal et al., 2023): Alternates between up-projection and down-projection to work in a wider space part of the time. The paper views this as another simplified instance within the broader VWN family, but one with a fixed alternation pattern rather than learned, content-dependent routing.

The paper positions both HC and AltUp as simplified instances within the broader VWN family (Section 1), implying that they capture part of the idea but not its full potential. The key missing element is a routing mechanism with sufficient expressivity to actually utilize the expanded capacity — not just create it.

Frac-Connections takes the opposite approach. Frac-Connections (FC) (Zhu et al., 2025) addresses a related but distinct problem: instead of enlarging the hidden size, it partitions the existing hidden dimension into smaller segments to create HC-like connectivity without increasing width. This is the inverse approach — rather than expanding capacity, it reorganizes existing capacity. The paper acknowledges FC as an influence but notes that its approach (shrinking per-segment dimension) is orthogonal to VWN's approach (expanding total dimension while keeping backbone computation fixed).

Multi-token prediction as an under-explored synergy. The paper also draws on recent work in multi-token prediction (MTP) (Gloeckle et al., 2024) and over-tokenization (Huang et al., 2025). These works demonstrated that predicting multiple future tokens — rather than just the next token — provides richer training signals and improves downstream performance. The Over-Tokenized Transformer framework introduced Over-Encoding (scaling input representations via multi-gram tokenization) and Over-Decoding (enhancing output supervision via MTP). The paper positions VWN as a natural complement to MTP: wider representations provide more degrees of freedom for the compositional modeling that MTP demands, while MTP's denser supervision helps the model actually utilize the expanded capacity. However, this synergy had not been explored prior to VWN; MTP was studied with standard-width models, and width expansion was studied without MTP.

No systematic scaling law for virtual width. Perhaps most critically, prior work had not established any scaling relationship for virtual width. Scaling laws are well-characterized for model parameters, training tokens, and compute (Kaplan et al., 2020; Hoffmann et al., 2022), and even for aspects like vocabulary size (Tao et al., 2024). But no prior work had asked: if we expand the embedding dimension while keeping the backbone fixed, how does performance scale with the expansion factor? Is it linear? Logarithmic? Does it saturate? The absence of such a characterization meant that virtual width expansion was an ad-hoc design choice rather than a principled scaling dimension. Without a scaling law, practitioners cannot predict how much benefit to expect from a given expansion factor, and cannot perform cost-benefit analysis to determine the optimal expansion ratio for a given budget.

How This Paper Positions Itself

The paper's positioning can be understood as filling the specific gap between "we know wider is better" and "we know how to scale compute-efficiently" with a mechanism that achieves wider representations without the quadratic penalty, combined with an empirical characterization of how the benefits scale.

The central insight is deceptively simple but has non-obvious implications. The key observation is that the embedding lookup operation — mapping token IDs to dense vectors — represents only a tiny fraction of total FLOPs in a Transformer. This means the input embedding dimension can be expanded dramatically without proportionally increasing compute, as long as the subsequent layers can process that wider representation without themselves being widened. The challenge then becomes: how do you feed an over-width representation through standard-width attention and FFN layers without either (a) losing the extra information (by simply projecting down and discarding it) or (b) incurring the quadratic cost (by widening the layers too)?

The paper's answer is Generalized Hyper-Connections (GHC), which the paper frames as a unification and generalization of prior connection mechanisms (Hyper-Connections, Frac-Connections). GHC acts as a learned, fixed-cost router that:

1. Compresses the over-width hidden states to backbone width before each attention/FFN sublayer (so the expensive operations run at the original, cheaper dimension).
2. Expands the sublayer outputs back to the over-width dimension.
3. Maintains a "depth cache" of information across layers via carry/forget operators, allowing information to persist through the network without being bottlenecked by the backbone width at any single layer.

This is formalized through the connectivity perspective in Section 4, where VWN is reinterpreted as implementing a linear-attention-like mechanism over the depth axis — each layer can attend to (and selectively retain) information from many previous layers, with the total memory budget controlled by the virtual width factor $r = n/m$.

The paper positions VWN not as a replacement for existing scaling methods but as a complementary dimension. MoE addresses the processing bottleneck (more FFN capacity per token); VWN addresses the representational bottleneck (wider hidden states). Depth scaling adds more layers; VWN adds more representational capacity per layer. Data scaling adds more training examples; VWN makes each example more information-rich. The paper explicitly frames virtual width as "a new dimension for scaling large models" (Section 1) that can be combined with existing approaches — indeed, all experiments use MoE backbones, demonstrating that VWN and MoE are complementary rather than competing.

The scaling law contribution is a deliberate effort to make virtual width a principled design choice. By demonstrating the log-linear relationship between virtual width factor $r$ and loss reduction (Figure 8, Section 5.2.1), the paper provides the first empirical basis for treating virtual width as a predictable scaling dimension alongside depth, width, and data. The fitted coefficient of $-0.0069$ per doubling of $r$ provides a concrete estimate that practitioners can use for cost-benefit analysis: doubling the virtual width reduces loss by approximately 0.007, which can be weighed against the modest additional FLOPs and memory cost of the wider embeddings and GHC routing.

The paper is candid about practical limitations, which strengthens rather than weakens its positioning. Section 6 explicitly acknowledges that very large virtual width expansions (8× and beyond) face practical deployment constraints due to communication and memory-access overheads on current hardware, and that the 1.5×–4× range is more immediately practical. This positions VWN not as a universal solution that makes width scaling free, but as a mechanism that shifts the Pareto frontier — you can get substantially wider representations at modest additional cost, but the extreme end of the expansion spectrum still requires hardware/software co-design to fully realize. This honest boundary-drawing distinguishes the paper from over-claimed results and provides a clear roadmap for what needs to improve (software stacks, memory layouts, interconnect strategies) for larger expansions to become practical.

In summary, the paper addresses a specific, well-motivated gap: the quadratic cost of width scaling prevents models from achieving the representational capacity they could benefit from, existing approaches either expand the wrong dimension (MoE) or provide insufficient routing expressivity (HC/AltUp), and no prior work had established whether virtual width expansion could be treated as a principled, predictable scaling dimension. VWN proposes to fill this gap with a learnable routing mechanism (GHC) that enables substantial virtual width expansion at near-constant backbone cost, and provides the first scaling law characterization to guide practical adoption.

3. Technical Approach

This is primarily a systems architecture paper whose core idea is that the representational benefits of wider Transformer hidden dimensions can be achieved without the quadratic computational cost, by expanding only the embedding space and routing the wider representations through standard-width backbone layers using a learned, fixed-cost mechanism.

3.1 Reader Orientation

VWN is a modification to the standard Transformer architecture that makes the token embeddings wider while keeping the expensive attention and feed-forward layers at their original width, using a learned routing mechanism called Generalized Hyper-Connections to compress, process, and expand representations at each layer. The system solves the problem that wider hidden dimensions improve model quality but incur quadratic growth in FLOPs — VWN breaks this coupling by treating representational width (how much information can be stored per token) as independent from backbone width (the dimension at which attention and FFN operations execute), achieving substantial effective width expansion with minimal additional compute.

3.2 Big-Picture Architecture (Diagram in Words)

The VWN architecture has five major components arranged in a standard Transformer stack, but with modified data flow at each layer:

1. Over-Width Embedding — the input token embedding is expanded from the standard hidden dimension $D$ to a wider dimension $D'$, where $D' = (n/m)D$ with integers $n > m$. This creates a richer initial token representation at negligible additional cost since embedding lookup is a tiny fraction of total FLOPs.

2. Generalized Hyper-Connections (GHC) — at each Transformer layer, a learned routing module takes the over-width hidden states (dimension $D'$) and performs two operations: a width connection that compresses the over-width states down to backbone width $D$ before feeding them into the attention or FFN sublayer, and a depth connection that expands the sublayer output back to over-width dimension $D'$ and mixes it with information carried forward from previous layers via learned carry/forget operators.

3. Standard-Width Backbone Layers — the attention and feed-forward sublayers operate exactly as in a standard Transformer, at the original backbone width $D$. They never "see" the over-width dimension — GHC handles all compression and expansion. This is where the computational savings come from: the $\mathcal{O}(D^2)$ operations run unchanged.

4. Reduce Operator — after the final Transformer layer, a learned linear projection maps the over-width hidden states (still at dimension $D'$) back down to the original width $D$, producing a standard-width representation that feeds into the unembedding layer for token prediction. Group normalization is applied before this projection to stabilize training when $D'$ is large.

5. Multi-Token Prediction Head — additional VWN layers stacked on top of the backbone predict not just the next token but the next several tokens, providing denser supervision that helps the model utilize the expanded representational capacity. A block-level linear mixing mechanism keeps the cost manageable even at large expansion ratios.

Information flows as follows: token IDs → over-width embedding lookup → (for each layer) GHC width connection compresses to $D$ → standard attention/FFN at $D$ → GHC depth connection expands back to $D'$ and mixes with depth cache → (after final layer) GroupNorm → Reduce projection to $D$ → unembedding → output logits. Simultaneously, the MTP head takes the final over-width hidden state, concatenates it with the next token's embedding, and predicts subsequent tokens through additional VWN layers.

3.3 Roadmap for the Deep Dive

• First, the core concept of decoupling embedding width from backbone width (Section 3.1), including why this is possible — the embedding lookup is cheap, but the backbone layers are expensive, so we can expand the former without proportionally expanding the latter.
• Second, the Over-Width Embedding mechanism (Section 3.2) and the segmentation scheme that partitions hidden states into $m$ backbone segments and $n$ over-width segments, establishing the notation and dimensional relationships.
• Third, the Generalized Hyper-Connections (Section 3.3) in full detail — the static and dynamic routing matrices, the width connection (compression and routing into the backbone), the depth connection (expansion and mixing with accumulated depth information), and the initialization scheme that ensures stable early training.
• Fourth, the Multi-Token Prediction integration (Section 3.4), including the block-level linear mixing that prevents the MTP head cost from scaling with the expansion ratio $r$.
• Fifth, the cost analysis (Section 3.5) quantifying the FLOPs and memory overhead of VWN relative to a standard Transformer, establishing that the overhead is modest (e.g., roughly 8.8% additional activation memory for a 1.5× expansion).

3.4 Detailed, Sentence-Based Technical Breakdown

The Core Insight: Decoupling Embedding Width from Backbone Width

The paper begins with a deceptively simple observation about where the computational cost lives in a Transformer. In a standard $L$-layer Transformer with hidden dimension $D$, the initial token representation $\mathbf{h}^0 \in \mathbb{R}^D$ is obtained through embedding lookup — an operation whose cost is proportional to the vocabulary size times $D$, which is negligible compared to the per-layer operations. The per-layer operations — attention and feed-forward networks — have computational complexity $\mathcal{O}(D^2)$ because they involve matrix multiplications between $D \times D$ weight matrices and $D$-dimensional vectors. This is where essentially all the FLOPs go.

The critical structural fact is that the embedding dimension and the hidden layer dimension are the same variable $D$ in a standard Transformer. There is no architectural reason they must be identical — the embedding lookup produces a $D$-dimensional vector, and the first Transformer layer expects a $D$-dimensional input. This coupling is a design convention, not a mathematical necessity.

The paper's key move is to replace this single variable $D$ with two independent variables: the embedding dimension $D'$ (which can be made large, since embedding lookup is cheap) and the backbone width $D$ (which stays moderate, since that's where the quadratic cost lives). This creates an immediate architectural challenge: how do you feed a $D'$-dimensional vector into a $D$-dimensional Transformer layer, and how do you take the $D$-dimensional output and use it to update the $D'$-dimensional representation, without either (a) simply discarding $D' - D$ dimensions of information at each layer (which would waste the expanded capacity) or (b) incurring the quadratic cost of widening the backbone (which would defeat the purpose)?

The answer has two parts: a segmentation scheme that organizes both the standard-width and over-width representations into blocks, and a routing mechanism (Generalized Hyper-Connections) that learns to compress, route, and expand information between these blocks at each layer. The segmentation scheme creates a structured interface: the hidden state is divided into $m$ backbone-facing segments of size $D/m$ and $n$ over-width virtual segments of size $D'/n$, with the backbone sublayers only ever operating on the $m$ segments (total dimension $D$) while the routing mechanism maintains information flow across all $n$ segments (total dimension $D'$).

Over-Width Embedding: Segmentation, Expansion, and Reduction

The Over-Width Embedding mechanism establishes the dimensional framework that the entire VWN architecture depends on. It defines how wide the token representations become and how that width relates to the backbone width through a structured partitioning scheme.

Partitioning the Standard Hidden State. The standard hidden state $\mathbf{h}^l \in \mathbb{R}^D$ at layer $l$ is partitioned evenly into $m$ disjoint segments, each of dimension $D/m$:

$\mathbf{h}^l = \begin{pmatrix} {\mathbf{h}^l_1}^\intercal & {\mathbf{h}^l_2}^\intercal & \dots & {\mathbf{h}^l_m}^\intercal \end{pmatrix}^\intercal$

where $\mathbf{h}^l_k \in \mathbb{R}^{D/m}$ for $k = 1, 2, \dots, m$.

This partitioning is not a learned operation — it is a structural rearrangement that views the $D$-dimensional vector as $m$ independent blocks of size $D/m$. The integer $m$ is called the fraction rate and controls how many backbone-facing segments exist. When $m = 1$, there is a single segment of size $D$ — no partitioning. When $m = 2$, the hidden state is split into two segments of size $D/2$, and so on. Each segment has the same dimensionality $D/m$, and together they sum to the total backbone width.

Defining the Over-Width Embedding. The over-width embedding $\mathbf{e} \in \mathbb{R}^{D'}$ is constructed with an expanded number of segments $n$, where $n > m$. The expanded dimension is $D' = \frac{n}{m}D$, meaning the virtual width expansion factor is $r = n/m$. Each over-width segment has dimension $D'/n = D/m$ — the same per-segment size as the backbone segments:

$\mathbf{e} = \begin{pmatrix} \mathbf{e}_1^\intercal & \mathbf{e}_2^\intercal & \dots & \mathbf{e}_n^\intercal \end{pmatrix}^\intercal, \quad \text{where } \mathbf{e}_k \in \mathbb{R}^{D'/n} \text{ with } D' = \frac{n}{m}D$

Note the crucial design choice: the per-segment dimension is identical for backbone segments and over-width segments — both are $D/m$. This is what makes GHC routing possible: the segments are interchangeable in size, and the routing mechanism only needs to learn which segments to route where, not how to change their dimensionality. The expansion is purely in the number of segments (from $m$ to $n$), not in their individual size.

For example, with $m = 2$ and $n = 8$, a backbone width $D = 4096$ yields per-segment dimension $4096/2 = 2048$, backbone segments $\mathbf{h}^l_1, \mathbf{h}^l_2$ each of size 2048, and over-width segments $\mathbf{e}_1, \dots, \mathbf{e}_8$ also each of size 2048. The total virtual width is $D' = (8/2) \times 4096 = 16384$, a 4× expansion, but every segment is the same 2048-dimensional vector.

At the input layer, the over-width embedding is used directly as the initial over-width hidden state: $\mathbf{h}'^0 = \mathbf{e}$. This means the first layer receives a representation that is $n/m$ times wider than the standard approach, with no change to the embedding lookup other than a larger output dimension.

Optional Embedding Expansion via Linear Projection. When the expansion ratio $r = n/m$ is large, directly learning a full embedding table of size $|\mathcal{V}| \times D'$ may be memory-intensive. The paper provides an optional alternative: learn a standard-width embedding table of size $|\mathcal{V}| \times D$ and expand it via a learned linear projection:

$\mathbf{E}_{\text{wide}} = \mathbf{W}_{\text{expand}} \mathbf{E}_{\text{base}}$

where $\mathbf{W}_{\text{expand}}$ maps from dimension $D$ to dimension $D'$. This is analogous to applying a low-rank decomposition to a very wide embedding table — the effective rank is constrained by the input dimension $D$, but the output can be arbitrarily wide. The paper notes that input-augmentation strategies from Huang et al. (2025) can also be applied to further enrich the widened representation by injecting more information per input than a single isolated token embedding.

The Reduce Operator. After the final Transformer layer, the over-width hidden state $\mathbf{h}'^L \in \mathbb{R}^{D'}$ must be mapped back to the original width $D$ before being fed into the unembedding layer — the vocabulary projection expects a $D$-dimensional input, not a $D'$-dimensional one. This is accomplished by a learned linear projection:

$\mathbf{h}^L_{\text{reduce}} = \mathbf{W}_{\text{reduce}} \mathbf{h}'^L$

where $\mathbf{W}_{\text{reduce}} \in \mathbb{R}^{D \times D'}$.

Why a linear projection rather than, say, average pooling or selecting the first $D$ dimensions? A learned projection allows the model to optimally aggregate information across all $n$ over-width segments when producing the final standard-width representation. Averaging would treat all segments equally; selection would discard most of the expanded capacity. The linear projection lets the model learn which combinations of virtual segments are most predictive for the downstream vocabulary projection, effectively performing a learned dimensionality reduction at the final step.

Group Normalization Before Reduction. When the expansion ratio $r$ is large, $D'$ becomes very large — for an 8× expansion of a 4096-dimensional backbone, $D' = 32768$. Normalizing a 32K-dimensional vector directly (e.g., with LayerNorm) can be unstable and computationally expensive. The paper instead uses Group Normalization (Wu and He, 2018) where the group size equals the original hidden size $D$. This means the $D'$-dimensional vector is divided into $r = D'/D$ groups, each of size $D$, and each group is normalized independently. This preserves the original $D$-dimensional normalization statistics that the backbone was designed around while handling the expanded dimensionality. For the 1.5× expansion experiments, group normalization is omitted — presumably because the expansion is small enough that direct normalization remains stable.

Why this segmentation scheme? The uniform per-segment size $D/m$ is the architectural invariant that makes Generalized Hyper-Connections feasible. Because backbone segments (from the compressed hidden state) and over-width segments (from the expanded embedding) have identical dimensionality, they can be mixed and routed using matrix operations over the segment indices without any per-segment shape transformations. The segmentation can be understood as creating a slot-based memory where each slot has the same capacity ($D/m$ dimensions) and the routing mechanism decides which slots to read from (compress) and which slots to write to (expand), with $m$ backbone-facing slots and $n$ total virtual slots.

Generalized Hyper-Connections: The Routing Mechanism

Generalized Hyper-Connections (GHC) is the central mechanism that enables VWN to function — it is the learned router that compresses over-width representations to backbone width, feeds them through standard Transformer sublayers, and expands the outputs back while maintaining a depth-wise information cache. Without GHC, the over-width embedding would simply be projected down to backbone width at the first layer, and all the extra capacity would be lost. With GHC, information can flow through the virtual segments across layers, accumulating and being refined over depth.

The GHC Matrix Structure. At each layer $l$, GHC defines a transformation matrix $\mathcal{GHC}^l \in \mathbb{R}^{2m \times (m+n)}$ that encodes all the routing decisions for that layer:

$\mathcal{GHC}^l = \begin{pmatrix} \mathbf{0} & \mathbf{B}^l \\ \mathbf{A}^l & \end{pmatrix} = \begin{pmatrix} \mathbf{0} & \mathbf{B}^l \\ \overset{\circ}{\mathbf{A}}^l & \hat{\mathbf{A}}^l \end{pmatrix}$

This matrix has a specific block structure that is worth unpacking carefully:

• The top block is $\begin{pmatrix} \mathbf{0}_{m \times m} & \mathbf{B}^l \end{pmatrix}$ where $\mathbf{B}^l \in \mathbb{R}^{m \times n}$. The $m \times m$ zero block in the top-left means that the top $m$ rows (which correspond to backbone-facing output slots) do NOT read from the backbone-facing input slots — they only read from the over-width virtual slots via $\mathbf{B}^l$. This ensures that the backbone sublayer input is constructed purely from the virtual slot information, not from a shortcut of the backbone slots.

• The bottom block is $\mathbf{A}^l \in \mathbb{R}^{n \times (m+n)}$, which is further partitioned into $\overset{\circ}{\mathbf{A}}^l \in \mathbb{R}^{n \times m}$ (the first $m$ columns) and $\hat{\mathbf{A}}^l \in \mathbb{R}^{n \times n}$ (the remaining $n$ columns). $\overset{\circ}{\mathbf{A}}^l$ controls how the backbone sublayer output (which has $m$ segments) is distributed to the $n$ virtual slots. $\hat{\mathbf{A}}^l$ controls how the previous layer's $n$ virtual slots are carried forward (or forgotten) — this is the depth-wise carry/forget operator.

This structure is not arbitrary. The top-left zero block enforces that the compression into backbone width is purely a function of the virtual representation — the backbone sublayer receives a summary constructed from the over-width state. The bottom-right $\hat{\mathbf{A}}^l$ block enables information to persist across layers in the virtual slots, creating a "depth cache" analogous to a KV cache but along the layer dimension rather than the sequence dimension.

The GHC Forward Pass. With the matrix structure defined, the forward pass of GHC at layer $l$ is:

$\mathbf{H}'^l = \mathcal{GHC}^l(\mathcal{T}^l, \mathbf{H}'^{l-1}) = {\mathbf{B}^l}^\intercal \mathcal{T}^l\left({\overset{\circ}{\mathbf{A}}^{l}}^\intercal \mathbf{H}'^{l-1}\right) + {\hat{\mathbf{A}}^l}^\intercal \mathbf{H}'^{l-1}$

where $\mathbf{H}'^{l-1} \in \mathbb{R}^{(D/m) \times n}$ is the previous layer's over-width hidden state reshaped into an $n$-column matrix (each column is a $D/m$-dimensional segment), and $\mathcal{T}^l$ is the backbone sublayer (attention or FFN).

What this equation computes, operation by operation:

1. Compression: ${\overset{\circ}{\mathbf{A}}^{l}}^\intercal \mathbf{H}'^{l-1}$ multiplies the $n \times m$ transposed routing matrix with the $n$-column virtual state, producing an $m$-column matrix. This is a learned linear combination of the $n$ virtual segments into $m$ backbone-facing segments. The result is the compressed representation that will be fed into the backbone sublayer.

2. Backbone processing: $\mathcal{T}^l\left({\overset{\circ}{\mathbf{A}}^{l}}^\intercal \mathbf{H}'^{l-1}\right)$ applies the standard Transformer sublayer (attention or FFN) to the compressed representation. This operation runs at dimension $D$ (the total of $m$ segments times $D/m$), incurring the standard $\mathcal{O}(D^2)$ cost — no wider than a normal Transformer.

3. Expansion: ${\mathbf{B}^l}^\intercal \mathcal{T}^l(\dots)$ multiplies the $m \times n$ transposed routing matrix with the $m$-column sublayer output, producing an $n$-column matrix. This is a learned linear combination that expands the backbone output back to the full virtual width, distributing information across all $n$ virtual segments.

4. Depth carry-forward: ${\hat{\mathbf{A}}^l}^\intercal \mathbf{H}'^{l-1}$ multiplies the $n \times n$ transposed carry matrix with the previous $n$-column virtual state. This passes information from the previous layer's virtual slots directly to the current layer's virtual slots, bypassing the backbone sublayer entirely. The diagonal (or near-diagonal) structure of $\hat{\mathbf{A}}^l$ determines how much of each previous virtual slot is retained versus forgotten.

5. Summation: The backbone-processed-and-expanded term and the depth carry-forward term are added element-wise to produce the new virtual state $\mathbf{H}'^l$.

Why this form? The separation into a backbone-processing path ($\mathbf{B}^l$ term) and a depth carry-forward path ($\hat{\mathbf{A}}^l$ term) is critical. If only the backbone-processing path existed without the depth carry-forward, information in virtual slots that are not routed into the backbone at a given layer would be lost — the model would have "amnesia" across layers for non-routed virtual slots. The depth carry-forward ensures that the full $n$-slot virtual state persists through the network, with each layer able to selectively retain or update information in each slot. This is analogous to how residual connections prevent information loss in standard Transformers, but operating at the segment level with learned per-slot retention rates.

The $\overset{\circ}{\mathbf{A}}^l$ matrix (compression) and $\mathbf{B}^l$ matrix (expansion) do not need to be inverses of each other — they can learn asymmetric routing where, for example, information from virtual slots 1 and 3 is compressed into backbone slot 2, and the backbone output from slot 2 is then expanded into virtual slots 5 and 7. This asymmetry enables the model to reorganize information across the virtual space at each layer, effectively implementing a learned attention-like mechanism over the segment indices.

Algorithm 1 in the paper provides the complete forward pass:

1. Over-width token embedding $\mathbf{e} \in \mathbb{R}^{D'}$ is reshaped to $\mathbf{H}'^0 \in \mathbb{R}^{(D'/n) \times n} = \mathbb{R}^{(D/m) \times n}$ — an $n$-column matrix of $m$-dimensional segments.
2. For each layer $l = 1$ to $L$:
   • $\mathbf{X}^l \leftarrow {\overset{\circ}{\mathbf{A}}^l}^\intercal \mathbf{H}'^{l-1}$ — compress $n$ virtual slots to $m$ backbone slots.
   • $\mathbf{z}^l \leftarrow \mathcal{T}^l(\texttt{Reshape}(\mathbf{X}^l, (D,)))$ — reshape to a flat $D$-dimensional vector, apply backbone sublayer.
   • $\mathbf{Z}^l \leftarrow \texttt{Reshape}(\mathbf{z}^l, (m, D/m))^\intercal$ — reshape output to $m$-column matrix.
   • $\mathbf{H}'^l \leftarrow {\mathbf{B}^l}^\intercal \mathbf{Z}^l + {\hat{\mathbf{A}}^l}^\intercal \mathbf{H}'^{l-1}$ — expand backbone output and add depth carry-forward.
3. After all layers: $\mathbf{h}^L \leftarrow \texttt{Linear}(\texttt{GroupNorm}(\mathbf{H}'^L))$ — reduce to $D$ dimensions.
4. $\mathbf{y} \leftarrow \texttt{Unembedding}(\texttt{Norm}(\mathbf{h}^L))$ — predict output tokens.

Dynamic Generalized Hyper-Connections (DGHC)

The static GHC described above uses fixed routing matrices $\mathbf{A}^l$ and $\mathbf{B}^l$ that are learned parameters of the model but do not depend on the input. This means the same routing pattern is applied regardless of what the input tokens are. To enable input-dependent routing — where the model can decide how to compress and expand based on what it's currently processing — the paper introduces Dynamic GHC (DGHC).

In DGHC, the routing matrices become functions of the input:

$\mathcal{GHC}(\mathbf{H}') = \begin{pmatrix} \mathbf{0}_{m \times m} & \mathcal{B}(\mathbf{H}') \\ \overset{\circ}{\mathcal{A}}(\mathbf{H}') & \hat{\mathcal{A}}(\mathbf{H}') \end{pmatrix}$

The dynamic parameters are generated through a lightweight linear projection network applied to the normalized over-width hidden states. The process for computing the dynamic $\mathcal{B}(\mathbf{H}')$ matrix is:

$\overline{\mathbf{H}'} = \texttt{norm}(\mathbf{H}')$

$\mathcal{B}(\mathbf{H}') = \mathbf{S}_\beta \circ \tanh\left(\frac{\overline{\mathbf{H}'} \mathbf{W}_\beta}{\tau}\right)^\top + \mathbf{B}$

where:

• $\overline{\mathbf{H}'} \in \mathbb{R}^{(D/m) \times n}$ is the input normalized per-segment (the norm is applied along the $D/m$ dimension within each segment).
• $\mathbf{W}_\beta \in \mathbb{R}^{(D/m) \times m}$ is a learned projection weight that maps each $D/m$-dimensional segment to an $m$-dimensional vector of routing coefficients.
• $\overline{\mathbf{H}'} \mathbf{W}_\beta \in \mathbb{R}^{n \times m}$ computes, for each of the $n$ virtual segments, an $m$-dimensional routing score indicating how much that segment should contribute to each of the $m$ backbone slots.
• $\tanh(\cdot / \tau)$ with $\tau = \sqrt{D/m}$ applies a scaled hyperbolic tangent activation. The scaling by $1/\sqrt{D/m}$ is a standard initialization heuristic to keep the variance of the pre-activation roughly constant regardless of segment size.
• The transpose makes the result $m \times n$, matching the shape of the static $\mathbf{B}$ matrix.
• $\mathbf{S}_\beta \in \mathbb{R}^{m \times n}$ is a learnable scaling matrix initialized to all ones, which allows the model to learn per-entry scaling of the dynamic contribution.
• The element-wise product $\circ$ with $\mathbf{S}_\beta$ and the addition of the static $\mathbf{B}$ implement a hybrid strategy where both static and input-dependent routing coexist.

The same pattern is used for $\mathcal{A}(\mathbf{H}')$:

$\mathcal{A}(\mathbf{H}') = \mathbf{S}_\alpha \circ \tanh\left(\frac{\overline{\mathbf{H}'} \mathbf{W}_\alpha}{\tau}\right) + \mathbf{A}$

where $\mathbf{W}_\alpha \in \mathbb{R}^{(D/m) \times (m+n)}$ and $\mathbf{S}_\alpha \in \mathbb{R}^{n \times (m+n)}$.

What this computes operationally: For each of the $n$ virtual segments, the model looks at the segment's content (its $D/m$-dimensional vector), passes it through a small linear projection, and produces routing coefficients that determine how that segment's information gets distributed during compression (via $\overset{\circ}{\mathcal{A}}$), how it gets carried forward (via $\hat{\mathcal{A}}$), and how backbone outputs get routed back to it (via $\mathcal{B}$). The tanh squashes the dynamic contribution to $[-1, 1]$, and the scaling matrix $\mathbf{S}$ controls the magnitude of the dynamic deviation from the static baseline.

Why the hybrid static + dynamic approach? Using purely dynamic routing (with no static $\mathbf{B}$ and $\mathbf{A}$) would mean the routing patterns are fully determined by the input at each layer. This could cause training instability early on, when the dynamic projection weights $\mathbf{W}_\beta$ and $\mathbf{W}_\alpha$ are randomly initialized and producing essentially random routing. The static matrices provide a stable, learned baseline routing pattern that ensures reasonable information flow from the start of training, while the dynamic component adds input-dependent flexibility that the model can gradually learn to use. This hybrid strategy follows the approach used in Zhu et al. (2024) and Zhu et al. (2025).

Computational cost of the dynamic component: The dynamic projection requires computing $\overline{\mathbf{H}'} \mathbf{W}_\beta$ (shape $n \times m$) and $\overline{\mathbf{H}'} \mathbf{W}_\alpha$ (shape $n \times (m+n)$), which are small matrix multiplications compared to the backbone attention and FFN operations. The paper quantifies this cost in Section 3.5: with modest settings of $m=2$ and $n=3$, the normalization, dynamic parameter calculation, and width connection steps amount to $42D$ FLOPs per token, while the depth connection requires $6D$ FLOPs — negligible compared to the $\mathcal{O}(D^2)$ backbone operations.

Initialization and Implementation

The initialization of GHC matrices is designed to ensure stable early training by providing a sensible default routing pattern before the model has learned to use the dynamic routing effectively.

Static $\mathbf{B}$ initialization. The $m \times n$ matrix $\mathbf{B}$ is initialized with a cyclic pattern:

$\mathbf{B}[i, j] = \begin{cases} 1, & \text{if } i = j \bmod m, \\ 0, & \text{otherwise} \end{cases}$

For $m = 2, n = 8$, this produces:

$\mathbf{B} = \begin{pmatrix} 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 \\ 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 \end{pmatrix}$

Why this pattern? Each backbone slot $i$ is initialized to read from virtual slots whose indices are congruent to $i$ modulo $m$ — that is, it reads from every $m$-th virtual slot. This ensures that (a) every virtual slot is read by exactly one backbone slot (no virtual slot is ignored at initialization), and (b) each backbone slot receives information from $n/m$ evenly spaced virtual slots. The cyclic pattern distributes the virtual capacity uniformly across the backbone slots, providing a balanced starting point from which the model can learn to specialize.

Static $\mathbf{A}$ initialization. The $n \times (m+n)$ matrix $\mathbf{A}$ is initialized as a block matrix with two cases:

When $n = m$ (no virtual expansion beyond partitioning): $\mathbf{A} = \begin{pmatrix} \mathbf{I}_{m \times m} & \mathbf{I}_{m \times m} & \mathbf{0}_{m \times r} \end{pmatrix}$

When $n > m$: $\mathbf{A} = \begin{pmatrix} \mathbf{I}_{m \times m} & \mathbf{I}_{m \times m} & \mathbf{0}_{m \times r} \\ \mathbf{0}_{r \times m} & \mathbf{0}_{r \times m} & \mathbf{I}_{r \times r} \end{pmatrix}$

where $r = n - m$ is the number of "extra" virtual slots beyond the backbone slot count.

What this does at initialization for the $n > m$ case (the common one):

• The top-left $m \times m$ block ($\overset{\circ}{\mathbf{A}}$) is the identity, meaning each backbone slot $i$ reads from virtual slot $i$ — a one-to-one routing.
• The top-middle $m \times m$ block is the identity, meaning each backbone slot $i$ also carries forward information from virtual slot $i$ — the backbone output is routed back to the same virtual slot it came from.
• The top-right $m \times r$ block is zero, meaning the first $m$ virtual slots do not carry forward information from the extra virtual slots $m+1$ through $n$.
• The bottom $r \times (m+n)$ block has identity in the rightmost $r \times r$ sub-block and zeros elsewhere, meaning the extra virtual slots $m+1$ through $n$ simply carry themselves forward (identity mapping) and do not initially interact with the backbone or the first $m$ virtual slots.

Why this initialization? At the start of training, the first $m$ virtual slots are initialized to behave like a standard Transformer with identity routing — information enters through slot $i$, gets processed by the backbone, and returns to slot $i$. The extra $r$ virtual slots are initialized to simply persist their content unchanged (identity carry-forward), meaning the over-width embedding information in those slots is preserved but not yet utilized. As training progresses and the dynamic routing learns, the model can begin to mix information between the backbone-facing slots and the extra slots, gradually learning to leverage the additional capacity. This initialization ensures that VWN starts with the same effective behavior as a standard Transformer (no worse performance early in training) and can only improve from there as the routing patterns are learned.

Dynamic parameter initialization. The dynamic projection weights $\mathbf{W}_\beta$ and $\mathbf{W}_\alpha$ are initialized to zero. This means that at the start of training, the dynamic component contributes nothing — the routing is purely static. The dynamic scaling matrices $\mathbf{S}_\beta$ and $\mathbf{S}_\alpha$ are initialized to all ones, so when the dynamic weights begin to move away from zero, the dynamic contributions enter at unit scale. This zero-initialization of dynamic parameters is a common stabilization technique: the model starts with the known-good static routing and gradually introduces input-dependent flexibility.

Weight decay. The static matrices $\mathbf{B}$ and $\mathbf{A}$ do not use weight decay, while the dynamic parameters do. This is because the static matrices have a specific structural role (providing baseline routing) that should not be regularized toward zero, while the dynamic parameters can benefit from regularization to prevent overfitting to spurious input-routing correlations.

Multi-Token Prediction Integration

VWN is paired with Multi-Token Prediction (MTP) to provide denser supervision that exercises the expanded representational capacity. The intuition (Section 3.4) is that MTP requires the model to maintain richer short-range compositional information — predicting the next two tokens rather than just the next one demands that the hidden state encode more about the current context — and the wider virtual representations from VWN provide more degrees of freedom for this encoding.

MTP architecture. The MTP head is implemented following the approach of DeepSeek-V3 (DeepSeek-AI, 2025): an additional stack of VWN layers is placed on top of the backbone model, and for predicting the $k$-th future token, the embedding of the $(k-1)$-th token is concatenated with the last-layer embedding of the preceding context and fed through a linear projection to produce logits. This is illustrated in the upper portion of Figure 3(c).

The cost challenge with VWN. In a standard-width model, the MTP mixing linear would map a $2D$-dimensional concatenated vector (hidden state plus next-token embedding) to a $D$-dimensional output. Under VWN with expansion factor $r$, this would naively become a $2rD \to rD$ mapping — for $r = 8$, a 16× increase in the linear layer's parameters and FLOPs compared to the $r = 1$ case. This would quickly become the dominant cost in the MTP head, undercutting VWN's efficiency advantages.

Block-level linear mixing. To avoid this cost explosion, the paper introduces a block-level linear mixing strategy. The $rD$-dimensional vectors are partitioned into $n = r \times m$ segments of size $D/m$, and the same small linear layer (mapping $2D/m \to D/m$) is applied independently to each segment:

"we partition the $rD$-dimensional vectors into $n = r \times m$ segments of size $D/m$, and apply the same small linear per segment with shape $(2D/m) \to (D/m)$... we fuse the hidden-state and embedding features locally within each segment, sharing the linear projector across all blocks."

What this means concretely: For each of the $n$ segments, the model takes the segment's portion of the hidden state ($D/m$ dimensions) and the segment's portion of the next-token embedding ($D/m$ dimensions), concatenates them locally to a $2D/m$-dimensional vector, and applies a learned linear projection to produce a $D/m$-dimensional output for that segment. The same projection weights are shared across all $n$ segments. The total cost is $n \times (2D/m \times D/m) = 2rD^2/m$ — comparable to the $r = 1$ cost of $2D^2$ when $m$ is chosen appropriately.

Why this works: The block-level mixing assumes that the fusion of hidden state and embedding information can be done locally per segment rather than globally across all segments. The global mixing (which would allow any hidden-state dimension to interact with any embedding dimension) is instead handled by the GHC routing in subsequent VWN layers of the MTP head — the GHC matrices can route information across segments after the local mixing. This is a form of factorized mixing: local fusion within segments followed by cross-segment routing via GHC, which together approximate the full dense mixing at a fraction of the cost.

Cost Analysis: FLOPs and Memory

The paper provides a detailed breakdown of the computational and memory overhead of VWN relative to a standard Transformer (Section 3.5), establishing that the additional cost is modest and primarily in operations that are not the bottleneck on modern GPU hardware.

FLOPs overhead. The dominant computational costs in a Transformer are the attention and FFN operations, which scale as $\mathcal{O}(D^2)$. VWN does not modify these operations — they run at the original backbone width $D$. The additional costs are:

• Normalization: $4\frac{n}{m}D$ FLOPs per token for RMSNorm applied to the over-width hidden states.
• Dynamic parameter calculation: $2\frac{(2m+n)n}{m}D$ FLOPs per token for computing $\mathbf{W}_\beta$ and $\mathbf{W}_\alpha$ projections.
• Width connection: $2\frac{(m+n)n}{m}D$ FLOPs for the matrix multiplications in the compression and expansion steps.
• Depth connection: $2nD$ FLOPs for the carry-forward matrix multiplication.

With modest settings of $m = 2$ and $n = 3$ (a 1.5× expansion), these sum to $42D$ (normalization + dynamics + width) plus $6D$ (depth) = $48D$ FLOPs per token per layer. Compare this to the attention cost alone, which is $\mathcal{O}(D^2)$ — for $D = 4096$, the attention FLOPs are on the order of $D^2 = 16.8$M per head, while the VWN overhead is $48 \times 4096 \approx 197$K, or roughly 1% of the attention cost.

Why this overhead structure matters: The additional FLOPs are in memory-bound operations (normalization, small matrix multiplications over segment indices) rather than compute-bound operations (large matrix multiplications). On GPUs, memory-bound operations are often not the throughput bottleneck — the GPU's compute units may be idle while waiting for data from memory. This means the practical wall-clock overhead of VWN can be even lower than the raw FLOPs count suggests, especially when the normalization, dynamic parameter calculation, and width connection are fused into a single GPU kernel (as the paper does to "minimize I/O").

Memory overhead during training. During training, intermediate activations must be stored for backpropagation. A vanilla Transformer layer using selective activation recomputation (Korthikanti et al., 2023) requires approximately $34D$ bytes of activation storage per token (assuming 16-bit floats). VWN adds the cost of saving inputs to the $\mathbf{A}$ and $\mathbf{B}$ routing matrices for gradient computation.

The paper's analysis: with $m = 2$, $n = 3$, and a saving ratio $\eta = 0.5$ (saving the width connection input for attention but recomputing it for FFN), the extra memory is $4\eta\frac{n}{m}D = 4 \times 0.5 \times (3/2) \times D = 3D$ bytes. This is approximately 8.8% of the vanilla Transformer's activation memory — a modest increase.

Why recomputation helps: The inputs to the width connection can be recomputed from the depth connection input at low cost (a small matrix multiplication), so the paper's strategy of saving only some width connection inputs and recomputing others reduces the memory footprint without a significant FLOPs penalty. The parameter $\eta$ controls this trade-off: $\eta = 1$ saves all width connection inputs (higher memory, lower recomputation), while $\eta = 0$ saves none (lowest memory, most recomputation). The paper's choice of $\eta = 0.5$ balances the two.

Inference memory. During inference, the additional memory comes only from the extra parameters (the static and dynamic GHC matrices, the expand and reduce projections, the MTP head parameters). The paper notes this is "a negligible amount compared to other memory consumption" — the KV cache, which scales with batch size, sequence length, and number of layers, dominates inference memory, and VWN does not affect the KV cache size since the backbone dimension $D$ is unchanged.

Practical limitations acknowledged. The paper is candid that for very large expansion ratios ($r = 8$ and beyond), communication and memory-access overheads become non-trivial, and "contemporary hardware is not particularly friendly to very wide activations and cross-device routing" (Section 6). The 1.5×–4× range is identified as more immediately practical. This honesty about deployment constraints provides important context: the theoretical FLOPs savings are real, but translating them to wall-clock speedups requires engineering effort to handle the wide activation layouts and inter-device communication patterns that large $r$ values introduce.

---

This completes the Technical Approach section. The remaining sections (Connectivity Perspective, Experiments, etc.) are not included per the instructions.

4. Key Insights and Innovations

Innovation 1: Virtual Width as an Independent, Predictable Scaling Dimension

The paper's most fundamental contribution is conceptual: it establishes virtual width — the capacity to expand representational dimensionality independently of computational dimensionality — as a new, predictable axis for scaling model efficiency, complementary to depth, width, and data scaling. This is not merely a new architectural trick; it is a reframing of what "width" means in a Transformer.

Prior to this work, "scaling width" meant increasing the hidden dimension $D$, which carried an inescapable quadratic cost in both attention and FFN layers. The scaling laws community (Kaplan et al., 2020; Hoffmann et al., 2022) had characterized how performance improves with model size and data, but width was treated as a monolithic variable: wider models are better, but quadratically more expensive. This created an implicit assumption — embedded in every standard Transformer implementation — that the dimension of the token representation and the dimension at which operations execute are the same thing. They are not required to be, but the architecture made them so.

VWN breaks this coupling. By treating the embedding dimension $D'$ and the backbone dimension $D$ as independent variables, the paper demonstrates that representational capacity can be scaled along a new axis — the virtual width factor $r = D'/D$ — with costs that grow sub-quadratically (roughly linearly in the additional normalization and routing operations, as shown in Section 3.5). This is a fundamental shift in the design space: prior work expanded either the processing capacity (MoE, by increasing FFN parameters per token) or the backbone width (naïve scaling, at quadratic cost). Neither approach recognized that representational capacity could be expanded through the embedding and routing layers alone, leaving the expensive operations untouched.

What elevates this from architectural cleverness to scientific contribution is the scaling law characterization in Section 5.2.1 and Figure 8. The paper demonstrates an approximately log-linear relationship between the virtual width factor $r$ and loss reduction, with the fitted relation $y = -0.0069 \cdot \log_2(x) + 1.6212$ and $R^2 = 0.9986$. Each doubling of virtual width yields approximately a 0.0069 loss decrease. While the effect size is modest, the existence of a clean, predictable scaling relationship — with monotonic improvements from $r = 2$ to $r = 4$ to $r = 8$ at fixed $m = 8$ (Table 1) — means that virtual width is not an ad-hoc knob but a principled design dimension. Practitioners can now perform cost-benefit analysis: "if I double the virtual width, I can expect roughly 0.007 loss improvement at the cost of X FLOPs and Y memory," directly analogous to how the Chinchilla scaling laws guide pretraining compute allocation.

This is not incremental because it opens an entirely new dimension for scaling that did not previously exist in the conceptual toolkit. The $R^2 = 0.9986$ fit in Figure 8 suggests the relationship is not noisy — it behaves like a law, not a trend. The ablation in Figure 6 further shows that the benefit is robust to the choice of fraction rate $m$ once $m > 4$, indicating the scaling relationship is primarily a function of $r$ rather than the specific $(m, n)$ parameterization. This is the hallmark of a genuine scaling dimension: the outcome depends predictably on the aggregate quantity ($r$) rather than on implementation details.

Innovation 2: The Connectivity Perspective — Depth as an Attention Axis

The paper introduces a novel interpretive framework in Section 4 that reframes VWN's routing mechanism as attention along the depth axis, where layers are treated as positions in a sequence and each layer maintains a compact "depth KV cache" of information from previous layers. This is not a new mechanism — the mechanism is GHC, detailed in Section 3.3 — but rather a diagnostic lens that reveals why the mechanism works and how to configure it.

The key move is Equation 17, which unrolls the GHC recurrence to show that the hidden state at layer $l$ is a linear combination of backbone-transformed features from all previous layers, weighted by products of the carry matrices $\hat{\mathbf{A}}$ and written via $\mathbf{B}$ at each step. This is structurally identical to linear attention over a compressed representation of past layers — the "query" is implicit in the learned routing matrices, the "keys" are the backbone-processed features, and the "values" are written into the current virtual state. Prior work had described residual connections as enabling information flow across depth, and dense connectivity (DenseFormer, Ma et al., 2023; Huang et al., 2017) as expanding the receptive field. But no prior work had formulated depth-wise connectivity as a learnable, fixed-cost attention mechanism with explicit control over the memory budget and the fidelity-layers tradeoff.

The practical value of this framing is that it provides a principled way to choose the hyperparameters $m$ and $n$. The paper shows that the total memory budget for depth information is $r = n/m$, measured in $D$-units, and that $m$ controls a fundamental tradeoff:

• With $m = 1$, the model stores up to $r$ layers at full $D$-dimensional fidelity (fewer layers, higher bandwidth per layer).
• With $m > 1$, the model stores up to $n = rm$ layers, each compressed to $D/m$ dimensions (more layers, lower bandwidth per layer).

This reframes the configuration problem: rather than treating $(m, n)$ as arbitrary hyperparameters to be tuned by grid search, one can reason about the depth receptive field and per-layer fidelity needed for a given model scale. The paper's hypothesis — that larger models need larger $m$ to maintain sufficient per-layer bandwidth for their deeper stacks, while smaller models saturate at low $m$ — is supported by the ablation in Figure 6, where $m = 4$ is sufficient for the 0.8B-activation model. This is a conceptual advance because it transforms VWN configuration from an empirical tuning exercise into a design problem with clear tradeoffs and scaling intuitions.

The distinction between hard routing (binary gates producing fixed-size windows) and soft routing (real-valued, potentially input-dependent matrices producing exponentially decayed access to older layers) further enriches this perspective. Prior work on connectivity patterns — residual connections, dense connections, HC, FC — can be understood as specific points in this design space. Residual connections correspond to a window of size 2 with identity routing. Dense connections correspond to an unconstrained window. Hyper-Connections correspond to scalar-weighted routing. VWN's GHC corresponds to learned, matrix-valued routing with a controllable memory budget. This unification is valuable because it shows VWN is not an arbitrary new connection pattern but a generalization that subsumes prior approaches and makes their design choices explicit parameters.

This is fundamental rather than incremental because it provides a new language for thinking about depth-wise information flow — one that is likely to influence future architecture design beyond VWN itself. The "depth as sequence, layers as tokens, routing as attention" analogy is a productive conceptual tool that did not exist in the literature before this paper.

Innovation 3: Demonstrating That Virtual Width and Multi-Token Prediction Are Synergistic, Not Merely Additive

The paper shows that VWN and Multi-Token Prediction (MTP) exhibit synergy — their combination yields improvements greater than the sum of their individual contributions, and the advantage amplifies over training rather than saturating. This is a specific empirical finding with implications beyond the VWN architecture itself.

The evidence for synergy is most clearly seen in the large-scale experiments (Figure 1 and Section 5.2.2). On the 3.3B-activation MoE model, VWN achieves a 2.5× token efficiency gain for next-token prediction but a 3.5× gain for next-2-token prediction — the multi-token objective benefits disproportionately from virtual width expansion. Meanwhile, the loss gap relative to the baseline grows over training: the next-token loss gap increases from $\Delta = 0.025$ early to $\Delta = 0.032$ at 3T tokens, and the next-2-token loss gap grows from $\Delta = 0.049$ to $\Delta = 0.056$. These trends indicate that MTP supervision and virtual width are not merely providing independent benefits that add — they interact such that the model becomes increasingly effective at utilizing the expanded capacity as training proceeds.

Why is this non-obvious? Prior work had studied MTP (Gloeckle et al., 2024) and width expansion independently. The Over-Tokenized Transformer framework (Huang et al., 2025) had proposed Over-Encoding and Over-Decoding as complementary strategies, but this was a conceptual proposal rather than an empirical demonstration of synergy. The default assumption would be that wider representations help next-token prediction (more capacity to encode context) and that MTP helps next-token prediction (richer training signal), but there was no reason to expect the ratio of improvement to differ across objectives, nor that the synergy would compound over training. The fact that it does — that the next-2-token loss sees a 3.5× speedup versus 2.5× for next-token — suggests a specific mechanistic interaction: the wider virtual space provides "more representational degrees of freedom for short-range compositional targets" (Section 5.2.2), and MTP's denser supervision provides the training signal needed to actually organize and utilize those degrees of freedom.

The 1.5× experiments (Figures 4 and 5) provide converging evidence at smaller scale. On the 0.4B/4B MoE, MTP alone slightly hurts next-token prediction loss (Figure 4, left), which is a known phenomenon — optimizing a multi-token objective can trade off against single-token performance when the model lacks sufficient capacity to serve both objectives simultaneously. But VWN+MTP achieves the lowest loss among augmented variants, suggesting that the virtual width expansion provides enough additional capacity to accommodate the MTP objective without degrading the primary next-token loss. On the 2.5B/25B MoE (Figure 5), this negative interaction disappears — MTP no longer hurts next-token loss when combined with VWN — and VWN+MTP consistently achieves the highest downstream accuracy. This scale-dependent interaction (MTP hurts at small scale, helps at large scale, but VWN makes it beneficial at all scales) is precisely the signature of a capacity bottleneck being relieved — and it was not predicted by prior work.

This is incremental as a finding (synergy between techniques is not a new category of contribution) but significant for practice because it establishes that virtual width expansion and multi-token prediction should be adopted together — the benefits are not merely additive, and deploying one without the other leaves efficiency gains on the table. For practitioners building large-scale training pipelines, this means that the decision to adopt VWN and the decision to adopt MTP are not independent; they should be evaluated jointly, with the expectation that the combined benefit exceeds the sum of individual ablation results.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses large-scale internal training datasets, with downstream evaluation on two collections of publicly available benchmarks. Collection A (used for the 1.5× experiments) aggregates scores across ARC Challenge, BBH, DROP, WinoGrande, Hellaswag, MMLU, MMLU-Pro, C-Eval, TriviaQA, Ape210K, GSM8K, MATH, MBPP, HumanEval, AGIEval, and GPQA (Table 2). Collection B (used for the large virtual width experiments) aggregates scores across MMLU, MMLU-Pro, C-Eval, AGIEval, BBH, DROP, KOR-Bench-Easy, MATH, MBPP+, HumanEval, McEval, TriviaQA, and Chinese SimpleQA (Table 3). The training data scale is substantial—ranging from 500B tokens for the 0.8B-activation MoE ablations to multi-trillion-token scales for the 3.3B-activation MoE experiments. No specific dataset names or sources are provided for the pretraining corpus beyond "large-scale internal datasets."

• Base model(s). All experiments use internal Mixture-of-Experts (MoE) Transformer models based on the ByteDance Seed architecture. Three model scales are studied: MoE 0.4B/4B (0.4B activation parameters, 4B total parameters), MoE 2.5B/25B (2.5B activation, 25B total), MoE-A0.8B (0.8B activation, used for virtual width scaling ablations and scaling law analysis), and MoE-A3.3B (3.3B activation, used for the headline large-scale experiments). The "A" designation distinguishes models used in the large virtual width experiments from the 1.5× experiments. The paper states that MoE architectures are chosen because they represent "a canonical approach to scaling model capacity" (Section 5), and the models are described as "representative of the capabilities of many contemporary LLMs" — though this claim is unverified since no external model families (e.g., LLaMA, GPT, PaLM) are tested.

• Metrics. The primary training metric is next-token prediction (NTP) loss and next-2-token prediction loss measured on held-out validation data, reported versus seen training tokens. This tracks optimization efficiency — how quickly the model learns to predict tokens. The primary downstream metric is average accuracy (%) on the benchmark collections (A or B), where scores are aggregated using "internally defined task weights" — the paper notes that "a difference of one point corresponds to a notable performance gap under this weighting scheme" (Figure 1 caption). Per-benchmark accuracy is reported for representative benchmarks in Figure 9. The loss differences (Δ values) reported in Table 1 and Section 5.2.2 represent absolute reductions in training loss relative to the non-VWN baseline at specific token counts.

• Baselines. The primary baseline in all experiments is the matched non-VWN MoE model — the same architecture (same backbone width, same number of layers, same MoE configuration) trained on the same data with the same hyperparameters, but without VWN's over-width embedding or GHC routing. This is a strong baseline because it isolates the effect of virtual width expansion while holding all other factors constant. For the 1.5× experiments, additional baselines include MTP-only (baseline model with multi-token prediction head but no VWN) and the combination VWN+MTP. For the large virtual width experiments, all models (including the baseline) include MTP by default, so the comparison is VWN versus non-VWN under otherwise identical MTP-augmented training. No external baseline models (e.g., standard dense Transformers without MoE) are compared.

• Generation budget / compute accounting. The paper does not use "generations" as a compute unit (this is a training paper, not an inference paper). Instead, compute is measured by training tokens seen — all models are compared at matched token counts, which is the standard approach for training efficiency comparisons. The key efficiency metric is token efficiency: how many fewer tokens VWN needs to reach a given loss or accuracy compared to the baseline. This is reported as speedup ratios (e.g., "2.5× fewer tokens" in Section 5.2.2). Additional cost analysis in Section 3.5 quantifies the per-token FLOPs and memory overhead of VWN relative to a standard Transformer, establishing that the per-token cost difference is modest (e.g., ~8.8% additional activation memory for 1.5× expansion). The paper does not report wall-clock training time or total FLOPs-to-completion, which would be the most direct measure of practical efficiency.

• Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported. All results appear to be from single training runs — the paper reports one curve per configuration in each figure, with no error bars, confidence intervals, or multiple seeds. The learning rate is kept constant throughout training in the large-scale experiments "to flexibly control the training length" (Section 5.2.2), which is a non-standard choice that may affect the comparability of loss values at different token counts (since standard cosine or step-decay schedules would produce different optimization dynamics). The paper does not discuss whether results are robust to different random seeds, data orderings, or hyperparameter choices beyond the specific ablations shown.

---

Main Quantitative Results

1.5× Virtual Width Expansion at Small and Medium Scale

The paper first establishes that even a modest 1.5× virtual width expansion provides consistent gains across two model scales, and that VWN synergizes with Multi-Token Prediction.

MoE 0.4B/4B results (Figure 4). On the training objective (Figure 4, left), VWN consistently reduces next-token prediction loss relative to the baseline throughout training. However, MTP alone slightly increases the NTP loss relative to baseline — a known phenomenon where multi-token objectives can trade off against single-token performance when capacity is limited. The combination VWN+MTP achieves the lowest loss among augmented variants, but the paper notes a residual gap of 0.016 versus the baseline "when MTP is included." On downstream evaluation (Figure 4, right, Collection A), MTP alone is reported as "comparable with the baseline," while VWN+MTP "delivers the highest gains in average accuracy throughout training." Exact accuracy values are not provided in the text, but the figure shows VWN+MTP consistently above both the baseline and VWN-only curves across the full training horizon.

MoE 2.5B/25B results (Figure 5). At this larger scale, the negative MTP interaction disappears. VWN reduces next-token loss relative to baseline, and "adding MTP on top of VWN does not degrade optimization at this scale" (Section 5.1). Both VWN and VWN+MTP achieve final losses "each approximately 0.015 below the baseline." On downstream evaluation (Figure 5, right, Collection A), both VWN variants outperform the baseline, with VWN+MTP consistently yielding the best average accuracy. The scale-dependent interaction — MTP hurts NTP loss at 0.4B/4B but not at 2.5B/25B — is consistent with a capacity bottleneck interpretation: the smaller model lacks sufficient representational capacity to serve both next-token and multi-token objectives simultaneously, and VWN's virtual width expansion partially relieves this bottleneck.

Key takeaway from 1.5× experiments: Even a fractional virtual width expansion (1.5×) provides consistent training and downstream improvements across model scales. The synergy with MTP is scale-dependent — VWN helps most when capacity is constrained, and the combination VWN+MTP becomes strictly beneficial at moderate scale. These results are qualitative (trend directions) rather than precisely quantified; the paper reports loss differences to two decimal places but does not provide exact accuracy numbers for the 1.5× experiments beyond what is visible in the figures.

5.2.1 Virtual Width Scaling Law: $r \in \{2, 4, 8\}$ at Fixed $m = 8$

The paper's most systematic analysis of virtual width scaling is performed on MoE-A0.8B with fixed fraction rate $m = 8$, sweeping the virtual width factor $r \in \{2, 4, 8\}$ ($n = r \cdot m \in \{16, 32, 64\}$) over a 500B-token training horizon. This is where the scaling law is established.

Table 1 summarizes the quantitative improvements over the non-VWN baseline at 500B tokens:

(All Δ values represent reductions in loss; accuracy measured on Collection B.)

Training dynamics (Figure 7, left and middle). The ordering VWN×8 > VWN×4 > VWN×2 > baseline is consistent throughout training for both next-token and next-2-token prediction loss. The curves do not cross — there is no regime where smaller virtual width temporarily outperforms larger virtual width. The gaps between curves are relatively stable after an initial transient, suggesting that the benefit of larger $r$ is present early and persists rather than emerging late. The next-2-token loss benefits more from virtual width expansion than next-token loss: at 500B tokens, VWN×8 reduces next-2 loss by 0.058 versus 0.035 for next-token loss — a 66% larger absolute improvement on the multi-token objective. This disproportionate benefit on the multi-token objective is evidence for the paper's claim that "the wider virtual space provides richer representational degrees of freedom for short-range compositional targets" (Section 5.2.2).

Downstream accuracy (Figure 7, right). The accuracy curves on Collection B show monotonic improvement with virtual width factor $r$, with VWN×8 achieving +4.16 points over baseline at 500B tokens. The paper notes that under their internal weighting scheme, a 1-point gain reflects "a notable performance difference" — so +4.16 is a substantial improvement. The accuracy ordering is consistent with the loss ordering (VWN×8 > VWN×4 > VWN×2 > baseline) with no observed regressions.

Scaling law analysis (Figure 8). The paper fits a log-linear function $y = -0.0069 \cdot \log_2(x) + 1.6212$ to the relationship between virtual width factor $r$ and training loss, achieving $R^2 = 0.9986$. This is based on only three data points ($r = 2, 4, 8$), which makes the $R^2$ value somewhat misleading — with three points, a two-parameter linear fit on log-transformed data will always achieve high $R^2$ unless there is strong curvature. The fitted coefficient of $-0.0069$ means each doubling of virtual width reduces loss by approximately 0.0069. The paper appropriately characterizes this effect as "modest" (Section 6), and the extrapolation to $r = 1$ (the baseline) is shown by the red data point in Figure 8 lying close to the fitted line.

Per-benchmark breakdown (Figure 9, Appendix 8). On individual benchmarks at 500B tokens, VWN×8 yields: +8.92 on DROP, +2.44 on HumanEval, +4.20 on MATH, +3.95 on MMLU, +5.25 on MMLU-Pro, and +7.45 on TriviaQA. The largest gains are on knowledge-intensive and reasoning-heavy benchmarks (DROP, TriviaQA, MATH), while HumanEval shows smaller gains — the paper attributes this to its "limited test size" (Section 8). The paper also notes that VWN "achieves particularly strong gains on tasks with relatively long context, such as DROP and TriviaQA, where modeling extended dependencies and multi-sentence evidence aggregation benefits most from the enlarged embedding space." The learning curves for all benchmarks show a "uniform left-shift" (Figure 9), indicating better sample efficiency rather than just a higher asymptote.

Ablation on fraction rate $m$ (Figure 6). Before fixing $m = 8$, the paper sweeps $m$ values at each $r$ to assess sensitivity to the partitioning granularity. At $r = 2$, increasing $m$ from 2 to 4 produces "a noticeable but modest gap." At $r = 4$, variants with $m = 8$ and $m = 16$ "nearly overlap, indicating negligible sensitivity to fraction rate." At $r = 8$, $m = 4$ and $m = 8$ are "similarly close, with marginal advantage for $m = 8$." The conclusion: "the effect of $m$ diminishes once $m > 4$, suggesting that, at this scale, partition granularity beyond 4 provides limited benefit." This is an important finding because it means the scaling relationship is primarily a function of the aggregate virtual width $r$, not the specific $(m, n)$ decomposition — you can vary $m$ within a reasonable range without substantially affecting performance, as long as $r = n/m$ is held constant.

5.2.2 Large-Scale VWN×8 on MoE-A3.3B

The headline results use a 3.3B-activation MoE with $(m, n) = (8, 64)$, corresponding to $r = 8$. The learning rate is kept constant throughout training (a non-standard choice, noted but not motivated in detail). All models include MTP by default.

Token efficiency (Figure 1, left and middle). VWN achieves the baseline's next-token loss using 2.5× fewer tokens, and the baseline's next-2-token loss using 3.5× fewer tokens. These are speedup ratios computed by finding the horizontal distance between the VWN and baseline curves at the baseline's final loss value. The ratios are loss-level-dependent: at earlier points in training (higher loss values), the speedup may differ. The paper does not report speedups at multiple loss levels or provide uncertainty estimates on these ratios.

Loss gap dynamics. The next-token loss gap relative to baseline grows from Δ = 0.025 at early stages to approximately Δ = 0.032 at 3T tokens. The next-2-token loss gap grows from Δ = 0.049 to Δ = 0.056 over the same period. Both gaps are still widening at the end of training — neither has saturated. This is evidence against the hypothesis that virtual width benefits are an early-training phenomenon that diminishes as the backbone catches up; instead, VWN's advantage "amplifies as training proceeds" and its "relative efficiency not only appears early but also strengthens over time" (Section 5.2.2).

Downstream accuracy (Figure 1, right). On Collection B, VWN achieves a peak average accuracy that is +2.16 points higher than the baseline. The paper states that "the performance gap persists and continues to widen with extended training," consistent with the growing loss gaps. The absolute accuracy values are not provided in the text, only the difference.

Comparison with 0.8B results. At the larger 3.3B scale, the absolute loss reductions for VWN×8 are comparable to those at 0.8B (Δ NTP loss ≈ 0.032 at 3T tokens for 3.3B vs. 0.035 at 500B tokens for 0.8B). However, direct comparison is confounded by different training horizons (3T vs. 500B tokens), different model scales, different $(m, n)$ configurations (8,64 vs. 8,64 — actually identical), and the constant learning rate schedule used at 3.3B versus whatever schedule was used at 0.8B (not specified). The paper does not attempt to derive a unified scaling law that accounts for both model size and virtual width simultaneously — the scaling law in Figure 8 is fit only to the 0.8B data at fixed $m = 8$.

---

Ablation Studies and Robustness Checks

Fraction rate $m$ under fixed virtual width $r$ (Figure 6): Sweeping $m \in \{2, 4\}$ at $r = 2$, $m \in \{8, 16\}$ at $r = 4$, and $m \in \{4, 8\}$ at $r = 8$ on MoE-A0.8B shows that performance is largely insensitive to $m$ once $m > 4$. At $r = 2$, the $m = 4$ variant slightly outperforms $m = 2$. At $r = 4$, $m = 8$ and $m = 16$ nearly overlap. At $r = 8$, $m = 4$ and $m = 8$ are close. This validates that virtual width $r$ is the primary scaling parameter, not the specific partitioning granularity.

Dynamic vs. static GHC: The paper uses a hybrid static + dynamic GHC configuration throughout, following prior work (Zhu et al., 2024; Zhu et al., 2025). However, no ablation is reported that isolates the contribution of the dynamic component — i.e., comparing static-only GHC against static + dynamic GHC. This is a significant omission: the dynamic component adds parameters (the projection weights Wβ and Wα) and computation (the tanh-based routing computations), and without an ablation, it is unclear whether the dynamic routing is necessary for the reported gains or whether static routing alone would suffice. Given that the dynamic weights are initialized to zero and the static initialization already provides sensible routing, it is possible that much of the benefit comes from static routing with learned weights, and the dynamic component adds marginal value. The paper's silence on this point leaves a genuine question about which aspect of GHC is doing the work.

MTP head design: The paper introduces block-level linear mixing for the MTP head to keep costs manageable at large $r$. No ablation compares block-level mixing against a naive dense mixing (which would be expensive but might be more expressive) or against no MTP head at all for the large-$r$ configurations. The claim that MTP and VWN are synergistic (Section 3.4) is supported by the disproportionate improvement on next-2-token loss (3.5× speedup vs. 2.5× for next-token), but the architectural choices in the MTP head are not ablated to determine whether the specific block-level design matters.

Group normalization before reduce operator: The paper states that group normalization is used before the reduce operator "when the expansion ratio $r$ is large" (Section 3.2), and that it is omitted in the 1.5× experiments. No ablation compares group normalization against layer normalization or no normalization for large $r$, so the necessity of this design choice is not empirically established.

VWN without MTP at large $r$: All large virtual width experiments ($r \in \{2, 4, 8\}$ on MoE-A0.8B and $r = 8$ on MoE-A3.3B) include MTP by default. The 1.5× experiments do include a VWN-only variant (no MTP), showing it outperforms the baseline. But for large $r$, no VWN-only baseline is reported. This means the claimed benefits of large virtual width are always in combination with MTP — the paper cannot disentangle how much of the 2.5× token efficiency gain on MoE-A3.3B comes from VWN alone versus from the VWN+MTP synergy. A VWN×8 without MTP baseline would clarify this, but is absent.

Expand projection (Equation 3): The paper mentions that "when the expansion ratio $n/m$ is large, a single linear projection can optionally be used to map the original 1× embedding to the wider dimension" (Section 3.2). No ablation reports whether this projection is used in the experiments, nor whether performance differs with versus without it. The implementation details in Algorithm 1 take an over-width embedding as input directly, suggesting the expand projection may not have been used in the reported experiments, but this is not stated explicitly.

Weight decay on static vs. dynamic parameters: The paper mentions that static matrices do not use weight decay while dynamic parameters do (Section 3.3, Initialization and Implementation). No ablation tests the sensitivity to this choice. Given that the static matrices are initialized with specific structural patterns (cyclic for B, block-identity for A), applying weight decay might distort these patterns, so the choice is intuitive. But without empirical evidence, it remains a design heuristic.

Constant learning rate in large-scale experiments: The MoE-A3.3B experiments use a constant learning rate "to flexibly control the training length" (Section 5.2.2). This is non-standard — most large-scale training uses cosine or step-decay schedules. No comparison with a standard schedule is provided. A constant learning rate means the model never enters a fine-tuning/annealing phase, which could affect both the absolute loss values and the rate at which the VWN-baseline gap evolves. The reported loss gaps ($Δ = 0.025$ growing to $Δ = 0.032$) may depend on this schedule choice.

Single training run per configuration: All results appear to be from single training runs with no error bars or multiple seeds. Given the scale of these experiments (3T tokens for MoE-A3.3B), replication is understandably expensive, but the absence of any variance estimates means we cannot distinguish genuine differences from run-to-run noise. For the scaling law fit in Figure 8 with $R^2 = 0.9986$, three data points from single runs provide no protection against overfitting the noise at those specific operating points.

---

Critical Assessment

Does VWN decouple representational width from backbone width at near-constant compute?

The paper demonstrates this architecturally: the over-width embedding is larger than the backbone width, and the GHC mechanism routes through standard-width attention and FFN layers. The cost analysis in Section 3.5 estimates the overhead at ~8.8% additional activation memory for a 1.5× expansion with modest $(m, n) = (2, 3)$. What is not demonstrated is that this overhead is small at large $r$. For $r = 8$ with $(m, n) = (8, 64)$, the normalization must handle 64 segments of size $D/8$ at each layer, the dynamic routing must compute 64 × (8+64) = 4608 routing coefficients per layer, and the depth connection must multiply an $n \times n$ matrix with the $n$-column virtual state. The paper acknowledges this tension in Section 6: "as hidden width grows, communication and memory-access overheads become non-negligible, and contemporary hardware is not particularly friendly to very wide activations and cross-device routing." The claim of "near-constant compute" therefore applies most strongly at modest expansion ratios (1.5×–4×). At $r = 8$, the paper provides no wall-clock timing data comparing VWN×8 training throughput against the baseline — the token efficiency gains are measured in training tokens, not in GPU-hours. If VWN×8 is 2.5× more token-efficient but 1.5× slower per token (due to the overheads the paper acknowledges), the net wall-clock speedup would be only ~1.7×, which is still substantial but qualitatively different from the headline "2.5× fewer tokens" framing.

Does virtual width follow a predictable scaling law?

The evidence for a scaling law is based on exactly three data points ($r = 2, 4, 8$) at a single model scale (MoE-A0.8B) with a single fraction rate ($m = 8$) on a single data distribution (internal training data). With three points and a two-parameter log-linear fit, the reported $R^2 = 0.9986$ is a measure of interpolation quality on the training points, not predictive accuracy. The paper does not test the fitted law on held-out $r$ values (e.g., $r = 3$ or $r = 6$) to verify that the log-linear relationship generalizes. It does not test whether the same coefficient (−0.0069 per doubling) holds at different model scales — comparing the 0.8B results (Δ NTP loss = 0.035 at $r = 8$, 500B tokens) with the 3.3B results (Δ NTP loss ≈ 0.032 at $r = 8$, 3T tokens) suggests the coefficient may be scale-dependent, but the paper does not address this. The scaling law should therefore be understood as an existence proof — virtual width scaling follows a regular, monotonic pattern — rather than a quantitatively reliable predictive tool. The paper's language is appropriately cautious: "an initial empirical basis and motivation for exploring virtual-width scaling" (Abstract), "the magnitude of the gain is modest" (Section 6).

Is VWN synergistic with Multi-Token Prediction?

The case for synergy rests on two observations: (1) the next-2-token loss benefits more from virtual width than next-token loss (Δ = 0.058 vs. 0.035 at $r = 8$ on 0.8B; 3.5× vs. 2.5× speedup on 3.3B), and (2) MTP alone slightly hurts NTP loss at small scale, but VWN+MTP does not. These are consistent with synergy but do not rule out additive effects — if virtual width provides $X$ benefit to NTP loss and MTP provides $Y$ benefit to next-2-token loss, the combination could produce $X+Y$ without any interaction effect. The disproportionate improvement on next-2-token loss (66% larger Δ for next-2 vs. next-token at 0.8B) suggests an interaction beyond pure additivity, since one would not expect a purely additive effect to differentially benefit one objective. However, the absence of a VWN×8 without MTP baseline at large $r$ makes it impossible to decompose the contributions. A cleaner test would be: (baseline), (baseline+MTP), (VWN×8), (VWN×8+MTP), all at the same $r$. If the interaction term (VWN×8+MTP − baseline) − [(VWN×8 − baseline) + (MTP − baseline)] is positive, synergy is demonstrated. This full 2×2 ablation is present for the 1.5× experiments (with the caveat that VWN×1.5+MTP at 0.4B shows a residual gap) but not for large $r$.

Do the experiments support the claim that VWN "accelerates optimization by over 2×"?

This claim (Abstract, Section 5.2.2) is supported for the specific configuration tested: MoE-A3.3B, $(m, n) = (8, 64)$, constant learning rate, MTP included, measured against a matched non-VWN baseline on the same data. The 2.5× and 3.5× speedup ratios are computed from Figure 1 by finding the horizontal distance between curves at the baseline's final loss. However, these ratios are specific to the loss level at which they are measured — at earlier points in training (higher loss values), the speedup may be different. The paper reports only a single speedup value per objective, rather than a curve of speedup versus loss level. Additionally, the speedup is measured in training tokens, not in FLOPs or wall-clock time. Since VWN adds per-token overhead (additional normalization, routing computations, wider activations), the FLOPs-matched speedup would be somewhat lower. The paper's cost analysis suggests this overhead is modest (~8.8% additional activation memory for 1.5×), but no FLOPs counting is provided for the $r = 8$ configuration, so the exact discount cannot be calculated from the paper's data.

Do the experiments support the claim that VWN's advantage "amplifies over training"?

Yes. The loss gap growth from Δ = 0.025 to Δ = 0.032 (next-token) and Δ = 0.049 to Δ = 0.056 (next-2-token) on MoE-A3.3B (Section 5.2.2) clearly shows widening gaps. The downstream accuracy gap also continues to widen. This is consistent across the 0.8B experiments as well (Figure 7, where the gaps between VWN curves and baseline do not shrink). The constant learning rate used at 3.3B complicates interpretation somewhat — with standard cosine decay, gaps might stabilize or narrow during the annealing phase — but the qualitative trend of non-saturating benefit is robustly demonstrated.

What key experiments are missing?

Several experiments would significantly strengthen the paper's claims but are absent:

1. VWN×N without MTP at large $r$. This is the most important missing ablation. It would isolate how much of the large-$r$ benefit comes from virtual width alone versus from the VWN+MTP interaction. If VWN×8 without MTP still provides, say, a 1.8× speedup, then virtual width alone is highly effective and MTP adds value on top. If VWN×8 without MTP provides only marginal benefit, then the synergy claim is not just complementary but essential — virtual width may only be useful in conjunction with dense multi-token supervision.

2. Static-only vs. static+dynamic GHC. The dynamic component adds parameters and computation. An ablation showing whether it contributes to performance would help practitioners decide whether to adopt the full DGHC or the simpler static GHC. The zero-initialization of dynamic weights means that early in training, the models are identical — if the final performance is similar, the dynamic component may be unnecessary.

3. Multiple training seeds for key configurations. At minimum, running the baseline and VWN×8 configurations with 2–3 different random seeds on a smaller scale (e.g., MoE-A0.8B) would provide variance estimates and confirm that the observed differences are not within run-to-run noise. The scaling law fit would be more credible with error bars on each data point.

4. FLOPs-matched comparison at large $r$. The paper reports token efficiency, but a FLOPs-matched comparison that accounts for VWN's per-token overhead would more directly address the practical value proposition. If VWN×8 uses 1.2× more FLOPs per token due to routing overhead, the 2.5× token efficiency becomes a 2.1× FLOPs efficiency — still substantial, but the exact number matters for cost calculations.

5. Scaling law validation at different model sizes. The log-linear fit is based on a single model (MoE-A0.8B). Testing whether the same coefficient (−0.0069 per doubling) holds at 0.4B/4B and 3.3B scales would determine whether virtual width scaling is a universal relationship or model-size-dependent. The 3.3B results provide one additional data point ($r = 8$), but with different training length and schedule, a direct coefficient comparison is not possible.

6. Comparison against simply widening the backbone. The paper's motivating claim is that VWN achieves wider representation benefits without quadratic cost. A direct comparison against a model with the backbone proportionally widened to achieve the same total hidden dimension (i.e., a dense model with $D \times r$ backbone width) would quantify how much of the wide-model benefit VWN captures. This comparison is absent. Without it, we know VWN improves over the baseline, but we don't know what fraction of the full width-scaling benefit it recovers — if a truly wider backbone would reduce loss by 0.070 instead of VWN's 0.035, then VWN captures only 50% of the possible gain (at much less than 50% of the cost, which could still be a good tradeoff, but the paper cannot make this case without the data).

7. Wall-clock timing data. Token efficiency is a useful abstract metric, but practitioners deploying these models need to know wall-clock training time. The paper's cost analysis is theoretical; empirical throughput measurements for different $(m, n)$ configurations would ground the efficiency claims in practical reality.

What conditions limit the paper's claims?

The claims are bounded in several important ways that the paper acknowledges with varying degrees of explicitness:

• Single model family (internal MoE). No results on dense Transformers or on other MoE architectures. The paper states the model is "representative" but provides no evidence for this claim. It is possible that VWN's benefits depend on the MoE structure — MoE models already have a wider parameter-vs-compute gap, and VWN's virtual width might be filling a representational bottleneck that is more acute in MoE than in dense models. Testing on a standard dense Transformer (e.g., a LLaMA-style architecture) would address this.

• Training data is internal and undisclosed. The absolute loss values and downstream accuracies cannot be compared to published models. The reported improvements are relative to internal baselines on internal data — the paper demonstrates that VWN improves over its own baseline, not that VWN-trained models are competitive with externally published models of similar scale. This is standard for industry papers but limits independent verification.

• Practical deployability favors modest expansions. The paper states in Section 6 that "virtual width expansions in the 1.5×–4× range are more feasible on today's stacks" and that larger expansions "may require co-design of software, memory layouts, and interconnect strategies." The headline 8× results are therefore partially aspirational — the algorithmic gains are demonstrated, but realizing them in production systems requires engineering work not yet done. This is an honest boundary but means the paper's most dramatic results (3.5× speedup on next-2-token loss) are not immediately actionable without additional infrastructure investment.

• Constant learning rate at large scale. The 3.3B results use a constant learning rate, which is non-standard. The paper states this is "to flexibly control the training length" (Section 5.2.2), but this choice may affect the reported loss gaps and speedup ratios. If a standard cosine schedule were used, the baseline might achieve lower final loss (due to annealing), potentially changing the speedup ratio computed at the baseline's final loss. The direction of this effect is unclear — it could either increase or decrease the apparent VWN advantage.

• No inference-time results. All experiments measure training-time metrics (training loss, downstream accuracy of the trained model). There is no evaluation of whether VWN-trained models have different inference characteristics — e.g., whether the over-width representations need to be maintained during autoregressive decoding, whether the reduce operator can be folded into the unembedding layer for deployment, or whether the dynamic routing adds inference latency. This is a training paper, so the omission is understandable, but practitioners need inference costs to make deployment decisions.

Overall, the experiments convincingly demonstrate that virtual width expansion improves training efficiency on internal MoE models, with monotonic and apparently predictable scaling behavior. The paper falls short of establishing that the specific $r = 8$ configuration is practically deployable, that the scaling law generalizes beyond the tested operating points, or that the dynamic GHC component is necessary for the gains. The synergy with MTP is well-motivated and empirically supported by the disproportionate multi-token improvement, but the absence of large-$r$ VWN-only baselines prevents a clean decomposition. The paper's contributions — identifying virtual width as a new scaling dimension, providing the GHC mechanism to realize it, and demonstrating a log-linear scaling relationship — are substantial and well-supported. The quantitative claims about speedup ratios and scaling coefficients should be treated as specific to the tested configurations rather than universal constants, pending broader validation.

6. Limitations and Trade-offs

The Scaling Law Rests on Three Data Points from a Single Model Scale

The assumption or constraint. The paper presents a log-linear scaling relationship between virtual width factor $r$ and loss reduction (Figure 8, Section 5.2.1), with $R^2 = 0.9986$ and a fitted coefficient of $-0.0069$ per doubling of $r$. This relationship is derived from exactly three data points — $r \in \{2, 4, 8\}$ — on a single model (MoE-A0.8B) at a single fraction rate ($m = 8$) trained on a single internal dataset for 500B tokens. The paper explicitly characterizes this as "an initial empirical basis" (Abstract) and notes that "the magnitude of the gain is modest" (Section 6), but does not test whether the log-linear relationship holds at other model scales, with other $m$ values, or for intermediate $r$ values not used in the fit.

The consequence. With three data points and a two-parameter log-linear model, $R^2 = 0.9986$ measures interpolation quality on the training points rather than predictive accuracy. There is no held-out validation — the fitted line is not tested against $r = 3$ or $r = 6$ to verify that the log-linear form generalizes, nor against other model scales to verify that the coefficient is universal. A practitioner cannot reliably use the $-0.0069$ coefficient to predict the benefit of, say, moving from $r = 4$ to $r = 12$ on a different model family, because the paper provides no evidence that the relationship extrapolates beyond the tested $r$ range or transfers across scales. Comparing the 0.8B results ($\Delta = 0.035$ at $r = 8$, 500B tokens) with the 3.3B results ($\Delta \approx 0.032$ at $r = 8$, 3T tokens) suggests the coefficient may be scale-dependent, since the same $r = 8$ produces meaningfully different absolute loss reductions at different model sizes and training lengths — but with different training horizons and schedules, no direct coefficient comparison is possible.

What evidence exists in the paper. The scaling law data is Figure 8 and Table 1, based on three configurations shown in Figure 7. The 3.3B results (Section 5.2.2, Figure 1) provide one additional point at $r = 8$ for a larger model, but at a different training length (3T vs. 500B tokens), a different $(m, n)$ configuration, and with a constant learning rate — these confounds prevent using this point to validate or update the fitted law. The ablation on fraction rate $m$ (Figure 6) shows that varying $m$ at fixed $r$ produces only minor differences, which supports the idea that $r$ is the dominant parameter, but does not test the functional form of the $r$-loss relationship.

Mitigation status. The paper does not attempt to validate the scaling law on held-out $r$ values, different model scales, or different training distributions. The language is appropriately cautious — "an initial empirical basis and motivation for exploring virtual-width scaling" (Abstract), and the $R^2$ value is reported without extrapolation claims. The limitation is acknowledged only implicitly through the modesty of the claims; the paper does not directly state that the scaling law is unvalidated or that its generality is unknown. Future work to establish whether the log-linear form and coefficient generalize would require sweeping $r$ at multiple model scales with controlled training length, which is computationally expensive but necessary to elevate the finding from an existence proof to a predictive tool.

---

Practical Deployment Overhead Is Not Accounted for in Headline Efficiency Numbers

The assumption or constraint. The headline efficiency gains — 2.5× fewer tokens for next-token prediction, 3.5× for next-2-token prediction on MoE-A3.3B with $r = 8$ (Section 5.2.2) — are measured in training tokens seen, not in FLOPs or wall-clock time. The paper provides a theoretical cost analysis in Section 3.5 estimating the per-token FLOPs and memory overhead of VWN relative to a standard Transformer, and concludes this overhead is modest for small $(m, n)$ configurations (e.g., ~8.8% additional activation memory for $m = 2$, $n = 3$). However, the paper explicitly acknowledges in Section 6 that for large expansion ratios:

"as hidden width grows, communication and memory-access overheads become non-negligible, and contemporary hardware is not particularly friendly to very wide activations and cross-device routing"

and that:

"virtual width expansions in the 1.5×–4× range are more feasible on today's stacks, while larger expansions may require co-design of software, memory layouts, and interconnect strategies to fully realize their potential."

No empirical throughput measurements (tokens per second, FLOPs utilization, wall-clock training time) are reported for any configuration.

The consequence. The headline speedup ratios (2.5×, 3.5×) are upper bounds on practical efficiency — they assume zero per-token overhead from VWN's additional operations. If VWN×8 with $(m, n) = (8, 64)$ runs, say, 1.5× slower per token due to the normalization of 64 segments per layer, dynamic routing coefficient computation for an $n \times (m+n) = 64 \times 72$ matrix per layer, and the I/O bottleneck of managing wide activations across devices, then the 2.5× token efficiency becomes a ~1.7× wall-clock speedup. The exact overhead depends on hardware, software stack maturity, and implementation quality — none of which are characterized in the paper. For a practitioner deciding whether to adopt VWN×8 in a production training pipeline, the missing throughput data means they cannot perform cost-benefit analysis in the currency that matters (GPU-hours or dollars). They must run their own benchmarking to determine whether the token efficiency gains survive translation to wall-clock time on their specific hardware.

What evidence exists in the paper. The paper provides theoretical FLOPs and memory analysis in Section 3.5, estimating $48D$ additional FLOPs per token per layer for $(m, n) = (2, 3)$ (a 1.5× expansion). For $(m, n) = (8, 64)$ ($r = 8$), the corresponding overhead would scale with the segment counts: normalization becomes $4 \times 8 \times D = 32D$ FLOPs, dynamic parameter calculation becomes $2 \times (16 + 64) \times 8 \times D = 1280D$ FLOPs, width connection becomes $2 \times (8+64) \times 8 \times D = 1152D$ FLOPs, and depth connection becomes $2 \times 64 \times D = 128D$ FLOPs — a total of approximately $2592D$ FLOPs per token per layer, compared to $48D$ for the 1.5× configuration. This is a ~54× increase in the routing overhead at $r = 8$ versus $r = 1.5$, though it remains small relative to the $\mathcal{O}(D^2)$ attention and FFN costs for typical $D$ values. The paper does not report whether this theoretical overhead translates to measured throughput degradation, nor whether the kernel fusion strategy described in Section 3.5 (fusing normalization, dynamics, and width connection into a single GPU kernel) is feasible for the 64-segment configuration on current hardware.

Mitigation status. The paper acknowledges this limitation with unusual candor in Section 6, explicitly identifying the 1.5×–4× range as practically feasible and noting that larger expansions require hardware/software co-design. However, this acknowledgement does not reduce the uncertainty for the $r = 8$ results — the headline numbers are reported without discounting for the acknowledged overhead. The paper suggests future work on "co-design of software, memory layouts, and interconnect strategies" but does not demonstrate any such optimizations. A practitioner interested in $r = 8$ deployment would need to independently characterize the throughput impact, which may be substantial depending on their hardware and software stack.

---

All Results Are on a Single Proprietary Model Family with Undisclosed Training Data

The assumption or constraint. Every experiment in the paper uses internal ByteDance Seed Mixture-of-Experts Transformer models trained on "large-scale internal datasets" (Section 5). The paper states that PaLM 2-S* from the reference example is "representative of the capabilities of many contemporary LLMs" — this paper makes no analogous claim of representativeness for its models, though the MoE architecture is described as "a canonical approach to scaling model capacity" (Section 5). No results are reported on dense Transformers, on publicly available model architectures (e.g., LLaMA, GPT, BLOOM), or on public training data (e.g., The Pile, C4, FineWeb). The downstream evaluations use internal task weights to aggregate benchmark scores, and the paper notes that "a difference of one point corresponds to a notable performance gap under this weighting scheme" (Figure 1 caption) — a metric that cannot be compared to any external baseline.

The consequence. The paper demonstrates that VWN improves over the ByteDance Seed MoE baseline on ByteDance's internal training data. It does not demonstrate that VWN would improve over a LLaMA- or GPT-style dense Transformer trained on public data, nor that the virtual width scaling relationship would hold for different architectures or data distributions. There are specific reasons to suspect architecture-dependence: MoE models already have a wide parameter-to-compute ratio (many FFN parameters per token, but fixed backbone width), and VWN's virtual width expansion may be addressing a representational bottleneck that is more acute in MoE than in dense models — where the backbone width is already proportionally larger relative to total parameters. If a dense Transformer already allocates sufficient representational capacity per token, the benefit of further virtual expansion might be smaller or even zero. Similarly, the internal training data may have specific properties (token distribution, sequence length distribution, domain composition) that affect how much virtual width benefits training. Without public baselines, the community cannot independently verify the claims, reproduce the scaling law, or determine whether VWN generalizes beyond the ByteDance ecosystem.

What evidence exists in the paper. All figures (Figures 1, 4, 5, 6, 7, 8, 9) report results on internal MoE models with internal data. The benchmark collections A and B (Tables 2 and 3) use public benchmarks evaluated with internal aggregation weights. The paper provides no comparison to any published external model at similar scale, nor any dense Transformer baseline. The model architecture details (number of layers, attention head count, MoE routing configuration, exact hidden dimension $D$) are not disclosed — only the activation parameter counts (0.4B, 0.8B, 2.5B, 3.3B) and total parameter counts (4B, 25B) are provided.

Mitigation status. The paper does not acknowledge this as a limitation. The use of internal models and data is standard for industry research papers and reflects practical constraints (training large models on public data is expensive and may not align with internal priorities). However, the absence of any public model or data baseline means the paper's claims should be understood as internal ablation results — VWN works on the authors' models and data — rather than as general architectural findings that transfer across model families and data distributions. External validation by other research groups using different architectures and public data would be necessary to establish generality.

---

Large Virtual Width Benefits Are Not Disentangled from Multi-Token Prediction

The assumption or constraint. All experiments with large virtual width factors ($r \in \{2, 4, 8\}$ on MoE-A0.8B in Section 5.2.1, and $r = 8$ on MoE-A3.3B in Section 5.2.2) include Multi-Token Prediction (MTP) by default. The paper states that "all models include a Multi-Token Prediction (MTP) head by default, jointly optimizing the standard next-token and MTP objectives" (Section 5.2). There is no VWN×N without MTP configuration tested at large $r$. The 1.5× experiments (Section 5.1) do include a VWN-only variant (no MTP) and show it outperforms the baseline, but at large $r$, only the VWN+MTP combination is evaluated.

The consequence. The claimed benefits of large virtual width — 2.5× token efficiency on next-token prediction, 3.5× on next-2-token prediction, $\Delta = 0.035$ loss reduction at $r = 8$ — cannot be attributed to virtual width alone. They reflect the combined effect of VWN+MTP, and the paper cannot say what fraction of the improvement comes from VWN, what fraction from MTP, and what fraction from their interaction (synergy). The paper argues for synergy based on the disproportionate improvement on the next-2-token objective (3.5× vs. 2.5× speedup) and on the 1.5× results showing that MTP alone slightly hurts NTP loss at small scale while VWN+MTP does not (Figures 4, 5). But these arguments establish plausibility, not a clean decomposition. For a practitioner deciding whether to adopt VWN, MTP, or both, the inability to estimate the standalone VWN benefit at the $r$ value they might want (e.g., $r = 4$) is a significant gap — they cannot perform cost-benefit analysis on VWN alone without running their own ablation.

What evidence exists in the paper. The 1.5× experiments (Figures 4 and 5) provide a 2×2 ablation (baseline, VWN, MTP, VWN+MTP) at small $r$, showing that VWN alone improves over baseline and VWN+MTP is best. The $r = 2, 4, 8$ experiments on MoE-A0.8B (Figure 7) have no VWN-only or MTP-only variants — all curves include MTP. The MoE-A3.3B experiments (Figure 1) also lack VWN-only or MTP-only baselines. The paper never reports a VWN×8 without MTP result at any scale.

Mitigation status. The paper does not acknowledge this as a missing ablation. The synergy with MTP is presented as a feature — "VWN synergizes with MTP, yielding consistent improvements" (Section 1 Contributions) — rather than as a confound that prevents clean attribution. The implication is that practitioners should adopt VWN and MTP together, which may be reasonable advice if both are beneficial, but leaves unanswered whether VWN alone at large $r$ would be worth the implementation complexity for teams that cannot or do not wish to adopt MTP. A single experiment — VWN×8 without MTP on MoE-A0.8B or MoE-A3.3B — would close this gap at the cost of one additional training run.

---

The Dynamic GHC Component Is Not Ablated, Leaving Its Contribution Unknown

The assumption or constraint. The paper uses a hybrid static + dynamic GHC configuration throughout all experiments, where the routing matrices are the sum of learned static parameters ($\mathbf{A}$, $\mathbf{B}$) and input-dependent dynamic components generated through small linear projections (Equations 13 and 14 in Section 3.3). The dynamic projection weights $\mathbf{W}_\beta$ and $\mathbf{W}_\alpha$ are initialized to zero, meaning "at the start of training, the dynamic component contributes nothing — the routing is purely static" (Section 3.3, Initialization and Implementation). No ablation is reported comparing static-only GHC against static + dynamic GHC. The paper notes that the dynamic design "integrates the advantages of both" HC and FC (Section 2), but provides no empirical evidence that the dynamic component improves over the static baseline.

The consequence. The dynamic component adds non-trivial implementation complexity and computational overhead. At each layer, for each of the $n$ virtual segments, the model must compute $D/m$-dimensional projections through $\mathbf{W}_\beta$ and $\mathbf{W}_\alpha$, apply tanh nonlinearities, and scale by learned matrices $\mathbf{S}_\beta$ and $\mathbf{S}_\alpha$. This introduces additional parameters (the projection weights and scaling matrices) that must be stored, loaded, and updated during training. If the dynamic component provides negligible benefit over static GHC — which already has learned parameters that can adapt routing patterns through gradient descent — then practitioners could drop the dynamic machinery, simplifying implementation and reducing per-token overhead. Conversely, if the dynamic component is essential for the reported gains, it represents a necessary complexity that adopters must implement faithfully. Without an ablation, neither conclusion can be drawn.

What evidence exists in the paper. None. The paper provides no experiment comparing static-only GHC against static + dynamic GHC at any scale or configuration. The ablation on fraction rate $m$ (Figure 6) varies $m$ values but always uses the hybrid static + dynamic configuration. The 1.5× experiments (Section 5.1) use the same hybrid approach. The paper's related work discussion (Section 2) argues that HC "often under-utilizes the expanded representations, since each extension is updated using only a few scalar weights," and that GHC's matrix-valued routing with dynamics provides "fine-grained control over capacity usage." This is a motivating argument, not empirical evidence — it explains why one might expect dynamics to help, but does not demonstrate that they do.

Mitigation status. The paper does not acknowledge the absence of this ablation. The dynamic component is presented as an integral part of the GHC design, and its contribution is assumed rather than tested. Given that the static matrices $\mathbf{A}$ and $\mathbf{B}$ are already learned parameters with expressive matrix-valued routing (unlike HC's scalar weights), it is plausible that static GHC alone could capture most of VWN's benefit, with the dynamic component providing only marginal improvement. A straightforward experiment — training VWN×4 with $(m, n) = (8, 32)$ using static-only GHC versus the full DGHC on MoE-A0.8B — would resolve this question at modest computational cost relative to the experiments already reported.

---

No Comparison Against Naïvely Widening the Backbone

The assumption or constraint. The paper's central motivation is that "naively increasing hidden dimensions leads to quadratic growth in parameters and compute" (Section 1), and that VWN captures the benefits of wider representations while avoiding this quadratic cost. However, the paper never empirically quantifies what fraction of the full width-scaling benefit VWN actually recovers. All comparisons are against the same-width backbone baseline — the model with $D$ unchanged. There is no experiment comparing VWN×$r$ (virtual width $rD$ at near-constant backbone cost) against a model with the backbone proportionally widened to $rD$ (truly wider representations at quadratic cost). Without this comparison, we know VWN improves over the narrow baseline, but we do not know how close it gets to the idealized wider model that it approximates.

The consequence. The paper's value proposition — "benefits of wider representations without the quadratic cost" — is evaluated only on one side of the equation. VWN's improvements over the baseline could represent 90% of what a truly widened model would achieve (excellent cost-benefit tradeoff), 50% (good but leaves gains on the table), or 20% (marginal benefit despite statistical significance). A practitioner deciding between (a) adopting VWN×4 with ~8.8% overhead or (b) simply training a 1.5× wider backbone with ~2.25× more FLOPs needs to know what they are trading off. The paper provides no data to inform this decision. The theoretical cost analysis shows VWN is dramatically cheaper than backbone widening, but without knowing the quality gap, cost-effectiveness cannot be assessed.

What evidence exists in the paper. None. The paper does not train or evaluate any model with a proportionally widened backbone. All baselines use the same backbone width $D$ as the VWN variants, with VWN differing only in the embedding dimension and GHC routing. The figures (Figures 1, 4, 5, 7) compare VWN at various $r$ against the non-VWN baseline at $r = 1$. A widened-backbone baseline (e.g., a dense or MoE model with hidden dimension $D \times r$) is absent from all experiments.

Mitigation status. The paper does not acknowledge this as a missing comparison. The FLOPs-matched comparison framework used in the reference example (comparing test-time compute vs. pretraining at matched total FLOPs) is not adopted here — VWN is simply compared against the same-backbone baseline at matched token counts. The paper's contribution is framed as establishing virtual width as a new scaling dimension, not as demonstrating that it Pareto-dominates backbone widening, so the missing comparison may be considered outside scope. However, for a practitioner making architecture decisions, the question "should I just make my model wider?" is the most natural alternative to "should I adopt VWN?" — and the paper provides no empirical guidance to answer it. A single experiment on MoE-A0.8B comparing VWN×4 against a model with 4× wider backbone at matched total parameters (or matched total FLOPs) would substantially strengthen the practical case for VWN adoption.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a genuinely new axis for scaling language models — virtual width — that had no prior existence in the conceptual toolkit of the field. Before VWN, the debate around model capacity was structured around a small set of options: scale depth (more layers), scale width (larger hidden dimensions, with quadratic cost), scale parameters via sparsity (MoE, expanding FFN inner dimensions while keeping backbone width fixed), or scale data. VWN breaks this taxonomy by showing that representational width and computational width are separable variables, and that expanding the former while holding the latter fixed yields predictable, monotonic improvements at modest additional cost. This is not an incremental tweak to an existing scaling axis — it is the creation of a new axis entirely, one that sits orthogonal to both depth scaling and MoE-based parameter scaling.

The conceptual shift is best understood by analogy to how MoE changed the conversation about parameter scaling. Before MoE, "more parameters" meant "more FLOPs per token" — the two were coupled. MoE demonstrated that total parameters (capacity) could be decoupled from per-token FLOPs (compute) through conditional computation, creating a new design dimension where practitioners could independently choose model capacity and inference cost. VWN achieves a parallel decoupling for representational width: "wider representations" no longer implies "quadratically more FLOPs." The embedding dimension and the backbone dimension become independent design choices, with GHC serving as the learned interface between them — analogous to how MoE routers serve as the learned interface between token representations and expert FFN parameters.

This is a new scaling dimension, not just a better architecture. The paper's most significant contribution is the empirical demonstration (Section 5.2.1, Figure 8) that virtual width follows an approximately log-linear scaling relationship, with $R^2 = 0.9986$ for the fit $y = -0.0069 \cdot \log_2(x) + 1.6212$. While the specific coefficient is derived from only three data points on a single model family and should not be treated as universal, the existence of a clean, monotonic relationship establishes virtual width as a principled design dimension rather than an ad-hoc architectural knob. This is the same qualitative step that Hoffmann et al. (2022) took for pretraining compute allocation and that Kaplan et al. (2020) took for model size scaling — identifying that a variable behaves predictably enough to inform resource allocation decisions. The paper's demonstration that performance is primarily a function of the aggregate virtual width factor $r = n/m$ rather than the specific $(m, n)$ decomposition (Figure 6, where varying $m$ at fixed $r$ produces only minor differences) further supports treating $r$ as the relevant scaling parameter.

The paper also resolves a latent tension in the MoE literature. MoE architectures achieve impressive parameter counts and strong performance, but practitioners have long observed that MoE models at a given total parameter count underperform dense models at the same total parameter count on certain metrics — a phenomenon often attributed to the "representation bottleneck" of the fixed backbone width. VWN provides both a diagnosis and a treatment: the bottleneck is real (wider hidden states help even when FFN capacity is already abundant), and virtual width expansion directly addresses it without sacrificing MoE's compute efficiency. This explains why VWN shows benefit even on top of strong MoE baselines (Figures 1, 4, 5, 7) — MoE solves the processing bottleneck but leaves the representation bottleneck untouched, and VWN fills that gap.

The connectivity perspective (Section 4) reframes depth-wise information flow. By unrolling the GHC recurrence (Equation 17) and showing that each layer's virtual state is a linear combination of backbone-transformed features from all previous layers, weighted by products of learned carry matrices, the paper provides a new language for thinking about depth connectivity. Prior work characterized residual connections as enabling gradient flow and dense connections as expanding receptive fields. VWN's framing — depth as a sequence, layers as positions, GHC as learned linear attention over a compressed depth cache — unifies these perspectives and provides a principled basis for choosing hyperparameters. The memory budget is $r = n/m$, measured in $D$-units. The fraction rate $m$ controls the fidelity-layers tradeoff: small $m$ means fewer layers remembered at higher fidelity; large $m$ means more layers remembered at lower fidelity per layer. This transforms VWN configuration from an empirical tuning exercise into a design problem with clear intuitions — deeper models likely need larger $m$ to maintain sufficient per-layer bandwidth, wider models can accommodate larger $m$ because their increased representational capacity provides more total bandwidth. The paper's hypothesis that "larger models tend to require higher $m$" (Section 5.2) is partially supported by the $m$ ablation (Figure 6), where the 0.8B-activation model saturates at $m = 4$, suggesting smaller models do not benefit from finer partitioning — a finding that would guide configuration choices at different scales.

Several research directions become more attractive. The demonstration that virtual width can be treated as a scaling dimension suggests that joint scaling laws — simultaneously optimizing backbone width, virtual width, depth, and data — are the natural next step. The current paper treats backbone width as fixed and varies only virtual width; a complete picture would characterize how the optimal virtual width factor $r$ depends on backbone width, model depth, and training budget. The synergy with MTP (disproportionate improvement on next-2-token loss, Section 5.2.2) suggests that virtual width and multi-token prediction may be co-adapted — future work on either technique should consider their interaction rather than treating them independently. The paper's candid acknowledgement of deployment challenges at large $r$ (Section 6) — communication overhead, wide activation layout, cross-device routing — identifies systems infrastructure as the binding constraint, redirecting attention from algorithmic innovation to software/hardware co-design for wide activations.

Some directions become relatively less attractive. The paper's demonstration that MoE alone does not solve the representation bottleneck (since it expands only FFN inner dimensions) suggests that purely sparsity-based approaches to capacity scaling — more experts, more fine-grained routing, dynamic expert allocation — will hit diminishing returns if the backbone hidden dimension remains fixed. The paper shows that even a modest 1.5× virtual width expansion provides consistent gains on top of MoE (Section 5.1), implying that further MoE innovations without corresponding representational width increases are working against a fundamental bottleneck. Similarly, the ablation finding that fraction rate $m$ matters little beyond $m = 4$ (Figure 6) suggests that research into more complex segmentation schemes or learned segment allocation is unlikely to yield significant benefits — the simple uniform segmentation with $r = n/m$ as the dominant parameter is sufficient.

---

Follow-Up Research This Work Enables

Joint scaling laws for virtual width, backbone width, and depth. The paper establishes that virtual width follows a log-linear relationship with loss at a single model scale (MoE-A0.8B, $m = 8$, 500B tokens). The natural extension is to characterize how this relationship depends on other scaling parameters. A systematic sweep would train models at multiple backbone widths (e.g., $D \in \{2048, 4096, 8192\}$), multiple depths (e.g., $L \in \{24, 48, 96\}$), and multiple virtual width factors (e.g., $r \in \{2, 4, 8, 16\}$) with controlled training budgets, fitting a unified parametric form (e.g., loss as a function of $D$, $L$, $r$, and tokens). The key question is whether the virtual width coefficient ($-0.0069$ per doubling in this paper) is universal or depends on backbone capacity — one might hypothesize that models with narrower backbones benefit more from virtual expansion (since the representation bottleneck is more acute), while very wide backbones saturate with smaller $r$. A clean negative result — finding that the optimal $r$ scales sub-linearly with backbone width, for instance — would establish boundary conditions on virtual width's utility and guide practical configuration choices.

Static-only versus static+dynamic GHC ablation at scale. The paper uses a hybrid static + dynamic GHC throughout but never ablates the dynamic component. An experiment on MoE-A0.8B with VWN×4, $(m, n) = (8, 32)$, comparing three conditions — (a) static-only GHC (only $\mathbf{A}$ and $\mathbf{B}$, no $\mathbf{W}_\beta$, $\mathbf{W}_\alpha$, $\mathbf{S}_\beta$, $\mathbf{S}_\alpha$), (b) full DGHC as described, (c) static GHC with learned $\mathbf{A}$, $\mathbf{B}$ but no input-dependent routing — would determine whether the dynamic routing contributes to performance or only to implementation complexity. If static-only matches DGHC, the architecture simplifies considerably (no tanh projections, no scaling matrices, no per-layer dynamic FLOPs), making adoption easier. If DGHC provides substantial gains over static, it establishes that input-dependent routing is necessary to fully utilize virtual width, and future work could explore more efficient dynamic mechanisms (e.g., low-rank dynamic updates, token-level rather than segment-level routing, or learned sparsity in the dynamic coefficients).

VWN on dense Transformers to test architecture-dependence. All experiments use internal MoE models. A direct test on a publicly available dense Transformer — for example, adapting a LLaMA-2 7B or OLMo 7B architecture to include VWN with $r = 4$ and $(m, n) = (4, 16)$, training on a public dataset like FineWeb or C4, and comparing against the unmodified baseline at matched tokens — would establish whether VWN's benefits transfer beyond MoE. The hypothesis is that dense Transformers have a proportionally wider backbone relative to total parameters than MoE models, so the representation bottleneck may be less acute, potentially reducing VWN's benefit. Quantifying this difference — perhaps finding that VWN provides, say, a 0.015 loss reduction on dense versus 0.028 on MoE at the same $r = 4$ — would help practitioners predict whether VWN is worth adopting for their specific architecture family. A negative result (VWN provides negligible benefit on dense models) would not invalidate the paper's MoE findings but would establish an important boundary condition and redirect VWN research toward MoE-specific applications.

Virtual width scaling combined with inference-time compute or model distillation. The paper focuses entirely on training-time benefits. Two natural extensions probe whether virtual width's advantages persist into deployment. First, VWN-trained models produce over-width hidden states at every layer — during autoregressive decoding, these must be maintained and routed through GHC at each step. Measuring inference latency and throughput for VWN models at different $r$ values (versus a same-backbone baseline and versus a proportionally widened backbone model with comparable loss) would determine whether the training-time token efficiency gains translate to inference. Second, the over-width hidden states could serve as rich targets for distillation — training a smaller student model to predict the VWN teacher's over-width representations (rather than just the output logits) might transfer more of the virtual width benefit to a compact deployment model. A concrete experiment: train a VWN×4 teacher and a same-backbone student with an auxiliary loss matching the teacher's over-width hidden states at intermediate layers, evaluating whether the student outperforms one distilled from a standard teacher.

Difficulty-conditioned or input-dependent virtual width allocation. The paper uses a fixed $r$ for all tokens and all layers. However, the connectivity perspective (Section 4) suggests that different layers might benefit from different effective virtual widths — early layers might need more representational capacity to encode rich token-level features, while later layers might need more depth-history bandwidth. A dynamic virtual width scheme, where the expansion ratio varies per layer or is conditioned on input difficulty (analogous to how the reference paper's compute-optimal scaling varies search strategy by prompt difficulty), could allocate the virtual width budget more efficiently. A concrete experiment: modify GHC to allow per-layer $r_l$ values with a total virtual width budget constraint ($\sum_l r_l = \text{constant}$), and learn the allocation via a lightweight meta-controller or via differentiable budget allocation. The prediction would be that early and middle layers benefit from higher $r$ (richer token representations) while later layers can operate with lower $r$ (more focused on task-specific features), potentially outperforming a uniform-$r$ model at the same total virtual width budget.

Over-width embedding as a target for representation learning objectives. The paper pairs VWN with next-token and next-2-token prediction — standard autoregressive objectives. The over-width embedding space, with its $n$ independent segments of dimension $D/m$, provides a natural structure for auxiliary representation-learning objectives. For example, a contrastive objective that encourages different virtual segments to encode complementary aspects of the token context (orthogonality constraints, mutual information maximization across segments, or slot-attention-style competition) could ensure the expanded capacity is fully utilized rather than learning redundant representations. The paper's finding that the static initialization already provides sensible routing (identity for the first $m$ segments, identity carry-forward for the remaining $r-m$ segments) suggests that early in training, the extra virtual segments may not be actively used. A concrete experiment: add an auxiliary loss that maximizes the mutual information between different virtual segments and different future tokens (e.g., segment $i$ predicts token $t+i$), training on MoE-A0.8B with VWN×8, and measuring whether per-segment utilization (quantified by the effective rank of the segment covariance matrix or by ablation — dropping individual segments and measuring loss impact) improves over the standard objective.

---

Practical Applications and Downstream Use Cases

Cost-efficient training of MoE language models in the 1B–10B activation range. The most directly actionable finding is that virtual width expansions in the 1.5×–4× range provide consistent, monotonic improvements on MoE architectures at modest additional cost (~8.8% activation memory overhead for $r = 1.5$, per Section 3.5). For teams training MoE models at the 1–10 billion activation parameter scale — a common regime for organizations with moderate compute budgets — adopting VWN×4 with $(m, n) = (8, 32)$ or similar configuration offers a clear efficiency gain: the 0.8B-activation results (Table 1) show $\Delta = 0.028$ next-token loss reduction and +3.5 accuracy points on Collection B at $r = 4$ versus baseline, and the $m$ ablation (Figure 6) indicates that $m$ can be chosen conservatively (e.g., $m = 4$ or $8$) without sensitivity to exact tuning. The implementation complexity is moderate — GHC requires modifying the residual connection logic at each layer, adding the dynamic projection parameters, and fusing normalization/routing kernels for efficiency — but the paper provides complete pseudocode (Algorithms 2 and 3) that can be adapted to existing Transformer implementations in frameworks like Megatron-LM, HuggingFace Transformers, or JAX-based training stacks. The paper's cost analysis provides concrete numbers for memory budgeting (e.g., $3D$ bytes additional activation memory for $(m, n) = (2, 3)$ with $\eta = 0.5$), enabling practitioners to determine whether their current hardware can accommodate the expansion without reducing batch size or sequence length.

Improving sample efficiency for multi-token prediction training pipelines. The paper demonstrates that VWN disproportionately benefits multi-token prediction objectives — the next-2-token loss sees 3.5× token efficiency gains versus 2.5× for next-token loss on MoE-A3.3B (Section 5.2.2), and the $\Delta$ values in Table 1 are consistently larger for next-2 loss than next-token loss (e.g., 0.058 vs. 0.035 for VWN×8). For teams already using or considering MTP — which has become increasingly common following DeepSeek-V3 and related work — adding VWN provides a complementary benefit that specifically targets the multi-token objective. The block-level linear mixing design (Section 3.4) keeps the MTP head cost manageable at large $r$: instead of a naive $2rD \to rD$ dense projection (which would scale quadratically with $r$), the shared per-segment $(2D/m \to D/m)$ projection ensures the MTP head cost scales linearly with $n$ but with small constant factor. A concrete deployment recipe: take an existing MoE+MTP training pipeline, add VWN with $r = 4$ and moderate $m$ (e.g., $m = 4$, $n = 16$), use block-level MTP mixing, and expect ~0.028–0.035 next-token loss reduction and ~0.045–0.058 next-2-token loss reduction at 500B+ token scale, with the gap widening over training (as demonstrated by the growing Δ values in Section 5.2.2). The constant learning rate finding in the large-scale experiments (which the paper suggests enables "flexible training length control") is not a requirement — practitioners can use standard cosine schedules.

Enhancing downstream performance on long-context and knowledge-intensive tasks. The per-benchmark results (Figure 9, Appendix 8) show that VWN's gains are largest on tasks requiring extended context modeling and multi-sentence evidence aggregation: +8.92 on DROP, +7.45 on TriviaQA, +4.20 on MATH at $r = 8$ on MoE-A0.8B at 500B tokens. This pattern is consistent across both the 0.8B and 3.3B scales (the paper notes "VWN achieves particularly strong gains on tasks with relatively long context"). For applications where these task types dominate — question-answering systems, retrieval-augmented generation pipelines, mathematical reasoning assistants — deploying VWN-trained models offers disproportionate benefit on the metrics that matter most. The mechanism is intuitive: longer contexts require maintaining more information in the hidden state (entities, relations, discourse structure), and the over-width embedding provides additional representational capacity per token to encode this information without compression or forgetting. A team building a document-grounded QA system using a MoE backbone could adopt VWN×8 with $(m, n) = (8, 64)$ (the configuration validated at 3.3B scale), expecting substantial improvements on benchmarks like DROP and TriviaQA without changing the backbone architecture, training data, or inference infrastructure (since inference memory overhead is limited to the additional GHC parameters, with no impact on KV cache size).

Training compute allocation in resource-constrained research settings. For academic labs or small companies training models with limited GPU budgets, VWN offers a way to improve model quality without increasing the backbone FLOPs budget. The paper's scaling law analysis (Figure 8, with the important caveat about limited data points) provides a rough heuristic: each doubling of virtual width reduces loss by ~0.007 at the cost of modest additional memory and routing FLOPs. A team that can afford to train a model with backbone width $D = 2048$ on, say, 100B tokens could instead train with $(m, n) = (4, 16)$ ($r = 4$, same backbone width) and, extrapolating from the paper's 0.8B results, expect a loss reduction of approximately $0.007 \times 2 = 0.014$ (two doublings from $r = 1$ to $r = 4$) — a meaningful improvement at the cost of additional activation memory (~8.8% or somewhat more for $r = 4$, depending on exact $(m, n)$) but no increase in the dominant attention and FFN costs. The trade-off is favorable when the primary constraint is FLOPs or training time rather than GPU memory — if the model already fits in memory with the additional activation overhead, VWN provides "free" quality improvement in the sense that the dominant compute costs are unchanged. The paper's implementation guidance (kernel fusion for normalization/dynamics/width connection, selective activation recomputation with $\eta$ tuning, group normalization before the reduce operator) provides a concrete playbook for efficient implementation that minimizes the practical overhead.

---

When to Prefer This Method

The paper does not explicitly position VWN against named alternatives with a clear decision rule or tradeoff matrix comparable to the reference example's "prefer test-time compute when $R \ll 1$ and problems are easy-to-medium difficulty; prefer pretraining when problems are hard or $R \gg 1$." The paper frames VWN as a complementary scaling dimension to be combined with existing approaches (MoE, MTP) rather than as an alternative that replaces them. The absence of comparisons against proportionally widened backbones or against alternative representation-expansion methods (e.g., deeper models with narrower layers, concatenation-based hidden state expansion) means there is no empirical basis in the paper for a "prefer VWN over X" recommendation. The paper's own characterization — that VWN is "a new dimension for scaling large models" (Section 1) and that "virtual width expansions in the 1.5×–4× range are more feasible on today's stacks" (Section 6) — suggests a default posture of adopting VWN within the feasible range as a general complement to existing architectures, rather than a contingent choice that depends on specific problem characteristics. The decision is primarily about whether the additional implementation complexity and memory overhead are acceptable for the expected loss reduction, which the scaling law (with appropriate caveats about generalizability) and cost analysis provide a basis for evaluating. I therefore omit a conditional preference matrix, as the paper does not provide the empirical comparisons needed to construct one with specific conditions.