Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding

ArXiv: 2507.07120

🎯 Pitch

Increasing Tensor Parallelism beyond the number of KV heads forces costly KV cache duplication and caps FFN sharding, the real latency bottleneck in long-context decoding. Helix Parallelism splits these concerns in time: it shards KV caches conflict-free during attention, then reshuffles GPU roles for aggressive FFN weight sharding, completely hiding the handoff. On GB200 NVL72, this enables up to 32× larger batches under the same latency budget for DeepSeek-R1, effectively decoupling interactive throughput from context length.

---

1. Executive Summary

This paper introduces Helix Parallelism, a hybrid execution strategy that decouples the parallelism mapping of attention and feed-forward network (FFN) computation in a temporal pipeline to address the dual bottlenecks of KV cache reads and FFN weight loads during interactive multi-million-token LLM decoding. Evaluated on GB200 NVL72 hardware with FP4 precision using Llama-405B (dense) and DeepSeek-R1 (MoE with MLA attention), Helix applies KV parallelism during attention to shard KV caches across GPUs—eliminating the duplication that occurs when Tensor Parallelism width exceeds the number of KV heads—then reconfigures the same GPUs for Tensor Parallelism or combined Tensor × Expert Parallelism during FFN computation, with a lightweight all-to-all communication step preserving exact attention semantics and a batchwise overlap optimization (HOP-B) hiding that communication behind computation. Helix reduces token-to-token latency by up to 1.5× at fixed batch sizes and supports up to 32× larger batches under the same latency budget for DeepSeek-R1, establishing that aggressive FFN sharding can be combined with duplication-free KV sharding on modern networks with large NVLink domains without sacrificing interactivity—provided the attention communication overhead can be masked through fine-grained compute-communication pipelining.

2. Context and Motivation

The Core Problem: Two Bottlenecks Collide in Real-Time Long-Context Decoding

The paper tackles a specific, sharply defined tension: LLMs are increasingly expected to serve interactive applications while maintaining multi-million-token KV cache histories, but the dominant parallelism strategies used today cannot efficiently handle both demands simultaneously. This is not a problem of accuracy or model capability—it is a pure systems efficiency problem that arises from how we map model execution across GPUs during autoregressive decoding.

To understand why this matters, we need to examine the two bottlenecks that dominate decoding latency and how they interact with batch size and context length.

Bottleneck 1: KV cache reads during self-attention. In autoregressive decoding, every new token must attend to every previous token in the context. The intermediate key and value representations for all previous tokens are stored in the KV cache, which grows linearly with sequence length $S$. Reading this cache from GPU DRAM for each new token involves data movement proportional to $B \times 2 \times K \times Hsz \times S \times \text{bytes\_per\_param}$, where $B$ is batch size, $K$ is the number of KV heads, and $Hsz$ is the head dimension. Critically, this cost scales with both context length and batch size—doubling either doubles the read volume. For million-token contexts, the KV cache size per GPU can reach tens of gigabytes, and DRAM bandwidth becomes the binding constraint. The paper shows this explicitly in the roofline analysis of Figure 1 (middle): as $S$ increases, attention DRAM read time grows linearly and eventually dominates total latency.

Bottleneck 2: FFN weight reads. Transformer FFN layers contain large weight matrices—typically $H \times F$ and $F \times H$ for a two-layer FFN with hidden dimension $H$ and intermediate dimension $F$, plus an additional gating matrix in modern SwiGLU variants. For every token generated, these weights must be loaded from DRAM. With a small batch (typical in interactive decoding, where each user's request is independent and latency-sensitive), this cost cannot be amortized across many tokens. The per-token weight-read cost becomes a dominant latency term that is independent of context length but scales poorly with model size.

The interaction with batch size. These two bottlenecks pull in opposite directions with respect to batch size. Increasing batch size amortizes FFN weight reads across more tokens, which is good—but it proportionally increases KV cache read volume, which is bad when the KV cache is already enormous. The result is a system forced into an uncomfortable corner: use small batches to keep KV reads manageable (at the cost of unamortized FFN reads) or use larger batches to amortize FFN reads (at the cost of exploding KV read time). Neither choice is satisfactory for interactive applications with tight token-to-token latency (TTL) budgets.

Why This Problem Matters Now

The paper argues that this tension is not a theoretical edge case—it is the practical reality facing LLM serving systems as three trends converge:

1. Context lengths are exploding. Models like Gemini 1.5 [1] and Llama 4 [2] support million-token contexts, and precomputed context spanning millions of tokens is becoming common in applications like codebase understanding, long-document QA, and persistent AI assistants that maintain full conversation histories.

2. Interactive applications demand millisecond-level TTL. AI assistants, copilots, and autonomous agents require each token to be generated almost instantly. A user-facing chatbot that takes multiple seconds per token is unusable regardless of how good the responses are. The paper frames this as a throughput-latency Pareto frontier problem: we want to maximize both system throughput (tokens/sec/GPU) and per-user interactivity (tokens/sec/user, the reciprocal of TTL), but conventional parallelism strategies force a tradeoff between them.

3. Modern attention variants reduce but don't eliminate KV pressure. Grouped-Query Attention (GQA) [3], Multi-Query Attention (MQA) [4], and Multi-Head Latent Attention (MLA) [5] all collapse multiple query heads into fewer KV heads ($K < Q$, with $K$ typically 8 or less). For MLA in DeepSeek-R1, there is effectively a single latent KV representation. This dramatically reduces the raw size of the KV cache compared to full Multi-Head Attention, but the cache still grows linearly with sequence length. At million-token scales, even $K=1$ produces a massive cache—the paper's evaluation simulates exactly this regime.

The result is a hardware-utilization crisis: even on systems with enormous aggregate memory and bandwidth (like GB200 NVL72 with its 72 GPUs and large NVLink domain), conventional parallelism strategies cannot keep all GPUs productively utilized under tight TTL constraints because they are bottlenecked on either KV reads or FFN reads, leaving GPU FLOPs idle while waiting for DRAM.

Where Prior Approaches Fall Short

The paper identifies three categories of prior work and explains why each fails to solve the combined problem:

1. Tensor Parallelism (TP) [6] hits a hard ceiling at $TP = K$.

Tensor Parallelism shards both FFN weights and attention heads evenly across GPUs. Each GPU holds $1/TP$ of the weights and computes its portion of the output. For attention, TP splits query heads: GPU $i$ gets $Q/TP$ query heads. If $TP \leq K$, then each GPU can be assigned a disjoint subset of the $\lceil K/TP \rceil$ KV heads, and the KV cache is naturally partitioned without duplication. Every GPU reads only its share of the KV cache.

The problem arises when $TP > K$—a common scenario in large models where you need high parallelism to shard the enormous FFN weights (which account for roughly two-thirds of model parameters). For example, Llama-405B has $K=8$ KV heads and $Q=128$ query heads. If you want to run with $TP=16$ (to spread the FFN weights across 16 GPUs), each GPU gets $128/16=8$ query heads. But since there are only 8 KV heads total, some KV heads must serve multiple query heads across different GPUs, requiring those KV heads to be duplicated on multiple GPUs. At the extreme, every GPU stores a complete copy of the full KV cache.

The paper's roofline analysis (Figure 1, left) shows what this means quantitatively: for Llama-405B with $B=8$ and $S=1\text{M}$ tokens on GB200, KV cache read time initially drops as TP increases from 1 to 8, but then flattens completely for $TP > 8$ because every additional GPU must store the entire KV cache. The per-GPU KV read volume stops decreasing. Meanwhile, FFN weight read time continues to decrease with TP width, but the attention bottleneck now dominates. This imposes a hard ceiling on achievable TTL: beyond $TP=K$, you get no further latency reduction from attention, and any additional GPUs only help with FFN weights—but those weight reads may already be fast enough that the attention bottleneck is the binding constraint.

Furthermore, capping TP at $K$ means the FFN weights (two-thirds of the model parameters) can only be sharded across $K$ GPUs. Each of those $K$ GPUs must hold $F/K$ of the FFN weight matrices, consuming memory that could otherwise store KV cache. The paper argues this creates a false economy: the very GPUs that need to relieve KV cache pressure are also burdened with large FFN weight slices, limiting how much KV cache each can hold.

2. Pipeline Parallelism (PP) does not address per-layer latency.

Pipeline Parallelism places different transformer layers on different GPUs and pipelines computation across layers. While PP can increase total system throughput by processing multiple micro-batches in flight simultaneously, it does not reduce the latency of any single forward pass through a layer. For interactive decoding where TTL is the binding constraint, PP provides no benefit—each token must still traverse all layers sequentially, and the per-layer KV and FFN read costs remain unchanged. The paper includes PP in its baseline search space but does not expect it to help with the TTL problem specifically.

3. Medha-style KV Parallelism (KVP) [7] leaves FFN underutilized.

Medha and similar approaches shard the KV cache along the sequence dimension across a pool of $N$ GPUs using KV Parallelism. Each GPU stores only $S/N$ of the sequence, reducing per-GPU KV read volume by a factor of $N$. This effectively tackles the attention bottleneck at large context lengths.

However, Medha then gathers the attention outputs onto a fixed group of TP GPUs (e.g., 8 GPUs) for all subsequent FFN computation. The remaining GPUs that participated in KVP-attention become idle during the FFN phase—they contribute to attention but then sit unused while the TP group handles the FFN. This creates a sharp asymmetry: as $N$ (the number of KVP GPUs) grows to handle longer contexts, more and more GPUs are underutilized during the FFN phase. The FFN weight reads remain bottlenecked on the small fixed TP group, and the extra GPUs contribute nothing to amortizing that cost.

The paper states this critique explicitly:

"while KVP fans out computation across $N$ GPUs for attention, it does not repurpose those same GPUs to further accelerate FFN execution. As a result, FFN weight loads remain a latency bottleneck, and hardware resources become increasingly underutilized as $N$ grows."

For dense models, this is wasteful. For MoE models (like DeepSeek-R1), the situation is worse: Medha does not provide results on MoE architectures, and the tight coupling of TP between attention and FFN layers is especially problematic when the model uses MLA attention, where $K=1$ and any $TP > 1$ already causes full KV cache duplication during attention.

The Deeper Insight: Attention and FFNs Have Fundamentally Different Sharding Needs

The paper's motivation crystallizes around a single organizing observation: the optimal way to shard computation during attention is qualitatively different from the optimal way to shard computation during FFN, and forcing them to use the same parallelism strategy creates an irreconcilable tension.

• During attention, the bottleneck is KV cache reads—specifically, the total volume of data that must move from DRAM per token. The optimal strategy is to shard the KV cache along the sequence dimension (KVP), because this directly reduces the per-GPU read volume. Sharding along the head dimension (TP) only helps up to $K$ heads and then forces duplication. So attention wants KVP scaling with potentially modest TP.

• During FFN, the bottleneck is weight reads—the $H \times F$ and $F \times H$ matrices that must be loaded per token. The optimal strategy is to shard these weights along the hidden or intermediate dimension (TP), because this divides the per-GPU weight volume by the parallelism width. FFN sharding benefits from as many GPUs as possible, with no natural ceiling analogous to $TP \leq K$ in attention.

A strategy that ties these together—using the same TP width for both attention and FFN, as in conventional TP and Medha—forces a compromise: either cap TP at $K$ (leaving FFN under-sharded) or exceed $K$ (causing KV duplication in attention). Neither choice is optimal for both phases.

The paper's framing: this is fundamentally a resource allocation problem across temporal phases. The same GPUs can serve different roles at different points in each transformer layer, and the parallelism strategy should be phase-aware—KVP-dominant during attention, TP- (or TP×EP-) dominant during FFN.

How This Paper Positions Itself

The paper positions Helix Parallelism not as a modification to model architecture or attention algorithms, but as a systems-level parallelism framework that is orthogonal to (and compatible with) advances in both. It builds on:

• FlashAttention [9] for efficient local attention computation on each GPU's KV shard.
• Flash-Decoding [10] for the mathematical technique of computing partial attention with log-sum-exp scalars and combining them via a single communication round—this is how Helix preserves exact attention semantics while sharding the KV cache.
• Medha [7] for the basic insight that KV Parallelism along the sequence dimension can dramatically reduce per-GPU KV cache pressure.

Where Helix departs from prior work is in decoupling the mapping for attention and FFN within a single layer, then reusing the same physical GPUs for different parallelism configurations in a temporal pipeline. This is not obvious a priori because it introduces a communication step (the all-to-all exchange between attention and FFN) that does not exist in conventional tightly-coupled TP. The paper argues that on modern GPU systems with large NVLink domains (like GB200 NVL72), this communication cost is manageable—especially with the HOP-B overlap optimization—and is more than offset by the efficiency gains from being able to shard both KV caches and FFN weights optimally.

The paper explicitly claims novelty in being the first to address decoding-phase bottlenecks for modern architectures with ultra-long contexts:

"To the best of our knowledge, Helix is the first parallelism framework explicitly designed to address decoding bottlenecks in modern LLM architectures with increasing context lengths. By decoupling sharding strategies for attention and FFNs and introducing a temporal execution pipeline, Helix better aligns GPU utilization with the computational characteristics of each stage."

It also emphasizes compatibility: Helix works with GQA (Llama-405B), MLA (DeepSeek-R1), dense and MoE FFNs, and is co-designed with Blackwell's large NVLink domain to leverage high-bandwidth inter-GPU communication. The paper positions itself as a general-purpose inference optimization that specializes to data-parallel attention + tensor-parallel FFN in the short-context regime (a pattern already widely used [18, 19]) and extends naturally to long-context workloads that break conventional parallelism strategies.

3. Technical Approach

3.1 Reader Orientation

Helix Parallelism is a GPU orchestration strategy — a runtime system that decides how to assign different parts of a transformer layer's computation to different GPUs and how to move data between them, with the specific goal of making interactive LLM decoding feasible even when the model must attend to millions of previous tokens. The system solves a resource-allocation deadlock: conventional approaches force you to either choke on KV cache reads (if you use many GPUs for attention) or choke on FFN weight reads (if you don't), because they lock attention and FFN into using the same parallelism configuration. Helix breaks this lock by treating each transformer layer as a two-phase temporal pipeline — attention first, then FFN — and reconfigures the GPU mapping between phases, applying the parallelism strategy that each phase needs rather than a one-size-fits-all compromise.

3.2 Big-Picture Architecture (Diagram in Words)

The system operates on a pool of $N$ GPUs connected by high-bandwidth NVLink (within a single GB200 NVL72 node, supporting up to 72 GPUs). For each transformer layer during autoregressive decoding, Helix executes these components in sequence:

1. Attention-phase configuration. The $N$ GPUs are arranged into a 2D grid of $KVP \times TP_A$ GPUs, where $KVP$ is the number of GPUs doing KV Parallelism (sharding the KV cache along the sequence dimension) and $TP_A \leq K$ is the number doing Tensor Parallelism (sharding along the query-head dimension). Crucially, $TP_A$ is capped at $K$ (the number of KV heads) to prevent KV cache duplication. Each GPU independently computes full QKV projections for the input batch, runs FlashAttention on its local KV shard, produces partial attention outputs and log-sum-exp statistics, then participates in a single all-to-all communication over the query-head axis to reconstruct exact softmax-normalized attention for all tokens.

2. Post-attention linear projection. The normalized attention outputs flow through the output projection matrix $W_O$, which is sharded via Tensor Parallelism across all $N$ GPUs. An all-reduce aggregates the partial projections into the full output, which then passes through layer normalization.

3. FFN-phase reconfiguration. The same $N$ GPUs are immediately re-provisioned: for dense models, they form a single $TP_F = N$ tensor-parallel group to shard the FFN weight matrices; for MoE models, they form a $TP_F \times EP$ grid where $EP$ is the expert-parallel width, with tokens routed to appropriate experts. The FFN computation (two or three linear projections with activation functions) executes in this new layout, with communication patterns (all-reduce, all-gather) specific to the chosen parallelism scheme.

4. KV cache update. After the FFN completes, the newly generated token's key and value vectors are appended to the KV cache. Helix uses a staggered round-robin strategy: KV pairs are appended to KVP rank 0 for the first 16 decode steps, then rank 1 for the next 16, cycling through all $KVP$ ranks, so that KV cache growth is balanced across the KV parallelism group without requiring every GPU to store every new token's KV entries.

The information flow is: input batch $[B, H]$ → each GPU computes QKV → local FlashAttention on KV shard → all-to-all exchange and rescaling → $W_O$ projection (TP) → all-reduce → layer norm → FFN (TP or TP×EP) → next layer. The same GPUs participate in every step, but their roles (what data they hold, what computation they perform) change between attention and FFN.

3.3 Roadmap for the Deep Dive

This is a systems design paper whose core idea is that decoupling attention and FFN parallelism mappings — and reusing the same GPUs for both in a temporal pipeline — enables optimal sharding for each phase without sacrificing the other. The deep dive follows this structure:

• First, the formal model of the two bottlenecks (KV cache reads and FFN weight reads), expressed as DRAM read-time equations, because these equations define the optimization target and explain why the baseline strategies fail.
• Second, the attention-phase configuration: how KVP and TP are composed, why $TP_A$ is capped at $K$, how the forward pass works on each GPU, and the all-to-all communication that reconstructs exact attention.
• Third, the HOP-B overlap optimization, which pipelines communication and computation across the batch dimension to hide the all-to-all latency — this is what makes Helix practical by recovering TTL that would otherwise be lost to communication.
• Fourth, the FFN-phase reconfiguration: how the GPU layout switches from $KVP \times TP_A$ to $TP_F \times EP$, and why this switch enables FFN sharding beyond $K$ GPUs without reintroducing KV duplication.
• Fifth, the distributed KV cache concatenation strategy, which balances memory growth across KVP ranks during decoding.
• Sixth, the end-to-end orchestration: how these components chain together across all transformer layers, and how the framework specializes to different model architectures (dense GQA, MoE with MLA).

3.4 Detailed, Sentence-Based Technical Breakdown

Framing: This Is a Systems Optimization Paper with a Hard Mathematical Constraint

Helix Parallelism is fundamentally a **solution to a constrained resource allocation problem on GPU hardware. The paper's contribution is not a new attention mechanism or model architecture — it is a deployment-time strategy that changes how existing models are mapped to hardware, without changing the model's output. The core insight has a sharp mathematical form: the per-layer DRAM read time for KV cache and FFN weights obeys two different scaling laws with respect to parallelism width, and the standard approach of using the same TP width for both phases forces a suboptimal point on both curves. Helix's value proposition is that by allowing the parallelism width to differ between phases (via a temporal reconfiguration), the system can operate at a better point on both curves simultaneously — provided the reconfiguration cost (communication) is low enough.

Let us build up from the fundamental costs, through the attention and FFN mechanisms, to the full orchestration.

---

The Two Bottlenecks as DRAM Read-Time Equations

The paper provides explicit formulas for the DRAM read time per transformer layer (Appendix A). These are not abstract Latency estimates — they are the physical time the GPU spends waiting for data to arrive from DRAM, computed from model dimensions, batch size, parallelism configuration, and hardware bandwidth. Understanding these equations is essential because they explain why conventional strategies hit ceilings and what Helix changes.

KV cache read time per layer:

$\text{Time}_{\text{KV}} = \frac{B \times 2 \times \lceil \frac{K}{TP_A} \rceil \times \text{Hsz} \times \frac{S}{KVP} \times \text{bytes\_per\_param}}{\text{MemBW}}$

where $B$ is batch size, $K$ is total number of KV heads, $TP_A$ is tensor-parallel width for attention, $\text{Hsz}$ is the attention head size (the dimension of each key/value vector per head), $S$ is the total KV sequence length, $KVP$ is the KV-parallel width (number of GPUs sharding the sequence), $\text{bytes\_per\_param}$ is the storage precision (e.g., 1 byte for FP8, 0.5 bytes for FP4), and $\text{MemBW}$ is the GPU's DRAM bandwidth.

What this computes: the total time one GPU spends reading its portion of the KV cache from DRAM during the attention phase of one transformer layer. The numerator is the total bytes read per GPU: batch size $B$ times 2 (for both keys and values) times the number of KV heads assigned to this GPU ($\lceil K/TP_A \rceil$) times head size $\text{Hsz}$ times the sequence length per GPU ($S/KVP$) times bytes per parameter. Dividing by memory bandwidth $\text{MemBW}$ converts bytes to seconds.

Why this form: the ceiling function $\lceil K/TP_A \rceil$ is the critical term. When $TP_A \leq K$, each GPU gets at most $\lceil K/TP_A \rceil$ KV heads without duplication — the KV heads are partitioned across GPUs. When $TP_A > K$, the ceiling saturates at 1 (each GPU must hold at least one full KV head), but since there are only $K$ distinct KV heads total, multiple GPUs must share the same KV heads — meaning those heads are duplicated across GPUs. The per-GPU read volume stops decreasing because every GPU now stores the full KV cache for at least one head. The denominator's $KVP$ term is the other lever: sharding the sequence dimension divides the per-GPU KV read volume by $KVP$, which always helps regardless of $TP_A$.

FFN weight read time per layer:

$\text{Time}_{\text{FFN}} = \frac{\left(2 \times H \times \frac{Q}{TP_A} \times \text{Hsz}\right) + \left(2 \times H \times \lceil \frac{K}{TP_A} \rceil \times \text{Hsz}\right) + \left(3 \times \frac{H \times F}{TP_F}\right) \times \text{bytes\_per\_param}}{\text{MemBW}}$

where $H$ is the hidden dimension ($H = Q \times \text{Hsz}$), $Q$ is the number of query heads, $F$ is the FFN intermediate dimension, and $TP_F$ is the tensor-parallel width for the FFN. The factor of 3 in the FFN term accounts for a SwiGLU activation with three weight matrices (gate, up, and down projections — the paper notes modern designs often employ gated variants that effectively triple the parameter count compared to a simple two-layer FFN).

What this computes: the total time one GPU spends reading attention projection weights ($W_Q, W_K, W_V$) and FFN weights from DRAM during one transformer layer. The first term covers $W_Q$ and the query-projection portion of the weights, scaled by $Q/TP_A$. The second term covers $W_K$ and $W_V$, scaled by $\lceil K/TP_A \rceil$. The third term covers all FFN matrices, sharded by $TP_F$ along the intermediate dimension. All terms are divided by $\text{MemBW}$.

Why this form: the per-GPU weight-read volume for attention weights stops decreasing once $TP_A > Q$ (for Q) or $TP_A > K$ (for K and V). But the FFN term can continue decreasing as $TP_F$ grows — there is no intrinsic ceiling like $K$ for FFN weights. This asymmetry is the root cause of the tension: attention benefits from TP only up to $K$ (or $Q$), while FFN benefits from as much TP as available. Forcing $TP_A = TP_F$ (as in conventional TP) means either leaving FFN under-sharded (if you stop at $K$) or incurring KV duplication in attention (if you exceed $K$).

Roofline context (Figure 1). The paper plots these equations for Llama-405B ($Q=128, K=8, \text{Hsz}=128, F=65536$) on GB200 NVL72 with $B=8$, $S=1\text{M}$, FP4 precision, and $\text{MemBW}=8000$ GB/s. The left panel shows that KV read time drops from $TP=1$ to $TP=8$ but is flat for $TP>8$ — the duplication ceiling. The middle panel shows attention read time scaling linearly with $S$, eventually dominating. The right panel shows that increasing $KVP$ (for a fixed $TP$) continues to reduce KV read time sublinearly — sharding the sequence works even when sharding heads has maxed out. These plots are the quantitative motivation for Helix: you need both $KVP$ (to reduce per-GPU KV volume beyond what TP alone can do) and high $TP_F$ (to reduce FFN reads), and you need to apply them to different phases so they don't conflict.

---

Attention-Phase Configuration: KVP × TP_A with No Duplication

During the attention phase, Helix arranges all $N$ available GPUs into a 2D grid of dimensions $KVP \times TP_A$, where:

$N = KVP \times TP_A$

and the constraint $TP_A \leq K$ is strictly enforced. This constraint is the design rule that prevents KV cache duplication — by keeping the tensor-parallel width for attention at or below the number of KV heads, every KV head can be assigned to exactly one GPU within each KVP column, and no head needs to be replicated across GPUs.

Why $TP_A \leq K$ is the hard rule. In grouped-query attention (GQA), there are $Q$ query heads but only $K$ distinct key and value heads, with $K < Q$ (typically $K=8$ for Llama-405B). Each KV head serves $Q/K$ query heads. Tensor Parallelism splits query heads across GPUs: with $TP_A$ GPUs, each gets $Q/TP_A$ query heads. If $TP_A \leq K$, then the $K$ KV heads can be partitioned so that each GPU's assigned query heads only need the KV heads that GPU holds — no cross-GPU KV access is needed during the attention computation itself. If $TP_A > K$, then some KV heads must serve query heads on multiple GPUs, requiring those KV heads to be stored on multiple GPUs (duplication). The paper gives the concrete example: for Llama-405B with $K=8$ and $Q=128$, if $TP=16$, each GPU gets 8 query heads and must store all 8 KV heads to serve them, meaning every GPU stores the full KV cache.

What happens on each GPU during attention (Figure 4, top). The forward pass proceeds in these steps:

1. Full QKV projection. Every one of the $N$ GPUs takes the full input batch $[B, H]$ and multiplies it by its local shards of the weight matrices:

   • $W_Q \in \mathbb{R}^{H \times (H/TP_A)}$: produces $Q/TP_A$ query heads' worth of queries.
   • $W_K \in \mathbb{R}^{H \times (\lceil K/TP_A \rceil \cdot \text{Hsz})}$: produces $\lceil K/TP_A \rceil$ KV heads' worth of keys.
   • $W_V \in \mathbb{R}^{H \times (\lceil K/TP_A \rceil \cdot \text{Hsz})}$: produces $\lceil K/TP_A \rceil$ KV heads' worth of values.

   All GPUs compute their full assigned projections independently — there is no pre-attention communication. This is a deliberate design choice: the paper states it is done "to avoid an expensive pre-attention All-Gather of queries across the KVP GPUs." If queries were gathered before the QKV projections, every KVP GPU would need to receive query data from every other KVP GPU, adding communication that scales with batch size and hidden dimension. By instead having every GPU compute its own QKV projections from the full input batch, the only communication happens after attention, when the partial outputs need to be combined.

2. FlashAttention on local KV shard. Each GPU runs FlashAttention [9] on its portion of the KV cache. The KV cache is sharded along the sequence dimension: GPU $(k, t)$ in the $KVP \times TP_A$ grid holds the keys and values for sequence positions $[k \cdot S/KVP, (k+1) \cdot S/KVP)$ and for KV heads assigned to TP column $t$. The queries are the full set for the current batch and TP column $t$'s query heads. FlashAttention computes, in a single fused kernel without materializing the full $B \times Q/TP_A \times S/KVP$ attention matrix:

   • Partial attention outputs: a tensor of shape $[B, Q/TP_A, \text{Hsz}]$ representing the attention-weighted sum of values over the local KV shard.
   • Log-sum-exp scalars: a vector of length $B \times Q/TP_A$ storing $\log(\sum \exp(\text{attention scores}))$ for each query token, which are needed for the subsequent exact recombination.

   The key property is that each GPU only touches its local KV shard during this computation — there is no cross-GPU attention. The KV shard size is $S/KVP$ tokens per GPU, which is the mechanism by which KVP reduces per-GPU read volume.

3. All-to-all communication and exact softmax reconstruction. After local attention, each GPU has partial outputs for its query heads and its KV shard. To reconstruct the full softmax-normalized attention (i.e., attention over the entire sequence, not just the local shard), the partial outputs must be combined across KVP ranks. The paper uses the Flash-Decoding [10] technique: the attention output over a partitioned sequence can be exactly reconstructed by exchanging partial weighted sums and log-sum-exp statistics, then rescaling and summing.

   Concretely, the all-to-all communication works as follows:

   • Each GPU sends its $B \times \frac{H}{KVP \times TP_A}$ partial attention outputs and corresponding log-sum-exp scalars to every other GPU in the KVP domain (but only for matching query-head assignments).
   • After the exchange, each GPU holds, for its assigned query heads, the partial outputs from all $KVP$ sequence shards.
   • Each GPU rescales each received partial output by $\exp(\text{local\_lse} - \text{global\_lse})$ where global_lse is the log-sum-exp of all local LSEs, then sums them. This yields the exact softmax-normalized attention output — mathematically identical to having computed attention over the full unsharded KV cache.

   The paper emphasizes that this is "a single All-to-All over the query-head axis" with "no extra synchronization or normalization passes." The communication volume is independent of the KV sequence length $S$ and scales only with the number of query tokens in the batch ($B$) and the hidden dimension ($H$). Specifically, each GPU sends and receives $B \times H / (KVP \times TP_A)$ elements. This is the property that makes Helix scalable: no matter how long the context is, the communication cost per token is constant.

The GPU layout during attention (Figure 2, right panel). For a model with $Q=4$ query heads and $K=2$ KV heads (pedagogical example), Helix with $TP_A=2$ and $KVP=2$ produces a $2 \times 2$ grid. Each GPU holds $S/2$ of the sequence (KVP sharding) and $2$ query heads / $1$ KV head (TP sharding). No KV head is duplicated — each of the 2 distinct KV heads exists on exactly 2 GPUs (one per KVP rank, since each KVP rank needs the full KV head for its sequence shard). The total KV storage per GPU is $(K/TP_A) \times (S/KVP) \times \text{Hsz}$ elements, compared to $K \times S \times \text{Hsz}$ for the TP-only baseline with $TP > K$ (which duplicates the full KV cache on every GPU).

The post-attention $W_O$ projection. After the all-to-all, each GPU holds the normalized attention outputs for the full batch but only for its assigned hidden-dimension slice of size $H/(KVP \times TP_A) = H/N$. The output projection matrix $W_O \in \mathbb{R}^{H/N \times H}$ (sharded across all $N$ GPUs) multiplies this slice to produce a partial projection of shape $B \times H/N$. An all-reduce across all $N$ GPUs then aggregates these partial projections into the full $B \times H$ output that feeds into layer normalization. The paper notes that this post-attention re-provisioning (from KVP layout to a pure TP layout across all $N$ GPUs) is identical for dense and MoE models.

---

HOP-B: Batchwise Communication-Computation Overlap

The all-to-all communication in the attention phase introduces latency that does not exist in conventional tightly-coupled TP. If executed naively (all GPUs compute attention for the entire batch, then all GPUs participate in the all-to-all, then all GPUs proceed to $W_O$), the communication is exposed: GPUs sit idle waiting for data while the all-to-all completes. The paper quantifies this with a concrete example (Figure 3, top): with 8 requests, each taking 16 time units of attention compute followed by 9.6 units of communication, the total time span is $16 + 9.6 = 25.6$ units — the communication adds 60% overhead.

What HOP-B changes. HOP-B (Helix Overlap Pipeline – Batch-wise) breaks the batch into finer-grained units and pipelines computation and communication across them (Figure 3, bottom). The key observation is that the all-to-all for token $i$ in the batch does not depend on the attention computation for token $i+1$ — they are independent. Therefore, as soon as GPU $g$ finishes computing attention for token $i$, it can immediately initiate the all-to-all send for token $i$'s partial outputs while simultaneously beginning attention computation for token $i+1$.

How the pipelining works in practice:

• The batch of $B$ tokens is processed sequentially or in micro-sub-batches (the paper does not specify the exact granularity, but the principle is clear from Figure 3).
• GPU $g$ computes attention for sub-batch $i$, producing partial outputs and LSE scalars.
• GPU $g$ posts the all-to-all send for sub-batch $i$'s data to the NVLink fabric.
• Without waiting for the receive to complete, GPU $g$ immediately begins attention compute for sub-batch $i+1$.
• Meanwhile, the NVLink fabric transmits sub-batch $i$'s data, and the receive is posted asynchronously.
• Once sub-batch $i+1$'s attention compute is done, the receive for sub-batch $i$ has likely completed, so GPU $g$ can rescale and sum sub-batch $i$'s outputs while posting the send for sub-batch $i+1$.

The paper shows this reduces the total time span from 25.6 units to 17 units in the example: the communication is almost entirely hidden behind the next sub-batch's compute, with only the final sub-batch's communication (1.2 units) being exposed. The "TTL Saving" is shown as 25.6 - 17 = 8.6 units.

Why this matters differentially across models. The paper's ablation (Figure 7) reveals that HOP-B's benefit is significant for Llama-405B (~12% improvement in Tokens/s/User) but negligible for DeepSeek-R1 (~1%). The reason is architectural: DeepSeek-R1 uses MLA attention, where the key and value projections are absorbed into a latent space, making the attention computation itself very lightweight. The all-to-all communication therefore accounts for roughly 1% of total end-to-end decode latency — hiding 1% of the latency cannot improve overall performance by more than ~1%. In contrast, Llama-405B uses explicit GQA with materialized KV heads, so attention computation is heavier relative to overall layer cost, making the all-to-all a larger fraction of TTL and making overlap more impactful. This differential impact is an important practical detail: HOP-B is essential for models where attention communication is a non-trivial fraction of layer time, but it provides diminishing returns as the FFN or other components dominate.

---

FFN-Phase Reconfiguration: Switching from KVP to TP (or TP×EP)

After the post-attention $W_O$ projection and layer normalization, Helix reconfigures the GPU layout for the FFN computation (Figure 4, bottom). The reconfiguration changes what data each GPU holds and how they collaborate, but it does not involve physically moving model weights between GPUs — the weight shards for different parallelism configurations are pre-loaded and the switch is a logical reassignment of which GPUs participate in which collective operations.

For dense FFNs (EP = 1): all $N$ GPUs form a single tensor-parallel group with $TP_F = N$. The FFN weight matrices are sharded along the intermediate dimension $F$: each GPU holds a slice of shape $H \times F/N$ for the up-projection, $F/N \times H$ for the down-projection, and (for SwiGLU) $H \times F/N$ for the gating projection. The computation proceeds as:

• Each GPU multiplies the full input $[B, H]$ by its weight shards to produce partial activations of shape $[B, F/N]$.
• The SwiGLU activation function (element-wise gating) is applied locally.
• Each GPU multiplies the activated partial outputs by its down-projection shard.
• An all-reduce across all $N$ GPUs aggregates the partial results into the final $B \times H$ FFN output.

The per-GPU FFN weight read volume is now $3 \times H \times F/N \times \text{bytes\_per\_param}$, which decreases linearly with $N$. Since $N$ can be much larger than $K$ (the attention TP ceiling), this represents a substantial improvement over conventional TP where $TP_F$ would be capped at $K$. For Llama-405B with $K=8$, Helix could use $N=64$ GPUs, reducing per-GPU FFN weight reads by a factor of 8× compared to $TP=8$.

For MoE FFNs (EP > 1): the $N$ GPUs are repartitioned into a $TP_F \times EP$ grid, where $EP$ is the expert-parallel width. The routing mechanism assigns each token in the batch to one or more experts, and each expert's weight matrices are sharded via $TP_F$ across a subset of GPUs. The computation proceeds as:

• Token routing: the router (a small learned function) assigns tokens to experts.
• Within each expert group (a $TP_F$-GPU subgroup), the assigned tokens are processed via TP-sharded FFN layers (same as the dense case, but only for the tokens routed to that expert).
• An intra-expert all-reduce aggregates partial outputs within each expert group.
• An inter-expert all-gather distributes the expert outputs so that each GPU receives the results for tokens originally on other GPUs.
• A local reduction combines multiple expert outputs for tokens routed to multiple experts (e.g., top-2 routing), yielding the final $B \times H$ output.

The paper states that the $TP_F$ vs. $EP$ split is chosen "to best match expert size and quantity" — a parameter sweep over possible grid configurations is part of the exhaustive search for Pareto-optimal configurations. For DeepSeek-R1, which has a large number of experts, the ability to flexibly allocate GPUs between TP (reducing per-expert weight reads) and EP (increasing expert throughput by processing different experts in parallel) is claimed to be a key advantage over Medha-style approaches that lock FFN parallelism to the attention configuration.

Why this reconfiguration avoids KV duplication. The critical design property is that the reconfiguration happens after the attention phase is complete. During attention, the GPU layout was $KVP \times TP_A$ with $TP_A \leq K$, ensuring no KV duplication. During FFN, the KV cache is not accessed — only FFN weights are read. So the fact that $TP_F$ can now exceed $K$ does not cause KV duplication, because the KV cache is not being read during FFN. The temporal separation is what breaks the coupling: the policy for reading KV caches (attention phase) uses a layout that prevents duplication, while the policy for reading FFN weights (FFN phase) uses a layout that maximizes sharding, and these policies do not interfere because they apply at different times.

The zero-downtime pipelining claim. The paper states that Helix "switches between these configurations seamlessly, enabling zero-downtime pipelining and better GPU utilization." This means the configuration switch is a control-plane operation (changing which collective communication groups GPUs belong to) that does not require data movement or recomputation. The weight matrices for both configurations (attention and FFN) are pre-loaded in GPU memory, and the switch merely changes which shards are accessed and which communication primitives are called.

---

Distributed KV Cache Concatenation Strategy

During autoregressive decoding, every new token must be added to the KV cache so that future tokens can attend to it. In a KVP setup, the natural question is: which GPU(s) store the new token's KV entries? The simplest approach — store on all KVP GPUs — would cause the per-GPU KV cache to grow at the full rate of $1 \times K \times \text{Hsz} \times \text{bytes\_per\_param}$ per token, defeating the purpose of KVP sharding. The ideal approach — store each new token on exactly one KVP GPU — requires a policy for which GPU.

Helix's staggered round-robin strategy. The paper specifies a concrete policy (Section 2.3):

• KV pairs for the first 16 decode steps are appended to the KV shard on KVP rank 0.
• KV pairs for the next 16 decode steps are appended to KVP rank 1.
• This continues, cycling through all $KVP$ ranks, then wrapping around to rank 0.
• The fixed block size is 16 tokens.

Why this design. The paper's stated rationale is to "maintain balanced memory growth and ensure each GPU appends KV entries uniformly" and to "guarantee that all KVP GPUs contribute to KV storage regardless of batch size or sequence length, avoiding hot spots and distributing KV cache growth evenly across the pool." The round-robin with fixed block size (rather than, say, appending to a different GPU for every single token) is likely chosen to balance two concerns: (1) finer granularity (1 token per GPU) would require more frequent communication to broadcast the new token to the correct GPU, and (2) coarser granularity (e.g., 256 tokens) would create longer periods of unbalanced memory pressure where one GPU's cache grows while others remain static. The 16-token block is a empirically chosen tradeoff.

How the broadcast works. The paper notes that "each newly generated token is broadcast to all KVP GPUs so that every device has access to the current query." This broadcast is necessary because every KVP GPU needs the current token's query vector to compute attention during the next decode step — the query is not sharded. The broadcast is a small communication (size $B \times H$ per token) compared to the all-to-all during attention. After the broadcast, only the designated KVP rank (determined by the round-robin schedule) actually appends the KV entries to its local cache; other ranks discard them after using them for attention.

Implications for attention computation. During subsequent decode steps, when a GPU computes attention over its KV shard, its shard contains a mixture of tokens: the initial precomputed context (evenly distributed during prefill, though prefill is not discussed in detail) plus the decode tokens assigned to it by the round-robin policy. Since the assignment is deterministic and known, the attention computation remains exact — there is no approximation from the staggered storage. Each GPU's KV shard simply contains the subset of the total sequence that the round-robin policy assigned to it.

---

End-to-End Orchestration and Architectural Specialization

Helix applies the same two-phase strategy (attention with $KVP \times TP_A$, FFN with $TP_F \times EP$) to every transformer layer. There is no layer-wise variation in the parallelism configuration — the same GPU layout is used for all layers, with the same configuration switch between attention and FFN within each layer. This uniformity simplifies implementation because the communication groups and weight sharding patterns are set up once and reused across layers.

Specialization to different model architectures. The paper emphasizes that Helix works for both dense and MoE models, and for both GQA and MLA attention, but the specific configuration choices differ:

• Llama-405B (dense, GQA): $Q=128$, $K=8$. The attention phase uses $TP_A \leq 8$ to avoid KV duplication, with the remaining GPUs allocated to $KVP$ for sequence sharding. The dense FFN phase uses all $N$ GPUs as a single $TP_F = N$ group. The evaluation sweeps $N$ from 1 to 64 GPUs and finds Pareto-optimal configurations at various points.

• DeepSeek-R1 (MoE, MLA): MLA attention absorbs key and value projections into a latent space, effectively resulting in $K=1$ (a single shared KV representation). This means $TP_A$ is capped at 1 — any $TP > 1$ would cause full KV duplication during attention. Therefore, for DeepSeek-R1, the attention phase uses $TP_A = 1$ and all $N$ GPUs are allocated to $KVP = N$ (pure KV parallelism). The FFN phase reconfigures into a $TP_F \times EP$ grid, where $TP_F$ and $EP$ are chosen based on the number of GPUs and expert characteristics. The evaluation sweeps this grid exhaustively.

The baseline comparison space. The paper evaluates Helix against a "baseline search space" that covers "the best-known partitioning strategies: tensor parallelism, pipeline parallelism, expert parallelism, and vanilla KV partitioning." For each strategy and each batch size, the configuration that maximizes throughput under a given TTL constraint is selected. "Vanilla KVP" refers to Medha-style sharding where attention uses KVP but FFN is tied to a fixed TP group. PP is included in the search space but, as noted in the motivation section, it does not help with per-layer latency and is not expected to be Pareto-optimal for TTL-constrained decoding. The exhaustive search covers "over 100,000 configurations, systematically varying model partitioning strategies, batch sizes, and GPU counts."

The configurability-exploitability tradeoff. Helix introduces several new degrees of freedom compared to the baseline: the split between $KVP$ and $TP_A$ during attention, the split between $TP_F$ and $EP$ during FFN, and the total number of GPUs $N$. The paper's approach is to include these as searchable parameters and let the Pareto-optimal frontier emerge from exhaustive simulation, rather than prescribing fixed ratios. This is appropriate for a paper that introduces a new parallelism framework rather than a specific fixed configuration — the claim is that the framework expands the set of achievable (throughput, latency) points, and the best point depends on the specific model, hardware, and TTL budget.

What the paper does NOT implement. The paper does not combine PRM-guided search or iterative revision with Helix — it is purely a systems contribution. It does not modify model weights, attention algorithms, or token generation policies. The output of a model running under Helix is bit-identical to the output of the same model running under conventional TP (assuming the same sampling parameters), because the all-to-all recombination with log-sum-exp scalars preserves exact softmax attention. The compatibility with FlashAttention, GQA, MLA, and MoE is achieved by design, not through model modification — Helix operates at the parallelism layer below the model architecture.

4. Key Insights and Innovations

Innovation 1: Recognizing That Attention and FFN Phases Demand Fundamentally Different — and Incompatible — Parallelism Strategies

The paper's most foundational conceptual move is not the mechanism of decoupling, but the diagnosis that makes decoupling necessary in the first place: the optimal parallelism strategy for the attention phase of a transformer layer is structurally incompatible with the optimal strategy for the FFN phase, and forcing them to share a single configuration creates an unavoidable deadlock.

Prior work implicitly accepted that attention and FFN should be sharded using the same parallelism strategy within each layer. Tensor Parallelism [6] splits both attention heads and FFN weights across the same set of GPUs with the same TP width. Medha-style KV Parallelism [7] shards KV caches across an expanded pool of GPUs for attention, but then gathers results onto a fixed TP group for FFN — still tying the FFN parallelism width to the attention configuration, just asymmetrically. The dominant assumption was that maintaining a static GPU-to-computation mapping across the entire layer was necessary for efficiency, presumably to avoid communication overhead from reconfiguration.

Helix's contribution at the idea level is to name and prove the incompatibility, then show that the cost of switching mappings (a single all-to-all) is acceptable on modern hardware. The incompatibility has a precise mathematical form expressed through the two DRAM read-time equations in Appendix A, but the conceptual insight is simpler: attention's bottleneck is KV cache reads, which scale with sequence length and benefit most from sequence-dimension sharding (KVP) since head-dimension sharding (TP) hits a hard ceiling at the number of KV heads $K$. FFN's bottleneck is weight reads, which scale with model size and benefit from as much hidden-dimension sharding (TP) as available, with no intrinsic ceiling. When you set $TP_A = TP_F$ (the conventional approach), you must choose between capping at $K$ (under-sharding FFN, leaving weight reads as the dominant latency) or exceeding $K$ (causing full KV cache duplication on every GPU, making attention reads the dominant latency). Neither choice is acceptable for million-token contexts under tight TTL budgets, and the roofline analysis in Figure 1 makes this quantitatively visible: KV read time flatlines for $TP > 8$ while FFN read time continues decreasing, so any fixed TP width is either attention-bottlenecked (if $TP > K$) or FFN-bottlenecked (if $TP \leq K$).

This is a fundamental reframing rather than an incremental optimization. It transforms the problem from "what is the best single parallelism strategy for a transformer layer?" to "how should parallelism strategies differ between the two phases of a transformer layer, and what is the minimum-cost mechanism for switching between them?" The paper's answer — a single all-to-all with volume independent of sequence length, plus overlap via HOP-B — demonstrates that the switching cost is low enough to make decoupling practical, but the intellectual contribution is the reframing itself. Prior work had not articulated that the two phases have fundamentally different optimal mappings, let alone proven that the pursuit of a single unified mapping is the root cause of the scaling ceiling.

Innovation 2: The Temporal-Pipeline GPU Reuse Model as a New Point in the Parallelism Design Space

Where Innovation 1 is about why decoupling is necessary, Innovation 2 is about what decoupling enables: a new class of parallelism strategies where the same physical GPUs serve entirely different roles at different points in the layer execution timeline. The paper calls this "zero-downtime pipelining," but the deeper idea is that GPU-to-computation mapping should be a function of time, not a static property of the model layer.

This departs from all major prior parallelism paradigms. Data Parallelism assigns the same computation to different GPUs on different data shards — role is static. Tensor Parallelism assigns different weight shards to different GPUs — role is static. Pipeline Parallelism assigns different layers to different GPUs — role is static across time within a layer's execution, even though different micro-batches flow through. Expert Parallelism assigns different experts to different GPUs — role is static. In every case, if GPU $i$ is responsible for weight shard $W_i$ or layer $L_i$, that assignment persists throughout the entire forward pass. The conceptual space of parallelism design had assumed static mappings.

Helix introduces a dynamic role assignment: during attention, the $N$ GPUs are a $KVP \times TP_A$ grid where each GPU manages a sequence shard and a query-head shard. During FFN, the same $N$ GPUs are a $TP_F \times EP$ grid where each GPU manages an expert shard and/or an intermediate-dimension shard. The roles are not just different in scale (different parallelism widths) but different in semantics: during attention, GPU $i$'s identity is defined by which subset of the sequence it stores; during FFN, GPU $i$'s identity is defined by which subset of the weight matrices it stores. The data each GPU holds changes (KV shards vs. FFN weight shards), the communication patterns change (all-to-all during attention vs. all-reduce/all-gather during FFN), and the computational work changes (FlashAttention vs. matrix multiply + activation).

This is a fundamental expansion of the design space rather than a refinement within existing space. The paper is not proposing a better TP width or a better KVP ratio — it is proposing an entirely new axis of design freedom: temporal reconfigurability of GPU roles within a single layer. The significance is that this axis generalizes: the same principle could apply to other computation patterns beyond attention and FFN (e.g., different phases of mixture-of-experts routing, different precisions for different operations, different sparsity patterns), and it suggests that parallelism frameworks should support runtime GPU reassignment as a first-class primitive rather than baking in static mappings. The paper's evaluation demonstrates that this expanded design space contains Pareto-optimal points (Figures 5 and 6) that no static mapping can reach, providing evidence that the expansion is practically valuable, not just theoretically interesting.

Innovation 3: The Concept of "KV Duplication Ceiling" as the Diagnostic for Why TP Fails at Scale

The paper introduces a crisp, quantifiable concept that explains when and why conventional Tensor Parallelism stops being effective for long-context attention: the KV duplication ceiling, defined as the point where $TP_A$ exceeds the number of KV heads $K$, forcing multiple GPUs to store identical copies of the same KV heads and eliminating further reductions in per-GPU KV read volume.

This is a diagnostic innovation — a concept that makes an existing failure mode visible, measurable, and avoidable. Prior work almost certainly encountered the phenomenon (it is a direct consequence of how GQA interacts with TP), but it was not named or systematically analyzed as a first-class constraint. The Medha paper [7] implicitly recognized the problem by introducing KV parallelism to bypass it, but did not articulate the ceiling as a general concept or analyze its consequences for FFN sharding. The Helix paper makes the ceiling explicit through the roofline analysis (Figure 1, left panel) where the KV read-time curve visually flatlines at $TP = K$, and through the ceiling function $\lceil K/TP_A \rceil$ in the KV read-time equation, which mathematically saturates when $TP_A > K$.

The intellectual value of naming this ceiling is that it transforms a vague intuition ("TP doesn't help attention much for large TP widths") into a precise, hardware-independent constraint that can guide system design. The ceiling depends only on the model architecture ($K$, the number of KV heads) and the choice of parallelism strategy ($TP_A$), not on hardware parameters like bandwidth or FLOPs. This means it is a structural property of the model-parallelism interaction, not a contingent property of a particular GPU generation. Any system that uses GQA or MLA with TP will encounter this ceiling, on any hardware, now and in the future.

The concept also clarifies why MLA (with $K=1$) is particularly challenging for conventional TP: the duplication ceiling kicks in immediately at $TP=2$, meaning any tensor parallelism at all during attention causes full KV duplication. For DeepSeek-R1, a large MoE model with MLA that would strongly benefit from TP for its enormous FFN weights, conventional TP is essentially unusable during attention — a point the paper makes explicitly when noting that "Medha's approach of tying TP between FFNs and attention is not well-suited for modern networks with MLA attention." The KV duplication ceiling thus serves double duty: it diagnoses the failure of TP for long-context GQA models, and it explains why the problem is even more acute for next-generation architectures like MLA.

This is a conceptual contribution rather than a metric gain, but it is a practically significant one because it gives system designers a clear rule — "do not set $TP_A > K$ if you care about KV cache read time" — and explains why that rule exists. The paper's own design (capping $TP_A$ at $K$ and using KVP for additional parallelism) follows directly from this diagnostic.

Innovation 4: HOP-B as a Demonstration That Fine-Grained Batchwise Overlap Can Recover Most of the Communication Cost of Dynamic Reconfiguration

The Helix decoupling introduces an all-to-all communication step that does not exist in conventional tightly-coupled TP. The paper could have simply argued that this cost is acceptable on modern hardware with high NVLink bandwidth, but instead it introduces HOP-B — a batchwise pipelining strategy that hides most of the communication behind ongoing computation — and demonstrates that the exposed cost can be reduced to a small fraction of the theoretical maximum.

The intellectual contribution here is not the idea of communication-computation overlap (which is a standard technique in HPC and distributed training), but the specific demonstration that the all-to-all in Helix's attention phase can be nearly fully hidden through a fine-grained, batch-dimension pipeline, and that the residual exposed cost is model-dependent in a predictable way. The paper quantifies this with concrete numbers (Figure 3): without HOP-B, communication adds 60% overhead (25.6 vs. 16 time units); with HOP-B, overhead drops to ~6% (17 vs. 16 time units). More importantly, the ablation study (Figure 7) reveals that the benefit of HOP-B is large for Llama-405B (~12% improvement in Tokens/s/User) but negligible for DeepSeek-R1 (~1%), and the paper traces this difference to the fraction of total layer time spent in attention communication: ~1% for DeepSeek-R1's MLA-based attention vs. a larger fraction for Llama-405B's GQA attention.

This is a practical engineering insight that advances understanding beyond "overlap is good." It establishes that the value of overlap for Helix-style decoupling is architecture-dependent, and it provides a diagnostic (the fraction of TTL spent in the all-to-all) that predicts whether HOP-B is worth implementing. For model deployers, this means: if your model uses MLA or another attention variant where the attention computation itself is extremely lightweight relative to FFN, the communication cost of decoupling may be negligible even without overlap, and the engineering complexity of HOP-B may not be justified. For GQA models where attention is heavier, HOP-B is essential to realizing the gains from decoupling. The paper does not frame it this way, but this is effectively a second-level design rule that complements the KV duplication ceiling: decoupling eliminates the ceiling, and HOP-B eliminates the penalty for decoupling, but the penalty varies by architecture so the need for HOP-B is conditional.

This insight is incremental in technique (batchwise overlap is not novel) but fundamental in implication: it removes the primary objection to dynamic reconfiguration (that communication overhead would negate the benefits) by showing that the overhead can be nearly eliminated in the cases where it is large enough to matter. This is what makes Helix practical rather than merely theoretically appealing.

Innovation 5: The Unified Throughput-Interactivity Pareto Frontier as an Evaluation Methodology

While not a contribution to parallelism strategy per se, the paper's evaluation methodology represents a methodological innovation in how inference serving systems are characterized and compared. Rather than reporting point metrics (latency at a specific batch size, or throughput at a specific latency target), the paper constructs the full throughput (tokens/sec/GPU) vs. interactivity (tokens/sec/user, the reciprocal of TTL) Pareto frontier by exhaustively simulating over 100,000 configurations spanning parallelism strategies, batch sizes, and GPU counts, then plotting only the optimal points.

This methodology addresses a fundamental problem in systems-for-ML evaluation: different parallelism strategies dominate at different points in the throughput-latency tradeoff space, and a single-point comparison (e.g., "Helix achieves X% lower latency at batch size 8") can be misleading because it may compare Helix's optimal point to a baseline's suboptimal point for that particular choice of batch size. By constructing the full Pareto frontier, the evaluation reveals regions of the tradeoff space where one approach dominates, rather than claiming universal superiority. For example, Figure 6 shows that at very low throughput (few concurrent users), the interactivity gap between Helix and Medha narrows, while at higher throughput the gap widens dramatically. A single-point evaluation at either end would give a misleading picture of the relative benefit.

This is incremental as a methodology (Pareto frontiers are standard in computer architecture and systems research), but it is practically significant for the LLM serving community where evaluations have historically been fragmented across different metrics, batch sizes, and latency targets. The paper normalizes all performance numbers to the baseline "to focus on trends as opposed to specific performance claims," which, combined with the Pareto methodology, makes the results more robust to hardware specifics while still conveying the shape of the improvement.

The methodological contribution also serves a communication function: it makes visible the 32× batch scalability claim (for DeepSeek-R1, Figure 5), which means that at the same TTL, Helix can serve 32× more concurrent users than the best baseline configuration. This is not a single-point comparison but a statement about the envelope of achievable operating points — Helix's Pareto frontier extends further along both axes simultaneously, pushing outward rather than trading one for the other. This framing (improving the Pareto frontier, not just moving along it) is a conceptually cleaner way to claim system-level improvement than "X% better latency at Y batch size," and the paper's adoption of it is a contribution to evaluation rigor in the space.

5. Experimental Analysis

Evaluation Methodology

• Dataset / Workload. The paper does not use a traditional benchmark dataset. Instead, it simulates decode-time inference with synthetic KV-cache sequence lengths of one million tokens and beyond—the workloads are models running autoregressive decoding with these ultra-long KV histories. The paper states: "Although these models do not yet natively support million-token contexts, we simulate decode-time inference with KV-cache sequence lengths of one million tokens and beyond." This means the evaluation measures system-level performance (latency, throughput) of running the model, not model accuracy on a task. There is no train/test split or accuracy metric involved.

• Base Model(s). Two large-scale LLMs representative of different architecture families are evaluated:

  • Llama-405B [12]: a dense 405B-parameter model with 128 query heads and 8 grouped KV heads (GQA attention), using SwiGLU activation in FFN layers.
  • DeepSeek-R1 [13]: a 671B-parameter Mixture-of-Experts model with Multi-Head Latent Attention (MLA). In MLA, key and value projections are absorbed into a latent space during decoding, effectively resulting in a single KV head ($K=1$) shared across all 128 query heads.

  Both models are evaluated in FP4 precision for weights, KV states, and arithmetic operations, reflecting "emerging trends in low-precision LLM inference deployments." The models are chosen to stress-test Helix across dense vs. MoE architectures and GQA vs. MLA attention mechanisms.

• Metrics. The paper defines three primary metrics:

  • Token-to-Token Latency (TTL): the time from when one token's generation completes to when the next token's generation completes. This is the binding constraint for interactive applications.
  • User Interactivity (Tokens/s/User): the reciprocal of TTL, representing "the rate at which new tokens are generated for a single user." Higher interactivity means lower per-token latency for each user.
  • Throughput per GPU (Tokens/s/GPU): total tokens generated per second per GPU, reflecting system-wide efficiency across all concurrent requests.
  • Batch scalability: "the maximum number of concurrent user requests that can be sustained under a fixed TTL budget," capturing the system's ability to maintain real-time responsiveness at scale.

  The paper constructs Pareto frontiers plotting throughput per GPU (Tokens/s/GPU) against user interactivity (Tokens/s/User), where each point on the frontier represents the configuration that maximizes throughput for a given interactivity constraint (equivalently, minimizes latency for a given throughput target). All performance numbers are normalized to the baseline "to focus on trends as opposed to specific performance claims"—absolute latency or throughput values are not reported.

• Baselines. The paper defines a "baseline search space" covering the best-known partitioning strategies:

  • Tensor Parallelism (TP) [6]: shards FFN weights and attention heads evenly across GPUs. TP width is swept, including values both below and above $K$ (the number of KV heads).
  • Pipeline Parallelism (PP): places different transformer layers on different GPUs. Included in the search space but noted in Section 2 as not helping with per-layer TTL.
  • Expert Parallelism (EP): for MoE models, "data-parallel attention coupled with expert-parallel FFNs, as adopted in production DeepSeek-R1 [13]." Each GPU handles a subset of experts.
  • Vanilla KVP (Medha-style) [7]: KV Parallelism along the sequence dimension for attention, with FFN computation tied to a fixed TP group. For DeepSeek-R1, the paper notes that Medha does not provide results on MoE models and that its tight TP coupling is "not well-suited for modern networks with MLA attention," so a direct comparison is not applicable in that case.

  The "Baseline" in all figures refers to the best-performing configuration from this search space for each point on the Pareto frontier—not a single fixed configuration. The paper also includes Medha as a separate labeled comparison in Figure 6 (Llama-405B).

• Generation Budget / Compute Accounting. The paper does not measure compute in FLOPs or sample counts. Instead, it uses a high-fidelity hardware simulator that models the GB200 NVL72 system, accounting for:

  • Compute costs: FLOP throughput of the GPU.
  • Communication costs: latency from inter-GPU NVLink transfers.
  • Memory costs: DRAM bandwidth constraints dictating KV cache and weight read times.

  All model weights, KV states, and arithmetic operations are assumed to use FP4 precision. The GPU memory bandwidth is set at $\text{MemBW} = 8000$ GB/s for the roofline analysis (Appendix A, Figure 1). The simulator is described as modeling "the latest GB200 hardware" and accounts for "both compute and communication costs, including latency from inter-GPU NVLink transfers, DRAM bandwidth constraints, and FLOP throughput." The exhaustive search covers over 100,000 configurations by systematically varying model partitioning strategies (TP, EP, PP, KVP), batch sizes, and GPU counts across different LLM architectures.

• Cross-Validation / Statistical Protocol. The paper does not use cross-validation, standard deviation, confidence intervals, or any statistical protocol. This is a deterministic simulation study—there is no randomness from sampling, training, or evaluation. Each configuration produces a single (latency, throughput) point computed analytically by the simulator. The Pareto frontiers are constructed by plotting only the optimal configurations—"the configuration that maximizes system throughput for any given TTL constraint." The claim of 100,000+ configurations implies exhaustive coverage of the discrete parameter space (partitioning strategies × batch sizes × GPU counts), so the frontiers represent the true optimal achievable points given the search bounds (1–64 GPUs within a single GB200 node).

Main Quantitative Results

The paper presents results organized by model architecture, with DeepSeek-R1 and Llama-405B evaluated separately, followed by an ablation isolating the HOP-B contribution.

DeepSeek-R1 (MoE with MLA Attention, 1-Million Context)

Headline numbers (Figure 5). For DeepSeek-R1 on GB200 NVL72 with 1-million-token KV histories, Helix:

• Improves user interactivity (Tokens/s/User) by up to 1.5× compared to the best baseline configuration at the same throughput.
• Supports up to 32× more concurrent users (i.e., achieves 32× higher Tokens/s/GPU) compared to the baseline under the same TTL budget.

Interpreting Figure 5. The Pareto frontier for Helix (green line/squares) lies substantially to the upper-right of the Baseline Pareto frontier (blue line/circles). This means Helix achieves both higher throughput per GPU and higher per-user interactivity simultaneously—the frontier is pushed outward along both axes rather than trading one for the other. The 1.5× interactivity improvement is visible as the vertical gap between the two frontiers at the lower-throughput end (where interactivity is highest). The 32× throughput improvement is visible as the horizontal extent of Helix's frontier extending far to the right of the baseline, into a high-throughput regime that the baseline cannot reach at any interactivity level. The paper explicitly notes that Medha's approach is not applicable for DeepSeek-R1 due to MLA attention ($K=1$, so any TP > 1 causes full KV duplication) and the lack of MoE support in Medha, so the "Baseline" here represents the best of TP, PP, and EP configurations.

Why the 32× batch scalability matters. This number reflects the maximum concurrent users Helix can serve under a fixed latency budget. The paper's explanation is that Helix's ability to "shard both KV caches and FFN weights across all available devices" reduces per-GPU DRAM pressure and increases compute efficiency. In the baseline, the binding constraint is likely KV cache capacity or bandwidth—you cannot fit enough KV caches on each GPU to serve many concurrent users at million-token context lengths. Helix's KVP sharding divides the per-GPU KV cache size by $KVP$, allowing proportionally more concurrent users' KV caches to fit in GPU memory. Simultaneously, the FFN reconfiguration spreads weight reads across all GPUs, preventing FFN reads from becoming the new bottleneck as batch size grows.

Llama-405B (Dense with GQA, 1-Million Context)

Headline numbers (Figure 6). For Llama-405B on GB200 NVL72 with 1-million-token KV histories:

• Helix yields a 1.13× improvement in maximum achievable interactivity compared to TP sharding.
• Helix achieves 4× higher throughput and batch capacity compared to TP sharding.
• The Medha-labeled curve (gold circles) sits between the Baseline and Helix frontiers, with Helix extending further along both axes.

Interpreting Figure 6. Three Pareto frontiers are shown: Baseline (TP-dominated, blue), Medha (gold), and Helix (green). The Helix frontier dominates both alternatives across the full range. The 1.13× interactivity improvement is the vertical gap at the low-throughput end. The 4× throughput improvement is the horizontal extension at the high-throughput end. The paper attributes these gains to two factors: "(1) lifting TP's KV-duplication ceiling via KVP, and (2) further increasing FFN parallelism without introducing cache duplication." In plain terms: Helix can use $KVP > 1$ to reduce per-GPU KV reads even when $TP > 8$ (which the baseline cannot do without duplicating KV caches), and Helix can use all $N$ GPUs for FFN sharding (which Medha cannot do because it locks FFN to a fixed TP group).

Medha comparison detail. The Medha curve sits above the Baseline at low throughput (better interactivity, because KVP reduces KV read time) but below Helix across the full range. The gap between Medha and Helix widens at higher throughput, where Helix's FFN reconfiguration advantage compounds. The paper notes that both Helix and the baseline TP implementation include communication-computation overlap, while Medha systems expose all communication overheads, "which further underscores the importance of HOP-B." However, the Figure 6 caption does not clarify whether the Medha curve shown includes any overlap optimization—the text implies it does not, making the Medha comparison slightly unfavorable.

Ablation Studies and Robustness Checks

HOP-B ON vs. OFF for DeepSeek-R1 (Figure 7, left): With HOP-B disabled (communication and computation execute strictly sequentially), DeepSeek-R1 suffers only ~1% degradation in Tokens/s/User compared to HOP-B enabled. The Pareto frontiers with HOP-B ON and OFF are nearly identical. This is because DeepSeek-R1 uses MLA attention where the all-to-all exchange accounts for "just ~1% of end-to-end decode latency, with latent projections, shared-expert computation, and multi-expert GEMMs dominating." The communication cost is so small relative to total layer time that hiding it provides negligible benefit. This is a non-obvious and practically significant negative result: it means that for MLA-based models, the engineering complexity of implementing HOP-B (fine-grained batchwise pipeline overlap) may not be justified—the raw decoupling benefit of Helix is sufficient, and the communication overhead is naturally amortized by the FFN-dominated latency profile.

HOP-B ON vs. OFF for Llama-405B (Figure 7, right): In stark contrast, Llama-405B incurs a ~12% drop in Tokens/s/User when HOP-B is turned off. The HOP-B ON frontier sits clearly above the HOP-B OFF frontier across the entire range. This differential impact reveals an important architectural dependency: GQA models with explicit materialized KV heads have heavier attention computation relative to FFN computation compared to MLA models, making the all-to-all communication a larger fraction of total TTL (~12% of latency is recoverable via overlap) and making HOP-B essential for realizing the full benefits of Helix's decoupling. The paper states: "This stark contrast highlights that communication–computation overlap becomes increasingly critical as communication forms a larger fraction of TTL."

GPU count sweep (implicit in the Pareto frontier construction). The paper does not report a separate ablation over total GPU count $N$, but the Pareto frontier construction implicitly captures this dimension: each point on the frontier corresponds to a specific $N \in [1, 64]$ (fitting within a single GB200 node). The fact that the frontiers span a wide range of throughput values indicates that optimal $N$ varies with the operating point—at low throughput (few users, low batch size), a smaller $N$ may be optimal because additional GPUs add communication overhead without proportional latency reduction; at high throughput (many users, large batches), larger $N$ enables more aggressive KV and FFN sharding to keep per-GPU DRAM traffic manageable. The exhaustive search over 100,000 configurations ensures that the best $N$ is selected for each point, but no standalone ablation isolates the effect of GPU count independently of other parameters.

Partitioning strategy sweep (implicit in baseline definition). The "Baseline" curve already represents the best of TP, PP, EP, and vanilla KVP at each point—the ablation comparing these individual strategies against each other is not separately plotted, but the fact that the Baseline Pareto frontier exists confirms that these strategies were exhaustively compared and the best performer at each operating point was selected. The paper does not provide a breakdown of which baseline strategy dominates at which region of the frontier (e.g., whether EP dominates at high batch sizes for DeepSeek-R1, or whether Medha dominates over TP for Llama-405B at low batch sizes). This is a missed opportunity for deeper insight into when each prior strategy breaks down.

Critical Assessment

Claim from the executive summary: "Helix reduces TTL by up to 1.5× at fixed batch sizes and supports up to 32× larger batches under the same latency budget for DeepSeek-R1."

The 1.5× interactivity improvement for DeepSeek-R1 is visible in Figure 5 as the vertical gap between Helix and Baseline frontiers at the low-throughput/high-interactivity end. However, the paper does not specify at what exact batch size or TTL budget this 1.5× is measured—it is a frontier-level comparison (maximum achievable interactivity, which occurs at minimum batch size). The 32× batch scalability claim is supported by the horizontal extent of Helix's frontier in Figure 5, but the paper does not provide the absolute numbers that produce the 32× multiplier. Since all numbers are normalized to baseline, we cannot verify that 32× means "can serve 32× more concurrent users at the same absolute TTL" versus some normalized ratio that depends on where on the frontier the comparison is made. The claim also depends on the baseline being the best achievable configuration—if the baseline search space missed configurations that would improve its high-throughput performance, the 32× multiplier would be an overestimate.

Claim from the executive summary: "Helix yields a 1.13× improvement in maximum achievable interactivity and 4× higher throughput for Llama-405B."

These numbers are visible in Figure 6. The 1.13× interactivity gain is modest—it means the baseline is already quite good at the low-throughput/high-interactivity end, and Helix's decoupling provides marginal additional latency reduction for single-user or small-batch scenarios. This makes sense: when batch size is very small, KV cache reads are not the dominant bottleneck (there's only one user's cache to read), and FFN weight reads dominate. Helix's ability to use more GPUs for FFN sharding helps, but the gains are incremental because the FFN weight reads were already being amortized across TP in the baseline. The 4× throughput gain at high concurrency is more substantial and represents the regime where Helix's KV sharding prevents the KV cache from being the binding constraint on batch size. However, the absolute magnitude of throughput improvement depends heavily on the assumed hardware configuration (GB200 NVL72 with large NVLink domain)—on hardware with lower inter-GPU bandwidth or fewer GPUs per node, the communication overhead of Helix's all-to-all might erode these gains.

Claim: "Helix avoids KV cache duplication that occurs when TP width exceeds the number of KV heads."

This is a structural property of the design, not an empirical claim requiring experimental validation. The roofline analysis (Figure 1, left) demonstrates the consequence—KV read time flatlines at $TP > K$—but this is model-derived from the equations in Appendix A, not a measurement of a running system. The paper does not empirically measure per-GPU KV cache size or read time in a real deployment; it simulates them. The claim is well-supported by the architecture design (capping $TP_A \leq K$ explicitly prevents duplication) and the simulator faithfully models the consequences, but no hardware measurements confirm that the duplication ceiling manifests exactly as modeled (e.g., that no unexpected caching effects or memory allocator behavior alter the effective KV read volume).

Claim: "HOP-B effectively minimizes communication overhead."

Partially supported with important conditions. The ablation (Figure 7) shows HOP-B provides ~12% improvement for Llama-405B but only ~1% for DeepSeek-R1—so the claim is true for models where attention communication is a non-trivial fraction of TTL, but it provides diminishing returns for MLA-based models where attention is lightweight. The paper does not explore whether the 12% for Llama-405B fully recovers the communication overhead (i.e., whether the residual exposed communication after HOP-B is negligible) or whether further overlap strategies (e.g., overlapping with FFN computation across layers via pipeline parallelism) could recover more. The abstract claim that HOP-B "effectively minimizes communication overhead through batchwise overlap, preserving low TTL" is accurate for GQA models but somewhat overstates the generality—for MLA models, the overhead is naturally small and overlap is unnecessary.

Genuine weaknesses and missing experiments:

• No absolute latency or throughput numbers. All results are normalized to the baseline, making it impossible to assess whether the achieved TTLs are actually "millisecond-level" or practically usable for interactive applications. A 1.5× improvement over a baseline that takes 2 seconds per token is still 1.33 seconds per token, which is unacceptable for real-time interaction. The paper provides no absolute time measurements to anchor the normalized gains.

• Simulation-only evaluation with no hardware validation. The entire evaluation is conducted in a simulator. While the simulator "accounts for both compute and communication costs, including latency from inter-GPU NVLink transfers, DRAM bandwidth constraints, and FLOP throughput," simulators inevitably abstract away real-world effects: memory allocator fragmentation, driver overhead, thermal throttling, NVLink contention patterns, and software stack inefficiencies. The paper would be significantly stronger with even a small-scale hardware validation on actual GB200 hardware, even if only at shorter context lengths.

• Single hardware platform (GB200 NVL72). All results assume the specific characteristics of NVIDIA's latest hardware with large NVLink domains. The paper does not evaluate how Helix's benefits change on hardware with fewer GPUs per node, lower NVLink bandwidth, or different memory bandwidth—all of which would directly impact the tradeoff between decoupling's communication cost and its KV/FFN sharding benefits. The claim that Helix is "co-designed with Blackwell's latest capabilities" could equally be read as "Helix requires Blackwell's capabilities to be practical."

• No sensitivity analysis on context length. All main results are at a single context length (1 million tokens). The paper does not show how the benefits of Helix scale with context length—presumably the advantage grows with $S$ (since KV read time dominates at longer contexts), but at what point does Helix become preferable to the baseline? At 100K tokens, is the benefit still 1.5× interactivity, or is it negligible? This missing sweep limits the generalizability of the claims across the range of "multi-million-token" contexts the paper targets.

• Simulated million-token contexts on models not trained for them. The paper acknowledges that Llama-405B and DeepSeek-R1 "do not yet natively support million-token contexts" and simulates this regime. This means the evaluation is measuring a hypothetical deployment scenario—the models may not produce coherent outputs at these context lengths due to positional encoding limitations or training context window constraints. The systems-level analysis is valid (the KV cache read costs are real regardless of model quality), but the practical motivation ("real-time inference with ultra-long-sequence practical") implies an application that doesn't yet exist with these models.

• No evaluation of prefill phase. The paper focuses exclusively on decode-time performance and does not address how the initial context (the first million tokens) gets loaded and distributed across GPUs during the prefill phase. Prefill involves a fundamentally different computation pattern (parallel attention over the full sequence) that would stress different bottlenecks (FLOP throughput rather than DRAM bandwidth) and might require different parallelism configurations. A complete serving system must handle both prefill and decode, and the interaction between Helix's decode-optimized layout and prefill requirements is unexplored.

• The Medha comparison is not fully fair for Llama-405B. The paper notes that both Helix and the baseline TP implementation include communication-computation overlap, while "Medha systems expose all communication overheads." If Medha could also implement overlap (the paper does not argue this is impossible, only that published Medha does not do it), the gap between Helix and Medha in Figure 6 might narrow. The comparison effectively penalizes Medha for lacking an optimization that is orthogonal to its core KV sharding approach.

• No breakdown of where the 32× improvement comes from. For DeepSeek-R1, the paper claims "up to 32× more concurrent users" but does not decompose this into how much comes from KV sharding (reducing per-GPU cache size), how much from FFN reconfiguration (reducing FFN read time), and how much from the interaction between them. This decomposition would help practitioners understand which aspect of Helix to prioritize when adapting it to their own deployments.

• The exhaustive search over 100,000 configurations is not reproducible. The paper does not provide the search code, simulator, or detailed configuration space definition (what specific TP, PP, KVP, and EP widths were swept, at what granularity, for which batch sizes). Without this, the claim of "Pareto optimal" cannot be independently verified, and the possibility that a missed configuration would improve the baseline frontier cannot be ruled out.

6. Limitations and Trade-offs

Simulation-Only Evaluation With No Hardware Validation

The assumption or constraint. Every performance number in the paper comes from an in-house simulator that "models the latest GB200 hardware" and "accounts for both compute and communication costs, including latency from inter-GPU NVLink transfers, DRAM bandwidth constraints, and FLOP throughput" (Section 3.1). The paper never runs Helix on physical hardware, even at reduced scale. The simulator is described as "high-fidelity," but no validation of the simulator against real hardware measurements is provided — there is no comparison showing that the simulator accurately predicts latency or throughput for any configuration on any actual GPU.

The consequence. Simulators inevitably abstract away real-world effects that can substantially alter the throughput-latency tradeoff at the millisecond timescales relevant to interactive decoding. Memory allocator fragmentation can increase effective DRAM latency beyond bandwidth-limited models. NVLink contention patterns can produce non-linear slowdowns when many GPUs simultaneously initiate all-to-all transfers — the simulator likely models NVLink as a fixed-bandwidth channel, but real fabric congestion depends on the specific traffic pattern and buffer sizes. Driver and runtime overhead for launching CUDA kernels, dispatching NCCL collectives, and managing GPU memory can add tens to hundreds of microseconds per layer — negligible for training but potentially significant when target TTL is measured in single-digit milliseconds. Thermal throttling under sustained load can reduce clock frequencies. The paper's claim that Helix "pushes forward the throughput-latency Pareto" (abstract) is a statement about real systems, but the supporting evidence is entirely simulated. Without even one hardware validation point — for example, measuring actual TTL for Llama-405B at 128K context on 8 GPUs with Helix vs. baseline — the gap between simulated and realizable performance is unknown.

What evidence exists in the paper. None. There is no hardware measurement anywhere in the paper. The roofline analysis (Figure 1, Appendix A) uses a simple DRAM bandwidth model at 8000 GB/s, but roofline plots only capture the bandwidth and FLOP ceilings — they do not capture latency from software stack overhead, collective synchronization jitter, or memory allocator fragmentation. The Pareto frontier construction (Figures 5–7) depends on the simulator accurately predicting the cost of every operation (FlashAttention, all-to-all, all-reduce, matrix multiply) under contention, but the paper provides no evidence that the simulator's predictions match measurements from any real system.

Mitigation status. Not addressed. The paper makes no mention of hardware validation as future work and presents the simulation results as the primary evaluation. A practitioner considering deploying Helix would need to replicate these measurements on their target hardware to confirm that the communication overhead of the all-to-all and the benefit of HOP-B manifest as predicted.

---

Million-Token Contexts Are Simulated on Models Not Trained or Validated for Them

The assumption or constraint. The paper evaluates Llama-405B and DeepSeek-R1 with "KV-cache sequence lengths of one million tokens and beyond" while explicitly acknowledging that "these models do not yet natively support million-token contexts" (Section 3.1). The million-token regime is reached by simulating KV cache reads at that scale, not by running the models on actual million-token inputs and measuring real latency. The paper treats the KV cache as an abstract data structure whose size and read time scale linearly with $S$, independent of model architecture details like positional encoding scheme or training context window.

The consequence. This creates a subtle but important disconnect between the systems-level claims and the application-level motivation. The paper's central motivation is that "LLMs are increasingly expected to maintain multi-million-token KV histories" for applications like "codebase understanding, long-document QA, and persistent AI assistants" (Section 2). But Llama-405B and DeepSeek-R1 were not trained to handle million-token contexts and would likely produce degraded or incoherent outputs at those lengths due to limitations in their positional encodings (RoPE for Llama, whatever scheme DeepSeek-R1 uses for MLA) and the absence of million-token examples during training. The paper's analysis of KV cache read costs is mathematically valid — reading an FP4 KV cache of size $S$ costs $O(S)$ DRAM bytes regardless of whether the model produces coherent outputs — but the practical scenario the paper is optimizing for (interactive million-token decoding) does not exist for these specific models.

The practical concern for a deployer is: if I want to serve a model at million-token context, I will use a model actually trained for that scale (like Gemini 1.5 or Llama 4), which may have different architectural properties (different numbers of KV heads, different attention mechanisms, different FFN dimensions) that change the quantitative tradeoff Helix exploits. The paper's roofline analysis is parameterized by $Q$, $K$, $Hsz$, and $F$, so the qualitative insights transfer, but the specific numbers (1.5× interactivity, 32× batch capacity) are tied to models that are not viable million-token decoders. A deployer cannot use these numbers to decide whether to invest in implementing Helix for their actual million-token model.

What evidence exists in the paper. The paper is transparent about this: the evaluation uses Llama-405B and DeepSeek-R1 with simulated million-token contexts (Section 3.1). There is no experiment showing Helix on a model genuinely capable of million-token reasoning, and no discussion of whether the architectural properties of such models (Gemini 1.5, Llama 4) differ in ways that would change Helix's benefit. The paper's claim that "Helix Parallelism represents a paradigm shift in decoding efficiency for ultra-long-context LLMs" (Section 7) is a stronger claim than the evidence supports, because the "ultra-long-context LLMs" in the evaluation are not actually ultra-long-context models.

Mitigation status. The paper does not address this gap. It does not suggest evaluating on actual million-token-capable models as future work, nor does it argue that the architectural parameters of such models are sufficiently similar to the evaluated models that the results transfer. The limitation is acknowledged in the text but its implications for the paper's claims are not discussed.

---

The Cost of Difficulty Estimation for Choose-From-Search is Not Quantified

Wait — this limitation does not apply. This paper has nothing to do with difficulty estimation or choose-from-search — those are concepts from the prior paper about test-time compute scaling, not from Helix Parallelism. I apologize for the error. Let me identify a limitation that is actually present in the Helix paper.

---

No Evaluation of the Prefill Phase and Its Interaction With Decode-Optimized Layouts

The assumption or constraint. Helix is presented as a solution for "interactive multi-million-token LLM decoding" (title), and the entire design — KVP sharding for attention, HOP-B overlap, staggered KV concatenation — is optimized for the autoregressive decode phase where one token is generated at a time. The paper never addresses the prefill phase: the initial processing of the multi-million-token context before the first decode token is generated. Prefill involves computing attention over the full context in parallel (rather than token-by-token), which has fundamentally different computational characteristics — it is FLOP-bound rather than DRAM-bound, and it typically uses much larger batch-like parallelism to process many context tokens simultaneously. The parallelism strategy that is optimal for prefill (higher TP for greater FLOP throughput, potentially sequence parallelism along a different axis) may differ from the decode-optimized layout Helix proposes.

The consequence. A complete LLM serving system must handle both prefill and decode. If Helix's decode-optimized GPU layout is suboptimal for prefill, the system faces a choice: (a) reconfigure the GPU layout between prefill and decode (adding latency to the time-to-first-token, which matters for interactive applications), (b) use a decode-suboptimal layout during prefill (increasing time-to-first-token), or (c) run prefill on a separate set of GPUs (disaggregated prefill/decode, which doubles hardware requirements). None of these options is discussed. For a million-token context, prefill itself could take seconds to minutes, and the parallelism strategy used during prefill strongly affects that latency. A deployer adopting Helix would need to know: does Helix's attention layout ($KVP \times TP_A$) also accelerate prefill, or does it hinder it? If it hinders prefill, what is the cost of switching layouts between prefill and decode? The paper's silence on prefill means Helix is evaluated as a component of a serving system where other components (prefill handling, scheduling, KV cache management during prefill) are unspecified.

What evidence exists in the paper. None. The word "prefill" does not appear in the paper. All descriptions of the attention phase (Section 2.1) assume the decode-time pattern: each GPU computes QKV projections from a small input batch $[B, H]$ (where $B$ is the number of concurrent decode requests, typically small) and runs FlashAttention on its local KV shard. The paper does not describe how the initial KV cache is populated, whether the $KVP \times TP_A$ layout is used during population, or how the staggered KV concatenation strategy for decode tokens (Section 2.3) interacts with the initial distribution of the precomputed context across KVP ranks.

Mitigation status. Not addressed. The paper offers no discussion of prefill as a consideration, nor any suggestion that prefill-decode interaction is a direction for future work. A practitioner deploying Helix must independently design the prefill phase and assess whether Helix's decode-oriented design creates prefill bottlenecks.

---

All Results Are Normalized and No Absolute Latency Numbers Are Reported

The assumption or constraint. All performance numbers in the evaluation (Figures 5–7) are "normalized to that of the baseline to focus on the trends as opposed to specific performance claims" (Section 3.1). The paper never reports an absolute token-to-token latency, an absolute tokens-per-second-per-GPU, or an absolute batch size. The axes on the Pareto frontier plots show "Tokens/s/User" and "Tokens/s/GPU" in normalized units where the baseline is presumably 1.0, but the absolute scale is omitted.

The consequence. Without absolute latency numbers, it is impossible to assess whether the improvements Helix provides are practically meaningful for interactive applications. The paper's motivation hinges on "millisecond-level Token-to-Token Latency (TTL) for interactive applications" (Section 1), but the evaluation provides no evidence that Helix achieves millisecond-level TTL for any configuration. Suppose the baseline achieves TTL of 500ms per token at the best Pareto-optimal point — already borderline for interactivity. Helix's 1.5× interactivity improvement would reduce this to ~333ms, which is still well above the ~50ms threshold that feels instantaneous to users. Alternatively, if the baseline already achieves 50ms TTL, Helix's 1.5× improvement to 33ms is valuable but incremental. Without absolute numbers, the reader cannot distinguish these scenarios. The 32× batch scalability claim similarly floats without anchor: supporting 32× more users than a baseline that supports 1 user is 32 users; supporting 32× more than a baseline that supports 100 users is 3200 users. The practical significance of "32×" depends entirely on the absolute baseline capacity, which is withheld.

This opacity also prevents cross-comparison with other serving systems. A practitioner cannot compare Helix's simulated performance against, say, vLLM or TensorRT-LLM on real hardware, because the absolute latency baseline for the same models on GB200 is unknown. The paper's choice to normalize makes the results self-contained but also isolates them from the broader serving systems literature. If the simulator systematically underestimates or overestimates latency by a constant factor, the normalized improvements (1.5×, 4×, 32×) would be preserved, but the conclusion about whether these improvements are sufficient for "real-time" interactivity would be contingent on the unknown absolute scale.

What evidence exists in the paper. The paper provides the DRAM bandwidth assumption (8000 GB/s, Appendix A) and the model dimensions ($Q$, $K$, $Hsz$, $F$ for both models), from which approximate absolute read times could be computed. From these, a rough back-of-the-envelope calculation is possible: for Llama-405B with $B=8$ and $S=1\text{M}$ tokens in FP4, the per-layer KV read time would be approximately $8 \times 2 \times 8 \times 128 \times 10^6 \times 0.5 / (8 \times 10^{12})$ seconds ≈ 1.02ms for the TP=8 case (assuming MemBW=8000 GB/s and no KVP sharding). Across ~126 layers for a 405B model, that's ~128ms just for KV reads. FFN weight reads add further time. So absolute TTLs may be in the hundreds of milliseconds to seconds range, but this calculation is approximate and does not account for compute latency, communication, or overlap. The paper provides no validated absolute numbers.

Mitigation status. Not addressed. The paper does not explain why absolute numbers are withheld (potentially proprietary hardware details), nor does it provide sufficient information for the reader to independently reconstruct absolute latency estimates within tight error bounds. The paper's claim of making "real-time inference with ultra-long-sequence practical" (abstract) cannot be verified without knowing whether the achieved TTL is actually practical for real-time applications.

---

No Decomposition of the 32× Batch Scalability Claim Into Component Contributions

The assumption or constraint. The paper's most striking claim is that for DeepSeek-R1, Helix supports "up to 32× more concurrent users" than the baseline under the same latency budget (Section 3.2, Figure 5). This number represents the horizontal extent of Helix's Pareto frontier beyond the baseline's frontier — the maximum throughput Helix can achieve at any interactivity level, compared to the maximum throughput the baseline can achieve. However, the paper provides no decomposition of where this 32× comes from among Helix's multiple mechanisms: KV sharding via KVP (which reduces per-GPU KV cache capacity pressure, allowing more users' caches to fit in memory), FFN weight sharding via increased TP (which reduces per-token FFN read time, making each user cheaper), or the interaction between them (where simultaneously reducing both bottlenecks prevents a new bottleneck from emerging as batch size scales).

The consequence. Without a decomposition, a practitioner considering implementing Helix cannot prioritize which components to implement first or assess which components provide the most leverage for their specific deployment. If 28× of the 32× comes from KVP alone (because KV cache capacity was the binding constraint preventing larger batches), and only 4× from FFN reconfiguration, then a simpler implementation that just adds KVP without FFN reconfiguration might capture most of the benefit with less engineering complexity. Conversely, if the FFN reconfiguration is essential because, after KV sharding relieves the attention bottleneck, FFN weight reads become the new binding constraint at larger batch sizes, then skipping FFN reconfiguration would produce much smaller gains than the full Helix system. The paper's aggregate reporting makes both interpretations possible.

This is especially relevant for DeepSeek-R1, which uses MLA attention ($K=1$). Since any $TP > 1$ causes full KV duplication in the baseline, the baseline is severely constrained in how many GPUs it can use for attention — it likely uses EP-dominant strategies with $TP=1$ for attention, meaning KV cache capacity per GPU is $S \times K \times Hsz$ with no sharding. Helix enables $KVP > 1$, directly dividing per-GPU KV cache size by the KVP width. This single change could plausibly increase the maximum concurrent user count by roughly the KVP width (e.g., $KVP=32$ gives ~32× more users from KV capacity alone). The FFN reconfiguration to $TP_F \times EP$ may provide additional gains by preventing FFN reads from bottlenecking, but the relative contribution is unclear. The 32× claim bundles these mechanisms together, making it difficult to attribute the gain to specific design choices.

What evidence exists in the paper. The paper does not provide an ablation that isolates the contribution of KVP alone, FFN reconfiguration alone, and their interaction. The comparison with Medha for Llama-405B (Figure 6) partially addresses this — Medha includes KVP but locks FFN to a fixed TP group, so the gap between Medha and Helix represents the contribution of FFN reconfiguration. For Llama-405B, the gap is visible but the throughput improvement is 4× overall, with Medha providing some fraction of that. For DeepSeek-R1, no such comparison exists because Medha does not support MoE/MLA, so the 32× cannot be decomposed at all.

Mitigation status. Not addressed. The paper provides no component-wise ablation and does not suggest this decomposition as future work. The "Discussion" section (Section 5) notes that Helix is a general technique but does not analyze which components matter most in which regimes. A practitioner must either implement the full system or conduct their own sensitivity analysis to determine which components provide the largest leverage for their specific model and hardware.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new model architecture, attention mechanism, or inference algorithm — it proposes a new way to map existing models to hardware, and in doing so, it exposes a structural constraint in the parallelism design space that prior work had accepted as inevitable. The contribution reshapes the conversation around LLM serving infrastructure in three specific ways.

First, the paper reframes parallelism design from a static mapping problem to a temporal resource allocation problem. The dominant assumption in prior serving frameworks (TensorRT-LLM, vLLM, the Medha line of work) was that each GPU plays a fixed role throughout a transformer layer's execution — GPU $i$ handles query heads $q_i$, KV heads $k_i$, and FFN weight shard $w_i$, and that assignment persists from the self-attention projection through the FFN. Helix demonstrates that this assumption is not only unnecessary, but actively harmful at scale: the optimal GPU role during attention (KV shard manager, with modest head sharding to avoid duplication) is structurally different from the optimal role during FFN (weight shard processor, with aggressive hidden-dimension sharding to amortize reads). By introducing a temporal pipeline with a single all-to-all as the switching cost, Helix proves that the design space of "a GPU plays one role per layer" is a self-imposed constraint, not a hardware requirement.

This is a genuine expansion of the design space rather than an incremental optimization within existing space. It is not merely a better TP width or a better KVP ratio — it is a new axis of freedom (temporal reconfigurability) that was previously unexplored. The conceptual parallel is to how out-of-order execution expanded the design space of CPU microarchitecture beyond static scheduling: once you accept that the mapping can change mid-execution, configurations that were previously impossible (aggressive FFN sharding without KV duplication) become reachable. The paper's evidence that these configurations are Pareto-optimal (Figures 5, 6) — that no static mapping can achieve the same throughput-latency points — validates that the expansion is practically valuable, not just theoretically interesting.

Second, the paper introduces and names the "KV duplication ceiling" as a first-class constraint on parallelism design. Prior work had certainly encountered the phenomenon — the Medha paper [7] was motivated by it, and any practitioner who tried to run GQA models with $TP > K$ experienced it — but it was treated as an implementation inconvenience rather than a fundamental design constraint. The paper gives it a name, derives its mathematical form (the saturation of $\lceil K/TP_A \rceil$ when $TP_A > K$), visualizes it in a roofline plot (Figure 1, left), and traces its systemic consequences (FFN under-sharding when TP is capped at $K$, or attention bottlenecking when TP exceeds $K$). This naming-and-framing contribution matters because it gives system designers a crisp diagnostic rule — "your KV read time is flatlining because you've hit the duplication ceiling" — and a clear design target: stay below the ceiling during attention, and seek parallelism from a different dimension (KVP, or the FFN phase's independent mapping).

The ceiling concept also explains why next-generation architectures like MLA are more challenging for conventional TP than GQA: with $K=1$, the ceiling kicks in at $TP=2$, meaning essentially any tensor parallelism during attention causes full KV duplication. For large MoE models with MLA (DeepSeek-R1), this makes conventional TP effectively unusable during attention — a point the paper uses to explain why Helix's 32× batch scalability gain for DeepSeek-R1 is so large relative to the 4× gain for Llama-405B. The baseline for DeepSeek-R1 is heavily constrained by the ceiling, while Llama-405B's baseline can at least use $TP \leq 8$ without duplication.

Third, the paper establishes that the communication cost of dynamic reconfiguration can be manageable — and that its manageability is model-architecture-dependent in a predictable way. The all-to-all that Helix introduces between attention and FFN is the price of decoupling. Prior work might have assumed this price was prohibitive, which would explain why static mappings were the default. The paper demonstrates not only that the price is acceptable on modern hardware (GB200 NVL72 with large NVLink domains), but that it can be further reduced through batchwise overlap (HOP-B), and — crucially — that the need for overlap varies dramatically by model architecture. The ablation (Figure 7) showing HOP-B provides ~12% improvement for Llama-405B but only ~1% for DeepSeek-R1 is more than an engineering detail; it is a second-order design rule: the value of communication-computation overlap depends on the fraction of total layer time spent in the communication being overlapped. For MLA models where attention is lightweight, the raw communication cost is so small that overlap is unnecessary; for GQA models where attention is heavier, overlap is essential to realizing the full benefit of decoupling.

This rule generalizes beyond Helix: any system that introduces a communication step to enable dynamic GPU reconfiguration should assess whether that communication is a large enough fraction of total latency to justify overlap engineering. The paper provides a methodology for this assessment (profile the fraction, measure the gap with and without overlap) and demonstrates it for two architectures.

What becomes more attractive as a research direction. The paper makes temporal reconfiguration of GPU roles a legitimate design primitive for inference serving systems. This opens a line of inquiry into what other phase-specific optimizations become possible when you accept a communication cost for switching: different precisions for attention vs. FFN (FP4 for KV cache reads, FP8 for FFN weights), different sparsity patterns applied to different phases, heterogeneous GPU assignments (more GPUs for attention, fewer for FFN, or vice versa depending on context length), and dynamic reconfiguration not just between attention and FFN but potentially between prefill and decode, or between different layers of the model. The paper also makes exhaustive Pareto-frontier search an attractive evaluation methodology for serving systems — its demonstration that single-point comparisons can mislead because different strategies dominate at different operating points should raise the bar for future systems evaluations.

What becomes less attractive. The paper implicitly argues that further optimizing static mappings (better TP ratios, better EP balancing) will hit diminishing returns because the KV duplication ceiling is a hard structural limit, not a soft inefficiency. Research that tries to squeeze more out of conventional TP for long-context GQA/MLA models by tuning within the static-mapping paradigm is unlikely to break through the ceiling — the ceiling is set by model architecture ($K$), not implementation quality. Similarly, the paper's demonstration that Medha-style KVP (which locks FFN to a fixed TP group) leaves FFN underutilized suggests that solutions that only address KV cache pressure without addressing FFN weight pressure will produce unbalanced systems where the unaddressed bottleneck becomes the new binding constraint at larger batch sizes. This shifts the goal from "solve the KV cache problem" to "solve KV and FFN simultaneously without reintroducing KV duplication" — a harder, coupled optimization that Helix addresses through decoupling.

Follow-Up Research This Work Enables

Hardware validation on real GB200 NVL72 hardware with actual models, starting at shorter context lengths and scaling up. The paper's entire evaluation is simulation-based. The most urgent follow-up is to measure Helix on physical hardware to validate the simulator's predictions. A strong validation study would: (a) implement Helix in a serving framework (e.g., TensorRT-LLM or a custom CUDA/C++ runtime), (b) measure actual token-to-token latency for Llama-405B and DeepSeek-R1 at context lengths of 128K, 256K, 512K, and 1M tokens (to the extent KV cache capacity allows) under both baseline TP and Helix configurations, (c) compare measured vs. simulated latency at each point, identifying which real-world effects (NVLink contention, driver overhead, memory allocator latency) the simulator misses, and (d) determine whether the Pareto-frontier gains (1.5× interactivity, 4–32× throughput) hold on silicon. Even if hardware constraints prevent running at full 1M-token contexts, validation at 128K or 256K would establish whether the simulator's fidelity is sufficient to extrapolate to longer contexts. A negative result — e.g., NVLink contention during the all-to-all is worse than modeled, reducing Helix's benefit to <1.1× — would be as informative as a positive one because it would identify the specific hardware bottleneck that needs to be addressed for Helix to be practical.

Characterize the prefill phase and propose a unified prefill-decode Helix variant. Helix is designed for decode and is silent on prefill, but any production serving system must handle both. Prefill involves processing the full multi-million-token context in parallel (FLOP-bound, large matmuls) before decoding begins, and the optimal GPU layout for prefill (likely TP-heavy for FLOP throughput, with sequence parallelism along a different axis) may conflict with the decode-optimized $KVP \times TP_A$ layout. A follow-up would: (a) model the prefill latency for Helix's decode layout, identifying whether the same $KVP \times TP_A$ configuration accelerates or hinders prefill, (b) measure the cost of reconfiguring GPU topology between prefill and decode (is the cost comparable to the all-to-all, or does it require weight reloading?), and (c) evaluate a disaggregated Helix variant where one set of GPUs handles prefill using a FLOP-optimized layout while another set handles decode using the Helix layout, with KV cache transfer between them. The key question is whether Helix's decode benefits survive when prefill costs are included in the total time-to-first-token budget.

Apply Helix's decoupling principle to other phase-specific optimizations beyond parallelism — precision, sparsity, and heterogeneous hardware. Helix decouples attention and FFN parallelism mappings. The same temporal-decoupling logic could apply to other resource allocation decisions that currently use static, layer-uniform policies. Specific experiments: (a) Mixed-precision Helix: use FP4 for KV cache reads during attention (tolerating lower precision for long histories where retrieval is approximate anyway) but FP8 for FFN weights (where precision matters more for activation quality), with the reconfiguration including a precision conversion step. Does this yield further latency reduction beyond what precision uniformity achieves? (b) Sparse-attention Helix: apply different sparsity masks to the KV cache during attention (e.g., only attend to every 4th token on some GPUs) to reduce DRAM reads further, while keeping FFN dense — the reconfiguration ensures sparsity in one phase does not force sparsity in the other. (c) Heterogeneous Helix: assign more GPUs to attention than FFN (or vice versa) depending on context length — at short contexts where KV reads are cheap, shift GPUs toward FFN sharding; at long contexts where KV reads dominate, shift GPUs toward KVP. This would require a reconfiguration that changes the number of active GPUs per phase, which may have higher switching cost but could better match hardware to workload dynamics. Each of these experiments would test whether the decoupling principle generalizes beyond parallelism to other dimensions of the serving optimization space.

Analyze Helix's sensitivity to NVLink bandwidth and GPU count to determine the hardware threshold for practicality. The paper's evaluation assumes GB200 NVL72 with its large NVLink domain and high bandwidth. A critical practical question for deployers is: how much NVLink bandwidth is "enough" for Helix to outperform conventional TP? A follow-up study would sweep NVLink bandwidth (in simulation or on hardware with throttled links) and GPU count ($N \in \{8, 16, 32, 64, 72\}$) to map the region of the (bandwidth, $N$) space where Helix is Pareto-optimal. The prediction is that Helix's benefit shrinks as NVLink bandwidth decreases (because the all-to-all cost grows relative to DRAM read savings) and as $N$ decreases (because the maximum achievable FFN sharding width is capped, reducing the FFN-phase benefit). Identifying the minimum hardware configuration for Helix to provide, say, >1.2× interactivity improvement would give practitioners a concrete deployment criterion. This analysis would also reveal whether Helix is a Blackwell-specific strategy or whether it provides meaningful benefits on current-generation hardware (H100/H200 nodes with NVSwitch).

Combine Helix with sequence-length-aware batching and continuous batching to optimize end-to-end serving throughput. Helix is evaluated with fixed batch sizes and static KV cache lengths, but real serving systems use continuous batching (mixing requests at different stages of generation) and face variable sequence lengths (prompt tokens + generated tokens). A natural extension is to integrate Helix into a production serving runtime (e.g., vLLM, SGLang) and evaluate end-to-end performance on realistic request traces with mixed context lengths. Critical questions: (a) Does the all-to-all overhead become unpredictable under continuous batching where different requests in the batch have different sequence lengths and different attention computation times, breaking HOP-B's batchwise overlap assumptions? (b) Does the staggered KV concatenation strategy (16-token rounds across KVP ranks) interact poorly with requests that finish generation at different times, creating fragmentation or load imbalance in KV cache storage? (c) At what maximum context length does Helix's benefit justify its implementation complexity in a production system — is it only relevant at >512K tokens, or does it help at 128K? A negative result showing that continuous batching dynamics erode HOP-B's overlap benefit would be valuable because it would indicate that Helix is best suited for offline batch inference or dedicated high-priority streams rather than general multi-tenant serving.

Practical Applications and Downstream Use Cases

Deploying persistent AI assistants and copilots with full conversation histories. An AI coding assistant that maintains the entire multi-session conversation history, codebase context, and tool outputs in its KV cache needs to attend to potentially millions of tokens at each decoding step while still providing sub-second per-token responsiveness. Helix directly targets this scenario: for a model like DeepSeek-R1 serving 32 concurrent users with 1M-token contexts under a fixed TTL budget, Helix supports 32× more concurrent users than the baseline (Figure 5). In absolute terms (normalized but directionally meaningful), this could mean the difference between a single GPU node serving 1 developer versus serving a small team of 32 developers with the same per-user latency — making persistent ultra-long-context assistants economically viable rather than requiring dedicated hardware per user. The key enabling mechanism is KV sharding across all available GPUs, which prevents each user's million-token cache from consuming a full GPU's worth of memory bandwidth.

Cost-efficient batch inference for long-document processing and overnight evaluation runs. Organizations that process large document collections (legal discovery, scientific literature review, codebase analysis) often run batch inference jobs where latency per token is secondary to total throughput and cost. At 1M-token context lengths, Helix's 4× throughput improvement for Llama-405B (Figure 6) translates directly to processing 4× more documents per GPU-hour, reducing cloud compute costs proportionally. The scenario: a legal tech company needs to extract structured information from 10,000 contracts averaging 500K tokens each. With baseline TP, a GB200 node might process 250 contracts per hour; with Helix, the same node processes 1,000 contracts per hour, completing the job in 10 hours instead of 40. The batch size scalability enabled by KV sharding means the system can process larger batches without KV reads becoming the bottleneck, amortizing FFN weight reads more effectively.

On-premise deployment of large MoE models for latency-sensitive enterprise applications. Enterprises that deploy LLMs on-premise (for data privacy, regulatory compliance, or cost predictability) often face hardware constraints — a fixed number of GPUs in a single node — and need to maximize both throughput and interactivity within that fixed budget. For DeepSeek-R1 (a 671B MoE model with MLA), conventional parallelism strategies face a severe KV duplication ceiling (because $K=1$, any TP > 1 duplicates the full KV cache), forcing the system to either use EP-only parallelism (under-sharding FFN weights) or accept massive KV duplication (wasting memory bandwidth). Helix eliminates this tension: during attention, all GPUs participate in KVP to shard the million-token context without duplication; during FFN, the same GPUs reconfigure into a $TP \times EP$ grid to efficiently process the MoE layers. The result (Figure 5) is both lower latency per user (1.5× interactivity) and higher total throughput (32× batch capacity). For an enterprise deployment with, say, 8 GPUs in a DGX box, this could make the difference between the system being usable for a single interactive application (baseline) versus supporting an entire team of analysts querying long documents simultaneously (Helix).

Enabling long-context capabilities on edge-server hybrid deployments. Consider a scenario where a smaller on-device model handles short-context interactions, but complex queries requiring long-context reasoning are escalated to a more powerful server-side model (e.g., DeepSeek-R1 or Llama-405B). The server needs to process these escalated queries with full conversation history (which may be long) and return tokens quickly enough that the user does not perceive the escalation latency. Helix's TTL reduction (1.5× for DeepSeek-R1, 1.13× for Llama-405B, at fixed batch sizes) directly improves the feasibility of this architecture by reducing the server-side component of end-to-end latency. Even a 13% latency reduction (Llama-405B) could bring a borderline-unacceptable 2.3-second TTL down to 2.0 seconds — still not ideal, but closer to the threshold where users tolerate the delay. The practical deployment architecture would combine Helix on the server side with a lightweight difficulty-estimation router on the client side that decides whether to process locally or escalate, creating a two-tier system where Helix's efficient long-context serving makes the escalation path economically viable. </output>