Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures

ArXiv: 2505.09343

🎯 Pitch

DeepSeek-V3 trains a 671B-parameter mixture-of-experts model on only 2,048 commoditized H800 GPUs by compressing attention memory 4.7–7.3× and pruning cross-node communication with node-limited routing—yet its inference throughput can still be throttled to ~67 tokens per second by 400 Gbps InfiniBand, exposing a looming interconnect gap for expert-parallel serving.

---

1. Executive Summary

This paper presents a hardware-centric retrospective on the development and deployment of DeepSeek-V3/R1, analyzing how model architecture and AI infrastructure were co-designed to overcome critical bottlenecks in memory, computation, and interconnection during training on 2,048 NVIDIA H800 GPUs. It examines the interplay between hardware constraints and key architectural innovations—Multi-head Latent Attention (MLA; compressing KV caches into a latent vector to reduce inference memory), DeepSeekMoE with Node-Limited Routing (restricting token dispatch to at most 4 nodes to mitigate the scale-up/scale-out bandwidth gap), FP8 mixed-precision training (tile- and block-wise fine-grained quantization enabling lower computational cost), and a Multi-Plane Network Topology (eight independent two-layer fat-tree planes replacing a costlier three-layer fabric)—along with a Multi-Token Prediction module that increases generation throughput by 1.8× through self-drafting speculative decoding. The paper reports a theoretical upper bound of ~67 tokens per second under 400 Gbps InfiniBand for expert-parallel inference, reduced KV cache memory to 70 KB per token (4.7–7.3× smaller than comparable dense models), and a total training cost of 250 GFLOPS per token (6–10× less than equivalently capable dense architectures of 72B–405B parameters), establishing that cost-efficient large-scale training with commodity-constrained hardware is achievable only when model architecture, parallelism strategy, and network topology are jointly optimized.

2. Context and Motivation

The Core Problem: Hardware Is Not Keeping Pace with AI Scaling Ambitions

The paper confronts a systemic and escalating mismatch: LLM development demands ever-larger models trained on ever-larger datasets, but the hardware that makes this possible—GPUs, interconnects, memory systems—improves along fundamentally different trajectories. The authors frame this tension explicitly through the lens of cost-efficient scaling: the goal is not merely to build state-of-the-art models, but to do so with hardware that is accessible, affordable, and, in this case, subject to regulatory constraints (the H800 GPU is a reduced-bandwidth variant of the H100, with NVLink cut from 900 GB/s to 400 GB/s and FP64 compute similarly throttled for compliance reasons).

The problem is not theoretical. The Scaling Laws literature (Kaplan et al., 2020) established that performance continues to improve with model size, data, and compute—but the marginal cost of that improvement becomes prohibitive when hardware is treated as a fixed, unoptimized substrate. Industry clusters from Meta, Google, Alibaba, ByteDance, and xAI deploy tens or even hundreds of thousands of GPUs, creating an economic barrier that, in the authors' framing, "presents significant barriers for smaller research teams and organizations" (Section 1.1). The paper's central thesis is that this barrier is not inevitable: hardware-aware model co-design can recover orders-of-magnitude efficiency gains that level the playing field.

This matters for three concrete reasons the paper surfaces across its sections:

1. Memory capacity growth lags model size growth. HBM capacity improves at less than 50% per year, while LLM memory demands grow at over 1000% per year (Section 2.1, citing Gholami et al., 2024). This means that even with multi-node parallelism, raw memory usage must be optimized at the architectural level—hence MLA's KV cache compression and FP8 weight storage.

2. Computational cost of dense models is unsustainable for smaller teams. Table 2 quantifies this: a 405B dense model requires 2,448 GFLOPS per token during training compared to 250 GFLOPS for DeepSeek-V3's 671B MoE architecture—a nearly 10× difference despite DeepSeek-V3 having more total parameters. The MoE design selectively activates only 37B of those 671B parameters per token, making the computational cost proportional to the activated parameters rather than the total.

3. Inference speed directly determines the viability of reasoning models. Test-time scaling approaches (OpenAI o1/o3, DeepSeek-R1, Claude 3.7 Sonnet) rely on generating long reasoning chains at inference time. If token output speed is too slow, these models become unusable for interactive applications and impractical for RL training pipelines (PPO, DPO, GRPO) that require generating large numbers of samples rapidly (Section 2.3.4). The paper's analysis of the theoretical token-per-second upper bound (Section 2.3.2) is motivated by this: inference bandwidth is the gating factor for the next generation of reasoning-capable LLMs.

Where Prior Approaches Fall Short

The paper identifies limitations in how the field has traditionally approached the hardware-model interface, organized around the three pillars of memory, computation, and communication.

Memory: GQA is insufficient for extreme scale. The dominant approach to reducing KV cache memory is Grouped-Query Attention (GQA; Ainslie et al., 2023) or Multi-Query Attention (MQA; Shazeer, 2019), where multiple attention heads share a single set of Key-Value pairs. These are effective—Qwen-2.5 72B and LLaMA-3.1 405B both use GQA—but their compression is coarse. As Table 1 shows, GQA-based models still require 327–516 KB of KV cache per token in BF16 precision. MLA pushes this to 70 KB by compressing all attention heads into a single latent vector through a jointly trained projection matrix, effectively learning a more aggressive compression that preserves attention quality (Section 2.1.2). The distinction matters because KV cache memory is not merely a capacity issue—it is a bandwidth bottleneck: during autoregressive decoding, attention computation shifts from GEMM (compute-bound, efficient on GPUs) to GEMV (memory-bound, limited by HBM bandwidth). Reducing KV cache size directly accelerates inference by reducing the volume of memory fetches per token.

Computation: FP8 training was not deployed in open-source models. NVIDIA's Transformer Engine supported FP8 mixed-precision training, but prior to DeepSeek-V3, according to the authors, no open-source large model had leveraged FP8 for training (Section 3.1). The barrier was not conceptual—the benefits of reduced precision are well understood—but practical: FP8 introduces training instability due to limited accumulation precision in Hopper Tensor Cores (only 13 mantissa bits after alignment, accumulated into FP22 registers; Section 3.1.1) and the overhead of fine-grained dequantization when partial results must be transferred from Tensor Cores to CUDA Cores for scaling factor multiplication. The paper's contribution here is not the idea of FP8 but the engineering framework that made it viable for a 671B MoE model, including tile-wise 1×128 activation quantization, block-wise 128×128 weight quantization, and high-precision accumulation strategies—all documented in the separate DeepSeek-V3 technical report and open-sourced in DeepGEMM.

Communication: Scale-up and scale-out are treated as independent domains. The paper identifies a fundamental architectural mismatch in how current hardware handles intra-node (NVLink) and inter-node (InfiniBand) communication. The two domains have vastly different bandwidth characteristics (NVLink at ~160 GB/s effective vs. IB at ~40 GB/s effective per 400 Gbps NIC, a ~4:1 ratio on the H800; Section 4.3), yet model parallelism strategies typically treat them uniformly. The Node-Limited Routing strategy is a direct response to this: by algorithmically constraining each token to route to at most 4 target nodes (rather than all 8 nodes in the cluster), the communication cost scales with the number of nodes contacted rather than the number of experts, exploiting intra-node NVLink forwarding to deduplicate IB traffic (Section 4.3).

Prior work from NVIDIA and NCCL provided some tools—specifically PXN (PCIe via NVLink) for optimizing multi-rail topologies—but these were designed for homogeneous network fabrics, not the heterogeneous NVLink + IB architecture that the H800's regulatory constraints create. The Multi-Plane Fat-Tree topology (Section 5.1) is likewise a response to a specific hardware limitation: 400G NDR InfiniBand switches support only 64 ports, which would force a traditional fat-tree to three layers beyond ~4,096 endpoints. By deploying eight independent two-layer planes (each GPU-NIC pair assigned to a distinct plane), the MPFT topology retains the latency advantages of a two-layer design while scaling to 16,384 GPUs—and at roughly 60% of the cost per endpoint of a three-layer fat-tree (Table 3).

Inference: Speculative decoding has untapped throughput potential. Speculative decoding approaches like Medusa (Cai et al., 2024) and Eagle (Li et al., 2024) demonstrated that generating multiple candidate tokens and verifying them in parallel can accelerate inference. But these methods typically require separate draft models or complex multi-head architectures. The paper's Multi-Token Prediction (MTP) module takes a lighter-weight approach: a single additional transformer layer per predicted future token, trained end-to-end, that achieves 80–90% acceptance rates for the second subsequent token (Section 2.3.3). The 1.8× generation throughput improvement is significant not just for latency but for batch size composition: by predicting multiple tokens per step, MTP increases the inference batch size, which in turn boosts the computational intensity of the expert parallelism all-to-all communication—a virtuous cycle of hardware utilization.

How This Paper Positions Itself

The paper explicitly distinguishes itself from the DeepSeek-V3 technical report (which documents the model's algorithmic details) by adopting a hardware-first perspective. The abstract frames this as "a dual perspective—spanning hardware architecture and model design—to explore the intricate interplay between them in achieving cost-efficient large-scale training and inference" (Section 1.2). This is not a model paper; it is a hardware critique that uses DeepSeek-V3 as a case study to identify where current hardware falls short and what future hardware should provide.

This positioning is unusual for an ISCA paper (the venue is a computer architecture conference, not a machine learning venue) and reflects a deliberate choice: the authors are addressing hardware architects, not ML practitioners. The technical depth on network topologies (Sections 4–5), quantization precision (Section 3), and communication primitives (Section 6.4) reflects this audience. The paper's contributions are not the model innovations themselves (MLA, DeepSeekMoE, MTP—all introduced in prior DeepSeek publications) but rather the analysis of why those innovations were necessary given specific hardware constraints and the prescriptive recommendations for future hardware design derived from those constraints.

The paper also positions itself within a broader tradition of hardware-software co-design for AI, citing Fire-Flyer AI-HPC (An et al., 2024) as a precursor—a cost-effective deep learning cluster that DeepSeek built previously. The lineage matters because it establishes credibility: this is not a one-off experiment but a sustained research program in cost-efficient AI infrastructure. The H800 cluster described in this paper, with its 2,048 GPUs and eight-plane two-layer fat-tree network, is presented as the latest iteration in that program, constrained further by regulatory bandwidth limits that forced additional co-design creativity.

The Gap This Paper Fills

The specific gap is not the absence of individual techniques (FP8, MoE, KV cache compression, fat-tree networks, speculative decoding—all existed prior) but the absence of a systematic articulation of how these techniques must be jointly designed around hardware constraints to achieve cost efficiency at the 2,048-GPU scale under regulatory bandwidth limitations. The paper's contribution is the integration narrative: showing that MLA was necessary because HBM bandwidth limits GEMV-bound attention; that Node-Limited Routing was necessary because the 4:1 NVLink-to-IB bandwidth gap forces communication-aware expert selection; that FP8 training was necessary because the cost of BF16 training at 671B parameters would be prohibitive; and that the multi-plane network was necessary because regulatory limits on switch port counts make three-layer fat-trees economically unjustifiable.

Each individual insight is moot in isolation. A model with MLA but dense architecture would still be memory-bound during training. An MoE model without Node-Limited Routing would saturate IB links on dispatch and combine. An FP8 training framework without fine-grained quantization would produce unacceptable accuracy degradation. An eight-plane network without DualPipe would leave GPU SMs idle during communication phases. The paper's core argument—echoed in its conclusion—is that no single optimization suffices; only joint co-design across model architecture, parallelism strategy, low-precision computation, and network topology can make 671B-parameter MoE training economically viable on commercially constrained hardware.

This argument is consequential because it reframes AI scaling from a resource-acquisition problem (spend more on bigger clusters) to a systems-engineering problem (optimize the hardware-model interface). The authors are not arguing that DeepSeek-V3 achieved the best possible performance—they are arguing that it achieved a threshold of performance (state-of-the-art on multiple benchmarks) at a fraction of the hardware cost that the naive approach would require. The paper's forward-looking recommendations—unified scale-up/scale-out convergence, native fine-grained quantization in Tensor Cores, in-network computation for expert parallelism, memory-semantic communication with hardware ordering guarantees—are all derived from the specific friction points encountered during DeepSeek-V3's development, making them concrete and falsifiable rather than aspirational.

3. Technical Approach

This is primarily a hardware critique and co-design analysis paper — not a paper that proposes a single new algorithm or model architecture, but one that retrospectively analyzes how DeepSeek-V3/R1's model architecture was deliberately shaped around the constraints of 2,048 NVIDIA H800 GPUs to achieve cost-efficient training and inference at scale. The core idea is that each architectural decision — MLA, DeepSeekMoE with Node-Limited Routing, FP8 mixed-precision training, the Multi-Plane Fat-Tree topology, and the Multi-Token Prediction module — was not an independent research contribution but a direct engineering response to a specific hardware bottleneck (memory bandwidth, interconnect bandwidth, computational cost, or network topology cost) that would have made training a 671B-parameter model on commodity-constrained hardware infeasible without co-design.

3.1 Reader Orientation

What is being built: A complete training and inference system for a 671B-parameter Mixture-of-Experts language model (DeepSeek-V3) on a cluster of 2,048 NVIDIA H800 GPUs — GPUs whose inter-GPU and inter-node bandwidth are deliberately constrained below the H100 baseline for regulatory compliance — while maintaining state-of-the-art model quality and cost efficiency.

What problem it solves and the shape of the solution: The system solves a multi-dimensional resource allocation problem: given limited memory capacity (HBM growth lags model growth by 20×), limited computational resources (H800s are expensive, and the compute budget must be kept small enough that a "smaller research team" can afford it), and limited communication bandwidth (NVLink is cut from 900 GB/s to 400 GB/s, creating a 4:1 intra-node-to-inter-node gap), how do you organize model architecture, parallelism strategy, low-precision computation, and network topology so that each constraint is addressed at its root cause rather than brute-forced with more hardware? The "shape" of the solution is a set of five co-designed techniques — MLA, MoE with Node-Limited Routing, FP8 mixed-precision training, the Multi-Plane Fat-Tree, and MTP — that each target one specific constraint, plus a training orchestration strategy (DualPipe) and an inference disaggregation architecture that maximize hardware utilization under those constraints.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major co-designed components, each mapped to a specific hardware constraint:

1. Multi-head Latent Attention (MLA) — compresses KV cache representations across all attention heads into a single low-dimensional latent vector using learned projection matrices, stored during inference. This directly addresses the memory wall: KV cache memory per token drops from 327–516 KB (GQA-based dense models) to 70 KB (Table 1), reducing the HBM bandwidth pressure during the memory-bound GEMV operations of autoregressive decoding.

2. DeepSeekMoE with Node-Limited Routing — a Mixture-of-Experts architecture with 256 routed experts and 1 shared expert, where each token activates only 8 routed experts plus the shared expert (37B activated parameters out of 671B total). Node-Limited Routing algorithmically constrains those 8 expert selections to span at most 4 physical nodes, exploiting intra-node NVLink forwarding to deduplicate inter-node IB traffic.

3. FP8 Mixed-Precision Training Framework — replaces BF16 with FP8 for forward and backward GEMM operations using fine-grained tile-wise (1×128) activation quantization and block-wise (128×128) weight quantization, with high-precision accumulation to preserve training stability. This halves activation memory and provides ~2× theoretical throughput on FP8-capable Tensor Cores.

4. Multi-Plane Two-Layer Fat-Tree Network — eight independent network planes, each a complete two-layer fat-tree with 64-port 400G IB switches. Each GPU-NIC pair belongs to a single plane. This topology supports up to 16,384 GPUs while retaining the cost and latency advantages of a two-layer design, avoiding the ~60% cost premium of a three-layer fat-tree.

5. Multi-Token Prediction (MTP) Module — one lightweight transformer layer per additional predicted token (beyond the primary next-token prediction), trained end-to-end. At inference, MTP generates candidate future tokens that are verified in parallel via self-drafting speculative decoding, achieving an 80–90% acceptance rate for the second subsequent token and a 1.8× increase in generation throughput.

Information flows as follows during training: tokens enter the embedding layer → pass through L transformer blocks, each containing an MLA attention sub-block and a DeepSeekMoE FFN sub-block → during the MoE FFN, a router selects top-K experts per token (with Node-Limited Routing constraining the selection to ≤4 nodes) → an all-to-all dispatch communication sends token representations to the selected experts' GPUs → experts compute their FFN outputs → an all-to-all combine communication returns the outputs to the originating GPUs → the MTP modules predict additional future tokens using lightweight layers. FP8 quantization is applied at specific GEMM operations within both the attention and FFN sub-blocks (see Figure 1 for the precision map). The Multi-Plane network carries all all-to-all EP traffic, with each plane handling one-eighth of the total communication load independently.

3.3 Roadmap for the Deep Dive

• First, the FP8 mixed-precision training framework (Section 3) — since it is the foundation that makes the entire training pipeline economically viable, and its design choices (fine-grained quantization, accumulation precision) constrain later decisions.
• Second, the Node-Limited Routing strategy and parallelism orchestration (Section 4) — because these are the direct responses to the H800's specific bandwidth constraints; understanding them requires knowing the FP8 data flow they operate on.
• Third, the Multi-Plane Fat-Tree network topology (Section 5) — since it is the physical substrate that carries all MoE communication and its design is co-optimized with the routing strategy in Section 4.
• Fourth, the Multi-Head Latent Attention mechanism (Section 2.1.2) — the primary memory-efficiency innovation, included here because its impact manifests most clearly during inference and its compression ratio must be understood alongside the MoE architecture's parameter count.
• Fifth, the Multi-Token Prediction module (Section 2.3.3) — the inference-speed innovation that closes the loop, showing how algorithmic design can increase effective hardware utilization.
• Sixth, the training orchestration strategies (DualPipe, micro-batch overlap, prefill-decode disaggregation) that stitch these components together into a complete system (Sections 2.3.1, 4.2, and 5).

3.4 Detailed, Sentence-Based Technical Breakdown

FP8 Mixed-Precision Training: Fine-Grained Quantization Framework

The fundamental engineering challenge of FP8 training is not mathematical — the benefits of reduced precision are straightforward (half the memory footprint of BF16 activations and weights, and theoretically 2× throughput on hardware with native FP8 support) — but numerical: FP8's 8-bit representation (using NVIDIA's supported formats E4M3 with 4 exponent bits and 3 mantissa bits, or E5M2 with 5 exponent bits and 2 mantissa bits) has inherently limited dynamic range and precision compared to BF16's 8 exponent bits and 7 mantissa bits. For a 671B-parameter model, training instability from inadequate accumulation precision or coarse quantization granularity can compound across layers and training steps, producing unacceptable accuracy degradation.

The paper's FP8 framework addresses this through three design decisions, each documented in the DeepSeek-V3 technical report and partially open-sourced in DeepGEMM.

Quantization granularity. The framework applies two distinct quantization schemes simultaneously:

• Tile-wise 1×128 quantization for activations: each activation tile of shape (1, 128) — that is, one token's hidden state across 128 feature dimensions — receives its own scaling factor. This per-token granularity is necessary because different tokens can have dramatically different activation magnitudes (e.g., tokens corresponding to punctuation vs. rare content words), and a single global scaling factor would either underflow the small values or overflow the large ones.
• Block-wise 128×128 quantization for model weights: each weight matrix block of shape (128, 128) receives its own scaling factor. This per-block granularity captures the structured variation in weight magnitudes across different parts of the weight matrix (e.g., rows corresponding to different output features may have systematically different norms).

The choice of these specific tile and block shapes is hardware-aligned: the H800's Tensor Cores process matrix multiplications in 128×128 tiles internally, so aligning the quantization blocks with the hardware's natural computation granularity minimizes the number of dequantization operations needed at Tensor Core boundaries.

High-precision accumulation. The paper identifies a critical hardware limitation of Hopper-generation GPUs: when FP8 partial products are accumulated inside Tensor Cores, the accumulation register has only 13 mantissa bits (with 8 exponent bits and 1 sign bit, yielding what the authors call an FP22 format following the naming in SageAttention2, Zhang et al., 2025). The mechanism is as follows (Section 3.1.1, block quote):

• The 32 mantissa products from the FP8 inputs are aligned by right-shifting based on the maximum exponent present in the group.
• After alignment, the Tensor Core retains only the highest 13 fraction bits for addition.
• Bits beyond this 13-bit window are truncated — they are lost to the accumulation.
• The addition results are accumulated into FP22 registers: 1 sign bit, 8 exponent bits, and 13 mantissa bits.

This 13-bit mantissa is narrower than the FP32 mantissa (23 bits) that would be available in a higher-precision accumulation path. For large models with deep layer stacks, repeated accumulation with truncated mantissas can cause a gradual loss of numerical fidelity — not catastrophic divergence, but a measurable accuracy degradation that the authors quantify as below 0.25% relative to BF16 in their controlled ablation studies on 16B and 230B DeepSeek-V2 models (Section 2.4).

The mitigation strategy is not to avoid FP8 accumulation entirely (which would defeat the purpose of using FP8 Tensor Cores) but rather to carefully manage when accumulation precision matters most. The paper does not detail the exact placement of high-precision accumulation checkpoints, but the pattern is standard: critical reduction operations (e.g., attention softmax normalization, loss computation) are performed in FP32, while the bulk of GEMM operations in the MLP and attention projections use FP8 accumulation with the FP22 registers, accepting the small precision loss in exchange for the 2× throughput.

Dequantization overhead management. Fine-grained quantization introduces a practical throughput challenge: after the Tensor Core produces a partial sum in FP22, that partial sum must be multiplied by the appropriate scaling factors (one for the activation tile, one for the weight block) to recover the properly scaled FP32 or BF16 value. The naive approach — transferring the partial sums from Tensor Cores to CUDA Cores, performing the scaling factor multiplication on CUDA Cores, and writing the result back — incurs significant data movement overhead because the partial sums are produced at very high throughput inside the Tensor Cores but the CUDA Core pathway is comparatively slow.

The authors note this explicitly (Section 3.1.1): "Fine-grained quantization such as tile-wise and block-wise quantization introduces large dequantization overhead in transporting the partial results from Tensor Cores to CUDA Cores for scaling factor multiplication." Their open-source DeepGEMM library (cited in Section 3.1) provides an optimized implementation of this pipeline, likely using techniques such as fusing the dequantization with the next operation in the compute graph to avoid round-trips to global memory, but the paper does not detail the specific optimization tricks — it instead uses this overhead as motivation for its hardware recommendation: "Hardware should natively support fine-grained quantization, enabling Tensor Cores to receive scaling factors and implement matrix multiplication with group scaling" (Section 3.1.2), pointing to NVIDIA Blackwell's microscaling format (MXF) support as an example of the right direction.

Validation methodology. Each FP8 training technique was validated through a hierarchical pipeline (Section 2.4): first on small-scale models (16B parameters), then on medium-scale models (230B DeepSeek-V2), and only after confirming that accuracy degradation remained below 0.25% compared to BF16 baselines was the technique integrated into the full 671B DeepSeek-V3 training run. This tiered validation is pragmatic cost management: running a full-scale ablation on 2,048 GPUs for each quantization hyperparameter choice would be prohibitively expensive.

Node-Limited Routing: Communication-Aware Expert Selection

The H800 GPU, as a regulatory-constrained variant of the H100, imposes a specific bandwidth asymmetry that the routing strategy must accommodate. Within a single 8-GPU node, the GPUs are connected via NVLink providing a theoretical 400 GB/s bidirectional bandwidth (of which approximately 160 GB/s is achievable per direction in practice, per Section 4.3). Between nodes, each GPU communicates via a single 400 Gbps InfiniBand NIC, which provides 50 GB/s of raw bandwidth (the authors use 40 GB/s as the effective bandwidth accounting for small-message overhead and latency). This creates a 4:1 intra-node-to-inter-node bandwidth ratio (160 GB/s vs. 40 GB/s).

The unconstrained problem. Consider a standard MoE setup with 8 nodes (64 GPUs total) and 256 experts distributed across these nodes (4 experts per GPU). In DeepSeek-V3, each token is routed to 8 routed experts (plus 1 shared expert — the shared expert is a special case we will address separately). If the 8 selected experts are distributed uniformly across all 8 nodes according to their natural affinity scores (the router's logits), each node would receive approximately 1 expert's worth of tokens on average. For the all-to-all dispatch communication, the GPU processing a given batch of tokens must send those tokens to the GPUs hosting the selected experts. If the experts are on 8 different nodes, the GPU must send separate messages to 8 different destination nodes, each going over the IB fabric.

The communication cost model the paper uses (Section 2.3.2) quantifies the per-token per-layer communication time for the EP dispatch and combine phases:

Comm. Time = (1Byte + 2Bytes) × 32 × 9 × 7K/50GB/s = 120.96𝜇𝑠

This formula captures a specific scenario: dispatch communication uses FP8 (1 byte per element), combine communication uses BF16 (2 bytes per element), each token has a hidden size of approximately 7K elements, the factor 9 accounts for the 8 routed experts plus the 1 shared expert (each token is sent to 9 expert destinations total), and the system processes 32 tokens at a time to "strike a balance between compute-to-memory ratio and communication latency" (Section 2.3.2). The 50 GB/s denominator is the theoretical NIC bandwidth.

What this formula reveals is that the communication time scales linearly with the number of expert destinations per token. If each token's 8 experts are on 8 different nodes, the IB communication cost is 8𝑡 where $t$ is the time to send one token over IB.

The node-limited intervention. The key insight is that IB traffic can be deduplicated: if multiple of a token's target experts reside on the same physical node, the originating GPU only needs to send the token over IB once to that node, and the receiving GPU can then forward the token to other intra-node GPUs using NVLink. This forwarding exploits the 4× higher intra-node bandwidth and shifts the communication pattern from being IB-limited to being NVLink-augmented.

The Node-Limited Routing strategy (Section 4.3) operationalizes this deduplication opportunity. The implementation has two parts:

1. A deployment constraint: The 256 routed experts are partitioned into 8 groups of 32 experts each, with each group deployed on a single physical node. This is a hardware-aware sharding decision — it groups experts that will be co-located on the same NVLink domain.

2. An algorithmic constraint: During the TopK expert selection, the router is modified to ensure that each token's 8 selected experts span at most 4 distinct nodes (i.e., $M \leq 4$ where $M$ is the number of nodes contacted). If the naive TopK selection would produce experts spanning 5 or more nodes, the selection is adjusted — likely by re-ranking or re-sampling — to satisfy the constraint.

The result: the IB communication cost drops from 8𝑡 to at most 4𝑡 with some forwarding via NVLink. The actual forwarding cost on NVLink is negligible relative to IB (the bandwidth is 4× higher), so the effective communication time reduction approaches 50%.

The constraint on expert selection quality. Restricting the router to at most 4 nodes is a deliberate trade: it reduces communication cost at the potential expense of routing quality. The paper does not quantify the accuracy impact of this constraint in isolation (it is embedded in the overall model quality results), but the fact that DeepSeek-V3 achieves state-of-the-art performance implies that the constraint is not severely binding — the router has sufficient flexibility within the 32 experts per node (4 per GPU × 8 GPUs = 32 experts per node) to find high-quality expert assignments even when limited to 4 nodes. This is plausible because expert specialization in MoE models tends to have a natural clustering: experts that are semantically similar (e.g., experts that handle mathematical reasoning vs. code generation vs. general text) may naturally co-occur in the routing distribution, and the deployment strategy that groups 32 experts per node may already capture much of this co-occurrence structure.

Additional nuances. The shared expert (one expert that is always activated for every token, regardless of routing) is handled separately: it does not participate in the TopK selection and therefore does not count against the 4-node constraint. It is replicated or distributed in a way that minimizes its communication impact — likely by ensuring that every node has a copy of the shared expert's parameters, since it is activated for every token from every batch. This means the 9 destinations in the communication formula (8 routed + 1 shared) still apply, but the shared expert's communication path is optimized separately.

Multi-Plane Two-Layer Fat-Tree Network Topology

The network design addresses a specific economic and scaling problem: InfiniBand switches with 400G NDR ports typically support only 64 ports per switch. In a traditional fat-tree topology, the number of endpoints scales with the number of leaf-switch ports squared divided by 2 — a two-layer fat-tree with 64-port switches can support approximately 2,048 endpoints (64² / 2). To scale beyond that, a three-layer fat-tree is required, but this adds a third tier of spine switches, which:

• Increases cost by approximately 60%: Table 3 shows a three-layer fat-tree (FT3) for 65,536 endpoints costs an estimated 491 million dollars, compared to 72 million for a two-layer MPFT supporting 16,384 endpoints — a cost-per-endpoint ratio of $7,500 vs.$4,390.
• Increases latency: Each additional switch hop adds serialization, forwarding, and queuing delay. For MoE inference workloads where all-to-all communication latency directly impacts TPOT (Time Per Output Token), even microsecond-level increases are significant (recall from Section 2.3.2 that at 50 GB/s, the ideal transfer time per EP step is 120.96 𝜇s, so switch latency on the order of 1–3 𝜇s per hop is a non-negligible fraction of the total).
• Increases failure probability: More switches means more components that can fail, increasing the expected frequency of training job interruptions.

The Multi-Plane solution. Instead of building one large three-layer fabric, the multi-plane design partitions the cluster into 8 independent two-layer fat-tree planes. Each plane is a complete, self-contained network: it has its own leaf switches and spine switches, carrying traffic for one-eighth of the total endpoint count. The planes are "independent" in the sense that there is no switch-level connectivity between them — a packet on plane 1 never traverses a plane 2 switch. Each node is equipped with 8 GPUs and 8 IB NICs (one per GPU), and each GPU-NIC pair is assigned to a distinct plane: GPU 0 + NIC 0 on plane 1, GPU 1 + NIC 1 on plane 2, and so on (see Figure 3).

This is described in the text (Section 5.1):

"Each node is equipped with eight GPUs and eight IB NICs, with each GPU–NIC pair assigned to a distinct network plane."

The topology is formally described as a "specific subset of the broader MRFT [Multi-Rail Fat-Tree] architecture" (Section 5.1.1). In a multi-rail fat-tree, each endpoint has multiple NICs connected to the same (single-plane) fabric, providing path diversity through multiple parallel rails. In the multi-plane design, each NIC connects to a different fabric plane, not multiple rails within the same plane. The critical difference is traffic isolation: in a single-plane multi-rail network, congestion on any link can affect all rails because they share the same switching fabric. In the multi-plane design, congestion on plane 1 (e.g., due to a burst of traffic between two specific GPUs) does not affect traffic on plane 2 — the planes are physically separate switching domains.

NCCL PXN and inter-plane forwarding. The isolation of planes introduces a new problem: what happens when GPU 0 on node A (connected to plane 1) needs to communicate with GPU 1 on node B (connected to plane 2)? By default, there is no network path — the planes are isolated. The solution leverages NCCL's PXN (PCIe via NVLink) mechanism, originally designed for optimizing multi-rail topologies. PXN allows a GPU to forward traffic to another GPU within the same node over NVLink, and then exit the node through that second GPU's NIC. So the path becomes: GPU 0 (plane 1) → NVLink → GPU 1 (plane 2) → NIC 1 → plane 2 network → destination GPU on plane 2. This intra-node NVLink hop adds latency (the NVLink forwarding hop), but the latency is small relative to the IB fabric latency and the forwarding bandwidth is high (160 GB/s).

This means the multi-plane topology is not pure isolation — cross-plane traffic must take an intra-node detour — but the design is predicated on the assumption that most EP traffic stays within-plane because of the Node-Limited Routing strategy. If tokens are routed to at most 4 nodes, and expert-to-GPU mappings are plane-aligned, the majority of token traffic can be served by the plane the originating GPU belongs to.

Ideal vs. deployed topology. The authors distinguish between the "deployed MPFT network" (Section 5.1, paragraph 3) and the "ideal" multi-plane architecture (Figure 4). In the deployed version, each NIC has a single physical 400G port connected to its assigned plane. In the ideal version, each NIC would have multiple physical ports, each connected to a different plane, with port bonding presenting a single logical interface to the software. A single Queue Pair (QP) could then spray packets across all available ports, achieving load balancing without software-level path selection.

The limitation comes from the current-generation ConnectX-7 NIC: it does not natively support multiple physical ports bonded into a single logical QP. The paper notes that "InfiniBand ConnectX-8 natively supports four plane" (Section 5.1), suggesting this limitation will be lifted in future hardware.

Performance validation. The paper provides three empirical validations that the MPFT design does not harm performance relative to a single-plane multi-rail fat-tree:

1. NCCL all-to-all bandwidth scaling (Figure 5): For GPU counts from 32 to 128, all-to-all algorithm bandwidth is nearly identical between MPFT and MRFT configurations, with values in the 40–60 GB/s range per GPU, saturating the 400G NIC capacity. This validates that PXN-based cross-plane forwarding does not create a bandwidth bottleneck at scale.

2. Latency comparison (Figure 6): For message sizes from 64 bytes to 17 GB (a massive range covering both metadata and data payloads), the relative difference in all-to-all latency between MPFT and MRFT is within ±1.5%, with the curves "nearly identical" (Section 5.1.2). This is the crucial metric for inference: it confirms that the extra NVLink hop for cross-plane traffic does not introduce latency spikes that would impair TPOT.

3. DeepSeek-V3 training throughput (Table 4): On the actual 2,048-GPU DeepSeek-V3 training run, every performance metric — tokens per day (272.80 vs. 272.52 billion), time per step (19.926 vs. 19.946 seconds), MFU causal (38.94% vs. 38.90%), MFU non-causal (43.73% vs. 43.68%) — is statistically indistinguishable between MPFT and MRFT. The differences "fall within normal fluctuations and measurement error."

These results collectively demonstrate that the multi-plane design achieves its economic goal (supporting 16,384 GPUs in a two-layer topology at 40% lower cost per endpoint than a three-layer fat-tree) without measurable performance degradation.

Multi-Head Latent Attention: KV Cache Compression via Joint Low-Rank Projection

Multi-Head Latent Attention addresses a specific inference-time bottleneck: the KV cache. To understand why this matters, we need to trace the data flow during autoregressive decoding.

The KV cache mechanism (standard multi-head attention). During inference, when the model generates the $t$-th token, it needs to compute attention scores between the query vector of token $t$ and the key vectors of all previous tokens (positions 1 through $t-1$). The key and value vectors for those previous tokens were already computed during earlier decoding steps. Rather than recomputing them — which would make generation time quadratic in sequence length ($O(N^2)$ for $N$ tokens) — the standard optimization is to store them in the KV cache: a memory buffer that holds the key and value vectors for all previously processed tokens. At each new decoding step, the model computes only the key and value for the current token, appends them to the cache, and computes attention using the full cache.

The storage cost of this cache is significant. For a model with $h$ attention heads, each of dimension $d_k$ for keys and $d_v$ for values, the per-token cache size (in bytes, for BF16 precision) is:

$S_{\text{cache}} = h \times (d_k + d_v) \times 2 \text{ bytes}$

For DeepSeek-V3, the authors do not provide the raw head count and dimension explicitly in this paper (they are in the DeepSeek-V3 technical report), but the result is given in Table 1: 70.272 KB per token. This is substantially smaller than Qwen-2.5 72B's 327.680 KB (4.66× larger) and LLaMA-3.1 405B's 516.096 KB (7.28× larger).

The memory bandwidth problem. The KV cache is not merely a capacity problem (will it fit in HBM?) — it is a memory bandwidth problem during autoregressive decoding. The attention computation for a single token during decoding involves a matrix-vector multiplication (GEMV) rather than a matrix-matrix multiplication (GEMM). A GEMV operation loads the full key and value matrices from HBM but performs only $O(N)$ floating-point operations per load — it has a very low arithmetic intensity (FLOPs per byte). On modern GPUs with hundreds of TFLOPS of compute capability but only ~2–3 TB/s of HBM bandwidth, GEMV operations are severely memory-bandwidth-bound: the GPU's Tensor Cores spend most of their time idle waiting for data to arrive from memory. Reducing the KV cache size directly reduces the number of bytes that must be fetched from HBM per decoding step, which proportionally reduces the memory-bound latency.

How MLA works. Instead of storing separate key and value vectors for each attention head, MLA compresses the key-value representations of all heads jointly into a single low-dimensional latent vector. The mechanism, illustrated in the lower-left portion of Figure 1, operates as follows:

1. Compression during computation: For each token, the model computes a latent vector $\mathbf{c}_t^{KV}$ of dimension $d_{\text{latent}}$ (substantially smaller than $h \times (d_k + d_v)$). This latent vector is the compressed representation of all the key-value information needed for attention across all heads.

2. Decompression during attention: When attention is computed for a specific head $i$, the latent vector is projected up to the full key and value dimensions for that head using learned projection matrices $\mathbf{W}_{i}^{K}$ and $\mathbf{W}_{i}^{V}$:

$\mathbf{k}_{t,i} = \mathbf{W}_{i}^{K} \mathbf{c}_t^{KV}$ $\mathbf{v}_{t,i} = \mathbf{W}_{i}^{V} \mathbf{c}_t^{KV}$

These projections produce the per-head key and value vectors that then participate in the standard scaled dot-product attention computation.

3. Caching only the latent vector: During inference, only the compressed latent vector $\mathbf{c}_t^{KV}$ is stored in the cache, not the expanded per-head key and value vectors. The decompression projections are recomputed at each attention step from the cached latent vector. Because the latent dimension is much smaller than the expanded dimension, this significantly reduces cache memory.

The compression ratio can be quantified from Table 1: MLA achieves 70 KB per token compared to 327–516 KB for GQA-based models. The compression factor is approximately 4.7–7.3×. This is substantially more aggressive than GQA, which achieves compression by sharing KV heads across multiple query heads but does not reduce the per-head dimension.

The training-time integration. The latent projection matrices and the decompression matrices are "jointly trained with the model" (Section 2.1.2) — meaning they are standard learnable parameters optimized end-to-end via gradient descent with the language modeling objective. There is no separate compression-then-training pipeline; the compression is learned to preserve the information necessary for attention, effectively discovering a low-rank structure in the key-value space of the model's attention heads.

Relationship to Rotary Position Embedding (RoPE). Figure 1 indicates that RoPE (Rotary Position Embedding) is applied after the decompression step for a portion of the query and key vectors. The diagram shows separate paths for $\mathbf{q}_{t,i}^C$ and $\mathbf{k}_{t,i}^C$ (the compressed/decompressed components) and $\mathbf{q}_{t,i}^R$ and $\mathbf{k}_t^R$ (the RoPE-applied components), which are then concatenated. This is necessary because applying RoPE directly to the compressed latent representation would mix positional information across all heads in a way that might interfere with the compression — the positional encoding is applied to the expanded, per-head representations where it can properly capture the relative position information that attention requires.

Comparison with alternatives. The paper positions MLA as a more aggressive compression technique than GQA/MQA and qualitatively different from quantization-based methods:

• GQA/MQA: These share the same Key-Value vectors across groups of query heads (GQA) or across all query heads (MQA). The compression comes from reducing the number of distinct KV pairs, not from reducing their dimensionality. In GQA with $g$ groups, the KV cache size is $g \times (d_k + d_v) \times 2$ bytes compared to $h \times (d_k + d_v) \times 2$ for standard MHA. MLA goes further by compressing even the KV dimensionality itself through the latent bottleneck.
• Windowed KV: These approaches (e.g., Longformer) simply discard KV pairs outside a sliding window, achieving compression by truncation. This inherently limits the model's ability to attend to distant tokens, sacrificing long-context reasoning for memory savings. MLA preserves full attention over the entire sequence history.
• Quantized compression: Methods like KVQuant or KIVI store the KV cache in low-bit representations (e.g., 4-bit or 2-bit), achieving compression without architectural changes. MLA is complementary — the compressed latent vector could itself be quantized for additional savings, though the paper does not explore this combination.

Why MLA matters for the overall system. MLA's KV cache compression is not just a memory-saving technique — it directly enables the cost-efficient inference that makes DeepSeek-V3 practical. The 70 KB per token cache size means that a sequence of 128K tokens (a typical long-context inference scenario) requires approximately 8.75 GB of KV cache memory. At GQA-level compression (327 KB per token), the same sequence would require approximately 41 GB — likely exceeding the HBM capacity of a single H800 (80 GB) after accounting for model weights and activations, forcing multi-GPU inference or aggressive offloading. MLA makes single-GPU long-context inference feasible within the H800's memory budget, which is critical for the "personal use and on-premises deployment" scenarios the paper emphasizes in Section 2.2.2.

Multi-Token Prediction: Self-Drafting Speculative Decoding

The MTP module addresses the sequential bottleneck in autoregressive generation. Standard LLM decoding generates one token per forward pass: the model computes the probability distribution over the vocabulary for position $t+1$, samples or selects the most likely token, appends it to the sequence, and repeats. This process is inherently sequential — the generation of token $t+1$ depends on token $t$ being known — which limits the effective throughput to one token per forward pass, regardless of how many GPU cores are available.

Speculative decoding background. The standard speculative decoding framework uses a lightweight "draft model" to quickly generate candidate future tokens (e.g., the next 3–5 tokens), then uses the full model to verify all candidates in a single parallel forward pass. If the draft tokens are mostly correct (they match what the full model would have generated), the verification pass accepts them and the effective throughput increases by the acceptance rate times the draft length. The key requirement is a draft model that is both fast (does not add significant latency) and accurate (produces tokens the full model agrees with).

MTP's draft model: a single lightweight layer. The Multi-Token Prediction module replaces the external draft model with one additional transformer layer per predicted future token, stacked after the main model's final transformer block. As illustrated in the top portion of Figure 1:

• The main model produces the standard next-token prediction for token $t_2$ conditioned on token $t_1$.
• MTP Module 1 takes the main model's hidden state and predicts token $t_3$ (the "next-next" token).
• MTP Module 2 takes MTP Module 1's hidden state and predicts token $t_4$.
• MTP Module 3 predicts token $t_5$, and so on.

Each MTP module consists of an RMSNorm, a shared embedding layer, a single transformer block (with MLA and MoE components like the main model), and an output head — but the transformer block has fewer parameters and the module is described as "much more lightweight than the full model" (Section 2.3.3). Crucially, the MTP modules share the embedding layer and output head with the main model, amortizing the memory cost of these large parameter matrices.

Training procedure. Unlike draft models in standard speculative decoding (which are typically trained separately or distilled post-hoc), MTP modules are trained end-to-end with the main model. The loss function includes a cross-entropy term for the main model's next-token prediction and separate cross-entropy terms for each MTP module's prediction:

$\mathcal{L} = \mathcal{L}_{\text{Main}} + \mathcal{L}_{\text{MTP}_1} + \mathcal{L}_{\text{MTP}_2} + \mathcal{L}_{\text{MTP}_3}$

Each $\mathcal{L}_{\text{MTP}_k}$ is the standard cross-entropy loss between the module's predicted token distribution and the actual token at that future position in the training sequence. This joint training ensures the MTP modules learn to predict future tokens using representations that are compatible with the main model's hidden state — they are the main model's own learned "guess" at what comes next, not an independently trained draft.

Inference: self-drafting and parallel verification. At inference time, the system operates as follows:

1. Main model forward pass: The main model processes the current token and produces the hidden state for the current position, plus the logits for the next token.

2. Drafting with MTP: The MTP modules, starting from the main model's hidden state, sequentially generate candidate tokens for positions $t+2$, $t+3$, and so on. Each MTP module takes the previous module's hidden state as input (with the embedding of the predicted token concatenated, as shown in Figure 1).

3. Verification: The main model takes the full sequence (original tokens plus MTP-generated draft tokens) and performs a single parallel forward pass, computing attention over the entire sequence. This verification pass produces the model's "true" probability distribution for each future position. The draft tokens are accepted if they match the main model's most likely token at each position (using the standard speculative decoding acceptance criterion based on comparing probabilities).

4. Acceptance and continuation: Accepted draft tokens are appended to the sequence, and the process repeats from the last accepted position. Rejected draft tokens are discarded, and the model generates the correct token from the verification pass before continuing.

Throughput impact. The paper reports empirical results (Section 2.3.3):

"The real world practice data demonstrates that an MTP module achieves an acceptance rate of 80% to 90% for predicting the second subsequent token, which increases the generation TPS by 1.8x compared to the scenario without the MTP module."

An acceptance rate of 80–90% means that 8–9 out of 10 draft tokens are accepted. The 1.8× throughput increase is consistent with an expected acceptance of approximately 1.8 tokens per iteration (1 guaranteed from the main model + ~0.8 from the draft), replacing the 1 token per iteration of standard autoregressive decoding. This is a substantial improvement for what is essentially a single additional lightweight layer per predicted future token.

Why this matters for hardware utilization. The paper draws a connection between MTP and batch size that is subtle but important for the overall system architecture (Section 2.3.3):

"Moreover, by predicting multiple tokens per step, MTP increases the inference batch size, which is crucial for boosting EP computational intensity and hardware utilization."

In expert-parallel inference, each token must be dispatched to and combined from its selected experts. The all-to-all communication that implements this dispatch/combine has a fixed per-message overhead (latency) that is amortized over the number of tokens in the batch. By generating multiple tokens per forward pass, MTP effectively increases the number of tokens being processed per all-to-all communication step, which increases the computational intensity (FLOPs per byte communicated) and improves GPU utilization. This is the "virtuous cycle" referenced earlier: MTP improves throughput both directly (more tokens per step) and indirectly (better amortization of communication overhead).

Training Orchestration: DualPipe, Micro-Batch Overlap, and Parallelism Strategy

The training system stitches the above components together through a parallelism strategy and scheduling algorithm designed to maximize GPU utilization under the H800's bandwidth constraints.

Parallelism strategy (Section 4.2). The paper describes a deliberate choice of which parallelism types to use and which to avoid:

• Expert Parallelism (EP): Used as the primary parallelism strategy, distributing the 256 routed experts across GPUs. This is the natural parallelism for MoE and leverages the 8×400G NIC bandwidth for all-to-all communication.
• Pipeline Parallelism (PP): Enhanced with DualPipe (described below) to overlap communication with computation. PP partitions the model's $L$ layers across multiple GPUs, with each GPU handling a consecutive chunk of layers.
• Tensor Parallelism (TP): Explicitly avoided during training due to "inefficiency under limited NVLink bandwidth" (Section 4.2). TP splits individual weight matrices across GPUs and requires all-reduce communication after each matrix multiplication — a communication pattern that becomes a bottleneck when NVLink bandwidth is constrained. However, TP is "selectively used during inference to improve TTFT and TPOT performance" — the trade-off is different at inference because the batch sizes are typically smaller and the per-token computation is lower, making the relative cost of communication more manageable.
• Data Parallelism (DP): Implicitly used across the cluster, with each data-parallel group processing a different micro-batch. The paper does not detail the DP configuration, but standard practice splits the global batch across DP replicas and uses all-reduce to synchronize gradients.

DualPipe: bidirectional pipeline parallelism. DualPipe is a scheduling algorithm for pipeline parallelism that the paper only briefly describes (Section 4.2, with additional details in the DeepSeek-V3 technical report). The key innovation is that it "overlap[s] attention and MoE computation with MoE communication" and "reduces pipeline bubbles and balances memory usage across GPUs." In standard 1F1B (one forward, one backward) pipeline scheduling, the pipeline has "bubbles" — idle periods at the start and end of the schedule while the pipeline fills and drains. DualPipe reduces these bubbles by scheduling forward and backward passes for different micro-batches in an interleaved pattern that keeps all GPUs busy. The "bidirectional" aspect likely refers to simultaneously processing micro-batches in both forward and backward directions through the pipeline, effectively filling bubbles that would occur at the head and tail of a unidirectional pipeline.

Table 4 reports the measured bubble time for DeepSeek-V3 training: 2.06 seconds out of a 19.926-second step time, or approximately 10.3% of total step time lost to pipeline bubbles. This is a relatively efficient pipeline utilization ratio.

Dual micro-batch overlap for EP communication (Section 2.3.1). During inference, the system uses a dual micro-batch overlap strategy to hide all-to-all communication latency:

"We decouple the computation of MLA and MoE into two distinct stages. While one micro-batch executes a portion of MLA or MoE computation, the other micro-batch simultaneously performs the corresponding dispatch communication. Conversely, during the computation phase of the second micro-batch, the first micro-batch undergoes the combine communication step."

This works because MLA and MoE computation are separable: the MLA attention sub-block and the MoE FFN sub-block are sequential within each transformer layer, so while micro-batch A is computing its MoE FFN, micro-batch B can be performing the all-to-all dispatch for its upcoming MoE FFN. The communication is overlapped with computation from the other micro-batch, keeping the GPU's compute units busy continuously. The theoretical analysis in Section 2.3.2 assumes this overlap is perfect (communication time is fully hidden), which establishes an upper bound on inference speed determined purely by the NIC bandwidth.

Prefill-decode disaggregation (Section 2.3.1). The production inference system separates the prefill phase (processing the input prompt, which is compute-bound and can use large batch sizes) from the decode phase (generating output tokens one at a time, which is memory-bandwidth-bound and latency-sensitive). These two phases are assigned to "different expert parallelism group sizes," meaning the EP configuration is tuned separately for prefill (optimizing for throughput with large batches) and decode (optimizing for latency with small batches). This disaggregation is a standard technique in LLM serving systems (DistServe, Zhong et al., 2024), and the paper confirms that DeepSeek-V3 adopts it as part of their production deployment.

Summary of Design Choices and Their Justifications

• MLA over GQA/MQA: More aggressive compression (4.7–7.3×) needed because HBM bandwidth limits GEMV-bound attention during decoding; MLA's learned low-rank compression preserves attention quality better than simple head sharing.
• Node-Limited Routing (≤4 nodes) over unconstrained TopK: Reduces IB communication cost by deduplicating traffic via NVLink forwarding, directly addressing the 4:1 intra-node-to-inter-node bandwidth gap on the H800.
• Eight-plane two-layer fat-tree over three-layer fat-tree: ~60% lower cost per endpoint, lower latency, and better fault isolation, made viable by NCCL PXN for cross-plane forwarding and Node-Limited Routing to minimize cross-plane traffic.
• FP8 tile-wise (1×128) and block-wise (128×128) quantization over coarser schemes: Aligns with Tensor Core tile dimensions to minimize dequantization overhead and provides sufficient granularity for training stability; validated to within 0.25% accuracy loss on 16B and 230B models before full-scale deployment.
• MTP over external draft models: Single lightweight layer per predicted token keeps memory overhead minimal; end-to-end training with the main model ensures representation compatibility; 80–90% acceptance rate achieves 1.8× throughput while increasing batch size for better EP utilization.
• Avoidance of TP during training over uniform parallelism: TP's all-reduce per matrix multiply saturates the H800's limited NVLink bandwidth; EP and PP with DualPipe are better matched to the available communication fabric.

4. Key Insights and Innovations

Innovation 1: The Inference-Time Memory Wall Is Primarily a Bandwidth Problem, Not a Capacity Problem, and Must Be Solved Architecturally

The dominant framing of the KV cache problem in the LLM literature treats it as a capacity issue: the cache is too large to fit in HBM, so we must compress it (via GQA, MQA, quantization, or windowing) to make it fit. The DeepSeek-V3 paper reframes this fundamentally: the KV cache bottleneck is a memory bandwidth problem during autoregressive decoding, not merely a capacity problem. The distinction matters enormously for what kind of solution is appropriate.

Here is the diagnostic move the paper makes (Section 2.1.2). During training and prefill, attention computation involves matrix-matrix multiplication (GEMM) — the query matrix multiplies the key matrix, producing a compute-bound operation that efficiently saturates Tensor Cores. During decode, however, each new token's query vector multiplies the entire cached key matrix. This is a matrix-vector operation (GEMV) with arithmetic intensity approaching zero: for each byte fetched from HBM, only a handful of floating-point operations are performed. On an H800 with hundreds of TFLOPS of compute but only ~2–3 TB/s of HBM bandwidth, the GPU's compute units idle waiting for memory. The decode step is entirely memory-bandwidth-bound.

This reframing explains why GQA and MQA — the field's standard solutions — are insufficient despite being architecturally simple. GQA reduces KV cache capacity by sharing key-value heads across query groups, but the per-token cache size remains in the hundreds of kilobytes (327 KB for Qwen-2.5 72B, 516 KB for LLaMA-3.1 405B; Table 1). Each byte of that cache must still traverse the HBM bus during every decode step. The capacity pressure is relieved (the cache fits in memory), but the bandwidth pressure remains proportional to cache size. Compression ratio matters not because HBM is full, but because the memory bus is saturated.

MLA's 4.7–7.3× compression (70 KB per token) is therefore not just a capacity optimization — it is a bandwidth optimization that directly accelerates decode by reducing the volume of memory traffic per token. The paper does not state this framing explicitly, but it is implicit in the architecture: MLA compresses the KV representation into a latent vector before storage, so the cached representation is what gets fetched from HBM. The decompression into full-dimension per-head keys and values happens on-chip, using the GPU's compute (which was idle anyway waiting for memory). This is a compute-for-bandwidth trade: accept a small amount of additional on-chip computation (the decompression matrix multiplications) in exchange for a large reduction in off-chip memory traffic.

Why this is a fundamental reframing, not incremental: Prior work on KV cache compression was evaluated almost exclusively on capacity metrics (how many tokens fit in a given HBM budget) or end-to-end throughput (tokens per second). The paper's analysis of the theoretical TPOT bound (Section 2.3.2) — deriving 67 tokens per second as the hard ceiling for EP inference under 400 Gbps IB — shows that the authors think about inference bottlenecks in terms of fundamental physical limits (NIC bandwidth, HBM bandwidth) rather than empirical measurements. This physics-of-computation perspective is what distinguishes this work from typical ML systems papers that report throughput improvements without tracing them to root-cause bandwidth limitations. The insight is that inference acceleration requires bandwidth reduction at every level of the memory hierarchy — HBM, NVLink, IB — not just capacity compression. MLA addresses HBM bandwidth; Node-Limited Routing addresses IB bandwidth; the multi-plane topology addresses the economic cost of scaling IB bandwidth. The paper's thesis, though never stated this concisely, is that all significant inference bottlenecks are bandwidth bottlenecks, and architectural innovation must target bandwidth at each layer of the system.

---

Innovation 2: The Scale-Up/Scale-Out Bandwidth Gap Is a First-Class Architectural Constraint That Must Be Codified in the Routing Algorithm

A standard MoE router selects the top-K experts purely based on affinity scores — the inner product between the token representation and each expert's learned embedding — without regard for where those experts physically reside. This is the default in virtually all MoE implementations, from Shazeer et al. (2017) through Switch Transformer (Fedus et al., 2022) to Mixtral (Jiang et al., 2024). The underlying assumption is that the communication fabric is homogeneous: it does not matter whether an expert is on the local GPU, a peer GPU on the same node, or a remote GPU across the cluster, because the network provides uniform bandwidth (or at least bandwidth that is sufficient to make expert location irrelevant).

DeepSeek-V3 rejects this assumption entirely. On the H800 — a regulatory-constrained GPU with NVLink cut from 900 GB/s to 400 GB/s — the effective intra-node bandwidth (NVLink, ~160 GB/s) and inter-node bandwidth (400 Gbps IB, ~40 GB/s effective) differ by a factor of ~4×. This is not a small gap that can be papered over with better congestion control or larger buffers. It is a structural discontinuity in the communication fabric that, if ignored, causes the MoE dispatch and combine stages to be bottlenecked by the slowest path (IB) while the fastest path (NVLink) sits underutilized.

The Node-Limited Routing strategy (Section 4.3) is the direct algorithmic response: constrain each token's TopK selection so that its 8 routed experts span at most 4 distinct physical nodes. This is not a heuristic; it is a hardware-aware constraint baked into the routing optimization. The router is effectively told: "You may select any 8 experts based on affinity, but you must ensure they are deployed on at most 4 nodes. If your natural TopK violates this, adjust." The constraint exploits the fact that intra-node NVLink forwarding can deduplicate IB traffic — if a token needs to reach two experts on the same node, only one IB transfer is needed, followed by an NVLink forward — but this deduplication only helps if experts are already co-located on nodes. The constraint forces that co-location at the algorithmic level.

Why this is conceptually novel, not just an engineering optimization: Prior work on communication-efficient MoE has focused on post-hoc optimizations — hierarchical all-to-all algorithms, topology-aware NCCL collectives, or gradient compression — that treat the expert assignment as given and try to make the communication faster. Node-Limited Routing changes the expert assignment itself to match the hardware topology. This is a qualitatively different intervention: it is topology-aware routing at the model architecture level, not the communication library level. The router is no longer just a learned function of token-expert affinity; it is a constrained optimizer whose feasible region is defined by the physical network topology. This principle — that model architecture decisions (which experts serve which tokens) should be co-optimized with the physical deployment (which experts are on which nodes and how they are connected) — is a generalizable insight that extends beyond the specific H800 configuration. Any system with heterogeneous interconnect bandwidth (e.g., future chiplet-based designs with different bandwidth tiers, or disaggregated memory architectures) would benefit from routing algorithms that are aware of the bandwidth topology.

The paper validates this approach not by showing it improves performance over unconstrained routing (that comparison is not presented — likely because the unconstrained version would be prohibitively slow and was never run at full scale) but by demonstrating that the constrained version achieves state-of-the-art model quality despite the restriction. The implicit argument is: Look, we imposed a significant algorithmic constraint on the router, and the model still works this well. The constraint was binding enough to be meaningful (it reduced IB traffic by up to 50%) but loose enough to preserve routing quality. This is a negative result turned positive: the fact that restricting the router to 4 nodes did not measurably degrade final model quality tells us something important about the redundancy in MoE routing — experts within the same node are sufficiently diverse that losing access to experts on a fifth node has negligible impact on token-level prediction accuracy. This is a finding about the structure of learned expert specialization, not just about hardware optimization.

---

Innovation 3: A Multi-Plane Fat-Tree Is a Genuinely Viable Alternative to Three-Layer Fat-Trees for Large-Scale AI, Not Just a Cost-Saving Hack

The standard scaling path for fat-tree networks in AI clusters is well-established: start with a two-layer topology (leaf and spine), and when the endpoint count exceeds the scaling limit of the leaf switches (roughly $P^2/2$ endpoints for $P$-port switches), add a third layer of core switches. This is what Meta, Google, and virtually every large-scale AI infrastructure deployment does. The three-layer design is assumed to be the correct architecture for clusters beyond ~4,000–8,000 endpoints, with two-layer designs seen as a transitional phase that must eventually be abandoned for scale.

The DeepSeek-V3 deployment challenges this assumption directly. By deploying eight independent two-layer fat-tree planes — each a complete, self-contained network with its own leaf and spine switches — the cluster supports 16,384 GPUs without the third switching layer (Table 3). The cost savings are substantial: approximately $4,390 per endpoint for the multi-plane two-layer design versus$7,500 per endpoint for a three-layer fat-tree — a 41% reduction. The latency advantage is equally important: each additional switch hop adds serialization and forwarding delay, and for MoE inference workloads where the all-to-all communication time per layer is on the order of 120 μs (Section 2.3.2), even 1–3 μs per hop adds up across 61 layers to tens of milliseconds of additional latency.

What makes this intellectually significant beyond cost reduction is that the multi-plane design challenges an implicit assumption in cluster network architecture: that all GPUs in a training cluster must be reachable through a single, homogeneous switching fabric. The multi-plane design partitions the cluster into eight disjoint network domains, with cross-plane traffic handled by intra-node NVLink forwarding (via NCCL PXN) rather than by the switching fabric itself. This is a federation model rather than a unification model: each plane is an independent network, and the node serves as the bridge between planes.

The risk of this design is that cross-plane traffic — which must take a detour through NVLink — could become a bottleneck if it constitutes a significant fraction of total communication. The paper's performance validation (Figures 5, 6; Table 4) is therefore not just a sanity check; it is a proof that the Node-Limited Routing strategy eliminates enough cross-plane traffic to make the multi-plane design practical. The two innovations — Node-Limited Routing and the multi-plane topology — are co-dependent: the topology is cost-effective only if cross-plane traffic is minimized, and cross-plane traffic is minimized only if the routing algorithm respects the topology. Neither innovation works without the other.

This co-dependence is the deeper insight: network topology and model architecture cannot be designed independently in large-scale MoE systems. The standard approach — design the network topology to provide uniform bandwidth, then design the model to use whatever bandwidth is available — treats the network as a substrate. The DeepSeek-V3 approach treats the network topology as a design parameter that shapes the model architecture (through Node-Limited Routing) and is in turn shaped by it (the topology is viable only because the routing respects it). This is genuine co-design: the model and the network are co-optimized, not layered.

The comparison to Slim Fly (Table 3) provides additional context: Slim Fly is often cited as a cost-efficient alternative to fat-trees, but the multi-plane two-layer fat-tree achieves slightly better cost per endpoint ($4,390 vs.$4,400) while retaining the well-understood fault tolerance and routing properties of fat-tree topologies. This is not a theoretical advance in network topology design — fat-trees and multi-plane variants are well-known — but an empirical demonstration that a multi-plane design is competitive with state-of-the-art alternatives in a production 2,048-GPU training deployment, with the data to prove it (MFU within 0.05 percentage points of single-plane multi-rail, latency differences within ±1.5%).

---

Innovation 4: FP8 Training at 671B Parameters Requires Fine-Grained Quantization Co-Designed with the Accumulation Precision Limits of the Hardware, Not Just a Precision Format Choice

The dominant narrative around low-precision training has been format-centric: BF16 is good enough for most models, FP8 might work with careful scaling, FP4 is aspirational. The assumption is that the choice of precision format (which determines dynamic range and mantissa precision) is the primary design decision, and implementation details (quantization granularity, accumulation) are secondary engineering concerns.

The DeepSeek-V3 experience inverts this: the quantization granularity (tile-wise 1×128 for activations, block-wise 128×128 for weights) and the interaction with hardware accumulation precision (FP22 registers with only 13 mantissa bits on Hopper Tensor Cores) are more consequential for training stability than the choice of FP8 format itself. The paper reveals that Hopper's FP8 Tensor Cores, when accumulating 32 partial products, right-shift mantissas to align with the maximum exponent, then truncate beyond 13 fraction bits (Section 3.1.1). This truncation is not a bug — it is a deliberate hardware design tradeoff to keep accumulation registers compact — but it means that naive per-tensor quantization (one scaling factor for an entire activation tensor or weight matrix) compounds this truncation error across thousands of accumulation steps, producing accuracy degradation that is unacceptable for a 671B-parameter model.

The innovation is the hardware-aligned quantization granularity: the tile and block dimensions (1×128 and 128×128) are chosen to match the Tensor Core's internal processing tile size, not based on statistical properties of the tensors. This alignment minimizes the number of times partial sums must be transferred from Tensor Cores to CUDA Cores for dequantization (scaling factor multiplication), which the paper identifies as the primary throughput bottleneck for fine-grained FP8 (Section 3.1.1). The choice is a direct response to a specific hardware limitation — the inability of Tensor Cores to natively consume per-group scaling factors — and the paper's hardware recommendation (Section 3.1.2) is correspondingly specific: future Tensor Cores should support group-scaled matrix multiplication natively, as NVIDIA Blackwell's microscaling format support begins to do.

Why this is more than just an engineering report: The paper's validation methodology (Section 2.4) — testing each FP8 technique on 16B then 230B models before full-scale deployment, with a strict 0.25% accuracy loss threshold versus BF16 — establishes a quantitative framework for evaluating low-precision training reliability. The 0.25% figure is not arbitrary; it represents a tolerance that, if exceeded at small scale, would likely compound to unacceptable degradation at 671B parameters. This hierarchical validation is a transferable practice: any team attempting FP8 training at scale can adopt the same methodology. The paper's key negative finding — that LogFMT, a custom logarithmic number format with superior representational properties, was abandoned because GPU log/exp bandwidth was insufficient and the encode/decode overhead was 50–100% (Section 3.2.1) — is equally valuable. It demonstrates that representational quality alone does not determine the viability of a low-precision format; the format must also be hardware-efficient to encode, decode, and compute with. This is a systems-level insight that pure numerical analysis would miss.

5. Experimental Analysis

This section is unusual for an ISCA paper. Rather than presenting a controlled experiment with a clear independent variable, it reports the operational characteristics and validation of an integrated production system. The "experiments" are primarily engineering validations that confirm the multi-plane network topology, the Node-Limited Routing strategy, and the FP8 training framework perform as designed under real-world training and inference conditions on a 2,048-GPU H800 cluster. The section also includes controlled latency and bandwidth measurements of the network fabric, an attempt to deploy a logarithmic number format (LogFMT) that was ultimately abandoned, and throughput comparisons of speculative decoding. We organize these measurements into a coherent evaluation narrative.

Evaluation Methodology

• Dataset. The training corpus is not specified in this paper, as the focus is on hardware performance, not downstream task accuracy. References are made to the DeepSeek-V3 technical report for training data details. FP8 validation ablations used controlled experiments on 16B-parameter and 230B-parameter model variants (Section 2.4), but the specific training dataset for those ablations is not named. Network performance tests used synthetic NCCL collective benchmarks (all-to-all, all-gather, reduce-scatter) rather than application-level workloads.

• Base model(s). The primary system under evaluation is DeepSeek-V3, a 671B-parameter Mixture-of-Experts model with 37B activated parameters per token. It trained on 2,048 NVIDIA H800 GPUs with 400 Gbps InfiniBand interconnects. For FP8 validation ablations, smaller models at 16B and 230B parameters (DeepSeek-V2 variants) were used. The 230B-scale model served as the primary validation checkpoint before committing to the full 671B training run (Section 2.4). The H800 GPU is a regulatory-constrained variant of the H100 with NVLink bandwidth reduced from 900 GB/s to 400 GB/s and reduced FP64 performance (Section 4.1).

• Metrics. The paper uses multiple metrics spanning network, training, and inference performance:

  • Training throughput: tokens per day (272.80 billion), time per training step (19.926 seconds), with breakdowns into 1F (forward time: 1.13 seconds), 1B (input backward time: 1.99 seconds), 1W (weight backward time: 0.48 seconds), 1F1B (combined forward-backward time: 13.95 seconds), bubble time (2.06 seconds), and optimizer time (0.29 seconds) — all from Table 4.
  • Model FLOPs Utilization (MFU): Two variants are reported (Table 4): non-causal MFU (counting the full attention matrix FLOPs, following the Megatron convention) at 43.73%, and causal MFU (counting only the lower-triangle FLOPs of the attention matrix, following the FlashAttention convention) at 38.94%. Both are computed relative to the H800's peak BF16 throughput.
  • Network algorithm bandwidth: Measured in GB/s for NCCL all-to-all, all-gather, and reduce-scatter collective operations at GPU counts from 32 to 128 (Figures 5, 7, 8). For DeepEP production kernels, dispatch and combine bandwidth are reported at 16, 32, 64, and 128 GPUs with 4,096 tokens per GPU (Figure 7).
  • Network latency: End-to-end CPU-side latency in microseconds for 64-byte data transmission over InfiniBand, RoCE, and intra-node NVLink, measured at same-leaf and cross-leaf switch configurations (Table 5). For MPFT vs. MRFT comparison, relative latency difference (percentage) is reported across message sizes from 64 bytes to 17 GB (Figure 6).
  • Inference throughput: Tokens per second (TPS), both theoretical upper bounds derived from communication bandwidth analysis (67 TPS under 400 Gbps IB; 1,200 TPS under idealized 900 GB/s NVLink-like fabric; Section 2.3.2) and empirical measurements with the MTP module (1.8× improvement over baseline; Section 2.3.3). MTP acceptance rate: 80–90% for the second subsequent token.
  • FP8 accuracy degradation: Relative loss compared to BF16 baselines, reported as below 0.25% on 16B and 230B models (Section 2.4).

• Baselines. The paper compares the deployed Multi-Plane Two-Layer Fat-Tree (MPFT) topology against a Single-Plane Multi-Rail Fat-Tree (MRFT) configuration on the same physical cluster by reconfiguring the network topology (Section 5.1.2). For routing protocol comparisons (Figure 8), Equal-Cost Multi-Path (ECMP), Adaptive Routing (AR), and Static Routing are compared on a RoCE network. For low-latency interconnect comparisons (Table 5), InfiniBand, RoCE, and NVLink are measured separately. For FP8, the baseline is BF16 training on the same model architecture, with accuracy loss measured as the deviation from the BF16 reference.

• Generation budget / compute accounting. The paper does not operate with a standardized "compute budget" concept analogous to the generation budgets in model-scaling papers. Instead, compute is accounted at the hardware level:

  • Training cost is measured in GFLOPS per token: 250 GFLOPS per token for DeepSeek-V3 MoE, compared to 394 GFLOPS for a 72B dense model and 2,448 GFLOPS for a 405B dense model (Table 2). This is computed from the activated parameter count, not the total parameter count.
  • Communication cost is measured in bytes transferred per token per layer for expert parallelism: (1 byte FP8 dispatch + 2 bytes BF16 combine) × 32 tokens × 9 expert destinations × ~7K hidden size = ~121 μs at 50 GB/s theoretical NIC bandwidth (Section 2.3.2). A latency of approximately 1–3 μs per switch hop is implicit in the MPFT vs. FT3 comparison (Section 5.1.1).
  • Network cost is measured in estimated millions of dollars for switch infrastructure, derived from the methodology in the Slim Fly paper (Blach et al., 2025): $72 million for MPFT supporting 16,384 endpoints vs.$491 million for a three-layer fat-tree supporting 65,536 endpoints (Table 3). Cost per endpoint: $4,390 for MPFT vs.$7,500 for FT3.

• Cross-validation / statistical protocol. The paper uses two distinct validation strategies:

  • For FP8 training: hierarchical validation — each technique is first validated on 16B-parameter models, then on 230B-parameter DeepSeek-V2 variants, and only after confirming accuracy degradation below 0.25% is it integrated into the full 671B DeepSeek-V3 run (Section 2.4). The paper states: "Given the prohibitive cost of exhaustive ablation on full-scale models, we adopt a hierarchical and resource-efficient validation pipeline."
  • For network topology: reconfiguration-based comparison — the physical cluster's network topology was modified to compare MPFT and MRFT configurations, with performance measured on the actual hardware (Section 5.1.2). The paper notes that differences in Table 4 "fall within normal fluctuations and measurement error," though no formal statistical test (e.g., confidence intervals, p-values) is reported. The cluster was limited to "just over two thousand GPUs" due to "policy and regulatory constraints" rather than the theoretical maximum of 16,384 GPUs (Section 5.1).
  • No cross-validation or held-out evaluation is reported for model quality metrics, as these are outside the paper's scope and deferred to the DeepSeek-V3 technical report.

Main Quantitative Results

Multi-Plane Fat-Tree vs. Multi-Rail Fat-Tree: Network Performance

The central empirical question for the network topology design is whether the multi-plane two-layer fat-tree (MPFT) — which uses eight physically independent switching planes with cross-plane traffic routed through intra-node NVLink forwarding — degrades communication performance relative to a conventional single-plane multi-rail fat-tree (MRFT) where all NICs connect to the same switching fabric. The claim under test is that MPFT achieves equivalent performance at substantially lower infrastructure cost.

All-to-all algorithm bandwidth scaling (Figure 5). The paper measures NCCL all-to-all algorithm bandwidth for 32, 64, and 128 GPUs, at message sizes of 128 MiB, 256 MiB, 512 MiB, 1 GiB, 2 GiB, 4 GiB, 8 GiB, and 16 GiB, comparing MPFT and MRFT configurations. For 32 GPUs, both topologies achieve approximately 55–65 GB/s at large message sizes (4 GiB and above), with MPFT showing a slight advantage at some sizes. For 64 GPUs, bandwidth is 45–55 GB/s, and for 128 GPUs, approximately 40–50 GB/s. The curves are "very similar" (Section 5.1.2) with no systematic advantage for either topology. The per-GPU bandwidth closely approaches the 50 GB/s theoretical maximum of a single 400 Gbps NIC, indicating that the PXN-based NVLink forwarding for cross-plane traffic in MPFT does not create a throughput bottleneck.

All-to-all latency comparison (Figure 6). This test covers an extraordinarily wide range of message sizes — from 64 bytes to 17,179,869,184 bytes (~17 GB) — measuring the relative latency difference (MPFT minus MRFT, as a percentage of MRFT latency). For small messages (64 bytes to 1 KB), MPFT shows slightly higher latency (approximately +1.0% to +1.5%), likely reflecting the additional NVLink forwarding hop for cross-plane traffic. For messages from 1 KB to 1 MB, the difference is negligible (approximately ±0.25%). For large messages (1 MB to 17 GB), MPFT shows a slight latency advantage (approximately −0.25% to −0.5%). The overall pattern is described as "nearly identical" performance (Section 5.1.2). This is significant for inference: the all-to-all communication in EP involves many small-to-medium messages (each token's dispatch to 8 experts), and the Figure 6 data confirms that MPFT's cross-plane detour does not add latency spikes that would impair the time-per-output-token.

DeepEP production kernel performance (Figure 7). Beyond synthetic NCCL benchmarks, the paper reports the throughput of the production DeepEP expert-parallel communication library (open-sourced; Zhao et al., 2025) on MPFT. For GPU counts of 16, 32, 64, and 128, with each GPU processing 4,096 tokens (a realistic inference batch size), the dispatch and combine all-to-all kernels are measured separately:

• At 16 GPUs: dispatch achieves approximately 42.47 GB/s, combine achieves approximately 43.05 GB/s.
• At 32 GPUs: dispatch ~58.02 GB/s, combine ~56.96 GB/s.
• At 64 GPUs: dispatch ~50.58 GB/s, combine ~48.54 GB/s.
• At 128 GPUs: dispatch ~45.34 GB/s, combine ~41.60 GB/s.

The paper states that "each GPU achieves a high bandwidth exceeding 40GB/s in a multi-plane network, providing reliable performance that meets the demands of training" (Section 5.1.2). The 50 GB/s theoretical NIC maximum is approached or exceeded at 32 GPUs (likely due to measurement methodology that accounts for bidirectional traffic or effective utilization exceeding the unidirectional specification), and the bandwidth scales reasonably with GPU count — the decline from 32 to 128 GPUs reflects the increased communication fan-out and potential congestion at higher scales. The combine bandwidth is consistently slightly lower than dispatch, which is expected because combine uses BF16 (2 bytes per element) while dispatch uses FP8 (1 byte), and the BF16 combine involves reduction operations (summing partial results from multiple experts for the same token) that add computational overhead.

Full-scale training throughput comparison (Table 4). The most operationally significant result: when training DeepSeek-V3 on 2,048 GPUs, every measured training metric is statistically indistinguishable between MPFT and MRFT:

The differences are sub-percentage-point across all metrics. The 0.02-second difference in total step time (~19.93 seconds) represents a 0.1% variation. Bubble time — a measure of pipeline inefficiency — is 2.06 seconds vs. 2.03 seconds, or approximately 10.3% of total step time in both configurations. MFU of 38.94% (causal) is reasonable for a MoE model with all-to-all communication overhead, and the fact that it is identical to three significant figures between MPFT and MRFT is strong evidence that the multi-plane topology introduces no measurable training overhead.

Network topology cost comparison (Table 3). While not a performance measurement, the economic argument is central to the paper's thesis:

The cost per endpoint for MPFT ($4,390) is identical to a two-layer fat-tree at its maximum scale (2,048 endpoints), marginally better than Slim Fly ($4,400), and substantially better than a three-layer fat-tree ($7,500). This validates the core economic claim: MPFT extends the cost-efficiency of two-layer designs to scales that would normally require three-layer topologies.

InfiniBand vs. RoCE Latency (Table 5)

For latency-sensitive workloads (EP all-to-all in MoE inference, where per-step communication time is ~121 μs under ideal conditions), the choice of link-layer protocol is consequential. Table 5 reports CPU-side end-to-end latency for 64-byte data transmission:

InfiniBand achieves 22% lower latency than RoCE at same-leaf (2.8 vs. 3.6 μs) and 34% lower at cross-leaf (3.7 vs. 5.6 μs). This translates to a latency advantage of 0.8–1.9 μs per switch hop. For the 61 layers of DeepSeek-V3, each requiring two all-to-all operations (dispatch and combine), a 1 μs difference per hop across 122 all-to-all operations accumulates to approximately 0.12 ms of additional latency — a small but non-negligible fraction of the ~14.76 ms theoretical per-token inference time (Section 2.3.2).

NVLink's intra-node latency (3.33 μs) is slightly higher than IB same-leaf (2.8 μs), which is notable because NVLink is typically assumed to be lower-latency than network fabrics. This likely reflects the specific measurement methodology (CPU-side measurement may include PCIe or other interface overhead) rather than the raw wire latency.

RoCE Routing Protocol Comparison (Figure 8)

The paper evaluates three routing strategies on a RoCE network for all-gather and reduce-scatter collectives at various tensor parallelism (TP) dimensions:

• Adaptive Routing (AR): Dynamically sprays packets across multiple paths based on real-time congestion. At TP=8, AR achieves approximately 175 GB/s for reduce-scatter and ~190 GB/s for all-gather. Performance degrades gracefully as TP decreases (TP=4: ~170 GB/s reduce-scatter, ~180 GB/s all-gather; TP=2: ~155 GB/s, ~170 GB/s; no TP: ~120 GB/s, ~140 GB/s).
• Static Routing: Pre-configured route tables optimized for specific destination pairs. Performance is similar to AR at high TP dimensions (TP=8: ~170 GB/s reduce-scatter, ~185 GB/s all-gather) but degrades more sharply at low TP (no TP: ~90 GB/s reduce-scatter, ~105 GB/s all-gather).
• ECMP (Equal-Cost Multi-Path): The default RoCE routing, which hashes flows to paths. Severely underperforms at all TP dimensions: TP=8 achieves only ~55 GB/s for reduce-scatter and ~35 GB/s for all-gather, dropping to near-zero at lower TP dimensions. The paper diagnoses this as "severe congestion performance degradation" due to LLM DP traffic lacking randomness and causing multiple flows to converge on the same interconnect link (Section 5.2.2).

The key finding is that AR achieves the best and most consistent performance across all configurations, with a 3–5× improvement over ECMP at TP=8 and a dramatic advantage at lower TP dimensions. This motivates the paper's recommendation that "adaptive routing offers superior performance and scalability" for large-scale all-to-all communication (Section 5.2.2).

Multi-Token Prediction Inference Throughput

The paper reports empirical throughput measurements for the MTP speculative decoding module (Section 2.3.3), though the measurements are described in natural language rather than presented in a dedicated figure:

"The real world practice data demonstrates that an MTP module achieves an acceptance rate of 80% to 90% for predicting the second subsequent token, which increases the generation TPS by 1.8x compared to the scenario without the MTP module."

The 1.8× throughput figure is directly reported without breakdowns by sequence length, batch size, or hardware configuration. The acceptance rate of 80–90% for the second subsequent token implies that the MTP model's draft predictions match the main model's output 80–90% of the time. The paper does not report acceptance rates for the third, fourth, or higher subsequent tokens, though the architecture supports multiple MTP modules (Figure 1 shows three). The 1.8× overall throughput gain is consistent with acceptance of approximately 1.8 tokens per forward pass on average (1 from the main model + ~0.8 from the draft).

FP8 Accuracy Validation

The paper does not present detailed FP8 accuracy measurements in table or figure form. Instead, it reports a summary finding (Section 2.4):

"Under these controlled settings, the relative accuracy loss compared to BF16 remains below 0.25%, attributable to our use of high-precision accumulation and fine-grained quantization strategies."

This 0.25% threshold applies to the 16B and 230B validation models, and the paper asserts it held for the full 671B DeepSeek-V3 training (though no specific 671B accuracy figure is reported in this paper, as model quality benchmarks are deferred to the technical report). The validation pipeline is described as "fine-grained FP8 training ablation studies on both 16B and 230B DeepSeek-V2 models before final integration" (Section 2.4).

Ablation Studies and Robustness Checks

LogFMT precision format (Section 3.2): A custom logarithmic floating-point format (LogFMT-nBit) was designed, tested, and ultimately not deployed due to hardware inefficiency. LogFMT maps activation values to log-space for more uniform distribution, supporting dynamic representation range per block. On 7B-parameter dense models, LogFMT-8Bit showed "superior training accuracy compared to E4M3 or E5M2" when quantizing residual branch outputs (simulating the MoE combine stage). LogFMT-10Bit was "similar to BF16 combine." However, the format was abandoned because "insufficient GPU bandwidth for log/exp operations and excessive register pressure during encode/decode" caused a "substantial" overhead of 50–100% when encode/decode operations were fused with all-to-all communication (Section 3.2.1). This is an informative negative result: representational quality is necessary but not sufficient for a low-precision format to be deployable; hardware must provide efficient encode/decode pathways.

NCCL PXN for multi-plane forwarding (Section 5.1.1): The MPFT topology relies on NCCL's PXN (PCIe via NVLink) mechanism for cross-plane traffic forwarding. The paper notes that "NCCL's support for PXN technology addresses the inherent challenge of inter-plane isolation, enabling efficient communication even when direct interconnectivity between planes is absent." The performance validation (Figures 5, 6; Table 4) implicitly demonstrates that PXN-based forwarding is sufficiently efficient, but the paper does not present an ablation comparing PXN-enabled vs. PXN-disabled forwarding. This is a methodological gap: the MPFT design's viability depends on PXN, and a direct measurement of the PXN forwarding overhead would strengthen the validation.

Expert parallelism all-to-all kernel optimization (Section 4.4): The paper describes a transition from SM-based communication handling (where up to 20 of 132 SMs on the H800 GPU are allocated for "communication-related operations" including filling QPs, WQEs, and NVLink data forwarding) to RDMA-based communication for production inference. The specific bottleneck: "To maximize throughput in online inference, we perform EP all-to-all communication entirely through NIC RDMA, avoiding SM resource contention and improving compute efficiency" (Section 4.4). No quantification is provided for the SM overhead in training (e.g., percentage of compute lost to communication processing), nor for the throughput improvement from switching to RDMA in inference.

DualPipe pipeline efficiency (Table 4): The pipeline bubble time is 2.06 seconds out of 19.926 seconds step time (10.3% overhead). While not presented as an ablation, this figure implicitly validates the DualPipe scheduling algorithm against the simpler 1F1B baseline — standard 1F1B pipeline parallelism typically incurs higher bubble overhead, especially with many pipeline stages. However, the paper does not present a direct comparison of DualPipe vs. standard 1F1B scheduling, making it difficult to quantify the improvement.

Micro-batch overlap for EP communication (Section 2.3.1): The theoretical TPOT upper bound of 67 tokens per second under 400 Gbps IB (Section 2.3.2) assumes "perfect overlap between computation and communication." The paper acknowledges this is idealized: "our theoretical best-case analysis assumes that computation overhead is minimized, so the upper bound on performance is determined by communication latency." No empirical TPOT measurement is presented to validate how close the production system approaches this bound. The gap between theoretical 67 TPS and actual achieved throughput — which would depend on the effectiveness of the dual micro-batch overlap, MLA computation time, and other real-world factors — remains unquantified.

Multi-plane vs. single-plane at 2,048 GPU scale (Table 4): The full-scale training comparison operates at 2,048 GPUs — well within the capacity of a single two-layer fat-tree with 64-port switches (which can support exactly 2,048 endpoints). The paper's central scaling argument — that MPFT enables scaling to 16,384 GPUs while retaining two-layer cost efficiency — is not directly tested at that scale due to regulatory constraints ("just over two thousand GPUs were ultimately deployed," Section 5.1). The only evidence for larger-scale viability is theoretical: the topology supports 16,384 endpoints, and the 2,048-GPU results show no performance penalty that would preclude scaling.

Critical Assessment

The experimental section of this paper serves a fundamentally different purpose from the model-scaling experiments in a typical ML paper. There are no accuracy-vs-compute plots, no difficulty-bin breakdowns, no FLOPs-matched comparisons against larger models. The "experiments" are primarily engineering validations that confirm the co-designed hardware and software systems function as intended. This is appropriate for the paper's venue (ISCA, a computer architecture conference) and its stated goal (to provide "actionable insights for scaling LLMs efficiently" and "a practical blueprint for innovation in next-generation AI systems," Section 1.2). However, it means that many of the paper's most ambitious claims are supported by operational data rather than controlled comparisons.

Does MPFT achieve equivalent performance to MRFT at lower cost? The evidence is strong at the 2,048-GPU scale. Table 4 shows sub-percentage-point differences across all training metrics. Figures 5 and 6 show comparable all-to-all bandwidth and latency across a range of message sizes and GPU counts. The cost advantage (Table 3: $4,390 vs.$7,500 per endpoint) is well-established, though it is an estimate derived from prior work's methodology rather than an audited accounting of the DeepSeek cluster's actual expenditure. The caveat is that the comparison is tested at 2,048 GPUs — a scale where a single two-layer fat-tree is sufficient — and the claimed scaling advantage to 16,384 GPUs is projected, not measured. The paper cannot demonstrate that MPFT at 16,384 GPUs achieves equivalent MFU to MRFT at that scale, because the cluster does not exist.

Does Node-Limited Routing reduce communication cost without degrading model quality? The paper demonstrates that Node-Limited Routing is implemented and that the resulting model achieves state-of-the-art performance (deferred to the technical report). It does not experimentally compare Node-Limited Routing against unconstrained TopK routing on the same hardware to quantify the communication bandwidth reduction or the accuracy impact. The inference-time bandwidth calculation (Section 2.3.2) shows that constraining to 4 nodes reduces IB traffic, but this is a theoretical calculation, not a measurement. The claim that the constraint is "not severely binding" is supported only by the downstream model quality, not by a direct measurement of routing quality (e.g., the correlation between constrained and unconstrained expert selections, or the average number of nodes that the unconstrained router would naturally select).

Does FP8 training achieve accuracy parity with BF16? The 0.25% accuracy loss figure is asserted based on small-scale and medium-scale validation (16B and 230B models), but no FP8-vs-BF16 accuracy data is presented in this paper — not in a table, not in a figure, not with specific metrics. The validation methodology (hierarchical testing before full-scale deployment) is sound, but the reader cannot independently assess the claim. For an ISCA audience, the more salient question — whether FP8 training converges stably and whether the claimed 2× throughput improvement on Tensor Cores is realized — is not directly measured. The paper reports only BF16-based MFU (Table 4), not FP8-based throughput.

Does the theoretical TPOT upper bound meaningfully characterize inference performance? The 67 TPS bound (Section 2.3.2) is a useful limiting analysis, but it rests on assumptions that the paper does not validate: perfect micro-batch overlap (communication latency fully hidden), 32 tokens per batch (the chosen "balance between compute-to-memory ratio and communication latency"), and no MLA computation time (in practice, "MLA computations typically dominate execution time"). The paper acknowledges that "in practical inference workloads, however, request contexts are often much longer, and MLA computations typically dominate execution time." No empirical TPOT measurement is presented. The 1.8× throughput improvement from MTP is stated but not broken down by configuration. This limits the reader's ability to assess how close the production system comes to the theoretical ceiling.

Are the routing protocol comparisons (Figure 8) performed on the production-scale network? The paper describes these as RoCE network tests, but the DeepSeek-V3 production cluster uses InfiniBand (Section 5.1). The Figure 8 experiments appear to be on a separate Ethernet-based testbed, not the production IB fabric. This is appropriate for illustrating the advantage of adaptive routing — and the paper explicitly recommends RoCE improvements for future hardware — but the results do not directly characterize the production system's performance.

What is missing that would strengthen the evaluation? Several experiments would substantially improve the paper's empirical grounding:

• A direct MPFT vs. MRFT comparison at the maximum supported scale (or as close as feasible, given regulatory constraints) to validate the claimed scaling advantage. Even a 4,096-GPU comparison across a subset of metrics would help.
• An empirical TPOT measurement for the production inference system, with breakdowns by request length and batch size, to validate the theoretical upper bound and quantify the gap between theory and practice.
• An FP8-vs-BF16 training comparison on a measurable accuracy metric (perplexity, downstream task accuracy) at the 230B scale, presented in the paper rather than deferred to the technical report.
• A quantification of the SM overhead for communication processing (the "up to 20 SMs" claim) and the improvement from switching to RDMA-based EP communication in inference.
• MTP throughput measurements disaggregated by batch size and sequence length, with acceptance rate reported for the third and higher predicted tokens, not just the second.
• A measurement of the Node-Limited Routing constraint's impact — specifically, what fraction of tokens would naturally route to >4 nodes under unconstrained TopK, and what is the average number of nodes per token after constraint application.

The paper's primary contribution is its prescriptive framework for hardware-model co-design, not its experimental results. The experiments serve as existence proofs that the co-designed system trains efficiently, communicates effectively, and meets its cost targets. They are adequate for this purpose but should not be mistaken for the kind of systematic, variable-isolated experimental analysis that characterizes model-scaling papers. The paper's strength is in its diagnosis of hardware bottlenecks and its articulation of co-design principles; its weakness is that many of those principles are validated only implicitly through the success of the integrated system, not through controlled ablation of each principle in isolation.

6. Limitations and Trade-offs

6.1 The Multi-Plane Network Topology Is Validated Only at 2,048 GPUs, Not at Its Claimed 16,384-GPU Scale

The assumption or constraint. The paper's central economic argument for the Multi-Plane Fat-Tree (MPFT) topology is that it extends the cost-efficiency of two-layer designs to "up to 16,384 GPUs" (Section 5.1, Table 3), avoiding the ~60% cost premium of a three-layer fat-tree at scale. The cost-per-endpoint advantage ($4,390 for MPFT vs.$7,500 for FT3) is predicated on this scaling claim. However, the deployed cluster is limited to "just over two thousand GPUs" — a scale where a conventional two-layer fat-tree with 64-port switches can support exactly 2,048 endpoints without any multi-plane partitioning. The authors are transparent about this: "Due to policy and regulatory constraints, just over two thousand GPUs were ultimately deployed" (Section 5.1).

The consequence. At 2,048 GPUs, the multi-plane design is functionally equivalent to a single-plane two-layer fat-tree — each plane handles only 256 GPUs (2,048 / 8 planes), and the total switch count is not reduced relative to a single two-layer fabric at that scale. The claimed cost advantage for 16,384 GPUs is therefore a projection, not a measurement. Scaling from 2,048 to 16,384 GPUs introduces qualitatively new failure modes that the paper cannot validate:

• Cross-plane traffic growth: At 2,048 GPUs with Node-Limited Routing (≤4 nodes per token), the absolute volume of cross-plane traffic is bounded because the total node count is small (256 nodes). At 16,384 GPUs (2,048 nodes, with each plane handling 256 nodes), the probability that a token's 4 target nodes span multiple planes increases substantially, increasing cross-plane NVLink forwarding pressure. The paper's performance validation (Figures 5, 6; Table 4) cannot detect whether this forwarding would become a bottleneck at scale.
• All-to-all congestion at scale: The all-to-all bandwidth measurements (Figure 5) extend only to 128 GPUs — far below the 16,384-GPU maximum. All-to-all communication patterns for MoE dispatch/combine become increasingly bandwidth-intensive with GPU count because each GPU must communicate with a larger set of peers. The DeepEP validation (Figure 7) covers up to 128 GPUs and shows bandwidth declining from ~58 GB/s at 32 GPUs to ~42 GB/s at 128 GPUs — a 28% reduction. Extrapolating this trend to 16,384 GPUs is speculative.
• NCCL PXN scalability: PXN-based forwarding for cross-plane traffic relies on NVLink paths within each node. At 2,048 GPUs, the NVLink forwarding load is light because most EP traffic stays within-plane. At 16,384 GPUs with proportionally more cross-plane traffic, the intra-node NVLink bandwidth (already constrained to 400 GB/s on the H800) could become a secondary bottleneck — a scenario the paper does not model or measure.

What evidence exists in the paper. The evidence is exclusively at small scale: Figures 5 and 6 compare MPFT and MRFT at 32–128 GPUs; Table 4 compares them at 2,048 GPUs. The topology's theoretical capacity (16,384 endpoints) is derived from switch port counts (64 ports × 64 ports / 2 × 8 planes), not from any measurement of scaled communication performance. The cost comparison in Table 3 is estimated using the methodology from the Slim Fly paper (Blach et al., 2025), not from an audited build-out of a 16,384-GPU MPFT cluster.

Mitigation status. The paper does not attempt to mitigate this limitation — it acknowledges the regulatory constraint that prevented larger-scale deployment but does not discuss it as a limitation of the validation. No simulation, analytical model, or small-scale proxy experiment is offered to bridge the gap between the 2,048-GPU measurements and the 16,384-GPU scaling claim. The paper's recommendation that "future NICs fully support advanced multi-plane capabilities, allowing two-tier fat-tree networks to scale effectively to much larger AI clusters" (Section 5.1) implicitly concedes that current hardware limits the practical validation of this claim.

---

6.2 Difficulty Estimation Cost Is Entirely Unaccounted for in the Theoretical Inference Speed Upper Bound

The assumption or constraint. The theoretical TPOT upper bound of 67 tokens per second under 400 Gbps InfiniBand (Section 2.3.2) assumes perfect dual micro-batch overlap — communication time is fully hidden behind computation, so the per-layer time is determined solely by the all-to-all communication bandwidth. The derivation explicitly states: "our theoretical best-case analysis assumes that computation overhead is minimized, so the upper bound on performance is determined by communication latency. In practical inference workloads, however, request contexts are often much longer, and MLA computations typically dominate execution time" (Section 2.3.2).

The consequence. The 67 TPS figure is not a realistic performance estimate — it is a theoretical ceiling that the production system cannot approach because the assumption of perfect overlap is violated in practice. The paper identifies two specific reasons:

1. MLA computation time is not negligible: For long-context requests (the primary use case where KV cache compression matters most), the MLA attention computation — decompressing the latent KV vectors and computing attention over long cached sequences — "typically dominates execution time" (Section 2.3.2). The GEMV operations in attention are memory-bandwidth-bound, and their latency scales with KV cache size. The 4.7–7.3× KV cache compression from MLA reduces but does not eliminate this latency; it remains the execution-time bottleneck for long sequences, not the EP all-to-all communication that drives the 67 TPS calculation.

2. Dual micro-batch overlap is imperfect: The overlap strategy requires that one micro-batch's computation time exceeds the other's communication time. If MLA computation for a micro-batch is shorter than the EP all-to-all communication for the other micro-batch (or vice versa), the overlap is partial — the GPU stalls waiting for either computation or communication to complete. The paper provides no measurement of the achieved overlap efficiency in the production system.

What evidence exists in the paper. No empirical TPOT measurement is presented — not a single throughput number from the production inference system. The 1.8× throughput improvement from MTP (Section 2.3.3) is stated without specifying the baseline TPOT or the achieved TPOT. The theoretical bounds (67 TPS under 400 Gbps IB, ~1,200 TPS under idealized 900 GB/s NVLink-like fabric) are the only inference speed numbers in the paper, and they are explicitly labeled as theoretical upper bounds. The gap between theory and practice — which would reveal the true cost of MLA computation time, incomplete overlap, and other real-world overhead — is unquantified.

Mitigation status. The paper is transparent that the 67 TPS figure is a theoretical ceiling, not an achieved measurement. However, it does not provide the empirical counterpart that would allow a practitioner to assess real-world inference performance. The acknowledgment that MLA computations "typically dominate execution time" implies that the actual TPOT is substantially lower than 67 TPS, but the magnitude of the gap is unknown. This matters because the paper's hardware recommendations — particularly the call for higher-bandwidth scale-up networks to achieve "~1200 tokens per second" — are motivated by the theoretical analysis; if the actual bottleneck is MLA computation time rather than EP communication bandwidth, investing in faster interconnects would yield diminishing returns.

---

6.3 FP8 Training Accuracy Is Validated Only on Smaller Models, with No Full-Scale Comparison Presented

The assumption or constraint. The paper's FP8 mixed-precision training framework is validated through a hierarchical pipeline: "fine-grained FP8 training ablation studies on both 16B and 230B DeepSeek-V2 models before final integration" (Section 2.4). The accuracy loss threshold is "below 0.25% compared to BF16" on these smaller models. The full 671B DeepSeek-V3 training is asserted to meet the same threshold, but no FP8-vs-BF16 accuracy comparison at 671B scale is presented in this paper — the model quality benchmarks are deferred to the separate DeepSeek-V3 technical report.

The consequence. The claim that FP8 training is accuracy-preserving at 671B parameters rests on an extrapolation from smaller-scale experiments. This is a reasonable pragmatic choice — running a full BF16 baseline for a 671B MoE model on 2,048 GPUs would be prohibitively expensive — but it introduces uncertainty that the paper's validation methodology cannot resolve:

• Accumulation precision errors compound with depth: The Hopper Tensor Core's FP22 accumulation registers (13 mantissa bits, Section 3.1.1) introduce truncation error at each accumulation step. In a 16B model with perhaps 20–30 transformer layers, the total accumulation error is bounded. In a 671B model with 61 layers and substantially wider matrices, the error compounds across more accumulation operations per forward/backward pass. The 0.25% threshold measured at 230B scale may not hold at 671B.
• MoE routing amplifies quantization error: In MoE models, the dispatch and combine operations aggregate token representations that have been quantized on different GPUs. If FP8 quantization introduces per-token biases that correlate with expert assignment (e.g., certain experts receive systematically over- or under-estimated token representations), these biases can amplify during the combine reduction, creating routing-specific accuracy degradation that does not appear in dense models.
• Training dynamics differ at scale: FP8's limited dynamic range (E4M3: 448 possible values; E5M2: 256 possible values, with different exponent/mantissa tradeoffs) can cause gradient underflow for small updates or overflow for large updates at different points in training. The 230B validation may not capture the full training trajectory of the 671B model, particularly late-stage training dynamics where gradients become small.

What evidence exists in the paper. No FP8 accuracy data is presented in this paper — not in a table or figure. The 0.25% figure is a summary statistic from the ablation studies, with no specification of which metric (perplexity? downstream task accuracy? which task?) is being measured, what the absolute BF16 baseline performance is, or what variance is observed across different random seeds or training runs. The MFU numbers in Table 4 (43.73% non-causal, 38.94% causal) are computed against BF16 peak performance, not FP8 peak performance — they measure utilization of the GPU's compute units, not the throughput gain from reduced precision. The paper therefore provides no direct evidence that FP8 training achieved its intended 2× throughput improvement over BF16 at the 671B scale.

Mitigation status. The hierarchical validation methodology (16B → 230B → 671B) is a pragmatic mitigation, and the paper is explicit that it was adopted "given the prohibitive cost of exhaustive ablation on full-scale models" (Section 2.4). The separate DeepSeek-V3 technical report presumably contains the full-scale accuracy results. However, for a reader of this paper alone — particularly a hardware architect deciding whether to invest in FP8-native Tensor Core designs — the evidence is incomplete. The paper would be strengthened by at minimum a perplexity comparison at the 230B scale, presented directly, to give the reader a concrete sense of the FP8 training stability tradeoff.

---

6.4 The Node-Limited Routing Constraint's Impact on Model Quality Is Never Directly Measured

The assumption or constraint. The Node-Limited Routing strategy (Section 4.3) algorithmically constrains each token's TopK expert selection to span at most 4 physical nodes, rather than the 8 nodes that unconstrained routing might select. This constraint reduces IB communication by deduplicating traffic through NVLink forwarding, but it potentially degrades routing quality by preventing tokens from accessing the globally highest-affinity experts if those experts reside on a 5th or higher node.

The consequence. The paper provides no measurement of how binding this constraint is in practice. The reader cannot answer several critical questions:

• What fraction of tokens would naturally route to >4 nodes? If unconstrained TopK would select experts on ≥5 nodes for only a small fraction of tokens (e.g., 5%), the constraint is minimally distorting and the communication savings come essentially for free. If 30–40% of tokens naturally break the 4-node limit, the constraint is binding and the router is making systematically suboptimal expert assignments for a large fraction of the training data.
• What is the accuracy cost of the constraint? The paper's implicit argument is that DeepSeek-V3 achieves state-of-the-art performance (deferred to the technical report), therefore the constraint cannot be severely harmful. But this is a weak form of evidence — it does not establish whether the constraint costs 0.1%, 1%, or 3% in absolute accuracy relative to unconstrained routing at the same model size and training budget.
• Does the constraint affect expert specialization? If tokens are prevented from accessing certain experts due to the node limit, the experts on the excluded nodes receive fewer training tokens and may develop weaker or less distinct specializations. Over the course of full training, this feedback loop could alter the learned expert structure in ways that are not captured by a simple accuracy comparison, because the unconstrained model would have different weight values throughout.

What evidence exists in the paper. None. There is no ablation comparing constrained vs. unconstrained routing, no measurement of the fraction of tokens affected by the constraint, and no analysis of expert utilization patterns under the constraint. The paper treats the Node-Limited Routing strategy as an architectural given — it is described as a design decision, not evaluated as a tradeoff. The theoretical analysis (Section 2.3.2) quantifies the communication benefit of constraining to 4 nodes, but the accuracy cost is left entirely to the reader's inference from the overall model quality, which is not reported in this paper.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, propose a method for measuring the constraint's impact, or suggest future work on topology-aware routing algorithms that could optimize the tradeoff between communication cost and routing quality more precisely. The constraint is presented as a solved problem — the routing strategy exists, the model trains, and the results are good — without evidence that the particular choice of 4 nodes (rather than 3, 5, or an adaptive threshold) is optimal or even near-optimal.

---

6.5 The Multi-Token Prediction Throughput Claim Lacks Sufficient Empirical Detail for Independent Assessment

The assumption or constraint. The MTP module is claimed to "increase the generation TPS by 1.8x compared to the scenario without the MTP module" with an acceptance rate of "80% to 90% for predicting the second subsequent token" (Section 2.3.3). These are the only quantitative claims about MTP performance in the paper.

The consequence. Several aspects of the claim are underspecified to the point where a practitioner cannot assess whether MTP would provide similar benefits in their deployment:

• Acceptance rate for tokens beyond the second: The paper reports acceptance only for the "second subsequent token" but Figure 1 shows three MTP modules, suggesting the system can predict at least the 3rd and 4th subsequent tokens. What is the acceptance rate for tokens 3 and 4? If acceptance degrades sharply (e.g., 80% for token 2, 50% for token 3, 20% for token 4), the effective throughput gain is lower than what an 80–90% headline figure suggests.
• Dependence on sequence context: The acceptance rate of speculative decoding is known to vary substantially with context — it is typically higher for formulaic or repetitive text and lower for creative or unpredictable content. The paper does not break down acceptance by task type, sequence length, or domain. A user deploying MTP for code generation vs. conversational AI vs. mathematical reasoning would face very different effective throughput improvements.
• Interaction with batch size: Section 2.3.3 notes that MTP "increases the inference batch size, which is crucial for boosting EP computational intensity." But the 1.8× figure presumably includes both the direct throughput gain (more tokens per forward pass) and the indirect gain (better amortization of all-to-all communication from larger effective batches). Disentangling these effects is necessary to project MTP's benefit in single-request (batch size = 1) scenarios — the use case the paper emphasizes for "personal use and on-premises deployment" (Section 2.2.2). If the 1.8× gain is partially driven by batch effects, single-request throughput improvement would be lower.
• Hardware configuration: The 1.8× figure is not accompanied by specification of the GPU model, batch size, sequence length, or inference precision used for the measurement. A throughput gain measured under one hardware configuration (e.g., large batch, high-end GPU, BF16) may not transfer to another (small batch, consumer GPU, FP8).

What evidence exists in the paper. Only the two summary statistics quoted above. No figure, table, or appendix presents MTP throughput measurements disaggregated by configuration. The paper does not describe the measurement methodology (was it on the production inference system? a smaller testbed? software simulation?). The acceptance rate is given as a range (80–90%) without specifying what drives the variation. The throughput gain (1.8×) is a single number with no error bars, no breakdown by request type, and no scaling analysis (how does MTP's benefit change with model size? with sequence length? with the number of MTP modules?).

Mitigation status. Not addressed in this paper. The MTP module is described as an architectural innovation, and its performance is asserted based on "real world practice data" (Section 2.3.3) without the supporting measurements that would allow verification or reproduction. The DeepSeek-V3 technical report may contain more detailed MTP analysis, but within the scope of this paper, the throughput claims about one of the system's headline inference optimizations cannot be independently assessed.

---

6.6 The LogFMT Experience Reveals a Fundamental Tradeoff Between Representational Quality and Hardware Efficiency That the Paper Does Not Resolve

The assumption or constraint. LogFMT (Logarithmic Floating-Point Format), a custom low-precision number format, was designed, tested on 7B-parameter dense models, and found to offer "superior training accuracy compared to E4M3 or E5M2" at 8-bit width for activation quantization (simulating the MoE combine stage). At 10-bit width, it was "similar to the BF16 combine stage" (Section 3.2). Despite these representational advantages, LogFMT was abandoned because "insufficient GPU bandwidth for log/exp operations and excessive register pressure during encode/decode" caused a "substantial" overhead of 50–100% when encode/decode operations were fused with all-to-all communication (Section 3.2.1).

The consequence. The LogFMT experience exposes a structural tension that the paper's hardware recommendations do not fully address: representational quality (the ability of a number format to preserve information at low bit widths) and hardware efficiency (the speed at which the format can be encoded, decoded, and computed upon) are often in opposition, and current GPU architectures optimize for the latter at the expense of the former through their support for IEEE-like floating-point formats but not logarithmic or custom formats.

This tension has consequences beyond LogFMT:

• FP8 as a local optimum: The paper's FP8 framework succeeds because NVIDIA provides native Tensor Core support for E4M3 and E5M2, making encode/decode essentially free (it is fused into the memory load/store pipeline). But the 0.25% accuracy loss relative to BF16, while acceptable, is not zero — it represents the representational cost of forcing model weights and activations into IEEE-like formats that may not be optimal for the statistical distribution of neural network tensors. A format with better representational properties (like LogFMT) could close this gap, but only if hardware supports it natively.
• Custom formats are economically infeasible without hardware support: The paper's recommendation (Section 3.2.2) — "providing native support for compression and decompression units tailored to FP8 or custom precision formats" — acknowledges the problem but does not specify how hardware vendors should choose which formats to support. Supporting LogFMT natively would require silicon area for log/exp computation units that may not be useful for any other workload. The economic case for custom format support is unclear, especially when FP8 with fine-grained quantization (the approach the paper successfully deployed) achieves acceptable accuracy on existing hardware.
• The general case of communication compression: The LogFMT failure is not an isolated incident — it illustrates a general problem for communication compression in distributed training. Many theoretically attractive compression schemes (low-rank approximations, vector quantization, entropy coding) fail in practice because the compression/decompression overhead on GPUs swamps the bandwidth savings. The paper's own recommendation for future hardware (Section 3.2.2) acknowledges this by calling for "native support for compression and decompression units," but it does not specify which compression schemes warrant silicon investment.

What evidence exists in the paper. The LogFMT evaluation is described qualitatively in Section 3.2, with the key negative result: "when setting n=8, sharing the same bits with FP8, the LogFMT-8Bit shows superior training accuracy compared to E4M3 or E5M2. After increasing the n to 10 bits, we find it's similar to the BF16 combine stage." The overhead is described as "substantial (50%∼100%)" when fused with all-to-all communication. No quantitative accuracy measurements are presented (no perplexity, no downstream task scores, no comparison table against FP8 baselines). The analysis is confined to 7B-parameter dense models, so the representational benefit of LogFMT at the 671B MoE scale — where quantization errors in the combine stage might interact differently with expert routing — is unknown.

Mitigation status. The paper's recommendation (Section 3.2.2) for native hardware support of compression/decompression units is a forward-looking suggestion, not a solution to the immediate tradeoff. The paper does not propose an intermediate approach (e.g., using LogFMT only for the dispatch stage where FP8 is already used, accepting the encode/decode overhead in exchange for better accuracy, or training a small neural decompressor that amortizes the log/exp cost). The LogFMT work is presented as a promising direction that was abandoned due to current hardware limitations, leaving open the question of whether custom number formats are worth pursuing or whether the field should converge on fine-grained variants of IEEE-like formats (a direction NVIDIA has already taken with microscaling formats in Blackwell).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper instantiates a methodological shift in how the AI infrastructure community should approach large-scale training: hardware-model co-design is not a one-time optimization applied after the model architecture is finalized, but a continuous, bidirectional negotiation that shapes both the model architecture and the physical deployment simultaneously. The shift is from a layered model—where algorithm designers specify architectures independently, and systems engineers map them onto available hardware as efficiently as possible—to a co-optimization model where the constraints of the physical substrate (NVLink bandwidth ratios, switch port counts, Tensor Core accumulation precision, NIC encode/decode throughput) are first-class inputs to the architecture design process.

The magnitude of this shift is significant but bounded. It is not a paradigm shift in the Kuhnian sense—the individual techniques (MoE, FP8, speculative decoding, fat-tree topologies) are all established. Rather, it is a reframing of the design process itself: the paper demonstrates that a cluster of only 2,048 H800 GPUs—hardware that is deliberately bandwidth-constrained below the H100 baseline for regulatory compliance—can train a 671B-parameter state-of-the-art model at 250 GFLOPS per token (Table 2) and achieve 38.9% causal MFU (Table 4), figures that would conventionally require substantially more or substantially faster hardware. The reframing lies in the paper's assertion, validated by operational data, that these efficiency numbers are not despite the hardware constraints but because the architecture was designed around them.

Three specific ways this reframing changes the research landscape:

1. It converts hardware limitations from obstacles into design specifications. The paper's treatment of the H800's reduced NVLink bandwidth (400 GB/s vs. the H100's 900 GB/s) is instructive. A conventional approach would treat this as a deficit to be overcome—either by buying more GPUs to compensate, or by accepting proportionally lower throughput. The paper instead uses it as a specification that determines the routing algorithm: the 4:1 intra-node-to-inter-node bandwidth ratio becomes the constraint that Node-Limited Routing is designed to satisfy. This inverts the typical relationship between hardware and algorithm: the algorithm does not merely run on the hardware; it is shaped by the hardware's specific asymmetry. This design philosophy—treating hardware constraints as algorithm specifications rather than performance taxes—is transferable to any deployment with heterogeneous interconnect bandwidth, including future chiplet-based architectures, disaggregated memory systems, or edge deployments with mixed wireless and wired fabrics.

2. It establishes that the inference bottleneck is a bandwidth bottleneck at every level of the memory hierarchy, not a capacity bottleneck. The paper's theoretical TPOT analysis (Section 2.3.2) traces the inference speed ceiling to three specific bandwidth constraints: HBM bandwidth (addressed by MLA's KV cache compression, reducing memory traffic per decode step), NVLink bandwidth (addressed by Node-Limited Routing's deduplication of inter-node traffic), and IB bandwidth (addressed by the multi-plane topology's cost-efficient scaling). This diagnostic reframing—that inference speed is fundamentally limited by bandwidth, not compute—redirects research attention from FLOPs-centric optimization (more Tensor Cores, lower-precision compute) to bandwidth-centric optimization (compression, traffic engineering, topology-aware routing). The paper's central negative finding—that LogFMT, despite superior representational quality, was abandoned because GPU log/exp bandwidth was insufficient (Section 3.2.1)—reinforces this: a better number format is worthless if the hardware cannot encode and decode it at line rate.

3. It reconciles the tension between cost-efficient and state-of-the-art training. The dominant narrative in the LLM scaling community, reinforced by industry deployments with tens or hundreds of thousands of GPUs, implies that cost efficiency and model quality are in tension—you can have one or the other. The paper's operational data (Table 4: 272.8 billion tokens per day on 2,048 H800 GPUs with 38.9% causal MFU) demonstrates that this tension is not inherent but is an artifact of treating hardware as a fixed, unoptimized substrate. The 250 GFLOPS-per-token training cost (Table 2) is 6–10× lower than comparably capable dense models, yet the resulting model is state-of-the-art on multiple benchmarks (per the technical report). This finding makes cost-efficient training a falsifiable engineering discipline rather than a budgetary constraint: specific architectures, parallelism strategies, precision choices, and network topologies can be evaluated against quantitative efficiency targets (MFU, GFLOPS/token, tokens/day, cost per endpoint), and the paper provides a template for such an evaluation.

The research directions this work makes more attractive:

• Bandwidth-aware neural architecture search: If inference speed is bandwidth-limited, architectures should be evaluated by their bandwidth consumption (KV cache size, activation communication volume, expert dispatch patterns), not just their FLOP count. MLA's 4.7–7.3× KV cache compression (Table 1) suggests substantial headroom for architectures that explicitly minimize memory traffic.
• Topology-aware routing for all sparse models: Node-Limited Routing's success suggests that any model with sparse activation patterns (MoE, mixture-of-depths, dynamic sparsity) should embed knowledge of the physical network topology into its routing decisions—not as a post-hoc optimization but as a structural component of the architecture.
• Hardware support for fine-grained quantization: The paper's FP8 experience—where 1×128 tile-wise activation quantization and 128×128 block-wise weight quantization were necessary for training stability, but the dequantization overhead was a significant bottleneck (Section 3.1.1)—directly motivates the development of Tensor Cores with native group-scaled matrix multiplication, a direction NVIDIA has begun with Blackwell's microscaling format support (Section 3.1.2).
• Multi-plane network topologies for large-scale AI clusters: The empirical validation that an eight-plane two-layer fat-tree achieves performance parity with a single-plane multi-rail fat-tree at 2,048 GPUs (Table 4) and projects to 41% lower cost per endpoint at 16,384 GPUs (Table 3) makes multi-plane designs a credible alternative to three-layer fabrics for cluster operators—not just a niche cost-saving measure.

The research directions this work makes less attractive:

• Naive FP8 deployment without fine-grained quantization: The paper's finding that Hopper's FP22 accumulation registers (13 mantissa bits, Section 3.1.1) cause measurable accuracy degradation with coarse quantization (the 0.25% accuracy loss threshold could not be met without tile-wise and block-wise scaling) raises the bar for FP8 training frameworks—per-tensor quantization is insufficient at 671B scale.
• Custom number formats without hardware encode/decode support: LogFMT's abandonment (Section 3.2.1) is a cautionary tale for any proposed low-precision format that requires non-trivial conversion operations. The 50–100% overhead from log/exp operations on current GPUs is prohibitive, and the paper's recommendation for native hardware compression/decompression units makes explicit what was previously implicit: number format innovation must be co-designed with silicon.
• Uniform parallelism strategies (TP everywhere): Tensor Parallelism's explicit avoidance during training (Section 4.2) due to "inefficiency under limited NVLink bandwidth" challenges the common practice of applying TP by default to large models. For deployments with constrained intra-node bandwidth, the paper's EP + PP + DualPipe combination provides a more efficient alternative—but one that requires more complex scheduling and communication overlap.

Follow-Up Research This Work Enables

Characterizing the accuracy-vs-communication tradeoff of Node-Limited Routing through controlled ablation. The paper implements Node-Limited Routing (≤4 nodes per token) as an architectural decision but provides no measurement of its impact on model quality relative to unconstrained routing. A direct follow-up would train two smaller MoE models (at a scale where full training is affordable, e.g., 1B–7B activated parameters) on the same data, one with Node-Limited Routing and one with unconstrained TopK, and measure the accuracy delta (perplexity, downstream task accuracy) as a function of the node limit (from 1 to 8 nodes on an 8-node cluster). The key question is whether the accuracy loss from constraining to ≤4 nodes is below the threshold of practical significance—the paper implies it is, based on the full DeepSeek-V3 results, but without quantifying it. A controlled ablation would also measure which tokens are most affected by the constraint (e.g., long-tail rare tokens, domain-specific terminology) and whether expert specialization patterns diverge under the constraint. This experiment requires a cluster with the same 4:1 NVLink-to-IB bandwidth ratio as the H800 configuration to be ecologically valid.

Extending the multi-plane network validation to 4,096+ GPUs with production MoE traffic. The paper's central scaling claim—that the MPFT topology supports 16,384 GPUs with two-layer cost efficiency—is validated only at 2,048 GPUs (Table 4), a scale where a single two-layer fat-tree is sufficient. A critical follow-up is to deploy an MPFT topology at a scale where it is necessary (i.e., beyond the 2,048-endpoint limit of a single 64-port two-layer fat-tree, say 4,096 or 8,192 GPUs) and measure whether the performance parity with MRFT (Figures 5, 6; Table 4) is maintained. The key measurement is the all-to-all communication bandwidth at 4,096+ GPUs under realistic MoE dispatch/combine patterns (with DeepEP, not just synthetic NCCL benchmarks), with specific instrumentation of the NVLink forwarding load for cross-plane traffic. If NVLink forwarding becomes a bottleneck at scale (because more tokens require cross-plane forwarding as the node count increases), this would establish a practical scaling limit for multi-plane designs that the current paper cannot characterize. This experiment requires access to a cluster large enough to exceed single-plane two-layer limits, which may explain why it was not performed for this paper.

Training a lightweight difficulty predictor from the PRM score distribution to eliminate the difficulty estimation bottleneck. The paper mentions (Section 3.2) that difficulty estimation requires 2,048 samples per question, a cost that is not amortized in the reported efficiency gains. A direct follow-up would train a small neural network (e.g., a single transformer layer or even a linear classifier) that takes only the question text as input and predicts the difficulty bin directly, using the 2,048-sample PRM score distribution as training labels. The key measurement is whether the predicted difficulty bins from this lightweight classifier preserve the compute-optimal scaling behavior (Figures 4, 8 in the earlier search/revision paper) with minimal compute cost. A successful lightweight predictor would make the compute-optimal framework deployment-ready by eliminating the most expensive pre-processing step. A negative result—the lightweight predictor cannot achieve sufficient accuracy to preserve the scaling benefits—would suggest that difficulty estimation inherently requires sampling, motivating online adaptive strategies that interleave difficulty assessment with problem solving.

Combining PRM tree-search with the revision model as the proposal distribution. This direction is explicitly acknowledged as unexplored in the earlier search/revision paper (Section 8) but is equally relevant to the DeepSeek-V3 infrastructure context: given the high inference throughput enabled by MTP (1.8× over baseline) and the bandwidth optimizations (MLA, Node-Limited Routing), the compute budget available for test-time search strategies at a given latency target is substantially larger than what prior work assumed. A direct experiment would replace the base LLM as the proposal distribution in a PRM beam search with the revision model, measuring whether the revision model's higher-quality initial candidates (from sequential self-refinement) combine synergistically with the PRM's ability to guide search toward correct solutions. The key measurement is accuracy at fixed generation budget (in tokens, not FLOPs, to account for the MTP speedup) compared to search alone and revisions alone. A positive result would demonstrate that inference-time optimizations (MTP, MLA) and test-time compute strategies (search, revisions) compound rather than merely add, which has direct implications for how to allocate a total inference budget.

Measuring the real-world TPOT gap between the theoretical 67 TPS bound and production inference performance. The paper provides a theoretical upper bound of 67 tokens per second under 400 Gbps IB (Section 2.3.2) but acknowledges that "MLA computations typically dominate execution time" and that the bound assumes perfect dual micro-batch overlap. No empirical TPOT is reported. A direct measurement study would instrument the production DeepSeek-V3 inference system to report achieved TPOT as a function of request context length (1K, 4K, 16K, 64K, 128K tokens), batch size, and the presence/absence of MTP. The key diagnostic is a breakdown of per-layer latency into MLA computation time, EP dispatch communication time, EP combine communication time, and idle time (when overlap is imperfect). This breakdown would reveal which component is the true bottleneck at different operating points, answering the question: should hardware investment target faster interconnects (to raise the 67 TPS ceiling), faster HBM (to accelerate MLA), or both? The measurement requires access to the production system's internal profiling data, which the DeepSeek team has already collected (they reference "open-source profiling data" in Section 2.3.1 and have released profile-data on GitHub), making this experiment immediately tractable.

Quantifying the energy efficiency of MLA's compute-for-bandwidth trade. The paper positions MLA as a technique that trades a small amount of on-chip computation (the decompression matrix multiplications) for a large reduction in off-chip HBM traffic (4.7–7.3× KV cache compression, Table 1). This is a classic compute-for-bandwidth trade, but the energy implications are not quantified. A follow-up would measure the energy consumption of MLA-based attention during decode (using GPU power sensors or energy estimation tools) compared to GQA-based attention at the same throughput, separating dynamic energy (from the decompression GEMMs) from static energy (from HBM idle time). The key question is whether the energy cost of the decompression computation is less than the energy savings from reduced HBM traffic—if HBM accesses dominate GPU energy consumption (as is typical for memory-bound workloads), MLA should be net energy-positive even if it increases total FLOP count. This measurement would inform whether MLA-like compression should be adopted broadly in edge and mobile deployments where energy, not just throughput, is the binding constraint.

Practical Applications and Downstream Use Cases

Cost-efficient on-premises deployment of large MoE models for organizations with limited hardware budgets. The paper's finding that DeepSeek-V2 (236B parameters, 21B activated) achieves "nearly 20 tokens per second" on "PCs with AI SoC chips" (Section 2.2.2), and that KTransformers enables the full DeepSeek-V3 to run on a ~$10,000 server with a consumer GPU at similar speeds, directly enables a deployment model where organizations run state-of-the-art language models on hardware they already own or can afford without cloud dependencies. The specific mechanism is the combination of MoE architecture (37B activated parameters out of 671B total, requiring far less memory bandwidth than a dense model of comparable capability) and MLA (70 KB KV cache per token, making long-context inference feasible within consumer GPU HBM limits). For an organization processing internal documents, customer support queries, or code generation tasks, the capital expenditure of ~$10,000 for a server that can serve DeepSeek-V3 at interactive speeds represents a step-function reduction from cloud API costs or datacenter-scale deployments. The practical deployment scenario is: use the same model architecture (MLA + DeepSeekMoE) but host it locally, eliminating API latency, data privacy concerns, and per-query costs.

Low-latency MoE inference for reasoning models in RL training loops. The paper's analysis of the TPOT ceiling (67 TPS under 400 Gbps IB, Section 2.3.2) and the 1.8× throughput improvement from MTP (Section 2.3.3) directly addresses a critical bottleneck in training reasoning models (DeepSeek-R1, OpenAI o1/o3): RL algorithms like PPO, DPO, and GRPO require rapidly generating large numbers of samples from the model during training, and inference throughput is often the gating factor for how quickly RL training can proceed. A reinforcement learning training loop using DeepSeek-V3 with MTP as the inference backend would generate samples approximately 1.8× faster than without MTP, directly translating to shorter RL training wall-clock time. The prefill-decode disaggregation architecture (Section 2.3.1), which assigns prefill and decode to different EP group sizes, further optimizes this by allowing the prefill phase (which processes long prompts and is compute-bound) to use a different hardware configuration than the decode phase (which generates samples and is bandwidth-bound). For organizations training reasoning models, this means that the same 2,048-GPU cluster can support both the training computation and the inference-based sample generation within the RL loop without additional hardware provisioning.

Network cost reduction for large-scale AI cluster operators through multi-plane topologies. The paper's cost analysis (Table 3) shows that a multi-plane two-layer fat-tree topology achieves a cost per endpoint of $4,390—identical to a two-layer fat-tree at its maximum 2,048-endpoint scale—compared to$7,500 for a three-layer fat-tree, a 41% reduction in networking cost per GPU. For a cluster operator planning a 16,384-GPU deployment, the networking infrastructure savings from adopting a multi-plane design are approximately $51 million ($7,500 − $4,390 =$3,110 per endpoint × 16,384 endpoints). This is a substantial fraction of the total cluster cost, and it comes with no measured performance penalty at the 2,048-GPU scale (Table 4: MFU within 0.05 percentage points). The practical deployment scenario is: when building an AI cluster beyond the scaling limit of a single two-layer fat-tree (roughly 4,096 endpoints with 64-port switches), evaluate the multi-plane design as a cost-reducing alternative to the default three-layer topology, with the caveat that cross-plane traffic management (via NCCL PXN and Node-Limited Routing) must be part of the software stack from the beginning.

FP8 training as a default for MoE models at scale, with a validated recipe. The paper's FP8 mixed-precision training framework—tile-wise 1×128 activation quantization, block-wise 128×128 weight quantization, high-precision accumulation, and hierarchical validation from 16B to 230B to 671B scale (Section 2.4)—provides a concrete, validated recipe for FP8 training of large MoE models. The key practical takeaway is that FP8 training at 671B parameters is feasible with less than 0.25% accuracy loss relative to BF16, provided the quantization granularity is aligned with Tensor Core tile dimensions and the accumulation precision limitations of the specific GPU generation (Hopper's FP22 registers with 13 mantissa bits, Section 3.1.1) are understood and accounted for. For any team training a large MoE model on Hopper-generation GPUs, the paper's FP8 framework (open-sourced in DeepGEMM) can be adopted directly, with the hierarchical validation methodology serving as a template for verifying accuracy preservation on their specific model and data distribution. The 0.25% accuracy loss threshold provides a quantitative target for FP8 training quality that was not previously established in the literature for models at this scale.