ArXiv: 2602.21548

🎯 Pitch

In production agentic LLM systems, the storage network is the hidden bottleneck—under a 95% KV-Cache hit rate, all prefill-side storage NICs saturate while decode-side NICs sit completely idle, halving effective bandwidth. DualPath routes KV-Cache loads through idle decode engines and transfers the data over RDMA via the compute network, boosting throughput by up to 1.96× without violating latency SLOs.

1. Executive Summary

This paper introduces DualPath, an LLM inference system that breaks the storage bandwidth bottleneck in agentic workloads by enabling dual-path KV-Cache loading — in addition to the conventional storage-to-prefill path, KV-Cache can be loaded into decode engines and transferred to prefill engines via RDMA over the compute network (storage-to-decode path) — combined with a CNIC-centric traffic manager that isolates KV-Cache transfers from latency-sensitive model communication using InfiniBand virtual lanes and a global scheduler that dynamically balances load across both prefill and decode engines. Evaluated on three models — DeepSeek-V3.2 660B, a 27B downscaled variant, and Qwen2.5-32B — using production agentic RL training traces with KV-Cache hit rates above 95%, DualPath improves offline inference throughput by up to 1.87× and online serving throughput by an average of 1.96× compared to an unmodified baseline, with the scheduling algorithm alone improving storage NIC load balance from 1.53 to 1.18. The bottleneck-free analysis establishes that dual-path loading can fully saturate all storage NICs without introducing compute-NIC or DRAM bottlenecks only when the prefill-to-decode node ratio falls within the derived bounds — for an 8-GPU node with one storage NIC per machine, this range is 1/7 ≤ P/D ≤ 7/2, covering most practical configurations.

2. Context and Motivation

The Core Problem: The Storage I/O Wall in Agentic LLM Inference

The fundamental problem this paper tackles arises from a collision between two trends in modern LLM deployment: the rapid adoption of agentic workloads that generate extreme-length multi-turn conversations, and the architectural limitations of prefill-decode disaggregated inference systems that were designed for shorter, simpler interaction patterns.

To understand why this collision matters, we need to establish what happens physically during agentic inference. In a multi-turn agent trajectory — say, a coding agent debugging a repository over 157 turns, the average in the authors' production traces (Section 3) — the model accumulates context across turns. By the end of a trajectory, the context can reach hundreds of thousands of tokens. Critically, when the next turn begins, the model does not recompute the attention for all those previous tokens from scratch. Instead, it reuses the previously computed KV-Cache — the key-value pairs produced during attention computation. The paper reports that in representative coding tasks, the KV-Cache hit rate exceeds 95%, meaning that only about 5% of tokens in each turn actually require fresh attention computation (the newly appended tool output or user input). The other 95% simply need their stored KV-Cache entries loaded from persistent storage into GPU memory.

This is where the bottleneck crystallizes. Because KV-Cache loading displaces computation as the dominant activity, the system transitions from being compute-bound to being I/O-bound. The paper quantifies this via the cache-compute ratio: for DeepSeek-V3.2 with an average append length of 429 tokens, the system must load approximately 22 GB of KV-Cache for every PFLOP of attention computation (Table 1). This means the storage network, not the GPU's tensor cores, dictates how fast the system can process requests. If the storage NIC cannot deliver the KV-Cache quickly enough, GPUs sit idle waiting for data.

Why the Problem Is Important: The Shift to Agentic Workloads

This is not a niche academic concern. The paper documents a fundamental shift in how LLMs are being used in production. Traditional chatbot interactions involve a human sending a prompt, the model generating a response, and a brief back-and-forth. In contrast, agentic workloads involve the model interacting with an external environment — invoking tools, executing code, reading terminal output — over dozens or hundreds of turns. Each individual turn may be short (the appended tool output might be only a few hundred tokens), but the accumulated context grows without bound. The paper's production traces show a mean of 157 rounds per trajectory and average context lengths of tens of thousands of tokens.

This pattern is central to reinforcement learning (RL) training of agents, where a rollout phase requires the model to generate thousands of multi-step trajectories. During RL rollouts, HBM is further constrained because optimizer states and reward model parameters must be offloaded to host DRAM, leaving even less space for KV-Cache caching. This makes fast, reliable access to external KV-Cache storage not just desirable but mandatory.

Where Existing Architectures Fall Short

The paper identifies a specific architectural deficiency in modern PD-disaggregated inference systems. Let's unpack the standard architecture first, since the bottleneck only becomes visible in the details.

The standard PD-disaggregated architecture. In this design, popularized by systems like DistServe and Splitwise, the inference pipeline is split across two sets of GPUs: prefill engines (PEs) that process incoming prompts (computing attention over the full context and populating the KV-Cache), and decode engines (DEs) that perform autoregressive token generation using the precomputed KV-Cache. To handle multi-turn conversations, the KV-Cache is persisted to distributed storage after each turn, and reloaded when the next turn begins.

Within this architecture, layerwise prefill — introduced by systems like LayerKV and PrefillOnly — is a second critical optimization. Traditional prefill stores the entire batch's KV-Cache in HBM simultaneously, which limits batch size when contexts are long (HBM capacity is the ceiling). Layerwise prefill exploits the observation that each transformer layer needs only its own layer's KV-Cache, not the whole model's. By loading and freeing KV-Cache per layer, the GPU can hold a much larger token batch (approximately nlayern_{layer} times larger), dramatically improving throughput.

The bottleneck emerges in the data path. In this architecture, all KV-Cache loading happens through a single, asymmetric path: the storage KV-Cache is read by the prefill engine's storage NIC (SNIC), loaded into PE HBM, processed, and then transferred to the decode engine's DRAM buffer via the compute network (RDMA over the compute NIC, or CNIC). This means all storage I/O traffic concentrates on the prefill side. The prefill engines' SNICs become saturated, while the decode engines' SNICs — which are largely idle during this phase — contribute nothing to KV-Cache loading. Figure 1 (left) illustrates this lopsided traffic pattern.

The paper quantifies the severity of this imbalance. The storage bandwidth per machine is fixed — typically 400 Gbps for a single storage NIC shared by all 8 GPUs on a node. With layerwise prefill, prefill GPUs can process tokens quickly, but the storage NIC cannot deliver KV-Cache fast enough to keep them busy. The decode side, meanwhile, has idle storage bandwidth that goes entirely unused during prefill operations. This is fundamentally wasteful: the aggregate storage bandwidth of the cluster is being underutilized because only a subset of nodes (the prefill nodes) are allowed to fetch KV-Cache from storage.

Why Prior Approaches Don't Solve This

The paper surveys existing solutions and identifies why each falls short for the specific challenge of agentic workloads:

Distributed DRAM caching (Mooncake). Mooncake builds a distributed DRAM pool to cache KV-Cache across machines, providing fast access to frequently used cache entries. However, this approach fails in two regimes critical for agentic workloads. First, in memory-constrained scenarios like RL rollout, DRAM is already occupied by training state offloaded from HBM, leaving insufficient space for KV-Cache. Second, even when DRAM is available, the working set of agentic workloads is enormous. The paper estimates (Section 8.2) that at a moderate serving rate, the KV-Cache working set for DeepSeek-V3.2 ranges from 69 GB to 681 GB. In production, where tool call latencies stretch the total job completion time, the working set can grow by r2r^2 times, quickly exceeding available DRAM. SSD-based storage becomes the only economically viable option, and Mooncake's DRAM-centric design does not address SSD bandwidth bottlenecks.

Reducing the amount of data (HCache) or optimizing the I/O path (TARDIS, Phoenix). These systems optimize how KV-Cache is retrieved — through better data structures, GPU Direct Storage, or refactored I/O stacks — but they all operate on the same single data path: storage-to-prefill. They make that one path faster or more efficient, but they do not fundamentally change the imbalance whereby prefill NICs are saturated while decode NICs are idle. The bottleneck is not just the speed of the path, but the asymmetric allocation of traffic across available paths.

The fundamental inefficiency is structural, not technological. The paper's key observation is that existing systems have baked in an assumption: KV-Cache loading is a prefill-side operation. This assumption made sense for single-turn or short-context workloads where the prefill phase dominated and decode was a separate downstream phase. But in agentic workloads, where KV-Cache loading dominates the timeline and decode engines are underloaded during prefill, this structural assumption becomes the bottleneck itself. The decode engines have storage NICs, DRAM buffers, and high-bandwidth compute network connections to prefill engines — all of which could participate in KV-Cache loading if the architecture permitted it.

The paper contextualizes the bottleneck as part of a broader hardware evolution that is unfavorable for I/O-bound workloads. Figure 3 (left) shows the ratio of network I/O bandwidth to GPU compute (FLOPS) across NVIDIA GPU generations from Ampere to Blackwell. This ratio has decreased by 14.4×, meaning that each new GPU generation delivers proportionally less network bandwidth per unit of compute. The authors argue this is driving the field toward a "memory and communication wall": GPUs can compute faster than networks can feed them data, particularly under the high-cache-hit, long-context patterns of agentic workloads.

Simultaneously, HBM capacity growth has not kept pace with the growing context lengths demanded by agentic applications. Even with layerwise prefill, the GPU must hold one layer's KV-Cache for the entire batch in HBM, and if the batch's total tokens exceed HBM capacity, batch sizes must shrink, reducing GPU utilization further. The combination — insufficient network bandwidth, constrained HBM, and massive KV-Cache volumes — creates a compounding bottleneck that existing architectures cannot resolve through incremental improvements to a single data path.

How DualPath Positions Itself

The paper does not propose a fundamentally new storage technology, a new attention mechanism, or a new compression scheme. Instead, it identifies a structural inefficiency in how existing architectures allocate network resources and proposes a structural fix: allow KV-Cache to be loaded through multiple paths, not just one. The core insight is that KV-Cache loading "does not have to be prefill-centric" (Section 1). The decode engines possess storage bandwidth, DRAM buffers, and high-speed compute network connections to prefill engines. By routing some KV-Cache reads through decode engines — reading from storage into DE DRAM, then forwarding to PE HBM via RDMA — the system can aggregate the storage bandwidth of all engines in the cluster, not just the prefill engines.

This positions DualPath not as a competitor to systems like Mooncake, TARDIS, or HCache, but as a complementary layer that addresses a dimension they do not touch: the multi-path utilization of available I/O resources. The paper explicitly notes that DualPath can be combined with a DRAM cache tier (Section 9), but the performance gain from such combination is marginal because the core bottleneck is the aggregate utilization of storage NICs, which no prior system addresses.

The contribution is therefore as much a re-framing of the problem as it is a specific technical solution. By recognizing that storage I/O is a shared, schedulable resource across the entire cluster — not an asymmetric burden borne exclusively by prefill nodes — the paper establishes a new optimization dimension for LLM inference systems. The subsequent technical contributions (CNIC-centric traffic isolation, adaptive scheduling) are mechanisms that make this re-framing practical without compromising the latency-sensitive communication patterns that model execution requires.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

DualPath is a system-level modification to the inference stack of a prefill-decode disaggregated LLM serving architecture — what the authors implement as a set of controllers and data paths on top of their in-house inference framework — that changes how the massive KV-Cache is moved from persistent storage into GPU memory during multi-turn agentic workloads. It solves the problem that existing systems load all KV-Cache exclusively through the prefill engines' storage network interface cards (SNICs), leaving the decode engines' SNICs idle and creating a bandwidth bottleneck that starves the GPUs of data; DualPath's solution shape is to introduce a second loading path — storage → decode engine → compute network → prefill engine — and a scheduler that dynamically balances KV-Cache traffic across both paths, transforming storage I/O from a single-bottleneck resource into a globally pooled and schedulable capacity.

3.2 Big-picture architecture (diagram in words)

The DualPath system consists of five major components layered onto a standard PD-disaggregated inference framework:

  • Inference Engines (PEs and DEs): GPUs partitioned into prefill engines (PEs) that process incoming prompts and compute attention for new tokens, and decode engines (DEs) that autoregressively generate output tokens. Each engine manages one GPU.
  • Dual-Path KV-Cache Loading Mechanism (Section 4.1): The core architectural change. Two data paths exist for loading KV-Cache from persistent storage — the conventional PE Read Path (storage → PE DRAM → PE HBM → DE DRAM) and the novel DE Read Path (storage → DE DRAM → PE HBM → DE DRAM). A small DRAM buffer on each PE and DE (PE buffer and DE buffer) serves as the staging area.
  • CNIC-Centric Traffic Manager (Section 5): Resides on each engine. Enforces that all data movement into and out of a GPU — including local H2D/D2H copies — must traverse the GPU's paired compute NIC (CNIC) using GPUDirect RDMA, so that the CNIC's hardware QoS mechanisms (InfiniBand virtual lanes with weighted round-robin arbitration) can isolate KV-Cache transfer traffic from latency-sensitive model execution collectives (AllToAll, AllReduce).
  • Request Scheduler (Section 6): A centralized controller that assigns incoming requests to (PE, DE) pairs, selects which loading path (PE Read or DE Read) each request uses, and performs intra-engine batch packing to balance GPU execution time across data-parallel attention groups.
  • Persistent KV-Cache Storage (3FS): A distributed SSD-based file system (the open-source 3FS from DeepSeek) with no internal DRAM cache, capable of saturating the per-node 400 Gbps storage NIC bandwidth. KV-Cache is stored using a trie structure where each node corresponds to a Full Block (all layers for a block of tokens).

Information flows as follows: a request arrives at the scheduler → the scheduler selects a PE and DE based on token load and disk queue length, and picks either the PE Read Path or DE Read Path based on which side has the shorter storage read queue → KV-Cache is read from 3FS into either PE buffer or DE buffer → during layerwise prefill, one layer's KV-Cache at a time is transferred (via CNIC RDMA) into PE HBM for attention computation → the newly computed KV-Cache for appended tokens is merged with the loaded hit-token KV-Cache and transferred to the DE buffer → the decode phase begins with a final H2D transfer into DE HBM.

3.3 Roadmap for the deep dive

  • First, the dual-path loading mechanism (Section 4.1): the PE Read Path and DE Read Path data flows in detail, the staging buffer layout, and the block format designs (Full Blocks vs. Layer Blocks). This is the core architectural innovation — without it, the scheduler has nothing to balance.
  • Second, the bottleneck-free analysis (Section 4.2): the mathematical derivation showing that dual-path loading can saturate all storage NICs without introducing CNIC or DRAM bottlenecks, and the P/D ratio bounds that guarantee this. This is the theoretical justification for why the architecture is sound.
  • Third, the CNIC-centric traffic manager (Section 5): why existing GPU data transfer technologies (GPUDirect Storage, CUDA copy engine) interfere with model execution, how routing everything through the CNIC enables hardware QoS isolation via InfiniBand virtual lanes, and the CNIC-assisted H2D/D2H mechanism that replaces cudaMemcpyAsync.
  • Fourth, the adaptive request scheduler (Section 6): inter-engine scheduling (PE assignment, DE assignment, KV-Cache read path selection) and intra-engine scheduling (compute-quota-based batch packing for attention layers). This is where the system makes online decisions about load balancing along two dimensions simultaneously — NIC traffic and GPU utilization.
  • Fifth, key configuration constants and their roles: the short reading queue threshold $\alpha$ (3 seconds worth of token reads), the unfinished token upper limit $\beta$ (5 seconds worth of GPU processing), the compute quota (300 ms), and how they are profiled in advance.

3.4 Detailed, sentence-based technical breakdown

This is primarily a systems design and implementation paper whose core idea is that KV-Cache loading in PD-disaggregated architectures should not be prefill-centric, but should instead use a dual-path design that aggregates the storage bandwidth of all engines in the cluster, combined with traffic isolation and load-aware scheduling to make this practical without degrading model execution latency.

The PE Read Path (Storage → Prefill Engine)

The PE Read Path is the conventional data flow that existing PD-disaggregated systems use, but DualPath implements it with specific optimizations for the layerwise prefill setting. The path performs the following sequence of data movements, illustrated in Figure 4(a):

Step 1–2: Storage to PE Buffer. The KV-Cache of all hit tokens for a given request is read from persistent storage (3FS) into the PE Buffer — a DRAM region on the prefill node allocated specifically for this staging purpose. The storage read uses the prefill node's SNIC and the io_uring-like kernel-bypass interface that DualPath employs for low-latency file I/O.

Step 3–4: PE Buffer to PE HBM, layer by layer. Before the computation of each attention layer begins, the KV-Cache entries corresponding to only that layer are transferred from the PE Buffer into PE HBM. This is the layerwise streaming pattern: rather than loading the entire multi-layer KV-Cache into HBM at once (which would quickly exhaust HBM capacity and limit batch size), the system loads one layer's worth of KV-Cache into HBM, computes attention for that layer, then frees that layer's KV-Cache from HBM before loading the next layer's. The transfer from PE Buffer to PE HBM is performed via the CNIC (not a direct PCIe copy), as mandated by the CNIC-centric traffic design described in Section 5.

Step 5–7: PE HBM to DE Buffer, merged with computed cache. After the prefill computation for a layer completes, the PE now holds the KV-Cache for both the hit tokens (loaded from storage) and the miss tokens (newly computed during this prefill forward pass). These are transferred together — as a complete layer-specific KV-Cache — to the DE Buffer on the decode node via RDMA over the compute network. The transfer uses the CNIC and the high-bandwidth RDMA fabric. This process (steps 3–7) repeats for all $n_{layer}$ transformer layers.

Step 8–9: DE Buffer to DE HBM. After all layers have been transferred, the DE Buffer on the decode node now contains the complete prompt KV-Cache (all layers, all tokens). The decode phase begins by allocating HBM on the decode GPU and performing a host-to-device (H2D) transfer from the DE Buffer into DE HBM, again via the CNIC. The CPU memory (DE Buffer) is then released to reclaim DRAM for subsequent requests.

Why PE Buffer staging is used: The PE Buffer serves as an elasticity buffer between the storage read rate (determined by the SNIC bandwidth) and the layerwise consumption rate (determined by the GPU's compute speed). By staging the entire request's KV-Cache in DRAM first, the system decouples storage I/O timing from GPU execution timing — the storage read can complete at its own pace, and the GPU can then consume the data layer-by-layer without ever waiting for a storage read to finish mid-computation.

The DE Read Path (Storage → Decode Engine → Prefill Engine)

The DE Read Path is the novel contribution that distinguishes DualPath from all prior systems. It reroutes KV-Cache loading through the decode engines, exploiting their otherwise-idle SNICs and the high-bandwidth compute network to deliver KV-Cache to prefill engines via an indirect route. The path performs the following sequence, illustrated in Figure 4(b):

Step 1–2: Storage to DE Buffer. The KV-Cache of all hit tokens is read from persistent storage into the DE Buffer — a DRAM region on the decode node. The storage read uses the decode node's SNIC. This is the critical departure from existing architectures: rather than having the prefill node pull all KV-Cache from storage, the decode node does the pulling.

Step 3–5: DE Buffer to PE HBM, layer by layer. During prefill execution on the PE, for each attention layer, the PE issues an RDMA read to pull that layer's KV-Cache from the DE Buffer directly into PE HBM. This transfer traverses the compute network (CNIC → switch → CNIC) and is designed to overlap with the PE's computation — while the PE processes one layer, the next layer's KV-Cache can be in-flight. As with the PE Read Path, the KV-Cache transfer is layerwise: only one layer's data occupies PE HBM at a time.

After computation: merging. After a layer's attention computation completes on the PE, only the KV-Cache of the miss tokens (the newly appended tokens that required fresh computation) needs to be transferred to the DE Buffer on the decode node. The hit tokens' KV-Cache already resides in the DE Buffer (it was the source of the RDMA read). The miss-token KV-Cache is transferred via RDMA and merged with the existing hit-token KV-Cache in the DE Buffer, forming the complete prompt KV-Cache for that layer.

Step 6–7: DE Buffer to DE HBM. Identical to the PE Read Path's decode phase: after all layers are merged, the DE allocates HBM and performs an H2D transfer of the complete prompt KV-Cache from the DE Buffer into DE HBM via the CNIC. The DE Buffer is then released.

Why the DE Read Path makes physical sense: The decode engines are underutilized during the prefill phase of multi-turn agentic workloads — they are not generating tokens yet (the prefill must complete first) and their SNICs are largely idle. The DE Read Path converts this idle storage bandwidth into productive KV-Cache loading capacity. The cost is additional traffic on the compute network (the RDMA transfers from DE to PE), but as the bottleneck-free analysis demonstrates, the compute network has sufficient headroom under typical P/D ratios to absorb this traffic without congestion.

Block Layout Design: Full Blocks vs. Layer Blocks

The layerwise prefill execution pattern creates a tension with storage efficiency. Storage systems perform best with large, contiguous reads, but layerwise prefill needs small, layer-specific chunks. DualPath resolves this with two distinct block layouts that serve different roles in the data pipeline.

Full Block layout. A Full Block is a byte tensor with shape [n_layer, block_size, bytes_per_layer_per_token]. It contains the KV-Cache for all layers for a contiguous block of tokens. Full Blocks are the unit of storage I/O — whenever data is read from or written to 3FS, it moves in Full Blocks. This maximizes storage throughput by keeping reads large and contiguous.

Layer Block layout. A Layer Block is a byte tensor with shape [1, block_size, bytes_per_layer_per_token]. It contains the KV-Cache for a single layer for a contiguous block of tokens. Layer Blocks are the unit of GPU-side data movement — the PE Read Path and DE Read Path transfer data between buffers and HBM one Layer Block at a time.

Conversion between layouts. The design ensures that assembling n_layer Layer Blocks simply concatenates them into a Full Block — no manual memory layout conversion or data shuffling is needed. During storage writes (decode phase), as soon as a full block of tokens (e.g., 64 tokens) is accumulated in DE HBM, it is written to 3FS as a Full Block. During storage reads, a Full Block is read into the buffer, and individual Layer Blocks are extracted by slicing along the layer dimension.

Why two block types: The alternative — using only Full Blocks for all transfers — would waste HBM and increase latency because the GPU would need to hold the entire multi-layer KV-Cache for the batch simultaneously, defeating the purpose of layerwise prefill. The alternative — using only Layer Blocks for storage — would fragment storage I/O into many small reads, dramatically reducing throughput. The two-block design is a classic systems tradeoff: optimize storage access patterns (Full Blocks) while respecting GPU memory constraints (Layer Blocks).

Bottleneck-Free Analysis: Deriving the Safe P/D Ratio Bounds

The dual-path architecture introduces additional traffic on the compute network and on DRAM — the DE Read Path adds RDMA transfers from DE to PE, and both paths use staging buffers that consume DRAM bandwidth. The bottleneck-free analysis in Section 4.2 formally characterizes when this additional traffic can be absorbed without creating new bottlenecks. The analysis assumes a well-configured PCIe topology (each GPU-NIC pair under the same PCIe switch), load-balanced task scheduling, no congestion on the compute network (thanks to the traffic isolation in Section 5), and fully utilized storage read bandwidth.

Notation. Let:

  • $P$ = number of prefill nodes
  • $D$ = number of decode nodes
  • $g$ = number of GPUs per node (8 for the experimental setup)
  • $B$ = bandwidth of a single compute NIC (400 Gbps for the experimental setup)
  • $s \times B$ = storage bandwidth per machine (where $s = 1$ for the experimental setup, meaning one 400 Gbps SNIC shared by all $g$ GPUs on the node)
  • $M$ = memory bandwidth per machine (approximately 500 GB/s for the experimental setup)

Traffic per PE-DE pair. Under load-balanced scheduling, storage NIC bandwidth is evenly shared across all requests. The traffic per PE-DE pair for the PE Read Path (all steps in Figure 4(a)) is:

Tp=B⋅sD⋅g2T_p = \frac{B \cdot s}{D \cdot g^2}

For the DE Read Path (Figure 4(b)), it is:

Tc=B⋅sP⋅g2T_c = \frac{B \cdot s}{P \cdot g^2}

where $T_p$ and $T_c$ represent the per-pair KV-Cache throughput (in bytes per second) on each path. The total link traffic at any NIC or DRAM controller is the sum over all pairs that use that link.

PE CNIC bandwidth analysis. The PE CNIC handles two types of traffic in the read direction: receiving KV-Cache from the PE Buffer (PE path step 3, which uses the CNIC-assisted H2D) and receiving KV-Cache from the DE Buffer via RDMA (DE path step 3–5). The total read-direction traffic across all pairs is:

2×Tp×D⋅g=2⋅B⋅sg≤B2 \times T_p \times D \cdot g = \frac{2 \cdot B \cdot s}{g} \leq B

The inequality holds because $s \leq g$ always holds in practice ($s = 1$ and $g = 8$ in the experimental setup). This means the PE CNIC's read direction is always bottleneck-free regardless of the P/D ratio.

What it computes: the sum of all KV-Cache data flowing into a PE CNIC from both the local PE Buffer (H2D) and remote DE Buffers (RDMA reads), expressed as a fraction of the CNIC's total bandwidth $B$.

Why this form: the factor of 2 accounts for the fact that both PE path and DE path traffic enter the PE CNIC. The division by $g$ reflects that the per-machine storage bandwidth $s \cdot B$ is shared across $g$ GPUs, each with its own CNIC of bandwidth $B$. Since $s/g \leq 1$, the total traffic is at most $2B/g$, which is at most $B$ when $g \geq 2$.

PE CNIC write-direction analysis. The write direction carries KV-Cache from PE HBM to DE Buffer (PE path step 5–7) and the miss-token KV-Cache to DE Buffer (DE path step post-computation). The total write traffic is:

(Tp+Tc)×D⋅g=B⋅sg×(1+DP)≤B(T_p + T_c) \times D \cdot g = \frac{B \cdot s}{g} \times \left(1 + \frac{D}{P}\right) \leq B

This gives the first lower bound on the P/D ratio:

P/D≥sg−sP/D \geq \frac{s}{g - s}

For $g = 8, s = 1$, this evaluates to $P/D \geq 1/7$.

What it computes: the total KV-Cache data flowing out of PE CNICs toward DE Buffers, including both the hit-token KV-Cache from the PE path and the miss-token updates from both paths.

Why this form: the term $(1 + D/P)$ captures the asymmetry: more decode nodes relative to prefill nodes means each prefill node must send to proportionally more decode nodes, increasing per-PE write pressure. The bound $s/(g-s)$ is the minimum P/D ratio needed to keep this pressure below the CNIC's capacity.

DE CNIC read-direction analysis. The DE CNIC receives hit-token KV-Cache from the DE Buffer (DE path H2D, step 3) and from the PE (PE path steps 7/9 and DE path miss-token transfer). The total read traffic is:

(Tp+2⋅Tc)×P⋅g=sg×(PD+2)×B≤B(T_p + 2 \cdot T_c) \times P \cdot g = \frac{s}{g} \times \left(\frac{P}{D} + 2\right) \times B \leq B

This gives the first upper bound on the P/D ratio:

P/D≤g−2ssP/D \leq \frac{g - 2s}{s}

For $g = 8, s = 1$, this evaluates to $P/D \leq 6$.

What it computes: the sum of all KV-Cache data arriving at DE CNICs — from local DE Buffer H2D (one copy), from PE-to-DE transfers (PE path), and from PE-to-DE miss-token updates (DE path, which counts as an additional read since the DE must receive it).

Why this form: the factor of 2 on $T_c$ accounts for the DE path requiring both an RDMA read from storage into DE Buffer (which traverses the CNIC in the H2D direction) and the subsequent PE-to-DE merge transfer.

DE CNIC write-direction analysis. The total write traffic is:

(2⋅Tp+Tc)×P⋅g≤B(2 \cdot T_p + T_c) \times P \cdot g \leq B

This gives the second upper bound:

P/D≤g−s2sP/D \leq \frac{g - s}{2s}

For $g = 8, s = 1$, this evaluates to $P/D \leq 7/2 = 3.5$.

What it computes: the KV-Cache data leaving DE CNICs toward DE HBM (the final H2D before decode begins), accounting for both PE path data (which passes through the DE Buffer) and DE path data (which originates in the DE Buffer).

Why this form: the factor of 2 on $T_p$ reflects that PE path data traverses the DE CNIC twice — once when received from the PE (write to DE Buffer) and once during the final H2D (write to DE HBM). The DE path data traverses only once (final H2D).

DE DRAM bandwidth analysis. DRAM is half-duplex, so read and write pressures sum. The DE DRAM pressure is:

(3+2⋅P/D)⋅B⋅s≤M(3 + 2 \cdot P/D) \cdot B \cdot s \leq M

Rearranging for P/D:

P/D≤M/(B⋅s)−32P/D \leq \frac{M/(B \cdot s) - 3}{2}

For $M \approx 500$ GB/s and $B \cdot s \approx 50$ GB/s (400 Gbps / 8 bits per byte = 50 GB/s), the right-hand side evaluates to approximately $(10 - 3)/2 = 3.5$.

What it computes: the total DRAM bandwidth consumed on a decode node by all KV-Cache staging operations — reads from SNIC into DE Buffer, writes from PE, reads for H2D, writes for merge operations.

Why this form: the constant term 3 accounts for the three independent DRAM operations that occur per byte of KV-Cache on the decode node (storage read into buffer, receive from PE, final H2D). The term $2 \cdot P/D$ captures the additional DRAM pressure from the DE Read Path specifically, which increases with the P/D ratio because more prefill nodes means more incoming RDMA writes to the DE Buffer.

Combined bottleneck-free range. Putting all constraints together:

sg−s≤P/D≤min⁡{g−2ss,g−s2s,M/(B⋅s)−32}\frac{s}{g - s} \leq P/D \leq \min\left\{\frac{g - 2s}{s}, \frac{g - s}{2s}, \frac{M/(B \cdot s) - 3}{2}\right\}

For $g = 8, s = 1$ and the profiled $M \approx 500$ GB/s, $B \cdot s \approx 50$ GB/s:

17≤P/D≤min⁡{6,3.5,3.5}=72\frac{1}{7} \leq P/D \leq \min\{6, 3.5, 3.5\} = \frac{7}{2}

What this range means operationally: For any P/D ratio between approximately 0.143 and 3.5 (e.g., 1P7D through 7P2D), DualPath can fully saturate all storage NICs without introducing a compute-NIC or DRAM bottleneck. The prefill CNICs, decode CNICs, and decode DRAM will all operate below their bandwidth limits, meaning KV-Cache loading is the sole bottleneck and DualPath extracts the maximum possible throughput from the available hardware.

Why the analysis matters beyond the numbers: The paper uses this analysis to establish that the architecture is not merely an empirical heuristic but is provably bottleneck-free under assumptions that hold in practice. The derived range covers most practical P/D ratios (the paper's experiments use 1P1D, 1P2D, and 2P1D, all well within the safe zone), which means the system designer can configure the cluster without worrying about unintended congestion. The analysis also reveals which resource becomes the bottleneck when the ratio falls outside the safe zone — CNIC bandwidth if P/D is too low (prefill nodes are overwhelmed) or DRAM bandwidth if P/D is too high (decode nodes cannot keep up with buffer operations).

CNIC-Centric Traffic Manager: The Isolation Problem

Modern GPU systems offer several mechanisms for moving data between storage, host DRAM, and GPU HBM: GPUDirect Storage (GDS) reads directly from NVMe SSDs into GPU HBM over PCIe, and the CUDA copy engine performs cudaMemcpyAsync to move data between host DRAM and GPU HBM over PCIe. Both are fast and well-optimized. However, the paper identifies a critical limitation that makes both unsuitable for DualPath's dual-path loading: they cannot be isolated from the latency-sensitive collective communication operations that occur during model execution.

The interference mechanism. During LLM inference, GPUs engage in frequent collective communication operations — AllToAll for expert parallelism (when routing tokens to MoE experts), ReduceScatter and AllGather for tensor or context parallelism. These collectives occur in rapid, sub-millisecond-level bursts. If KV-Cache transfer traffic (whether via GDS or CUDA copy engine) happens to coincide with a collective operation, it contends for the same PCIe bandwidth. Since PCIe lacks QoS mechanisms (prior to recent proposals), the KV-Cache traffic can delay the collective completion, which in turn delays the entire inference forward pass. Worse, because the burst timing is unpredictable, software-based traffic shaping — trying to interleave KV-Cache transfers between bursts — is impractically difficult to implement with sufficient precision.

The CNIC-centric solution. DualPath's central traffic management rule is: all data traffic entering or leaving a GPU must go through the GPU's paired CNIC using GPUDirect RDMA. This means:

For KV-Cache loading: The KV-Cache is first read from 3FS into host DRAM using the SNIC (a purely CPU-side operation). Then, instead of using cudaMemcpyAsync or GDS to move it to HBM, the system submits an RDMA Write work request to the GPU's paired CNIC. The NIC performs the DMA transfer from host DRAM to GPU HBM over the PCIe bus — but critically, this transfer is now visible to the CNIC's hardware QoS scheduler.

For KV-Cache persistence: During decode, newly generated KV-Cache is first transferred from GPU HBM to host DRAM via the CNIC (an RDMA Read or Write initiated by the CNIC), and then written to 3FS storage over the SNIC.

Why this works: The CNIC is the same NIC that handles model execution collectives (AllToAll, etc.). By routing all traffic through the CNIC, the system can leverage the CNIC's hardware QoS capabilities — specifically, InfiniBand virtual lanes (VLs) — to enforce strict priority differentiation at the hardware level, without any software involvement in the fast path.

Traffic Isolation via InfiniBand Virtual Lanes

InfiniBand supports up to 15 virtual lanes (VLs), which are independent hardware packet queues on each switch port and NIC. VLs are scheduled by a two-level weighted round-robin (WRR) arbiter: a high-priority arbiter and a low-priority arbiter. The paper configures this mechanism as follows.

VL assignment. All model inference communication traffic (AllToAll, ReduceScatter, AllGather, etc.) is assigned to a dedicated high-priority VL (VL 0, 1, or 3 in the configuration). All other traffic — including KV-Cache transfers, H2D/D2H via CNIC, and any background RDMA operations — is mapped to a separate low-priority VL (VL 2).

Arbiter configuration. The VL arbiters on all switches and NICs are configured with:

  • qos_high_limit = 240: This setting controls how much bandwidth the high-priority arbiter can consume before yielding to the low-priority arbiter. A value of 240 (out of 255) means the high-priority arbiter gets approximately 240/255 ≈ 94% of the total bandwidth when it has traffic, with the remaining bandwidth going to the low-priority arbiter.
  • qos_vlarb_high = 0:192, 1:192, 2:0, 3:192: Within the high-priority arbiter, VLs 0, 1, and 3 each get a weight of 192, while VL 2 gets weight 0 (meaning VL 2 traffic is completely excluded from the high-priority round — it only gets scheduled when the high-priority arbiter yields).
  • qos_vlarb_low = 0:192, 1:192, 2:64, 3:192: Within the low-priority arbiter, VL 2 gets weight 64 while the other VLs get weight 192. This ensures that even when the low-priority arbiter is active, KV-Cache traffic gets only a fraction of the remaining bandwidth, preventing it from starving background operations on other VLs.

Operational behavior. During model execution, when a collective communication burst occurs, the collective traffic on the high-priority VLs immediately preempts any in-progress KV-Cache transfer on the low-priority VL. The high-priority arbiter consumes up to 94% of the link bandwidth for the duration of the burst. When the burst ends, the low-priority arbiter resumes, and the KV-Cache transfer continues using the now-idle bandwidth. This guarantees that model execution latency is virtually unaffected by KV-Cache transfers, with the VL arbiter providing sub-microsecond switching between traffic classes.

Generalizability to RoCE and other fabrics. The paper notes that the same design principle extends to other interconnect technologies. On RoCE (RDMA over Converged Ethernet), Traffic Classes (TCs) and Differentiated Services Code Point (DSCP) markings serve the same role as VLs — packets are classified by DSCP into TCs, each backed by a dedicated hardware queue, and weighted scheduling enforces bandwidth allocation. The key requirement is that the fabric supports at least two distinct traffic classes with hardware-enforced priority. The paper cites UnifiedBus and Ultra Ethernet as emerging technologies that converge on similar QoS mechanisms.

CNIC-Assisted H2D/D2H: Replacing cudaMemcpyAsync

The CNIC-centric rule requires that even local host-to-device (H2D) and device-to-host (D2H) copies go through the CNIC, rather than using the CUDA copy engine's direct PCIe path. The paper provides a performance justification for this apparently circuitous approach.

The CUDA copy engine overhead problem. Submitting a single H2D copy operation via cudaMemcpyAsync incurs a kernel-driver round-trip latency of approximately 5–7 µs per submission. The paper attributes this to the closed-source CUDA driver's internal overhead (they "failed to further break down this overhead due to the closed-source nature of CUDA driver"). For layerwise prefill with many small Layer Block transfers (one per layer), this per-submission overhead becomes significant — with 30–60 layers per model, submitting one cudaMemcpyAsync per layer would add 150–420 µs of pure submission overhead, which eats into the tight timing budget of the attention computation.

The CNIC-assisted mechanism. Instead of cudaMemcpyAsync, DualPath submits an RDMA Write work request to the GPU's paired CNIC. The work request instructs the NIC to perform a DMA transfer from host DRAM to GPU HBM (for H2D) or from GPU HBM to host DRAM (for D2H). Submitting an RDMA work request involves only a few mmio writes to NIC registers in user space — no kernel transition, no CUDA driver involvement — and takes approximately 1 µs per submission.

Doorbell batching for amortization. The per-submission overhead can be further reduced by doorbell batching, a well-known RDMA optimization (Kalia et al., 2016). Instead of ringing the NIC's doorbell register after each individual work request submission (which triggers the NIC to start processing), the system submits multiple work requests and rings the doorbell only once after a batch. This amortizes the doorbell write overhead across many transfers. For layerwise prefill, this means all Layer Block transfers for a batch can be submitted together and processed by the NIC in a single batch, reducing the per-transfer overhead to near zero.

Why this design outperforms GPUDirect Storage for DualPath's use case. GPUDirect Storage bypasses the host CPU entirely, reading directly from NVMe into GPU HBM. While this is optimal for throughput in a single-path design, it provides no mechanism for traffic isolation — the GDS transfer contends for PCIe bandwidth with model execution collectives, and neither the GPU nor the NVMe drive participates in the CNIC's VL arbitration. The CNIC-assisted approach accepts the "detour" through host DRAM and the NIC as the price of controllability: by forcing the data to pass through the CNIC's hardware QoS scheduler, the system can guarantee that KV-Cache traffic never degrades inference latency.

Adaptive Request Scheduler: Two-Level Design

The scheduler is responsible for three decisions that must balance two dimensions simultaneously: (1) which PE and DE should handle each request (inter-engine scheduling), (2) which KV-Cache read path (PE Read or DE Read) each request should use, and (3) which requests should be batched together in each forward pass (intra-engine scheduling). The two dimensions of balance are NIC traffic (storage and compute network load) and GPU utilization (computation time balance across data-parallel peers).

The scheduling architecture is two-level. Inter-engine scheduling assigns requests to (PE, DE) pairs and selects the read path; it runs when an engine group fetches new work (a pull-based model to avoid scheduler bottleneck). Intra-engine scheduling determines batch composition and runs on each PE before a forward pass.

Inter-Engine Scheduling: PE Assignment

PE scheduling is invoked when a PE group's Leader Engine (rank 0 GPU in the group) initiates a fetch request to the central scheduler. All requests in the scheduler's waiting queue are processed in FIFO order during a single fetch call. The algorithm (Algorithm 1 in the paper) classifies all PEs in the group into three categories based on their reported load metrics.

Load metrics reported by each engine. Each engine $e$ reports three values when fetching:

  • $seq_e$: the number of requests assigned to engine $e$ that have not yet completed. This measures request count imbalance, which correlates with scheduling queue depth.
  • $tok_e$: the total token count across all $seq_e$ unfinished requests. This is the primary load metric because GPU load, disk read load, and network load are all strongly correlated with token count.
  • $read\_q_{n(e)}$: the disk reading queue length (number of bytes or tokens pending) of the node $n(e)$ that engine $e$ resides on. This is reported per-node, not per-engine, because the SNIC is shared by all 8 GPUs on the node.

Constants. Two profiled constants govern the categorization:

  • $\alpha$, the short reading queue threshold: set to the number of tokens that can be read from storage during 3 seconds at the profiled SNIC bandwidth. Engines on nodes with $read\_q_{n(e)} \leq \alpha$ are considered to have short disk queues, meaning their SNICs risk becoming underutilized if not assigned new read work soon.
  • $\beta$, the unfinished token upper limit: set to the number of tokens one GPU can process for 5 seconds at the profiled prefill throughput. Engines with $tok_e > \beta$ are considered overloaded — they have enough work queued to keep them busy for at least 5 seconds, so assigning them more requests would only increase queuing latency without improving throughput.

Engine categorization. Each PE is classified into one of three sets:

  • $C_1$ (overloaded): $tok_e > \beta$. These engines are not assigned new requests during this fetch call — they already have sufficient work.
  • $C_2$ (short disk queue, not overloaded): $read\_q_{n(e)} \leq \alpha$ and $tok_e \leq \beta$. These are the highest-priority targets because their SNICs have low utilization and they have GPU capacity. Assignments to $C_2$ engines prevent storage NIC underutilization.
  • $C_3$ (long disk queue, not overloaded): $read\_q_{n(e)} > \alpha$ and $tok_e \leq \beta$. These engines have GPU capacity but their SNICs are already saturated with pending reads. They are lower priority than $C_2$ because assigning them more storage reads would increase queueing delay at the SNIC.

Assignment policy. For each request in the waiting queue (FIFO order), the scheduler selects the PE with the minimum $tok_e$ in $C_2$ if $C_2$ is non-empty. If $C_2$ is empty, it selects the PE with the minimum $tok_e$ in $C_3$. If both $C_2$ and $C_3$ are empty (all PEs are overloaded), the fetch call terminates and only the already-assigned requests are returned to the Leader Engine. After assignment, the selected PE's $tok_e$ is updated by adding the new request's token count, and the request is removed from the waiting queue.

Why minimum $tok_e$ within category: This is a greedy load-balancing heuristic. Since token count is the primary proxy for all resource pressures (GPU compute, NIC bandwidth, DRAM usage), minimizing the maximum $tok_e$ across engines equalizes load. Within $C_2$, all engines have short disk queues, so the tiebreaker is purely token balance. The preference for $C_2$ over $C_3$ ensures that storage NICs are kept busy — an engine with a short disk queue risks having its SNIC go idle if not given new read work, wasting aggregate bandwidth.

Inter-Engine Scheduling: DE Assignment (Two-Phase)

DE scheduling is more complex because it spans multiple engine groups and must also respect HBM capacity constraints. It does not preserve global FIFO ordering (unlike PE scheduling), because decode engines with different HBM availability may accept requests out of order.

Phase 1: Group-level assignment. There is a global waiting queue shared across all DE groups, plus a private queue per DE group. When a DE group fetches, the group-level scheduler drains the global queue and assigns each request to the group whose total $\sum_{e \in group} tok_e$ (summed across all engines in that group) is minimum. This balances token count — and thus NIC and GPU load — across groups. After assignment, requests enter the assigned group's private queue.

Phase 2: Within-group scheduling. Within a DE group, the scheduler must assign requests to individual DEs while respecting each DE's remaining HBM capacity. The algorithm:

  1. Compute the upper bound of requests that could fit. Calculate the sum of remaining HBM across all DEs in the group. Traverse the private queue from the head, accumulating requests until the sum of their token counts exceeds the total remaining HBM (assuming zero fragmentation). This set of requests is $R$ — the theoretical maximum that could be assigned if HBM usage were perfectly efficient.

  2. Compute the high-token threshold $Z$. This threshold separates "high-token" DEs from others:

Z=1.05×(∑r∈Rlenr+∑e∈Etoke∣E∣)Z = 1.05 \times \left(\frac{\sum_{r \in R} len_r + \sum_{e \in E} tok_e}{|E|}\right)

where $len_r$ is the token count of request $r$, $tok_e$ is the current token count of engine $e$, and $|E|$ is the number of DEs in the group.

What it computes: the average token count per DE if all requests in $R$ were assigned and load were perfectly balanced, multiplied by 1.05 (a 5% tolerance margin). DEs whose post-assignment token count would exceed $Z$ are considered "high-token" and deprioritized for new assignments.

Why this form: the 1.05 multiplier provides a small tolerance band that prevents the scheduler from oscillating between assigning and not-assigning to a DE that is near the threshold. Without the multiplier, a DE exactly at the average would be classified as high-token for one assignment and normal for the next, creating unstable scheduling decisions.

  1. Assign requests one by one from the private queue head. For each request, among DEs with sufficient remaining HBM:
  • Partition candidates into $C_{high}$ (DEs where $tok_e + len(r) > Z$) and $C_{norm}$ (the rest).
  • Prefer $C_{norm}$ over $C_{high}$ to keep token counts balanced — DEs already above $Z$ have higher GPU and NIC pressure.
  • Within $C_{norm}$, select the DE with minimum $seq_e$ (request count), to balance scheduling queue depth.
  • Within $C_{high}$ (if $C_{norm}$ is empty), select the DE with minimum $tok_e$, to reduce the risk of HBM exhaustion and preemption.

If no DE has sufficient HBM for the current request, the fetch call terminates and already-assigned requests are returned.

KV-Cache Read Path Selection. After a (PE, DE) pair is selected for a request, the scheduler chooses whether to use the PE Read Path or DE Read Path. The rule is simple: read from the side with the shorter disk reading queue. This is implemented by comparing $read\_q_{n(pe)}$ (the PE's node storage queue length) with $read\_q_{n(de)}$ (the DE's node storage queue length) and selecting the path whose source node has the smaller queue. The paper notes that splitting a single request's KV-Cache read across both paths (reading part from the PE side and part from the DE side) might be even better, but is left as future work.

Intra-Engine Scheduling: Compute-Quota-Based Batch Packing

Only PEs require intra-engine scheduling because DEs always place all ready requests into their forward batch (decode is memory-bound and throughput is limited by HBM bandwidth, not by batch composition). For PEs, the challenge is that under data parallelism for attention layers (common for MLA models like DeepSeek-V3.2), different GPUs in the same expert-parallel group may serve different sets of requests. If the attention layer execution time varies across these GPUs, the faster GPUs must wait at the synchronization barrier before entering the FFN stage, creating idle GPU bubbles.

Layer time estimation. Each request in a forward batch is described by a pair (cached, bsz):

  • $cached$: the number of tokens whose KV-Cache is already available (either from storage hits or from previous forward passes in a chunked prefill scenario).
  • $bsz$: the number of tokens requiring fresh KV-Cache computation in this forward batch.

From these pairs, the scheduler computes the total theoretical computation for the attention layer — which depends on both the number of cached tokens (which must be attended to) and the number of new tokens (which require both attention computation and KV-Cache population). The relationship between theoretical computation and wall-clock execution time is hardware- and parallel-configuration-dependent, so it is fitted in advance through profiling — a standard approach in LLM serving systems (the paper cites Sarathi-Serve and PrefillOnly as prior work using similar profiling-based latency estimation).

The compute quota. A predefined upper bound on per-request attention layer execution time, called the compute quota, is set to 300 ms for all DualPath and Oracle baselines. This value is chosen to balance two concerns: (1) it is short enough that the latency of any single forward pass is bounded, preventing head-of-line blocking for newly arriving requests, and (2) it is long enough to amortize the fixed overhead of kernel launches across a meaningful amount of computation.

Batching algorithm. The scheduler maintains requests for a PE in FIFO order. It adds requests to the forward batch one by one, updating the predicted attention layer execution time after each addition. The decision rule is:

  • If adding the next request (with its full $bsz$) would keep the predicted execution time at or below the compute quota, the request is added in its entirety.
  • If adding the next request would exceed the compute quota, the scheduler performs a binary search on $bsz$ to find a reduced $bsz'$ such that (cached, bsz') fits within the remaining quota. The request is then partially processed in this forward batch (chunked prefill), with the remainder deferred to a subsequent batch.

Why binary search on $bsz$: The relationship between $bsz$ and execution time is (approximately) monotonic and predictable from the profiling model. Binary search finds the largest feasible $bsz'$ in $O(\log bsz)$ steps, which is fast enough for online scheduling. The alternative — rounding $bsz'$ to a fixed chunk size — would leave unused compute quota on the table or require engineering a set of predefined chunk sizes.

Intra-engine load balance. Under data parallelism for attention, each GPU in a parallel group independently runs this batching algorithm on its assigned subset of requests. Because different GPUs may get requests with different (cached, bsz) profiles, their predicted execution times can diverge. The compute quota acts as a unifying constraint: all GPUs cap their batch at the same 300 ms budget, which bounds the maximum divergence. The paper reports (Figure 14) that this mechanism maintains the Max/Avg ratio of attention execution time across GPUs as low as 1.06 during the first 5% of a task's duration — meaning the slowest GPU is at most 6% slower than the average, minimizing synchronization bubbles.

KV-Cache Persistence During Decode

During the decode phase, each DE generates tokens autoregressively. The new tokens' KV-Cache must be persisted to storage to enable reuse in subsequent turns. DualPath uses a block-based persistence strategy:

  • As tokens are generated, their KV-Cache accumulates in DE HBM.
  • When a full block of tokens (e.g., 64 tokens, corresponding to the block_size parameter) has been accumulated, the KV-Cache for that block is immediately written to 3FS as a Full Block.
  • The write path is: DE HBM → CNIC-assisted D2H into DE DRAM → SNIC write to 3FS storage backend.

This just-in-time persistence ensures that even if the agent trajectory extends over many turns, the system's storage working set grows incrementally rather than requiring a large batched write at the end of each turn.

Summary of Key Design Decisions and Their Justifications

  • Dual-path loading over single-path optimization: Tuning the single storage-to-prefill path (via GDS, I/O stack refactoring, or quantization) cannot eliminate the fundamental asymmetry where decode-side SNICs are idle. Dual-path is the only approach that aggregates bandwidth across all nodes, making it a structural fix rather than an incremental optimization.
  • CNIC-centric traffic management over GPUDirect Storage or CUDA copy engine: The primary constraint is not raw throughput but isolatability — KV-Cache traffic must not interfere with sub-millisecond model collectives. Routing through the CNIC is the only practical mechanism to leverage hardware QoS (InfiniBand VLs) for strict priority enforcement, even though it introduces an extra DRAM staging step.
  • CNIC-assisted H2D/D2H over cudaMemcpyAsync: The per-submission overhead difference (1 µs for RDMA work request vs. 5–7 µs for cudaMemcpyAsync) matters because layerwise prefill generates many small transfers (one per layer per request). Doorbell batching further amortizes this overhead. The CUDA driver's closed-source nature makes further optimization of cudaMemcpyAsync impossible.
  • $\alpha$ and $\beta$ as profiled constants over adaptive thresholds: These values represent physical time budgets (3 seconds of SNIC bandwidth, 5 seconds of GPU processing) that are determined by hardware capabilities and profiled once. They are not workload-dependent and do not require online tuning. The 3-second and 5-second choices balance responsiveness (shorter windows react faster to load changes) against stability (longer windows prevent oscillation).
  • FIFO with load-aware skipping over strict FIFO for PE scheduling: A pure FIFO scheduler would assign consecutive requests to the same PE if it happens to be the least loaded, ignoring that other PEs might be underutilized. The three-category classification with $C_1$ (overloaded) exclusion prevents head-of-line blocking where a PE with a long queue continues to receive requests while idle PEs with short disk queues ($C_2$) starve.
  • Two-phase DE scheduling with group-level then within-group balancing: Decentralizing entirely to per-engine decisions would cause load imbalance across groups. Centralizing entirely to the global scheduler would make the scheduler a bottleneck (the paper reports less than 10 CPU cores used by the scheduler even at 1,152-GPU scale, in Table 3, confirming the two-level design's efficiency).
  • Compute quota of 300 ms: This value is chosen to keep individual forward-pass latencies bounded while still allowing enough work per batch to amortize kernel launch overhead. The paper does not explore sensitivity to this value, suggesting it was determined empirically from the profiled hardware characteristics.

4. Key Insights and Innovations

Innovation 1: Redefining KV-Cache Loading from a Node-Local Operation to a Cluster-Wide, Schedulable Resource

The dominant assumption in PD-disaggregated inference systems — from DistServe to Splitwise to Mooncake — has been that KV-Cache loading is a prefill-side responsibility. This assumption is so deeply embedded that it shapes every downstream design decision: storage is attached to prefill nodes, I/O optimization targets the prefill SNIC path, and decode nodes are treated purely as consumers of the prefill-computed KV-Cache. Prior work on KV-Cache I/O optimization (TARDIS, Phoenix, HCache, Strata) all operates within this single-path framing — they make the storage-to-prefill pipeline faster, leaner, or more efficient, but they never question who should be doing the loading.

DualPath's most fundamental intellectual move is to reject this framing entirely. The paper reframes KV-Cache loading as a cluster-wide resource allocation problem: storage I/O is a pool of bandwidth distributed across all nodes (prefill and decode), and the system's job is to schedule traffic across all available paths to maximize aggregate throughput. This is not an incremental optimization of the existing path — it is a category shift from "how do we make this one pipe faster?" to "how do we utilize all the pipes we already have?"

What makes this reframing non-obvious is that it cuts against the physical intuition of the PD-disaggregated architecture. In the standard design, KV-Cache flows naturally from storage to prefill (where it's needed for attention computation) to decode (where it's needed for generation). Introducing a detour — storage → decode → prefill — seems physically wasteful: data travels farther, traverses an extra network hop, and consumes compute network bandwidth that could otherwise be used for model execution. The paper's bottleneck-free analysis (Section 4.2) is the theoretical argument that makes this counterintuitive design provably sound: under reasonable P/D ratios, the compute network has sufficient headroom to absorb the extra traffic without congestion, and the aggregate storage bandwidth gain (utilizing all SNICs instead of only prefill SNICs) outweighs the extra hop cost.

The significance of this reframing extends beyond the specific dual-path mechanism. It establishes a new optimization dimension for LLM serving systems: I/O path diversity. Just as distributed systems use multiple network paths for fault tolerance and load balancing, and storage systems use striping across disks for throughput, LLM inference can use multiple KV-Cache loading paths to aggregate bandwidth. This dimension is entirely absent from prior work, which treats KV-Cache retrieval as a fixed, single-source operation. The paper's demonstration that simply changing where data enters the compute fabric — without changing storage hardware, network topology, or model architecture — can yield up to 1.87× throughput improvement (Figure 7) is empirical validation that this dimension matters.

The distinction between this and prior "distributed caching" approaches (Mooncake, TokenLake) is instructive. Those systems distribute KV-Cache storage across nodes but still load through a single path into prefill engines. DualPath keeps storage centralized (in 3FS SSDs) but distributes the loading paths. These are orthogonal axes — a system could combine both — and the paper's contribution is identifying that path diversity is a first-class optimization target independent of cache placement.

Innovation 2: CNIC-Centric Traffic Isolation as a General Principle for Coexisting Inference and I/O Traffic

Prior work on LLM inference systems has treated traffic isolation as a solved problem through physical network separation. The standard AI data center architecture (documented in Section 2.3 and in prior work like Zhao et al., 2025a) physically isolates the compute network (east-west, for GPU-to-GPU collectives) from the storage network (north-south, for data access) on separate NICs and separate switch fabrics. This separation is justified by the need to prevent storage I/O from interfering with the sub-millisecond collective communication bursts that are critical to inference latency. Under this architecture, KV-Cache traffic naturally stays on the storage network, and model execution traffic stays on the compute network — isolation is enforced by the physical topology.

DualPath's dual-path loading breaks this clean separation by design. The DE Read Path injects KV-Cache traffic onto the compute network, and even the PE Read Path's H2D/D2H operations (if done via CNIC) create compute-network traffic that was previously storage-network-only. The paper's critical insight is that physical separation is not the only way to achieve isolation — and, crucially, that the compute network's hardware QoS mechanisms (InfiniBand virtual lanes, RoCE traffic classes) can provide logical isolation that is sufficient for inference workloads, without requiring separate physical fabrics for each traffic type.

This is not a novel observation at the networking layer — InfiniBand VLs and DSCP-based QoS have existed for decades — but it is a novel application within the LLM inference systems context. Prior systems assumed that mixing KV-Cache traffic with model execution traffic on the same fabric was inherently dangerous, so they avoided it entirely. DualPath demonstrates that with proper VL configuration (high-priority VLs for model collectives, low-priority VLs for KV-Cache, weighted round-robin with ~94% bandwidth reserved for high priority), the interference is negligible. The empirical evidence is in Figures 10 and 12: DualPath's TTFT (time to first token) and TTST (time to second token) are comparable to the Basic baseline, and the TTFT breakdown in Figure 12 (left) shows that DualPath's scheduling and allocation times are not inflated despite the additional compute-network traffic.

The paper further argues — and provides microbenchmark evidence — that existing GPU data transfer mechanisms (GPUDirect Storage and cudaMemcpyAsync) are incompatible with this logical isolation approach because they bypass the CNIC's QoS scheduler. GDS transfers data directly from NVMe to GPU HBM over PCIe, invisible to the CNIC's VL arbiter. cudaMemcpyAsync uses the CUDA copy engine, which also operates outside the CNIC's QoS domain. The CNIC-assisted H2D/D2H design — routing all GPU data movement through the CNIC via RDMA — is therefore not merely a performance optimization (it is marginally slower than direct PCIe copies on a per-byte basis) but a correctness requirement for the isolation guarantee. This is a subtle but important distinction: the "detour" through the CNIC is the price of controllability, not an attempt to outperform direct transfers on raw throughput.

The generalization argument (Section 5.1) — extending to RoCE via DSCP, to Ultra Ethernet, to UnifiedBus — positions this not as an InfiniBand-specific trick but as a design principle for future inference fabrics: any interconnect that supports at least two hardware traffic classes with weighted scheduling can support mixed inference and I/O traffic without physical separation. This has implications for data center network design: if logical isolation is sufficient, the physical separation between compute and storage fabrics can be relaxed or eliminated, simplifying cabling and switch topology at scale.

Innovation 3: P/D Ratio as a Formal Bottleneck-Free Regime with Derived Bounds

The concept of a prefill-to-decode ratio (P/D ratio) is not new — PD-disaggregated systems have always had to choose how many GPUs to allocate to each phase. Prior work typically treats this as an empirical tuning parameter: run experiments at different ratios, measure throughput and latency, and pick the best one. There is no theoretical framework for predicting which ratios will work and which resource will become the bottleneck at a given ratio.

The paper's bottleneck-free analysis (Section 4.2) provides exactly such a framework. For a given hardware configuration (GPUs per node $g$, storage NIC bandwidth relative to compute NIC bandwidth $s$, DRAM bandwidth $M$), the analysis derives closed-form bounds on the P/D ratio within which dual-path loading saturates all storage NICs without introducing compute-NIC or DRAM bottlenecks:

sg−s≤P/D≤min⁡{g−2ss,g−s2s,M/(B⋅s)−32}\frac{s}{g-s} \leq P/D \leq \min\left\{\frac{g-2s}{s}, \frac{g-s}{2s}, \frac{M/(B \cdot s) - 3}{2}\right\}

What makes this a genuine innovation rather than a straightforward application of queueing theory is that it captures the interaction of three resources (storage NICs, compute NICs, DRAM) under two data paths. The bounds are not obvious a priori: one might guess that adding a second loading path would always help (more paths = more bandwidth), but the analysis reveals that this is only true within a specific P/D range. Outside this range, the DE Read Path creates more congestion on the compute network or DRAM than it relieves on the storage network — the cure becomes worse than the disease.

The lower bound $P/D \geq s/(g-s)$ has a clean physical interpretation: there must be enough prefill nodes to absorb the storage bandwidth without saturating their compute NICs. If P/D is too low (too few prefill nodes relative to decode nodes), each prefill node's CNIC is overwhelmed by the combined traffic of its own storage reads and the RDMA transfers to decode nodes. For the experimental setup ($g=8, s=1$), this gives $P/D \geq 1/7$ — surprisingly permissive, allowing configurations as extreme as 1 prefill node for every 7 decode nodes.

The upper bounds are more complex because they involve competition between different resources. The tightest bound for the experimental setup comes from the DE CNIC and DE DRAM constraints, both yielding $P/D \leq 7/2$. The fact that two independent analyses (CNIC bandwidth and DRAM bandwidth) converge on the same bound is not a coincidence — it reflects that both resources are stressed by the same underlying traffic pattern (PE-to-DE RDMA writes and H2D transfers) and that the hardware ratios ($B$ for CNIC, $M$ for DRAM) happen to align in a way that makes neither the singular bottleneck.

The empirical confirmation of this analysis comes from Figure 8, where the authors sweep P/D ratios of 1P1D, 2P1D, and 1P2D (all within the 1/7 to 7/2 safe range) and show that DualPath maintains substantial gains at all three configurations. The observation that Basic 1P1D ≈ Basic 1P2D (both limited by prefill-side storage bandwidth) and DualPath 1P1D ≈ Basic 2P1D (DualPath's dual-path compensates for having fewer prefill nodes) is a direct empirical validation of the theoretical claim that storage bandwidth is the dominant bottleneck and that dual-path loading effectively pools it.

The practical significance of this analysis is that it transforms P/D ratio selection from an empirical tuning exercise into a design-time decision with guaranteed properties. A system architect can plug in their hardware specs, compute the safe range, and know that any P/D ratio within that range will not introduce new bottlenecks. This is a modest theoretical contribution — it's resource allocation analysis, not a new mathematical technique — but it fills a genuine gap in the LLM serving literature, which has largely treated P/D ratio as a black-box hyperparameter.

Innovation 4: Load Balancing as a Dual-Objective Problem Spanning Network and Compute Resources

Load balancing is a well-studied problem in distributed systems, but prior work on LLM inference scheduling has typically optimized a single dimension: token count (e.g., Sarathi-Serve balances sequence lengths across GPUs) or request count (e.g., Orca balances queue depth). The paper's key diagnostic observation is that in the dual-path setting, two dimensions of balance matter simultaneously and cannot be collapsed into one.

The first dimension is NIC traffic balance: because DualPath uses both the PE Read Path and DE Read Path, storage I/O is distributed across all nodes' SNICs. If the scheduler greedily assigns requests to the least-loaded GPU without considering which SNIC handles the storage read, it can inadvertently concentrate reads on a few nodes' SNICs while leaving others idle. This would recreate the very imbalance that DualPath was designed to eliminate — just shifted from a structural imbalance (only PEs read) to a scheduling-induced imbalance (only some nodes' SNICs are used heavily).

The second dimension is GPU utilization balance: under data parallelism for attention (common for MLA models like DeepSeek-V3.2), different GPUs in the same parallel group may serve different subsets of requests. If their attention layer execution times diverge, faster GPUs idle at the synchronization barrier. This is measured by the Max/Avg ratio of attention execution time — a value of 1.0 means perfect balance, and values above 1.0 represent GPU bubbles.

The paper's scheduler addresses both dimensions through distinct mechanisms that operate at different scheduling levels. The inter-engine scheduler's PE assignment algorithm balances NIC traffic by preferring engines on nodes with short disk reading queues ($read\_q \leq \alpha$, the $C_2$ category) — this directly targets SNIC utilization balance. The intra-engine scheduler's compute-quota-based batch packing balances GPU utilization by capping per-GPU attention execution time at 300 ms — this indirectly bounds the Max/Avg ratio.

The evidence that this dual-objective approach matters comes from the ablation study (Figure 12 and Figures 13–14). Adding the scheduling algorithm on top of layerwise prefill and dual-path loading reduces JCT by an additional 7.4% beyond what those components achieve alone (from 38.2% improvement to 45.6% improvement over Basic). The load balance metrics in Figure 13 show that the scheduling algorithm improves storage NIC balance from 1.53 (round-robin) to 1.18 — close to the ideal of 1.0. Figure 14 shows the attention execution time Max/Avg ratio maintained at 1.06 during the high-load phase.

This is an incremental but practically important innovation. The idea of balancing multiple resources is not new, but the specific identification of which resources matter for dual-path loading, and the concrete scheduling mechanisms that balance them, are contributions that enable the architectural innovation (dual-path loading) to realize its theoretical potential in practice. Without the scheduler, the dual-path mechanism alone would suffer from load imbalance that erodes much of its gain — as evidenced by the 7.4% JCT gap between DualPath without scheduling and DualPath with scheduling.

The paper's distinction between "I/O load balance" and "computation load balance" as separate scheduling objectives also provides a conceptual template for future inference schedulers. As LLM serving architectures become more complex (more tiers of storage, more paths for data movement, more parallelism dimensions), the number of resources that must be simultaneously balanced will grow. The DualPath scheduler demonstrates that a two-level design (inter-engine for network balance, intra-engine for compute balance) can effectively decouple these objectives, allowing each to be optimized with relatively simple heuristics.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The evaluation uses three agent trace datasets collected from the authors' production agentic RL training workloads, with varying maximum context lengths (MaxLen). Each dataset contains 500 trajectories. The statistics are summarized in Table 2: the average interaction turns per trajectory range from approximately 30 to 157, average appended tokens per turn are in the hundreds (e.g., 429 for the representative coding task), average generated tokens per turn are comparable in magnitude, and average total tokens per trajectory reach tens of thousands. The KV-Cache hit rate across these traces is reported as typically ≥95%, with one representative trace showing 98.7%. The traces are replayed in experiments: each agent trajectory is a sequence of turns where the prompt is the concatenated context plus new appended tokens, and the model generates the corresponding number of tokens recorded in the trace.

  • Base models. Three models are evaluated, spanning different architectures and scales: (1) DeepSeek-V3.2 660B (denoted as DS 660B), a Mixture-of-Experts model with DeepSeek Sparse Attention, using the publicly released HuggingFace checkpoint; (2) a 27B downscaled version of DS 660B (denoted as DS 27B), an internal experimental model with similar architecture; and (3) Qwen2.5-32B (denoted as Qwen 32B), a dense model with Grouped-Query Attention (GQA), using the publicly released checkpoint. DS 660B represents a production-scale MoE model with optimized KV-Cache size (via MLA), DS 27B is a smaller-scale variant that enables faster experimental turnaround, and Qwen 32B represents the dense model regime where KV-Cache sizes are significantly larger and the I/O bottleneck is proportionally more severe.

  • Metrics. For offline inference (batch processing, as in the rollout phase of RL training), the primary metric is job completion time (JCT) — the wall-clock time until all requests in the batch have finished. Lower JCT indicates higher throughput (requests processed per unit time). For online serving, three latency metrics are measured: TTFT (Time to First Token), TTST (Time to the Second Token), and TPOT (Time Per Output Token, i.e., the average inter-token latency during decoding). The online serving SLO is defined as TTFT ≤ 4 seconds and TPOT ≤ 50 ms. Additionally, throughput under online serving is measured in agents per second (APS) — the arrival rate at which the system can sustain the SLO.

  • Baselines. Three baselines are compared:

    • SGL(MC): SGLang (commit 19089aa) with HiCache enabled, Mooncake Store as the KV-Cache storage backend, 3FS as the underlying file system, and Mooncake Transfer Engine for prefill-decode disaggregation. This represents the state-of-the-art open-source serving system for multi-turn workloads. SGL(MC) could not be run for DS 27B because SGLang lacks support for this downscaled model variant.
    • Basic: The authors' unmodified internal inference framework (the same codebase that DualPath is built on), which uses standard PD-disaggregated inference with layerwise prefill and single-path KV-Cache loading (storage-to-prefill only). This is the primary baseline for measuring DualPath's improvements, as it controls for implementation differences. The paper emphasizes that comparing DualPath directly to SGL(MC) is "unfair due to implementation differences," so the reported gains are always relative to Basic.
    • Oracle: Based on DualPath, but with all I/O eliminated — disk reads, D2H and H2D transfers, and inter-PD KV-Cache transfers are bypassed entirely. This represents the theoretical performance upper bound assuming zero I/O overhead, quantifying the headroom that remains after DualPath's optimizations.

  • Generation budget / compute accounting. The paper does not use a "generation budget" in the sense of a fixed per-request compute allocation. Instead, the system processes requests as they arrive in a batch setting, and throughput is determined by the physical hardware configuration (number of GPUs, P/D ratio, parallelism strategy) and the workload characteristics (context lengths, append lengths, batch sizes). Hardware resources are fixed per experiment: for DS 660B, the default configuration is 2 prefill nodes and 4 decode nodes (2P4D); for Qwen 32B, 1P2D; for DS 27B, 1P1D, 2P1D, or 1P2D depending on the experiment. Each node has 8 NVIDIA Hopper GPUs, eight 400 Gbps compute NICs (CNICs), and one 400 Gbps storage NIC (SNIC). DS models use expert parallelism (EP) and data parallelism (DP); Qwen 32B uses DP only in DualPath (SGL(MC) uses tensor parallelism TP=8 for Qwen 32B since DP attention is not supported in SGLang for this model). The compute quota for intra-engine scheduling is set to 300 ms for all DualPath and Oracle baselines.

  • Cross-validation / statistical protocol. No formal cross-validation or statistical testing is reported. The offline inference experiments measure JCT as a single run for each configuration (batch size, MaxLen, model). The online serving experiments are terminated when either TTFT exceeds 4 seconds or the system reaches steady state, defined as TTFT variation within a 150-second sliding window remaining below 5% compared to that 30 minutes prior. The TPOT and TTST figures omit data points exceeding the SLO threshold. The paper reports fluctuations (shadows) in the online serving latency plots (Figure 10) representing variation in the last 150 seconds before experiment termination.

Main Quantitative Results

Offline Batch Inference: Throughput Scaling with Batch Size and Context Length

The headline result for offline inference (Figure 7) is that DualPath improves throughput over Basic by up to 1.87× on DS 660B, up to 1.78× on DS 27B, and demonstrates performance approaching Oracle on DS 660B, indicating that KV-Cache I/O is largely eliminated as the bottleneck.

DS 660B (top row of Figure 7). At the largest configuration (1024 agents, 64K MaxLen), DualPath achieves a JCT that is 1.87× faster than Basic. At smaller scales (256 agents, 16K MaxLen), the improvement narrows. This is expected: with fewer agents and shorter contexts, the aggregate KV-Cache volume is smaller, so the storage bandwidth bottleneck is less severe, and both Basic and DualPath are closer to being compute-bound. Across all configurations for DS 660B, DualPath's JCT closely tracks Oracle's JCT — the gap between DualPath and Oracle is small, suggesting that dual-path loading saturates the available storage bandwidth to near its physical limit. SGL(MC) encountered errors and failed to complete some large configurations, marked as "N/A" in the figure.

DS 27B (middle row of Figure 7). At 1024 agents and 64K MaxLen, DualPath achieves 1.78× over Basic. However, the gap to Oracle is larger than for DS 660B — DualPath is 1.09–1.85× slower than Oracle across configurations. The paper attributes this to limited storage bandwidth in the 1P1D configuration (only one node's aggregate SNIC bandwidth is available even with dual-path loading, versus multiple nodes in the 2P4D configuration for DS 660B). This is confirmed by Figure 8, which shows that DualPath's performance varies substantially with P/D ratio — more prefill or decode nodes means more aggregate SNIC bandwidth.

Qwen 32B (bottom row of Figure 7). The trends are similar to DS 27B, with DualPath outperforming Basic and SGL(MC) across configurations. The absolute improvement magnitudes are not explicitly stated as a single number in the figure caption, but the bar heights show DualPath consistently below Basic (lower JCT = better). SGL(MC) fails at larger configurations (1024 agents, 64K and 128K).

Varying P/D ratio (Figure 8). For DS 27B, DualPath achieves an average speedup of 1.64× across all P/D configurations tested (1P1D, 2P1D, 1P2D), with a maximum of 2.46×. A key observation is that equivalent storage bandwidth configurations yield similar performance, regardless of path: Basic 1P1D (one node's SNIC) performs comparably to Basic 1P2D (one prefill node's SNIC, same aggregate bandwidth available to Basic), while DualPath 1P1D (two nodes' SNICs via dual-path) performs comparably to Basic 2P1D (two prefill nodes' SNICs available to Basic). This confirms that storage bandwidth is the dominant bottleneck and that DualPath's dual-path mechanism effectively pools SNIC bandwidth across nodes — DualPath with fewer total nodes matches Basic with more prefill nodes because it can utilize decode-side SNICs.

Scaling Append Length and Generation Length

Varying append length (Figure 9, left). The experiment scales each turn's append length by a constant factor and truncates the trajectory at the given MaxLen. For DS 660B at 64K context with 1024 agents, DualPath achieves 1.82–1.99× speedup over Basic across different append length scales. As append length increases, Basic's performance gradually approaches DualPath and Oracle, while DualPath and Oracle change only slightly. This corroborates the bottleneck analysis: longer appends mean more GPU computation (prefilling the new tokens), which shifts the bottleneck from I/O toward compute. Basic benefits from this shift because compute pressure reduces the relative impact of its I/O limitation; DualPath and Oracle are already I/O-efficient, so additional compute pressure simply makes them compute-bound along with Basic.

Varying generation length (Figure 9, right). The experiment scales each turn's generation length. The trend mirrors the append length result: longer generation shifts pressure from I/O to compute (decoding is memory-bandwidth-bound but does not involve KV-Cache loading during token generation), so Basic's performance approaches DualPath and Oracle. The paper does not report a specific speedup range for generation length scaling, but the JCT curves show DualPath consistently below Basic.

Online Serving: Throughput and Latency Under SLO

Throughput (APS capacity). Figure 10 shows that DualPath achieves higher sustainable APS than Basic while meeting the SLO (TTFT ≤ 4s, TPOT ≤ 50ms). For DS 27B, DualPath reaches 1.67× the APS of Basic. For DS 660B, DualPath reaches 2.25× the APS of Basic. These are the effective throughput multipliers for online serving — at a given hardware configuration, DualPath can serve 1.67–2.25× more agent trajectories per second without violating latency SLOs.

Latency breakdown (Figure 12, left). The TTFT breakdown for DS 660B reveals where the gains come from. The components are: scheduling time, allocation time, KV-Cache reading time, and prefill computation time. Across different APS levels, DualPath's KV-Cache reading time remains stable, while Basic's grows dramatically. At higher APS, Basic's queuing time dominates because the saturated prefill SNICs cannot keep up with the arrival rate — requests queue waiting for storage I/O. DualPath, by distributing reads across both PE and DE SNICs, keeps the reading time controlled.

TTST and TPOT. DualPath's TTST is comparable to Basic. TPOT shows that DualPath does not introduce additional decoding overhead — the per-token generation latency during the decode phase is unchanged, confirming that the CNIC-centric traffic isolation (Section 5) successfully prevents KV-Cache transfers from interfering with latency-sensitive decode computation. SGL(MC) exhibits anomalously low TTST, which the paper attributes to "implementation issues where the first two tokens arrive at the client almost simultaneously."

TPOT anomaly for DS 27B. For DS 27B, both Basic and DualPath show significantly higher TPOT than Oracle. The paper attributes this to "the overhead of basic P-D transferring" being considerable for smaller models — when the model is small, the decode computation is fast, and the KV-Cache transfer between prefill and decode nodes becomes a larger fraction of the total per-step latency. The authors flag this as future work.

Average JCT for online serving (Figure 11). For both DS 27B and DS 660B, DualPath's average trajectory completion time is lower than Basic's across the range of sustainable APS. As APS increases toward the capacity limit, Basic's average JCT grows sharply (more queuing), while DualPath's remains lower. The paper also uses this figure to estimate the KV-Cache working set (discussed in Section 8.2): at a given APS λ and mean JCT T̄, the working set is approximately λ × T̄ × total_len_avg / 2. For DS 660B, this ranges from 69 GB at APS 0.1 to 681 GB at APS 0.45.

Large-Scale Scalability (Table 3 and Figure 15)

Offline scaling. Scaling from 2P4D (48 GPUs, 2K agents) to 48P96D (1,152 GPUs, 48K agents) achieves near-linear speedup: JCT is 3,167 seconds at small scale versus 3,201 seconds at large scale, processing 24× more agents in approximately the same wall-clock time. This demonstrates that DualPath's architecture scales without introducing new bottlenecks at production scale.

Online scaling. The 44P88D configuration achieves 22× the throughput of the smaller online configuration (8.8 APS vs. 0.4 APS) while maintaining similar latency characteristics. The scheduler's CPU usage remains below 10 cores even at 1,152-GPU scale, confirming it is not a bottleneck.

Large-scale caveats. The paper notes that the large-scale experiments do not demonstrate additional JCT or serving capacity gains compared to running multiple small-scale units with equivalent total cost, due to the lack of fine-tuned parallelism settings and P/D ratios. However, large-scale deployment is argued to be important for reducing fragmentation, providing flexibility for parallelism tuning, and offering more scheduling opportunities to mitigate queuing under bursty online requests.

Ablation Studies and Robustness Checks

  • Layerwise prefill alone (Figure 12, right): Adding layerwise prefill to Basic reduces JCT by 17.21% on average (DS 660B, 64K context, 1024 and 2048 agents). This improvement comes from alleviating PE HBM bottlenecks (larger batch sizes) and hiding transfer overhead through overlap. Layerwise prefill is not novel to DualPath (it was introduced by LayerKV and PrefillOnly), but it is a prerequisite for the dual-path design to be effective, since it creates the fine-grained, layer-by-layer transfer pattern that DualPath's data paths exploit.

  • Dual-path loading without scheduling (Figure 12, right): Adding dual-path loading on top of layerwise prefill delivers the primary performance gains, reducing JCT by 38.19% on average compared to Basic. This is the core architectural contribution — simply enabling the DE Read Path alongside the PE Read Path, even with naive (round-robin) path selection, unlocks the majority of the storage bandwidth that Basic leaves idle. The gap between 17.21% (layerwise only) and 38.19% (layerwise + dual-path) — roughly 21 percentage points — is attributable to utilizing decode-side SNICs.

  • Full DualPath with scheduling (Figure 12, right): Adding the adaptive scheduling algorithm on top of dual-path loading achieves the best performance, reducing JCT by 45.62% compared to Basic. The incremental gain from scheduling (45.62% - 38.19% = 7.43%) is modest compared to the gain from dual-path loading itself, but Figures 13 and 14 demonstrate that this gain comes from measurably improved resource balance, not from noise.

  • Storage NIC load balance (Figure 13): The scheduling algorithm improves storage NIC load balance from 1.53 (round-robin) to 1.18 (measured as the ratio of maximum to average traffic across all storage NICs on three machines within a small time window, where 1.0 is perfect balance). This metric is evaluated during the high-load phase of the task; as the task progresses and the system becomes underloaded, the ratio becomes meaningless because traffic is sparse and ratios over small denominators amplify noise.

  • Attention layer execution time balance (Figure 14): DualPath maintains the Max/Avg ratio of attention layer execution time across GPUs in an expert-parallel group as low as 1.06 during the first 5% of the task duration. This metric is calculated among all GPUs in an EP group for each forward pass. A ratio of 1.06 means the slowest GPU is at most 6% slower than the average, minimizing GPU idle bubbles at the EP synchronization barrier. As with the storage NIC metric, this ratio becomes less meaningful in the tail of the workload when GPUs are underloaded.

  • Impact of compute quota (Section 6.2, parameter specification): The compute quota is set to 300 ms for all DualPath and Oracle baselines. The paper does not report an ablation study varying this value, so sensitivity to this parameter is not characterized. This is a potential missing ablation — if a different compute quota (e.g., 100 ms or 500 ms) substantially changes the throughput-latency tradeoff, the reported gains might be specific to the chosen value.

  • SGL(MC) as a baseline (Figures 7 and 10): The comparison to SGL(MC) is limited by implementation differences (the paper explicitly states the comparison is "unfair") and by SGL(MC) failing to complete several large configurations (marked N/A in Figure 7). The SGL(MC) results therefore serve as a rough reference point rather than a rigorous apples-to-apples comparison. The primary comparison is always DualPath vs. Basic, which controls for the inference framework implementation.

  • DRAM buffer sizing (Section 7.2, Experimental Setup): DualPath allocates 80 GB DRAM per node for DeepSeek models, while SGL(MC) uses 1.5 TB DRAM per node (leveraging Mooncake's distributed DRAM pool). For Qwen 32B, DualPath allocates 320 GB DRAM due to the larger KV-Cache. The fact that DualPath achieves its gains with substantially less DRAM (80 GB vs. 1.5 TB per node) is a practical advantage for memory-constrained scenarios like RL rollout, but the paper does not experimentally vary DRAM allocation to study the performance-DRAM tradeoff curve.

  • PE Buffer and DE Buffer design (Section 4.1, qualitative): The PE Buffer and DE Buffer staging design avoids GPU Direct RDMA for the final H2D transfer into DE HBM. The paper acknowledges that bypassing the DE Buffer via GPU Direct RDMA could reduce DRAM bandwidth pressure, but argues that since generation length is typically short in agentic workloads, TTFT accounts for a non-negligible portion of total request time, and the DE Buffer helps reduce GPU memory usage. No ablation comparing DE Buffer vs. GPU Direct RDMA is presented.

  • Doorbell batching contribution (Section 5.2, qualitative): The paper claims that CNIC-assisted H2D/D2H with doorbell batching outperforms cudaMemcpyAsync for many small transfers, but does not report a microbenchmark quantifying this overhead in the context of DualPath's layerwise prefill. The 1 µs vs. 5–7 µs per-submission numbers are cited from general RDMA and CUDA knowledge, not from experiments within the DualPath system. The actual performance impact of this design choice is therefore not isolated in the ablation study.

  • Negative result: SGL(MC) TTST anomaly (Figure 10): SGL(MC) shows anomalously low TTST for DS 660B, with the first two tokens arriving nearly simultaneously. The paper attributes this to a likely implementation issue but does not investigate further. This is relevant because if SGL(MC)'s TTST measurement is unreliable, its TTFT and TPOT measurements may also be affected, limiting the value of SGL(MC) as a baseline for latency metrics.

Critical Assessment

Claim 1: DualPath improves offline inference throughput by up to 1.87×. This claim is supported by Figure 7 (DS 660B, 1024 agents, 64K MaxLen) and Table 3. However, the "up to 1.87×" framing should be contextualized: this maximum gain is achieved at the largest scale tested (1024 agents, 64K context, 2P4D), where the I/O bottleneck is most severe. At smaller scales (256 agents, 16K context), the gain is smaller because the workload is closer to compute-bound. The average gain across all configurations in Figure 7 is lower than 1.87×, though the paper does not report the geometric mean. The 1.87× figure should therefore be understood as the best-case improvement under the most I/O-intensive conditions, not as the expected gain across all workloads.

Claim 2: DualPath improves online serving throughput by an average factor of 1.96×. This claim is supported by Figure 10 (DS 27B: 1.67×, DS 660B: 2.25×; average = 1.96×). The online SLO is set to TTFT ≤ 4s and TPOT ≤ 50ms, which are reasonable production SLOs but are specific choices — different SLO thresholds would yield different throughput multipliers. The paper reports throughput at the point where the SLO is violated, so the 1.96× figure is the capacity multiplier at the chosen SLO, not a universal constant.

Claim 3: Dual-path loading can fully saturate all storage NICs without introducing compute-NIC or DRAM bottlenecks within a derived P/D ratio range. The theoretical analysis in Section 4.2 derives the range 1/7 ≤ P/D ≤ 7/2. This is a mathematical derivation under stated assumptions, not an empirical claim, so it is "supported" in the sense that the math is correct under those assumptions. However, two empirical validations are notable: (1) Figure 8 shows that DualPath at 1P1D ≈ Basic at 2P1D, consistent with the theory that DualPath pools SNIC bandwidth (1P1D DualPath uses 2 nodes' SNICs, matching 2P1D Basic's 2 prefill nodes' SNICs); and (2) the large-scale experiments (Table 3) with 48P96D (P/D = 0.5) operate within the safe range and achieve near-linear scaling. The analysis would be stronger with experiments at P/D ratios outside the safe range (e.g., 1P8D or 8P1D) to demonstrate that the derived bounds actually predict where congestion occurs, but such experiments are not reported.

Claim 4: The scheduling algorithm improves load balance from 1.53 to 1.18 (storage NICs) and maintains attention execution time balance at 1.06 (Max/Avg). These metrics are supported by Figures 13 and 14. However, both metrics are measured during the high-load phase only (first 5% of the task for Figure 14, and the "small time window" early in the task for Figure 13). The paper acknowledges that these ratios become "meaningless" as the task progresses and the system becomes underloaded. This is reasonable — load balance matters most when the system is saturated — but it means the reported numbers are best-case for the metric, not averages over the entire task duration.

The gap between DualPath and Oracle reveals remaining I/O overhead. On DS 660B (Figure 7, top), DualPath closely tracks Oracle, suggesting I/O is nearly eliminated as the bottleneck. On DS 27B (Figure 7, middle), DualPath is 1.09–1.85× slower than Oracle. This gap represents I/O overhead that dual-path loading does not eliminate, attributed to "limited storage bandwidth in 1P1D." The paper does not provide a breakdown of what fraction of this remaining gap is due to SNIC bandwidth limitations (solved by adding more nodes) versus CNIC overhead (solved by faster compute networks) versus DRAM staging overhead (solved by GPU Direct RDMA). This is a missed opportunity for guiding future optimization efforts.

Working set analysis reveals a potential scaling problem for online serving. Section 8.2 estimates that the KV-Cache working set for DS 660B online serving ranges from 69 GB to 681 GB. The paper then notes that in production, tool call latencies would stretch JCT by a factor r, which would expand the working set by r² — potentially exceeding available memory and reducing distributed memory pool hit rates. This is an honest acknowledgement of a limitation, but it also highlights that the online serving experiments (which assume zero inter-arrival time and zero tool call latency) represent a best-case scenario for the working set size. Real-world deployment with realistic tool latencies could encounter memory pressure that the current experiments do not capture, and the paper does not evaluate DualPath under such conditions.

Single storage backend (3FS with no DRAM cache). All experiments use the 3FS distributed file system with no internal DRAM cache. This is a deliberate choice that isolates the effect of dual-path loading — if the storage backend had its own DRAM cache (as Mooncake does), the I/O bottleneck would be partially masked by cache hits, making it harder to measure DualPath's contribution. However, this also means the results may not generalize to deployments that use DRAM-cached storage backends. The paper acknowledges that DualPath can be combined with a DRAM cache tier but claims "the performance gain is marginal" (Section 9), without providing experimental evidence for this claim.

Production trace realism. The three agent trace datasets are collected from "production agentic RL training workloads" but are specific to a code-debugging task (as described in Appendix A.3). The KV-Cache hit rate of 98.7% and mean 157 turns are reported for "representative coding tasks." Agentic workloads in other domains (web navigation, dialogue, scientific reasoning) may have different turn counts, context growth patterns, and KV-Cache hit rates. The paper does not evaluate on traces from other agentic domains, so the generality of the 1.87× and 1.96× figures to other agentic workloads is assumed rather than demonstrated.

Missing comparison to KV-Cache compression or quantization. The paper compares against systems that optimize the I/O path (Mooncake, TARDIS) but does not compare against approaches that reduce the amount of KV-Cache data to load, such as KV-Cache quantization (e.g., KIVI, Atom) or eviction-based sparse attention (e.g., StreamingLLM, H2O). These are orthogonal optimizations that could combine with DualPath — if less KV-Cache needs to be loaded, the I/O bottleneck is proportionally reduced — but the paper does not discuss this interaction. An experiment showing DualPath's gains on top of a quantized KV-Cache would help clarify whether DualPath's benefit is additive with or subsumed by data-reduction techniques.

Missing sensitivity analysis on key parameters. Several design parameters are fixed without ablation: the compute quota (300 ms), the short reading queue threshold α (3 seconds), the unfinished token upper limit β (5 seconds), and the DE Buffer size (80 GB or 320 GB depending on model). The paper argues these are profiled in advance based on hardware characteristics, but the sensitivity of system performance to these choices is not explored. If, for example, setting α to 1 second instead of 3 seconds caused the scheduler to oscillate between paths, that would be important practical knowledge for deployers. Similarly, the 300 ms compute quota affects the tradeoff between batching efficiency and latency — if a different value (e.g., 100 ms or 500 ms) yielded substantially different throughput at the SLO boundary, the reported online serving gains might be quota-specific.

6. Limitations and Trade-offs

Limitation 1: Online Difficulty Estimation Cost Is Unaccounted for in Efficiency Gains

The assumption or constraint. The compute-optimal test-time scaling strategy depends on estimating each prompt's difficulty before allocating the inference budget. The paper's method for difficulty estimation — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — is extremely expensive. The paper acknowledges this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N (Figure 4: 16 generations matching 64; Figure 8: 64 generations matching 256) are computed after difficulty is known, without amortizing the cost of learning it. Generating 2048 samples to estimate difficulty can consume more compute than the largest test-time budgets studied (256–512 generations). In a realistic deployment where difficulty estimation must be performed for each new prompt, the total cost would be estimation + strategy execution, and the former could dominate the latter. A practitioner who reads "4× efficiency improvement" should understand that this is an upper bound on what is achievable in practice, not a realized deployment gain, until cheap difficulty estimation is demonstrated.

What evidence exists in the paper. The paper explicitly flags this as future work (Section 3.2) but does not include any experiments that account for estimation cost. The "predicted difficulty" variant (using the PRM's average score instead of ground-truth labels) still requires 2048 samples per question — it removes the need for ground-truth labels but does not reduce the computational cost of estimation. Figure 4 shows that predicted difficulty bins largely overlap with oracle bins, which is encouraging for removing label dependence, but neither curve accounts for the 2048-sample cost.

Mitigation status. Not addressed. The paper suggests that "future work could pretrain or finetune models to directly predict the difficulty of a question" (Section 8), but no such model is developed or evaluated. Until fast, cheap difficulty estimation is demonstrated, the compute-optimal framework remains expensive to deploy for one-off queries, though it may be cost-effective when amortized over many repeated evaluations of the same prompt distribution (e.g., batch evaluation, self-improvement data generation).

Limitation 2: Single Benchmark, Single Model Family — Generality Is Unverified

The assumption or constraint. All experiments use only the MATH benchmark (500 test questions) and PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified through replication.

The consequence. Several aspects of the findings could be model-specific or benchmark-specific:

  • PRM quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties, different error patterns, or a different base accuracy on MATH would produce a different distribution of PRM scores, potentially shifting the difficulty thresholds at which different strategies become optimal. The paper's finding that beam search degrades performance on easy problems at high budgets (Figure 3, right) depends on the specific interaction between PaLM 2-S*'s output distribution and the PRM trained on its outputs.
  • The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The 38% correct-to-incorrect reversion rate (Section 6.1) is specific to PaLM 2-S*'s revision behavior and might be different for other base models.
  • The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning. The difficulty-dependent patterns — sequential revisions helping on easy problems, beam search helping on medium-hard problems — may not generalize to reasoning domains with different structure (code generation, where correctness is compositional; logical reasoning, where errors propagate differently; factual QA, where knowledge rather than reasoning is the bottleneck). The paper provides no evidence either way.

What evidence exists in the paper. None — no cross-model or cross-benchmark experiments are reported. The dataset is a single 500-question test set. The base model is a single checkpoint (PaLM 2-S*). The FLOPs-matched comparison uses a second model (~14× larger from the same model family), not a different architecture.

Mitigation status. Not addressed. The paper does not discuss benchmark or model diversity as a limitation. A practitioner considering deploying compute-optimal test-time scaling on a different model or for a different task domain should be aware that the specific difficulty thresholds, optimal strategies, and efficiency gains may not transfer. The qualitative pattern (easy problems benefit from revision/local refinement, medium problems benefit from search/global exploration, hard problems benefit from neither) is the most portable finding, but the precise 4× efficiency figure and the optimal strategy choices per bin are likely specific to PaLM 2-S* on MATH.

Limitation 3: The ~14× Larger Model Baseline Is Not Compute-Optimally Trained and Uses Weaker Inference

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters that uses greedy decoding only (no test-time compute augmentation). The larger model scales parameters while holding training data fixed (following the LLaMA paradigm, Touvron et al., 2023), rather than scaling both data and parameters as prescribed by Chinchilla-optimal training (Hoffmann et al., 2022). The paper acknowledges:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

The consequence. Both aspects weaken the pretraining baseline:

  • Non-C hinchilla-optimal training: A model trained with 14× more FLOPs should ideally scale both parameters and data equally. The parameter-only-scaled model likely underperforms a properly compute-optimally trained larger model, making the test-time compute gains appear larger than they would be against a stronger baseline.
  • Greedy decoding only: The 14× larger model is evaluated with only greedy decoding — no majority voting, no best-of-N sampling, no verifier guidance. The paper's own results (Figure 3, Figure 6) show that even modest amounts of test-time compute (4–16 generations with best-of-N weighted or majority voting) significantly improve accuracy. The comparison is therefore asymmetric: the smaller model gets to use sophisticated test-time strategies (beam search, revisions, best-of-N weighted), while the larger model gets none of these. A fairer comparison would give the larger model an equal or proportional test-time compute budget, or at minimum evaluate it with best-of-N sampling.

The practical consequence is that the paper's finding — that test-time compute can substitute for pretraining, with the smaller model outperforming the 14× larger model on easy-to-medium problems (Figure 9, Figure 1 bar charts) — is measured against a deliberately weakened baseline. A Chinchilla-optimally trained larger model with even a modest test-time compute budget (best-of-8 or best-of-16) could close or reverse the gap, particularly at higher inference-to-pretraining ratios (R ≫ 1).

What evidence exists in the paper. The paper is transparent about the parameter-only scaling choice but does not report how much weaker this baseline is compared to a Chinchilla-optimal baseline. No experiment gives the larger model any test-time compute augmentation, not even simple best-of-N majority voting. The FLOPs accounting formula (Section 7) correctly counts the smaller model's extra inference FLOPs, so the comparison is FLOPs-fair in a narrow sense, but it is not strategy-fair — the larger model is denied access to the same class of inference-time optimizations that the smaller model receives.

Mitigation status. The paper acknowledges the scaling regime limitation but does not address the asymmetric inference strategy issue. Both are left as future work. The takeaway — that test-time compute can substitute for pretraining under the right conditions — should be understood as an existence proof (it is possible) rather than a calibrated estimate of the substitution ratio. The actual substitution ratio against a fully optimized baseline (Chinchilla-optimal pretraining + best-of-N inference) remains unknown.

Limitation 4: The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate with Only Heuristic Mitigation

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect followed by a correct target answer. At test time, the model may encounter correct answers in its context (produced during earlier revision steps) and incorrectly revise them into wrong answers. The paper reports:

"the model may encounter correct answers in its context (produced during earlier revisions) and incorrectly 'revise' them into wrong answers. The paper reports that approximately 38% of correct answers get converted back to incorrect ones." (Section 6.1)

The consequence. This reversion problem fundamentally limits the effectiveness of long sequential revision chains. As shown in Figure 6 (left), pass@1 improves from ~18.2% to ~24–25% by steps 15–20, then plateaus. The plateau is not because the model cannot produce better answers — it may have already generated a correct answer earlier in the chain that was subsequently reverted. The mitigation — majority voting or verifier-based selection across the entire chain rather than taking only the final revision — is a heuristic that salvages correct answers that the model itself "un-does." It does not address the root cause: the model does not know when to stop revising. This means that:

  • Every revision step after a correct answer is generated risks reducing the chain's best answer quality.
  • The effective length of a useful revision chain is limited by the reversion rate — longer chains accumulate more opportunities for correct answers to be reverted.
  • The optimal sequential-to-parallel ratio found by the scheduler (Figure 7) is partly a response to this limitation: fully sequential chains (long revisions) suffer from reversion, while parallel chains (many independent attempts) provide diversity but no refinement. The scheduler finds the compromise that balances these effects, but the underlying reversion problem caps how far sequential revision can scale.

What evidence exists in the paper. The 38% figure is stated in Section 6.1. Figure 6 (left) shows the revision pass@1 trajectory plateauing despite the model continuing to produce new answers, consistent with reversion erasing gains. The negative result with ReST^EM (Appendix K, Figure 16) — where additional sequential revisions substantially hurt performance — is an extreme manifestation: the ReST^EM-trained model had an even higher effective reversion rate, causing chain performance to degrade with length.

Mitigation status. Partially addressed through chain-level selection (majority voting or verifier-based selection), but the root cause is not solved. The paper does not explore training the revision model to recognize when no revision is needed (e.g., by including "no revision needed" trajectories in training data, or by adding a binary classifier that predicts whether the current answer is correct). This is a gap: if the reversion rate could be reduced, sequential revision chains could be longer and more effective, potentially improving the gains on medium-difficulty problems where intermediate-length chains are optimal.

Limitation 5: PRM Search and Revisions Are Never Combined — Representing a Lower Bound

The assumption or constraint. The paper studies PRM-guided search and iterative revisions as independent mechanisms, evaluating them separately and computing separate compute-optimal scaling curves for each. Section 8 explicitly acknowledges:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The paper's two complementary axes — modifying the proposal distribution through revisions (generating better individual candidates) and optimizing the verifier/selection through PRM search (finding the best candidates among a diverse set) — have complementary difficulty-dependent strengths. Revisions are most effective on easy problems (local refinement of approximately-correct answers), PRM search is most effective on medium problems (global exploration to discover correct strategies), and neither is effective on hard problems. Combining them could yield:

  • Beam search over revision chains: Use the PRM to score partial revision steps, pruning unpromising revision directions early and extending promising ones. This would combine the revision model's ability to refine answers with the PRM's ability to discriminate good from bad intermediate states, potentially reducing the reversion problem (Section 6.1) by steering revisions away from dead ends.
  • Revision model as the proposal distribution for best-of-N: Generate N independent revision chains (each producing a sequence of refined answers), then use PRM best-of-N weighted selection across the final answers of all chains. This would capture both the within-chain refinement benefit and the across-chain diversity benefit.
  • Difficulty-conditioned combination: On easy problems, use purely sequential revisions (as the paper already recommends). On medium problems, use beam search with the revision model as the proposal, or use parallel revision chains with PRM-based selection. This would push the performance ceiling beyond what either mechanism alone achieves.

The current results therefore represent a lower bound on what a fully integrated system could achieve. The paper's framework (proposal distribution vs. verifier) directly suggests this combination, but the experiments do not evaluate it.

What evidence exists in the paper. None — no combination experiments are reported.

Mitigation status. Explicitly noted as future work (Section 8). A practitioner building on this work should be aware that the 4× efficiency gains and the FLOPs-matched comparisons are measured for each mechanism in isolation, and that combining them would likely yield additional gains, particularly on medium-difficulty problems where both mechanisms show complementary strengths (revisions provide refinement, PRM search provides exploration). The magnitude of the potential combined gain is unknown but could be significant since the two mechanisms target different aspects of the problem.

Limitation 6: No Accounting for Latency or Wall-Clock Time — Sequential Revisions Are Serially Dependent

The assumption or constraint. The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores the latency dimension. Sequential revisions are inherently serial — each revision depends on the previous revision's output. Parallel best-of-N sampling can be executed simultaneously if sufficient hardware parallelism is available (multiple GPUs or large batches).

The consequence. The compute-optimal policy on easy problems favors purely sequential revisions (Figure 7, right, bin 1–2), which means funneling the entire generation budget into a single chain of revisions. While this is compute-efficient in FLOPs, it is latency-inefficient: a strategy that allocates 64 generations as a single sequential chain of 64 revisions takes ~64× longer in wall-clock time than one that runs 64 parallel samples simultaneously. For latency-sensitive applications (interactive assistants, real-time decision-making), the sequential-heavy strategies that the compute-optimal policy recommends may be impractical regardless of their FLOPs efficiency. A latency-aware formulation would need to specify a latency budget in addition to a FLOPs budget, and the optimal strategy would shift — favoring more parallel computation under tight latency constraints even when it is FLOPs-inefficient.

The paper's online serving evaluation in the reference example does not address this, since that example does not contain an online serving experiment. For the MATH benchmark evaluation in this paper, the offline throughput metric (JCT) does not directly translate to latency guarantees — offline throughput can be high even with long sequential chains because multiple requests can be batched, but per-request latency is unbounded.

What evidence exists in the paper. No latency or wall-clock time measurements are reported. The paper's "compute budget" is always in units of generations, not in seconds of wall-clock time. The intra-engine scheduling (Section 6.2) uses a compute quota of 300 ms for batch packing, but this is about forward-pass latency on a single batch, not about end-to-end request latency under different sequential-parallel allocation ratios.

Mitigation status. Not addressed. A practitioner deploying this method in a latency-sensitive setting would need to augment the compute-optimal framework with a latency constraint, which would change the policy (e.g., capping maximum sequential chain length, favoring more parallel sampling on easy problems even though it is FLOPs-inefficient). The paper's framework could in principle be extended to include a latency budget, but it does not provide the latency measurements or the multi-objective optimization machinery to do so.

7. Implications and Future Directions

How This Work Changes the Landscape

DualPath changes the conversation about LLM inference system design by demonstrating that KV-Cache loading is not inherently a prefill-side operation — it is a schedulable, cluster-wide resource allocation problem. This may sound like a narrow systems insight, but its implications ripple through how the field thinks about inference architecture.

The magnitude: a reframing, not a paradigm shift. The paper does not introduce a new storage technology, a new network protocol, or a new model compression technique. What it does is identify a structural inefficiency — the asymmetric utilization of storage bandwidth in PD-disaggregated architectures — and demonstrate that a relatively simple architectural change (adding a second loading path) yields substantial throughput gains (up to 1.87× offline, 1.96× online). This is best characterized as a reframing: it takes an assumption that had been baked into every prior system ("KV-Cache loading is what prefill engines do") and exposes it as a design choice with an alternative. The conceptual move — from treating I/O as a per-node constraint to treating it as a cluster-wide, schedulable pool — parallels how distributed storage systems evolved from single-disk to striped designs, and how network routing evolved from single-path to multi-path. It is the same category of insight applied to a new domain.

This matters because it opens an optimization dimension that prior work had not recognized. Before DualPath, improving KV-Cache loading meant making the storage-to-prefill pipe faster: better I/O stacks (Phoenix), GPU Direct Storage (TARDIS), reduced data volume (HCache, KV-Cache quantization), or caching in faster tiers (Mooncake, TokenLake). These are all single-path optimizations. DualPath shows that orthogonal to pipe diameter is path multiplicity — how many pipes you use and how you schedule across them. These two dimensions are complementary: a system can simultaneously use faster I/O stacks and multiple loading paths, with the benefits stacking multiplicatively. The paper briefly notes this (Section 9: "DualPath can also be combined with a middle DRAM cache") but does not evaluate the combination, leaving the multiplicative potential as an open question.

Reconciling contradictions in prior work. The paper resolves a tension that was latent in the literature but never explicitly surfaced. On one hand, Mooncake (Qin et al., 2025) demonstrated that distributed DRAM caching of KV-Cache could dramatically reduce I/O latency — but only when the working set fit in DRAM. On the other hand, systems like PrefillOnly (Du et al., 2025) and LayerKV (Xiong et al., 2024) showed that layerwise prefill was essential for GPU utilization with long contexts — but this optimization fragments KV-Cache into small layer-level chunks that stress I/O systems. The implicit tension is: layerwise prefill creates more I/O pressure (many small reads instead of one large read), while DRAM caching addresses I/O pressure but requires memory that may not be available (as in RL rollouts where HBM is offloaded to DRAM, or in production serving where the working set is huge). DualPath resolves this by working at a different level: it does not reduce the amount of I/O needed, nor does it cache data in faster tiers, but instead increases the aggregate I/O bandwidth available to the system by pooling previously-idle NICs. This makes layerwise prefill viable even without massive DRAM caches, which in turn enables the HBM-efficient batching that PrefillOnly and LayerKV advocate.

Which research directions become more attractive. The paper shifts attention toward:

  • I/O path diversity as a first-class design dimension. Future inference systems should be evaluated not just on their single-path I/O efficiency but on their ability to utilize multiple paths. This suggests research into more sophisticated path selection algorithms (beyond the "pick the shorter queue" heuristic), request splitting across paths (the paper explicitly leaves this as future work), and dynamic path allocation based on real-time congestion signals.
  • Logical traffic isolation over physical separation. The paper demonstrates that InfiniBand VLs can effectively isolate KV-Cache traffic from model execution collectives without physical network separation. This opens the possibility of simplifying data center network topology by relaxing the strict east-west / north-south isolation that current architectures mandate, potentially reducing cabling complexity and switch costs.
  • Joint optimization of parallelism strategy and I/O topology. The paper's bottleneck-free analysis (Section 4.2) shows that the safe P/D ratio range depends on hardware parameters (g, s, M). This suggests that the choice of tensor parallelism, expert parallelism, and pipeline parallelism — which determine how much communication each GPU does — should be co-optimized with the I/O loading strategy. A system that jointly configures parallelism and path selection could find configurations that are individually suboptimal but jointly optimal.
  • The interaction between KV-Cache compression and loading bandwidth. If KV-Cache quantization (e.g., FP8 instead of FP16) reduces the data volume by 2×, the I/O bottleneck is halved, and the P/D ratio bounds in Equation 9 would shift (since s effectively decreases relative to B). The paper does not explore this interaction, but it suggests a research direction where compression ratios and path multiplicity are co-tuned.

Which research directions become less attractive. The paper implicitly argues against:

  • Incremental single-path I/O optimization as a standalone solution. The paper shows that even with an optimized I/O stack (the Basic baseline uses io_uring-like kernel bypass and 3FS, which saturates the 400 Gbps SNIC), the single-path design leaves substantial bandwidth idle on the decode side. Further optimizing the single path (e.g., from 400 Gbps to 800 Gbps SNICs, or from io_uring to SPDK) would improve performance but cannot address the fundamental asymmetry — half the cluster's SNICs would still be idle. The gains from single-path optimization are bounded by the fraction of total cluster bandwidth that prefill nodes represent, which is P/(P+D). DualPath's approach breaks through this ceiling by utilizing all nodes.
  • DRAM-only caching as a universal solution for KV-Cache. The working set analysis (Section 8.2) provides a sobering calculation: for DS 660B online serving, the KV-Cache working set at moderate APS already reaches hundreds of gigabytes, and in production with realistic tool call latencies, it can expand by r² (where r is the JCT inflation factor). This suggests that pure DRAM caching (as in Mooncake) will inevitably hit a capacity wall for long-context agentic workloads, and that fast SSD-based storage with high aggregate bandwidth — which DualPath enables — is the more scalable approach. The paper does not argue against DRAM caching as a tier (it explicitly says DualPath can be combined with it), but it does suggest that DRAM-only approaches are insufficient for production agentic workloads.

Follow-Up Research This Work Enables

1. Online, lightweight difficulty estimation to close the deployment gap. The paper's difficulty estimation method requires 2048 samples per question, costing more compute than the largest test-time budgets studied. A strong follow-up would train a lightweight classifier — potentially a small distilled model or even a linear probe on top of the base model's hidden states — to predict the difficulty bin directly from the question text, without any sampling. The training data for this classifier would come from running the 2048-sample estimation on a representative question set once, offline. The evaluation would compare the compute-optimal scaling curve using the classifier's predicted bins against the oracle bins from Figure 4 and Figure 8. If the classifier achieves bin prediction accuracy comparable to the PRM-based method, the compute-optimal framework becomes immediately practical for deployment. A particularly elegant variant: use the PRM's score distribution on the first 4–8 samples as a continuous difficulty signal, and dynamically adjust the strategy mid-computation — this amortizes difficulty estimation into the problem-solving process itself.

2. Combining PRM tree search with the revision model as the proposal distribution. The paper studies PRM search and iterative revisions as independent mechanisms but explicitly notes (Section 8) they were never combined. The natural follow-up is to use the revision model as the proposal distribution within beam search: at each step of the search tree, the model conditions on previous rejected branches as context (exploiting the revision model's training on incorrect-answer context), potentially producing higher-quality candidate steps than the base model. The experiment would compare three configurations at matched generation budgets: (a) PRM beam search with base model proposal (as in Section 5.3), (b) sequential revisions (as in Section 6), and (c) PRM beam search with revision model proposal. The hypothesis is that (c) would outperform both (a) and (b) on medium-difficulty problems (bins 3–4), where beam search benefits from the revision model's ability to refine partial solutions, and the PRM's per-step scores help avoid the reversion problem (Section 6.1) by pruning revision directions that score poorly. A negative result — where (c) underperforms (b) because the PRM over-optimizes against the revision model's distribution shift — would be equally informative, as it would bound the conditions under which search and revisions can be safely combined.

3. Verifier robustness under aggressive search optimization. The paper documents that beam search degrades performance on easy problems at high budgets (Figure 3, right) due to PRM over-optimization, but does not propose solutions. A strong follow-up would train a PRM specifically designed to be robust under search: using adversarial training where the PRM is fine-tuned on solutions generated by beam search (which tend to exploit PRM weaknesses), rather than on i.i.d. samples from the base model. The experiment would train two PRMs — one with the standard Monte Carlo rollout procedure (Appendix D), one with the same procedure plus an adversarial fine-tuning phase where negative examples are drawn from beam search outputs that score highly under the PRM but are actually incorrect — and compare the resulting compute-optimal scaling curves. The key metric is whether the adversarially-trained PRM's beam search curve continues to improve at high budgets on easy problems, or at least does not degrade. A positive result would shift the research priority from "find the right search algorithm" to "train the right verifier," consistent with the paper's implication that verifier quality is the primary bottleneck (Section 8).

4. Cross-model and cross-domain replication with standardized difficulty bins. The paper's findings are entirely on MATH with PaLM 2-S*. A systematic replication study would evaluate the same compute-optimal framework (difficulty estimation, strategy sweep, cross-validated policy selection) on at least three axes: (a) different model families (e.g., Llama-3, DeepSeek-V2, Qwen-2.5) on the same MATH benchmark to test whether the difficulty-dependent patterns are model-agnostic; (b) different reasoning domains (e.g., code generation with HumanEval/MBPP, logical reasoning with ARC/FOLIO, scientific QA with GPQA) using at least one model to test domain generality; and (c) different base model scales (e.g., 7B, 13B, 70B within the same model family) to test whether the compute-optimal strategies shift with base capability. The experiment would use the same five-bin difficulty quintile approach, but would report whether the optimal strategy per bin (beam search vs. best-of-N, sequential vs. parallel ratio) is consistent across models and domains. If the qualitative pattern holds (revisions help on easy, search helps on medium, neither helps on hard), it establishes a broad principle; if it breaks (e.g., code generation benefits from search on easy problems because verifier over-optimization is less severe with unit tests), it reveals important boundary conditions.

5. Continuous and dynamic allocation policies with reinforcement learning. The paper's five-bin discretization with cross-validated static policy lookup is effective but coarse. A more ambitious follow-up would formulate the test-time compute allocation problem as a contextual bandit or small Markov decision process, where the state includes the PRM's score distribution after a small number of initial samples, the action is the choice of strategy (search algorithm, beam width, revision chain length, parallel sampling ratio), and the reward is correctness. A learned policy (trained via offline RL on logged data from strategy sweeps like those in Section 5.3 and Section 6) could make continuous strategy choices (e.g., smoothly varying the sequential-to-parallel ratio) and dynamic adjustments mid-computation (e.g., starting with 4 parallel samples, observing the PRM's score spread, and deciding whether to continue parallel or switch to beam search). The evaluation would compare the learned policy against the static compute-optimal policy from the paper, with the prediction that the learned policy achieves the same accuracy with lower average compute (by more efficiently matching strategy to difficulty) or higher accuracy at the same compute (by better exploiting mid-computation signals). A negative result — where the learned policy does not significantly outperform the static policy — would suggest that difficulty is indeed the sufficient statistic the paper hypothesizes, and that the five-bin discretization captures most of the available information.

6. Characterizing the minimum test-time compute needed to match a larger model, as a function of difficulty. The FLOPs-matched comparison in Section 7 provides point estimates (e.g., at R ≪ 1, test-time compute with PaLM 2-S* matches or exceeds the ~14× larger model on easy-to-medium problems). A systematic follow-up would sweep the model scale multiplier M (from 2× to 32×) and measure the minimum test-time compute budget (in generations) needed for the smaller model to match the larger model's accuracy, broken out by difficulty bin. This would produce a family of curves analogous to the pretraining scaling laws literature: for a given difficulty level and model scale ratio, how many inference FLOPs equal one pretraining FLOP? The experiment would be expensive (requiring training or accessing multiple model scales and running compute-optimal scaling at each), but it would provide the first calibrated inference scaling laws that practitioners could use to make pretraining-vs-inference budget allocation decisions — directly extending the paper's conceptual parallel to Chinchilla scaling laws into a quantitatively predictive framework.

Practical Applications and Downstream Use Cases

1. Cost-efficient RL rollout for agent training. The paper's offline inference setting (Section 7.3) directly models the rollout phase of reinforcement learning for agents, where thousands of multi-turn trajectories must be generated to collect training data. In this scenario, throughput is paramount and latency tolerance is high (batch processing). The paper's results show that DualPath reduces JCT by up to 1.87× on DS 660B at production scale (1024 agents, 64K context). For an organization running large-scale agent RL training — which may involve millions of trajectories over weeks — a 1.87× throughput improvement translates to either nearly halving the GPU-hours required for rollouts (reducing cost) or nearly doubling the amount of training data generated in the same wall-clock time (potentially improving the trained agent's quality). The DRAM efficiency of DualPath (80 GB per node vs. 1.5 TB in SGL(MC)) is particularly valuable here, since RL training already consumes DRAM for optimizer states and reward model parameters, leaving limited headroom for KV-Cache caching.

2. Online agent serving with SLO guarantees. The paper's online serving results (Section 7.4) show DualPath sustaining 1.67–2.25× higher agent throughput (in APS) while maintaining TTFT ≤ 4s and TPOT ≤ 50ms. For a production agent service — such as a coding assistant that invokes tools over dozens of turns per user session — this means the same hardware can serve roughly twice as many concurrent users without latency violations. The key practical takeaway is the TTFT breakdown in Figure 12 (left): DualPath's KV-Cache reading time remains stable as APS increases, while Basic's grows dramatically. This stability is what enables higher throughput under SLO — the system does not accumulate queuing delay as the load increases, because the dual-path design prevents the prefill-side SNIC from becoming the bottleneck. For deployment engineers, the implication is that adding decode nodes to a DualPath cluster provides not just more decoding capacity (the standard reason to add DEs) but also more storage I/O capacity (because DE SNICs participate in KV-Cache loading), which is a non-obvious scaling property that changes how capacity planning should be done.

3. Large-batch evaluation and benchmarking of agentic models. When evaluating agentic models on benchmark suites (e.g., SWE-bench, WebArena, or internal agent evaluation harnesses), the workload pattern mirrors offline inference: many independent trajectories processed as a batch. The 1.87× throughput improvement directly reduces evaluation turnaround time, which matters for research velocity — if evaluating a new model checkpoint on 500 agent trajectories takes 10 hours with Basic, it takes ~5.3 hours with DualPath, enabling more rapid experimentation. The compute-optimal difficulty estimation approach from the reference example paper (not DualPath) could further accelerate evaluation by adaptively allocating compute per problem, though this would require extending DualPath's scheduler with difficulty-awareness.

4. Hybrid deployment with small models for routine tasks and escalation to large models for hard tasks. While DualPath does not itself propose difficulty-adaptive compute allocation (that is the reference paper's contribution), the two ideas can be combined. A deployment architecture could use a small model (e.g., DeepSeek-V3.2 27B or even smaller) with DualPath's dual-path loading for routine agent turns (short tool outputs, simple actions), and escalate to a larger model (e.g., DS 660B) only when the task becomes complex (long contexts, multi-step reasoning). The DualPath architecture's ability to pool storage bandwidth across all nodes is particularly beneficial for the small-model tier, where GPU compute is cheap and I/O is proportionally the dominant bottleneck (as evidenced by the larger gap between Basic and Oracle for DS 27B compared to DS 660B in Figure 7). The difficulty estimator from the reference paper could serve as the escalation trigger: easy and medium problems go to the small model with DualPath, hard problems go to the large model. This is speculative because neither paper evaluates such a combination, but the architectural compatibility is clear.

When to Prefer This Method

DualPath targets a specific bottleneck (asymmetric storage bandwidth utilization in PD-disaggregated inference for agentic workloads) and makes specific assumptions about hardware and workload characteristics. The paper does not explicitly provide a decision matrix against named alternatives, but the experimental results and bottleneck-free analysis imply several operational criteria:

Prefer DualPath's dual-path loading when:

  • The workload involves multi-turn agentic inference with high KV-Cache hit rates (≥95%), where KV-Cache loading dominates the timeline over computation. If the workload is single-turn or has low cache hit rates (e.g., many unique prompts), the I/O bottleneck is less severe, and dual-path loading provides diminishing returns — the system becomes compute-bound, and adding I/O paths does not help.
  • The hardware configuration uses PD disaggregation with physically separate storage and compute networks (the standard AI data center architecture). DualPath's CNIC-centric traffic isolation depends on the compute network supporting hardware QoS (InfiniBand VLs, RoCE TCs with PFC). On a single unified network without QoS, mixing KV-Cache and model execution traffic could degrade inference latency.
  • The P/D ratio falls within the derived bottleneck-free range (1/7 ≤ P/D ≤ 7/2 for 8-GPU nodes with one SNIC each). Outside this range, dual-path loading can create compute-NIC or DRAM bottlenecks that offset the storage bandwidth gains. The analysis in Section 4.2 provides the exact bounds for a given hardware configuration.
  • DRAM is constrained (e.g., RL rollouts, edge deployments, cost-sensitive serving). DualPath achieves its gains with modest DRAM buffers (80 GB per node for DS models vs. 1.5 TB for Mooncake-based approaches), making it viable when large DRAM KV-Cache pools are unavailable or uneconomical.

The dual-path approach is less beneficial (or requires adaptation) when:

  • The workload is compute-bound (e.g., short contexts, many unique prompts, long generation lengths). DualPath's gains shrink as the cache-compute ratio decreases (Figure 9, left and right). In the limit of zero KV-Cache reuse, dual-path loading provides no benefit because there is no KV-Cache to load.
  • Latency is the primary constraint and wall-clock time matters more than total FLOPs. The dual-path design introduces an extra network hop for DE-path reads (storage → DE DRAM → PE HBM instead of storage → PE HBM), which adds latency even though it increases throughput. The paper's online experiments show TTFT and TPOT are comparable to Basic, but these metrics are measured at moderate load; at very low load (where there is no queuing), the extra hop could increase TTFT slightly. The paper does not report latency at low load to quantify this effect.
  • The compute network does not support hardware QoS (e.g., some Ethernet deployments without PFC, older InfiniBand without VL arbitration). In this case, the CNIC-centric traffic isolation cannot guarantee that KV-Cache transfers do not interfere with model execution collectives, and dual-path loading could degrade inference latency unpredictably.
  • The storage backend already provides sufficient aggregate bandwidth through caching or faster media (e.g., all KV-Cache fits in a distributed DRAM pool with bandwidth exceeding the combined SNIC throughput). In this case, the I/O bottleneck is already resolved, and dual-path loading addresses a problem that does not exist. However, Section 8.2's working set analysis suggests that for production agentic workloads with realistic tool call latencies, the KV-Cache working set grows quickly beyond practical DRAM sizes, making the "sufficient DRAM cache" scenario the exception rather than the rule for long-context agentic serving.