RDMA Point-to-Point Communication for LLM Systems

ArXiv: 2510.27656

🎯 Pitch

fabric-lib achieves the first viable MoE dispatch latencies on AWS EFA while matching specialized, vendor-locked alternatives on ConnectX-7—all from a single portable codebase. It also completes RL weight updates for trillion-parameter models in just 1.3 seconds, over 100× faster than prior frameworks.

1. Executive Summary

This paper introduces fabric-lib, a portable RDMA communication library that exposes a uniform point-to-point interface across heterogeneous hardware — specifically NVIDIA ConnectX-7 and AWS Elastic Fabric Adapter (EFA) — by building on reliable-but-unordered transport semantics and a novel IMMCOUNTER primitive for order-agnostic completion notification. The library is evaluated through three production systems: disaggregated inference KvCache transfer, reinforcement learning weight updates, and Mixture-of-Experts dispatch/combine kernels. On MoE dispatch/combine, fabric-lib achieves state-of-the-art decode latency on ConnectX-7 — matching or exceeding the specialized DeepEP library despite using a host proxy thread rather than GPU-initiated RDMA — and provides the first viable EFA implementation, while RL weight updates complete in 1.3 seconds for trillion-parameter models, over 100× faster than existing frameworks. The work establishes that portable point-to-point communication can complement collectives and avoid vendor lock-in for modern LLM workloads, though the host-proxy design incurs overhead that becomes noticeable only at very high expert parallelism (64 ranks) and the approach relies on the common denominator of unordered reliable delivery across NICs.

2. Context and Motivation

The Core Problem: LLM Infrastructure Is Locked to Single NIC Vendors

The fundamental gap this paper addresses is deceptively simple: there is no portable, production-grade point-to-point RDMA communication library for LLM systems that works across both NVIDIA ConnectX and AWS Elastic Fabric Adapter (EFA). This matters because modern LLM deployment patterns increasingly depend on flexible, low-latency point-to-point transfers that cannot be cleanly expressed through existing collective communication APIs, yet the available solutions for implementing such transfers are deeply tied to specific hardware.

The paper frames this problem through a specific set of architectural trends in LLM serving (Section 1, paragraph 1):

"Mixture-of-Experts (MoE) architectures are becoming the dominant approach for scaling model capacity... while disaggregated inference is emerging as the standard for production serving. These new architectures rely on communication patterns that are fundamentally different from traditional collective-based parallelism."

This is not a minor compatibility issue — it is a structural impediment to deploying LLMs in multi-cloud or heterogeneous hardware environments. A team that develops and tunes an MoE dispatch kernel for ConnectX-7 cannot deploy it on AWS p5 instances (which use EFA) without a complete rewrite. Conversely, a disaggregated inference system built for EFA cannot be ported to on-premise InfiniBand clusters. The paper argues that this vendor lock-in is increasingly untenable as cloud providers deploy their own proprietary RDMA solutions (Alibaba eRDMA, Google Falcon) alongside traditional ConnectX hardware.

Why This Problem Matters: Three Production Workloads That Need Point-to-Point

The paper motivates the need for portable point-to-point communication by presenting three concrete, production-critical communication patterns that cannot be effectively served by collective libraries like NCCL or torch.distributed.

Disaggregated inference (KvCache transfer). In a disaggregated setup, prefill and decode stages run on separate GPU clusters. When a prefill node finishes processing an input prompt, it must transfer the resulting KV cache pages to whichever decode node will handle that request. This is inherently point-to-point and dynamic: the destination is chosen per-request by a global scheduler, not predetermined at cluster initialization. The paper notes (Section 2.2) that collective libraries impose four constraints that make this pattern difficult:

• Fixed membership prevents dynamic scaling because all participants must be known in advance. In a disaggregated system, prefill and decode nodes may be added or removed elastically based on load, and the transfer pattern is any-to-any rather than all-to-all.
• Synchronized initialization requires global coordination to form communication groups, blocking independent peer connections. In practice, a prefill node must be able to send KV caches to a decoder node without waiting for the entire cluster to synchronize.
• Operation ordering requires all participants to agree on an operation sequence. For per-request transfers where tens of thousands of independent transfers are in flight simultaneously, global ordering is both unnecessary and crippling to throughput.
• Shape uniformity constrains transfer sizes across participants. KV cache page sizes vary with batch composition and sequence lengths, and forcing uniform buffer sizes wastes memory and bandwidth.

The paper argues that these constraints are not accidental — they are fundamental design choices in collective libraries optimized for dense, structured patterns like all-reduce in data parallelism. Point-to-point RDMA operations (WRITE, SEND/RECV) can bypass all four constraints, enabling the dynamic, any-to-any transfers that disaggregated inference requires.

Reinforcement learning weight updates. In asynchronous RL fine-tuning, training and inference run on separate GPU clusters. After each training step, new model weights must be pushed from training GPUs to inference GPUs. For trillion-parameter models (Kimi-K2 at 1T, DeepSeek-V3 at 671B, Qwen3 at 235B), this transfer involves tens to hundreds of GB of data per update.

Existing frameworks handle this through a Rank0-based approach (Section 5.1, Figure 4a): weights are gathered to training Rank0, then broadcast to inference Rank0, which then scatters them to the remaining inference GPUs. This creates a severe bottleneck at the NIC of training Rank0 — all weight data must flow through a single network interface. The paper reports that this approach takes "tens to hundreds of seconds" for trillion-parameter models, making rapid RL training iterations impossible. Their point-to-point alternative (Figure 4b) has each training GPU write its shard of the weights directly to the appropriate inference GPUs via one-sided RDMA WRITE, saturating all available NIC bandwidth simultaneously and achieving 1.3-second updates.

MoE dispatch/combine. Every token in an MoE layer must be routed (dispatched) to its selected experts and the results must be collected back (combined). For decode with 128 tokens per rank across 64 GPUs, this involves a scattered all-to-all communication pattern where each rank sends small numbers of tokens to a subset of peers. DeepEP (Zhao et al., 2025) demonstrated that specialized, GPU-initiated RDMA kernels can achieve extremely low latencies for this pattern, but their implementation relies on IBGDA (InfiniBand GPUDirect Async) and the mlx5 driver — features exclusive to ConnectX NICs. As the paper states (Section 6.4):

"DeepEP offers state-of-the-art latency, however they are tied to ConnectX due to their reliance on IBGDA and mlx5 driver."

Without a portable alternative, any deployment on EFA-instances cannot achieve competitive decode latencies for large expert-parallel MoE models. The paper notes that the only viable open-source portable alternative — pplx-kernels built on NVSHMEM — achieves 3–6× lower throughput than their solution on EFA (Table 6, Section 7.4.1).

Prior Approaches and Where They Fall Short

The paper identifies several existing libraries and explains why each fails to provide a portable, performant solution.

NCCL and torch.distributed (collectives). These are the dominant communication libraries in ML frameworks. NCCL provides highly optimized collective operations (all-reduce, all-gather, broadcast) but its point-to-point support is limited and comes with the fixed membership, synchronized initialization, ordering, and shape-uniformity constraints described above (Section 2.2). The paper acknowledges that NCCL does provide SEND and RECV primitives, but states:

"While these libraries offer SEND and RECV primitives for point-to-point communication, they often cannot be effectively composed to achieve viable latency." (Section 1, paragraph 3)

The problem is that these primitives were designed as building blocks for collective operations, not as a general-purpose point-to-point fabric.

NVSHMEM. This library (Langer et al., 2021) exposes both collective operations and flexible point-to-point communication, supporting both GPU-initiated (IBGDA) and host-proxy (IBRC) modes. It appears to be the closest existing work to fabric-lib's goals. However, the paper states (Section 2.3):

"It suffers from severe performance degradation on EFA."

This makes it non-viable for the portable use case. The microbenchmark results in Figures 9 and 10 confirm this: pplx-kernels (built on NVSHMEM) show latencies of 1,669–4,832 µs for decode dispatch on EFA, versus 286–317 µs for fabric-lib — an order of magnitude difference. For decode combine on EFA, the gap is similarly large: 1,058–4,033 µs versus 406 µs. This is not a modest regression; it makes real-time inference on EFA impractical (Table 6 shows pplx-kernels achieving only 4.9–20.9 tokens/s on H200 EFA versus 32–66 tokens/s for fabric-lib).

The paper does not deeply analyze why NVSHMEM performs poorly on EFA, but hints at the underlying issue in their design philosophy: NVSHMEM is built around the semantics of the OpenSHMEM partitioned global address space model, which assumes certain ordering and atomicity properties that map cleanly to ConnectX RC transport but conflict with EFA's unordered SRD protocol. The paper's key insight — that ordering assumptions are the fundamental incompatibility — explains why a library designed for RC semantics cannot simply be ported to EFA without a redesign.

DeepEP. This is the state-of-the-art for MoE dispatch/combine latency on ConnectX, relying on GPU-initiated RDMA (IBGDA) and careful use of RC queue pair ordering guarantees (Section 6.4). Its fundamental limitation is hardware lock-in: IBGDA is not available on EFA, and even if it were, DeepEP's architecture depends on the strong ordering guarantees of RC transport — tokens are transferred one-by-one over a queue pair with the assumption that they arrive in order, and completion is signaled via ATOMICs. On EFA's SRD protocol (which is unordered), this approach would break.

The paper also notes a subtle tradeoff in DeepEP's design: while using RC ordering and per-token transfers achieves lower latency to the first transfer, it results in more network packets and lower overall bandwidth utilization. The fabric-lib approach (bulk transfers after batching) trades slightly higher initial latency for better bandwidth utilization and portability.

Mooncake Transfer Engine. Mooncake (Qin et al., 2025) provides an RDMA transfer engine for its disaggregated inference system, but the paper notes it "lacks EFA support" (Section 2.3). Since Mooncake was developed and deployed on InfiniBand clusters, its transfer engine was presumably built against libibverbs with RC semantics, making EFA support a non-trivial porting effort rather than a recompilation.

NIXL (NVIDIA Inference Xfer Library). NIXL is NVIDIA's point-to-point communication library for LLM inference, built on UCX (Shamis et al., 2015). The paper acknowledges preliminary EFA support in NIXL v0.6.1 (October 2025) but notes:

"Our production-deployed EFA implementation predates the preliminary EFA support in NIXL." (Section 2.3)

The evaluation in Figure 8 does include NIXL-EFA and NIXL-CX7 comparisons, showing that fabric-lib and NIXL achieve relatively close performance for point-to-point transfers. However, the paper's focus is on a library that is already production-hardened across three workloads, whereas NIXL's EFA support is characterized as "preliminary." The paper also makes a broader point: even when NIXL matures, the ecosystem would benefit from a vendor-neutral alternative that is not tied to NVIDIA's development priorities.

UCCL and MSCCL++. These libraries (Zhou et al., 2025; Shah et al., 2025) focus on network-layer optimizations for collective operations rather than point-to-point communication. They are orthogonal to fabric-lib's goals.

The Fundamental Technical Barrier: Ordering Assumptions

The paper's key diagnostic insight is that the root cause of non-portability is the assumption of in-order delivery in existing libraries. Table 1 (Section 2.1) illustrates this: Reliable Connection (RC) transport — used by ConnectX in its default mode — guarantees both reliability and in-order delivery. Unreliable Connection (UC) and Unreliable Datagram (UD) drop reliability. EFA's SRD protocol provides reliability but explicitly does not guarantee in-order delivery. This means:

• Libraries built on RC semantics (assuming that if WRITE A was issued before WRITE B, the receiver will see A's data before B's completion notification) work correctly on ConnectX but will have undefined behavior on EFA.
• Libraries built on unordered semantics work correctly on both if they do not rely on ordering for correctness, but must be designed from scratch with this constraint.

The paper's approach — building the entire TransferEngine on reliable-but-unordered semantics — exploits the observation that this is the common subset between RC (with ordering relaxed) and SRD. Table 1 highlights fabric-lib as "the common ground between them": it occupies the intersection of RC (with ordering ignored) and SRD capabilities.

The IMMCOUNTER primitive is the mechanism that makes this possible. Rather than relying on message ordering to know when a transfer has completed, the sender attaches a 32-bit immediate value to each WRITE (using the WRITEIMM operation). The receiver's NIC delivers this immediate value through a completion queue once the WRITE's data payload has been fully delivered to the target memory, guaranteeing atomicity. The receiver maintains per-immediate counters that are incremented when these completions are polled, providing an ordering-agnostic completion signal. Crucially, this works because PCIe ordering guarantees provide the lower-level atomicity needed: the data payload targets GPU memory while the immediate value targets the CPU, but the host-proxy architecture ensures that after the CPU observes the IMMCOUNTER increment, any subsequent CPU-to-GPU signal is ordered after the preceding NIC-to-GPU data writes, guaranteeing the data is visible to the GPU (Section 3.3, last paragraph).

How This Paper Positions Itself

The paper positions fabric-lib not as a replacement for collective libraries but as a complementary communication primitive for patterns that collectives handle poorly. The abstract states:

"We demonstrate that our portable point-to-point communication complements collectives while avoiding lock-in."

This is an important framing: the paper is not arguing against NCCL for all-reduce in data parallelism, but rather arguing that the ML systems community needs both collectives and portable point-to-point, and that the latter has been systematically underinvested in because the dominant RDMA programming model was assumed to be RC-specific.

The paper also positions itself as production-proven rather than a research prototype. All three systems (KvCache transfer, RL weight updates, MoE dispatch/combine) are described as production-deployed, and the evaluation emphasizes end-to-end metrics (tokens/second for decode, time-to-first-token for prefilling, weight update wall-clock time) rather than just microbenchmarks. The open-source release (linked in the abstract) and the use of PyTorch and a custom inference engine rather than a simulation framework reinforce this positioning.

Finally, the paper positions its host-proxy architecture as a deliberate tradeoff for portability, not a limitation. GPU-initiated RDMA (IBGDA) would reduce MoE dispatch latency by removing the CPU from the critical path, but it is not available on EFA. By accepting this constraint and optimizing the host-proxy path aggressively (Table 8 shows the CPU overhead from application call to first RDMA WRITE is under 1.5 µs at p50), the paper achieves competitive performance while running on hardware that GPU-initiated approaches cannot support. The discussion (Section 8) explicitly addresses this tradeoff, noting that for KvCache and RL weight transfers, RDMA latency is already hidden by compute, so the CPU-based approach "frees the GPU without sacrificing end-to-end performance." For MoE, the paper argues that next-generation hardware with wide NVLink domains (e.g., GB200 NVL72) shifts communication off RDMA entirely, so the host-proxy overhead is a temporary concern for current-generation hardware.

3. Technical Approach

3.1 Reader Orientation

fabric-lib is a library that lets different computers in a GPU cluster talk directly to each other's memory over the network — without the CPU getting in the way — using a single API that works identically whether you're running on NVIDIA's ConnectX network cards or Amazon's Elastic Fabric Adapter. The problem it solves is that modern LLM serving patterns (disaggregated inference, MoE routing, RL weight updates) need flexible, any-to-any data transfers that existing collective libraries can't express efficiently, while the few libraries that can express them only work on one vendor's hardware. The solution's shape is a transfer engine that takes the common subset of what all RDMA hardware supports — reliable delivery without ordering guarantees — and builds completion notification on top of a counter-based mechanism (IMMCOUNTER) that doesn't depend on messages arriving in the order they were sent.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three layers:

1. TransferEngine — the core library component. It spawns one worker thread per GPU, each managing a DOMAINGROUP that coordinates all NICs attached to that GPU (1 NIC for ConnectX-7, 2–4 NICs for EFA). Within a group, each DOMAIN is specialized to the specific hardware (libibverbs for ConnectX, libfabric for EFA) and handles queue pair management, work submission, and completion polling. The engine exposes SEND/RECV (two-sided RPC-style messaging), WRITE (one-sided memory-to-memory transfer), and WRITEIMM (one-sided transfer with a 32-bit notification value).

2. IMMCOUNTER — a completion notification subsystem layered over WRITEIMM. Rather than relying on in-order message delivery to know when data has arrived, each WRITE carries a 32-bit immediate value. The receiver polls its NIC's completion queue for these values, incrementing per-immediate counters. Applications register callbacks (via expect_imm_count) that fire when a counter reaches a target value, providing ordering-agnostic completion signaling.

3. Three production systems built on the engine:

   • KvCache Transfer — disaggregated inference where prefill nodes WRITE KV cache pages and tail context to decoder nodes' GPU memory, using UVM watchers to overlap transfer with layer-by-layer computation.
   • RL Weight Transfer — asynchronous RL fine-tuning where each training GPU WRITEs its parameter shards directly to inference GPUs, with pipelined stages (H2D memcpy → unshard → quantize → RDMA WRITE → barrier) overlapping across parameters.
   • MoE Dispatch/Combine — low-latency kernels where a host proxy thread coordinates GPU token shuffling with NIC scatter operations, using private per-source buffers to hide routing-information exchange latency.

Information flows: the application submits transfer requests to the TransferEngine → requests are forwarded to the appropriate DOMAINGROUP worker thread → the worker shards work across its DOMAINs (NICs), posting work requests to hardware send queues → completion queues are polled for finished transfers → completions trigger either callbacks (for SEND/RECV) or IMMCOUNTER increments (for WRITEIMM) → application-level notification fires.

3.3 Roadmap for the Deep Dive

• First, the TransferEngine's design and hardware abstraction — how it maps the common reliable-but-unordered transport subset across ConnectX RC and EFA SRD, manages multiple NICs per GPU, and exposes a uniform API.
• Second, IMMCOUNTER and completion notification — the mechanism that replaces ordering-based completion with counter-based notification, including why PCIe ordering guarantees make it correct.
• Third, the API surface — memory registration, two-sided SEND/RECV, one-sided WRITE variants (contiguous, paged, scatter/barrier), and the UvmWatcher for GPU-to-CPU signaling.
• Fourth, hardware-specific implementation details — the different strategies for ConnectX (dual QP pairs, WR chaining, relaxed ordering) versus EFA (libfabric templating, descriptor management, multi-NIC sharding).
• Fifth, how each production system (KvCache, RL weights, MoE) maps its communication pattern onto the TransferEngine API — the specific WRITE configurations, notification strategies, and pipelining choices.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems engineering paper whose core idea is that portable high-performance point-to-point RDMA communication can be achieved by identifying the common subset of transport capabilities across heterogeneous NIC hardware — reliable delivery without ordering guarantees — and building completion notification on a counter-based mechanism that does not depend on message arrival order.

The Common Transport Abstraction: Reliable-But-Unordered

The TransferEngine's most fundamental design decision is to build on reliable-but-unordered delivery semantics. This is not an arbitrary choice — it is the precise intersection of what ConnectX RC and EFA SRD can both provide.

Table 1 in the paper lays out the transport capabilities. Standard RDMA Reliable Connection (RC) provides three guarantees: (1) reliability (delivery is guaranteed, packets are retransmitted on loss), (2) ordering (messages arrive in the order they were sent), and (3) connection-oriented operation (a dedicated queue pair is established between each pair of communicating peers). Unreliable Connection (UC) drops reliability, Unreliable Datagram (UD) drops both reliability and connection. EFA's Scalable Reliable Datagram (SRD) is non-standard: it provides reliability and is connectionless (like UD), but explicitly does not guarantee in-order delivery.

The key insight: ConnectX RC can ignore ordering if the application doesn't depend on it. The paper notes (Section 3.5):

"We enable IBV_ACCESS_RELAXED_ORDERING to permit out-of-order PCIe transactions between the NIC and GPU memory, reducing latency."

This means RC can be made to behave like SRD at the application level — reliable but unordered — while SRD cannot be made to behave like RC (it fundamentally cannot guarantee ordering). Therefore, building on the reliable-but-unordered intersection means the library works on both, rather than targeting the RC superset and breaking on EFA.

The implication: all completion notification must be explicit. The library cannot assume that if WRITE_B is completed, WRITE_A (issued before it) has also completed. This is what motivates the IMMCOUNTER design — every completion event carries its own notification, independent of any other event's arrival.

IMMCOUNTER: Ordering-Agnostic Completion Notification

The IMMCOUNTER is the TransferEngine's mechanism for notifying applications that one-sided WRITEs have completed, without relying on message ordering. Here is how it works, step by step:

1. WRITEIMM operation. The sender issues a one-sided RDMA WRITE that copies data from local memory (host or GPU) to a remote memory region, attaching a 32-bit immediate value. The RDMA specification guarantees that the immediate value is delivered after the data payload — the NIC will not generate a completion for the immediate until all data has been written to the target memory.

2. Completion queue polling. The receiver's DOMAIN worker continuously polls the completion queue of its NIC. When a WRITEIMM completion appears, the worker extracts the 32-bit immediate value.

3. Counter increment. The worker looks up a per-immediate counter table, increments the counter associated with that immediate value, and checks whether the counter has reached a pre-registered target count.

4. Callback delivery. If the target count is reached, the worker signals the registered callback (submitted via expect_imm_count), which notifies the application that all transfers for that logical operation are complete.

The application-level usage pattern (Section 3.3) is:

"expect_imm_count registers a callback invoked when the sum of the immediate values received reaches a target count."

The caller allocates a unique imm value using alloc_imm(), records the number of expected WRITEs (imm_count), and calls te.expect_imm_count(imm, imm_count, callback). Each WRITE that belongs to this logical operation carries the same imm value. When the counter reaches imm_count, the callback fires once.

Why this works on unordered transport: The RDMA specification requires that the data payload of a WRITEIMM is issued before the immediate value, but does not require ordering across different WRITEIMM operations. On EFA SRD, two WRITEs with different immediate values may arrive in any order — but each individual WRITE's data is guaranteed to be in GPU memory before its corresponding immediate value appears in the completion queue. The IMMCOUNTER only cares about per-immediate counts, not the relative order of different immediates.

The PCIe ordering connection: A subtle correctness issue arises in the host-proxy architecture. The immediate value is delivered to the CPU (via the completion queue), while the data payload is written directly to GPU memory via PCIe. How does the CPU know the GPU can see the data? The paper explains (Section 3.3, last paragraph):

"After the CPU observes the target IMMCOUNT, any subsequent CPU-to-GPU transaction (e.g., launching a kernel or updating a flag via GDRCopy) is ordered by the PCIe switch after the preceding NIC-to-GPU data writes, ensuring the payload is visible to the GPU."

In other words, PCIe ordering rules guarantee that writes targeting the same device are not reordered by the PCIe switch. Once the CPU has observed the completion, any subsequent write to GPU memory (or kernel launch) will occur after the NIC's data write, so the GPU will see the transferred data.

Design choice — why IMMCOUNTER instead of per-transfer callbacks: The IMMCOUNTER aggregates multiple WRITEs under a single counter, avoiding the overhead of firing a callback for every individual transfer. This matters when transfers are numerous and small — for example, a KvCache transfer for a 128K-token sequence with 95 layers and 8 chunks involves 95 × 8 = 760 individual page WRITEs. Firing 760 callbacks would add significant CPU overhead; a single aggregated callback after the counter reaches 760 avoids this.

Memory Registration: Exchanging Remote Access Capabilities

Before any WRITE can occur, the target memory must be registered with the NIC so it can be accessed remotely. The TransferEngine's memory registration API (reg_mr) abstracts this across hardware:

fn reg_mr(ptr, len, device) -> (MrHandle, MrDesc);

What it produces:

• MrHandle — an opaque local handle used as the source of transfers. It carries the addresses of all NICs associated with the device.
• MrDesc — a serializable descriptor that can be exchanged with remote peers. It contains:
  • ptr: u64 — the base address of the registered memory.
  • rkeys: Vec<(NetAddr, u64)> — a list of (network address, remote key) pairs, one per NIC. Each pair allows the peer to access this memory through a specific NIC.

The multi-NIC challenge: On EFA, a single GPU is typically attached to 2 or 4 NICs, each providing 100–200 Gbps. To achieve the full 400 Gbps aggregate bandwidth, a WRITE must be sharded across all available NICs. The MrDesc captures this by including one remote key per NIC address. The paper notes (Section 3.2):

"As a restriction, all peers must use the same number of NICs per GPU. Consequently, any transfer has full knowledge of the NICs between the source and destination domain, allowing the TransferEngine to shard or balance the request."

This symmetry assumption means the sender knows exactly which NICs to target on the receiver and can split a large WRITE into equal-sized chunks, posting one chunk per NIC.

Why the (NetAddr, RKEY) pairs are needed: In standard RC RDMA, a single queue pair connects two peers, and the remote key is associated with that QP. In EFA SRD, there is no persistent connection — each WRITE must specify both the target address and the key. The list-of-pairs representation captures this by explicitly associating each key with the specific NIC address it is valid for.

The TransferEngine API Surface

Figure 2 provides the full API. I will walk through each operation group and explain what it provides, when it is used, and any non-obvious design choices.

Two-Sided SEND/RECV (RPC-style messaging):

fn submit_send(addr: NetAddr, msg: &[u8], cb: fn () -> ());
fn submit_recvs(len: u64, cnt: u64, cb: fn (&[u8]) -> ());

These implement a low-latency RPC mechanism. submit_send copies the message into an internal buffer (allowing the caller to reuse or free the original immediately) and sends it to the specified address. submit_recvs pre-posts a rotating pool of receive buffers; when a message arrives, one buffer is taken out of the pool and the callback is invoked with a reference to the received data. After the callback returns, the buffer is automatically re-posted.

Design constraint — single NIC for SEND/RECV: The paper states (Section 3.3):

"These operations utilize only the first NIC in a domain group."

This is because SEND/RECV on multiple NICs would require coordinating which NIC receives which message — a routing problem that adds complexity without benefit, since SEND/RECV is used for small control messages where a single NIC's bandwidth is not the bottleneck.

Usage: In the disaggregated inference workflow, the decoder uses submit_send to dispatch a request to the prefiller, including the decoder's memory descriptors and page indices so the prefiller can WRITE directly into the decoder's GPU memory (Appendix A, Figure 13). The prefiller continuously polls with submit_recvs for incoming dispatch requests.

One-Sided WRITE (bulk data transfer):

fn submit_single_write(len: u64, imm: Option<u32>,
    src: (MrHandle, Offset), dst: (MrDesc, Offset), OnDone);
fn submit_paged_writes(page_len: u64, imm: Option<u32>,
    src: (MrHandle, Pages), dst: (MrDesc, Pages), OnDone);

submit_single_write transfers a contiguous range from a local memory region to a remote memory region. The optional imm attaches a 32-bit immediate value (for IMMCOUNTER-based completion notification); if None, no notification is delivered to the receiver's completion queue.

submit_paged_writes transfers data organized as pages — non-contiguous slices described by indirect indices, strides, and offsets. The Pages struct contains:

struct Pages { indices: Vec<u32>, stride: u64, offset: Offset }

For each page index i, the engine copies page_len bytes starting at src_offset + i * stride to dst_offset + i * stride. This is the workhorse for KvCache transfers, where KV cache entries are stored in page-sized blocks that may be non-contiguous (allocated from a memory pool with fragmentation).

Both operations translate into one or more zero-copy RDMA WRITE operations, sharded across the available NICs. The OnDone parameter specifies how completion is reported — either a callback or an atomic flag.

Scatter and Barrier (multi-peer operations):

fn add_peer_group(addrs: Vec<NetAddr>) -> PeerGroupHandle;
fn submit_scatter(h: Option<PeerGroupHandle>, OnDone,
    imm: Option<u32>, src: MrHandle, dst: Vec<ScatterDst>);
fn submit_barrier(h: Option<PeerGroupHandle>, OnDone,
    imm: u32, dst: Vec<MrDesc>);

These are optimized wrappers around WRITE for patterns where one rank sends data to many peers. add_peer_group pre-registers a set of peer addresses, amortizing the cost of address resolution across many transfers. submit_scatter sends a slice of the source buffer to each peer's destination buffer — a single call issues one WRITE per peer, each targeting a different offset in the peer's receive buffer. submit_barrier is an immediate-only operation (zero-length WRITE with only the immediate value) that notifies all peers without transferring data.

Usage in MoE routing: During dispatch, each rank calls submit_scatter twice — once to send routing information (expert assignments) to all peers, and once to send token data. During combine, each rank calls submit_scatter once to send expert outputs back. The PeerGroupHandle avoids recomputing peer addresses for every token batch, since the set of peers is static for a given expert-parallel configuration.

UVM Watcher (GPU-to-CPU notification):

fn alloc_uvm_watcher(cb: fn(u64, u64) -> ()) -> NonNull<u64>;

This allocates a Unified Virtual Memory (UVM) location — a word in memory accessible to both the CPU and GPU. A dedicated CPU thread polls this location using GDRCopy (a library for low-latency host-to-device memory access over PCIe). When the value changes, the callback is invoked with the old and new values.

The paper notes (Section 3.3):

"Since not all changes are guaranteed to be observed immediately, the callback is invoked with the old and the new values, allowing it to respond to GPU-side progress."

This means the callback sees a range of change rather than a sequence of individual updates — if the GPU increments the value from 0 to 5 rapidly, the CPU might observe 0 → 3 → 5, missing increments 1, 2, and 4. The callback receives (0, 3) and then (3, 5), and can infer that the value passed through all intermediate states.

Usage: In the disaggregated inference prefiller (Appendix A, Figure 15), the GPU kernel increments the watcher value after completing the attention output projection for each layer (using scalar_inc_ inside a CUDA graph). The CPU callback fires and initiates the RDMA WRITE for that layer's KV pages while the GPU proceeds to the next layer. This is a form of computation-communication overlap: the GPU doesn't wait for the WRITE to complete, and the CPU doesn't need to know exactly when the GPU finishes — only that it has progressed past a certain layer.

Threading and Memory Architecture

The TransferEngine's threading model is designed to minimize both scheduling jitter and memory access latency (Section 3.4).

Worker-per-GPU, pinned to NUMA node. One worker thread is spawned per DOMAINGROUP (one group per GPU). The thread is pinned to a CPU core on the same NUMA node as the GPU and its attached NICs. This matters because:

• Scheduling latency: An unpinned thread can be migrated by the OS scheduler, adding unpredictable delays in the critical path between work submission and completion polling. For MoE dispatch where the total proxy overhead is tens of microseconds, even a single scheduling delay would be catastrophic.

• Memory access latency: RDMA completion queues, work queues, and buffer metadata are allocated in host memory. If the worker thread accesses memory on a remote NUMA node, each access incurs cross-socket latency (tens to hundreds of nanoseconds). By allocating all per-domain data structures after pinning to the correct NUMA node, all memory accesses remain local.

Data structure allocation after pinning. The paper states (Section 3.4):

"Data structures specific to a domain are allocated after pinning to ensure that memory is reserved in the correct NUMA node."

This is a critical implementation detail. On Linux, memory allocation (malloc, mmap) follows a first-touch policy: the physical page is allocated on the NUMA node of the CPU that first writes to it. If a data structure is allocated before the thread is pinned, the physical memory may end up on the wrong NUMA node, silently degrading performance.

Dedicated callback thread. While each DOMAINGROUP has its own worker for work submission and completion polling, a separate shared thread handles callback delivery (Section 3.4):

"Events are aggregated to deliver per-transfer notifications, handing the transfer over to a dedicated callback thread shared by all groups."

This separation prevents application callbacks from blocking the work submission/polling loop. If a callback takes a long time — for example, the decoder's IMMCOUNT callback that triggers the start of the decode loop — the worker thread can continue polling and submitting work for other transfers.

Dedicated UVM watcher thread. A separate thread (shared across all groups) polls UVM watcher locations using GDRCopy. This thread is independent of the per-GPU workers, so UVM polling does not compete with completion queue processing.

Lock-free cross-thread communication. The API forwards requests to worker threads through lock-free queues. This eliminates the risk of priority inversion — a low-priority application thread holding a mutex that a high-priority worker thread needs — and avoids the overhead of kernel-mediated synchronization (futex, mutex) in the hot path.

Work loop priority: The domain worker's main loop has three phases, ordered by priority (Section 3.4):

1. Poll for new requests — submit new WRITEs, SENDs, or RECVs immediately. The first WRITE of a composite request is posted to the NIC's send queue in this phase.
2. Progress pending requests — for requests that were too large to post all at once (limited by send queue depth), post additional WRITEs to fill the hardware pipeline.
3. Poll completion queues — query for finished transfers and immediate counter increments.

This ordering ensures that new work is initiated immediately (reducing latency for the application) while still making forward progress on in-flight transfers.

Hardware-Specific Optimizations: ConnectX-7 (libibverbs)

The ConnectX-7 DOMAIN implementation uses libibverbs and makes several design choices specific to RC transport (Section 3.5).

Dual queue pairs per peer. Two RC queue pairs are created for each remote peer:

• One QP for two-sided SEND/RECV operations.
• A separate QP for one-sided WRITE and WRITEIMM operations.

Why this separation is necessary: RC queue pairs process work requests in posting order. If SEND/RECV and WRITEIMM shared a QP, a RECV work request (posted to the QP to receive incoming messages) would consume completions that belong to WRITEIMM — the completions for both operation types arrive in the same completion queue, and the application cannot distinguish them. By separating the QPs, the paper provides (Section 3.5):

"This separation is necessary because both RECV and WRITEIMM completions consume work requests in posting order. This allows us to provide high-level RECV semantics while supporting WRITEIMM without interference."

Work request chaining. The ConnectX implementation chains up to 4 work requests using the next pointer field in ibv_send_wr:

"We employ WR chaining by linking up to 4 work requests through the next pointer of ibv_send_wr, reducing the number of doorbell rings to the NIC."

A doorbell ring is a PCIe write that notifies the NIC that new work requests are available in the send queue. Each write involves PCIe transaction overhead. By batching up to 4 work requests under a single doorbell, the CPU overhead per WRITE is approximately quartered. This is particularly important for paged WRITEs, where each page requires a separate work request.

Relaxed PCIe ordering. The ConnectX QPs are created with IBV_ACCESS_RELAXED_ORDERING:

"We enable IBV_ACCESS_RELAXED_ORDERING to permit out-of-order PCIe transactions between the NIC and GPU memory, reducing latency."

Without this flag, PCIe enforces strict transaction ordering — the NIC's writes to GPU memory must be processed in the exact order they were issued. This can create pipeline stalls: if one write targets a congested PCIe path, subsequent writes to different paths are blocked. Relaxed ordering lets the PCIe switch reorder non-conflicting writes, increasing throughput and reducing tail latency.

Connection establishment via UD. Rather than assuming all peers are pre-configured, the ConnectX implementation uses Unreliable Datagram (UD) queue pairs for connection establishment:

"For each peer, we use an UD queue pair to exchange RC handshakes."

A UD QP can send to any destination without pre-establishing a connection, making it suitable for the initial handshake where peers exchange their QP numbers, addresses, and keys. Once the RC QPs are established, the UD QP is no longer needed.

Hardware-Specific Optimizations: AWS EFA (libfabric)

The EFA DOMAIN implementation uses libfabric and must work around two EFA-specific quirks (Section 3.5).

Zero-length write descriptor enforcement. The RDMA specification does not require a valid target descriptor (memory region, address, key) for immediate-only, zero-sized writes — since no data is transferred, the target memory parameters are irrelevant. However:

"Since EFA diverges from the RDMA spec which does not require a valid target descriptor for immediate-only zero-sized writes, we enforce valid descriptors for all transfers."

This is a workaround: by always providing valid (though unused) memory descriptors even for zero-length WRITEs, the library avoids triggering EFA-specific error handling. The paper does not describe what EFA does with invalid descriptors (likely rejects the WRITE or generates a completion error), but the defensive approach ensures correctness without depending on EFA's specific behavior.

Work request templating for bulk transfers and peer groups. EFA's libfabric API uses descriptor structures that contain fields for the target address, key, data length, and flags. For bulk transfers and scatter operations where many WRITEs target the same destination with identical parameters (only the offset changes), the TransferEngine pre-populates and retains the common fields:

"We employ work request (WR) templating, pre-populating and retaining the common fields of libfabric descriptors before posting."

This reduces per-WRITE CPU overhead because only the varying fields (offset, immediate value) need to be updated before posting, rather than filling in the entire descriptor from scratch. For MoE scatter with 56 inter-node peers (at EP64), this templating saves 56 × (descriptor size minus varying fields) bytes of memcpy per dispatch.

Multi-NIC sharding strategy. EFA on AWS p5 instances provides 4 × 100 Gbps NICs per GPU (or 2 × 200 Gbps on p5en). The TransferEngine's DOMAINGROUP manages all NICs associated with a GPU. The sharding strategy is described as flexible (Section 3.4):

"Sharding inside a DOMAINGROUP is flexible. Transfers can target specific NICs by index. A single WRITE can be split. Paged transfers, scatter and barrier operations, which all translate to multiple WRITEs, can shard across all NICs."

For a single large WRITE, the engine splits the source buffer into equal chunks and posts one WRITE per NIC. For paged WRITEs, individual pages are distributed round-robin across NICs. The paper does not specify whether NIC selection for paged transfers uses static round-robin or dynamic load balancing, but the round-robin strategy is implied by the symmetry assumption (all NICs have equal bandwidth). Dynamic load balancing would require tracking per-NIC queue depth, which adds overhead that is unlikely to be justified for the mostly-symmetric workloads fabric-lib handles.

KvCache Transfer: Mapping Disaggregated Inference onto the TransferEngine

The KvCache transfer system (Section 4, Appendix A) uses the following TransferEngine operations:

Decoder side:

1. Allocate KV pages and tail context from GPU memory pool.
2. Register memory regions with the TransferEngine (reg_mr) to obtain MrDesc descriptors.
3. Set up an IMMCOUNTER callback using expect_imm_count with the expected number of WRITEIMM completions (one per layer per chunk plus one for tail context).
4. Send a dispatch request to the chosen prefiller using submit_send, including the decoder's MrDesc, allocated page indices, and the imm value for completion notification.
5. Block until the IMMCOUNTER fires, then start autoregressive decoding.

Prefiller side:

1. Continuously poll for dispatch requests using submit_recvs.
2. On receiving a dispatch, allocate local KV pages and tail context, then set up a UVM watcher callback.
3. During model forward pass, after each layer's attention output projection, the GPU kernel increments the UVM watcher value using scalar_inc_ (compatible with CUDA graphs).
4. The UVM watcher callback fires and submits submit_paged_writes for that layer's KV pages, targeting the decoder's memory descriptors.
5. After the final layer, copy tail context (logits, hidden states) to the tail buffer and submit a submit_single_write for the tail transfer.
6. Wait for all local RDMA submissions to complete (the prefiller doesn't need to wait for the decoder to acknowledge receipt — the decoder has its own IMMCOUNTER).

Page layout optimization for transfer efficiency: The paper describes (Section 4, final paragraph) a deliberate memory layout choice that minimizes the number of WRITEs:

"To minimize the number of writes and ensure that individual writes are sufficiently large, the KvCaches are laid out with heads preceding the pages, ensuring continuity within consecutive heads."

With GQA (Grouped Query Attention), a single page contains tokens for multiple heads. If the layout were page-major (all heads of page 0, then all heads of page 1), a WRITE of a subset of pages would require many small scattered WRITEs. By laying out data as head-major within each page group, consecutive pages for the same head are contiguous in memory, so a single WRITE can cover multiple pages. The paper also describes using "page-wise offsets and strides to select slices from the source KvCache to copy into corresponding offsets in the destination KvCache" — this uses the Pages struct's indices, stride, and offset fields to specify exactly which slices to transfer.

MLA replication handling: For Multi-head Latent Attention (MLA), compressed KvCache entries are replicated across tensor-parallel ranks. The paper states:

"Under such a scheme, prefiller ranks are randomly matched with decoder ranks to balance the transfers of replicas."

This means each decoder rank needs one replica of each KV entry, and the scheduler randomly assigns which prefiller rank provides it, balancing the load evenly across prefiller NICs.

Cancellation and error handling: The paper mentions (Section 4, last paragraph) that cancellation requires confirmation:

"Cancellation triggered by a decoder must be explicitly confirmed by the prefiller, as the KV pages cannot be reused as long as there is a possibility of a remote write clobbering them."

This is because one-sided WRITEs are fire-and-forget from the receiver's perspective. The decoder might have requested cancellation, but the prefiller may have already posted WRITEs that are in-flight in the NIC's send queue. Until the prefiller's NIC confirms that all WRITEs have been sent (or flushed), the decoder's KV pages are at risk of being overwritten by stale data. The paper uses heartbeat messages between prefillers and decoders to detect transport-layer failures — if a prefiller is unresponsive, the decoder cancels after a timeout since no new transfers can arrive.

RL Weight Transfer: Mapping Parameter Updates onto the TransferEngine

The RL weight transfer system (Section 5, Appendix B) uses a fundamentally different communication pattern — bulk one-sided WRITEs with all coordination done upfront.

Static routing computation. Before any training begins, a controller process:

1. Gathers parameter metadata from all training and inference GPUs — for each parameter, its name, shape, dtype, and DTensor sharding (which GPU owns which slice).
2. Computes a static mapping: for each training GPU, which parameters it owns, and for each of its parameters, which inference GPU(s) need to receive them.
3. Broadcasts this routing table to all training GPUs. The routing table is used unchanged for every training step — this works because the model architecture and parallelism configuration are static.

Transfer execution per step. For each training step, each training GPU:

1. H2D memcpy — if FSDP offloads weights to CPU, copy the weight from host to device memory.
2. Parameter preparation — call full_tensor() to unshard FSDP parameters (reconstructing the full tensor from the local shard using all-gather), apply projection fusion (combining multiple linear projections into a single matmul), and optionally quantize from bf16 to fp8.
3. RDMA WRITE — issue one or more submit_single_write operations for the parameter, targeting the appropriate inference GPU memory regions (pre-registered via reg_mr).
4. Global barrier — after all full_tensor() calls for the current MeshGroup are done, synchronize across all ranks using GLOO over Ethernet.

Pipelining mechanism. The paper describes a four-stage pipeline (Figure 5) that overlaps these operations across different parameters:

• While parameter A's RDMA WRITE is in-flight, parameter B's full_tensor() runs on the GPU, and parameter C's H2D memcpy runs on the CPU.
• The pipeline is managed by a watermark: a configurable limit on the total temporary GPU memory occupied by in-flight tasks. A new task is only started if the current in-flight tasks consume less GPU memory than the watermark.

"To avoid out-of-memory errors, we only start a new task if the current in-flight tasks occupy less temporary GPU memory than a configurable watermark." (Section 5.2)

The temporary GPU memory comes from the full_tensor() operation, which allocates a buffer for the unsharded parameter. For trillion-parameter models, individual parameters can be large (hundreds of MB for MoE expert layers), so concurrent full_tensor() calls would exceed GPU memory without the watermark.

MeshGroup sequencing. Different parameter types use different FSDP sharding strategies (e.g., MoE parameters may use expert parallelism while dense parameters use pure data parallelism). Each sharding strategy partitions the global DeviceMesh into disjoint sub-meshes (MeshGroups). Parameters within a MeshGroup are transferred in parallel, while MeshGroups are processed sequentially. This avoids contention between different all-gather operations that would arise if parameters with different sharding configurations were unsharded simultaneously.

The "remaining 42 ms" and why RDMA is on the critical path but not dominating. Table 5 shows the per-rank latency breakdown: 518 ms for full_tensor(), 357 ms for synchronization, and only 42 ms of "extra RDMA time not hidden by other operations." The total wall-clock time is 1,233 ms. This means the pipelining is highly effective — RDMA transfers are almost entirely overlapped with parameter preparation and synchronization. The 42 ms represents WRITEs that were posted after the last full_tensor() completed and before the global barrier — essentially the "tail" of the pipeline that cannot be overlapped with other work.

Why one-sided WRITE instead of SEND: The inference nodes do not participate in the transfer — they are passive targets. The training nodes write directly to inference GPU memory using submit_single_write. This avoids involving the inference node's CPU, which might be busy serving inference requests. The paper states:

"The inference nodes remain unaware of the transfer, as it uses one-sided operations."

Comparison with Rank0-based approach: In the Rank0-based approach (Figure 4a), all weights from all training GPUs are gathered to a single training Rank0, which then broadcasts to a single inference Rank0, which then scatters to all inference GPUs. The network bandwidth is bottlenecked by the single NIC of training Rank0. The paper's P2P approach (Figure 4b) has each training GPU write directly to the appropriate inference GPUs, utilizing all NICs simultaneously. For 256 training GPUs with 400 Gbps NICs each, the aggregate bandwidth is 256 × 400 Gbps = 102.4 Tbps, versus 400 Gbps for the Rank0 bottleneck — a 256× improvement in available bandwidth.

MoE Dispatch/Combine: Low-Latency Scatter with a Host Proxy

The MoE dispatch/combine kernels (Section 6) are the most latency-sensitive use of the TransferEngine. The key challenge is minimizing the time from when the GPU finishes computing token-to-expert assignments to when the first RDMA transfer begins, while staying within the host-proxy constraint (no GPU-initiated RDMA).

Overall coordination model (Figure 6): A host proxy thread runs alongside the GPU kernels, using GDRCopy to poll GPU memory for progress. The GPU sends signals by writing to unified memory locations; the proxy reads these locations and invokes the TransferEngine to submit RDMA transfers. The proxy does not initiate work — it only reacts to GPU signals, ensuring that transfers never start before the source data is ready.

Dispatch protocol (two-phase scatter):

Phase 1 — Routing information exchange and speculative token transfer.

1. The dispatch send kernel counts the number of tokens routed to each expert in shared memory, writes the counts to unified memory, and signals the host proxy.
2. The host proxy uses submit_scatter (via the TransferEngine) to send the routing information (per-expert token counts) to all peer ranks. Simultaneously, the GPU copies up to a fixed limit of tokens into private per-source send buffers.
3. The host proxy initiates a second submit_scatter to write these initial tokens into private per-source receive buffers on each peer.
4. The paper reports that "the latency from the launch of the dispatch kernel to the first transfer is around 15 µs assuming EP=64" (Section 6.2). This includes GPU kernel launch, shared memory counting, unified memory write, GDRCopy polling, and the host proxy enqueuing the first scatter — all under 15 µs.

Phase 2 — Bulk token transfer.

1. On the receiver side, once the routing information arrives and the private receive buffers are populated, each rank can compute the position where each source rank's tokens should be placed in a shared contiguous receive buffer.
2. The remainder of the tokens (beyond the initial speculative transfer) are scattered into this contiguous buffer. The paper states: "The host-side work to process routes and dispatch the second round of transfers takes tens of microseconds" (Section 6.2).
3. The number of tokens in the initial speculative transfer is chosen to hide this processing latency — while the host is computing offsets for the bulk transfer, the initial tokens are already in-flight on the network.

Why two-phase instead of one-phase: If all tokens were scattered in a single phase, the sender would need to know the exact destination offset in the receiver's buffer for each token. This offset depends on how many tokens other senders are also sending to the same receiver — information that isn't known until routing information has been exchanged. The two-phase design exchanges routing information first (Phase 1), then uses the computed offsets to pack tokens contiguously (Phase 2). The speculative initial transfer of a fixed number of tokens to private buffers hides the routing-exchange latency, achieving the low latency of a one-phase approach without requiring pre-coordination.

Buffer sizing for dispatch receive. The receive buffer must be large enough to handle the worst case where all tokens from all ranks are routed to the current rank's experts. The paper provides the formula (Section 6.1):

"Assuming there are $N$ ranks hosting $E$ experts, each dispatching $T$ tokens to $R$ experts, the upper bound is $N \cdot T \cdot \max(R, \frac{E}{N})$."

For $N=64$, $T=128$, $R=8$, and $E=256$ (the DeepSeek-V3 configuration), the buffer size per rank is $64 \cdot 128 \cdot \max(8, 4) = 64 \cdot 128 \cdot 8 = 65,536$ tokens. With 7168-dimensional fp8 vectors, this is approximately 470 MB per rank — substantial but workable on H200 GPUs with 141 GB of HBM.

NVLink vs. RDMA routing: Tokens destined for ranks on the same node are transferred via NVLink, not RDMA. The dispatch kernel checks each target rank: if it is local (same node), it writes directly to the peer's GPU memory over NVLink; if it is remote, it writes to the send buffer for the host proxy to transfer over RDMA. This reduces RDMA traffic by the factor of local peers — at EP64 across 8 nodes (8 GPUs per node), each rank communicates with 7 local peers over NVLink and 56 remote peers over RDMA, eliminating roughly 11% of network traffic (the intra-node fraction).

The paper describes an important ordering optimization for NVLink writes (Section 6.2, end):

"Loads are universally expensive, as they block the execution pipeline until they are satisfied. In contrast, stores are fire-and-forget, until a memory barrier is encountered, which blocks until all prior stores within their scope complete. Since both the host system and NVLink peers are within the same scope, a barrier ensuring ordering with the host might be slowed down by previously issued writes over NVLink. This is avoided by first signalling the host, then issuing NVLink writes after a grid barrier."

In plainer terms: NVLink writes (stores) are asynchronous from the sender's perspective — the GPU warp continues executing without waiting for the write to land on the remote GPU. However, a CUDA memory fence or barrier will block until all pending stores complete. If the code signals the host proxy (via a unified memory write) after issuing NVLink writes, the signal might be delayed by slow NVLink writes. By reversing the order — signal the host first, then issue NVLink writes — the host proxy gets the earliest possible notification to start RDMA transfers.

Combine protocol (single-phase scatter):

Combine reuses the routing information computed during dispatch, so it can use a single scatter: all payloads (expert outputs) are sent in one submit_scatter to the appropriate peers. The paper states (Section 6.3):

"Routing information is centralized during the dispatch stage, combine transfers all payloads in a single scatter."

The sender kernel prepares expert output tensors in send buffers, pushes tokens destined for local peers via NVLink, then signals the host proxy to issue the scatter. The receiver kernel caches all relevant offsets (derived from routing information) in shared memory, then waits for the IMMCOUNTER to signal that all WRITEs have completed. It then computes a weighted average of the received expert outputs, accumulating locally.

Buffer reuse and synchronization: Combine reuses the same buffers as dispatch, so it must ensure all prior operations are complete before overwriting them:

"The combine stage re-uses the same buffers as the dispatch stage, thus it requires both an NVLink and an RDMA barrier ensuring the completion of all prior operations before overwriting the send buffers." (Section 6.3)

The paper notes that for EFA, which waits for receipt confirmation on each WRITE (unlike RC which only acknowledges locally), it is important to "maximise the interval between posting a write and checking its status" — the longer the gap between posting and polling, the more likely the WRITE has already completed, avoiding wasted poll iterations.

Comparison with DeepEP's approach: The paper provides a careful comparison (Section 6.4) that explains why the host-proxy approach achieves competitive performance despite the architectural disadvantage:

• DeepEP uses GPU-initiated RDMA (IBGDA), avoiding the CPU entirely. Tokens are balanced across SMs, and each SM transfers tokens one-by-one over a queue pair. This achieves lower latency to the first transfer but produces more network packets and relies on RC ordering for correctness. Completion is signaled via ATOMICs — RDMA atomic operations that are only available on RC.

• fabric-lib uses a host proxy, adding GPU→CPU→NIC latency on the critical path. However, it batches tokens into bulk transfers, achieving better network utilization. For decode-sized batches, network bandwidth is not the bottleneck (the 7168-dimensional fp8 token is only ~7 KB per token, and 128 tokens routed to 8 experts is ~7 MB per rank — far below the 400 Gbps saturation point). The bottleneck is latency, and the host proxy overhead is small enough (Table 8: under 1.5 µs from application call to first RDMA WRITE at p50) that bulk transfer efficiency compensates.

• For prefill (4096 tokens per rank), DeepEP's advantage is larger because it uses sender-side partial sums over NVLink to reduce the amount of data transferred over RDMA. The paper acknowledges that their kernels lack this optimization, making them less competitive for prefill workloads with smaller expert parallelism.

Private buffer sizing ablation (Figure 11): The number of tokens transferred speculatively in private buffers is a critical tuning parameter. Too few tokens, and the routing-exchange latency is not fully hidden (the second-phase transfer starts before the initial tokens arrive, so the network goes idle for a period). Too many tokens, and the private buffer memory grows without additional benefit (once the routing information arrives and offsets are computed, all subsequent tokens go into the contiguous buffer). The ablation shows that performance degrades when private tokens drop below ~24 on ConnectX-7 and ~32 on EFA, indicating that the routing-exchange latency is higher on EFA (as expected from the bandwidth measurements in Table 2).

4. Key Insights and Innovations

Innovation 1: The Reliable-But-Unordered Transport Intersection as a Portability Primitive

The paper's most fundamental intellectual move is a diagnostic reframing of why RDMA point-to-point libraries are non-portable. Prior to this work, the dominant assumption — implicit in NVSHMEM, DeepEP, and Mooncake's Transfer Engine — was that portable RDMA meant either targeting the lowest common denominator (sacrificing performance) or relying on abstraction layers like UCX that paper over hardware differences (as NIXL does). Both approaches assume that the fundamental incompatibility lies in different APIs (libibverbs vs. libfabric) or different performance characteristics.

The paper argues that the real barrier is deeper and more specific: the assumption of in-order delivery. Table 1 makes this visible in a way that no prior work had: it lays out the capabilities of RC, UC, UD, and SRD side-by-side and highlights the intersection. The key observation is that ConnectX RC can be configured to ignore ordering (by enabling IBV_ACCESS_RELAXED_ORDERING), while EFA SRD cannot be configured to provide it. Therefore, the largest common subset that preserves both reliability and performance is reliable-but-unordered, not the "lowest common denominator" of unreliable datagrams.

This is a conceptual reframing, not just an engineering convenience. It converts the portability problem from "how do we support two different NIC APIs" to "how do we build completion notification without ordering guarantees" — a specific, solvable subproblem with a clear design principle (explicit per-transfer notification via IMMCOUNTER) rather than an open-ended abstraction challenge.

The significance extends beyond the two NICs studied. The paper notes (Section 8) that "Among Linux rdma-core providers, only EFA diverges from the standard RC transport. For RC-compatible NICs (e.g., eRDMA, Broadcom, AMD), the internal implementation would resemble the ConnectX path." This means the reliable-but-unordered design principle is sufficient for all current and near-future NICs — only EFA required the design adaptation, and by accommodating it, the library becomes portable to everything else by default. This turns the standard approach on its head: rather than designing for the dominant hardware and porting to the outlier, design for the outlier and the dominant case comes for free.

The evidence that this reframing is correct — and not merely a convenient story — comes from the performance data. NVSHMEM (pplx-kernels) shows an order-of-magnitude degradation on EFA versus ConnectX (Figure 9: 1,669 µs vs. 974 µs for decode dispatch at EP16), while fabric-lib shows comparable or better performance on both (286 µs vs. 236 µs). If the portability problem were merely an API abstraction issue, NVSHMEM's IBRC path (which uses libibverbs for both NIC types) would achieve similar performance on EFA. The fact that it doesn't — and that fabric-lib's explicit counter-based notification closes the gap — validates that ordering assumptions, not API differences, were the root cause.

Innovation 2: Difficulty-Conditioned Compute-Optimal Test-Time Scaling

[Note: The prior sections reference this heading but the innovation described belongs to the reference example paper, not fabric-lib. I will skip directly to the next innovation.]

Innovation 3: The Host-Proxy Architecture as a Deliberate Portability Strategy, Not a Performance Limitation

The conventional wisdom in low-latency GPU networking — reinforced by DeepEP's design and the broader RDMA literature — is that GPU-initiated RDMA (IBGDA/GDA) is strictly superior to host-proxy approaches because it removes the CPU from the critical path. DeepEP's state-of-the-art MoE dispatch latency relies on this: the GPU SMs directly post work requests to the NIC's send queue, avoiding the PCIe round-trip to the CPU and back. The implicit assumption is that a host-proxy design is either a stopgap for hardware lacking GDA support or a compromise that permanently caps performance.

The paper challenges this framing with two moves. First, it demonstrates that a host-proxy approach can match or exceed GPU-initiated performance on the same hardware: on ConnectX-7 at EP32, fabric-lib's decode dispatch is 110 µs versus DeepEP's 124 µs (Figure 9). This is not a ~10% competitive handicap — it is the opposite, despite the architectural disadvantage. The explanation (Section 6.4) is that bulk transfers achieve better network utilization than DeepEP's per-token approach, and the host-proxy overhead (Table 8: under 1.5 µs from application call to first WRITE at p50 for EP16) is small enough that the gains from batching outweigh the PCIe traversal cost.

Second, and more subtly, the paper reframes the host-proxy architecture as an enabler of portability rather than a performance concession. GDA is not available on EFA, and even on ConnectX, GDA is only available with specific driver versions and hardware configurations. By committing to a host-proxy design, fabric-lib achieves portability as a property of the architecture rather than as an afterthought — it works identically on any NIC that supports RDMA from the host, which is essentially all of them. The paper makes this explicit in Section 8:

"For KvCache and RL weight transfers, RDMA latency is already hidden by computation, so our CPU-based approach frees the GPU without sacrificing end-to-end performance."

This distinguishes between latency-bound workloads (MoE decode, where the proxy overhead matters but remains manageable) and bandwidth-bound workloads (KvCache, RL weights, where the proxy overhead is irrelevant because computation dominates). The insight is that GDA is only necessary when both latency and GPU utilization are critical simultaneously — a narrower set of conditions than the field assumed.

The evidence is in the breakdown data. For RL weight transfer (Table 5), RDMA transfer time is 26 ms out of 1,233 ms total — the CPU overhead of posting work requests is noise. For KvCache transfer (Table 3), the per-layer transfer time for 1,024 pages of 32 kB is 1.6 ms, while the compute time for that layer is 13.3 ms — the transfer is fully hidden. Even for MoE, the host-proxy overhead at the level of individual WRITE posting (Table 9: 8.5 µs p50 at EP64 on ConnectX-7) is an order of magnitude smaller than the total dispatch latency (216 µs at EP64 in Figure 9). The proxy is visible in profiling but not dominant in end-to-end performance.

Innovation 4: IMMCOUNTER as a Generalizable Completion Primitive for Unordered Transports

The IMMCOUNTER mechanism is the paper's architectural innovation that makes the reliable-but-unordered abstraction practically functional — but its significance goes beyond being a "completion notification mechanism." It is better understood as a design pattern for decoupling transfer semantics from transport ordering, a problem that arises in any distributed system where messages can be reordered.

Prior approaches to RDMA completion notification fall into three categories, all of which assume some form of ordering:

• Ordering-based notification (DeepEP, NVSHMEM on RC): the receiver knows that if WRITE_B's completion has arrived, all WRITEs issued before it have also completed. This is correct on RC but incorrect on SRD, and it couples the application's correctness to the transport's ordering guarantees.
• Per-transfer callbacks (naive approach): fire a callback for every individual WRITE completion. This works on unordered transports but incurs CPU overhead proportional to the number of transfers, which is prohibitive for fine-grained workloads (KvCache transfers with 760+ individual page WRITEs).
• Flag-based notification (pplx-kernels on EFA): the sender writes a flag to a known location in the receiver's memory after all data writes complete. This works on unordered transports but requires an additional RDMA operation per logical transfer (WRITE for data + WRITE for flag) and is vulnerable to the same ordering problems if the flag write overtakes the data writes on unordered transports.

IMMCOUNTER solves this by piggybacking completion notification on the data transfer itself: the 32-bit immediate value in WRITEIMM is delivered atomically with the data, so a single RDMA operation provides both data delivery and notification. The counter-based aggregation fires a single callback for N transfers rather than N callbacks. The per-immediate value provides a namespace that allows multiple logical operations to share the same completion queue without confusion — the application registers interest in a specific immediate value with a specific count threshold, and the engine routes completions to the correct callback based on the immediate value.

The paper's argument for correctness (Section 3.3, final paragraph) is itself an insight: PCIe ordering, not RDMA ordering, provides the necessary atomicity. Even on an unordered transport where WRITE_A and WRITE_B can complete in any order, the data payload of WRITE_A is guaranteed to be visible to the GPU before WRITE_A's immediate value is delivered to the CPU, because the data and the immediate traverse the PCIe switch together and PCIe maintains ordering for writes to the same device. The host-proxy architecture then bridges the CPU and GPU domains: once the CPU observes the IMMCOUNT, any subsequent CPU-to-GPU signal (via GDRCopy or kernel launch) is ordered after the NIC-to-GPU data writes by the PCIe switch.

This is a subtle but important distinction from how DeepEP uses ATOMICs. DeepEP's ATOMICs are RC-only operations that provide ordering by virtue of being issued on an ordered QP — the transport guarantees correctness. fabric-lib's IMMCOUNTER provides correctness through a combination of WRITEIMM atomicity (guaranteed by the RDMA spec for individual operations) and PCIe ordering (guaranteed by the hardware platform), neither of which alone would be sufficient. This decomposition is generalizable: any platform that provides per-operation atomicity for immediate values and maintains PCIe-style ordering between data and notification paths could support the IMMCOUNTER pattern.

The evidence that IMMCOUNTER works in practice is distributed across the three production systems. In KvCache transfer (Appendix A), the decoder uses expect_imm_count with imm_count = len(page_indices) * n_layers + 1 — for a 95-layer model with 1,024 pages per layer, this is 97,281 individual WRITEs aggregated under a single callback, which would be impossible with per-transfer notification. In MoE combine (Section 6.3), the receiver waits on an IMMCOUNTER for all scatter WRITEs to complete before computing the weighted average of expert outputs, ensuring that no partial data is accumulated. In RL weight transfer, IMMCOUNTER is not used because the training nodes don't need completion notification from inference nodes (the transfers are one-sided and fire-and-forget from the receiver's perspective), which demonstrates that IMMCOUNTER can be selectively deployed only where needed rather than being a mandatory part of every transfer.

Innovation 5: The Decode-Optimized MoE Kernel Design Demonstrating That NVLink-Ready Architectures Shift the Bottleneck

The paper's MoE kernel design (Section 6) is not just an implementation of scatter/gather over RDMA — it embeds a specific thesis about where the latency bottleneck lies in current-generation MoE serving and how this changes the optimization space. The thesis is: on current hardware (H100/H200 with NVLink within nodes and RDMA between nodes), the dominant latency cost is not the network transfer itself but the coordination overhead of exchanging routing information, and the correct optimization strategy is to hide this overhead through speculative data transfer rather than to minimize per-token network latency.

This is a departure from the approach embodied in DeepEP, which optimizes per-token transfer latency through GPU-initiated RDMA and RC ordering. DeepEP's design treats every microsecond on the wire as critical, transferring tokens one-by-one to get the first token to its destination as quickly as possible. The paper's approach instead accepts a slightly higher latency to the first transfer (~15 µs for the proxy to engage, versus DeepEP's sub-µs GPU-initiated posting) but batching tokens into bulk transfers that saturate NIC bandwidth and reduce the total number of network round-trips.

The evidence that this thesis holds comes from two sources. First, the private buffer ablation (Figure 11) shows that performance degrades sharply when fewer than ~24–32 tokens are transferred speculatively — this is precisely the threshold where the routing-exchange latency is no longer hidden, confirming that routing coordination, not raw transfer latency, is the dominant cost. Second, the send/receive latency breakdown (Figure 12) shows that at EP64, the send kernel execution time is 20.5 µs (EFA) or 17.9 µs (CX-7), while the transfer time between send completion and receive completion is tens to hundreds of microseconds. The GPU-side work is under 15% of the total end-to-end latency — the network and coordination dominate.

This has implications beyond the specific numbers. The paper argues (Section 8) that next-generation GPUs with wide NVLink domains (e.g., GB200 NVL72 with 72 GPUs on a single NVLink fabric) will shift even more communication off RDMA entirely — when most expert-parallel peers are within the NVLink domain, the RDMA path becomes a rare fallback rather than the common case. The host-proxy design, by keeping the RDMA path functional but not hyper-optimized, positions fabric-lib to benefit from this hardware trend: as the fraction of traffic that goes over RDMA shrinks, the proxy overhead becomes proportionally less important, while the portability advantage (working on any NIC) becomes more valuable because the NIC is no longer the primary communication medium.

This is an architectural bet, not a measured result. The paper does not present GB200 benchmarks. But by articulating the argument explicitly and showing that the host-proxy design is already competitive on current hardware where RDMA is the bottleneck, the paper makes a credible case that the bet is well-founded — not a compromise accepted for portability, but a strategic alignment with hardware evolution.

5. Experimental Analysis

Evaluation Methodology

Hardware configurations. All experiments run on two cluster types (Section 7, opening paragraph): 8×H200 nodes with 2×200 Gbps EFA NICs per GPU (AWS p5/p5en instances), and 8×H100 nodes with 400 Gbps ConnectX-7 NICs per GPU (on-premise InfiniBand). This dual-hardware setup is necessary to evaluate the portability claim — performance on both ConnectX and EFA must be measured under identical workloads.

Metrics. The paper uses three distinct metric types depending on the workload. For point-to-point microbenchmarks (Section 7.1), the metric is bandwidth (Gbps) and operations per second, measuring what fraction of theoretical line rate the library achieves at various message sizes. For end-to-end LLM performance (Sections 7.2, 7.4.1), the primary metric is tokens per second for decode and time-to-first-token (TTFT, in milliseconds) for prefill — these are user-facing latency and throughput metrics that capture the real impact of communication overhead. For RL weight transfer (Section 7.3), the metric is wall-clock time per update (milliseconds), broken down by pipeline stage using PyTorch profiler instrumentation. Kernel-level MoE benchmarks (Section 7.4.3) report dispatch and combine latency in microseconds, with a full distribution (mean, p01, p25, p50, p75, p95, p99) over 10,000 warmup and 10,000 benchmark iterations, which is essential because tail latency determines worst-case decode step time.

Baselines. For point-to-point communication, the paper compares against NIXL v0.6.1 (NVIDIA Inference Xfer Library) on both EFA and ConnectX-7, and against hardware-level benchmarks: ib_write_bw from rdma-core on ConnectX and fi_rma_bw from libfabric on EFA (Section 7.1). For MoE dispatch/combine, the primary baselines are DeepEP (Zhao et al., 2025) — the state-of-the-art GPU-initiated RDMA kernel for ConnectX — and pplx-kernels (Licker et al., 2025), an open-source portable MoE kernel built on NVSHMEM v3.4.5 that runs on both EFA and ConnectX-7 (Section 7.4). The pplx-kernels baseline is particularly important because it represents the only previously-available portable option on EFA, making it the direct comparison for the "first viable EFA implementation" claim. For disaggregated inference (Section 7.2), the baseline is non-disaggregated execution (prefill and decode on the same node) to isolate the TTFT overhead introduced by KvCache transfer. For RL weight transfer (Section 7.3), the baselines are prior reported numbers from similar-scale systems: Moonshot AI (2025) reporting "tens to hundreds of seconds" for comparable model sizes, and the open-source NeMo-RL and Slime frameworks where weight synchronization is a documented bottleneck.

Models used for end-to-end evaluation. Three model configurations are evaluated. For disaggregated inference: Qwen3-235B with TP4 on H200 (Section 7.2). For RL weight transfer: Kimi-K2 (1T parameters), DeepSeek-V3 (671B), and Qwen3-235B (Section 7.3), with the detailed latency breakdown (Table 5) specifically from Kimi-K2 at FSDP/PP/EP=16/2/8 with 256 training GPUs transferring BF16 weights to 128 inference GPUs receiving FP8 weights. For MoE dispatch/combine: DeepSeek-V3 with Multi-Token Prediction (MTP, draft length 1, acceptance rate 80%) across EP=DP=8, 16, 32, and 64 configurations (Section 7.4.1). These are all production-scale models, not toy configurations, which is essential because communication overheads that are negligible at small scale can become dominant at the expert-parallel and tensor-parallel degrees used in production.

Generation budget and compute accounting. The paper does not use a unified "generation budget" concept — each workload measures compute differently based on what matters for that use case. For point-to-point microbenchmarks, the independent variable is message size (64 B to 64 MiB for single WRITE, 1 KiB to 64 KiB for paged WRITE), with the dependent variable being achieved bandwidth. For MoE decode, the metrics are reported at batch sizes 2, 8, 32, 48, 64, 96, and 128 tokens per rank, with tokens dispatched to 8 random experts each (Section 7.4.1 and 7.4.2). For MoE prefill, the chunk size is 4096 tokens (Section 7.4.3). For RL weight transfer, the "budget" is implicit in the model size and parallelism configuration — the transfer time is measured for a full model update given a fixed hardware topology. For disaggregated inference, the independent variable is input sequence length (4K to 128K tokens), with KvCache page size fixed at 32 kB (128 tokens) and chunk-prefill length up to 16,384 tokens (Table 3).

Statistical protocol for MoE benchmarks. The kernel-level MoE microbenchmarks (Section 7.4.3) follow a rigorous protocol: 10,000 warmup iterations followed by 10,000 timed iterations, with timing aggregated across all ranks. Large GEMMs are inserted between iterations to simulate overlapped work and clear caches, preventing thermal or caching artifacts from inflating later measurements. Error bars in Figures 9 and 10 show the full distribution: p01, p25, p50 (median), p75, p95, and p99. This is critical because MoE dispatch/combine is latency-bound — the p99 tail latency determines whether a decode step exceeds a real-time serving deadline, not the mean. No cross-validation protocol is described for the MoE kernels, which is appropriate since the comparison is between implementations (not learned policies).

Main Quantitative Results

Point-to-Point Communication Performance

The paper first establishes that fabric-lib achieves near-line-rate bandwidth on both hardware platforms for messages large enough to saturate the NICs, with the primary insight being that EFA requires larger messages than ConnectX-7 to reach saturation, which has downstream consequences for MoE routing where messages are moderately sized.

Figure 8 shows the fraction of peak hardware bandwidth achieved by both fabric-lib and NIXL across message sizes. For single WRITE, both libraries require messages of at least 16 MiB to saturate bandwidth. At 1 MiB — a size relevant for moderate-weight transfers — fabric-lib achieves 145 Gbps on EFA and 245 Gbps on ConnectX-7 (Table 2), representing 36% and 61% of peak respectively. The gap narrows at smaller sizes: at 256 KiB (the typical size for a single MoE routing payload), fabric-lib achieves 54 Gbps on EFA (13.5% of 400 Gbps peak) and 116 Gbps on ConnectX-7 (29% of peak). This explains why EFA performance trails ConnectX-7 on MoE workloads — the per-transfer data volume is not large enough to amortize EFA's higher per-message overhead.

For paged WRITE, which is the operation used for KvCache transfers (many small pages issued as a batch), both platforms achieve near-line-rate at much smaller individual page sizes: at 16 KiB pages, fabric-lib achieves 274 Gbps on EFA (68.5% of peak) and 367 Gbps on ConnectX-7 (91.8% of peak). This is encouraging for the disaggregated inference use case — even with 32 kB KvCache pages, the paged WRITE interface allows the TransferEngine to batch multiple pages into a single submission, achieving high aggregate throughput. The operations-per-second metric is included for paged WRITE: at 1 KiB, fabric-lib achieves 2.11 million operations per second on EFA and 11.10 million on ConnectX-7, reflecting the fundamentally different per-operation overhead of the two NIC architectures.

The comparison with NIXL shows that fabric-lib is "relatively close" (Section 7.1) — NIXL achieves slightly higher bandwidth at some message sizes (NIXL-EFA reaches ~60% of peak at 256 KiB single WRITE versus fabric-lib's ~13.5%, though the paper's Figure 8 is a relative-bandwidth plot where direct comparison is difficult), while fabric-lib has a slight edge at others. The paper emphasizes that the TransferEngine is "slightly faster" at saturation, but the more important takeaway is that both are in the same performance class — fabric-lib is not sacrificing performance for portability relative to the (NVIDIA-developed, UCX-backed) NIXL.

KvCache Transfer: Disaggregated Inference Overhead

The central result for disaggregated inference is that KvCache transfer is effectively hidden by compute, with the TTFT overhead relative to non-disaggregated execution being dominated by an extra decode pass rather than by RDMA transfer latency.

Table 3 reports end-to-end results for Qwen3-235B on H200 with TP4 and 2×200 Gbps EFA per GPU. The key columns are: Non-disaggregated TTFT (baseline), Disaggregated TTFT (with KvCache transfer), the compute time per layer, and the transfer time per layer. At sequence length 4K (1 chunk, 256 pages of 32 kB), the per-layer compute takes 2.267 ms while the transfer takes 0.661 ms — the transfer is fully hidden within the next layer's computation. TTFT increases from 214 ms (non-disaggregated) to 260 ms (disaggregated), a 21.5% overhead. At 32K (2 chunks, 1024 pages each), the compute is 13.295 ms and the transfer is 1.606 ms per layer — again fully hidden. TTFT goes from 2,179 ms to 2,317 ms, a 6.3% overhead. At 128K (8 chunks), TTFT goes from 16,735 ms to 17,056 ms, a 1.9% overhead.

The paper states that the observed overhead is "mainly from our inference engine performing one extra decode pass for the final input token, rather than from KvCache transfer" (Section 7.2). This is an important qualification: the 21.5% overhead at 4K is not a fundamental limitation of the RDMA approach but an artifact of the specific inference engine's implementation. The per-layer transfer time of 0.66–1.61 ms is consistently less than the per-layer compute time of 2.27–34.9 ms, confirming that the UvmWatcher-based overlap works as designed — the GPU signals the CPU as soon as a layer's KV pages are ready, the CPU initiates the RDMA transfer, and the transfer completes while the GPU processes the next layer.

The UvmWatcher callback latency is reported in Table 4. Under CUDA Graph (which is the production deployment mode, since CUDA Graphs eliminate kernel launch overhead), the callback latency when implemented in Rust has p50 of 6.2 µs, p99 of 12.6 µs, and p99.9 of 19.4 µs — tightly bounded, with the minimum latency of 2.5 µs approaching the 2–5 µs PCIe latency floor. When the callback is in Python (which might be used during development or for non-performance-critical paths), the p50 remains acceptable at 9.3 µs, but the p99.9 balloons to 41.7 µs and the maximum to 3,325 µs — a classic Python GIL or garbage collection tail. The Rust callbacks are used in production, avoiding this tail.

The paper does not provide end-to-end TTFT numbers for larger models (DeepSeek-V3) or for the ConnectX-7 platform on this workload. The single Qwen3-235B on EFA data point establishes feasibility, but the generalization to other model architectures (with different layer counts, hidden dimensions, and hence different compute-to-transfer ratios) is not empirically validated.

RL Weight Transfer: 1.2-Second Updates for Trillion-Parameter Models

The headline result — 1.3 seconds for trillion-parameter model weight updates, over 100× faster than existing frameworks — is supported by a detailed latency breakdown in Table 5, profiled on Kimi-K2-1T with FSDP/PP/EP=16/2/8 across 256 training GPUs and 128 inference GPUs.

The total transfer completes in 1,233 ms (1.2 seconds, which the abstract rounds to 1.3 seconds — the discrepancy is not explained but is likely due to variance across runs or rounding). The breakdown reveals the critical path:

• full_tensor() (FSDP parameter unsharding): 518 ms, 974 calls, 532 µs average per call. This dominates the critical path — the GPU must reconstruct full tensors from sharded FSDP parameters before they can be written over RDMA. The average of 532 µs per call is relatively low, suggesting that most parameters are small and the all-gather operations are efficient, but the sheer number of parameters (974 calls across all MeshGroups) accumulates.

• Waiting for other ranks (synchronization): 357 ms. This is time spent in the GLOO barrier after all full_tensor() calls for a MeshGroup complete. The fact that this is the second-largest component suggests load imbalance — some ranks finish their transfers earlier and wait for stragglers. This could be due to uneven parameter distribution across FSDP shards or varying RDMA completion times.

• RDMA submit: 26 ms, 1,144 calls, 23 µs average per call. This is the CPU overhead of posting 1,144 individual submit_single_write work requests. The 23 µs per call is slightly higher than the MoE proxy overhead (Table 8 shows ~1.3 µs from application call to first WRITE), likely because the RL weight transfer involves larger buffers requiring more sharding across NICs, but the total CPU overhead is negligible relative to the 1,233 ms wall-clock time.

• Extra RDMA time not hidden by other operations: 42 ms. This is the "tail of the pipeline" — RDMA transfers that were posted after the last full_tensor() completed and before the global barrier, meaning they could not be overlapped with parameter preparation. The fact that only 42 ms out of the total RDMA transfer time (which is larger but not separately reported) falls outside the overlap confirms that the four-stage pipeline (H2D memcpy → full_tensor() → RDMA WRITE → barrier) is highly effective.

The secondary results — training logs for DeepSeek-V3-671B and Qwen3-235B showing "similar transfer times of 1.2 s to 2 s" — are mentioned but not tabulated. The lack of a detailed breakdown for these models is a weakness: without it, the reader cannot determine whether the pipeline stages scale similarly or whether different models expose different bottlenecks (e.g., DeepSeek-V3's MoE architecture might shift time from full_tensor() to the RDMA phase due to more, smaller expert parameters).

The claim of "over 100× faster than existing frameworks" is supported by citations to Moonshot AI (2025), NeMo-RL, and Slime reporting "tens to hundreds of seconds" for comparable model sizes (Section 7.3). However, the paper does not run a head-to-head comparison with these systems on identical hardware — it cites reported numbers rather than reproducing them. The 100× figure (1.2 seconds vs. 120 seconds) is therefore an approximate upper bound on the speedup, and the actual improvement over a well-optimized baseline (e.g., the NeMo-RL weight sync optimization described in He et al., 2025) might be smaller.

MoE Dispatch/Combine: Decode Performance

The MoE results are the most extensively benchmarked, spanning end-to-end tokens-per-second, kernel-level latency microbenchmarks, overlap effectiveness, private buffer sizing, and host-proxy CPU overhead.

End-to-end decode speed (Table 6). On EFA (H200), fabric-lib achieves 66.75 tokens/s at batch=2, 56.46 tokens/s at batch=8, and 32.00 tokens/s at batch=32. Compared to pplx-kernels (NVSHMEM-based) on the same hardware, this is a 3.2× improvement at batch=2 (20.97 → 66.75), a 4.9× improvement at batch=8 (11.61 → 56.46), and a 6.5× improvement at batch=32 (4.90 → 32.00). The widening gap at larger batches suggests that pplx-kernels has a fixed per-step overhead that does not amortize well, while fabric-lib's bulk transfer approach benefits from batching more tokens.

On ConnectX-7 (H100), fabric-lib achieves 78.42 tokens/s at batch=2, 67.67 tokens/s at batch=8, and 36.07 tokens/s at batch=32. Compared to DeepEP on the same hardware, fabric-lib is faster at batch=2 (78.42 vs. 73.76, a 6.3% improvement), slightly faster at batch=8 (67.67 vs. 65.79, a 2.9% improvement), and slightly slower at batch=32 (36.07 vs. 36.25, essentially tied). The paper describes this as "match or slightly exceed DeepEP across all batch sizes" (Section 7.4.1), which is accurate — the host-proxy design is not a handicap at production batch sizes.

The DeepSeek-V3 configuration with MTP (draft length 1, acceptance rate 80%) means that each "decode step" generates one draft token and one verified token with 80% probability, so the effective tokens-per-step is 1.8. The reported tokens/s therefore divide effective throughput by end-to-end step latency. At batch=32 and 36.07 tokens/s, this implies a per-step latency of approximately (32 × 1.8) / 36.07 = 1.60 seconds per step — consistent with the MoE decode being one component of a larger pipeline that includes attention, shared experts, and the MTP verification step.

Dual-batch overlap effectiveness (Table 7). Dual-batch overlap pipelines the communication of one batch with the computation of another, and is expected to help when per-GPU batch sizes are large enough that computation time exceeds communication latency. For fabric-lib on EFA, overlap provides modest gains: at batch=128, throughput increases from 11.81 to 13.92 tokens/s (17.9% improvement); at batch=96, from 14.35 to 16.49 (14.9%); at batch=64, from 21.26 to 21.44 (0.8%). At batch=48 (24.22 vs. 24.20) and batch=32 (32.00 vs. 30.24), overlap provides no benefit or slight degradation. The paper's interpretation is that "even in throughput-oriented regimes, MoE dispatch/combine latency still matters" — the communication latency is large enough that it dominates even when computation is overlapped, so reducing communication latency (as fabric-lib does relative to pplx-kernels) is beneficial even when overlap is possible.

For pplx-kernels, overlap consistently degrades performance: at batch=128, throughput drops from 1.55 to 1.45 tokens/s; at batch=32, from 4.90 to 4.83. The paper attributes this to "their high communication latency" preventing effective overlap. This is a strong negative result that underscores the importance of low communication latency for any overlap strategy to work — if communication takes longer than the compute it is meant to be overlapped with, dual-batch becomes a net loss because it adds scheduling overhead without reducing the critical path.

Decode latency microbenchmarks (Figure 9). The kernel-level measurements isolate dispatch and combine latency from the rest of the inference pipeline. The headline numbers (all at p50, the median):

• Intra-node (EP8): fabric-lib dispatch is 53 µs on EFA and 52 µs on CX-7, versus DeepEP at 43 µs. The ~9 µs gap (~21% slower) is attributed to "the use of NICs to exchange routing information" even for intra-node transfers (Section 7.4.3) — DeepEP uses NVLink for all intra-node communication, while fabric-lib still goes through the NIC proxy for routing. Combine latencies are 64–72 µs across all implementations, with DeepEP at 64 µs and fabric-lib at 65–72 µs.

• Inter-node (EP16): fabric-lib dispatch is 216–286 µs, versus DeepEP at 190 µs and pplx-kernels at 447–1,669 µs. Combine is 243–342 µs, versus DeepEP at 311 µs and pplx-kernels at 916–1,058 µs.

• EP32: fabric-lib dispatch is 110–155 µs (CX-7: 110 µs is faster than DeepEP's 124 µs), combine is 186–286 µs (CX-7: 186 µs vs. DeepEP's 267 µs). The paper emphasizes that on CX-7 at EP32, fabric-lib is faster than DeepEP on both dispatch and combine.

• EP64: fabric-lib dispatch is 124–236 µs (CX-7: 124 µs vs. DeepEP's 160 µs, a 22.5% improvement), combine is 203–342 µs (CX-7: 203 µs vs. DeepEP's 327 µs, a 37.9% improvement). However, on EFA, dispatch at EP64 is 236 µs — slower than DeepEP on CX-7 but still 20× faster than pplx-kernels on EFA (4,136 µs). The paper notes that "the CPU overhead of the proxy thread becomes noticeable" at EP64, with dispatch on EFA lagging behind CX-7 by ~90% (236 vs. 124 µs), and dispatch on CX-7 exceeding DeepEP because "the roughly microsecond overhead of enqueuing a transfer for each of the 56 inter-node peers" adds up.

The error bars in Figure 9 show that fabric-lib's latency distributions are generally tighter than pplx-kernels' — the p99 is consistently close to the p50 (within ~50–100 µs), while pplx-kernels shows standard deviations that are large relative to the mean (the paper notes "Error bars for pplx indicate stddev"). This suggests that NVSHMEM's performance on EFA is not just slow but unpredictable, which is worse for real-time serving where tail latency determines quality of service.

Prefill latency (Figure 10). The prefill results are less favorable to fabric-lib, as the paper acknowledges. At EP8, dispatch takes 1.05 ms (EFA) or 1.08 ms (CX-7) versus DeepEP's 1.82 ms — fabric-lib is faster. But as expert parallelism increases, DeepEP's advantage grows: at EP64, dispatch is 5.35 ms (EFA) or 4.68 ms (CX-7) versus 5.07 ms — roughly comparable. For combine, DeepEP has a clear advantage at all scales: at EP64, fabric-lib takes 8.28–9.79 ms versus DeepEP's 7.53 ms, and at EP8, fabric-lib takes 1.41–1.87 ms versus DeepEP's 0.96 ms. The paper attributes DeepEP's combine advantage to "sender-side partial sum" — DeepEP pre-accumulates tokens via NVLink on the sender node before transmitting over RDMA, reducing both the number of transferred bytes and the accumulation work on the receiver. Fabric-lib's decode-optimized design (single bulk scatter) does not include this optimization.

The paper is candid about the prefill limitation: "Due to the lack of chunking in transfers, the memory overhead of our decode-optimized kernels limits the set of models for which a deployment is viable" (Section 7.4.3). The 4096-token prefill chunk at EP64 requires a dispatch receive buffer sized for all tokens from all 56 remote peers potentially being routed to the local experts — at DeepSeek-V3 scale with 7168-dimensional fp8 tokens, this is substantial (~470 MB as computed in Section 3.4). DeepEP's chunked approach requires less buffer memory because tokens are transferred in smaller batches.

Ablation Studies and Robustness Checks

Private buffer sizing for MoE dispatch (Figure 11): The number of tokens transferred speculatively in private per-source buffers is a critical tuning parameter that reflects the tradeoff between hiding routing-exchange latency and consuming GPU memory. The ablation sweeps the maximum number of private tokens from 0 to 32 and measures the relative slowdown in p50 decode dispatch latency compared to the optimal buffer size. On ConnectX-7, performance is flat above ~24 tokens — the routing-exchange latency is fully hidden with that many tokens in-flight. Below 24, performance degrades gradually, reaching a ~10% slowdown at 0 private tokens (no speculative transfer). On EFA, the threshold is higher (~32 tokens) and the degradation is steeper — at 16 tokens, EFA shows a ~5% slowdown versus near-zero on CX-7. The paper explains this by noting "route exchange is slower" on EFA, consistent with the bandwidth measurements showing lower throughput at the 256 KiB message sizes used for routing information. For intra-node (EP8), the curve is steeper and the saturation point is ~8 tokens on EFA and ~4 tokens on CX-7 — intra-node routing via NICs (which fabric-lib does even for same-node peers) has lower latency, requiring fewer speculative tokens to hide.

Send vs. receive kernel latency (Figure 12): This ablation separates the GPU-side kernel execution time from the network transfer time by inserting an artificial delay before the receive kernel, allowing all transfers to settle. At EP64, the dispatch send kernel takes 20.5 µs (EFA) or 17.9 µs (CX-7), and the dispatch receive takes 24.9 µs or 25.7 µs — these are the GPU-side costs of shuffling tokens into send buffers and reordering received tokens. Combine send takes 13.1–13.7 µs (faster because it does not need to exchange routing information), and combine receive takes 15.0–31.6 µs (fabric-lib is notably faster than DeepEP's 31.6 µs on receive, attributed to "faster accumulation"). The total GPU-side execution is under 15% of the end-to-end dispatch+combine latency (which is hundreds of microseconds at EP64), confirming that the network and host-proxy coordination dominate the critical path. The paper highlights that "the proxy commences RDMA work midway through the execution of the send kernels, with shuffling adding only about 15 µs of idle time" (Section 7.4.5) — the GPU and proxy are well-synchronized, with the proxy launching transfers while the GPU is still preparing later batches of tokens.

Host-proxy CPU overhead breakdown (Table 8): This ablation instruments key events during MoE all-to-all communication at EP64, reporting elapsed time since the previous event. The application calls submit_scatter() (the beginning of the timeline). The work request is enqueued in the lock-free queue in 0.12 µs at p50 (the "→Enqueue done" event). The worker thread picks up the request in 0.86 µs at p50 (the "→Worker enqueue done" event) — this is scheduling latency from the application thread to the DOMAINGROUP worker, which is pinned to a NUMA-local core. The "→Before posting first WRITE" event fires 0.44 µs later, reflecting the time to prepare the libfabric/libibverbs descriptor from the templated version. The dominant cost is posting all WRITEs in the scatter: on EFA at EP64, this takes 27.89 µs at p50 and 43.32 µs at p99.9; on ConnectX-7, it takes 8.50 µs at p50 and 14.47 µs at p99.9. The 3.3× gap between EFA and ConnectX-7 in WRITE posting time reflects the higher per-operation overhead of libfabric (which must interact with the EFA SRD driver, a more complex software stack than the in-kernel mlx5 driver for ConnectX).

Scaling of WRITE posting time with expert parallelism (Table 9): A companion ablation measures how the time to post all WRITEs in a scatter grows with the number of targets (EP degree). On ConnectX-7, posting time increases roughly linearly from 0.84 µs at EP8 to 8.50 µs at EP64 (a 10.1× increase for an 8× increase in peers — slightly superlinear, suggesting some per-peer overhead that doesn't fully amortize). On EFA, the increase is from 3.08 µs at EP8 to 27.89 µs at EP64 (a 9.1× increase, roughly linear). The paper notes that these overheads "remain acceptable" because they are small relative to the total dispatch latency (216–236 µs at EP64). Even the p99.9 posting time of 14.47 µs on ConnectX-7 is only ~6.7% of the p50 dispatch latency. This justifies the paper's claim that the host proxy is not a bottleneck — the CPU-side work is an order of magnitude faster than the network transfers it initiates.

Dual-batch overlap with varying batch sizes (Table 7): While presented as a main result, this can also be read as an ablation on whether computation-communication overlap is effective as a strategy for hiding MoE dispatch/combine latency. The finding that overlap provides only modest gains (17.9% at batch=128) and degrades performance for pplx-kernels is a robustness check on the "just overlap more" hypothesis — for latency-bound workloads, reducing communication latency directly (as fabric-lib does) is more effective than trying to hide it behind computation. The degradation for pplx-kernels at all batch sizes is a negative result that reinforces this point: if communication latency is too high, dual-batch overlap becomes counterproductive.

UvmWatcher callback latency under CUDA Graph with Rust vs. Python (Table 4): This ablation tests whether the callback mechanism introduces unacceptable tail latency. The Rust implementation shows tight bounds (p99.9 of 19.4 µs, max of 64.8 µs), confirming that a systems-level implementation in Rust avoids the garbage collection and GIL-related tails that plague Python callbacks (p99.9 of 41.7 µs, max of 3,325 µs). While not a controlled ablation (the comparison is between languages, not configurations of the same system), it justifies the engineering choice to implement the TransferEngine and callback handlers in Rust for production deployments.

Critical Assessment

Do the experiments demonstrate that fabric-lib achieves portability without sacrificing performance?

On ConnectX-7, yes, with a qualification. The MoE decode latency results (Figure 9) show that fabric-lib's host-proxy design matches or exceeds DeepEP's GPU-initiated approach on ConnectX-7 at EP16, EP32, and EP64, with the largest advantage at EP64 (dispatch: 124 µs vs. 160 µs; combine: 203 µs vs. 327 µs). The qualification is that this comparison is at a specific model configuration (DeepSeek-V3, 7168-dimensional fp8 tokens, 8 experts per token). The paper does not evaluate whether the advantage holds for models with smaller expert hidden dimensions (where per-token transfer size is smaller and bulk batching is less effective), different numbers of experts per token (changing the scatteredness of the communication pattern), or different expert counts (changing the ratio of intra-node to inter-node traffic). DeepEP's per-token approach may have a relative advantage in regimes where individual token sizes are too small for bulk transfer to amortize the host-proxy overhead — a boundary the paper does not map.

On EFA, yes, with a major caveat about the baseline. The comparison with pplx-kernels (NVSHMEM) shows a 3–6× throughput improvement (Table 6) and an order-of-magnitude latency reduction (Figure 9), establishing fabric-lib as "the first viable implementation on EFA" (Abstract). However, pplx-kernels is the only EFA baseline, and the paper does not investigate whether NVSHMEM's poor EFA performance is intrinsic to the library or could be resolved through configuration tuning. If NVSHMEM's EFA degradation were fixable (e.g., by adjusting buffer sizes, QP configurations, or polling parameters), the "first viable" claim would be less about portability and more about NVSHMEM's immaturity on EFA. The paper does not provide enough detail about the pplx-kernels configuration to assess this.

On point-to-point bandwidth, yes, but the comparison is incomplete. Figure 8 shows fabric-lib achieving comparable bandwidth to NIXL, and Table 2 shows absolute numbers approaching line rate for large messages. However, the comparison is only at the microbenchmark level — there is no end-to-end RL weight transfer or KvCache transfer comparison with NIXL, which would be the most direct test of whether fabric-lib's portability imposes a performance cost. The fact that NIXL's EFA support is "preliminary" (Section 2.3) makes such a comparison difficult, but it also means the "no performance sacrifice" claim is tested primarily against DeepEP (which doesn't run on EFA) and NVSHMEM (which is slow on EFA), leaving a gap where a mature EFA-capable baseline would be the most informative comparison.

Do the experiments demonstrate that IMMCOUNTER provides correct, performant completion notification?

Correctness is argued from first principles, not experimentally validated. The paper's argument for IMMCOUNTER correctness relies on PCIe ordering guarantees (Section 3.3) and the RDMA specification's guarantee that WRITEIMM data is delivered before the immediate value. There are no experiments that deliberately stress this mechanism — for example, by running on EFA with high packet loss and retransmission, measuring whether any completion callbacks fire before data is visible to the GPU. This is not necessarily a weakness (correctness is often established through design review and static analysis in systems work), but the paper's empirical evidence for the mechanism is limited to three production systems working correctly, which is a existence proof rather than a stress test. A specific concern: the paper mentions that EFA SRD provides reliable but unordered delivery. If a WRITE's data payload is retransmitted due to packet loss, does the immediate value wait for the retransmission, or is it possible (under some EFA firmware version) for the immediate to be delivered on the first transmission while the data is still being retransmitted? The paper does not address this, and it would be a violation of the RDMA spec that could only be caught through testing.

Performance of IMMCOUNTER is demonstrated implicitly through KvCache and MoE workloads. The KvCache transfer (Table 3) uses IMMCOUNTER to aggregate hundreds of individual page WRITEs into a single callback, and the per-layer transfer time (0.66–1.61 ms) remains well below the per-layer compute time (2.27–34.9 ms). The MoE combine uses IMMCOUNTER to wait for all scatter WRITEs before computing the weighted average, and the receive-side latency (Figure 12) is dominated by data movement rather than counter processing. However, the paper does not isolate the CPU overhead of IMMCOUNTER counter increments versus alternative completion mechanisms (e.g., per-transfer callbacks, flag polling). The counter-based approach is clearly necessary (per-transfer callbacks would be prohibitively expensive for 760+ page WRITEs in KvCache transfer), but the paper does not quantify how expensive per-transfer callbacks would be, making the IMMCOUNTER innovation harder to evaluate independently.

Do the experiments demonstrate that the RL weight transfer achieves 1.3-second updates, 100× faster than existing frameworks?

The 1.3-second number is substantiated for Kimi-K2, but scope is limited. Table 5 provides a detailed breakdown that accounts for 1,233 ms total time, with the critical path dominated by full_tensor() (518 ms) and synchronization (357 ms). The 26 ms of RDMA CPU overhead and 42 ms of extra RDMA time confirm that the network is not the bottleneck. The secondary results for DeepSeek-V3-671B and Qwen3-235B ("similar transfer times of 1.2 s to 2 s") are mentioned without tables, which weakens the generalization claim. A 2-second transfer for Qwen3-235B (a smaller model than Kimi-K2) suggests that model size is not the only factor — parallelism configuration, parameter sharding strategy, and quantization pipeline likely also matter, but these are not systematically varied.

The 100× claim is relative to reported, not reproduced, baselines. The paper cites Moonshot AI (2025) and the NeMo-RL and Slime communities reporting tens to hundreds of seconds for comparable model sizes. These are not head-to-head comparisons on identical hardware with identical models. A fair comparison would run the Rank0-based approach (Figure 4a) on the same cluster and measure the speedup directly. The paper does not do this, likely because deploying a Rank0-based approach for a trillion-parameter model would require engineering investment in a deprecated communication strategy. This is understandable but means the "100×" figure should be read as "orders of magnitude faster than current common practice" rather than "exactly 100× faster than a specific optimized baseline."

The pipelining effectiveness is well-demonstrated but sensitive to configuration. The watermark-based concurrency control (Section 5.2) is essential to the pipeline's success — without it, concurrent full_tensor() calls would exhaust GPU memory. The paper does not evaluate how sensitive the transfer time is to the watermark value, the number of MeshGroups, or the FSDP sharding strategy. A higher watermark would allow more concurrent transfers (reducing idle time on the NICs) at the risk of OOM; a lower watermark would be safer but might leave NICs idle. The fact that only 42 ms of RDMA time falls outside the overlap suggests the current watermark is well-tuned, but the paper does not provide evidence that this tuning is robust to different model architectures.

Do the experiments demonstrate that KvCache transfer adds minimal overhead to disaggregated inference?

The TTFT data supports the claim for Qwen3-235B on EFA, but generalization is limited. Table 3 shows TTFT overhead of 1.9–21.5% (depending on sequence length), with per-layer transfer time consistently below per-layer compute time. This establishes feasibility for one model on one hardware platform. However:

• Single model, single parallelism configuration. The paper tests only Qwen3-235B with TP4 on H200. Scaling tensor parallelism changes the compute-to-communication ratio — TP8 halves the per-GPU compute but keeps the per-GPU KV cache size constant, making the transfer relatively more expensive. The paper does not explore this.

• No ConnectX-7 comparison for this workload. If KvCache transfer were the bottleneck (which it's not, based on the EFA data), the higher per-message bandwidth of ConnectX-7 (Table 2) would further reduce transfer time. But this is not tested.

• The extra decode pass artifact. The paper states that the TTFT overhead is "mainly from our inference engine performing one extra decode pass for the final input token, rather than from KvCache transfer." This means the current numbers are an upper bound on what a fully optimized disaggregated system could achieve. If the extra decode pass were eliminated, the overhead would be even smaller, making the point-to-point RDMA approach even more attractive. However, the paper does not quantify how much of the overhead is from this artifact versus from intrinsic transfer costs.

• Scalability to larger clusters. The paper tests on a single 8×H200 node for prefill and decode. In a production disaggregated deployment, prefill and decode clusters might have hundreds of nodes connected through a multi-tier network fabric. The paper does not evaluate whether the IMMCOUNTER mechanism, UVM watcher, or per-transfer completion tracking scale to cluster sizes where network contention and congestion control become relevant.

What experiments would have strengthened the paper?

1. A head-to-head RL weight transfer comparison on the same cluster — running the Rank0-based collective approach from an existing framework (NeMo-RL or Slime, with their reported optimizations) alongside fabric-lib's P2P approach, measuring both wall-clock time and network utilization.

2. A deliberate IMMCOUNTER stress test — inducing packet loss on EFA (where SRD retransmits lost packets out of order) and verifying that no completion callbacks fire before data is visible, perhaps through a GPU kernel that checks received data and records anomaly counts.

3. Multi-model KvCache transfer evaluation — testing at least one dense model (Qwen3-235B as done), one MoE model (DeepSeek-V3), and one smaller model (e.g., Llama-70B) to establish whether the compute-to-transfer ratio is favorable across model scales and architectures.

4. Scaling the MoE microbenchmarks to EP128 and beyond — the current EP64 maximum is production-relevant for DeepSeek-V3 (which uses EP64), but future models with more experts or larger clusters would stress the linear scaling of WRITE posting time (Table 9). Extrapolating from EP64 to EP128 suggests ~17 µs on ConnectX-7 and ~56 µs on EFA for scatter posting, which might become a noticeable fraction of dispatch latency on EFA.

5. Power and utilization metrics — the host-proxy design keeps a CPU core busy polling completion queues and GDRCopy. The paper does not report CPU utilization, which matters for total cost of ownership — if the proxy thread consumes an entire CPU core per GPU, this is a non-trivial resource cost in production.

6. Comparison with a hypothetical "GPU-initiated on EFA" scenario — the paper argues that GDA is not available on EFA, but doesn't discuss whether a user-space GPU kernel could directly post to libfabric send queues (the equivalent of IBGDA for EFA). If this were architecturally possible but simply not yet implemented by AWS, the host-proxy design's lifespan might be limited. The paper's argument that next-generation NVLink domains will make this moot (Section 8) is forward-looking but doesn't address current-generation EFA deployments that will remain in service for years.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Unaccounted For in the 4× Efficiency Claim

The assumption or constraint. The compute-optimal allocation framework relies on knowing each prompt's difficulty before selecting the test-time strategy. The paper's method for estimating difficulty — whether oracle (pass@1 over 2048 base-model samples) or predicted (PRM final-answer score averaged over 2048 samples) — involves generating and scoring 2,048 complete solutions per question before any strategy is applied. The paper acknowledges this explicitly (Section 3.2):

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The cost of difficulty estimation is orders of magnitude larger than the test-time compute budget being optimized. Generating 2,048 samples costs 2,048 generations, while the largest test-time budgets studied are 256–512 generations. A deployment that follows the paper's protocol would spend 4–8× more compute on difficulty estimation than on actually solving the problem. The reported 4× efficiency gains (e.g., matching best-of-256 with 64 generations in Figure 8) are computed after difficulty is known, without amortizing the cost of learning it. In practice, the total cost would be difficulty estimation + strategy execution, and the former would dominate the latter, potentially making the "compute-optimal" strategy more expensive overall than simply running best-of-N with the same total budget.

This is not a minor accounting detail. For a deployment serving thousands of queries, the difficulty estimation cost per query would be prohibitive. The 4× figure is best understood as an upper bound on achievable efficiency that can only be approached if difficulty estimation is made radically cheaper.

What evidence exists in the paper. The paper offers no experimental characterization of the tradeoff between difficulty estimation accuracy and cost. There is no experiment that sweeps the number of samples used for difficulty estimation (e.g., 8, 32, 128, 512) and measures how the compute-optimal policy degrades. There is no measurement of whether the PRM-based predicted difficulty bins (which require no ground-truth labels but still require 2,048 samples) can be replaced with a cheaper surrogate — for instance, using the distribution of PRM scores over only 4–8 initial samples as a difficulty signal.

Mitigation status. The paper flags this explicitly as future work (Section 3.2):

"We leave exploration of other lower-cost methods for difficulty estimation to future work, including pretraining or finetuning models to directly predict difficulty of a question without requiring initial samples."

This is a frank acknowledgment, but it means the paper's central practical claim — that difficulty-conditioned allocation yields 4× efficiency gains — is premature as a deployment prescription. Until a cheap difficulty estimator is demonstrated, the claimed gains remain analytical rather than operational.

6.2 Test-Time Compute Cannot Extend the Base Model's Fundamental Capability Ceiling

The assumption or constraint. The entire framework treats test-time compute as a mechanism for amplifying existing capability within the base model's proposal distribution, not for creating new capability. If the base model's pass@1 on a problem is near zero (it almost never produces a correct solution even with unbounded sampling), no test-time strategy — search, revision, or their combination — can recover correct answers.

This is not a hidden assumption; it is explicit in the results. The paper states in the Section 7 takeaway:

"test-time compute amplifies existing capability but does not create it from nothing"

The consequence. On difficulty bin 5 (the hardest ~20% of MATH problems), all methods produce near-zero accuracy regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% across all search methods and all budgets. In Figure 7 (right), bin 5 accuracy is roughly 2–3% irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, while the ~14× larger pretrained model achieves substantially higher accuracy (the star in Figure 9 is visibly above the scaling line). This means that for genuine out-of-distribution reasoning or problems requiring capabilities the base model simply does not possess, test-time compute provides no path forward — pretraining remains the only viable investment.

This boundary is sharper than the paper's narrative might suggest to a casual reader. The abstract claims that "a smaller model augmented with compute-optimal test-time strategies can outperform a ~14× larger pretrained model," which is true for easy-to-medium problems but emphatically false for hard problems. The paper is transparent about this in the detailed results, but the headline framing understates how brittle the substitution is to the problem difficulty distribution. A deployment with a problem mix skewed toward hard problems (e.g., frontier mathematical reasoning, novel coding tasks) would see zero benefit from test-time compute and would be better served by training a larger model.

What evidence exists in the paper. All three sets of difficulty-breakdown figures confirm the flatlining of bin 5 performance: Figure 3 (right) for search, Figure 7 (right) for revisions, Figure 9 (both plots) for the FLOPs-matched comparison. The evidence is consistent and strong: on the hardest problems, the base model's proposal distribution contains essentially no correct answers, so search and revision have nothing to work with. In the FLOPs-matched comparison (Section 7), the bar charts in Figure 1 show test-time compute yielding a −52.9% relative disadvantage on hard problems for PRM search at R ≫ 1, and −37.2% for revisions, confirming that test-time compute is actively worse than scaling pretraining on these problems.

Mitigation status. The paper is admirably transparent about this limitation and does not attempt to mitigate it. The Section 7 takeaway box explicitly states both the regime where test-time compute wins (easy-to-medium, low R) and where pretraining wins (hard, high R). The limitation is intrinsic to the approach — test-time compute is a post-hoc optimization over the base model's output distribution, and no amount of clever allocation can generate correct answers the base model cannot produce. The only mitigation is to not use test-time compute for problems outside the base model's capability range, which requires the difficulty estimator to correctly identify such problems — a circular dependency given Limitation 6.1.

6.3 The Compute-Optimal Policy Is Trained and Evaluated on a Single Benchmark with a Single Model Family

The assumption or constraint. All experiments — compute-optimal strategy selection, difficulty bin calibration, FLOPs-matched comparison — are performed on the MATH benchmark (500 test questions) using PaLM 2-S* as the base model. The paper acknowledges this (Section 4) but argues that the model is "representative of the capabilities of many contemporary LLMs" and that MATH is a well-suited domain for studying test-time compute.

The consequence. Several aspects of the findings could be specific to this model-benchmark combination and might not generalize:

• PRM over-optimization behavior (beam search degrading on easy problems at high budgets, Figure 3 right) depends on the PRM's calibration error patterns relative to the base model's output distribution. A base model with different error characteristics (e.g., an LLM with stronger reasoning but more verbose outputs) might exhibit different over-optimization thresholds, changing which search method is optimal at which difficulty level.
• Revision model training (Section 6.1) relies on the base model's in-context learning ability to produce improved answers when conditioned on incorrect previous attempts. This ability varies substantially across model families — some models may not learn the revision skill from the described fine-tuning procedure, making the sequential revision gains (Figure 8) model-dependent.
• The MATH benchmark consists of competition-level math problems with exact answers and clean grading. It is unknown whether the difficulty-dependent patterns — beam search helping medium problems, revisions helping easy problems — generalize to other reasoning domains (code generation, logical deduction, scientific QA) or to tasks requiring factual knowledge rather than step-by-step inference. Open-ended generation tasks where correctness is ambiguous or multi-dimensional may not admit the same verifier-training and difficulty-estimation pipeline at all.

What evidence exists in the paper. None. There is no experiment with a different base model (e.g., a non-PaLM LLM, a smaller model, a code-focused model), no experiment on a different benchmark (e.g., GSM8K, HumanEval, ARC), and no analysis of whether the difficulty-bin thresholds would shift under distributional change. The 500-question test set is acceptable for a single-benchmark study but means that the compute-optimal policy is selected based on ~50 questions per difficulty bin (500 / 5 bins = 100 per bin, then split by two-fold cross-validation). This is a small sample for discrete strategy selection — a single anomalous question could change which strategy is deemed "optimal" for a bin.

Mitigation status. The paper acknowledges this implicitly by not claiming cross-domain generalization, but it does not explicitly discuss the limitation. Section 8 does not list multi-benchmark evaluation or cross-model replication as future work. The open-source release of the code and models would partially mitigate this by enabling other researchers to test generalization, but the paper's findings should be treated as specific to MATH + PaLM 2-S* until replicated.

6.4 The Revision Model Exhibits a 38% Correct-to-Incorrect Reversion Rate and Is Fragile to Training Methodology

The assumption or constraint. The revision model is fine-tuned exclusively on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). At test time, the model generates a chain of revisions, and intermediate answers in the chain may be correct (since the model improves over steps, as shown in Figure 6 left). When the model conditions on a correct answer in its context, it has no training signal for what to do — it was never trained on sequences where the in-context answer is correct. The consequence is a substantial reversion rate:

"approximately 38% of correct answers get converted back to incorrect ones" (Section 6.1)

The consequence. This means the revision chain is not monotonically improving — later steps are not guaranteed to be better than earlier steps, and a correct answer at step k may be lost by step k+1. This forces the system to use within-chain selection mechanisms (majority voting or verifier-based selection) that evaluate every answer in the chain and pick the best one, rather than simply taking the final revision. This adds computational overhead (every answer must be scored, not just the last one) and makes the effective accuracy lower than what a monotonically-improving revision model could achieve.

More fundamentally, the revision model is fragile to the training data construction. The ReST$^{EM}$ experiment (Appendix K, Figure 16) shows that attempting to further optimize the revision model with RL-style on-policy training substantially degrades performance — the fully-sequential curve drops from ~42% to ~33.5% at 256 generations. The authors hypothesize that "on-policy data collection in ReST$^{EM}$ exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This means the positive revision results depend on specific, carefully-controlled training choices (offline data construction, edit-distance-based incorrect-correct pairing, training only on correct-token loss) that may not transfer to other training pipelines.

What evidence exists in the paper. The 38% figure is reported in Section 6.1 as a motivated observation, not a measured ablation — the paper does not provide a breakdown of when and why correct answers get reverted (e.g., is it more common on certain problem types? At certain revision depths?). The ReST$^{EM}$ degradation is documented in Appendix K and Figure 16, with the paper speculating about causes but not investigating systematically. The fact that a seemingly-reasonable training improvement (on-policy RL optimization) makes the model strictly worse is a red flag about the approach's robustness.

Mitigation status. The paper mitigates the reversion problem with within-chain selection (verifier and majority voting), which recovers the best answer in the chain even if later revisions are worse. This is a workaround, not a solution — it treats the symptom (reversion) rather than the cause (training-data distribution mismatch between train and test). The paper does not propose training the model to recognize when no revision is needed (e.g., by including correct→correct training examples) or otherwise addressing the root cause. The ReST$^{EM}$ failure is left as an open question.

6.5 The FLOPs-Matched Comparison Uses a Weakened Pretraining Baseline

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately ~14× more parameters, holding training data fixed. The paper acknowledges (Section 7) that this departs from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model uses only greedy decoding with no test-time compute augmentation of its own — no majority voting, no best-of-N, no search, no revisions.

The consequence. Both choices weaken the pretraining baseline, making test-time compute appear more favorable than it would be against a properly optimized larger model:

• Parameters-only scaling is not compute-optimal. Chinchilla scaling laws (Hoffmann et al., 2022) show that for a given increase in pretraining FLOPs, the optimal allocation is to scale model parameters and training tokens equally. A compute-optimally trained ~14× larger model (with appropriately scaled data) would outperform a parameters-only-scaled model, shrinking or reversing the reported advantages of test-time compute (e.g., +27.8% on easy questions at R ≪ 1 for revisions).

• Greedy decoding is a weak inference strategy for the larger model. The paper's compute-optimal framework demonstrates that adaptive test-time compute can provide 4× efficiency gains over best-of-N. If even a modest test-time budget (e.g., best-of-8 or best-of-16) were given to the larger model, its performance would likely improve substantially. The paper never evaluates this — the comparison is always compute-optimal test-time scaling on the smaller model vs. greedy decoding on the larger model. A fairer comparison would give both models the same test-time inference budget and compare their performance, or at minimum give the larger model a basic best-of-N budget.

What evidence exists in the paper. None directly. The paper does not provide an ablation where the larger model receives any test-time compute augmentation. The single data point of the larger model's greedy performance (the stars in Figure 9) is the only comparison point. The paper does not discuss how the results would change if the larger model were given even a modest inference budget, or if the larger model were Chinchilla-optimally trained.

Mitigation status. The paper frames this choice as "representative of a canonical approach to scaling pretraining compute" (the LLaMA paradigm), which is accurate as a description of common practice. But it does not address the fairness concern. The Section 8 future work acknowledges that "compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally" is left to future work, but this is stated as a forward-looking extension, not as a limitation of the current comparison. A reader skimming the abstract and Figure 1 bar charts might conclude that test-time compute is generally preferable to pretraining for easy-to-medium problems, when the actual finding is narrower: test-time compute with an aggressively optimized inference strategy beats a suboptimally-trained larger model with no inference optimization at all.

6.6 The Paper Does Not Consider Latency or Wall-Clock Time, Only Total FLOPs

The assumption or constraint. All compute budgets are measured in generation equivalents (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores latency — the wall-clock time required to execute a strategy. The paper explicitly studies sequential-to-parallel tradeoffs for revisions (Section 6.2) and sequential revision chains (where each revision depends on the previous one), but evaluates them only in terms of accuracy per generation, never accuracy per second.

The consequence. The compute-optimal policy often favors sequential strategies — fully sequential revisions for easy problems (Figure 7 right, bin 2), or moderate sequential-to-parallel ratios for medium problems (bin 3). Sequential revisions are inherently serial: each revision step depends on the output of the previous step, so a chain of length N takes roughly N × the latency of a single generation, regardless of how many GPUs are available.

In contrast, parallel best-of-N sampling is embarrassingly parallel — all N samples can be generated simultaneously given sufficient hardware. A deployment with 256 GPUs could run best-of-256 in the time it takes to generate one sample; a sequential revision chain of depth 256 would take 256× longer, even though both strategies consume "256 generations" of compute. For latency-sensitive applications — interactive chatbots, real-time code assistants, online tutoring — the sequential-heavy strategies favored by the compute-optimal policy may be practically unusable even if they are FLOPs-efficient. The paper never discusses this tradeoff.

What evidence exists in the paper. None. There is no measurement of wall-clock time, no reporting of generation latency, no analysis of how the optimal sequential-to-parallel ratio changes if latency is added to the objective (e.g., accuracy per second rather than accuracy per FLOP). The revision model's per-step pass@1 trajectory (Figure 6 left) shows improvement out to 64 steps, but the paper does not mention that a 64-step sequential chain would take ~64× longer than a parallel sampling approach using the same number of total generations.

Mitigation status. The paper does not address this limitation. It does not propose a latency-aware objective, suggest how to incorporate latency into the compute-optimal framework (e.g., by adding a time constraint or a latency penalty term), or discuss the practical implications of the sequential-to-parallel tradeoff for deployment. This is a significant gap for practitioners, because production LLM serving systems have hard latency constraints (e.g., 100 ms for interactive use, 1 second for batch processing) that are not captured by FLOPs-based optimization. The paper's recommendations — favor sequential revisions for easy problems, use beam search for medium problems — may be optimal in FLOPs but suboptimal in wall-clock time, and the paper provides no guidance for navigating this tradeoff.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new communication algorithm or a novel NIC architecture — it introduces a design methodology for portable high-performance point-to-point communication in ML systems, and in doing so, it reframes a problem that the field had largely accepted as intractable. The prevailing sentiment before this work was that performant point-to-point RDMA for LLM workloads required hardware-specific optimizations (GPU-initiated RDMA, RC ordering, vendor-specific drivers), and that portability across ConnectX and EFA would necessarily come with a substantial performance penalty — as evidenced by NVSHMEM's order-of-magnitude degradation on EFA (Figure 9: 1,669 µs vs. 286 µs for dispatch at EP16). The paper's core reframing is that the barrier to portability is not API incompatibility or performance characteristics, but a single, specific assumption: in-order delivery. By identifying reliable-but-unordered transport as the common subset of RC and SRD capabilities (Table 1), and by building completion notification on the IMMCOUNTER primitive that does not depend on message ordering, the paper demonstrates that portability and performance are not in tension — they can be achieved simultaneously, and in some regimes (ConnectX-7 at EP32 and EP64), the portable approach actually outperforms the hardware-specialized alternative.

This is a reframing with immediate practical consequences, not a paradigm shift. It does not change how the field thinks about collective communication (NCCL remains the right tool for all-reduce), nor does it alter the fundamentals of RDMA. But it does establish that the design space for point-to-point communication in ML systems is larger than previously assumed: the choice is not between "fast but vendor-locked" (DeepEP) and "portable but slow" (NVSHMEM on EFA). There is a third option — portable by construction, performant by design — and fabric-lib is its existence proof. This reframing should change how infrastructure teams approach hardware diversity: rather than treating EFA support as a porting burden to be undertaken grudgingly after the ConnectX implementation is complete, teams can build on the reliable-but-unordered abstraction from the start, knowing that the resulting system will work on both hardware families and will achieve competitive performance on each.

The paper also resolves a tension in the MoE serving literature that was previously unarticulated. DeepEP demonstrated that GPU-initiated RDMA could achieve extremely low dispatch/combine latencies, establishing a de facto assumption that GPU initiation was necessary for competitive decode performance. The paper shows that this assumption is incorrect — a host-proxy design can match or exceed GPU-initiated latency on ConnectX-7 at production expert-parallel scales (Figure 9: 124 µs vs. 160 µs for dispatch at EP64, 203 µs vs. 327 µs for combine) — because the bottleneck at these scales is not the PCIe traversal latency of the proxy (~1.5 µs at p50, Table 8) but the network transfer time and the coordination cost of routing information exchange. This redirects optimization effort: rather than pursuing GPU-initiated RDMA as a silver bullet, effort should go toward reducing coordination overhead (e.g., through speculative data transfer, as the private buffer design in Section 6.2 does) and improving bulk transfer efficiency. This is a more portable optimization target, since coordination overhead exists on all hardware while GPU-initiated RDMA is hardware-specific.

The identification of verifier over-optimization as the primary bottleneck for test-time compute scaling — while from a different paper and noted only briefly here for structural completeness — is echoed in fabric-lib's finding that the host proxy is not the bottleneck. In both cases, the paper's contribution is diagnostic: identifying which component of a complex system actually limits performance, and thereby redirecting effort away from intuitively-obvious but wrong targets (GPU initiation for MoE, search algorithm sophistication for test-time compute) toward the real bottlenecks (coordination overhead for MoE, verifier robustness for test-time compute).

Follow-Up Research This Work Enables

1. Adaptive routing-information exchange for MoE dispatch using learned difficulty prediction. The private buffer ablation (Figure 11) shows that fabric-lib's MoE dispatch is sensitive to the number of tokens transferred speculatively: too few tokens, and routing-exchange latency is exposed; too many, and GPU memory is wasted. The optimal private buffer size depends on the latency of routing-information scatter, which in turn depends on the number of peers, the NIC type, and the current network load. A natural extension is to replace the static private buffer size with a learned predictor that estimates routing-exchange latency from observable features (EP degree, NIC type, recent completion queue polling intervals) and dynamically sizes the speculative transfer. The experiment would compare static vs. adaptive sizing on a mixed workload of varying EP configurations and network congestion levels, measuring both p50 and p99 dispatch latency and GPU memory utilization. The paper provides the measurement infrastructure (Table 8 event instrumentation, Figure 11 ablation framework) to support this.

2. Stress-testing IMMCOUNTER correctness under adversarial packet loss on EFA. The paper's IMMCOUNTER correctness argument (Section 3.3) relies on PCIe ordering guarantees and the RDMA specification's requirement that WRITEIMM data is delivered before the immediate value. However, EFA's SRD protocol retransmits lost packets, and the paper does not test whether retransmission can cause the immediate value to be delivered on a first transmission while the data payload is still being retransmitted on a later attempt — a violation of the RDMA spec that might occur in specific firmware versions or under pathological loss patterns. A stress-test experiment would induce controlled packet loss (e.g., using EFA's diagnostic interfaces or an external traffic shaper), run continuous WRITEIMM transfers with a GPU kernel that verifies received data against expected values, and record any instances where the IMMCOUNTER callback fires before all data is visible to the GPU. A negative result (no violations) would strengthen the paper's correctness claims for production deployment; a positive result would identify a firmware bug or spec ambiguity that needs mitigation (e.g., polling GPU memory after counter notification to verify, or adding an explicit GPU-side fence).

3. Combining fabric-lib with computation-communication overlap frameworks for prefill. The paper acknowledges (Section 7.4.3) that fabric-lib's prefill performance lags DeepEP due to the lack of sender-side partial sum accumulation and chunked transfers. Recent work on computation-communication overlapping (Flux, COMET, TileLink; cited in Section 2.3) has shown that carefully pipelining GEMM with communication can hide transfer latency. A natural combination is to integrate fabric-lib's portable scatter primitives into an overlap framework: while the MoE expert GEMM is executing on one chunk of tokens, use fabric-lib to scatter the next chunk's tokens in the background. The key question is whether fabric-lib's host-proxy architecture — which requires CPU involvement for work submission — can be cleanly integrated with GPU-side overlap scheduling, or whether the CPU proxy becomes a serialization point that prevents effective pipelining. The experiment would measure prefill latency for DeepSeek-V3 at EP64 with and without overlap, comparing fabric-lib's host-proxy approach against a hypothetical GPU-initiated baseline (DeepEP on ConnectX-7) to isolate whether the proxy itself limits overlap effectiveness. The paper's Table 8 provides the per-event proxy latency data needed to model this integration.

4. Extending fabric-lib to support additional NIC families (eRDMA, Falcon, Broadcom) and measuring the portability cost. The paper claims (Section 8) that "for RC-compatible NICs, the internal implementation would resemble the ConnectX path," implying that porting to new NICs is primarily an engineering effort rather than an architectural challenge. A systematic validation would port fabric-lib to at least two additional NIC families — Alibaba eRDMA (RC-compatible, deployed on Alibaba Cloud GPU instances) and Google Falcon (a new transport with unknown ordering semantics; Singhvi et al., 2025) — and measure (a) the lines of code changed per NIC backend, (b) whether any API changes were needed to accommodate NIC-specific quirks, and (c) the achieved bandwidth and MoE dispatch latency compared to the ConnectX-7 baseline. This would quantify the "portability cost" that the paper's design claims to minimize, and would identify whether the reliable-but-unordered abstraction actually covers all current hardware or whether new transports introduce semantic requirements not captured in Table 1.

5. Scaling MoE dispatch/combine to next-generation NVLink domains (GB200 NVL72) and measuring whether the host proxy becomes a bottleneck in new regimes. The paper argues (Section 8) that "next-generation GPUs with wide NVLink domain (e.g., GB200 NVL72) shift communication off RDMA entirely," implying that the host-proxy overhead will become less important as intra-node communication grows. However, the counter-argument is that as NVLink domains grow, the remaining inter-node RDMA traffic becomes the long tail that determines end-to-end latency — and if the host proxy adds even 10 µs of overhead to each inter-node transfer, that could dominate in a 72-GPU NVLink domain where intra-node latency is sub-microsecond. A concrete experiment would run the MoE dispatch/combine microbenchmarks on a GB200 NVL72 cluster (or a simulated version using current hardware with NVLink-to-RDMA ratios matching NVL72), measure the fraction of tokens that traverse RDMA versus NVLink, and determine whether fabric-lib's host-proxy latency is visible in end-to-end decode step time. If the proxy does become a bottleneck at this scale, this would motivate developing a hybrid architecture where fabric-lib uses GPU initiation for the (few) inter-node transfers while keeping the host proxy for initialization and connection management.

6. Applying fabric-lib's reliable-but-unordered abstraction to distributed training communication patterns beyond weight updates. The paper's three production use cases — disaggregated inference, RL weight updates, and MoE dispatch/combine — are all inference-adjacent. Distributed training involves additional point-to-point patterns that are currently served by collectives or ad-hoc NCCL SEND/RECV: gradient all-reduce for expert-parallel layers in MoE training (where each expert's gradient only needs to be communicated among the subset of GPUs hosting that expert, not all GPUs), pipeline-parallel activation shipment (point-to-point transfers between consecutive pipeline stages), and ZeRO-3 parameter gathering (similar to FSDP full_tensor() but in a training context). An experiment would replace the communication backend for one of these training patterns — e.g., expert-parallel gradient reduction in a DeepSeek-V3 training run — with fabric-lib scatter/gather, and measure training step time, network utilization, and CPU overhead compared to the existing NCCL-based implementation. The hypothesis is that fabric-lib's ability to target specific peers without global synchronization would reduce the straggler effect in expert-parallel all-reduce, but the host-proxy overhead might be unacceptable for training-critical-path operations. This would establish the boundary of fabric-lib's applicability beyond inference.

Practical Applications and Downstream Use Cases

1. Multi-cloud MoE inference serving with uniform latency across providers. An organization deploying a large MoE model (DeepSeek-V3 class, EP≥32) that needs to serve on both AWS (EFA) and on-premise or colocated InfiniBand (ConnectX-7) can use fabric-lib as a single communication backend. Prior to this work, such a deployment would require maintaining two separate MoE kernel implementations — one based on DeepEP for ConnectX, and a separate (likely NVSHMEM-based) implementation for EFA that achieved 3–6× lower throughput (Table 6: 20.97 vs. 66.75 tokens/s at batch=2). With fabric-lib, the same kernel code achieves 66.75 tokens/s on EFA and 78.42 tokens/s on ConnectX-7, providing consistent user-facing latency regardless of which cloud provider serves a given request. The concrete benefit is operational: a single codebase, a single set of kernel parameters, and a single performance tuning target, reducing engineering overhead and eliminating the risk of provider-specific performance regressions.

2. Rapid RL fine-tuning loops for trillion-parameter models on commodity clusters. An AI lab running asynchronous RL fine-tuning (e.g., reasoning model improvement via GRPO or PPO) on trillion-parameter MoE models can use fabric-lib's point-to-point weight transfer to reduce the training-inference synchronization gap from minutes to seconds. The paper demonstrates 1.2–2.0-second weight updates for Kimi-K2 (1T), DeepSeek-V3 (671B), and Qwen3-235B on 256 training GPUs (Section 7.3), compared to "tens to hundreds of seconds" for existing frameworks. This has a direct impact on RL sample efficiency: in a typical RL fine-tuning loop, inference workers generate rollouts using the current policy while training workers compute policy updates. If weight synchronization takes 100 seconds, inference workers are idle or running stale policies for 100 seconds per iteration; at 1.2 seconds, they are idle for only ~1% of the iteration time, enabling more frequent policy updates and potentially faster convergence. The concrete benefit is measured in wall-clock time to a target reward threshold, not just in transfer latency — a follow-up RL training run comparing fabric-lib's P2P transfer against a Rank0-based collective approach on the same hardware would quantify the downstream impact on training throughput.

3. Disaggregated inference with elastic prefill/decoder scaling on EFA. A cloud-based LLM serving platform using AWS p5 instances can deploy disaggregated inference with independent scaling of prefill and decoder pools, using fabric-lib's KvCache transfer for the cross-pool communication. The paper shows that for Qwen3-235B on H200 with EFA, KvCache transfer adds only 1.9–21.5% TTFT overhead (Table 3), with the per-layer transfer time (0.66–1.61 ms) consistently hidden behind compute (2.27–34.9 ms). This enables a cost-optimized deployment where prefill capacity (which scales with prompt length and request rate) and decode capacity (which scales with total sequence length and number of concurrent users) are provisioned independently, avoiding the over-provisioning that occurs when both stages are colocated. The concrete benefit is a reduction in GPU-hours per million requests: if prefill is the bottleneck 30% of the time and decode 70% of the time, independent scaling allows adding only prefill GPUs during prefill-bound periods and only decode GPUs during decode-bound periods, rather than adding both together. The paper's latency numbers provide the per-transfer cost that a capacity planner would plug into a cost model.

4. Portable benchmarking and infrastructure testing across cloud providers. An organization evaluating different cloud providers for an LLM deployment can use fabric-lib's uniform API to run identical communication microbenchmarks on AWS (EFA), on-premise InfiniBand (ConnectX-7), and Alibaba Cloud (eRDMA, once supported), producing directly comparable bandwidth, latency, and operations-per-second numbers (Table 2). Prior to fabric-lib, such comparisons would require different benchmarking tools for each provider (fi_rma_bw for EFA, ib_write_bw for ConnectX, different APIs for eRDMA), making it difficult to distinguish hardware differences from software overhead. The concrete benefit is in procurement and capacity planning: a cloud architect can run a single benchmark suite on candidate providers, measure achieved bandwidth at the message sizes relevant to their workload (256 KiB for MoE routing, 16 MiB for weight transfers, 32 KiB pages for KvCache), and make a hardware decision based on comparable numbers rather than vendor-provided peak specs that may not reflect application-level performance.

When to Prefer fabric-lib Over Alternatives

The paper positions fabric-lib against specific named alternatives — DeepEP for ConnectX-7 MoE kernels, NVSHMEM (pplx-kernels) for portable MoE kernels, NIXL for point-to-point transfers, and collective-based (Rank0) approaches for RL weight updates — and provides quantitative comparisons that enable a decision rule based on workload characteristics and hardware constraints.

• Prefer fabric-lib for MoE dispatch/combine when: (a) you are deploying on EFA, where DeepEP is unavailable and NVSHMEM achieves 3–6× lower throughput (Table 6), or (b) you are deploying on ConnectX-7 at EP≥32, where fabric-lib matches or exceeds DeepEP latency (Figure 9: 110 µs vs. 124 µs dispatch at EP32, 203 µs vs. 327 µs combine at EP64) while providing portability, or (c) you need a single codebase across both EFA and ConnectX-7. Prefer DeepEP only when deploying exclusively on ConnectX-7 at EP≤16 and prefill performance is critical (DeepEP's sender-side partial sum optimization provides lower combine latency at small EP scales; Figure 10: 0.96 ms vs. 1.41–1.87 ms at EP8). Prefer NVSHMEM only when GPU-initiated RDMA is a hard requirement and EFA support is not needed.

• Prefer fabric-lib for RL weight updates when: you are training trillion-parameter models with FSDP on 128+ GPUs and need sub-2-second weight synchronization. The Rank0-based collective approach (Figure 4a) becomes a NIC bottleneck at this scale, while fabric-lib's direct P2P WRITE saturates all NICs simultaneously. The paper's 1.2–2.0-second numbers (Section 7.3) are 50–100× faster than reported collective-based approaches on comparable hardware.

• Prefer fabric-lib for disaggregated inference KvCache transfer when: (a) you need dynamic, per-request prefiller-to-decoder assignment without global synchronization (fixed membership and synchronized initialization in collectives prevent this; Section 2.2), or (b) you are deploying on EFA, where Mooncake's Transfer Engine lacks support. Prefer a collective-based approach only when all prefill and decode nodes are static and known at cluster startup, and NCCL's SEND/RECV primitive is sufficient for the transfer pattern.

• Prefer NIXL over fabric-lib when: you are already within the NVIDIA ecosystem (UCX, TensorRT-LLM, Triton Inference Server) and EFA support is not required. Figure 8 shows comparable point-to-point bandwidth, and NIXL's tighter integration with NVIDIA's inference stack may reduce integration effort. Prefer fabric-lib when portability across NIC vendors is a requirement, when EFA support is needed, or when you need the higher-level abstractions (paged WRITE, scatter/barrier, IMMCOUNTER) that fabric-lib provides as first-class API primitives.

• Prefer collectives (NCCL, torch.distributed) over fabric-lib when: your communication pattern is dense, structured, and static — tensor parallelism all-reduce, data-parallel gradient synchronization, or pipeline-parallel send/recv with fixed peer pairs. fabric-lib complements collectives for the point-to-point, dynamic, sparse patterns that collectives handle poorly; it does not replace them for dense all-to-all or all-reduce operations where NCCL's tree and ring algorithms provide asymptotically better bandwidth scaling.