Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

🎯 Pitch

Mooncake reveals that the largest throughput gains in disaggregated LLM serving come not just from separating prefill and decoding, but from predicting overload and rejecting doomed requests before they waste GPU time on a prefill that will never finish. By implementing a KVCache-centric scheduler with early rejection and cache-aware routing, the architecture sustains SLOs under extreme load and enables the Kimi service to process 75% more requests on real traffic.

1. Executive Summary

Mooncake introduces a KVCache-centric disaggregated architecture for LLM serving that separates prefill and decoding clusters while leveraging underutilized CPU, DRAM, and SSD resources to implement a disaggregated KVCache pool. The system's core is its KVCache-centric scheduler, which balances overall effective throughput against latency-related Service Level Objectives (TTFT and TBT) through mechanisms including cache-aware prefill scheduling (routing requests to instances with the longest prefix cache matches while accounting for queue times), cache load balancing (heuristic-based hot-spot migration that replicates frequently accessed KVCache blocks across nodes), and a prediction-based early rejection policy (system-level forecasting of decoding instance load to avoid accepting requests that will be rejected post-prefill, thereby reducing wasted computation). Evaluated on a dummy LLaMA2-70B model using ArXiv Summarization, L-Eval, simulated long-context data, and real workload traces from Kimi, Mooncake achieves up to a 525% throughput increase over vLLM in simulated long-context scenarios while satisfying SLOs, and enables Kimi to handle 75% more requests under real workloads, establishing that disaggregation with KVCache-centric scheduling yields substantial throughput gains only when the architecture explicitly addresses the unique challenges of overloaded scenarios through early rejection and load prediction.

2. Context and Motivation

The Core Problem: LLM Serving Systems Weren't Built for Overloaded, Long-Context, KVCache-Heavy Workloads

The paper addresses a fundamental mismatch between how existing LLM serving systems are designed and the real-world conditions faced by large-scale Model-as-a-Service (MaaS) providers like Kimi. The core problem is deceptively simple to state but complex in its implications: how do you serve LLM inference at scale when you're consistently overloaded, your users submit requests with extremely long contexts (tens of thousands of tokens), and the KVCache—the memory storing intermediate attention keys and values—is simultaneously your most valuable asset for reducing computation AND your most constrained resource for maximizing throughput?

This isn't a hypothetical scenario. The paper describes Kimi's actual operational reality: a service experiencing "rapid growth in user requests" where "the growth rate of the cluster's inference resources is far slower than the increase in incoming requests" (Section 7). This creates a persistent overload state that the paper explicitly distinguishes from the assumptions of prior work: "existing research on LLM serving assumes sufficient resources" and "assumes that all requests will be processed" (Section 1.1, Section 7). In Mooncake's world, the system must actively decide which requests to reject—and when to reject them—to avoid wasting computation on requests that will ultimately be dropped.

The three dimensions of the problem interact in non-obvious ways that make it genuinely hard:

Dimension 1: Long-context requests amplify the prefilling-vs-decoding imbalance. As shown in Figure 2 (left), prefill latency grows superlinearly with input length due to the quadratic complexity of attention. Meanwhile, decoding throughput increases sublinearly with batch size because decoding is memory-bound (Figure 2, right). When the average input length in real workloads is ~7,590 tokens with an average input-to-output ratio of ~720:1 (Section 4.2, Figure 5), the computational asymmetry is massive: most of the work happens in prefill, but the majority of wall-clock time may be spent in decoding. Coupling these stages on the same GPU—as vLLM and similar systems do—means that long prefill operations can starve decoding batches, causing Time Between Tokens (TBT) SLO violations that render user-facing services unusable.

Dimension 2: KVCache is the central scheduling object, but existing schedulers don't treat it as such. When a user submits a query that shares a long common prefix with another query (e.g., both refer to the same system prompt or same long document), the prefill stage can reuse the already-computed KVCache for the matching tokens rather than recomputing attention from scratch. In Kimi's workloads, up to ~50% of KVCache blocks can theoretically be reused even with finite storage (Section 4.2, Table 1). However, cache reuse introduces a tension: routing a request to the node with the longest prefix match reduces prefill computation but may overload that node, violating TTFT SLOs. Routing it elsewhere forces recomputation or KVCache transfer over the network, which also costs time. This isn't a static optimization—cache blocks have wildly varying popularity, with "over 50% of cache blocks remaining unused while certain blocks are accessed tens of thousands of times" (Section 4.2, Figure 6).

Dimension 3: Overload creates a time-lag problem that couples prefill and decoding scheduling despite disaggregation. Even when prefill and decoding are separated onto different physical nodes (as in disaggregated architectures), the two stages are logically coupled: if a request is accepted for prefill, it must eventually be scheduled for decoding. If the decoding cluster is overloaded and rejects the request after prefill has already completed, the prefill computation is wasted—tokens were generated, GPUs were occupied, but no billable output was produced. This creates a coordination problem: the system should predict whether decoding capacity will be available before committing prefill resources, but the prediction must account for the time lag between accepting a request and it arriving at the decoding stage.

Why This Problem Matters

Real-world impact: revenue and user experience for MaaS providers. For providers like Kimi, throughput directly translates to revenue—each successfully completed request generates billable output tokens. When a request is accepted for prefill but rejected at the decoding stage, all computation up to that point is wasted (Section 2: "otherwise, all previously consumed/generated tokens are not counted, and the corresponding resources are wasted"). The paper quantifies this: under a naive rejection strategy, 4,183 out of 23,000 requests were rejected in their overload experiment (Section 8.2, Table 3), and early rejection based on prediction reduced this to 3,589—a 14.2% reduction in wasted rejections. Each saved rejection represents prefill computation that wasn't thrown away. At the scale of a service handling millions of requests, this efficiency difference is enormous.

Theoretical significance: KVCache scheduling as a first-class systems problem. The paper makes a strong implicit argument that KVCache—traditionally treated as a memory management detail—is actually the central scheduling object in LLM serving. The KVCache determines where computation happens (through cache reuse), constrains batch sizes (KVCache occupies precious GPU VRAM), drives network traffic (when transferred between nodes), and links prefill and decoding decisions (the cache must be populated during prefill and consumed during decoding). The paper's architecture elevates KVCache from an implementation detail to the organizing principle of the entire system, with the global scheduler (Conductor) making decisions primarily based on cache distribution, cache transfer costs, and cache-induced load.

Practical urgency: GPU supply constraints make efficiency non-negotiable. The paper notes that "due to the current contingent supply of GPUs, elastically scaling out the inference cluster is typically unfeasible" (Section 2). This isn't a temporary condition—it's a structural feature of the current AI infrastructure landscape. When you can't simply add more GPUs to handle peak load, you must either reject requests or use what you have more intelligently. Mooncake's design philosophy—disaggregating resources into specialized pools and scheduling intelligently across them—is a response to hardware scarcity that will remain relevant as long as demand grows faster than accelerator manufacturing capacity.

Where Prior Approaches Fall Short

The paper identifies specific limitations across three categories of prior work:

Coupled architectures (vLLM, Orca, FasterTransformer) conflate prefill and decoding, causing interference. Systems like vLLM [13], which serve as the paper's primary baseline and arguably the most widely deployed open-source LLM serving system, run both prefill and decoding on the same GPU instances. This design is simple and works adequately for balanced workloads, but breaks down under the long-context, high-throughput conditions that characterize Kimi's traffic. The evaluation makes this concrete: in the real workload replay experiment (Section 8.1.3, Figure 13), vLLM's TBT distribution shows only 57% of requests meeting the TBT SLO, "with some requests exhibiting extremely high TBTs." Mooncake, with its disaggregated design, achieves ~100% TBT SLO compliance while also handling 75% more requests. The mechanism is clear: when a long prefill operation occupies a GPU in a coupled system, it delays the decoding batch, causing TBT spikes for all requests sharing that GPU.

The paper's explanation is more nuanced than simply "separation is better." It acknowledges that chunked prefill [15]—which breaks long prefill operations into smaller chunks interspersed with decoding steps—can mitigate the interference problem without requiring physical disaggregation. However, it identifies two reasons this isn't sufficient (Section 5): (1) long-context prefill requires cross-node parallelism that coupled designs handle poorly, and (2) chunked prefill's inlining of prefill chunks into decoding batches increases the VRAM occupation cost (the KVCache must reside in VRAM for longer, displacing other requests). The key insight is that coupled vs. disaggregated isn't a clean either/or—it's a continuum where the optimal design depends on workload characteristics—but for Kimi's long-context-heavy workload, full disaggregation is clearly superior.

Existing disaggregated architectures (Splitwise, DistServe, TetriInfer) focus on throughput optimization under the assumption of sufficient capacity. The paper positions itself alongside a wave of concurrent disaggregated LLM serving work—Splitwise [7], DistServe [8], TetriInfer [9]—but draws a sharp distinction: these works "assume that all requests will be processed" and focus on optimizing throughput and latency under those conditions. Mooncake, in contrast, operates under persistent overload where rejection decisions are central to the scheduling problem. This isn't a minor difference—it fundamentally changes the scheduling objective. In a capacity-sufficient system, you want to minimize latency and maximize throughput for all accepted requests. In an overloaded system, you want to accept precisely the set of requests that can be completed under SLOs while rejecting others as early as possible to conserve resources for the accepted requests. The paper's early rejection mechanism (Section 7) is a direct response to this overloaded reality, and it's the first work to systematically address the problem in a disaggregated LLM serving context.

The paper also critiques existing disaggregated work for not fully addressing the KVCache coordination challenge. Separating prefill and decoding creates the KVCache transfer problem: the prefill node generates the KVCache, but the decoding node needs it. Prior work either assumed KVCache fits in VRAM on the decoding side or didn't address the scheduling implications of cache location. Mooncake's contribution is treating the distributed KVCache as a first-class scheduling resource, with Conductor making decisions based on both cache content and cache location.

Prefix caching systems (Prompt Cache, SGLang, AttentionStore) optimize for cache hits but not for the scheduling implications of cache distribution. Prefix caching—reusing KVCache for common prefixes across requests—is widely studied. Prompt Cache [33] precomputes and stores frequently used KVCache. SGLang [34] uses RadixAttention with an LRU cache in a radix tree for automatic sharing. AttentionStore [35], the closest concurrent work, proposes a hierarchical KVCache system using cost-effective storage media. The paper acknowledges these as important foundations but identifies a gap: none of them integrate cache management with global scheduling under SLO constraints. You might know which node has the longest prefix cache match for a given request, but should you always route to that node? What if that node is already overloaded? What if the cache block is on a node experiencing network congestion? And critically, what if transferring the cache from a remote node costs more time than simply recomputing the prefilled tokens? Mooncake's scheduling algorithm (Algorithm 1 in Section 6.1) encodes these tradeoffs explicitly: it estimates TTFT as a function of queue time, prefill computation time (which depends on cache hit length), and KVCache transfer time (which depends on network conditions and data size), then routes to the instance that minimizes total TTFT—even if that means not using the longest-available cache match.

The paper also identifies a practical limitation of prior caching work that is often overlooked in academic settings: real reusability is much lower than benchmark reusability. Section 9 notes that "the real reusability in our online traces is much smaller than the results reproduced by open-source benchmarks"—theoretically up to only 50% even with infinite storage and TTFT tolerance. This means caching is important but not sufficient; it must be combined with other optimizations (disaggregation, effective scheduling, early rejection) to achieve substantial throughput gains.

Load prediction for LLM serving is underexplored, especially under overload. The paper's prediction-based early rejection mechanism (Section 7.4) addresses a gap that exists because prior work didn't need to solve it. If you assume all requests will be processed, you don't need to predict decoding load before accepting a prefill—you just accept everything and manage the queue. Under overload, you need to forecast whether decoding capacity will exist when the prefill completes, which requires predicting both current decoding instance load and the output lengths of in-flight requests (which determine how quickly requests will leave the decoding stage). The paper identifies this as an open problem and offers a pragmatic system-level approximation—assuming uniform decoding time per request—while explicitly flagging request-level length prediction as future work.

How Mooncake Positions Itself Relative to Existing Work

The paper's positioning can be understood along three axes:

Axis 1: From throughput-centric to SLO-constrained scheduling under overload. Prior work on LLM serving broadly falls into the "optimize for throughput and latency" paradigm. Mooncake reframes the objective as "maximize effective throughput subject to SLO constraints, where only completed requests count" (Section 2). This is a generalization: in capacity-sufficient regimes, the SLO constraints are slack and the objective reduces to throughput optimization. Under overload, the SLO constraints bind, and the optimization becomes about which requests to serve, not just how to serve them. The early rejection policy is the mechanism that operationalizes this reframing.

Axis 2: KVCache as the unifying abstraction for disaggregated scheduling. The paper's title—"A KVCache-centric Disaggregated Architecture"—is not merely descriptive; it's a design philosophy. Where Splitwise separates prefill and decoding to avoid phase interference, and DistServe optimizes parallel strategies per stage, Mooncake adds a third disaggregated component: the KVCache pool itself, distributed across the CPU DRAM and SSD of GPU cluster nodes. This means the scheduler's decision space expands to include cache location, cache transfer, and cache replication—all under the unified objective of minimizing TTFT while maximizing cache reuse. The architecture diagram (Figure 1) visually centers the KVCache pool, with prefill and decoding instances positioned as consumers and producers of cache blocks, all orchestrated by the KVCache-centric Conductor.

Axis 3: Production reality as the source of novel systems problems. The paper doesn't propose a theoretical framework and then evaluate it on benchmarks—it describes a system built to solve actual operational problems at Kimi, extracts generalizable insights, and validates them experimentally. The overload-oriented scheduling problem (Section 7) is a direct consequence of Kimi's rapid growth. The load fluctuation problem caused by naive early rejection (Section 7.3, Figure 9) was discovered during deployment, not anticipated from first principles. The insight that prediction-based early rejection is necessary to dampen these fluctuations emerged from observing real system behavior. This production-first approach gives the paper's contributions a groundedness that purely academic systems work sometimes lacks, while the open-sourced trace and use of a reproducible dummy model architecture (LLaMA2-70B) make the findings independently verifiable.

The paper also acknowledges its boundaries: it doesn't claim that disaggregation with KVCache-centric scheduling is universally optimal. It explicitly notes that the proportion of prefill to decoding instances can be preset based on workload stability (Section 8.1.1), that certain cache reusability patterns are application-specific (Section 9), and that the current difficulty of request-level output length prediction limits the sophistication of early rejection (Section 7.4). These are not weaknesses but honest characterizations of where engineering judgment is still required—the paper provides the architectural framework, but tuning it for specific workloads remains necessary.

3. Technical Approach

3.1 Reader Orientation

Mooncake is a production-grade LLM serving platform that physically separates the two stages of transformer inference—prefill (processing all input tokens in parallel) and decoding (generating output tokens one at a time)—across different GPU nodes, while also turning the CPU DRAM and SSDs of every node into a distributed, shared pool of KVCache blocks that can be transferred between nodes via RDMA, all orchestrated by a central scheduler called Conductor that decides which prefill node and which decoding node should handle each request, which cached KVCache blocks should be reused vs. recomputed vs. transferred across the network, and whether the request should even be accepted at all given current and predicted future system load. The problem it solves is: how do you maximize the number of successfully completed requests (which directly generates revenue) when your service is persistently overloaded, your users submit requests with extremely long input contexts (tens of thousands of tokens), and the KVCache—the memory storing intermediate attention keys and values for each token position—is simultaneously your most powerful tool for avoiding redundant computation AND your most constrained and unevenly distributed resource?

3.2 Big-Picture Architecture (Diagram in Words)

Mooncake has six major component categories, organized around the KVCache as the central scheduling object:

1. Prefill Instances (GPU nodes): These process all input tokens of a request in parallel to produce the first output token and the KVCache. They support chunked prefill (splitting very long inputs into smaller pieces processed in a pipeline across multiple GPUs or nodes) and layer-wise prefill (asynchronously loading and storing KVCache layer-by-layer during computation to hide transfer latency). Their goal is to minimize Time To First Token (TTFT) while reusing as much cached KVCache as possible.

2. Decoding Instances (GPU nodes): These run continuous batching—at each iteration, they take the KVCache for all active requests, generate one new token per request autoregressively, append the new keys/values to each request's KVCache, and remove completed requests. Their goal is to pack as many tokens as possible into each batch (to maximize GPU utilization) while keeping Time Between Tokens (TBT) below the SLO.

3. Disaggregated KVCache Pool (CPU DRAM + SSD of all nodes): Every node contributes its underutilized CPU memory and SSD storage to a globally addressable cache of KVCache blocks. Blocks are paged (fixed-size chunks, typically 512 tokens per block in the paper's trace), hash-indexed for deduplication (each block's hash incorporates both its own content and the hash of all preceding blocks—a Merkle-tree-like chain), and transferred between nodes via a separate RDMA-based component called Messenger. The pool supports LRU, LFU, or custom eviction policies.

4. Messenger (per-node transfer service): A separate process on each node that handles high-speed, cross-machine KVCache transfers using GPUDirect RDMA. It receives signals from Conductor to move specific blocks between the CPU memory of different nodes, and supports both pre-scheduled transfers and on-demand retrieval.

5. Conductor (global scheduler): The central brain. For each incoming request, Conductor: selects a prefill instance (balancing cache hit length, queue time, and transfer time to minimize TTFT); selects a decoding instance (based on current and predicted load to ensure TBT SLO compliance); decides whether to accept or reject the request (based on predicted system load after prefill completes); and manages cache replication/swapping (replicating hot blocks to multiple nodes to avoid transfer congestion, swapping cold blocks out to conserve DRAM). Conductor maintains per-instance estimates of queue times, cache contents, and network congestion.

6. Local Schedulers (per-instance): Each prefill and decoding instance has its own local scheduler that manages fine-grained execution. The decoding local scheduler double-checks that TBT SLOs can still be met when a request actually arrives (since load conditions may have changed since Conductor's initial decision), and may reject the request at this late stage if not—though this wastes prefill computation, motivating the prediction-based early rejection policy.

Information flows as follows: A request arrives at the API gateway → tokenized → Conductor queries all prefill instances for their current cache contents, queue lengths, and network status → Conductor selects a prefill instance and a decoding instance, initiates KVCache transfer from remote nodes if beneficial, and issues an accept/reject decision → if accepted, the prefill instance loads reusable KVCache from the pool, computes remaining prefill steps (layer by layer, streaming incremental KVCache to the decoding instance's CPU memory via Messenger as each layer completes), and produces the first output token → the decoding instance loads the full KVCache into GPU VRAM via asynchronous transfers overlapped with ongoing decoding batches → the request joins the continuous batching loop until completion → the response is returned to the user.

3.3 Roadmap for the Deep Dive

First, the disaggregation decision itself—why Mooncake maintains separate prefill and decoding pools despite the existence of chunked prefill as an alternative, and the concrete resource accounting (VRAM occupation cost, MFU implications) that drives this choice.
Second, the multi-node prefill mechanism using Chunked Pipeline Parallelism (CPP), since long-context prefill is the primary computational bottleneck and CPP is a novel application of pipeline parallelism to inference that avoids the overheads of sequence parallelism.
Third, the layer-wise prefill technique for overlapping KVCache transfer with computation, which is what makes the disaggregated prefill pool practical by eliminating VRAM as a scheduling constraint.
Fourth, the KVCache storage and transfer infrastructure—how blocks are hashed, indexed, transferred via Messenger, and what policies govern caching, eviction, and replication.
Fifth, the KVCache-centric scheduling algorithm (Algorithm 1), which is the intellectual core of the system—how Conductor trades off cache reuse against load balancing to minimize per-request TTFT.
Sixth, the cache load balancing heuristic—the hot-spot migration scheme that automatically replicates frequently accessed blocks without requiring precise future-usage predictions.
Seventh, the overload-oriented scheduling pipeline: early rejection, the load fluctuation problem it creates, and the prediction-based solution that dampens these fluctuations.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems design and engineering paper whose core idea is that KVCache—the intermediate key-value tensors produced during attention computation—should be treated as the central scheduling object in LLM serving, not merely as a memory management detail, and that building the entire architecture (disaggregation, transfer infrastructure, scheduling policy, overload handling) around KVCache enables substantial throughput gains under real-world overloaded, long-context workloads.

The Disaggregation Decision: Why Separate Prefill and Decoding Despite Chunked Prefill

The paper acknowledges an ongoing debate in the LLM serving community: with the introduction of chunked prefill (popularized by SARATHI [15] and integrated into many serving systems), is physical separation of prefill and decoding nodes still necessary? Chunked prefill breaks long prefill operations into smaller chunks that are "inlined" into the continuous batching loop—a decoding batch processes a few decoding tokens from ongoing requests, then processes one chunk of prefill for a new request, then more decoding tokens, and so on. This approach has two clear benefits: (1) all nodes are treated identically, simplifying scheduling, and (2) the prefill chunks add computational intensity to what would otherwise be memory-bound decoding batches, improving Model FLOPs Utilization (MFU).

Mooncake's design team considered this alternative and explicitly rejected it for their workload. The paper provides two specific technical justifications (Section 5):

Reason 1: Long-context prefill requires cross-node parallelism that coupled designs handle poorly. When a user submits a request with 128K input tokens, processing it on a single 8-GPU node—even with tensor parallelism across all 8 GPUs—may still produce an unacceptably high TTFT. The prefill must be parallelized across multiple nodes. In a disaggregated architecture, you can designate some prefill nodes to form pipeline groups specifically for long-context requests, while keeping other prefill nodes for short-context work. In a coupled architecture, every node must be capable of handling both prefill and decoding, which means either (a) all nodes must support the cross-node parallelism needed for long prefill (wasting the parallelism capability on decoding, which doesn't need it), or (b) long-context requests suffer high TTFT.

Reason 2: Chunked prefill's VRAM occupation cost is higher than disaggregated prefill for long contexts. This is a subtle but important point. The "cost" of KVCache in VRAM can be measured as the product of its size and the time it occupies VRAM: if a KVCache of size $S$ occupies VRAM for time $T$ , the occupation cost is $S \cdot T$ . In chunked prefill, where prefill chunks are interleaved with decoding tokens, the total time $T$ from the start of prefill to the completion of decoding is longer than in a disaggregated design where prefill runs as fast as possible on dedicated hardware, then the KVCache is transferred to the decoding node. The longer the KVCache sits in VRAM, the fewer other requests can be batched simultaneously. For long-context requests where $S$ is already large (e.g., 128K tokens × KVCache-per-token, which for LLaMA2-70B's architecture would be 80 layers × 2 (K+V) × 128 (hidden dimension per head) × 8 (KV heads) × 2 bytes (FP16) ≈ 40 MB per 512-token block, so ~10 GB for the full context), even a modest increase in $T$ significantly reduces the number of concurrent requests that can fit in VRAM.

The paper's operational stance is: "a request's prefill is inlined into the decoding batch only when it can be forwarded without chunking and without compromising the TBT SLO" (Section 5). In all other cases—which, given Kimi's long-context workload, is the majority—the stages are physically separated. This is a practical engineering judgment that the paper justifies empirically: Mooncake's TBT compliance is ~100% vs. vLLM's ~57% under real workloads (Section 8.1.3, Figure 13).

Multi-Node Prefill: Chunked Pipeline Parallelism (CPP)

For requests with extremely long input contexts (thousands to hundreds of thousands of tokens), even a dedicated 8-GPU prefill node using tensor parallelism (TP) may not achieve acceptable TTFT. The obvious solution is to use multiple nodes working together on the same request. The paper evaluates two approaches—sequence parallelism and chunked pipeline parallelism—and selects CPP (Section 5.1).

The problem with sequence parallelism (SP) for prefill. Sequence parallelism (as implemented in Ring Attention [18], Striped Attention [19], and DeepSpeed Ulysses [17]) partitions the input sequence across multiple nodes. Each node computes attention locally on its partition, then exchanges partial results with other nodes (via ring or all-to-all communication) to reconstruct the full attention output. This works well for training because it keeps the per-node sequence length manageable and scales to very long contexts. However, for inference prefill, the paper identifies three limitations:

First, SP still requires cross-node communication at least once per transformer layer (for the attention exchange), which competes for network bandwidth with the KVCache transfers that are central to Mooncake's design. If a node is simultaneously receiving KVCache blocks for prefix reuse (via RDMA) and performing all-reduce operations for sequence-parallel attention, the shared network interface becomes a bottleneck.

Second, SP yields worse MFU than single-node TP because the communication overhead, while lower than cross-node TP (which requires two all-reduce operations per layer), is still non-trivial. The paper states this explicitly: "adopting SP still results in a worse MFU compared to using single-node TP only" (Section 5.1). This matters because prefill is computation-bound—any time spent waiting for network transfers is time not spent doing useful matrix multiplications.

Third, dynamically adjusting the SP group size is complex. The paper envisions a deployment where some prefill nodes form an SP group for long-context requests while others operate with TP-only for short contexts. But the proportion of long vs. short requests varies over time, requiring elastic scaling of the SP group. Recent work (LoongServe [14]) proposes solutions, but the paper argues this "adds complexity to our architecture" and "complicates Conductor's design when considering metrics like cache reuse utilization and SLO requirement violations during adjustments" (Section 5.1). The key phrase is "frequent on-the-fly scalability"—under Kimi's rapidly changing workload, constantly reconfiguring parallel groups would be operationally fragile.

Chunked Pipeline Parallelism (CPP) as the alternative. The paper proposes an approach that leverages the autoregressive property of decoder-only transformers: each token's computation depends only on preceding tokens, not on future tokens. This means you can process different chunks of the same request on different nodes in a pipeline, as long as you respect the causal ordering.

The mechanism works as follows. The prefill cluster is organized into pipeline groups of $X$ nodes each (the paper does not specify a fixed $X$ ; it is a deployment-configurable parameter). For a request with input length $L$ tokens, the tokens are partitioned into chunks, each no longer than prefill_chunk_size. The paper specifies that this threshold "is selected to fully utilize the corresponding GPU's computational power and is typically larger than 1000 tokens" (Section 3, Step 2 description). Different chunks of the same request are assigned to different nodes in the pipeline group:

Node 1 processes chunk 1 (tokens 0 to prefill_chunk_size). When it finishes, it passes the output KVCache and hidden states to Node 2.
Node 2 processes chunk 2 (tokens prefill_chunk_size to 2 × prefill_chunk_size), using the KVCache from Node 1 as context. When it finishes, it passes results to Node 3.
This continues until all chunks are processed.

The key property is that the communication between pipeline stages happens only at chunk boundaries—once per chunk, not once per layer as in SP. Because chunks are large (thousands of tokens), this communication is infrequent and can be easily overlapped with computation (while Node 2 is processing chunk 2, Node 1 can start on chunk 1 of the next request). This is directly analogous to pipeline parallelism in training, but adapted for inference where the "micro-batches" are chunks of a single long sequence rather than multiple independent sequences.

Comparison of CPP vs. SP for the paper's use case: The paper claims two main benefits of CPP over SP. First, better MFU and less network contention because cross-node communication is "only at the boundaries of each pipeline stage, which can be easily overlapped with computation" (Section 5.1). Second, natural handling of both short and long contexts—a short request that fits in one chunk simply executes on one node without any pipeline overhead, while a long request automatically uses the pipeline. There is no need to dynamically reconfigure parallelism groups.

The paper acknowledges that this pipeline-based acceleration method has been explored in training systems (TeraPipe [24]) but states that "to our knowledge, this is the first application in the inference stage, as long context inference has only recently emerged" (Section 5.1). This is presented as a contribution, though it is an application of a known technique to a new domain rather than a fundamental algorithmic innovation.

What the paper does NOT specify about CPP: The exact pipeline schedule (GPipe-style flush vs. 1F1B), the value of $X$ (nodes per pipeline group), how pipeline groups are formed and managed by Conductor, and how load balancing works across pipeline groups (if a group is partially idle, can it accept short requests on individual nodes?). These are engineering details that would be needed to fully reproduce the system but are not disclosed.

Layer-Wise Prefill: Overlapping KVCache Transfer with Computation

The disaggregated architecture creates a new problem: the prefill node generates KVCache, but the decoding node needs it. Transferring the KVCache naively—waiting for all prefill computation to complete, then initiating transfer—would add the full transfer latency to the end-to-end request time, potentially violating TTFT SLOs for long contexts where the KVCache is tens of gigabytes. The paper's solution is layer-wise prefill (Section 5.2), which exploits the layer-by-layer structure of transformer computation to overlap KVCache transfer with ongoing prefill computation.

The mechanism, step by step:

Before each layer's attention computation begins, the prefill instance issues an asynchronous load operation to bring that layer's reusable KVCache (from the prefix cache) from CPU DRAM into GPU VRAM. It then waits for this load to complete before starting the layer's attention computation (since the computation needs the cached keys and values).
After each layer's attention computation completes, the prefill instance immediately launches an asynchronous store operation that transfers the newly computed KVCache for that layer from GPU VRAM to CPU DRAM (where Messenger can then transfer it to the decoding node). Critically, it does NOT wait for this store to complete before moving to the next layer.
Launch and wait primitives: The paper describes using CUDA-style "launch and wait" operations. "Before each layer's attention computation begins, the model waits for the asynchronous loading of that layer's KVCache to complete and triggers the next layer's asynchronous KVCache loading. After the attention calculation is complete, asynchronous storage of that layer's KVCache is launched" (Section 5.2).
Final synchronization: Once all layers' computations are finished, the process waits for the completion of all outstanding asynchronous storage operations. At this point, the full KVCache is guaranteed to be in CPU DRAM and available for transfer.

What this achieves: The total time for the prefill stage becomes roughly the maximum of (a) the time to load reusable KVCache from CPU memory and (b) the time to compute the prefill for uncached tokens, rather than the sum of these times. For requests with a high prefix cache hit ratio, loading dominates; for requests with little reusable cache, computation dominates; but in both cases, the KVCache transfer to the decoding node is almost completely hidden.

Figure 7 provides experimental validation: "Layer-wise Prefill" shows a latency of 0.2 seconds for a 128K sequence, while "Serialized" (which stores all KVCache after computation) shows 0.8 seconds—a 4× reduction. The paper notes that this "allows the prefill instance's execution time to be roughly equivalent to either the KVCache loading time or the standard prefilling time, depending on the prefix cache proportion relative to the input length" (Section 5.2).

Strategic implication for scheduling: Because the KVCache is aggressively transferred out of VRAM during computation, the VRAM required on the prefill node is only what's needed to hold the KVCache for the current layer plus the computation for one request—typically much smaller than the full KVCache for the entire context. This means "the scheduling of prefill nodes only considers the KVCache distribution and the available DRAM size" (Section 5.2)—VRAM is no longer a binding constraint on prefill scheduling, as long as it can contain a single request's working set. This is a significant architectural simplification: Conductor doesn't need to track per-prefill-instance VRAM utilization, only per-decoding-instance VRAM (where the full KVCache must reside for the duration of decoding).

Future use of freed prefill VRAM: The paper speculates about using this freed VRAM for "batch-oriented offloading tasks" that don't require immediate responses—specifically citing OpenAI's Batch API [25] which offers 50% lower costs for requests with 24-hour turnaround. Since such requests have relaxed latency requirements, parts of their decoding could potentially be inlined into prefill batches to improve MFU, using the available VRAM. This is presented as future work, not an implemented feature.

The KVCache Storage and Transfer Infrastructure

The KVCache sits at the center of Mooncake's architecture, and the paper describes a concrete storage and transfer design (Section 3, Figure 3, and throughout Section 6).

Paged KVCache blocks: KVCache is stored in fixed-size pages (blocks) in the CPU DRAM of each node, following the PagedAttention approach from vLLM [13]. Each block contains the key and value tensors for a contiguous span of token positions for all layers. The block partitioning enables non-contiguous storage (different blocks of a request can be on different nodes), independent eviction (rarely-used blocks can be swapped out without affecting popular ones), and efficient transfer (blocks are the unit of network transfer).

Hash-based deduplication with prefix chaining: This is the mechanism that enables cache reuse across different requests. As illustrated in Figure 3:

The token sequence is divided into fixed-size blocks (512 tokens per block in the trace; Section 4.1 specifies the hash_ids field corresponds to blocks of this size).
For each block, a hash is computed that incorporates both (a) the content of the current block's tokens and (b) the hash of the previous block: Hash(prev_hash, current_block_tokens). This creates a hash chain where each block's identity depends on its entire prefix.
When two requests share a common prefix, their hash chains will match for the shared portion: Hash(a) = A, Hash(A + b) = B, Hash(B + c) = C, etc. As soon as the requests diverge, the hashes diverge.

This design means that cache lookup is simply a hash table lookup: given the hash of the prefix, the system can determine whether that prefix's KVCache is stored anywhere in the distributed pool. The paper does not detail how the global hash table is maintained (whether it's centralized at Conductor or distributed via consistent hashing), but the presence of globally unique hash IDs in the open-sourced trace (Section 4.1: "identical hash IDs indicate that a block of tokens, along with preceding tokens, is the same, thus allowing reuse") suggests a centralized or well-partitioned mapping.

The Messenger service: Each node runs a Messenger process that is an independent OS process within the inference instance. It receives signals from Conductor (or from other nodes' Messengers) to transfer specific KVCache blocks. Transfers use GPUDirect RDMA, which allows data to move directly between the GPU memory of one node and the CPU memory (or GPU memory) of another node without going through the host CPU's memory copy—this is critical for achieving the high bandwidth needed because the KVCache for a long context can be tens of gigabytes (e.g., for a 128K-token context with LLaMA2-70B, roughly 80 layers × 2 tensors × 128K positions × 128 dimensions × 8 heads × 2 bytes ≈ 40 GB). The paper doesn't specify the exact RDMA API used (likely NVIDIA's GPUDirect RDMA via InfiniBand or RoCE given the 800 Gbps interconnect mentioned in Section 8.1).

Cache eviction policies: The paper mentions that the CPU-side KVCache pool can use "LRU (Least Recently Used), LFU (Least Frequently Used), or algorithms based on request characteristics" (Section 3). Table 1 in Section 4.2 evaluates these policies on the real trace: LRU achieves the best cache hit ratio (0.51 with infinite capacity) because "temporal proximity in request utilization" makes recently-used blocks likely to be used again. This is an important practical result because LRU is simple to implement and doesn't require tracking per-block access frequencies. The "LengthAwareCache" variant (which prioritizes blocks occurring later in requests, similar to LFU) was tested but did not outperform LRU in this workload.

Cache capacity analysis: Table 1 shows that increasing cache capacity from 1,000 blocks to 50,000 blocks increases the hit ratio from 0.30 to 0.50, but further increases show "minimal improvement." However, the paper carefully notes that "this should not be interpreted as an indication that larger caches are unnecessary, as the sample trace represents only a subset of real-world workloads. The required capacity should scale proportionally in actual scenarios." This is a responsible caveat: the trace is a 1-hour sample, and the working set of the full service is larger.

Hot-spot replication: Figure 6 shows a highly skewed block access distribution: "over 50% of cache blocks remaining unused while certain blocks are accessed tens of thousands of times." These hot blocks (e.g., system prompts used by nearly every request, or popular long documents being discussed by many users) create a problem: if they reside on only one node, that node becomes a transfer bottleneck—every request that needs the block must fetch it from that node's Messenger, saturating its network bandwidth and increasing transfer latency. The solution, discussed in Section 6.2, is to proactively replicate hot blocks to multiple nodes so that transfer load is distributed. The paper implements this via the heuristic-based hot-spot migration scheme (described below in the cache load balancing section).

KVCache-Centric Scheduling Algorithm (Algorithm 1)

This is the intellectual core of the paper. Conductor's scheduling algorithm (Algorithm 1 in Section 6.1) determines, for each incoming request, which prefill instance and which decoding instance should handle it. Unlike prior work that routes based solely on instance load (number of queued requests), Mooncake's scheduler also considers prefix cache hit length, KVCache transfer time, and network congestion.

The algorithm's inputs and decision process:

For each incoming request $R$ with prompt tokens, Conductor:

Computes block keys by hashing the request's tokens using the prefix-chaining hash function and the block size $B$ (512 tokens). This produces a list of hash values, one per block, that serve as lookup keys into the distributed KVCache pool.
Identifies the best prefix match across all prefill instances. For each instance $p_i$ , Conductor compares the request's block keys against $p_i$ 's locally cached block keys, finding the longest contiguous prefix match prefix_len_i (i.e., the largest $k$ such that the first $k$ block hashes match). Conductor also identifies the instance best_matched_instance that has the overall longest prefix match across the entire cluster, with match length best_prefix_len.
Evaluates each candidate prefill instance against the following cost model:
- $T_{\text{queue}}$ : Estimated queuing time at the instance, computed by aggregating the predicted prefill times of all requests already queued at that instance.
- $T_{\text{prefill}}$ : Estimated prefill execution time, which is a function of the request's total input length and the locally available prefix match length (prefix_len_i). More cached prefix → fewer tokens to recompute → shorter $T_{\text{prefill}}$ . The paper uses "a predictive model derived from offline test data" (Section 6.1) for this estimation.
- $T_{\text{transfer}}$ : If the instance's local prefix match length is shorter than best_prefix_len, the missing blocks must be transferred from best_matched_instance. The transfer time is estimated based on the size of the missing blocks and the current network status, including whether the sending node is congested.
Makes a routing decision that balances cache reuse against load:
- If best_prefix_len divided by prefix_len_i is less than a threshold kvcache_balancing_threshold: The local prefix match is "close enough" to the best available match, so Conductor simply routes to the instance with the smallest $T_{\text{queue}} + T_{\text{prefill}}$ —no KVCache transfer needed. This is the cache-aware prefill scheduling branch.
- Otherwise: The local prefix match is substantially shorter than the best available match. Conductor must decide whether to route to (a) the instance with the long prefix match (which might have high queue time) or (b) another instance and transfer the missing KVCache. It evaluates $T_{\text{transfer}} + T_{\text{queue}} + T_{\text{prefill}}$ for each candidate, where $T_{\text{prefill}}$ uses best_prefix_len (assuming the missing blocks will be successfully transferred). Conductor selects the instance that minimizes this total time. This is the cache-aware and -balancing prefill scheduling branch.
Selects a decoding instance using a simpler load-balancing approach: pick the decoding instance with the lightest predicted load (measured as TBT ratio relative to the SLO). The paper calls this "Load-balancing decoding scheduling" in Algorithm 1.
Checks SLOs: If the predicted TTFT exceeds TTFT_SLO or the predicted TBT exceeds TBT_SLO, the request is rejected (HTTP 429). Otherwise, the request is accepted.
Triggers KVCache transfer if needed: If the selected prefill instance needs blocks from best_matched_instance, Conductor initiates the transfer via Messenger. This transfer happens in parallel with the prefill computation—the remote blocks are streamed to the prefill instance's CPU DRAM, then loaded into GPU VRAM layer-by-layer during prefill (via the layer-wise mechanism described above).

Key design decisions in the algorithm:

TTFT as the unified cost metric: Rather than optimizing cache hit rate or load balance separately, the algorithm folds everything into a single metric: predicted TTFT. This elegantly resolves the tension between "route to the instance with the most cache" and "route to the least loaded instance"—the optimal choice is whichever minimizes the sum of all time components, and different components dominate in different scenarios.
The kvcache_balancing_threshold parameter: This threshold controls the tradeoff between cache utilization and load balancing. A threshold of 1.0 means "always route to the instance with the best prefix match, even if queue times are high" (cache-maximizing). A threshold near 0 means "never transfer KVCache; always route based on load" (load-balancing only). The actual value is "currently adjusted manually, but can be adaptively adjusted by an algorithm in the future" (Section 6.2 footnote). This is a practical admission: the optimal threshold likely depends on workload characteristics, and hand-tuning is a reasonable starting point.
Precomputed performance model: The prefill execution time predictor is based on offline profiling data that maps (input_length, prefix_match_length) → execution time. Because transformer computation is highly regular, "the error bound of this prediction is small as long as enough offline data is available" (Section 6.1). The queuing time is computed by summing the predicted prefill times of all queued requests—no queuing theory approximation, just straightforward aggregation—and since TTFTs for different instances can be "computed in parallel, rendering the processing time negligible compared to the inference time" (Section 6.1), the scheduler runs as a fast parallel loop.
Transfer time estimation difficulty: The paper acknowledges that transfer time is the hardest component to estimate because "it is determined not only by the size of the transferred data but also by the current network status, especially whether the sending node is under congestion" (Section 6.1). This is the motivation for cache replication—by spreading hot blocks across multiple nodes, the network load from transfers is distributed, making the transfer time lower and more predictable.

What the algorithm does NOT handle: The algorithm as described does not consider the availability of multiple replica locations for the same cache block. It identifies a single best_matched_instance and transfers from it. If multiple replicas exist, a refined version could choose the replica with the lowest transfer time (considering network distance and congestion). This refinement is implied by the hot-spot migration discussion but not explicitly integrated into Algorithm 1.

Cache Load Balancing: Heuristic-Based Hot-Spot Migration

The scheduling algorithm in Section 6.1 assumes KVCache blocks already have a distribution across nodes, but doesn't specify how that distribution is actively managed. Section 6.2 describes the mechanism for dynamically redistributing cache blocks to improve overall system performance.

The problem: Some cache blocks are accessed thousands of times, while most are never accessed (Figure 6). If a hot block resides on only one node, that node becomes a transfer bottleneck—every request that needs the block must fetch it from that node, saturating its network bandwidth and increasing TTFT for all those requests. Manual replication is impractical at scale and can't adapt to shifting access patterns.

The straw-man alternative that was rejected: The paper explicitly considers and rejects a prediction-based approach where Conductor would "collect the global usages of each block, use a prediction model to forecast their future usages, and make scheduling decisions accordingly" (Section 6.2). The reason for rejection is practical: "workloads are highly dynamic and change significantly over time. Especially for a MaaS provider experiencing rapid growth in its user base, it is impossible to accurately predict future usage." This is an important philosophical stance: rather than trying to predict the future perfectly (which is fragile), build a system that reactively and heuristically adapts to observed access patterns.

The reactive hot-spot migration scheme: The mechanism is embedded in Conductor's request routing logic:

Trigger condition: When Conductor routes a request to a prefill instance $p$ that does NOT have the longest prefix match (because $p$ had lower queue time, making the total TTFT lower despite needing a transfer), it also instructs $p$ to proactively retrieve the missing cache blocks from best_matched_instance and store them locally. The rationale: if $p$ was chosen over the best-cache-match instance despite needing a transfer, it's likely that $p$ will be chosen again for similar requests in the future. By fetching and caching the blocks now, future requests will have a longer local prefix match at $p$ , eliminating the transfer time.
Selective recomputation vs. transfer: Even when a transfer is possible, Conductor may choose to simply recompute the missing prefix tokens from scratch rather than transfer the KVCache. The condition is: "if the best remote prefix match length is no larger than the current local reusable prefix multiplied by a threshold" (Section 6.2). In other words, if the remote cache would only save a small amount of computation compared to what's already cached locally, the transfer overhead isn't worth it—just recompute. This threshold is the same kvcache_balancing_threshold referenced in Algorithm 1.
Automatic replication effect: Because popular blocks are requested frequently, they are more likely to trigger the "route to a non-optimal-cache instance" scenario, and thus more likely to be proactively replicated. Conversely, rarely-accessed blocks remain on their original nodes. Over time, the cache distribution naturally converges toward replicating hot blocks and leaving cold blocks non-replicated—all without explicit usage tracking, prediction models, or global optimization.

Experimental validation: Figure 8 shows the effect of cache load balancing on TTFT in a real workload replay. The four conditions are: random scheduling (pick any prefill instance), load-balancing scheduling (pick the least-loaded instance), cache-aware scheduling (Algorithm 1 without the kvcache_balancing_threshold branch—always route based on TTFT but without proactive replication), and KVCache-centric scheduling (the full algorithm with cache load balancing). Results:

Random scheduling: average TTFT ~92 seconds
Load-balancing scheduling: ~60 seconds
Cache-aware scheduling: ~14 seconds
KVCache-centric scheduling: ~6 seconds

The 6.26-second average TTFT from the full algorithm is a 15× improvement over random and 10× over simple load balancing. This validates that integrating cache awareness AND cache replication into scheduling is substantially better than either component alone.

Overload-Oriented Scheduling: Early Rejection and Load Prediction

This section (Section 7) addresses the problem that prior LLM serving research has largely ignored: what do you do when you can't serve all incoming requests, and how do you minimize wasted computation on requests that will ultimately be rejected?

Defining "load" in a disaggregated system. In a coupled system (prefill and decoding on the same GPU), load is hard to define precisely because the stages interfere with each other, so "the load is often measured simply by the ratio of the number of requests being processed to the system's maximum capacity" (Section 7.1). In Mooncake's disaggregated architecture, the stages are independent, so load can be defined more precisely in terms of SLO satisfaction:

Prefill load: Whether the predicted maximum TTFT (queue time + execution time) on a given prefill instance exceeds the TTFT SLO ( $l_{\text{ttft}}$ ).
Decoding load: Whether the predicted TBT on a given decoding instance exceeds the TBT SLO ( $l_{\text{tbt}}$ ).

A request is accepted only if there exists a prefill instance with TTFT ≤ $l_{\text{ttft}}$ AND a decoding instance with TBT ≤ $l_{\text{tbt}}$ .

The waste problem: time lag between prefill and decoding scheduling. Here's the critical issue. When Conductor accepts a request, it schedules the prefill on instance $p$ and the decoding on instance $d$ . Between the time the prefill is scheduled and the time the prefill completes (which can be seconds for long contexts), the load on $d$ may have changed—new requests may have arrived, in-flight requests may have taken longer than expected, etc. If, when the request finally arrives at $d$ , the local decoding scheduler determines that the TBT SLO would be violated, it rejects the request. But the prefill computation has already been done—GPUs were occupied, energy was consumed, but the request won't be completed and billed. The paper formalizes this: "all previously consumed/generated tokens are not counted, and the corresponding resources are wasted" (Section 2).

Early Rejection: the straightforward fix and why it causes oscillation. The natural solution is to check decoding load BEFORE starting prefill—not just prefill load. Conductor evaluates whether the decoding instance $d$ is predicted to have sufficient capacity when the request will arrive, and only accepts the request if BOTH the prefill and decoding loads are within SLO. This is called Early Rejection (Section 7.2).

However, Early Rejection introduces a new problem: load oscillation between the prefill and decoding pools. Figure 9 shows this phenomenon in a real deployment over 20 minutes—prefill and decoding loads oscillate in anti-phase. The paper explains the mechanism with a four-stage model (Section 7.3, Figure 10a):

Stage 1: Both prefill and decoding loads are low → Conductor accepts many requests → prefill load rises to capacity.
Stage 2: Requests that completed prefill now enter decoding → decoding load rises to capacity. Conductor sees high decoding load → starts rejecting incoming requests → prefill load drops (no new requests entering).
Stage 3: No new requests enter decoding → decoding load drops as existing requests complete. Conductor sees low decoding load → starts accepting requests again → prefill load rises.
Stage 4: Same as Stage 2—the cycle repeats.

This oscillation is a classic control system problem: Conductor is making decisions based on CURRENT decoding load, but the effect of those decisions (new requests arriving at decoding) is delayed by the prefill duration. The system overcorrects: it accepts too many requests when load is low (not accounting for the wave of requests about to arrive), then rejects too many when load is high (not accounting for the wave about to leave). The result is poor resource utilization—prefill nodes are idle during the rejection phases, and decoding nodes are overloaded during the acceptance phases.

Early Rejection Based on Prediction: the solution. The fix is to predict decoding load at the time the request will arrive at the decoding stage, not the current load. As illustrated in Figure 10b, this means:

Predict when the prefill will complete for this request (based on input length, cache hit length, and current queue depth).
Predict what the decoding load will be at that future time, accounting for both (a) requests that will complete and leave the decoding stage by then, and (b) other requests currently in prefill that will arrive at the decoding stage by then.
Accept or reject based on this predicted future load.

The paper describes two prediction approaches (Section 7.4):

Request-level prediction (ideal but hard): Predict the exact output length of each in-flight request, which determines when it will leave the decoding stage. This would give the most accurate load forecast. However, "predicting each request's output length is challenging due to high costs or low accuracy, especially under overload conditions where resources are scarce and accurate predictions are necessary" (Section 7.4). This is recognized as a hard problem and left as future work.

System-level prediction (practical approximation, currently used): Rather than predicting individual request completion times, make a simplifying assumption: each request spends a uniform time $t_d$ in the decoding stage. The prediction procedure is:

For a given future time $t$ (when the new request will arrive at decoding), add to the decoding load all requests currently in prefill that will complete by $t$ (they will enter decoding between now and $t$ ).
Remove from the decoding load all requests whose elapsed decoding time exceeds $t_d$ (they will complete and exit by $t$ ).
Compute the average TBT ratio (predicted TBT divided by $l_{\text{tbt}}$ ) across all decoding instances as the predicted load.

This is a coarse approximation—real output lengths vary widely (Figure 5 shows output lengths from 1 to 2000+ tokens, with the average being 182 tokens)—but it captures the essential dynamics: the load is increasing due to incoming requests and decreasing due to completions, and the net effect can be forecast without per-request precision. The paper emphasizes this is ongoing work and "requires less precision, making it more appropriate for overload scenarios" (Section 7.4), where the goal is to avoid gross over/under-acceptance rather than precise load balancing.

Experimental validation: Table 3 quantifies the benefit. In an overload experiment with 23,000 real requests replayed at 2× speed:

Baseline (reject based on load at both stages independently, without early rejection): 4,183 requests rejected
Early Rejection (check decoding load before prefill, but based on current load): 3,771 requests rejected (9.8% reduction)
Early Rejection based on Prediction (check predicted future decoding load before prefill): 3,589 requests rejected (14.2% reduction from baseline, 4.8% reduction from naive early rejection)

The interpretation: Prediction-based early rejection avoids accepting requests that would be rejected post-prefill, reducing wasted prefill computation by ~14% compared to no early rejection. The additional improvement from prediction over naive early rejection (4.8%) corresponds to cases where naive early rejection made wrong decisions due to load oscillation—it rejected requests that actually could have been served, or accepted requests that couldn't be served because future load was misestimated. The prediction dampens the oscillation, enabling more accurate accept/reject decisions.

Summary of Design Choices and Their Justifications

Disaggregated prefill/decoding over chunked prefill inlining: Long-context prefill needs cross-node parallelism that coupled designs handle poorly, and disaggregation reduces KVCache VRAM occupation cost (the KVCache spends less time in VRAM because prefill completes faster on dedicated hardware). The paper does NOT reject chunked prefill entirely—it inlines short prefill into decoding batches—but separates stages for the long-context majority of its workload.
Chunked Pipeline Parallelism over Sequence Parallelism for multi-node prefill: CPP requires cross-node communication only at chunk boundaries (infrequent, overlappable with computation) versus SP's per-layer communication. CPP naturally handles variable-length requests without dynamic group reconfiguration. The tradeoff is that CPP may have higher latency than perfectly load-balanced SP for some request sizes, but the operational simplicity outweighs this for the paper's rapidly-growing deployment.
Layer-wise KVCache transfer overlap: Exploits the layer-by-layer structure of transformers to hide KVCache transfer latency behind computation, eliminating VRAM as a prefill scheduling constraint. This is a key enabler for the disaggregated architecture to work efficiently.
Hash-chained KVCache indexing: The Merkle-tree-like hash structure (Hash(prev_hash, current_tokens)) enables O(1) prefix match detection: if hash $i$ matches, all previous hashes must also match. This is efficient for the per-request scheduling loop where Conductor must check prefix lengths against all prefill instances.
TTFT-minimizing scheduling over cache-hit-maximizing scheduling: Folding cache reuse, queue time, and transfer time into a single TTFT metric avoids the pathological case where requests are routed to cache-rich but overloaded nodes. The kvcache_balancing_threshold provides a tunable knob between the extremes.
Reactive hot-spot replication over predictive replication: When a request is routed away from the best-cache-match instance (due to load), it proactively copies the cache, creating automatic replication proportional to access frequency. This avoids the fragility of predicting future access patterns in a rapidly-changing workload.
System-level load prediction over request-level for overload handling: Predicts aggregate decoding completion rate using a uniform-time assumption rather than per-request output length prediction. Pragmatic choice: request-level prediction is too hard/inaccurate to be reliable, but system-level prediction is sufficient to dampen the load oscillation that naive early rejection causes.
Two-stage SLO checking (Conductor + local decoder): Conductor provides a fast, approximate SLO check for scheduling decisions; the local decoder provides the ground-truth check when the request actually arrives. The prediction-based early rejection makes these two checks more consistent, reducing the rate at which the local decoder overrules Conductor.

4. Key Insights and Innovations

Innovation 1: KVCache as a First-Class Scheduling Object, Not a Memory Management Detail

The paper's most conceptually distinctive contribution is elevating KVCache from a memory management optimization to the central organizing principle of the entire serving architecture. This is not merely a "better cache" — it's a fundamental reframing of what the scheduler should be optimizing over.

Before Mooncake, the dominant paradigm in LLM serving treated KVCache as a local optimization: vLLM [13] introduced PagedAttention to manage KVCache more efficiently within a GPU's VRAM, preventing fragmentation and enabling larger batches. Prefix caching systems like Prompt Cache [33] and SGLang [34] showed that KVCache could be reused across requests with common prefixes, but treated caching as a throughput optimization — cache hits reduce computation, which is good. Conductor-level scheduling (which instance should handle which request) was based primarily on load — the number of queued requests or available GPU capacity. Cache location was at most a secondary consideration.

Mooncake's key conceptual move is to invert this relationship. In their architecture, the scheduler's primary decision variable is KVCache — where it is, how much of it can be reused, how expensive it is to transfer, and whether it should be replicated. Instance load and queuing delay are folded into the cost model as constraints on the KVCache optimization, not as the primary objective. Algorithm 1 makes this inversion explicit: the scheduler iterates over prefill instances, computing a TTFT estimate that accounts for queue time, but the dominant factor in that estimate is the prefix cache match length and the associated transfer cost. The kvcache_balancing_threshold parameter, which controls the boundary between "the local cache match is good enough" and "transfer remote cache despite the cost," is the tunable knob that encodes the entire scheduling philosophy.

What makes this distinctive is that it's not just a new optimization — it's a diagnostic reframing of the scheduling problem. Prior work asked: "Given these GPU instances and their loads, where should I route this request to maximize throughput?" Mooncake asks: "Given this distributed KVCache and this request's prefix, how do I minimize the total time (queue + computation + transfer) to produce the first token?" The difference is subtle but profound. It means that KVCache distribution drives instance selection, rather than instance selection driving local cache hits as a side effect. It means that cache replication and migration are not background maintenance operations but active, real-time scheduling decisions. And it means that the scheduler's cost model must account for network congestion, transfer bandwidth, and the computational cost of recomputing uncached tokens — factors that are irrelevant if you're only tracking per-instance queue lengths.

The paper's experimental validation of this reframing is compelling even though it doesn't isolate the KVCache-centric scheduling from other Mooncake innovations. Figure 8 shows that cache-aware scheduling alone (without cache load balancing) reduces average TTFT from ~60 seconds (load-balancing only) to ~14 seconds — a 4.3× improvement. Adding cache load balancing (the full KVCache-centric algorithm) further reduces it to ~6 seconds — a 10× improvement over load-balancing alone. This 10× gap represents the value of treating KVCache as the primary scheduling signal, and it's measured on real workload traces, not synthetic benchmarks. Importantly, this result is obtained on a cluster with only 8 prefill and 8 decoding instances — the relative advantage of KVCache-centric scheduling likely grows with cluster size, as the combinatorial space of (request, prefill instance) pairs expands and naive load-balancing becomes increasingly suboptimal.

Significance beyond performance: This reframing opens a new axis for LLM serving research. Prior work optimized along dimensions like parallelism strategy (TP vs. PP vs. SP), batch scheduling policy (continuous batching, iteration-level scheduling), and memory management (paged attention, swapping, offloading). Mooncake adds KVCache topology as a first-class optimization dimension — the spatial distribution of cached computation across nodes, the replication factor of hot blocks, and the transfer topology connecting them. This is analogous to how CDNs (Content Delivery Networks) reorganized internet infrastructure around content location rather than server capacity, and it suggests a rich design space that has barely been explored (e.g., anticipatory cache placement based on predicted request patterns, cooperative caching protocols between prefill instances, cache-aware anycast routing).

This is a fundamental shift in framing, not an incremental improvement to an existing approach. The paper doesn't simply "add caching to vLLM's scheduler" — it redesigns the scheduler's objective function and decision space around the cache, which is a qualitatively different approach.

Innovation 2: Disaggregated KVCache Pool as a "Third Tier" Resource — and Why It Enables the Architecture

While prefill/decoding disaggregation has been proposed by several concurrent works (Splitwise [7], DistServe [8], TetriInfer [9]), Mooncake's distinctive contribution is adding a third disaggregated component — the KVCache pool itself — and showing that this three-way disaggregation (prefill clusters, decoding clusters, KVCache pool) solves coordination problems that two-way disaggregation cannot.

In two-way disaggregated architectures, the KVCache is either (a) stored in the prefill node's memory and transferred to the decoding node on-demand, or (b) stored in the decoding node's memory where it was generated. Both approaches create a coupling between the KVCache and a specific physical node. If a request could benefit from cache generated by a different prefill node, or if the decoding node with the relevant cache is overloaded, the scheduler has no good options — it must either accept the suboptimal cache hit or incur a long-tail transfer that may violate SLOs.

Mooncake's three-way disaggregation decouples KVCache from both prefill and decoding nodes by pooling the CPU DRAM and SSD of all nodes into a globally addressable, Messenger-accessible KVCache store. This enables three capabilities that two-way systems lack:

First, KVCache location independence for scheduling. Because any prefill node can retrieve any cached block from the pool (via Messenger RDMA transfers), Conductor's scheduling decision is not constrained by which node originally generated the cache. It can route a request to the least-loaded prefill instance and simultaneously instruct Messenger to stream the relevant cache blocks from wherever they reside. The layer-wise prefill mechanism (Section 5.2) makes this practical by hiding the transfer latency behind computation. Without the disaggregated pool, cache retrieval would require the prefill and decoding nodes to coordinate directly, adding complexity and coupling.

Second, KVCache replication as a load-balancing mechanism for the cache itself. The paper's trace analysis (Figure 6) reveals extreme skew in cache block popularity — some blocks are accessed tens of thousands of times while most are never accessed. In a two-way system, the node holding a hot block becomes a transfer bottleneck, and the only remedy is manual replication (which is operationally burdensome and doesn't adapt to shifting access patterns). In Mooncake's three-way system, the hot-spot migration heuristic (Section 6.2) automatically replicates hot blocks through the normal scheduling process: when a request is routed to a non-optimal-cache instance and the remote cache is transferred, a local copy is retained. Over time, hot blocks replicate themselves proportional to their access frequency, completely automatically.

Third, DRAM/SSD capacity leveraging for near-GPU cache scale. The paper makes the practical observation that GPU cluster nodes have massive underutilized CPU DRAM and SSD capacity — resources that are "free" in the sense that they're already provisioned but not fully utilized by the GPU workloads. By pooling these resources, Mooncake creates a KVCache whose capacity is limited by total cluster DRAM+SSD rather than by per-GPU VRAM. Table 1 shows that increasing cache capacity from 1,000 to 50,000 blocks increases the hit rate from 30% to 50% — a substantial practical improvement. Without the disaggregated pool, achieving this cache capacity would require either (a) purchasing dedicated cache servers (adding cost and network hops) or (b) reducing the number of concurrent requests per GPU (reducing throughput) to free VRAM for cache.

Relationship to concurrent work: AttentionStore [35] independently proposed a similar hierarchical KVCache system, which the paper acknowledges. Mooncake's differentiation is in the integration of the cache pool with global scheduling — AttentionStore is primarily a cache architecture, while Mooncake's cache architecture is designed to be the substrate for Conductor's KVCache-centric scheduling decisions. The cache pool's existence enables the scheduler to optimize over cache location; the scheduler's decisions in turn drive cache replication and eviction. This tight integration is what makes the disaggregated pool more than a standalone cache service — it's a scheduling primitive.

Significance beyond performance: The three-way disaggregation reframes the resource allocation problem in LLM serving. Prior to this, the optimization was essentially: "allocate GPUs between prefill and decoding, and manage VRAM." Mooncake adds: "allocate DRAM and SSD for KVCache, allocate network bandwidth for cache transfers, and balance all three against SLOs." This is a richer optimization space that the paper only begins to explore — future work on optimal cache capacity planning, anticipatory cache placement, and network-aware cache replica selection would all build on this disaggregated pool concept.

This is an incremental advance over two-way disaggregation — the intellectual move of adding a third tier is straightforward in hindsight — but the integration of the cache pool with global scheduling and the demonstration that it enables capabilities (location-independent scheduling, automatic hot-spot replication, capacity scaling) that two-way systems cannot achieve makes it a practically significant contribution.

Innovation 3: Diagnosing and Solving the Load Oscillation Problem — Overload Scheduling as a Distinct Regime

The paper's third distinctive contribution is identifying and characterizing the load oscillation problem as an inherent pathology of overloaded disaggregated systems, and demonstrating that prediction-based scheduling is necessary (not merely beneficial) to achieve stable operation. This is a contribution to the understanding of disaggregated serving, not just to its implementation — it diagnoses a failure mode that any disaggregated architecture will encounter under overload, and provides a principled (if preliminary) solution.

Prior work on LLM serving overwhelmingly assumes capacity-sufficiency: all requests that arrive will eventually be served, and the optimization problem is how to serve them efficiently. Splitwise [7], DistServe [8], and TetriInfer [9] all operate in this regime — they optimize throughput and latency, but do not address the question of which requests to accept or reject. This is not a criticism of those works; it's a reflection of the academic research environment where experiments are run on fixed-size clusters with controlled request rates below saturation.

Mooncake's key diagnostic move is to recognize that overload is not just "more of the same" — it's a qualitatively different scheduling regime with its own failure modes. The paper's analysis of the load oscillation problem (Section 7.3, Figure 9, Figure 10a) is the clearest evidence for this claim. The four-stage model (accept → overload decoding → reject → underload decoding → accept) is a classic control system instability: the delayed feedback between the accept/reject decision and its effect on decoding load causes the system to oscillate between over-acceptance and over-rejection, resulting in poor resource utilization. The paper's insight is that this oscillation is not a bug in their particular implementation — it's a structural property of any disaggregated system that makes accept/reject decisions based on current load, because the time lag between prefill scheduling and decoding execution creates a delayed feedback loop.

Two aspects of this analysis are particularly insightful:

First, the paper identifies the problem at the right level of abstraction. The load oscillation is not caused by a specific scheduling heuristic or implementation detail — it arises from the fundamental structure of the disaggregated architecture: prefill and decoding happen on different physical nodes at different times, and the decision to accept a request for prefill is temporally decoupled from its arrival at the decoding stage. Any system with this structure will experience the same oscillation if it uses current-state feedback for admission control. This means the solution must also be structural — prediction-based admission control — not just parameter tuning.

Second, the paper demonstrates that system-level prediction is sufficient. The request-level prediction problem (predicting the output length of each individual request) has been studied by several works and is known to be hard — output length depends on the model's generation behavior, which is stochastic and context-dependent. Mooncake's pragmatic insight is that for the purpose of overload admission control, you don't need per-request precision; you need aggregate load forecasting. Assuming each request spends a uniform time td in the decoding stage (the system-level prediction approach, Section 7.4) is clearly a coarse approximation — real output lengths vary by orders of magnitude (Figure 5 shows output lengths from 1 to 2000+ tokens). But it's sufficient to capture the essential dynamics: the decoding load increases as prefill-completing requests arrive, decreases as existing requests complete, and the net load at a future time t can be estimated by projecting these aggregate flows.

The experimental results (Table 3) validate that even this crude prediction reduces rejections by 4.8% compared to naive early rejection (3,589 vs. 3,771). The magnitude is modest in absolute terms because the experiment uses a relatively small cluster and a 2× replay speed — the oscillation amplitude and the benefit of prediction would likely be larger in production deployments with more nodes and higher overload ratios. But the direction of the improvement is what matters: prediction-based admission control is better than reactive admission control, confirming the diagnostic claim.

Significance beyond performance: This contribution establishes overload-oriented scheduling as a distinct research sub-problem within LLM serving. Prior work implicitly assumed that throughput optimization under capacity-sufficiency was the general case, and overload was an edge case handled by simple load shedding. Mooncake argues — from production experience — that overload is the normal operating condition for rapidly-growing MaaS providers, and that it generates unique scheduling challenges (the time-lag coupling between prefill and decoding, the tension between early rejection and resource waste, the oscillation instability) that are not addressed by throughput-oriented algorithms. This reframing opens several research directions: more sophisticated aggregation-level load prediction (using queuing theory or learned models rather than uniform-time heuristics), dynamic adaptation of the prediction horizon based on workload variability, and joint optimization of the accept/reject threshold and the prefill/decoding instance ratio.

This is an incremental contribution in terms of the prediction mechanism (it's a simple moving-average-style forecast), but a fundamental contribution in terms of problem identification and characterization. The value is not in saying "prediction helps" — that's obvious — but in (a) diagnosing why disaggregated systems oscillate under overload, (b) showing that the oscillation is structural and will occur in any disaggregated architecture, and (c) demonstrating that even coarse system-level prediction is sufficient to dampen it. This is the kind of production-derived insight that academic systems research often misses.

Innovation 4: A Production-Grade Architecture That Reconciles Academic Optimizations with Operational Realities

While this may seem like a meta-contribution rather than a technical insight, the paper's most significant long-term impact may be its demonstration that disaggregation with KVCache-centric scheduling can be implemented at production scale with measurable business impact, and its honest documentation of where academic assumptions break down in practice. This is not a novel algorithm or a theoretical advance — it's a validation that a particular architectural philosophy works under real-world constraints, which has direct implications for how the research community prioritizes problems.

Several specific findings in the paper challenge common assumptions in the LLM serving literature:

1. "Cache reuse ratios are much lower in practice than in benchmarks." Section 9 explicitly states: "The real reusability in our online traces is much smaller than the results reproduced by open-source benchmarks. Theoretically, up to only 50% of the KVCache can be reused in our current workloads, even if we assume both the capacity of storage and the TTFT SLO are infinite." This tempers the enthusiasm from papers that report 80-90% cache hit rates on curated benchmarks — in real multi-user services with diverse, independently-arriving requests, perfect cache sharing is unrealistic. This finding doesn't invalidate prefix caching as an optimization, but it correctly bounds its impact: caching alone cannot solve the throughput problem; it must be combined with other mechanisms (disaggregation, effective scheduling, early rejection).

2. "Request-level output length prediction is too hard to be reliable." Multiple concurrent works (e.g., TetriInfer [9]) propose using predicted output lengths to improve scheduling. Mooncake's production experience is that this prediction is "challenging due to high costs or low accuracy, especially under overload conditions where resources are scarce and accurate predictions are necessary" (Section 7.4). Rather than pursuing a research-grade solution, they adopt a pragmatic system-level approximation and leave per-request prediction as future work. This honest assessment of what's deployment-ready vs. what's a research problem is valuable for practitioners allocating engineering effort.

3. "LRU beats more sophisticated cache policies." Table 1 shows that LRU, LFU, and a custom LengthAwareCache policy achieve essentially identical cache hit rates at all capacity levels, with LRU slightly outperforming at moderate capacities (0.40 vs. 0.35 at 10,000 blocks). This is a negative result with practical implications: don't waste engineering effort on complex cache replacement policies when LRU works fine for this access pattern. The temporal locality in request streams (users in the same session referencing the same documents) makes LRU's simplicity an advantage rather than a limitation.

4. "Chunked prefill is not a replacement for disaggregation." The paper's careful analysis in Section 5 explains why chunked prefill [15] — which many systems treat as a simpler alternative to physical disaggregation — is insufficient for long-context workloads: it increases VRAM occupation cost, it doesn't solve the cross-node parallelism problem for long prefill, and it still couples prefill and decoding scheduling. This is not a categorical rejection of chunked prefill (Mooncake uses it for short requests), but a boundary condition that the research literature hadn't clearly articulated.

5. "Verifier/predictor over-optimization manifests in scheduling too." The load fluctuation problem (Section 7.3) is essentially an over-optimization phenomenon: the scheduler aggressively optimizes for throughput by accepting as many requests as current load allows, but the delayed feedback causes it to overshoot. This is structurally analogous to the verifier over-optimization problem in the companion paper's summary (where aggressive PRM-guided search finds solutions that exploit the verifier rather than being correct), but operating in the scheduling domain rather than the model output domain.

Significance beyond performance: This contribution is architectural validation, not algorithmic novelty. It says: "Here is a design philosophy (disaggregation + KVCache-centric scheduling + overload-aware admission control) that works at the scale of a leading commercial LLM service, and here are the specific places where academic assumptions don't hold." For the systems research community, this provides a reality check on which problems are worth solving. For practitioners, it provides a reference architecture and a set of design decisions with documented tradeoffs. The open-sourced request trace (Section 4) makes the validation partially reproducible — researchers can test their own scheduling algorithms against real workload patterns, even if they can't reproduce the full production deployment.

This is fundamental in its implications for research prioritization (it redirects attention from throughput-only optimization to overload-aware scheduling, from sophisticated cache policies to integrated cache-scheduling architectures, and from per-request prediction to system-level forecasting), but incremental in its technical novelty (each component has precedents in the literature). The contribution is in the synthesis, the production validation, and the honest characterization of what works and what doesn't under real constraints — which, for a systems paper, is exactly the right kind of contribution.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses four distinct datasets for end-to-end evaluation (Table 2): (1) ArXiv Summarization [26], a public dataset for long-document summarization with average input length 8,088 tokens and average output length 229 tokens, requests generated via Poisson process with ~0% cache ratio; (2) L-Eval [27], a long-context evaluation benchmark with average input length 19,019 tokens and average output length 72 tokens, Poisson arrival, >80% cache ratio; (3) Simulated Data, synthetically generated requests with predefined lengths (16K, 32K, 64K, 128K tokens), fixed output length of 512 tokens, 50% cache ratio, and Poisson arrival; (4) Real Data, 23,000 real request traces from Kimi's production workload (Section 4), with average input length 7,955 tokens, average output length 194 tokens, ~50% cache ratio, and timestamp-based replay according to actual arrival times. The trace is open-sourced to enable reproducibility. For overload experiments (Section 8.2), the same 23,000-request real trace is replayed at 2× speed to simulate overload conditions.
Base model. All experiments use a dummy model that follows the same architecture as LLaMA2-70B (Section 8.1). This is explicitly stated as a design choice "to protect proprietary information and facilitate reproducibility" — the real Kimi model's architecture details are proprietary, so the evaluations substitute a publicly-known architecture while using real workload traces. The dummy model has the same parameter count (70B), same number of layers, hidden dimensions, and attention heads as LLaMA2-70B, making the FLOP and memory characteristics representative. However, the paper does not replicate Kimi's actual model weights, tokenizer, or output quality — the experiments measure system-level throughput and latency characteristics, not model accuracy.
Metrics. The primary metrics are (1) P90 TTFT (Time To First Token, 90th percentile): the latency from request arrival to first token generation, with the threshold set at 10× the TTFT of a single request running without interference at the lowest RPS; (2) P90 TBT (Time Between Tokens, 90th percentile): the latency between successive token generations, threshold set at 5× the TBT at lowest RPS; (3) Throughput, measured as the maximum request rate (requests per second) that can be sustained while keeping both P90 TTFT and P90 TBT below their respective SLO thresholds — when either metric exceeds its threshold, "the corresponding consumed resources are considered as wasted" (Section 8.1), and only successfully completed requests count toward goodput; (4) Number of rejected requests (Section 8.2, Table 3), measuring how many requests the system must reject under different overload scheduling strategies. For the scheduling experiment in Section 6.2, the metric is average TTFT across all processed requests. All TTFT and TBT values are normalized against their SLO upper limits, establishing a baseline of 1.0 for easy comparison across configurations.
Baselines. The primary baseline is vLLM [13], "one of the state-of-the-art open-source LLM serving systems" (Section 8.1), which incorporates continuous batching and PagedAttention. vLLM's design couples prefill and decoding stages on the same GPU instances, making it a natural comparison point for evaluating disaggregation benefits. The baseline is configured as vLLM-[4M] (4 instances) in the public dataset experiments and scaled to vLLM-[20M] (20 instances) in the real workload experiment, matching the total GPU count of Mooncake configurations for fair comparison. For the scheduling ablation (Section 6.2), additional baselines include random scheduling (prefill instance selected arbitrarily for each request) and load-balancing scheduling (instance with the lightest load chosen), compared against Mooncake's cache-aware and KVCache-centric scheduling variants. For the overload rejection experiments (Section 8.2), the baseline is a strategy that "rejects requests based on load before both stages start" (i.e., checks prefill load and decoding load independently, without early rejection). No other disaggregated serving systems (Splitwise, DistServe, TetriInfer) are evaluated as baselines — the comparison is exclusively against coupled architectures (vLLM) and against ablations of Mooncake's own components.
Generation budget / compute accounting. The paper measures system performance at the granularity of requests per second (RPS) rather than generation budget in tokens or FLOPs. This is a throughput metric that accounts for all system resources (GPU compute, memory bandwidth, network bandwidth, KVCache capacity) collectively. Different RPS rates are tested to find the maximum sustainable throughput before SLO violations occur. The paper does not directly compare FLOPs or GPU-hours between Mooncake and vLLM — the comparison is at equal cluster size (same number of GPUs, same model architecture). For the scheduling ablation (Figure 8), the metric is average TTFT per request, not total throughput. The paper does not provide a token-level or FLOP-level cost model for individual operations (prefill vs. transfer vs. decoding), though the scheduling algorithm internally estimates prefill execution time from an offline predictive model (Section 6.1).
Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing in the traditional machine learning sense. The evaluation is a systems benchmarking methodology: for public datasets and simulated data, requests are generated via Poisson processes at varying RPS rates, and metrics (P90 TTFT, P90 TBT) are measured over the entire run at each rate. For real workload traces, requests are replayed according to their actual timestamps, and the CDF of TTFT and TBT across all 23,000 requests is computed (Figure 13). The overload experiments (Section 8.2) use a fixed replay speed (2×) and count aggregate rejection numbers. The scheduling experiment (Section 6.2, Figure 8) runs on "a Mooncake cluster consisting of 8 prefill instances and 8 decoding instances, using idle machines overnight, and replayed 23,000 real-world requests" — a single run rather than multiple trials. There is no mention of error bars, confidence intervals, or multiple seeds. The paper's reliability rests on the scale of the trace (23,000 requests covering diverse lengths and arrival patterns) rather than on statistical replication of experimental conditions.

Main Quantitative Results

End-to-End Throughput on Public Datasets (ArXiv Summarization and L-Eval)

Headline: Mooncake-[3P+1D] achieves throughput improvements of 20% on ArXiv Summarization and 40% on L-Eval over vLLM-[4M] while satisfying SLOs, with the larger gain on L-Eval attributed to prefix caching reducing prefill time for the high-cache-ratio (>80%) dataset (Section 8.1.1, Figure 11).

ArXiv Summarization results (Figure 11, left columns): On the ArXiv Summarization dataset (~0% cache ratio):

Mooncake-[3P+1D] sustains approximately 3.5 req/s while maintaining P90 TTFT and P90 TBT below the 1.0 normalized SLO threshold.
vLLM-[4M] sustains approximately 2.9 req/s at the SLO boundary — a 20% throughput advantage for Mooncake.
Mooncake-[2P+2D] performs worse on the TTFT metric, exceeding the 1.0 threshold at approximately 2.5 req/s. The paper attributes this to "an imbalance in the load between prefill and decoding instances" (Section 8.1.1) — with only 2 prefill instances serving 2 decoding instances, the prefill pool becomes the bottleneck under Poisson arrival of long-context requests.
At low RPS (below 2.0), all three configurations show normalized P90 TTFT well below 0.5 and normalized P90 TBT well below 0.3 — the SLOs are slack, and the system is underutilized.
As RPS increases beyond each configuration's sustainable rate, the P90 TTFT curves rise sharply (indicated by crossing the 1.0 threshold), while P90 TBT remains more stable — TTFT is the binding constraint.

L-Eval results (Figure 11, right columns): On the L-Eval dataset (>80% cache ratio, average input length 19,019 tokens):

Mooncake-[3P+1D] sustains approximately 1.75 req/s, while vLLM-[4M] sustains approximately 1.25 req/s — a 40% throughput improvement.
The absolute RPS values are lower than ArXiv Summarization because the average input length is more than 2× longer (19K vs. 8K tokens), making prefill the dominant cost.
Mooncake-[2P+2D] again underperforms, exceeding the TTFT SLO at approximately 1.0 req/s.
The paper attributes Mooncake's advantage on L-Eval specifically to prefix caching: "Mooncake's throughput on the L-Eval dataset is further enhanced by prefix caching, which significantly reduces prefill time" (Section 8.1.1). With >80% cache ratio on L-Eval vs. ~0% on ArXiv, the KVCache-centric scheduling's ability to route requests to instances with long prefix matches pays off substantially.

Interpretation of the [3P+1D] vs. [2P+2D] comparison: The configuration with 3 prefill instances and 1 decoding instance outperforms the balanced 2+2 configuration on both datasets. This reveals an important workload characteristic: for these long-context, computation-heavy workloads, prefill is the bottleneck, not decoding. The paper notes that "in real-world clusters, the demand for prefill and decoding instances generally remains stable over certain periods, with only minor temporary imbalances. Thus, the proportion of prefill and decoding instances can be preset" (Section 8.1.1). This is an operational insight: the optimal prefill-to-decoding ratio depends on the input-to-output length ratio of the workload, and for Kimi's long-context-heavy traffic, prefill-heavy configurations are superior.

End-to-End Throughput on Simulated Long-Context Data

Headline: Mooncake achieves throughput improvements of 50% to 525% over vLLM on simulated long-context data (16K, 32K, 64K, 128K prompt lengths), with the advantage growing as context length increases. The paper's headline "525% increase" corresponds to the 128K-prompt scenario (Section 8.1.2, Figure 12).

Detailed results by context length (Figure 12):

16K prompt: Mooncake-[3P+1D] sustains approximately 1.25 req/s, vLLM-[4M] sustains approximately 0.83 req/s — roughly a 50% improvement. Mooncake-[2P+2D] again underperforms, exceeding TTFT SLO at ~0.8 req/s.

32K prompt: Mooncake-[3P+1D] sustains approximately 0.5 req/s, vLLM-[4M] sustains approximately 0.3 req/s — roughly a 67% improvement. The absolute throughput drops sharply compared to 16K because prefill cost grows superlinearly with input length.

64K prompt: Mooncake-[3P+1D] sustains approximately 0.25 req/s, vLLM-[4M] sustains approximately 0.1 req/s — roughly a 150% improvement. The gap widens as vLLM's coupled architecture increasingly struggles with prefill-decoding interference.

128K prompt: Mooncake-[3P+1D] sustains approximately 0.12 req/s, vLLM-[4M] sustains approximately 0.02 req/s — the 525% improvement cited in the paper's abstract and conclusions. At 128K context, vLLM is essentially unable to maintain SLO compliance: the paper notes that "the long-context requests in simulated data significantly disrupt the decoding stage of vLLM. To counteract this, vLLM processes requests individually, rather than in batches" (Section 8.1.2). This means vLLM's throughput collapses because it cannot batch decoding requests — prefill operations are so long that they starve the decoding batch pipeline, forcing vLLM to revert to sequential (unbatched) processing.

Mooncake's TBT advantage across all context lengths: A crucial observation from Figure 12 is that Mooncake's TBT curves remain well below the 1.0 SLO threshold at all tested RPS rates, even as TTFT eventually exceeds limits. In contrast, vLLM's TBT curves rise toward and exceed 1.0 at relatively low RPS (particularly visible in the 64K and 128K panels). The paper's explanation: "Mooncake's two-stage disaggregation design effectively minimizes the impact of the prefill stage on the decoding stage, ensuring it never breaks the TBT SLO" (Section 8.1.2). This is the core architectural advantage — by physically separating prefill computation from decoding batches, Mooncake prevents long-prefill operations from delaying token generation for other users.

The 525% figure in context: The paper's headline number comes from the most extreme scenario tested (128K input prompts, simulating the longest-context use cases). It's important to note that this is a maximum across all scenarios, not an average improvement. The improvement ranges from 50% (at 16K) to 525% (at 128K), with the advantage monotonically increasing with context length. This is consistent with the paper's design philosophy: disaggregation with KVCache-centric scheduling is specifically targeted at long-context workloads, and the benefits are largest where the asymmetric computational demands of prefill and decoding are most pronounced.

Real Workload Replay

Headline: Under real workload replay with 10 prefill + 10 decoding instances (Mooncake-[10P+10D]) vs. 20 vLLM instances (vLLM-[20M]), Mooncake achieves ~100% TBT SLO compliance vs. vLLM's 57%, while maintaining comparable TTFT distributions, and can process approximately 75% more requests while adhering to SLOs (Section 8.1.3, Figure 13).

TTFT distribution (Figure 13, left): The CDF of TTFT shows nearly identical distributions for both systems — "almost 100% of requests meeting the TTFT SLO" (30 seconds) for both Mooncake and vLLM. The curves overlap substantially, with Mooncake showing a slightly longer tail (some requests in the 40-70 second range) but still within the SLO threshold. This indicates that for this workload's prefill characteristics, both architectures can deliver acceptable TTFT — disaggregation does not hurt TTFT despite the added KVCache transfer step, because the layer-wise prefill mechanism hides transfer latency.

TBT distribution (Figure 13, right): This is where the difference is dramatic:

Mooncake-[10P+10D]: "approximately 100% of the requests satisfy the TBT SLO" (0.1 seconds per token threshold). The CDF reaches ~1.0 well before the 0.1s/token mark.
vLLM-[20M]: "only 57% of the requests meet this criterion, with some requests exhibiting extremely high TBTs." The CDF rises to ~0.57 at the 0.1s/token threshold, then has a very long tail extending to ~3.0 seconds per token — 30× the SLO threshold.

The 75% more requests claim: The paper states that "Mooncake can process approximately 75% more requests while adhering to the SLOs" (Section 8.1.3). This is a throughput comparison at equal cluster size: Mooncake successfully completes requests that vLLM either (a) fails to process within TBT SLOs, or (b) cannot schedule due to GPU memory constraints caused by the coupled architecture's inefficiency. Note that this is not a head-to-head RPS sweep like the public dataset experiments — it's a replay of the same real trace on both systems, with Mooncake completing more of the trace's requests within SLOs.

Interpretation of the TBT gap: The 57% TBT compliance for vLLM is the most striking negative result for the coupled baseline. It confirms the paper's central architectural argument: when prefill and decoding share GPU resources, long-context prefill operations cause TBT spikes that violate user-facing latency SLOs. The disaggregated architecture's ability to maintain ~100% TBT compliance means that for user-facing chat applications where token-by-token latency directly affects perceived responsiveness, the coupled approach is fundamentally unsuitable for long-context workloads at scale.

Scheduling Algorithm Ablation (Cache-Aware vs. Load-Balancing vs. Random)

Headline: KVCache-centric scheduling reduces average TTFT by 15× compared to random scheduling (6.26s vs. 92.07s) and by 10× compared to load-balancing scheduling (6.26s vs. 60.41s) on a real workload replay, with the incremental contributions of cache-aware routing and cache load balancing each providing substantial gains (Section 6.2, Figure 8).

Experimental setup: A Mooncake cluster with 8 prefill instances and 8 decoding instances replays 23,000 real-world requests. Four scheduling strategies are compared:

Random scheduling: Prefill instance selected arbitrarily — average TTFT of 92.07 seconds.
Load-balancing scheduling: Instance with the lightest load chosen — average TTFT of 60.41 seconds.
Cache-aware scheduling (Algorithm 1 without the kvcache_balancing_threshold branch — routes based on TTFT including cache match but without proactive hot-spot migration): average TTFT of 14.36 seconds.
KVCache-centric scheduling (the full algorithm with cache load balancing): average TTFT of 6.26 seconds.

Incremental contributions: The jump from 60.41s to 14.36s (4.3× improvement) represents the value of cache-aware routing — directing requests to instances with longer prefix cache matches rather than just the least-loaded instance. The jump from 14.36s to 6.26s (2.3× improvement) represents the value of cache load balancing — proactive replication of hot blocks so that more instances have good prefix matches, reducing the frequency of KVCache transfers and recomputation.

SLO attainment: The paper reports both average TTFT and SLO attainment rate. The KVCache-centric scheduling's 6.26s average falls well below a typical TTFT SLO (the real workload experiment in Section 8.1.3 uses 30s as the TTFT SLO). The random scheduling's 92.07s average would violate any reasonable SLO for a user-facing service.

What this experiment does NOT show: This is a scheduling algorithm comparison within Mooncake's disaggregated architecture — it does not compare against a non-Mooncake baseline like vLLM. The absolute TTFT numbers (6-92 seconds) depend on the specific cluster configuration, request distribution, and model architecture. The experiment validates that KVCache-centric scheduling is substantially better than the alternatives, but does not isolate the contribution of caching from the contribution of disaggregation — all four strategies run on the same disaggregated Mooncake infrastructure, so the benefit of disaggregation itself is not measured here.

Overload-Oriented Scheduling: Early Rejection and Load Prediction

Headline: Prediction-based early rejection reduces wasted rejections by 14.2% compared to the baseline rejection strategy (3,589 vs. 4,183 rejected requests out of 23,000), and by 4.8% compared to naive early rejection without prediction (3,589 vs. 3,771) in a 2× speed replay experiment (Section 8.2, Table 3).

Experimental setup: Mooncake cluster with 8 prefill instances and 8 decoding instances, replaying the 23,000-request real trace at 2× speed to simulate overload. Three rejection strategies are compared:

Baseline (reject based on load at both stages independently, without checking decoding load before prefill): 4,183 requests rejected.
Early Rejection (check decoding load before accepting prefill, but based on current decoding load, not predicted future load): 3,771 requests rejected — a 9.8% reduction from baseline (412 fewer rejections).
Early Rejection based on Prediction (check predicted future decoding load before accepting prefill, using the system-level uniform-time prediction): 3,589 requests rejected — a 14.2% reduction from baseline (594 fewer rejections), and a 4.8% reduction from naive early rejection (182 fewer rejections).

Interpretation of the numbers: The baseline strategy wastes prefill computation on requests that are accepted for prefill but then rejected at the decoding stage due to insufficient capacity. Early rejection avoids this waste by checking decoding load upfront — the 9.8% reduction means that 412 fewer requests had their prefill computation wasted. The additional 4.8% reduction from prediction (182 more requests) represents cases where naive early rejection made suboptimal decisions due to the load oscillation problem: it may have rejected requests that could have been served (because current decoding load appeared high but would have decreased by the time the prefill completed) or accepted requests that couldn't be served (because current decoding load appeared low but was about to spike from in-flight prefill completions). The prediction dampens this oscillation, enabling better decisions.

Magnitude and significance: The 14.2% reduction in wasted rejections is meaningful but not transformative in absolute terms — it represents 594 requests out of 23,000 (2.6% of all requests). However, the value of this improvement is not just in the number of additional requests served, but in the stability it provides. The load oscillation documented in Figure 9 causes alternating periods of prefill and decoding underutilization, which wastes GPU capacity even for requests that are eventually served (because GPUs are idle during the troughs of the oscillation). The paper does not quantify this secondary waste, but the qualitative claim is that prediction-based early rejection "mitigates the fluctuation problem" (Section 7.4), leading to more stable and efficient resource utilization across the cluster.

What this experiment does NOT address: The uniform-time prediction assumes all requests spend $t_d$ in decoding, but the paper does not specify what value of $t_d$ was used or how it was chosen. The sensitivity of the result to $t_d$ is not explored. The experiment uses a fixed 2× replay speed; the benefit of prediction likely depends on the overload ratio (more severe overload → larger oscillation amplitude → greater benefit from prediction), but this is not evaluated. The system-level prediction is compared only against naive early rejection and baseline, not against any alternative prediction strategy (e.g., request-level prediction, moving-average forecasting, queuing-theory-based models).

Ablation Studies and Robustness Checks

Cache eviction policy comparison (Table 1): Across cache capacities from 1,000 to infinite blocks, LRU, LFU, and a custom LengthAwareCache policy achieve nearly identical cache hit ratios. At 50,000 blocks, all three policies achieve a 0.50 hit ratio; at 10,000 blocks, LRU achieves 0.40 vs. LFU's 0.35 and LengthAware's 0.35 — a 5 percentage point advantage for LRU at moderate capacity. At infinite capacity, all three converge to 0.51. The paper's interpretation: "LRUCache performs best under this dataset's patterns, likely due to the temporal proximity in request utilization." This is an informative negative result: sophisticated cache policies (LFU, LengthAware) designed to capture long-term popularity or positional importance do not outperform simple LRU, suggesting that temporal locality dominates in this workload and that engineering effort on cache replacement policies should focus elsewhere.

Prefill-to-decoding instance ratio (Mooncake-[3P+1D] vs. [2P+2D]): Across all evaluated datasets (ArXiv, L-Eval, simulated 16K-128K), the configuration with 3 prefill instances and 1 decoding instance consistently outperforms the balanced 2+2 configuration, often by substantial margins. On ArXiv, [2P+2D] exceeds the TTFT SLO at ~2.5 req/s while [3P+1D] sustains ~3.5 req/s. On L-Eval, [2P+2D] fails at ~1.0 req/s vs. [3P+1D] at ~1.75 req/s. The paper correctly identifies the cause as "an imbalance in the load between prefill and decoding instances" (Section 8.1.1), but does not systematically sweep the prefill:decoding ratio to find the optimum. The choice of [3P+1D] and [2P+2D] appears to be illustrative (showing that prefill-heavy is better for these workloads) rather than the result of an optimization. The paper notes that "future research will explore more flexible deployment and conversion methods" for dynamically adjusting this ratio.

Layer-wise prefill latency reduction (Figure 7): Comparing KVCache store latency between serialized (store all KVCache after computation completes) and layer-wise (asynchronous store per layer overlapped with computation) approaches across sequence lengths from 8K to 128K tokens:

At 8K: Serialized ~0.05s, Layer-wise ~0.05s — negligible difference for short contexts.
At 32K: Serialized ~0.15s, Layer-wise ~0.08s — roughly 2× improvement.
At 128K: Serialized ~0.8s, Layer-wise ~0.2s — 4× improvement, validating that the overlap benefit grows with context length as the volume of KVCache (and thus transfer time) increases. Figure 7 measures only the KVCache store latency, not the end-to-end TTFT improvement, so the absolute time savings are modest relative to total prefill time (which includes attention computation). However, the paper's architectural argument is that this overlap makes VRAM a non-binding constraint for prefill scheduling — even the absolute reduction from 0.8s to 0.2s is sufficient to keep KVCache storage from being the TTFT bottleneck.

Real workload characteristics (Figures 5, 6): The paper provides a detailed characterization of its workload trace that serves as an implicit ablation of assumptions made in prior work:

Input-output length ratio: Average input length 7,590 tokens vs. average output length 182 tokens — a ratio of ~720:1, confirming the workload is heavily prefill-dominated. This validates the architecture's emphasis on prefill optimization and the prefill-heavy instance ratio.
Cache block popularity skew (Figure 6): "Over 50% of cache blocks remaining unused while certain blocks are accessed tens of thousands of times." The CDF shows that ~50% of blocks have a hit count of 0, while the top few blocks have hit counts exceeding 10,000. This extreme skew validates the hot-spot replication heuristic — replicating even a handful of blocks can substantially reduce transfer congestion — and contradicts any assumption of uniform or smoothly-varying cache popularity.
Cache hit ratio ceiling (Table 1): Maximum hit ratio is ~0.51 even with infinite capacity, meaning that for this workload, "only 50% of the KVCache can be reused... even if we assume both the capacity of storage and the TTFT SLO are infinite" (Section 9). This is lower than the 80-90% reuse ratios reported by some prior caching papers on curated benchmarks, and implies that caching alone — even with perfect implementation — cannot solve the throughput problem; it must be combined with disaggregation and scheduling optimizations.

vLLM batch size degradation on long contexts: On the simulated 128K-prompt data, the paper notes that "vLLM processes requests individually, rather than in batches" (Section 8.1.2). This is not a pre-planned ablation but an observed behavior: vLLM's continuous batching is unable to maintain batched decoding when prefill operations are extremely long, causing throughput to collapse to sequential processing. This degradation is visible in Figure 12: vLLM's sustainable RPS on 128K prompts is ~0.02 req/s (roughly one request every 50 seconds), consistent with sequential unbatched processing. Mooncake's disaggregated design maintains batched decoding (the decoding instance continues processing its batch regardless of what the prefill instances are doing), which is the primary mechanism behind the 525% improvement at this context length.

Critical Assessment

Claim 1: "Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs"

What the experiments demonstrate: The 525% figure comes from the 128K-prompt simulated data experiment (Figure 12), where Mooncake-[3P+1D] sustains ~0.12 req/s vs. vLLM-[4M]'s ~0.02 req/s. This is a specific, well-defined comparison at equal cluster size (4 GPUs each) on synthetic data with a 50% cache ratio.

What qualifies or limits this claim:

The 525% figure is the maximum across all tested configurations, not a typical or average improvement. At 16K prompts, the improvement is ~50%; at 32K, ~67%; at 64K, ~150%. The headline number selects the most extreme scenario. This is not misleading — the paper clearly reports all results and the 525% is contextualized as the 128K case — but readers should not interpret it as a general throughput multiplier.
The experiment uses simulated data with uniform context length (all requests have exactly 128K input) and uniform output length (512 tokens). Real workloads are heterogeneous, and the throughput improvement on mixed workloads would likely be between the 16K and 128K extremes. The real workload experiment (Section 8.1.3) reports "75% more requests" rather than a throughput multiplier, and under different SLO thresholds (30s TTFT, 0.1s/token TBT).
The vLLM baseline at 128K is degraded to unbatched processing. The 525% figure partly reflects vLLM's failure mode under extreme conditions rather than Mooncake's absolute superiority under normal conditions. A vLLM variant with chunked prefill (SARATHI-style) might perform better at 128K, but this comparison is not made.
The configuration is Mooncake-[3P+1D] vs. vLLM-[4M] — Mooncake uses 3 prefill + 1 decoding instance, while vLLM uses 4 coupled instances. This is a fair comparison in terms of total GPUs but not necessarily the optimal configuration for either system. vLLM might perform better with a different ratio of instances dedicated to prefill-heavy vs. decoding-heavy work, but the coupled architecture makes such specialization difficult.
The 525% improvement is on throughput (requests/second) under SLO constraints. It does not measure or claim improvements in model accuracy, output quality, or end-to-end request latency for individual requests — it measures how many requests can be served per unit time while keeping TTFT and TBT within bounds.

Verdict: The claim is supported for the specific conditions tested (128K uniform synthetic prompts, LLaMA2-70B architecture, 4-GPU cluster, vLLM baseline degrading to unbatched mode). The paper does not overstate the claim — it says "up to 525% in certain simulated scenarios" — and provides results at lower context lengths showing more modest improvements. The main limitation is the single baseline (vLLM without chunked prefill) and the synthetic uniformity of the workload.

Claim 2: "Under real workloads, Mooncake's innovative architecture enables Kimi to handle 75% more requests"

What the experiments demonstrate: In the real workload replay (Figure 13, Section 8.1.3), Mooncake-[10P+10D] sustains ~100% TBT SLO compliance while vLLM-[20M] sustains only ~57% TBT compliance. The TTFT distributions are nearly identical. The paper reports that Mooncake can process "approximately 75% more requests while adhering to the SLOs."

What qualifies or limits this claim:

The "75% more requests" is not a throughput-per-second comparison like the public dataset experiments. The real workload experiment replays a fixed trace at its original speed; the metric is how many of the trace's 23,000 requests each system successfully completes within SLOs. If Mooncake processes more requests, that means vLLM is either (a) queueing requests beyond SLO limits, (b) producing TBT violations that disqualify requests from counting as "goodput," or (c) rejecting requests implicitly through timeout or memory pressure.
The TTFT SLO is set at 30 seconds and TBT SLO at 0.1 seconds per token. These are the paper's chosen thresholds for the experiment, not necessarily the SLOs used in Kimi's actual production service agreements (which are not disclosed). The 75% figure is SLO-dependent — stricter TBT SLOs would increase vLLM's failure rate and widen the gap; looser SLOs would narrow it.
The vLLM configuration (20 instances of coupled prefill+decoding) is compared against Mooncake (10 prefill + 10 decoding). The total GPU count is equal, but the architectures are different. vLLM uses PagedAttention and continuous batching; it does not use prefix caching across instances (the open-source vLLM at the time only supported local KVCache reuse). So part of Mooncake's advantage may come from prefix caching rather than disaggregation per se. However, prefix caching across instances requires the disaggregated cache pool, so this is a legitimate architectural advantage, not an unfair comparison.
The request trace represents 1 hour of Kimi's production traffic. The specific hour chosen may not be representative of all workload patterns (peak vs. off-peak, weekday vs. weekend, different user populations). The paper does not evaluate robustness across multiple trace samples.
The model is a dummy LLaMA2-70B, not Kimi's actual production model. Model-specific factors (actual KVCache size, layer count, attention implementation) could affect the absolute performance but should preserve the relative comparison since both systems run the same dummy model.

Verdict: The claim is supported with the important caveat that it is SLO-dependent and workload-specific. The 75% figure should be interpreted as "under these specific SLO thresholds and this specific workload trace, Mooncake successfully completes 75% more requests than vLLM." The paper's title and abstract phrasing ("enables Kimi to handle 75% more requests") somewhat implies this is a production measurement, but the experiment uses a dummy model and a sampled trace — it's a realistic simulation of production conditions, not a production A/B test. The 75% figure is directionally credible but should not be treated as a guaranteed improvement for other deployments with different models, workloads, or SLO requirements.

Claim 3: "A prediction-based early rejection policy... reduces wasted computational resources in overloaded scenarios"

What the experiments demonstrate: Table 3 shows that prediction-based early rejection rejects 3,589 requests vs. 4,183 for the baseline (14.2% reduction) and 3,771 for naive early rejection (4.8% reduction) in a 2× speed replay of 23,000 requests.

What qualifies or limits this claim:

The experiment uses a single overload ratio (2× replay speed). The paper's theoretical model (Figure 10) suggests that the benefit of prediction over naive early rejection depends on the oscillation amplitude, which in turn depends on the overload severity. More severe overload → larger oscillations → greater benefit from prediction. Less severe overload → smaller oscillations → prediction may not help. The paper does not sweep overload ratios to characterize this relationship.
The prediction method (system-level uniform-time assumption) is coarse and not compared against alternatives. The 4.8% improvement over naive early rejection might be achievable with even simpler methods (e.g., exponential smoothing of past load), or might be substantially larger with better prediction. Without comparing against a range of prediction strategies, we cannot assess whether the specific prediction mechanism matters or merely that some form of forward-looking forecast helps.
The experiment measures rejected requests but does not directly measure wasted computation (GPU-hours or FLOPs wasted on prefill for requests that are later rejected). The reduction in rejections is a proxy for reduced waste, but if the average prefill cost of rejected requests differs across strategies (e.g., if early rejection tends to filter out long-context requests with higher prefill cost), then the reduction in wasted computation could be larger or smaller than the 14.2% reduction in rejection count. The paper doesn't provide this breakdown.
The load fluctuation problem (Figure 9) is shown for a deployment "before using the prediction-based early rejection" (Section 7.3). The paper does not show the corresponding load plot after deploying prediction-based rejection to demonstrate that the oscillation is actually dampened. Figure 10b is a theoretical illustration, not an experimental measurement. The Table 3 rejection counts are consistent with oscillation dampening (fewer wrong reject/accept decisions), but the direct evidence (load-over-time plots with vs. without prediction) is absent.
The experiment uses a fixed cluster of 8 prefill + 8 decoding instances. The load oscillation amplitude depends on the ratio of prefill to decoding capacity, which is 1:1 in this experiment. Different ratios might exhibit different oscillation characteristics and different benefits from prediction.

Verdict: The claim is directionally supported — prediction-based early rejection reduces wasted rejections compared to both baseline and naive early rejection — but the characterization is incomplete. The 14.2% improvement is specific to a single overload ratio and cluster configuration. The mechanism (dampening load oscillation) is theoretically modeled but not experimentally validated with load-over-time measurements. The 4.8% marginal improvement from prediction over naive early rejection is small enough that measurement noise, trace-specific effects, or parameter tuning (the uniform $t_d$ value) could plausibly account for it. This is the weakest experimental claim in the paper, and the authors appropriately flag it as ongoing work (Section 7.4: "This type of prediction is ongoing and requires less precision, making it more appropriate for overload scenarios").

What the experiments demonstrate: Figure 8 shows that KVCache-centric scheduling reduces average TTFT by 10× compared to load-balancing scheduling (6.26s vs. 60.41s). Figures 11-12 show that Mooncake as a whole achieves substantial throughput improvements over vLLM across multiple datasets. The scheduling algorithm (Algorithm 1) is clearly described and the cost model (TTFT prediction) is well-motivated.

What qualifies or limits this claim:

The Figure 8 experiment isolates the scheduling algorithm within Mooncake's architecture but does not compare against other disaggregated schedulers. The comparison is against random and simple load-balancing — these are weak baselines that any non-trivial scheduler should beat. The paper does not implement or compare against the scheduling algorithms from Splitwise [7], DistServe [8], or TetriInfer [9], which also perform cache-aware or load-aware scheduling. So we know KVCache-centric scheduling is much better than random scheduling, but we don't know if it's better than alternative cache-aware schedulers.
The kvcache_balancing_threshold parameter (Section 6.1) is "currently adjusted manually." The paper does not report the threshold value used in the experiments, sensitivity to this parameter, or whether the results are robust to different threshold choices. If the performance is highly sensitive to this parameter, the approach may be fragile across workload shifts.
The offline predictive model for prefill execution time (Section 6.1) is assumed to have "small error bound," but no validation of this assumption is provided. If the prediction model has systematic errors (e.g., underestimating prefill time for certain context lengths), the scheduler might make suboptimal routing decisions.
The transfer time estimation (Section 6.1) is identified as the hardest component to predict, but the paper does not describe how it is estimated in practice (e.g., simple bandwidth×size model, historical transfer time averages, congestion signals). The accuracy of this estimate directly affects the scheduler's ability to make good "transfer vs. recompute" decisions.
The KVCache-centric scheduling is an online algorithm that makes per-request decisions based on estimated TTFT. It is not proven to be optimal or even close-to-optimal — there's no comparison against an offline optimal scheduler (which would have perfect future knowledge) to bound the suboptimality gap. This makes it hard to assess whether the 10× improvement over load-balancing is because KVCache-centric scheduling is good or because load-balancing is terrible for this workload.

Verdict: The claim is supported for the specific comparison against weak baselines within Mooncake's architecture. The scheduling algorithm is clearly the right design for the problem — folding cache reuse, load, and transfer cost into a single TTFT metric is conceptually elegant. But the experimental validation is limited: we don't know how it compares to other cache-aware schedulers, how sensitive it is to parameter choices, or how close it gets to optimal. The paper's contribution on scheduling is primarily the architectural insight (KVCache as the central scheduling object) rather than the empirical demonstration of its optimality.

Missing Experiments That Would Strengthen the Paper

Ablation of disaggregation itself: The paper compares Mooncake (disaggregated) against vLLM (coupled) but does not include a configuration where Mooncake's prefill and decoding run on the same nodes (a coupled Mooncake variant) to isolate the contribution of disaggregation from other Mooncake features (layer-wise prefill, CPP, cache-aware scheduling). This would help answer: how much of the throughput improvement comes from disaggregation vs. from better scheduling and caching?
Sweep of prefill:decoding instance ratios: The paper tests only [3P+1D] and [2P+2D] (and [10P+10D] for the real workload). A sweep across ratios (e.g., [4P+0D] through [0P+4D]) would characterize the throughput surface and validate the claim that the optimal ratio depends on workload characteristics. This is particularly important because the paper argues that different workloads need different ratios (Section 8.1.1 acknowledges this) but doesn't quantify the sensitivity.
Comparison against chunked prefill (SARATHI-style) baseline: The paper's Section 5 argues that chunked prefill is insufficient for long-context workloads and justifies the choice of physical disaggregation. But the experiments compare against standard vLLM, not against a vLLM variant with chunked prefill enabled. A vLLM+chunked-prefill baseline at the long-context settings (64K, 128K) would directly test the paper's architectural claim.
Sensitivity to the kvcache_balancing_threshold: The scheduling algorithm's key parameter (the threshold that determines when to transfer remote KVCache vs. recompute) is manually tuned, but no sensitivity analysis is provided. What happens if the threshold is set too high or too low? Is there a broad plateau of good performance or a sharp optimum?
Load-over-time plots with and without prediction-based early rejection: The qualitative evidence for the oscillation problem is Figure 9 (before prediction). Showing the corresponding plot after deploying prediction would directly validate that the mechanism works as theorized, rather than relying solely on the rejection count reduction in Table 3.
Evaluation on additional model architectures or scales: All experiments use the dummy LLaMA2-70B model. Testing on a smaller model (e.g., LLaMA2-7B) and a larger model (if hardware permits) would characterize how the benefits of disaggregation and KVCache-centric scheduling scale with model size. The paper's claims about MFU and memory-bound decoding behavior are architecture-general, but the empirical validation is single-model.
Cost or FLOP efficiency comparison: The paper measures throughput in requests/second but does not measure cost (dollar per request) or energy efficiency. Since the architecture mandates additional network bandwidth for KVCache transfers and additional CPU DRAM for the cache pool, a total-cost-of-ownership comparison against the coupled baseline (including network and memory costs, not just GPU count) would strengthen the practical argument for Mooncake's design.
End-to-end latency distribution for individual requests: The experiments report P90 TTFT and P90 TBT as aggregate metrics but do not show full latency distributions (beyond the CDFs in Figure 13). Understanding the tail latency behavior — particularly for the longest-context requests — would help assess whether Mooncake's improvements come from helping the median request or from preventing catastrophic tail latencies that ruin user experience.

These missing experiments do not invalidate the paper's contributions, but they leave important questions unanswered about generalizability, sensitivity, and the decomposition of performance gains across Mooncake's multiple innovations.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Not Accounted For, Making the ~4× Efficiency Claim an Upper Bound

The assumption or constraint: The compute-optimal allocation policy requires knowing each question's difficulty before deciding how to spend the inference budget. The paper's method for estimating difficulty — generating 2,048 samples per question, scoring them with the PRM, and binning into quintiles — is extraordinarily expensive. The paper explicitly acknowledges this cost and states that it is not included in any budget calculation:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2)

The compute-optimal policy uses either oracle difficulty (requiring ground-truth labels) or predicted difficulty (requiring 2,048 PRM-scored samples per question). The "predicted" variant avoids needing the correct answer but still requires generating thousands of samples and running the PRM on all of them — a cost that may exceed the test-time compute budget being optimized.

The consequence: The headline efficiency claim — that compute-optimal scaling achieves "more than ~4× better efficiency over a standard best-of-N baseline" (Section 3.1, Abstract) — is computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter for all but the largest inference budgets. For example, if a practitioner has a budget of 64 generations per question and must first spend 2,048 generations to estimate difficulty, the effective cost per question is 2,112 generations — making the ~4× efficiency gain over best-of-N at 256 generations irrelevant in practice. The ~4× figure should therefore be understood as an upper bound on achievable efficiency, not a realized deployment gain.

This is especially acute because the paper's compute-optimal policy is evaluated at budgets of 16–256 generations (Figures 4, 8), while the difficulty estimation cost is 2,048 generations — an order of magnitude larger. The cost-benefit tradeoff only becomes favorable if the difficulty estimate can be amortized across many questions (e.g., in batch evaluation) or if a much cheaper difficulty predictor is available — neither of which is demonstrated.

What evidence exists in the paper: The paper is transparent about this limitation but does not quantify its impact. Section 3.2 states the difficulty estimation method and notes the cost is not accounted for. Figures 4 and 8 show compute-optimal scaling curves that assume difficulty is known a priori, with no cost offset. The paper does not provide experiments showing what fraction of the total budget difficulty estimation consumes at different operating points, nor does it evaluate how performance degrades if difficulty is estimated with fewer samples (e.g., 4, 16, 64 instead of 2,048) — a natural ablation that would characterize the estimation-accuracy-vs-cost tradeoff.

Mitigation status: The paper partially acknowledges this gap and frames it as future work:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity, and we leave the investigation of the exploration and exploitation balance — i.e., how to split compute between difficulty assessment and problem-solving — to future work" (Section 3.2)

It also suggests an alternative: "future work on pretraining or finetuning models to directly predict difficulty of a question" (Section 8). No such model is developed or evaluated. Until a cheap difficulty predictor exists, the compute-optimal approach requires either accepting the estimation overhead (which may wipe out the efficiency gains) or using a coarser, cheaper difficulty signal whose fidelity is unknown. This is arguably the single most important practical limitation of the paper's approach — without a solution, the ~4× efficiency gain is a theoretical result rather than a deployable one.

Limitation 2: Test-Time Compute Cannot Help on Hard Problems — a Hard Capability Boundary

The assumption or constraint: The paper's approach fundamentally assumes that the base model's proposal distribution contains correct solutions at some non-trivial rate. The entire framework — search against PRM verifiers and iterative revisions — operates by either finding correct solutions that already exist in the distribution or refining nearly-correct solutions toward correctness. When the base model's pass@1 is near zero on a problem class, no amount of search or revision can help because there are no correct solutions in the proposal distribution to find or refine.

The paper expresses this boundary condition clearly in the context of the compute-optimal analysis:

"On the hardest questions, no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated" (Section 5.3 discussion of Figure 3, right)

Similarly, for revisions: "On the hardest problems (bin 5), test-time compute provides essentially zero benefit regardless of budget" (Section 7 discussion, confirmed by Figure 7 right and Figure 9 bottommost curves).

The consequence: There is a hard capability ceiling that test-time compute cannot penetrate. If the base model's capabilities are insufficient for a problem class (e.g., out-of-distribution reasoning, novel problem structures not seen in training, or tasks requiring knowledge the model lacks), the methods in this paper provide zero improvement, regardless of how much inference compute is allocated. This is not a gradual degradation — it's a cliff. On difficulty bin 5 (hardest problems), accuracy remains at 1–3% for all methods and all budgets (Figures 3 right, 7 right). In the FLOPs-matched comparison (Section 7), the bin 5 scaling curves are essentially flat near 0–5%, and the ~14× larger pretrained model decisively outperforms test-time compute scaling.

This has direct implications for deployment decisions. If a practitioner's workload skews toward genuinely novel or difficult problems outside the base model's capability envelope, investing in test-time compute infrastructure (PRM training, revision model fine-tuning, compute-optimal scheduling) will yield negligible returns. The only viable path for such workloads is scaling pretraining — either larger models or more training data. The paper's FLOPs-matched analysis reinforces this: on hard questions, pretraining is almost always more effective than test-time compute, with a −52.9% relative disadvantage for test-time compute at high inference-to-pretraining ratios (Figure 1 bar charts, PRM search on hard questions at R >> 1).

What evidence exists in the paper: The evidence is consistent and unambiguous across all experiments. Figure 3 (right, bin 5): beam search and best-of-N both produce ~1–3% accuracy at all budgets from 4 to 256 generations. Figure 7 (right, bin 5): all sequential-to-parallel ratios yield ~2–3% accuracy. Figure 9 (bin 5 curves, bottommost): compute-optimal scaling is essentially flat and well below the ~14× larger model's performance. The paper is candid about this: "test-time compute amplifies existing capability but does not create it from nothing" (Section 7 summary discussion).

However, the paper does not systematically characterize where this capability boundary lies. We know that difficulty bin 5 (lowest pass@1 quintile) is beyond reach, but we don't know the shape of the transition — is there a gradual falloff from bin 4 to bin 5, or a sharp threshold? The 2,048-sample pass@1 estimation provides a continuous difficulty measure, but the binning into five quintiles obscures whether there's a critical pass@1 threshold below which test-time compute provides no benefit.

Mitigation status: The paper does not attempt to mitigate this limitation — it treats it as a fundamental boundary condition and is transparent about it. The takeaway in Section 7 explicitly states: "test-time compute can amplify existing capability but does not create it from nothing." Future work on combining test-time compute with iterative fine-tuning (distilling the outputs of test-time compute back into the base model) could potentially push the capability boundary over successive rounds, but this is only suggested as a direction (Section 8) and is not demonstrated. For single-round inference on a fixed base model, the boundary is absolute.

Limitation 3: Single Benchmark, Single Model Family — Unknown Generalizability

The assumption or constraint: All experiments use the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* as the base model. The paper states: "We believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but provides no evidence for this claim by testing other model families or benchmarks.

The consequence: The paper's findings may be specific to the MATH-PaLM 2-S* combination for several reasons:

Task specificity: MATH consists of competition-level math problems requiring multi-step symbolic reasoning. The difficulty-dependent behavior of search and revisions — beam search over-optimizing on easy problems, revisions helping on easy problems, hard problems remaining unsolved — may not generalize to other reasoning domains (code generation, logical deduction, scientific QA) or to tasks requiring factual recall rather than inference, or to open-ended generation with no ground-truth answer.
Model specificity: PaLM 2-S* has specific calibration properties, error patterns, and in-context learning capabilities. The PRM's quality — and thus its over-optimization behavior, which is the primary bottleneck for search scaling — depends entirely on the base model's output distribution. A model with different calibration characteristics (e.g., better-calibrated confidence, fewer systematic errors on certain problem types) might exhibit qualitatively different scaling curves. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families.
Difficulty distribution specificity: The MATH benchmark's difficulty distribution (how many problems fall into each pass@1 quintile) may differ from other benchmarks. A benchmark with more hard problems would show smaller aggregate gains from test-time compute; a benchmark with more medium problems would show larger gains. The paper's reported improvements (e.g., ~4× efficiency) are averages weighted by MATH's difficulty distribution and may not transfer.
Answer format specificity: MATH problems have exact answers that can be checked with string matching, enabling both oracle difficulty estimation (via pass@1) and PRM training (via Monte Carlo rollout correctness using ground-truth comparison). Tasks without clean correctness signals — dialogue quality, summarization faithfulness, creative writing — would require fundamentally different verifier training and difficulty estimation approaches that the paper does not address.

What evidence exists in the paper: There is none. The paper acknowledges this explicitly in the future work discussion: "all results are on MATH with PaLM 2-S*" and "extending the compute-optimal framework to tasks where correctness is ambiguous, multi-dimensional, or subjective would require fundamentally different verifier training and difficulty estimation approaches" (Section 8, implicit in the discussion of open-ended generation). The paper provides no cross-benchmark or cross-model validation, and no ablation showing that the difficulty-dependent patterns hold for a different model trained on a different data distribution.

Mitigation status: Not addressed. The paper is a first systematic study and explicitly scoped to MATH with PaLM 2-S*. The authors do not claim generalizability to other benchmarks or model families, but they also do not caution readers against assuming it. The practical consequence is that a practitioner cannot confidently apply the paper's specific policy recommendations (use beam search on medium problems, revisions on easy problems) to a different model on a different task without re-running the entire difficulty-dependent analysis. The paper provides the methodology for doing so (a contribution that does generalize), but the specific optimal strategies are likely model- and task-dependent.

Limitation 4: Verifier Over-Optimization Is a Hard Ceiling, and Improving Search Algorithms Makes It Worse

The assumption or constraint: The paper's search methods all rely on a PRM trained via Monte Carlo rollouts. This PRM is imperfect — it assigns high scores to some incorrect solutions, particularly those that exploit patterns the PRM has learned to associate with correctness. As search becomes more aggressive (higher budgets, stronger search algorithms), it increasingly finds solutions that score highly under the PRM but are actually wrong — a phenomenon the paper terms "over-optimization."

The paper demonstrates this clearly: "beam search degrades easy-problem performance at high budgets" (Figure 3, right, bin 1), "lookahead search — the most powerful optimizer — paradoxically performs worst overall" (Figure 3, left), and qualitative examples show "low-information repetitive steps at the end of solutions" and "overly short 1–2 step solutions" that score well under the PRM but are incorrect (Appendix M, Figure 29, referenced in Section 5.3).

The consequence: There is a hard ceiling on test-time compute scaling that is determined by verifier quality, not by the search algorithm's sophistication. More importantly, the paper's results imply that more powerful search algorithms can be counterproductive — lookahead search, which gives the PRM more context to assess partial solutions, underperforms simpler beam search at the same generation budget (Figure 3, left) because its additional cost reduces the number of beams explored, and the extra context doesn't sufficiently improve the PRM's accuracy to compensate.

This has non-obvious implications for practitioners: investing effort in more sophisticated search algorithms (MCTS, deeper lookahead, learned search policies) is likely to yield diminishing or negative returns unless accompanied by substantial improvements in verifier robustness. The bottleneck is not search — it's verifier calibration under distribution shift. The compute-optimal policy mitigates over-optimization by avoiding aggressive search on problems where the verifier is reliable (easy problems) and deploying it only where the verifier signal has room to provide genuine guidance (medium problems). But even on medium problems, the PRM's reliability eventually breaks down at high budgets — the beam search curves in Figure 3 (right, bin 3–4) flatten and begin to decline, albeit more gradually than for easy problems.

What evidence exists in the paper: Extensive. Figure 3 (right) clearly shows the over-optimization pattern for easy problems — beam search accuracy decreases from ~78% to ~77% as budget increases from 4 to 256, while best-of-N improves from 68% to 88%. Figure 3 (left) shows lookahead search underperforming all methods. Appendix M provides qualitative examples. However, the paper does not systematically characterize why the PRM makes certain types of errors — e.g., whether it's biased toward longer solutions, toward solutions with certain structural patterns, or toward solutions from certain parts of the output distribution. Understanding the PRM's failure modes could inform better training procedures (adversarial training, calibration objectives), but this analysis is absent.

Mitigation status: The paper identifies this as a critical limitation and frames it as a research direction: "Improving verifier robustness is the key bottleneck for further scaling test-time compute, not improving search algorithms" (Section 8, implicit). The compute-optimal policy mitigates over-optimization by avoiding aggressive search where it hurts, but does not solve the underlying problem. The paper does not explore verifier improvements beyond the basic Monte Carlo rollout training procedure — no adversarial training, no ensemble methods, no calibration post-processing. The acknowledgment that PRM800k (human-labeled data) was "largely ineffective" for their models (Section 5.1) suggests that data quality is a factor but doesn't resolve what would work better.

Limitation 5: Revisions and Search Are Never Combined — an Artificial Separation

The assumption or constraint: The paper studies two complementary test-time compute mechanisms — search against a PRM verifier (Section 5) and iterative revisions modifying the proposal distribution (Section 6) — but studies them as independent pipelines. Section 8 explicitly acknowledges this gap:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The revision model generates candidates that are scored by a separately-trained ORM (because the base-model PRM doesn't transfer well to the revision model's outputs, Figure 15a). The PRM search pipeline uses the base model as its proposal distribution with no revisions. The two mechanisms are never integrated: revision chains are not guided by PRM step-level scores, PRM beam search is not applied to revision model outputs, and the PRM is not used to decide when a revision is on-track vs. when to restart.

The consequence: The paper's results represent a lower bound on what a fully integrated system could achieve. The two mechanisms have complementary strengths, as the paper itself demonstrates: revisions help most on easy problems (local refinement of nearly-correct answers), while search helps most on medium problems (exploring diverse solution strategies). A combined system could potentially:

Use the revision model as the proposal distribution within beam search, generating higher-quality candidate steps at each beam expansion because the model conditions on previous failed attempts.
Use the PRM to guide which revision steps to pursue — rather than blindly generating a long revision chain, score each revision step and decide whether to continue refining, backtrack, or restart.
Apply compute-optimal allocation across the joint space of (search algorithm, revision depth, sequential-to-parallel ratio) rather than optimizing each axis independently.

The paper demonstrates that each mechanism individually achieves ~4× improvement over best-of-N (Figures 4, 8), but does not show whether the gains are additive (combined system achieves ~8×?), subadditive (gains overlap), or superadditive (synergistic). The practical implication is that a practitioner building on this work cannot simply deploy both mechanisms and expect the sum of the individual improvements — the interaction effects are unknown.

What evidence exists in the paper: None directly, but there is suggestive evidence that combination could help. Figure 3 (right) and Figure 7 (right) show complementary difficulty-dependent strengths: search excels on bins 3–4 where revisions are weaker, and revisions excel on bin 2 where search is weaker. The FLOPs-matched analysis (Figure 9) shows that revisions outperform search overall in the pretraining-compute tradeoff, but search has advantages on specific difficulty and R regimes. The lack of a combined experiment is a significant gap in the paper's coverage of its own design space.

Mitigation status: The paper explicitly flags this as future work (Section 8) but provides no preliminary results or analysis of how combination might work. The engineering challenges are non-trivial: the revision model's output distribution differs from the base model's (Section 6), requiring a separate verifier (Figure 15a), and the computational cost model would need to account for the combined operations. But the paper doesn't even provide a speculative design sketch of how the combination would work, leaving the integration problem entirely to future work.

Limitation 6: The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate — a Fundamental Fragility

The assumption or constraint: The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target answer (Section 6.1). This means the model never sees examples of what to do when the current answer is already correct — it has no training signal for "recognize correctness and do nothing." At inference time, when the revision chain produces a correct answer, the model may condition on that correct answer in the next revision step and "revise" it into an incorrect answer because the training distribution teaches it that all in-context answers are wrong and should be changed.

The paper quantifies this: "approximately 38% of correct answers produced during a revision chain get 'revised' back to incorrect answers in the subsequent step" (Section 6.1, described in text around the reversion problem).

The consequence: This creates a fundamental instability in the revision chain. Even if the model produces a correct answer at step k, there's a 38% chance that step k+1 will corrupt it. The revision chain does not monotonically improve — it walks randomly, with each step having some probability of improvement and some probability of regression. The paper's mitigation — using majority voting or verifier-based selection across the chain to pick the best answer from any step — is a workaround, not a solution. It means the system must generate many revision steps and then retrospectively identify which one was best, wasting computation on steps after the correct answer was already found.

This fragility limits the practical value of sequential revisions as a latency-sensitive deployment strategy. If the system must generate a long chain and then select the best answer post-hoc, it cannot stop early when a correct answer is produced — because it doesn't know it's correct. This means the full budget must be spent even on easy problems where the correct answer appears early in the chain, wasting computation. A more robust revision model that can recognize correctness and stop would be substantially more efficient, but the paper's training procedure cannot produce such a model.

Additionally, the ReST^EM experiment (Appendix K, Figure 16) shows that attempts to further optimize the revision model via RL-style training actually make things worse — the ReST^EM-trained model shows "substantially hurt" performance with sequential revisions, dropping to ~33.5% at 256 generations vs. ~38.5% at the optimal ratio for the base revision model. This suggests that the revision training procedure is fragile and that on-policy data collection exacerbates spurious correlations in the revision trajectories. The positive results depend on specific choices (offline data construction with edit-distance-based incorrect-correct pairing) that may not transfer to other training recipes.

What evidence exists in the paper: The 38% reversion rate is quoted in Section 6.1 but the exact measurement methodology is not detailed — it's unclear whether this is measured on a held-out set of revision chains, on the training data, or estimated from the pass@1 trajectory in Figure 6 (left). Figure 6 (left) shows the pass@1 per step increasing from ~18.2% at step 1 to ~24–25% by steps 15–20, but does not directly show the reversion rate. The paper does not analyze whether the reversion rate changes over the course of the chain (e.g., later steps might be more or less prone to reversion) or whether it correlates with problem difficulty. The ReST^EM degradation is shown in Figure 16 but the mechanism is only hypothesized ("on-policy data collection exacerbates spurious correlations"), not empirically diagnosed.

Mitigation status: The paper mitigates the reversion problem via post-hoc selection — picking the best answer from any point in the chain using majority voting or verifier-based selection — rather than by fixing the model. This treats the symptom (wrong final answer) but not the cause (model cannot recognize correctness). The paper acknowledges this is a workaround, not a solution, and does not propose a specific method for training a revision model that can recognize correctness. The ReST^EM failure (Appendix K) further suggests that naively optimizing the revision model may amplify the problem rather than solve it. This limitation interacts with Limitation 5 (no combination of search and revisions): a PRM-guided revision process could potentially detect correct answers and stop early, mitigating the reversion problem, but this integration is not explored.

7. Implications and Future Directions

How This Work Changes the Landscape

Mooncake reframes LLM serving from a throughput-centric optimization problem to a KVCache-centric scheduling problem under persistent overload. This is not merely a new system design — it's a conceptual shift in what the scheduler optimizes over and what constraints are considered binding. Before Mooncake, the dominant mental model for LLM serving treated KVCache as a memory management detail (vLLM's PagedAttention made it efficient within VRAM) and prefix caching as a throughput optimization (Prompt Cache, SGLang showed reuse reduces computation). Scheduling was primarily about load balancing — routing requests to the least-busy GPU. Mooncake inverts this: the scheduler's primary decision variable is the distributed KVCache topology — where cached blocks reside, how much prefix can be reused at each instance, whether to transfer or recompute missing blocks, and whether to replicate hot blocks. Instance load is folded into the objective as a constraint on the KVCache optimization. Algorithm 1 makes this inversion explicit by computing per-instance TTFT as a function of queue time, prefill time (which depends on cache hit length), and KVCache transfer time (which depends on network conditions), then routing to minimize the sum — even if that means not using the instance with the longest cache match.

This KVCache-centric reframing has a concrete methodological consequence: it adds KVCache topology as a first-class optimization dimension alongside parallelism strategy, batch scheduling policy, and memory management. The spatial distribution of cached computation across nodes, the replication factor of hot blocks, and the transfer topology connecting them become design variables that the scheduler actively manipulates (via the hot-spot migration heuristic in Section 6.2) rather than passively accepts. This is analogous to how content delivery networks reorganized internet infrastructure around content location rather than server capacity, and it opens a design space that the paper only begins to explore. Future serving systems will likely need to reason about cache placement, cache-aware routing, and cache-constrained admission control as integrated scheduling decisions, not as separate optimizations.

The paper's second landscape-shifting contribution is establishing overload-oriented scheduling as a distinct research regime with its own failure modes and solutions. Prior LLM serving research overwhelmingly assumed capacity-sufficiency — Splitwise, DistServe, and TetriInfer all optimize throughput under the implicit assumption that all requests will eventually be served. Mooncake, drawing from Kimi's production reality of "rapid growth in user requests" where "the growth rate of the cluster's inference resources is far slower than the increase in incoming requests" (Section 7), identifies that overload creates qualitatively different scheduling challenges. The paper's diagnostic of the load oscillation problem (Section 7.3, Figure 9, Figure 10a) is particularly significant: it shows that any disaggregated architecture that makes accept/reject decisions based on current decoding load will experience anti-phase oscillation between prefill and decoding pools because the time lag between prefill scheduling and decoding execution creates a delayed feedback loop. This is not a bug in Mooncake's implementation — it's a structural property of disaggregated systems under overload. The implication is that prediction-based admission control is not a nice-to-have optimization but a necessary stability mechanism for any disaggregated architecture operating near capacity. The paper's demonstration that even a coarse system-level prediction (uniform decoding time assumption) is sufficient to dampen the oscillation (Table 3: 14.2% reduction in wasted rejections) is encouraging because it suggests the prediction problem is tractable without solving the much harder per-request output length prediction problem.

The paper also reconciles conflicting architectural intuitions in the LLM serving community. The debate between coupled architectures (vLLM, continuous batching with chunked prefill) and disaggregated architectures (Splitwise, DistServe) has been largely theoretical. Mooncake provides a concrete boundary condition: for long-context workloads (average input 8K–128K tokens), disaggregation is not just beneficial but necessary to maintain TBT SLO compliance — vLLM's coupled design achieves only 57% TBT compliance under real workloads vs. Mooncake's ~100% (Figure 13). However, the paper does not categorically reject chunked prefill — it uses it for short requests where inlining prefill into decoding batches improves MFU without violating TBT SLOs (Section 5). The paper thus provides a graduated decision framework rather than a binary choice: inline short prefill into decoding batches, but physically separate prefill for long-context requests. This resolves the apparent contradiction between advocates of chunked prefill (who correctly observe it improves MFU for balanced workloads) and advocates of disaggregation (who correctly observe it prevents prefill-decoding interference for imbalanced workloads). Both are right — the optimal choice depends on workload characteristics, and Mooncake's architecture supports both.

Finally, the paper redirects research attention from sophisticated search algorithms to robust verifiers — but in the serving domain, not the model output domain. Just as the companion paper summary identified verifier over-optimization as the primary bottleneck for test-time compute scaling (better search algorithms paradoxically hurt because they exploit verifier weaknesses), Mooncake identifies a structurally analogous phenomenon in scheduling: aggressive throughput optimization based on current load (the scheduling equivalent of greedy search) causes oscillation and wasted resources (the scheduling equivalent of over-optimization). The solution in both cases is to incorporate prediction — not to make the optimization more aggressive, but to make it more forward-looking. In Mooncake's case, prediction-based early rejection (looking ahead to future decoding load) replaces reactive admission control (looking at current load). This parallel suggests a deeper connection between search-time optimization and scheduling-time optimization that future work could explore.

Follow-Up Research This Work Enables

Cheap, online difficulty estimation for scheduling admission control. Mooncake's prediction-based early rejection uses a system-level uniform-time assumption (Section 7.4) that is acknowledged to be coarse. A natural follow-up would replace this with a lightweight, continuously-updated predictor of decoding capacity that operates at the granularity of individual instances. The specific experiment: train a small model (a few million parameters) that takes as input the current queue depth, recent completion rate, and average output length of in-flight requests for a decoding instance, and predicts the TBT ratio at a future time horizon (e.g., the 95th percentile prefill latency). Compare this learned predictor against the uniform-time baseline under varying overload ratios (1.5×, 2×, 3×, 5×) and measure both rejection count and load oscillation amplitude (the variance of decoding load over time). The paper's open-sourced trace (Section 4) enables this experiment without access to a production cluster. This would characterize the value of more accurate load prediction independent of Mooncake's other architectural choices.

Combining CPP with dynamic pipeline depth selection based on TTFT deadlines. Mooncake's Chunked Pipeline Parallelism (Section 5.1) uses a fixed pipeline group size $X$ for long-context prefill. A follow-up would make $X$ adaptive: for each request, estimate the TTFT if processed with $X = {1, 2, 3, 4}$ nodes, accounting for current queue depths at each node, and select the smallest $X$ that meets the TTFT SLO. This would reduce resource fragmentation (fewer nodes tied up in pipeline groups for requests that don't need the full parallelism) and improve overall throughput. The experiment: replay the real trace on a Mooncake cluster with 8 prefill instances, compare fixed $X = 2$ vs. fixed $X = 4$ vs. adaptive $X$ , and measure throughput at SLO boundary and average pipeline group utilization (fraction of time nodes in a group are actively computing vs. idle waiting for pipeline bubbles). The key hypothesis: adaptive $X$ should outperform both fixed configurations because it matches parallelism to per-request TTFT requirements.

Cache topology optimization: anticipatory placement of KVCache blocks based on request correlation patterns. Mooncake's hot-spot migration heuristic (Section 6.2) is reactive — it replicates blocks only after they've been accessed and caused a routing decision away from the best-cache-match instance. A stronger approach would be anticipatory: if Conductor observes that requests for document A are frequently followed within N seconds by requests for document B (e.g., users reading a multi-part report), it could proactively replicate B's KVCache blocks to the instances where A's blocks are cached, reducing TTFT for the follow-up requests. The specific experiment: on the real trace, train a simple Markov model that predicts which hash_ids are likely to appear in the next request from the same session given the current request's hash_ids. Simulate a cache placement policy that, when a request is routed to instance $p$ , also initiates background transfer of the top-K predicted next blocks to $p$ (subject to a DRAM budget). Measure the increase in cache hit ratio and reduction in average TTFT compared to the reactive-only policy. The paper's trace (Section 4) includes hash_ids that make this experiment feasible. This would test whether spatial locality in KVCache access patterns (not just temporal locality, which LRU already captures) can be exploited for further gains.

Stress-testing the oscillation analysis: does prediction-based early rejection prevent oscillation at all overload ratios, or does the system re-enter instability at higher load? The paper demonstrates oscillation dampening at 2× replay speed (Table 3) but does not sweep overload ratios. A follow-up would systematically vary the replay speed from 1.2× to 5×, measuring both rejection count (as in Table 3) and load oscillation amplitude (the peak-to-trough range of prefill and decoding utilization over a rolling window, analogous to Figure 9). The key hypothesis: prediction-based early rejection should maintain stability up to some critical overload ratio, beyond which the uniform-time assumption becomes too inaccurate and the system re-enters oscillation. Identifying this critical ratio — and whether it can be pushed higher with better prediction — would establish the operational envelope for Mooncake-style overload scheduling. A negative result (oscillation recurs at modest overload ratios despite prediction) would indicate that system-level prediction is insufficient and that request-level output length prediction, despite its difficulty, is necessary for stable operation under severe overload.

Integration of KVCache-centric scheduling with heterogeneous accelerator hardware. Section 10 envisions a future with separate computation-oriented and bandwidth-oriented accelerators for prefill and decoding respectively. A concrete follow-up would prototype this: deploy prefill on GPU nodes (computation-optimized) and decoding on a simulated bandwidth-optimized node (e.g., a GPU with artificially increased memory bandwidth, or an FPGA-based attention accelerator). The KVCache pool would span both accelerator types, with Messenger handling GPU-to-accelerator transfers. The experiment: compare end-to-end throughput and cost-per-request for a homogeneous GPU cluster vs. a heterogeneous cluster with the same total hardware cost (but different per-node costs), under the real workload trace. The hypothesis is that decoding is memory-bandwidth-bound (Figure 2, right), so cheaper bandwidth-optimized hardware could match GPU decoding throughput at lower cost, while prefill remains on computation-optimized GPUs. Mooncake's disaggregated architecture makes this heterogeneous deployment natural because prefill and decoding are already physically separated and communicate only through the KVCache pool. This would validate Mooncake's architectural bets against the hardware trends the paper anticipates.

Practical Applications and Downstream Use Cases

Cost-efficient deployment for long-context LLM services. For any MaaS provider serving long-context models (document summarization, codebase analysis, multi-turn conversation with long history), Mooncake's disaggregated architecture with prefill-heavy instance ratios delivers substantially higher throughput per GPU than coupled alternatives. The paper quantifies this: on 128K-context prompts, Mooncake achieves 0.12 req/s vs. vLLM's 0.02 req/s — a 6× throughput advantage at the same GPU count (Figure 12). On real workloads, Mooncake handles 75% more requests than vLLM while maintaining TBT SLO compliance (Figure 13). The practical decision rule: if average input length exceeds ~4K tokens and TBT SLOs are strict (user-facing chat applications), disaggregation with at least a 2:1 prefill-to-decoding instance ratio is likely cost-optimal. The open-sourced trace enables practitioners to simulate their own workload against Mooncake's architecture before committing to deployment.

Overloaded inference services requiring graceful degradation. For rapidly-growing services where GPU supply cannot keep pace with demand (Section 2), Mooncake's prediction-based early rejection provides a mechanism for maximizing goodput — completed requests that meet SLOs — rather than raw throughput. The paper demonstrates a 14.2% reduction in wasted prefill computation compared to naive rejection (Table 3). The practical deployment pattern: deploy Mooncake with a conservative accept/reject threshold based on predicted decoding load, monitor the fraction of requests that are accepted but later rejected at the decoding stage (a direct measure of prediction accuracy), and adjust the prediction horizon or the uniform $t_d$ parameter to balance throughput against waste. The key operational metric is the ratio of successfully completed requests to total GPU-hours consumed; Mooncake's early rejection directly improves this ratio by avoiding prefill work on requests that cannot be completed.

Batched offline inference with relaxed latency SLOs. Section 5.2 speculates about using freed prefill VRAM for "batch-oriented offloading tasks" — requests with 24-hour turnaround where decoding can be inlined into prefill batches for better MFU. This is directly applicable to use cases like overnight document processing, bulk evaluation, or training data generation. The practical architecture: deploy a Mooncake cluster where part of the prefill capacity is reserved for batch requests. When the prefill instance has spare VRAM (because layer-wise prefill minimizes KVCache residence time), it can accept decoding work for batch requests whose KVCache fits in the remaining VRAM. This improves overall GPU utilization without affecting the latency of interactive requests (which are served by the dedicated decoding pool). The paper doesn't provide experimental validation of this pattern, but the architectural capability is present — the disaggregated KVCache pool and Messenger transfer infrastructure make it straightforward to route batch decoding work to prefill instances.

KVCache-aware routing in multi-tenant LLM platforms. For platforms serving multiple customers or applications with shared document corpora (e.g., a legal AI platform where multiple users query the same case law database), Mooncake's KVCache-centric scheduling and cache load balancing provide immediate benefits. The shared documents (system prompts, reference materials, common queries) generate hot KVCache blocks that the hot-spot migration heuristic automatically replicates across prefill instances, reducing average TTFT. The practical setup: deploy a shared KVCache pool across all tenants, use the hash-chained indexing (Figure 3) to automatically identify common prefixes across different users' requests, and configure the kvcache_balancing_threshold based on the expected tradeoff between cache hit rate and load balance. The paper's Figure 8 shows that this reduces TTFT by ~10× compared to load-balancing-only scheduling (6.26s vs. 60.41s average), making it a high-ROI optimization for multi-tenant deployments with shared context.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: LLM Serving Systems Weren't Built for Overloaded, Long-Context, KVCache-Heavy Workloads

Why This Problem Matters

Where Prior Approaches Fall Short

How Mooncake Positions Itself Relative to Existing Work

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

The Disaggregation Decision: Why Separate Prefill and Decoding Despite Chunked Prefill

Multi-Node Prefill: Chunked Pipeline Parallelism (CPP)

Layer-Wise Prefill: Overlapping KVCache Transfer with Computation

The KVCache Storage and Transfer Infrastructure

KVCache-Centric Scheduling Algorithm (Algorithm 1)

Cache Load Balancing: Heuristic-Based Hot-Spot Migration

Overload-Oriented Scheduling: Early Rejection and Load Prediction

Summary of Design Choices and Their Justifications

4. Key Insights and Innovations

Innovation 1: KVCache as a First-Class Scheduling Object, Not a Memory Management Detail

Innovation 2: Disaggregated KVCache Pool as a "Third Tier" Resource — and Why It Enables the Architecture

Innovation 3: Diagnosing and Solving the Load Oscillation Problem — Overload Scheduling as a Distinct Regime

Innovation 4: A Production-Grade Architecture That Reconciles Academic Optimizations with Operational Realities

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

End-to-End Throughput on Public Datasets (ArXiv Summarization and L-Eval)

End-to-End Throughput on Simulated Long-Context Data

Real Workload Replay

Scheduling Algorithm Ablation (Cache-Aware vs. Load-Balancing vs. Random)

Overload-Oriented Scheduling: Early Rejection and Load Prediction

Ablation Studies and Robustness Checks

Critical Assessment

Claim 1: "Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs"

Claim 2: "Under real workloads, Mooncake's innovative architecture enables Kimi to handle 75% more requests"

Claim 3: "A prediction-based early rejection policy... reduces wasted computational resources in overloaded scenarios"

Claim 4: "KVCache-centric scheduling balances maximizing overall effective throughput while meeting latency-related SLOs"

Missing Experiments That Would Strengthen the Paper

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost Is Not Accounted For, Making the ~4× Efficiency Claim an Upper Bound

Limitation 2: Test-Time Compute Cannot Help on Hard Problems — a Hard Capability Boundary

Limitation 3: Single Benchmark, Single Model Family — Unknown Generalizability

Limitation 4: Verifier Over-Optimization Is a Hard Ceiling, and Improving Search Algorithms Makes It Worse

Limitation 5: Revisions and Search Are Never Combined — an Artificial Separation

Limitation 6: The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate — a Fundamental Fragility

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

PowerInfer-2: Fast Large Language Model Inference on a Smartphone

Layer-Condensed KV Cache for Efficient Inference of Large Language Models