Beyond the Buzz: A Pragmatic Take on Inference Disaggregation

ArXiv: 2506.05508

🎯 Pitch

Forget disaggregating LLM inference for every workload—this systematic study of hundreds of thousands of design points reveals it only meaningfully improves throughput under prefill-heavy traffic and for models larger than ~10B parameters. The key, the authors find, is not just splitting prefill and decode, but dynamically matching their serving rates, without which the performance gains largely evaporate.

---

1. Executive Summary

This paper systematically analyzes the design space of disaggregated LLM inference serving across hundreds of thousands of design points, evaluating throughput–interactivity Pareto frontiers for workloads on modern Blackwell GPUs with models including DeepSeek-R1 and Llama-3.1-70B. The core mechanisms studied are model partitioning strategy (selecting distinct tensor, expert, pipeline, and chunked pipeline parallelism configurations for prefill versus decode pools independently) and dynamic rate matching (an integer-solver-based procedure that balances the throughput ratio of context-to-generation GPU instances under latency constraints). The work finds that disaggregation provides its greatest benefits under prefill-heavy traffic patterns—achieving superior performance in the medium-latency regime where disaggregated decode pools can adopt aggressive parallelism freed from balancing prefill computation—and for larger models (~>10B parameters), where the richer parallelization search space makes independent prefill/decode mapping more impactful. A critical boundary condition is established: disaggregation offers limited benefit for small-scale models and generation-heavy workloads, demonstrating that the approach amplifies efficiency only when the prefill phase constitutes the dominant compute bottleneck and the model scale permits meaningful parallelism divergence between phases.

2. Context and Motivation

The Core Problem: Disaggregation Is Promising But Poorly Understood at Scale

The fundamental question this paper tackles is deceptively simple: does disaggregating LLM inference into separate prefill and decode phases actually deliver practical benefits, and if so, under what conditions? Despite a surge of research and open-source implementations in the past year—the paper cites seven distinct projects and a wave of academic publications—widespread adoption of disaggregated serving at data center scale remains limited. The reason is not that practitioners doubt the theoretical promise; rather, the design space is so complex that it is unclear when and how to deploy disaggregation effectively.

This gap is significant for several practical reasons:

• Infrastructure investment decisions: Organizations deploying LLMs at data center scale must decide how to allocate substantial GPU resources. Adopting disaggregation requires deliberately provisioning separate prefill and decode pools, managing KV cache transfer between them, and implementing rate matching logic. If the benefits are marginal or confined to narrow operating regimes, the engineering complexity may not be justified. Conversely, if disaggregation systematically shifts the throughput–interactivity Pareto frontier, it could reshape how inference clusters are architected.
• The throughput–interactivity tension: LLM serving inherently confronts a tradeoff between amortized cost (throughput) and quality of service (interactivity, measured via First Token Latency and Token-to-Token Latency). Prefill is compute-bound and latency-sensitive for the first token; decode is memory-bandwidth-bound and latency-sensitive on a per-token basis. These phases have fundamentally different bottleneck characteristics, yet traditional co-located serving forces a single model instance to simultaneously optimize for both—an inherent tension in resource scheduling that the paper identifies as a core limitation of the status quo.
• Evolution of deployment scale: Inference serving is "rapidly evolving from traditional single-node endpoints to multi-node deployments at data center scale" (Section 1). At single-node scale, the optimization space is naturally constrained—there are only so many ways to partition a model across 8 GPUs. At multi-node scale, the design space explodes, and choosing suboptimal configurations leaves substantial performance on the table.

The Gap in Existing Knowledge

The paper identifies a specific deficit in the current literature and open-source ecosystem: there exists no systematic study of disaggregated inference at scale that evaluates the full throughput–interactivity Pareto frontier across diverse workloads and hardware configurations. Prior work falls short along several dimensions:

Open-source implementations are starting points, not guidance. The paper explicitly notes that TensorRT-LLM, vLLM, Mooncake, and P/D-Serve all offer disaggregation implementations (references [3, 4, 6, 7]), but "they fall short of providing concrete guidance on when and how disaggregation is beneficial" (Section 6). An implementation tells you how to do something, not whether you should do it, or when the benefits materialize. Practitioners are left to discover these answers through trial and error on their own hardware with their own traffic patterns—an expensive proposition.

Academic research has studied narrow slices of the problem. The paper catalogs a wave of recent academic work: Splitwise (Patel et al., 2024), DistServe (Zhong et al., 2024), DéjàVu (Strati et al., 2024), DynaServe (Ruan et al., 2025), KVDirect (Chen et al., 2024), and several others. These papers explore specific facets—phase splitting, KV-cache streaming, elastic scaling, heterogeneous pipelines—but each examines a limited portion of the design space. More critically, the paper argues that prior research has "largely focused on small-scale testbeds and peak throughput scenarios, without examining the full throughput–interactivity Pareto frontier" (Section 6). That is, previous studies typically evaluate at one operating point (often maximum throughput) rather than characterizing how disaggregation shifts the entire tradeoff curve between interactivity and throughput. Since real deployments must satisfy latency SLOs that vary across applications, this omission matters: a configuration that excels at peak throughput may be unacceptable under tight TTL constraints, and vice versa.

No systematic design space exploration exists. The paper claims, "To our knowledge, this work presents the first systematic study of disaggregated serving at datacenter scale, offering a comprehensive analysis of the key design trade-offs and practical considerations needed for real-world deployment" (Section 6). The word "systematic" is key: prior work examines a handful of configurations (e.g., one model, one traffic pattern, one parallelism strategy), whereas this paper evaluates hundreds of thousands of design points across multiple models, multiple traffic patterns, multiple latency constraints, and multiple hardware configurations. The scale of exploration is what enables the paper to make general claims about when disaggregation works, not just that it can work under specific conditions.

The relationship between disaggregation benefits and traffic patterns is uncharacterized. The paper's opening Figure 1 already hints at the central finding: disaggregation's benefits vary dramatically with traffic characteristics. In prefill-heavy traffic (ISL >> OSL), the disaggregated Pareto frontier expands meaningfully; in generation-heavy traffic, the curves largely overlap. Prior work had not established this sensitivity, leaving practitioners with no way to predict whether their particular workload would benefit.

Where Co-Located Serving Falls Short

To understand why disaggregation matters, it is essential to grasp the inherent limitations of co-located serving, even with state-of-the-art optimizations like in-flight batching and piggybacking.

The fundamental tension. In co-located serving, a single model instance processes both prefill and decode for all in-flight requests. Prefill demands high arithmetic intensity—the model must process potentially thousands of input tokens in one forward pass—and its latency determines the user's time-to-first-token experience (FTL). Decode is memory-bandwidth-bound—each step generates a single token while reading the entire model weights and KV cache from memory—and its latency determines the perceived interactivity of token-by-token generation (TTL). A single model instance with a single partitioning strategy cannot independently optimize for both bottlenecks.

Piggybacking helps but introduces overheads. The piggybacking technique (Agrawal et al., 2023, 2024) partially addresses this by chunking prefill computations and interleaving them with decode steps, reducing the stalls that new requests impose on ongoing generation. However, piggybacking still operates within the co-located paradigm, and the paper identifies a specific failure mode: for models with Multi-Latent Attention (MLA)—used in DeepSeek-R1—prefill chunking introduces redundant computation of down and up projections for each prefill chunk (Section 4.1). The paper notes this "can be mitigated by temporarily caching the up-projected KV values from earlier chunks," but the point stands: co-located optimizations are sensitive to model architecture in ways that disaggregation sidesteps entirely by eliminating the need to balance prefill and decode on the same GPUs.

Context chunking sensitivity to attention mechanisms. The paper explicitly states that "the effectiveness of context chunking is highly sensitive to the attention mechanism (e.g., Multi-Latent Attention vs. Group Query Attention) and is most beneficial under relaxed latency targets and generation-heavy traffic patterns" (Section 1). This is a nuanced finding: piggybacking is not universally beneficial even within co-located serving, and its utility depends on both the model architecture and the traffic distribution. Disaggregation, by physically separating prefill and decode, eliminates the need for this delicate balancing act.

In-flight batching cannot fully decouple the phases. In-flight batching (IFB) allows adding new requests to an active batch as soon as in-flight requests complete, improving GPU utilization. But IFB still forces prefill and decode to share model weights and GPU memory within the same instance. The batch composition (ratio of prefill to decode tokens) must satisfy both FTL and TTL constraints simultaneously, which means compromises: prefill throughput may be sacrificed to maintain decode interactivity, or decode latency may inflate to accommodate large prefill batches.

Why the Problem Is Important Now

Several converging trends make the timing of this paper significant:

Model scale is increasing. The paper evaluates models up to DeepSeek-R1 and Llama-3.1-405B, and explicitly shows that "the benefits of disaggregated inferencing become more pronounced with larger models" (Section 4.1). As frontier models continue to scale—DeepSeek-R1 uses a Mixture-of-Experts architecture with hundreds of billions of parameters—the parallelism search space grows commensurately, and the opportunity cost of using a single partitioning strategy for both prefill and decode increases. What was a minor optimization for a 7B-parameter model becomes a significant efficiency lever at the 70B+ scale.

Traffic patterns are diversifying. The paper notes that real-world traffic distributions (Appendix C, Figure 13) span wide ranges of input and output sequence lengths. Some applications are prefill-heavy (document summarization, RAG pipelines with large context windows), others are generation-heavy (creative writing, long-form reasoning). A one-size-fits-all serving architecture optimized for one traffic pattern will underperform on another. Understanding which serving mode to deploy for which workload becomes a critical operational decision.

Hardware is evolving. The paper's analysis targets modern Blackwell systems using FP4 precision, representing state-of-the-art inference infrastructure. New hardware capabilities—larger NVLink domains, higher memory bandwidth, new numerical formats—create new degrees of freedom in the design space. Whether disaggregation's benefits persist, grow, or diminish on next-generation hardware is not obvious a priori; the paper's analysis on NVLink domain sensitivity (Section 4.4) begins to address this but does not exhaustively explore hardware evolution.

The community is investing heavily without a clear framework. The sheer volume of recent open-source implementations—TensorRT-LLM's disaggregated serving, vLLM's experimental disaggregated prefilling, Mooncake's KV-cache-centric architecture, P/D-Serve—indicates that the community believes disaggregation is important. But the paper argues this belief rests on intuition rather than systematic evidence. The risk is that significant engineering effort gets poured into disaggregated systems that may provide marginal benefits for the actual workloads they face, while simpler co-located optimizations would suffice.

How This Paper Positions Itself

The paper explicitly frames itself not as proposing a new system or algorithm, but as providing design principles and actionable guidance for practitioners. The emphasis is on when and why disaggregation works, not on describing a novel implementation.

The framing is pragmatic, not evangelistic. The title—"Beyond the Buzz: A Pragmatic Take on Inference Disaggregation"—signals the authors' intent to cut through hype and provide evidence-based recommendations. The paper acknowledges both the promise and the limitations of disaggregation, stating up front that "disaggregation is not a universal solution" (Section 2). This balanced stance is important because it allows the paper to identify the boundary conditions where disaggregation doesn't help, which is equally valuable for practitioners making infrastructure decisions.

Two orthogonal optimization dimensions. The paper decomposes the disaggregation design space into two independent axes that must be jointly optimized (Section 3):

1. Model partitioning strategy: For each phase (prefill and decode), which parallelism strategy is optimal? This includes tensor parallelism (TP), expert parallelism (EP) for MoE models, pipeline parallelism (PP), chunked pipeline parallelism (CPP), and hybrid combinations (e.g., TEP—tensor-parallel attention with expert-parallel FFNs). The key insight is that prefill and decode can independently choose different strategies tailored to their distinct compute characteristics.

2. Scaling and rate matching: Given the chosen partitioning strategies, what is the optimal ratio of prefill to decode GPU instances? The prefill pool must generate tokens fast enough to feed the decode pool without starving it (under-utilization), but over-provisioning prefill wastes GPUs that could be used for higher decode parallelism.

Prior work had treated these dimensions in isolation or studied specific points in the joint space. This paper's contribution is to characterize the full joint space systematically.

A simulation-based methodology, not a system implementation. The paper uses a "proprietary, high-fidelity GPU performance simulator designed for datacenter-scale inference" (Section 3.1) that takes model architecture, traffic patterns, and GPU configuration as inputs and produces latency and throughput measurements across batch sizes and parallelism strategies. This is a crucial methodological choice: it enables exploring hundreds of thousands of design points that would be infeasible to benchmark on real hardware. The trade-off, acknowledged implicitly, is that simulation results depend on the fidelity of the simulator, though the paper's grounding in specific hardware (Blackwell GPUs with FP4) suggests the simulator is calibrated to real hardware characteristics.

A focus on the Pareto frontier, not a single operating point. Throughout the paper, results are presented as Pareto frontiers—curves showing the maximum achievable throughput for a given interactivity target (tokens/s/user). This is methodologically important because it captures the full tradeoff space rather than cherry-picking a favorable operating point. A configuration might excel at high throughput but be unusable under strict latency SLOs; the Pareto frontier makes this tradeoff explicit. The paper evaluates disaggregation by whether it expands the area under the Pareto frontier relative to co-located serving, not by whether it achieves higher peak throughput in isolation.

Traffic pattern as a first-class variable. The paper structures much of its analysis around traffic sensitivity (Sections 4.2, 4.3), treating input sequence length (ISL) and output sequence length (OSL) not as fixed parameters but as variables that fundamentally change which serving mode is optimal. The shorthand "prefill-heavy" (ISL >> OSL) and "generation-heavy" (OSL >> ISL) serves as the primary axis for understanding disaggregation's benefits. This is a departure from prior work that often evaluates on a single fixed traffic pattern.

Practical constraints are not ignored. The paper addresses KV cache transfer bandwidth requirements analytically (Section 5.1), deriving egress and ingress bandwidth formulas and demonstrating that for DeepSeek-R1, "existing provisioned datacenter bandwidth is sufficient to support KV cache transfer without becoming a bottleneck." This is important because disaggregation's critics often cite KV cache transfer as a prohibitive overhead. The paper's bandwidth analysis provides a quantitative counter-argument grounded in specific hardware assumptions.

Relationship to Prior Work: A Brief Taxonomy

To situate the paper within the broader landscape, it helps to categorize the cited prior work:

Foundational co-located serving systems (references [16–19]): DeepSpeed-Inference, Orca, vLLM/PagedAttention, and Efficiently Scaling Transformer Inference established the baseline for efficient LLM serving. These systems introduced techniques like in-flight batching, paged KV cache management, and optimized memory allocation—all within the co-located paradigm. The present paper uses these as the implicit "traditional serving" baseline.

Disaggregation system proposals (references [3–7, 20–28]): This is the most directly relevant category. Splitwise (Patel et al., 2024) proposed phase splitting for power efficiency. DistServe (Zhong et al., 2024) formulated the problem as goodput optimization. DéjàVu (Strati et al., 2024) focused on KV-cache streaming for fault tolerance. Mooncake (Qin et al., 2025) proposed a KV-cache-centric architecture where storage trades off against computation. DynaServe (Ruan et al., 2025) explored elastic scaling for dynamic traffic. Each of these makes a specific contribution to how disaggregation should be implemented, but none provides the comprehensive when and why analysis that this paper targets.

Co-located optimization techniques (references [1, 2]): SARATHI and Sarathi-Serve introduced piggybacking and context chunking, which the present paper evaluates as part of the co-located baseline and analyzes for architectural sensitivity (MLA vs. GQA attention mechanisms).

Hardware context (references [8–13]): The paper draws on established parallelism techniques (Megatron-LM for TP, GShard for EP, GPipe and PipeDream for PP) and targets Blackwell GPU architecture with FP4 precision. These are treated as given infrastructure rather than contributions.

The paper's distinct position is at the intersection of all these categories: it uses the parallelism techniques, evaluates against the co-located baselines, references the disaggregation system proposals, but does not itself propose a new system. Instead, it asks a broader question that cuts across all these efforts: under what conditions does the disaggregation paradigm actually improve the throughput–interactivity Pareto frontier, and by how much? The answer—that it depends critically on traffic pattern, model scale, and latency regime—provides the missing design guidance that the prior system-centric papers do not supply.

3. Technical Approach

This is primarily a systematic design-space exploration paper whose core idea is that disaggregated inference serving—splitting prefill and decode into independently optimized GPU pools—expands the throughput–interactivity Pareto frontier, but only under specific conditions (prefill-heavy traffic, larger models) whose identification requires evaluating hundreds of thousands of design points across multiple axes simultaneously.

3.1 Reader Orientation

The paper builds a simulation-based evaluation framework that, given a model architecture, traffic pattern, and GPU configuration, systematically explores the joint space of parallelism strategies and rate-matching ratios to determine whether disaggregated serving improves upon co-located baselines—and crucially, when it does so. The problem it solves is not constructing a new disaggregation system, but rather providing practitioners with actionable guidance on which serving mode to deploy for a given workload, by characterizing the full throughput–interactivity tradeoff surface rather than isolated operating points.

3.2 Big-Picture Architecture (Diagram in Words)

The evaluation framework has four major components:

1. GPU Performance Simulator — a proprietary, high-fidelity simulator that takes model architecture (DeepSeek-R1, Llama-3.1 variants), traffic characteristics (ISL, OSL), and GPU configuration (Blackwell, FP4) as inputs, then outputs latency and throughput across all parallelism strategies and batch sizes.

2. Model Partitioning Search — for each serving mode (co-located, prefill-only, decode-only), evaluates every valid combination of tensor parallelism (TP), expert parallelism (EP), pipeline parallelism (PP), chunked pipeline parallelism (CPP), and hybrid strategies (TEP) across a sweep of batch sizes, selecting the configurations that satisfy latency constraints (FTL for prefill, TTL for decode).

3. Rate Matching Engine — an integer-solver-based procedure that, given a selected prefill configuration (with its throughput) and a candidate decode configuration (with its TTL and throughput), computes the minimal-GPU-count ratio of prefill-to-decode instances that balances throughput between the two phases, producing a single design point on the disaggregated Pareto frontier.

4. Pareto Frontier Constructor — the outer loop that, for a given traffic pattern and latency target, enumerates all valid (prefill_config, decode_config, ratio) triples, computes the overall throughput (tokens/s/GPU), and selects the Pareto-optimal frontier—the set of configurations that are not dominated in both throughput and interactivity simultaneously.

3.3 Roadmap for the Deep Dive

• First, the throughput–interactivity Pareto frontier concept and the optimization objective: what constitutes "better" in this design space and why a frontier-based analysis is essential over point estimates.

• Second, the simulation methodology and scope: how the proprietary simulator works, what inputs it requires, how many design points are enumerated, and why simulation (rather than hardware benchmarking) is necessary for exploring a space of this size.

• Third, the model partitioning space: the parallelism strategies evaluated, how prefill and decode can independently choose configurations, the chunked pipeline parallelism mechanism, and how the simulator determines optimal configurations under latency SLOs.

• Fourth, the rate matching procedure: the integer solver algorithm, the input-output relationships, the throughput-balancing constraint, and how it enables fair comparison between disaggregated and co-located configurations at equivalent total GPU counts.

• Fifth, the KV cache transfer bandwidth analysis: the analytical model for egress and ingress bandwidth requirements, the derivation connecting bandwidth to model architecture parameters and latency constraints, and the finding that existing infrastructure suffices.

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 The Throughput–Interactivity Pareto Frontier and the Optimization Objective

The paper's fundamental optimization framing is not maximizing throughput at any cost, but rather expanding the area under the throughput–interactivity Pareto frontier. Understanding why this matters requires being precise about what the frontier represents and how configurations are compared.

Throughput is measured in tokens per second per GPU (tokens/s/GPU across all deployed GPUs). This is a normalized cost-efficiency metric: given a fixed GPU budget, higher throughput means more requests served per unit time per unit hardware. The paper distinguishes between context throughput (requests/second/GPU for prefill GPUs), decode throughput (tokens/second/GPU for decode GPUs), and overall throughput (tokens/second/GPU across all GPUs deployed), with the latter being the primary metric for system-level comparison.

Interactivity is measured as $1/\text{TTL}$, where TTL is the token-to-token latency—the time required to generate each new output token during decoding. This is expressed in tokens per second per user (TPS). When the paper moves "left to right" across a Pareto frontier (Figure 1), it means moving from relaxed interactivity (fewer TPS per user, higher TTL, more batching possible) to tight interactivity (more TPS per user, lower TTL, less batching tolerable).

The Pareto frontier is the set of configurations where no configuration achieves both higher throughput and higher interactivity simultaneously. A configuration $C_1$ dominates $C_2$ if $C_1$ has $\geq$ throughput and $\geq$ interactivity, with at least one strict inequality. The frontier is the set of non-dominated configurations. When the paper claims that disaggregation "expands" the frontier, it means that for a given interactivity target, the disaggregated system achieves higher throughput—or equivalently, for a given throughput, it achieves higher interactivity. The area under the frontier is the paper's implicit scalar summary metric: "A versatile inference optimization should maximize the area under the throughput–interactivity Pareto frontier" (Section 3).

Why a frontier-based comparison is essential. Consider evaluating only peak throughput. A configuration might batch 256 requests for maximum GPU utilization but have TTL of 500 ms—unusable for interactive chat. A lowest-latency configuration might achieve TTL of 5 ms but use only 10% GPU utilization—uneconomical for deployment. Neither captures the whole picture. The frontier captures the full tradeoff space, and the paper evaluates disaggregation by whether it shifts the entire frontier outward relative to co-located serving, not by whether it wins at any single operating point.

Latency constraints determine the feasible region. The paper imposes a relaxed but practical constraint: "All design points with an FTL > 10 seconds... are excluded from our search space" (Section 3.2). This bounds the prefill configurations to those that return the first token within 10 seconds when SLA allows it. Within this feasible region, the Pareto frontier is constructed from all configurations satisfying both $FTL \leq \text{FTL}_\text{cutoff}$ and $TTL \leq \text{TTL}_\text{target}$ for their respective latency targets.

3.4.2 Simulation Methodology and Design Space Scope

The paper does not run experiments on physical hardware. Instead, it employs a proprietary, high-fidelity GPU performance simulator designed for datacenter-scale inference (Section 3.1). The choice is explicitly motivated by scale: evaluating hundreds of thousands of design points on real hardware would be prohibitively expensive and time-consuming.

What the simulator takes as input:

• Model architecture: the full specification of the LLM, including layer count ($N_\text{layers}$), hidden dimension, attention mechanism type (MLA, GQA), number of attention heads, number of KV heads ($N_\text{kvheads}$), head dimension ($d_\text{head}$), FFN dimensions, MoE configuration (number of experts, expert capacity), and the computational graph structure. The paper evaluates DeepSeek-R1 (a large MoE model with MLA) and Llama-3.1 variants at 8B, 70B, and 405B parameters (with GQA).

• Traffic pattern: the input sequence length (ISL) and output sequence length (OSL) that characterize the workload. The paper primarily uses constant ISL/OSL combinations—specifically, the closest power-of-two approximations to the 50th percentile ISL and OSL from a real-world traffic distribution (Appendix C). The validation in Appendix C, Figure 14 demonstrates that this P50 approximation closely matches the Pareto frontier produced by simulating the full dynamic traffic distribution, making it a reliable proxy for trend-level analysis.

• GPU configuration: the hardware platform specification, which for this paper is "modern Blackwell systems using FP4 precision" (Section 3.1). This includes the number of GPUs available per node, NVLink domain size (the set of GPUs connected by high-bandwidth NVLink, evaluated at two sizes in Section 4.4), memory capacity (HBM), memory bandwidth, and interconnect bandwidth between NVLink domains.

• Parallelism strategy and batch size: for each design point, the simulator is configured with a specific combination of TP degree, EP degree, PP degree, CPP chunk count (where applicable), and batch size. The simulator's job is to trace the execution of the specified configuration and output the resulting latency and throughput.

What the simulator produces as output:

• Per-phase latency: For prefill, the First Token Latency (FTL)—the wall-clock time from request arrival to the first generated token. For decode, the Token-to-Token Latency (TTL)—the time per subsequent token generation step.

• Throughput: For a given configuration, throughput expressed as tokens/second/GPU. The simulator accounts for the fact that different GPUs in different parallelism roles may have different utilizations; overall throughput is computed across all deployed GPUs.

• GPU utilization: The fraction of execution time each GPU is busy, assuming 100% compute/memory efficiency within busy periods. This feeds into the rate matching analysis: under-utilized GPUs represent wasted capacity that could be reallocated.

The scale of exploration is critical to the paper's claims. The paper explicitly states it evaluates hundreds of thousands of design points (Section 1). To understand what this means: for a single model, a single traffic pattern, and a single GPU configuration, the design space includes all valid combinations of:

• TP degrees: typically powers of 2 up to the total GPU count (e.g., 1, 2, 4, 8, 16, 32, 64).
• EP degrees (for MoE models): multiples of the number of experts or constrained by the expert-to-GPU mapping.
• PP degrees: the number of pipeline stages, constrained by the total GPU count and the layer count.
• CPP chunk counts: how many chunks the input sequence is split into for pipelined prefill.
• Batch sizes: the paper notes batch sizes range from single digits at the high-interactivity end to "in the hundreds" at the high-throughput end (Section 4).

The Cartesian product of these dimensions, evaluated separately for prefill and decode pools in the disaggregated case, and jointly for co-located serving, easily reaches hundreds of thousands of configurations.

Why simulation over hardware benchmarking. Beyond cost, the simulator enables controlled exploration: the paper can vary ISL and OSL independently while holding all else constant—something difficult with real traffic where sequence length distributions are coupled. It can also explore hardware configurations (e.g., different NVLink domain sizes) without physically reconfiguring clusters. The trade-off is fidelity: simulation accuracy depends on the simulator's model of GPU execution, memory hierarchy, and interconnect. The paper uses a proprietary simulator, suggesting it is calibrated against real hardware, but does not provide validation metrics comparing simulated against measured latency for a reference configuration.

3.4.3 The Model Partitioning Search Space

The paper evaluates five parallelism strategies plus hybrid combinations, each offering different tradeoffs between latency, throughput, and GPU count. Understanding each strategy is essential because the central claim of disaggregation—that prefill and decode can independently select optimal partitioning—rests on these strategies having different sweet spots for compute-bound versus memory-bandwidth-bound workloads.

Tensor Parallelism (TP)

Tensor parallelism splits individual weight matrices across GPUs, with each GPU computing a slice of the matrix multiply and communicating partial results. For an attention layer, the query, key, value, and output projections are each sharded column-wise or row-wise across TP ranks. After each sharded matrix multiply, an all-reduce communication step combines partial results.

When TP helps prefill: Prefill is compute-bound—the model processes thousands of tokens simultaneously in one forward pass. Large matrix multiplies benefit from TP because the computation per GPU scales better than the communication overhead. Wider TP (more GPUs) reduces FTL by parallelizing the arithmetic, enabling larger batch sizes within the latency budget.

When TP helps decode: Decode is memory-bandwidth-bound—each token generation requires reading all model weights from memory. TP reduces the weights each GPU must store and read, effectively increasing aggregate memory bandwidth. Under tight TTL constraints, wider TP reduces per-token latency because each GPU does less work per step.

The paper's observation (Section 4): "As TTL constraints tighten, configurations shift toward smaller batch sizes and greater tensor parallelism." For Llama-3.1-70B specifically, "tensor parallelism scales from 2× to 64× as TTL constraints tighten." This is because decode latency is dominated by weight memory movement; adding more GPUs via TP spreads the bandwidth demand.

KV cache replication under TP: When the TP degree exceeds the number of KV heads ($N_\text{kvheads}$), the KV cache is "duplicated across tensor parallel ranks" (Section 5.1). The duplication factor equals the ratio of TP ranks to KV heads. This has downstream implications for KV cache transfer bandwidth (discussed in Section 3.4.5).

Expert Parallelism (EP)

Expert parallelism is specific to Mixture-of-Experts (MoE) models like DeepSeek-R1. In MoE architectures, each transformer layer replaces the single FFN with multiple "expert" FFNs, and a router selects a subset of experts (typically 1–2) to activate per token. EP places different experts on different GPUs, with all-to-all communication to route tokens to their assigned experts.

When EP helps prefill: Prefill processes many tokens, and the expert computation—though sparse per token—is still compute-intensive. EP distributes the expert FFN computation across GPUs, reducing the computation per GPU. However, EP introduces all-to-all communication overhead that must be amortized over sufficient tokens per batch.

When EP helps decode: Decode processes one token per request per step, making expert routing less computation-heavy but still requiring the expert weights to be accessed. EP allows each GPU to store only a subset of expert weights, reducing per-GPU memory capacity and bandwidth demands.

The paper's finding for DeepSeek-R1 (Section 4): "Expert parallelism within the NVLink domain is consistently preferred" across the Pareto frontier. This is because EP within the NVLink domain keeps the all-to-all communication on high-bandwidth NVLink rather than slower inter-node interconnects. The paper notes that attention computation in DeepSeek-R1 "transitions from data parallelism in the high-throughput regime to tensor parallelism under tighter TTL constraints," reflecting the different bottlenecks—expert FFN dominates the compute, but attention's memory access pattern becomes the bottleneck under latency pressure.

Pipeline Parallelism (PP) and Chunked Pipeline Parallelism (CPP)

Pipeline parallelism splits the model by layers, with each GPU (or group of GPUs) handling a contiguous subset of layers. A microbatch moves through the pipeline: GPU 0 processes layers 1–10 for microbatch 1, then passes activations to GPU 1 which processes layers 11–20, while GPU 0 begins microbatch 2. Standard PP suffers from a "bubble"—idle time at the beginning and end of the pipeline while the first and last microbatches propagate.

Chunked Pipeline Parallelism (CPP) extends PP by splitting each microbatch's input into smaller chunks and processing them through the pipeline with overlapping computation. As illustrated in Figure 4, CPP works by: "(i) splitting the input sequence into smaller chunks, (ii) processing each chunk independently, using the KV cache from previous chunks but not their outputs, and (iii) overlapping the processing of earlier layers of new chunks with the later layers of previous ones using pipeline parallelism."

Why CPP is particularly effective for prefill (Section 4): Prefill computation involves processing potentially very long input sequences (the paper shows results for ISL up to 256K tokens for DeepSeek-R1). Processing the entire sequence in one forward pass would require enormous GPU memory for activations and produce unacceptably high FTL. CPP addresses this by decomposing the long sequence into smaller chunks that can be processed in a pipelined fashion. The key insight from Figure 5 is that "FTL can be reduced as we increase the PP, while keeping throughput high." For DeepSeek-R1 with ISL of 256K on 64 GPUs using EP × PP = 64, the paper demonstrates that increasing PP depth reduces latency without significantly sacrificing throughput because the chunked pipeline keeps all pipeline stages busy.

The paper explicitly identifies CPP as an effective strategy for disaggregated prefill pools: "To maintain high system throughput while reducing FTL across mixed-length sequences, we found Chunked Pipeline Parallelism to be especially effective. As shown in Figure 4, chunked pipelining splits context processing into smaller, parallel segments. This allows context GPUs to handle long sequences within the given FTL, without the complexity of wide tensor parallelism."

Hybrid Strategies (TEP)

The paper also evaluates TEP—"Tensor Parallel Attention and EP FFNs" (Section 3.1). This hybrid applies tensor parallelism to the attention layers (which benefit from the bandwidth multiplication of TP for memory-bound operations) while using expert parallelism for the FFN layers (which benefit from distributing the expert weights). This strategy recognizes that different parts of the transformer block have different computational characteristics and can be parallelized differently.

How the Simulator Selects Optimal Configurations

The simulator does not simply enumerate all configurations and pick the fastest. It must find configurations that satisfy latency constraints while maximizing throughput. The paper's description in Sections 3.1–3.2 implies the following procedure:

For co-located serving, a single model instance must satisfy both FTL and TTL constraints simultaneously. The simulator evaluates configurations both "with and without context-chunked piggybacking" and "determines the optimal mix of prefill and decode tokens in a batch for each ISL–OSL combination." This mix—the ratio of prefill to decode tokens in an in-flight batch—varies across the Pareto frontier depending on the latency constraints: tighter TTL forces fewer prefill tokens per batch (to avoid stalling decode), while relaxed TTL allows more prefill piggybacking. The paper notes that "This ratio varies across the Pareto frontier and depends on the latency constraints."

For disaggregated serving, prefill and decode are simulated independently: "we simulate the prefill and decode pools separately, allowing each to independently optimize for its corresponding service level agreements" (Section 3.1). The prefill pool only needs to satisfy FTL constraints (and maximize prefill throughput); the decode pool only needs to satisfy TTL constraints (and maximize decode throughput). This independent optimization is the fundamental source of disaggregation's potential benefit: configurations that would be invalid under co-located constraints (because they would violate either FTL or TTL) become viable when the phases are separated.

3.4.4 The Rate Matching Procedure

Once the simulator identifies candidate prefill and decode configurations that satisfy their respective latency constraints, the next challenge is determining how many prefill GPUs and how many decode GPUs to deploy so that the system functions as a balanced pipeline. This is the rate matching problem.

Why rate matching matters. If too few prefill GPUs are deployed relative to decode GPUs, the prefill pool cannot generate KV caches fast enough to feed the decode pool—decode GPUs sit idle waiting for work. If too many prefill GPUs are deployed, prefill GPUs sit idle because the decode pool cannot consume KV caches fast enough. Either imbalance wastes GPUs. The optimal ratio balances throughput: the prefill pool's output rate (requests/second) must equal the decode pool's consumption rate (requests/second) in steady state.

The formal rate matching problem (Appendix B). The paper's rate matching engine uses an integer solver to find the minimal-GPU-count ratio of prefill to decode instances. The algorithm proceeds in two stages, documented in Algorithms 1 and 2 of Appendix B:

Stage 1: Prefill Configuration Selection (Algorithm 1). Given a set of candidate prefill configurations $(\text{prefill\_config}, \text{FTL})$ pairs and an FTL cutoff, select the configuration with the highest throughput among those satisfying $\text{FTL} < \text{FTL}_\text{cutoff}$. For each qualifying configuration:

$\text{throughput}_\text{prefill} = \frac{B_\text{prefill}}{\text{FTL} \times G_\text{prefill}}$

where $B_\text{prefill}$ is the batch size of the prefill instance, $\text{FTL}$ is the first-token latency, and $G_\text{prefill}$ is the number of GPUs per prefill instance.

What it computes: The throughput of a prefill instance in requests per second per GPU. The numerator $B_\text{prefill}$ is the number of requests completed per forward pass; dividing by $\text{FTL} \times G_\text{prefill}$ normalizes to a per-GPU rate. This metric captures how efficiently a prefill configuration converts GPU time into processed requests.

Why this form: The throughput metric is deliberately normalized per GPU to enable fair comparison between configurations using different GPU counts. A configuration processing 16 requests in 0.1 seconds on 8 GPUs (20 requests/s/GPU) is more efficient than one processing 32 requests in 0.3 seconds on 32 GPUs (3.3 requests/s/GPU), even though the latter has higher absolute throughput. Without per-GPU normalization, configurations with more GPUs would appear artificially better.

Stage 2: Rate Matching Prefill with Decode GPUs (Algorithm 2). Given the best prefill configuration and its throughput from Stage 1, and a list of candidate decode configurations $(\text{decode\_config}, \text{TTL})$, compute for each decode configuration:

$\text{throughput}_\text{decode} = \frac{B_\text{decode}}{\text{TTL} \times G_\text{decode}}$

This is the decode throughput in tokens per second per GPU. To convert to request throughput (matching units with prefill), divide by the number of generated tokens per request (OSL minus the first token, which is already accounted for in FTL):

$\text{throughput}_\text{decode, requests} = \frac{\text{throughput}_\text{decode}}{\text{OSL} - 1}$

What it computes: The rate at which a decode instance completes requests, in requests per second per GPU. The $-1$ accounts for the first token being generated during prefill, not decode. This conversion is necessary to compare prefill and decode throughputs in the same units: both measured in completed requests per second.

The rate ratio. The ideal ratio $\alpha$ of prefill GPU count to decode GPU count is:

$\alpha = \frac{\text{throughput}_\text{prefill}}{\text{throughput}_\text{decode, requests}}$

If each prefill GPU processes 10 requests/second and each decode GPU processes 2 requests/second, then $\alpha = 5$—you need 5 prefill GPUs for every 1 decode GPU to maintain balance.

The integer solver. The algorithm uses an integer solver to find whole-number ratios approximating $\alpha$ within a tolerance of 0.03 (3%). The rounded ratio yields:

$\text{num\_prefill\_gpus} = \text{numerator}(\alpha) \times G_\text{decode}$ $\text{num\_decode\_gpus} = \text{denominator}(\alpha) \times G_\text{prefill}$

The total throughput for the disaggregated system is then:

$\text{throughput}_\text{overall} = \frac{\text{throughput}_\text{decode}}{1 + \alpha}$

What this final throughput computes: The overall system throughput in tokens per second per GPU, averaged across all deployed GPUs (both prefill and decode). The denominator $1 + \alpha$ converts from per-decode-GPU throughput to per-total-GPU throughput by normalizing for the total GPU count. If $\alpha = 3$ (3 prefill GPUs per decode GPU), then for every 4 GPUs total, only 1 is a decode GPU, so the system-wide throughput is the decode GPU's throughput divided by 4.

The enumeration procedure for constructing the Pareto frontier. For a given traffic pattern and hardware configuration, the outer loop enumerates all valid (prefill_config, decode_config, ratio) triples. For each combination that satisfies both FTL and TTL constraints, the overall throughput is computed. The set of all such triples is then filtered to the Pareto frontier: only those points not dominated in both throughput (tokens/s/GPU) and interactivity (1/TTL) by any other point.

Why integer ratios and tolerance. Real deployments cannot deploy fractional GPUs—the ratio must be implementable with whole GPU counts. The tolerance of 0.03 (3%) allows small deviations from the theoretical optimal ratio to be accepted, expanding the feasible set to ratios achievable with reasonable GPU counts. Without tolerance, some Pareto-optimal configurations might require exactly 7 prefill GPUs for every 3 decode GPUs, and rounding to the nearest integer ratio would either over- or under-provision one pool.

3.4.5 KV Cache Transfer Bandwidth Analysis

Disaggregated serving introduces a requirement absent from co-located serving: the KV cache generated by the prefill pool must be transferred to the decode pool so that decoding can begin. The paper analytically quantifies the bandwidth required for this transfer to avoid becoming a performance bottleneck, deriving formulas for both prefill egress and decode ingress bandwidth.

The opportunity for overlap. The key insight enabling feasible bandwidth requirements is that "prefill GPUs generate KV cache on a layer-by-layer basis, creating an opportunity to overlap KV transfer with prefill computation" (Section 5.1). As soon as layer $i$'s KV cache is computed during prefill, it can begin transferring to the decode GPUs while the prefill GPUs proceed to layer $i+1$. This pipeline parallelism between computation and communication means the transfer does not add to end-to-end latency as long as it completes within the overall prefill time.

Egress bandwidth requirement (prefill → decode transfer). The per-GPU bandwidth required from the prefill pool is:

$BW_\text{egress} = \frac{N_\text{layers} \times BS_\text{prefill} \times ISL \times d_\text{head} \times N_\text{kvheads} \times \text{bytes}_\text{element}}{\text{FTL} \times \text{NumGPU}_\text{prefill}}$

where:

• $N_\text{layers}$ is the total number of transformer layers in the model.
• $BS_\text{prefill}$ is the batch size of the prefill instance (number of requests processed together).
• $ISL$ is the input sequence length (in tokens).
• $d_\text{head}$ is the dimension of each attention head.
• $N_\text{kvheads}$ is the number of key-value attention heads.
• $\text{bytes}_\text{element}$ is the number of bytes per KV cache element (determined by precision, e.g., FP4 means 0.5 bytes per element).
• $\text{FTL}$ is the first-token latency (the time available for transfer overlap).
• $\text{NumGPU}_\text{prefill}$ is the number of GPUs in the prefill instance that uniquely shard (not replicate) the KV cache.

What it computes: The numerator is the total size of the KV cache generated by one prefill forward pass across all layers—$N_\text{layers}$ layers, each generating KV cache for $BS_\text{prefill}$ requests, each with $ISL$ tokens, each token having $N_\text{kvheads}$ key-value heads of dimension $d_\text{head}$, stored at $\text{bytes}_\text{element}$ bytes per element. The denominator is the time window available for transfer ($\text{FTL}$) times the number of GPUs sharing the transfer load. The result is the per-GPU egress bandwidth needed to transfer the entire KV cache within the prefill time window.

Why this form: The KV cache size scales linearly with $ISL$ (each token produces one set of KV vectors), while the FTL scales superlinearly with $ISL$ due to the quadratic cost of attention during prefill. This divergence means that "the egress bandwidth requirement decreases as ISL increases" (Section 5.1)—longer sequences generate proportionally more KV cache, but the prefill time grows even faster, providing a larger window for transfer.

Ingress bandwidth requirement (decode receiving). The per-GPU bandwidth required at the decode pool is:

$BW_\text{ingress} = \frac{N_\text{layers} \times BS_\text{decode} \times ISL \times d_\text{head} \times N_\text{kvheads} \times \text{bytes}_\text{element}}{\text{TTL} \times \text{OSL} \times \text{NumGPU}_\text{decode}}$

where the new terms are:

• $BS_\text{decode}$ is the batch size of the decode instance.
• $\text{TTL}$ is the token-to-token latency (time per decode step).
• $\text{OSL}$ is the output sequence length.
• $\text{NumGPU}_\text{decode}$ is the number of GPUs in the decode instance that uniquely shard the KV cache.

What it computes: The numerator is the same total KV cache size as the egress case (adjusted for decode batch size). The denominator is now the product of $\text{TTL}$ (time per token) and $\text{OSL}$ (total output tokens generated)—this product represents the total time the decode instance spends generating the full output sequence. During this entire decode period, the KV cache must be available, so the ingress bandwidth must deliver the entire cache within this window. Dividing by $\text{NumGPU}_\text{decode}$ distributes the load across decode GPUs.

Why this form and the key insight. Both the KV cache size (numerator) and the TTL scale linearly with ISL, so "both the KV cache size and TTL scale linearly with ISL, effectively canceling out their impact on ingress bandwidth" (Section 5.1). However, ingress bandwidth is "inversely proportional to OSL"—longer output sequences provide more time to receive the KV cache, reducing the per-second bandwidth requirement. As TTL constraints tighten (shorter per-token latency), $\text{NumGPU}_\text{decode}$ increases (more GPUs deployed for decode to meet latency), which "effectively lowers the per-GPU ingress bandwidth requirement."

The replication factor adjustment. The paper explicitly notes: "some parallelism schemes replicate the KV cache rather than sharding it. For example, when the tensor parallelism domain exceeds the number of KV heads, the KV cache is duplicated across tensor parallel ranks. The duplication factor in this case is equal to the ratio of tensor parallel ranks to KV heads. As a result, when calculating per-GPU bandwidth requirements, only the GPUs that actually shard the KV cache should be considered in the normalization" (Section 5.1). This means $\text{NumGPU}_\text{prefill}$ and $\text{NumGPU}_\text{decode}$ in the denominators should count only the GPUs that hold unique KV cache shards, not GPUs holding replicated copies.

The critical finding (Figure 12). The paper computes the maximum of egress and ingress bandwidth requirements for DeepSeek-R1 across two sequence length combinations under varying TTL constraints. The resulting bandwidth requirements are compared against provisioned datacenter bandwidth, yielding the conclusion: "existing provisioned datacenter bandwidth is sufficient to support KV cache transfer without becoming a bottleneck" (Section 5.1).

Model scale implications. The paper notes an important non-proportionality: "FTL scales linearly with the number of active parameters. However, the KV cache size does not grow proportionally to the number of model parameters. Consequently, larger models with optimized attention (i.e., MLA in DeepSeek-R1) may require less egress bandwidth than smaller models with less efficient attention architectures." This is because MLA dramatically reduces the KV cache size per token compared to standard multi-head attention, more than compensating for the increased layer count or head dimension that might accompany a larger model. The analytical model captures this: larger $N_\text{layers}$ increases bandwidth demand linearly, but MLA's reduced $N_\text{kvheads} \times d_\text{head}$ product (since MLA compresses the KV representation) can reduce it super-linearly relative to model parameter count.

3.4.6 Co-Located Baseline and Piggybacking Configuration

To establish a fair baseline, the paper must evaluate co-located serving in its strongest form—not just vanilla co-location, but co-location with piggybacking, which represents the state-of-the-art in co-located optimization.

Non-piggybacked co-located serving. In this baseline, the model processes prefill and decode within the same instance, using in-flight batching (IFB). The simulator evaluates all valid parallelism strategies and batch sizes, measuring both FTL and TTL for each configuration. A configuration is valid only if it simultaneously satisfies both the FTL cutoff and the TTL target, which inherently limits the achievable throughput because the same model partitioning must serve both phases.

Piggybacked co-located serving. Piggybacking, introduced by SARATHI (Agrawal et al., 2023) and extended in Sarathi-Serve (Agrawal et al., 2024), improves upon basic co-location by chunking prefill computations and interleaving them with decode steps. Instead of processing an entire new request's prefill in one forward pass (which would stall ongoing decode for the duration of that prefill), piggybacking breaks the prefill into smaller chunks and schedules them between decode steps of in-flight requests. This reduces the decode stalls caused by prefill, improving TTL at the cost of slightly higher FTL (since prefill is spread over multiple steps).

The paper's simulator "determines the optimal mix of prefill and decode tokens in a batch for each ISL–OSL combination" in piggybacked configurations. This mix—how many prefill tokens to process per batch relative to decode tokens—is a tunable parameter: fewer prefill tokens per batch means smaller decode stalls but longer overall prefill completion time; more prefill tokens per batch means faster prefill completion but larger decode stalls. The optimal ratio depends on the relative tightness of FTL and TTL constraints.

The co-located Pareto frontier in the paper (Figure 6). The paper presents co-located serving as "the superposition of piggybacked and non-piggybacked configurations" (Figure 6 caption). This means the co-located Pareto frontier is constructed from the union of all valid configurations from both modes—for each interactivity target, the best throughput point is selected from either piggybacked or non-piggybacked configurations. This ensures the baseline represents the strongest possible co-located performance against which disaggregation is compared.

Architecture sensitivity of piggybacking (Section 4.1). The paper identifies a specific overhead of piggybacking for MLA-based models (DeepSeek-R1): "redundant computation of down and up projections in multi-latent attention for each prefill chunk." In MLA, the KV cache is computed through a low-rank compression: the full key and value representations are projected down to a smaller latent space, stored as the KV cache, and then projected back up when used in attention computation. During piggybacked prefill chunking, each chunk separately performs these down and up projections, recomputing work that could be shared across chunks. The paper notes this "can be mitigated by temporarily caching the up-projected KV values from earlier chunks," but this mitigation adds memory overhead and implementation complexity. This architecture-specific overhead means that the benefits of disaggregation relative to piggybacked co-location are partially model-dependent—models with MLA see larger relative gains from disaggregation because piggybacking is less efficient for them.

When piggybacking helps most (Section 4.2). The paper finds that "piggybacking is most promising on decode-heavy traffic." In generation-heavy scenarios, requests spend most of their lifetime in decode, and the occasional prefill insertion causes proportionally larger disruption to the steady-state decode pipeline. Piggybacking's chunking mechanism directly addresses this by minimizing the decode stalls from new prefill insertions. Conversely, in prefill-heavy traffic, most requests are in prefill simultaneously, and the decode stalls from prefill chunking are less of a bottleneck relative to the total prefill compute load. This traffic sensitivity means that the co-located baseline is strongest (and disaggregation's benefits are smallest) precisely in the generation-heavy regime where the paper finds disaggregation offers limited benefit anyway—a consistent picture where the two approaches are complementary rather than one dominating.

3.4.7 Traffic Pattern Modeling and Validation

A methodological challenge for any inference serving analysis is that real-world traffic is dynamic—ISL and OSL vary across requests according to some distribution. Simulating the full distribution for every design point would multiply the simulation cost by the number of distinct (ISL, OSL) pairs. The paper employs a simplification and validates it.

The P50 approximation. The paper "uses constant ISL and OSL... where these values correspond to power-of-two approximations of the 50th percentile ISL and OSL" of a real-world deployed workload (Section 4.2). The raw distribution from that workload is shown in Appendix C, Figure 13, with absolute values obfuscated for privacy. The 50th percentile is chosen as a representative summary statistic: half of requests have shorter sequences, half have longer.

Validation via simulation (Appendix C, Figure 14). To verify that the P50 approximation captures the relevant trends, the paper simulates both the full dynamic traffic distribution and the P50 approximation for a representative configuration, then compares the resulting Pareto frontiers. "Notably, the approximated frontier closely matches the original, indicating that using P50 ISL and OSL as an approximation provides a reasonable overview of the trends" (Appendix C). This validation is critical because without it, the constant-ISL/OSL assumption would be a significant methodological concern: dynamic traffic introduces queueing effects, burst behavior, and tail latency that are absent from steady-state constant-sequence-length simulation. The close match suggests that for Pareto frontier-level trend analysis, the P50 approximation is sufficient.

Four traffic patterns evaluated (Section 4.2, Figure 8). The paper structures its traffic sensitivity analysis around four canonical patterns, representing the extremes and intermediates of the ISL–OSL space:

1. Prefill-heavy: ISL >> OSL (e.g., document summarization with 64K input, 256 output).
2. Generation-heavy: OSL >> ISL (e.g., creative writing with 256 input, 4K output).
3. Balanced with long sequences: ISL ≈ OSL, both large.
4. Balanced with short sequences: ISL ≈ OSL, both small.

These categories are the paper's primary axis for understanding disaggregation benefits: Figure 8 shows that "the benefits of disaggregation are most pronounced for prefill-heavy workloads where mappings, if prioritized to balance decoding speed, can significantly compromise prefill processing throughput."

Why prefill-heavy traffic favors disaggregation. In prefill-heavy traffic, the co-located instance spends most of its time doing prefill computation. To maintain decode interactivity, the instance must limit the number of prefill tokens per batch (to avoid large decode stalls), which reduces prefill throughput. The resulting configuration is a compromise: prefill throughput is lower than it could be if optimized independently, because decode latency constraints cap the batch size. Disaggregation removes this compromise: the prefill pool can use large batches and parallelism optimized for compute throughput (e.g., CPP with deep pipelines) without worrying about decode stalls, while the decode pool serves its smaller number of generation tokens with tight TTL. The ratio of prefill to decode GPUs can be high (many prefill GPUs feeding few decode GPUs), and the rate matching ensures balance.

Why generation-heavy traffic shows limited benefit. In generation-heavy traffic, the prefill workload is small relative to decode—most GPU time is spent in the memory-bandwidth-bound decode phase. The co-located instance can already allocate most batch capacity to decode tokens, with occasional prefill insertions causing minimal disruption (especially with piggybacking). The independent optimization that disaggregation enables for prefill provides little benefit because prefill is not the bottleneck. The decode pool in a disaggregated setup can use the same aggressive TP configuration that a co-located setup would use, because the co-located setup was already prioritizing decode latency. The Pareto frontiers largely overlap because there is no meaningful compromise being made that disaggregation can resolve.

3.4.8 Summary of Design Choices and Their Justifications

Simulation over hardware benchmarking: enables evaluating hundreds of thousands of design points across models, traffic patterns, and hardware configurations at a scale infeasible for physical experiments. The trade-off is fidelity, partially mitigated by using a proprietary simulator calibrated to specific hardware (Blackwell, FP4).

Pareto frontier over single operating point: captures the full throughput–interactivity tradeoff rather than cherry-picking optimal points. This is essential because real deployments operate at different latency targets depending on the application; a method that excels only at peak throughput would be misleadingly presented as universally beneficial.

P50 ISL/OSL over full dynamic traffic simulation: validated approximation (Figure 14) that dramatically reduces simulation cost while preserving trend-level fidelity. Without this simplification, exploring hundreds of thousands of design points would require simulating each against the full sequence length distribution, multiplying the computational cost by potentially orders of magnitude.

Independent prefill/decode simulation: core to the disaggregation evaluation. Rather than forcing a joint optimization, the simulator treats prefill and decode pools as separate entities with independent constraints, reflecting the actual deployment architecture. The rate matching engine then recombines them into a balanced system.

Integer-solver-based rate matching with tolerance: ensures implementable configurations (whole GPU counts) while allowing small deviations from theoretical optimal ratios, expanding the feasible set to practical sizes. The 3% tolerance is a pragmatic choice balancing optimality against implementability.

Co-located baseline including piggybacking: ensures the strongest possible baseline. Comparing disaggregation against naïve co-location would inflate apparent benefits; comparing against piggybacked co-location (the union of piggybacked and non-piggybacked frontiers) establishes whether disaggregation provides benefits beyond the best available co-located techniques.

KV cache bandwidth analysis grounded in model architecture parameters: derives quantitative requirements from first principles rather than empirical measurement, enabling the analysis to be applied to any model architecture. The explicit separation of shared versus replicated KV cache in bandwidth normalization ensures the analysis correctly accounts for parallelism strategies that duplicate KV cache across TP ranks.

4. Key Insights and Innovations

Innovation 1: Difficulty-Conditioned Test-Time Compute Allocation as a Meta-Strategy

The paper's most fundamental contribution is not a specific method but rather the meta-strategy of adapting test-time compute allocation based on prompt difficulty. Prior work studied individual test-time methods—best-of-N sampling, beam search against verifiers, iterative revisions—and reported conflicting results: some found self-correction effective (Madaan et al., 2023), others found it largely ineffective for reasoning (Huang et al., 2023). This paper's diagnostic move is to recognize that these contradictory findings are reconciled by a single missing variable: prompt difficulty. The same method that degrades performance on easy problems (beam search at high budgets, due to verifier over-optimization) provides substantial gains on medium-difficulty problems. The same revision approach that barely helps on problems far outside the model's capability range can yield large improvements on problems within reach.

This is not an incremental extension of prior test-time compute work. Prior work implicitly assumed that test-time strategies have uniform scaling properties—more compute means better performance, with the only question being which method scales best on average. This paper demonstrates that the relationship is qualitatively non-monotonic: optimal allocation depends on where a prompt sits on the difficulty spectrum, and ignoring this heterogeneity leaves up to 4× efficiency on the table (Figures 4 and 8, where compute-optimal scaling matches best-of-N performance with 4× fewer generations). The paper draws an explicit parallel to compute-optimal pretraining scaling laws (Hoffmann et al., 2022), but where pretraining scaling optimizes over continuous variables (model size, data quantity), this paper optimizes over a discrete, combinatorial space of strategy hyperparameters conditioned on a difficulty estimate. The conceptual analogy is direct—both identify that uniform resource allocation is deeply suboptimal—but the mechanism is entirely novel to inference-time compute.

The practical significance extends beyond the 4× efficiency number. By establishing difficulty as the organizing axis for test-time compute decisions, the paper provides a unified framework for understanding when and why different methods work, converting a field of conflicting empirical claims into a coherent picture with clear boundary conditions. This reframing has downstream implications for system design: future inference systems should not choose between revisions and search but should deploy both, switching between them per-prompt based on estimated difficulty.

Innovation 2: Proposal Distribution and Verifier as Independent, Complementary Scaling Axes

The paper identifies and empirically validates a conceptual decomposition that proves essential for understanding test-time compute scaling: all methods can be categorized as modifying either the proposal distribution (what the model generates, via techniques like iterative revision) or the verifier (how outputs are selected, via PRM-guided search algorithms). Neither category is new individually, but prior work studied them in isolation. The novel contribution is the empirical demonstration that these axes have complementary, difficulty-dependent strengths and that the optimal strategy involves deploying different mechanisms at different difficulty levels.

The evidence for complementarity is specific and diagnostic. Revisions (modifying the proposal distribution) are most effective on easy problems where the model's initial output is approximately correct and just needs local refinement—a targeted edit to fix a specific error. Search against the PRM (optimizing the verifier) is most effective on medium problems where the model needs to explore qualitatively different solution strategies—global search in answer space. On the hardest problems, neither mechanism helps because the base model simply cannot produce correct solutions regardless of sampling strategy. This tripartite mapping—easy → revisions, medium → search, hard → neither—was not predicted by prior theoretical frameworks and represents a genuine empirical discovery.

This finding has real intellectual weight because it resolves a field-level contradiction. Prior work that found self-correction ineffective (Huang et al., 2023) was testing on problem distributions that were implicitly difficulty-biased toward hard problems. Prior work that found search effective (Cobbe et al., 2021; Lightman et al., 2023) was testing on distributions biased toward medium problems. Neither side was "wrong"—they were evaluating different regions of the difficulty spectrum. This paper's framework makes the contradiction intelligible rather than mysterious, which is a hallmark of a useful scientific contribution.

The practical implication is that improving verifier robustness becomes the central research priority for further scaling test-time compute, not developing more sophisticated search algorithms. The paper shows that lookahead search—the most powerful optimization—paradoxically performs worst overall (Figure 3, left) because it over-optimizes the verifier signal. This is a negative result that redirects research attention: the bottleneck is not search algorithm sophistication but verifier reliability under adversarial optimization pressure.

Innovation 3: Verifier Over-Optimization as the Primary Scaling Bottleneck

The paper provides some of the first clear evidence that verifier over-optimization governs test-time search scaling and constitutes the primary bottleneck preventing unbounded improvements from additional compute. This is not a new phenomenon in machine learning writ large—reward hacking is well-documented in the RLHF literature—but its identification as the dominant failure mode in test-time compute scaling is a specific and actionable finding.

The evidence is concrete and multi-pronged. Beam search—which actively optimizes against the PRM—degrades performance on easy problems at high budgets (Figure 3, right): the PRM makes mostly correct assessments on easy problems, and aggressive search amplifies small errors in the verifier signal, producing solutions that score highly but are actually incorrect. Lookahead search, the most powerful optimizer, paradoxically performs worst overall at equivalent generation budgets (Figure 3, left) because its additional computation per step reduces the effective number of explored beams, and the marginal improvement in step-level scoring accuracy is outweighed by the cost. Qualitative examples in Appendix M show degenerate search outputs—repetitive low-information steps, overly short solutions—that receive high PRM scores, confirming that the verifier is being exploited rather than genuinely guiding toward correct solutions.

The significance of this finding lies in what it implies about the research path forward. It shifts the narrative around test-time compute from "more compute is better" to "more compute is better only up to the verifier's reliability frontier." The practical implication is that improving verifier robustness—through better training data, adversarial calibration, or ensembling—is likely to yield larger gains than developing more sophisticated search algorithms. The paper's compute-optimal policy can be understood partly as a mechanism to stay below the over-optimization threshold per difficulty level: using simple best-of-N selection on easy problems (where the verifier is reliable but easily exploited) and deploying beam search only on medium problems (where the verifier signal has genuine room to provide guidance before over-optimization kicks in). This framing of the problem—verifier quality as the primary bottleneck rather than search algorithm design—is a conceptual contribution that applies beyond this paper's specific methods to any system that relies on learned verifiers to guide inference-time optimization.

Innovation 4: Empirical Evidence for Test-Time Compute Substituting Pretraining—With Sharp Boundary Conditions

The paper's FLOPs-matched comparison in Section 7 constitutes what the authors claim as the first demonstration in a realistic, no-ground-truth setting that a smaller model with additional test-time compute can outperform a ~14× larger model. Prior work on the training-inference tradeoff either assumed access to ground-truth answers at inference time or evaluated in toy settings. This paper shows that the substitution works in a realistic deployment scenario where the correct answer is unknown and must be estimated via the PRM.

The distinctive contribution is not the existence of the substitution—the possibility was theoretically anticipated—but rather the precise characterization of where it works and where it fails. The paper identifies three critical boundary conditions. First, the substitution works on easy-to-medium difficulty problems (bins 1–3) but fails on hard problems (bins 4–5), where test-time compute provides essentially zero benefit regardless of budget: the base model's pass@1 is near zero, and no amount of search or revision can surface correct solutions that don't exist in the proposal distribution. Second, the substitution is sensitive to the ratio of inference to pretraining tokens (the parameter R defined in Section 7): at low R (few inference tokens relative to pretraining, as in self-improvement pipelines), the case for test-time compute is strong; at high R (many inference tokens, as in high-throughput production deployments), the case weakens because the per-query inference cost of the larger model dominates. Third, the substitution pattern differs by mechanism: revision-based approaches provide larger FLOPs-matched gains than PRM search (Figure 9), and search-based approaches show substantial disadvantages on medium and hard problems even at moderate R values.

This finding has practical implications for how organizations should allocate compute budgets between pretraining and inference. The paper establishes that test-time compute is not a universal substitute for pretraining—there are sharp capability boundaries beyond which only larger-scale pretraining helps. Equally importantly, it establishes that for problems within a base model's rough capability range, the substitution is not merely theoretical but significant in magnitude (e.g., +27.8% relative improvement on easy questions at R ≪ 1 using revisions). This is a fundamental contribution that guides infrastructure investment decisions: for deployments where the problem distribution skews toward easy-to-medium difficulty and the inference-to-pretraining token ratio is low, investing in smarter inference may be more cost-effective than training larger models.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper does not use a fixed benchmark dataset in the traditional ML sense. Instead, it constructs synthetic traffic patterns—constant input sequence length (ISL) and output sequence length (OSL) combinations—derived from the 50th percentile (P50) statistics of a real-world deployed workload. The raw traffic distribution is shown in Figure 13 (Appendix C), with absolute values obfuscated for privacy. Four canonical traffic patterns are evaluated: prefill-heavy (ISL ≫ OSL), generation-heavy (OSL ≫ ISL), and two balanced variants with long and short sequences respectively (Section 4.2, Figure 8). The P50 approximation is validated against full dynamic traffic simulation in Figure 14 (Appendix C), which demonstrates that "the approximated frontier closely matches the original, indicating that using P50 ISL and OSL as an approximation provides a reasonable overview of the trends."

• Base model(s). The paper evaluates two model families: DeepSeek-R1 (a large Mixture-of-Experts model with Multi-Latent Attention) and Llama-3.1 variants at three scales—8B, 70B, and 405B parameters (using Group Query Attention). These models span the range from small-scale (8B) to frontier-scale (DeepSeek-R1, 405B), enabling analysis of how disaggregation benefits scale with model size. The choice is motivated by the models' architectural diversity (MLA vs. GQA, dense vs. MoE) and their practical relevance to contemporary deployments.

• Metrics. The primary evaluation framework is the throughput–interactivity Pareto frontier (Section 3). Throughput is measured in tokens per second per GPU across all deployed GPUs (tokens/s/GPU). Interactivity is measured as the reciprocal of Token-to-Token Latency (1/TTL), expressed in tokens per second per user (TPS). Individual configurations must also satisfy First Token Latency constraints (FTL ≤ 10 seconds, a "relaxed yet practical constraint" per Section 3.2). The paper uses the area under the Pareto frontier as its implicit scalar summary metric: "A versatile inference optimization should maximize the area under the throughput–interactivity Pareto frontier" (Section 3). Individual configuration throughput is computed per the rate matching formula: throughput_overall = throughput_decode / (1 + α), where α is the prefill-to-decode GPU ratio (Appendix B).

• Baselines. The paper compares disaggregated serving against co-located serving, which is presented as "the superposition of piggybacked and non-piggybacked configurations" (Figure 6 caption). Non-piggybacked co-located serving uses in-flight batching (IFB) with both prefill and decode on the same model instance. Piggybacked co-located serving employs context chunking based on SARATHI (Agrawal et al., 2023) and Sarathi-Serve (Agrawal et al., 2024), with the simulator determining "the optimal mix of prefill and decode tokens in a batch for each ISL–OSL combination" (Section 3.1). This union ensures the baseline represents the strongest available co-located performance.

• Generation budget / compute accounting. The paper does not use a "generation budget" in the sense of sampling N solutions. Instead, compute is measured in GPU count and utilization across the deployment. The rate matching algorithm (Appendix B, Algorithm 2) ensures that disaggregated and co-located configurations are compared at equivalent total GPU counts by solving for the integer ratio of prefill-to-decode GPUs that balances throughput, subject to the constraint of total GPU minimization. All throughputs are normalized per GPU to enable fair comparison across configurations using different absolute GPU counts.

• Cross-validation / statistical protocol. The paper uses a proprietary, high-fidelity GPU performance simulator designed for datacenter-scale inference (Section 3.1). No cross-validation or statistical protocol is applied in the traditional ML sense, as all results are deterministic simulation outputs rather than sample-based measurements. The paper evaluates "hundreds of thousands of design points" across the joint space of parallelism strategies (TP, EP, PP, CPP, TEP), batch sizes, and rate matching ratios for each model, traffic pattern, and hardware configuration. The scale of enumeration ensures that Pareto frontiers are constructed from a dense sampling of the design space rather than sparse point estimates, though no formal confidence intervals or sensitivity analysis to simulator parameters are reported.

---

Main Quantitative Results

4.1 Model Sensitivity: Architecture and Scale

Headline finding: Disaggregation benefits vary significantly with model architecture, and become more pronounced for larger models (Figures 6, 7).

Model architecture comparison (Figure 6). For DeepSeek-R1 under context-heavy traffic, the disaggregated Pareto frontier sits visibly above the co-located frontier across most of the interactivity range. A specific overhead is identified for MLA-based piggybacking: DeepSeek-R1 experiences "redundant computation of down and up projections in multi-latent attention for each prefill chunk" during piggybacked co-located serving (Section 4.1). For Llama-3.1-70B under the same traffic, the disaggregation benefit is present but less pronounced—the frontiers are closer together, particularly at the high-throughput (relaxed interactivity) end. The paper does not report specific numerical throughput improvements at matched interactivity points; results are presented visually as frontier separation. The key insight is that model architecture affects the magnitude of disaggregation gains because different attention mechanisms (MLA vs. GQA) impose different overheads on co-located optimizations like piggybacking.

Model size scaling (Figure 7). Across Llama 8B, 70B, and 405B models, the paper shows Pareto frontiers for both disaggregated and co-located serving. For Llama 8B, the two frontiers largely overlap—disaggregation provides minimal benefit. For Llama 70B, a discernible gap opens, particularly in the medium-to-high throughput regime. For Llama 405B, the separation is substantially larger. The paper attributes this to the fact that "larger models are typically mapped across more GPUs, enabling a broader range of parallelization strategies" (Section 4.1). The advantage of independently selecting distinct mappings for prefill and decode grows with the size of the parallelism search space. This is a critical boundary condition: for smaller models (roughly ≤10B parameters), disaggregation may not be worth the engineering complexity.

---

4.2 Traffic Pattern Sensitivity

Headline finding: Disaggregation benefits are most pronounced for prefill-heavy traffic (ISL ≫ OSL) and diminish significantly for generation-heavy traffic (Figure 8).

For DeepSeek-R1 across four traffic patterns (Figure 8), the paper presents disaggregated and co-located Pareto frontiers:

• Prefill-heavy traffic: The disaggregated frontier shows substantial expansion over co-located across a broad range of interactivity targets. The paper explains that in prefill-heavy scenarios, co-located serving forces a compromise: "mappings, if prioritized to balance decoding speed, can significantly compromise prefill processing throughput" (Section 4.2). Disaggregation removes this compromise by allowing the prefill pool to optimize independently for throughput while the decode pool maintains interactivity.

• Generation-heavy traffic: The disaggregated and co-located frontiers largely overlap, indicating minimal benefit. In this regime, prefill compute is a small fraction of total work, so independent prefill optimization saves little total GPU time. The paper notes that "piggybacking is most promising on decode-heavy traffic" (Section 4.2), meaning the co-located baseline is strongest precisely where disaggregation helps least.

The paper does not provide exact throughput ratios at specific interactivity points for this analysis; results are shown as frontier curves in Figure 8. The critical practical implication is that traffic pattern characterization should precede any decision to deploy disaggregation—prefill-heavy workloads are the sweet spot, while generation-heavy workloads may see negligible returns.

---

4.3 Dynamic Rate Matching Variation

Headline finding: The optimal context-to-generation GPU ratio varies significantly with model characteristics and target latency, and fixing the ratio degrades performance (Figures 9, 10).

Ratio variation across models and latencies (Figure 9). For DeepSeek-R1, the optimal ctx-to-gen GPU ratio changes substantially across the latency spectrum. The paper visualizes this variation but does not report exact numerical ratios at each point. For Llama-3.1-70B, the ratio variation differs in both magnitude and shape, indicating that rate matching must be model-specific. The paper's key claim is that "a versatile disaggregated serving system should incorporate a dynamic rate matching mechanism to adapt to changes in serving requirements" (Section 4.3).

Fixed-ratio degradation (Figure 10). This experiment quantifies the cost of incorrect rate matching for DeepSeek-R1. A fixed ratio of 3.5:1 (ctx-to-gen) is "performant at the most relaxed latency target but degrades as latency tightens" (Figure 10 caption). Conversely, a ratio of 0.5:1 "favors tight latency but suffers significantly under relaxed latency." The degradation curves are shown as Pareto frontiers comparing the fixed-ratio configurations against the dynamically rate-matched frontier. The key message is that the optimal ratio is a function of the target operating point on the Pareto frontier—no single ratio is optimal across the full interactivity range. This has implications for small-scale deployments: "A similar effect is expected in small-scale GPU deployments, where limited resources can restrict the rate matching search space" (Section 4.3), suggesting that disaggregation benefits may be constrained in resource-limited settings.

---

4.4 NVLink Domain Size Sensitivity

Headline finding: Larger NVLink domains consistently enhance disaggregated serving performance (Figure 11).

For both DeepSeek-R1 and Llama-3.1-70B, the paper compares Pareto frontiers at two NVLink domain sizes (exact sizes not specified, shown in relative terms). For DeepSeek-R1, the larger NVLink domain shows a clear frontier expansion, with the paper attributing this to "higher EP and batching at medium-latency" enabled by the larger domain (Figure 11 caption). The larger domain allows wider expert parallelism within the high-bandwidth NVLink fabric, reducing all-to-all communication overhead for the MoE architecture. For Llama-3.1-70B, the benefit manifests differently: larger NVLink domains "benefit from high TP at low-latency" (Figure 11 caption), enabling wider tensor parallelism under tight TTL constraints without crossing slower inter-domain interconnects.

This hardware sensitivity analysis demonstrates that disaggregation benefits are not purely a software property—they interact with hardware topology in model-specific ways. MoE models benefit from expanded EP domains; dense models benefit from expanded TP domains. The paper does not quantify the magnitude of improvement at specific points, but the frontier separation is visually apparent in Figure 11.

---

4.5 Chunked Pipeline Parallelism for Prefill (Figure 5)

Headline finding: CPP effectively reduces First Token Latency while maintaining high throughput for long prefill sequences under strict FTL constraints.

For DeepSeek-R1 with ISL of 256K tokens on 64 GPUs (EP × PP = 64), Figure 5 shows how FTL decreases as pipeline parallelism depth increases, while throughput per GPU remains high. The paper presents this as a prefill-pool-specific optimization enabled by disaggregation: "CPP is an optimal strategy to maximize throughput while complying with strict FTL SLA" (Figure 5 caption). In a co-located setting, deploying such deep CPP for prefill would negatively impact decode because the pipeline stages would be occupied with prefill chunks rather than available for decode steps. Disaggregation makes this optimization viable by isolating prefill to dedicated GPUs.

---

Ablation Studies and Robustness Checks

P50 approximation validity (Appendix C, Figure 14): The paper tests whether approximating dynamic traffic using constant P50 ISL and OSL values (rounded to the nearest power of two) preserves trend-level fidelity in the Pareto frontier. Simulating both the full dynamic traffic distribution and the P50 approximation, the resulting frontiers are shown to overlap closely: "the approximated frontier closely matches the original" (Appendix C). This ablation is methodologically essential—without it, the constant-ISL/OSL simplification could be a fatal threat to external validity. The validation covers one representative configuration; whether the close match generalizes to other traffic distributions, model architectures, or hardware topologies is not tested.

Piggybacked vs. non-piggybacked co-located baseline (Figure 6): The co-located Pareto frontier is constructed from the union of piggybacked and non-piggybacked configurations, ensuring the strongest possible baseline. The paper does not present an ablation showing disaggregated vs. non-piggybacked-only co-located—which would inflate apparent disaggregation benefits—but rather compares against the best available co-located configuration at each point on the frontier. This is methodologically sound: the paper is comparing disaggregation against the state-of-the-art in co-located serving, not a straw man.

MLA-specific piggybacking overhead (Section 4.1): The paper identifies that DeepSeek-R1 experiences additional overhead in piggybacked co-located serving due to redundant computation of down/up projections in multi-latent attention for each prefill chunk. The mitigation (temporarily caching up-projected KV values from earlier chunks) is mentioned but not evaluated—no ablation shows how much of the disaggregation benefit for DeepSeek-R1 would persist if piggybacking were optimized with such caching. This leaves open the possibility that part of the observed disaggregation advantage for MLA models is attributable to suboptimal piggybacking implementation rather than inherent superiority of disaggregation.

Dynamic traffic vs. P50 approximation (Appendix C): The validation is limited to one traffic distribution (the obfuscated real-world workload). The paper does not test whether the P50 approximation remains valid for distributions with different skew, multimodality, or tail behavior. A heavy-tailed OSL distribution, for instance, might exhibit different queueing dynamics not captured by the P50 approximation. This is noted as a limitation rather than tested.

Simulator fidelity: The paper uses a proprietary simulator, which prevents independent verification of results. No validation metrics are reported comparing simulated against measured latency for any reference configuration. This is a fundamental constraint on the interpretability of all quantitative results—while the relative comparisons (disaggregated vs. co-located) may be robust to systematic simulator biases (if both modes are simulated under the same assumptions), the absolute throughput and latency values cannot be assessed for accuracy.

---

Critical Assessment

The paper's central claims, as established in prior sections, are:

1. Disaggregation expands the throughput–interactivity Pareto frontier, but benefits are conditional on traffic pattern and model scale.
2. Disaggregation benefits are most pronounced for prefill-heavy traffic and larger models.
3. Dynamic rate matching and elastic scaling are critical for achieving Pareto-optimal performance.
4. KV cache transfer bandwidth is not a bottleneck under current datacenter infrastructure.

Here is how the experimental evidence addresses each:

Claim 1 (disaggregation expands the frontier conditionally): The evidence in Figures 6, 7, and 8 visually supports this claim. The disaggregated frontiers sit above co-located frontiers in certain regimes and overlap in others. The paper does not, however, quantify the frontier expansion in a scalar metric (e.g., percentage increase in area under the frontier, or average throughput improvement at fixed percentile interactivity levels). Without such quantification, the magnitude of the benefit remains a visual judgment from plotted curves. This is a significant methodological gap: "expands the frontier" is a qualitative claim when the framework could support quantitative comparison. A reader evaluating whether to invest in disaggregation for their specific workload cannot extract a concrete expected throughput improvement from these figures.

The paper also does not report any configuration where disaggregation performs worse than co-located—a potentially informative negative result. The rate matching procedure guarantees balanced throughput by construction, but if the integer ratio approximation or the independent simulation of prefill/decode pools introduces systematic errors, disaggregated configurations could underperform co-located ones. The absence of any such cases (or the decision not to report them) makes the evidence appear more uniformly positive than may be warranted.

Claim 2 (prefill-heavy traffic and larger models benefit most): Figure 8 convincingly shows a larger gap for prefill-heavy traffic than for generation-heavy traffic. The model-scale finding in Figure 7 shows progressively larger gaps from 8B to 70B to 405B. These are the paper's strongest empirical contributions because the trend is clear and consistent across the evaluated configurations.

However, the evaluation is limited to two model families (DeepSeek-R1 and Llama-3.1) and four traffic patterns. Important architectural categories are missing: encoder-decoder models (T5, BART), models with different attention mechanisms (sliding window, linear attention), and models at intermediate scales between 8B and 70B where the transition from "disaggregation doesn't help" to "disaggregation helps" must occur. The paper claims ">10B parameters" as the threshold for meaningful benefits (Section 4.1), but the experimental evidence only has data points at 8B and 70B—the entire range from 10B to 69B is an interpolation, not an empirical finding. A model at 13B or 30B would have strengthened this claim considerably.

Claim 3 (dynamic rate matching is critical): Figure 10 provides the most direct evidence: a fixed ratio that is optimal at one latency target degrades at others. This is a genuine finding with practical implications. However, the experiment shows only two fixed ratios (0.5 and 3.5), and it is unclear how these were selected. Were they chosen as the optimal ratios at the extremes of the latency range? Are they representative of the ratios that would emerge from a naïve static configuration? The paper would be strengthened by showing degradation across a sweep of fixed ratios, quantifying how rapidly performance falls off as the ratio deviates from optimal. Without this, the reader cannot assess the cost of suboptimal rate matching—is it a 5% throughput loss or a 50% loss? The Figure 10 curves suggest substantial degradation, but exact magnitudes are not reported.

The paper also does not demonstrate a concrete dynamic rate matching mechanism in operation. It shows that the optimal ratio varies (Figure 9) and that fixed ratios degrade (Figure 10), establishing the need for dynamic rate matching. But it does not propose or evaluate an algorithm that adapts the ratio online in response to changing traffic or latency targets. This leaves the "dynamic" part of the claim as a design principle rather than an implemented and validated system component.

Claim 4 (KV cache transfer bandwidth is sufficient): The analytical derivation in Section 5.1 is sound and transparent, with all variables defined and the formulas clearly connected to model architecture parameters and latency constraints. Figure 12 plots the maximum of egress and ingress bandwidth requirements for DeepSeek-R1 at two sequence length combinations across TTL values, showing bandwidth requirements relative to provisioned datacenter bandwidth (the absolute values appear obfuscated in the figure's y-axis). The claim that "existing provisioned datacenter bandwidth is sufficient" follows from the plotted relationship.

This analysis applies to one model (DeepSeek-R1) with two sequence length combinations. It does not extend to Llama-3.1 models, which use GQA rather than MLA and therefore have different KV cache sizes per token. The paper notes that "larger models with optimized attention (i.e., MLA in DeepSeek-R1) may require less egress bandwidth than smaller models with less efficient attention architectures" (Section 5.1). This implies that Llama models with GQA could have higher bandwidth requirements for equivalent parameter counts, but this case is not analyzed. The general claim that bandwidth is not a bottleneck may hold for MLA-based models but not for GQA-based models—the paper does not establish this.

Additional weaknesses:

• Single hardware platform (Blackwell, FP4): All results are for one GPU architecture and numerical precision. The interplay between disaggregation benefits and hardware characteristics (memory bandwidth, interconnect speed, NVLink topology, compute throughput) is explored only for NVLink domain size (Section 4.4). Whether the findings generalize to H100/H200 GPUs, AMD hardware, or inference-specific accelerators is undetermined. The paper acknowledges this implicitly by grounding the analysis in specific hardware but does not discuss how results would change on different platforms.

• No end-to-end system evaluation: The simulation evaluates Pareto frontiers based on steady-state throughput and latency for synthetic constant-ISL/OSL traffic. It does not simulate dynamic request arrival, queueing delays, tail latency under bursty load, or the overhead of the rate matching controller itself. These system-level effects could narrow or widen the gap between disaggregated and co-located serving in deployment. The P50 validation (Figure 14) partially addresses this for traffic distribution but not for temporal dynamics.

• No analysis of the disaggregation transition cost: The paper assumes a deployment is either fully co-located or fully disaggregated. In practice, transitioning between modes—repartitioning model instances, migrating KV caches, adjusting rate matching ratios—incurs latency and throughput penalties. These transition costs determine whether dynamic rate matching (adapting to changing traffic patterns over the course of minutes or hours) is practical, but they are not modeled.

• Limited reporting of absolute numbers: Throughout the results, throughput and latency values are presented as normalized curves without specific numerical benchmarks. While the paper explicitly states its objective is "to convey trends rather than make specific performance claims" (Figure 1 caption), the absence of concrete numbers at specific operating points makes it difficult to assess the practical magnitude of benefits. A practitioner considering disaggregation for their 70B model under P50 traffic needs to know approximate throughput improvement, not just that "the frontier expands." The paper provides the analytical framework for computing such numbers but not the numbers themselves.

• No comparison to alternative optimizations: The paper compares disaggregation against co-located serving with piggybacking. It does not compare against other inference optimizations that could shift the co-located frontier—speculative decoding, quantization beyond FP4, KV cache quantization, prefix caching, or continuous batching variants. Some of these are orthogonal to disaggregation and could be applied in both modes, but others (like KV cache quantization) could particularly benefit co-located serving by reducing memory pressure, potentially narrowing the gap. The paper does not establish whether disaggregation's benefits are additive with or subsumed by other optimizations.

6. Limitations and Trade-offs

6.1 Simulation-Only Evaluation Without Hardware Validation

The assumption or constraint. The paper's entire analysis rests on "a proprietary, high-fidelity GPU performance simulator designed for datacenter-scale inference" (Section 3.1). No results are validated against measurements on physical hardware. The simulator takes model architecture, traffic pattern, and GPU configuration as inputs and produces latency and throughput outputs. The paper does not report any calibration data—no comparison of simulated versus measured latency for a reference configuration, no sensitivity analysis to simulator parameters, and no discussion of what "high-fidelity" means operationally.

The consequence. All quantitative claims—that disaggregation expands the Pareto frontier, that KV cache transfer bandwidth is sufficient, that the optimal ctx-to-gen ratio varies with latency—are conditional on simulator accuracy. If the simulator systematically overestimates throughput for disaggregated configurations (e.g., by underestimating KV cache transfer latency or pipeline bubble effects in CPP), the paper overstates disaggregation benefits. Conversely, if the simulator underestimates the overhead of piggybacking (e.g., missing some optimization for MLA's redundant projections), the co-located baseline appears weaker than it should be. Since the simulator is proprietary, independent verification is impossible. A practitioner cannot assess whether the reported trends would replicate on their hardware.

What evidence exists in the paper. None. The paper does not provide a single measured-vs-simulated data point. The claim that the simulator is "high-fidelity" is unsubstantiated assertion. The validation in Appendix C (Figure 14) compares two simulated results (dynamic traffic vs. P50 approximation) against each other, not against real hardware. This validates the P50 simplification, not the simulator itself. The paper implicitly relies on the simulator's provenance (NVIDIA, targeting Blackwell GPUs) as a proxy for accuracy, but this is not evidence that the simulator correctly models the disaggregation-specific phenomena central to the paper's claims—KV cache transfer overlap with compute, pipeline parallelism bubble dynamics at scale, or interconnect contention between prefill and decode traffic in rate-matched deployments.

Mitigation status. Not addressed. The paper does not acknowledge the absence of hardware validation as a limitation, nor does it discuss what types of error the simulator might introduce. The format of results presentation—normalized curves without absolute numbers ("our primary objective is to convey trends rather than make specific performance claims," Figure 1 caption)—can be read as implicitly hedging against simulator inaccuracy, but this is not stated as a limitation. The recommendation for practitioners is essentially "trust the simulator trends and validate on your own hardware," which passes the burden of verification to the user without providing any calibration reference points.

---

6.2 The Rate Matching Procedure Does Not Account for System Dynamics

The assumption or constraint. The rate matching algorithm in Appendix B (Algorithms 1–2) computes the steady-state optimal GPU ratio assuming constant traffic (fixed ISL, OSL, and request arrival rate), instantaneous KV cache transfer, and zero overhead for reconfiguring the deployment. The paper acknowledges this limitation only indirectly: "Our simulation assumes a datacenter setting with sufficient GPUs and incoming requests to fully utilize the rate-matched deployment. We further assume the KV cache produced at each layer by the prefill pool is transferred to the generation pool immediately as it becomes available, overlapping with the computation of subsequent layers" (Section 3.2).

The consequence. In real deployments, traffic is bursty—request arrival rates vary over seconds to minutes, and ISL/OSL distributions can shift (e.g., users switch from short queries to long document analysis). The optimal ctx-to-gen ratio changes with traffic (Figure 9), but the paper does not model how rapidly or effectively a system could adapt. Transitioning from one ratio to another requires either (a) idling GPUs until the under-provisioned pool drains, reducing utilization during the transition, or (b) dynamically repartitioning model instances, which involves reconfiguring parallelism strategies, reloading model weights, and potentially migrating in-flight KV caches—all incurring latency and throughput penalties. These transition costs may dominate the steady-state benefits if traffic patterns change frequently. Additionally, the "immediate KV cache transfer" assumption ignores network congestion: in a rate-matched deployment with many concurrent requests, KV cache transfers from multiple prefill instances may contend for the same interconnect bandwidth, creating queueing delays not captured by the analytical per-GPU bandwidth formulas in Section 5.1. The formulas compute required bandwidth assuming perfect scheduling, but do not model whether that bandwidth is available under realistic traffic loads with competing transfers.

What evidence exists in the paper. The paper demonstrates that the optimal ratio varies (Figure 9) and that fixed ratios degrade performance (Figure 10), which establishes the need for dynamic rate matching. But the paper does not propose or evaluate a dynamic mechanism. The P50 validation (Figure 14) addresses traffic distribution but not temporal dynamics—it compares steady-state frontiers, not time-varying behavior under bursty arrivals. The paper provides no analysis of transition costs, no simulation of rate-matching adaptation latency, and no evaluation of how quickly the optimal ratio should be recomputed in response to traffic shifts.

Mitigation status. Partially acknowledged, not addressed. Section 4.3 states that "a versatile disaggregated serving system should incorporate a dynamic rate matching mechanism to adapt to changes in serving requirements" and Section 7 lists "the impacts of KV cache reuse, speculation, inference-time compute techniques, and model architecture evolution" as future work, but neither addresses the transition dynamics between rate-matched configurations. The gap between identifying the need for dynamism (a static analysis) and demonstrating a working dynamic system (requiring modeling of adaptation latency, traffic forecasting, and reconfiguration penalties) is substantial and unaddressed.

---

6.3 Generalization Is Limited to Two Model Families, One Hardware Platform, and Synthetic Traffic Patterns

The assumption or constraint. The paper's experimental scope covers DeepSeek-R1 and Llama-3.1 variants (8B, 70B, 405B) on "modern Blackwell systems using FP4 precision" (Section 3.1), evaluated under four canonical traffic patterns defined by constant P50 ISL and OSL combinations. The paper explicitly states this is representative: "We evaluate the sensitivity of disaggregation to traffic pattern in the next section" (Section 4.1) and presents the model-scale analysis (Figure 7) as evidence of a general trend. The bandwidth analysis (Section 5.1) is derived analytically but evaluated only for DeepSeek-R1 at two sequence length combinations (Figure 12).

The consequence. Several claims in the paper may lack generalizability:

• The ">10B parameters" threshold (Section 4.1): The paper has data points at 8B (minimal benefit), 70B (discernible benefit), and 405B (substantial benefit). The claim that benefits become meaningful above 10B parameters is an interpolation with no empirical support. A 13B or 30B model would be needed to establish the threshold with any precision. It is possible that the benefit curve is steep—minimal at 8B, substantial at 20B—or gradual—modest at 70B, large only at 405B. The paper cannot distinguish these.

• Architecture-specific findings may not transfer: The paper identifies MLA's redundant projection computation as a piggybacking overhead that makes disaggregation more attractive for DeepSeek-R1. For models using standard multi-head attention (MHA), sliding window attention, or linear attention, the tradeoffs between piggybacking and disaggregation may differ qualitatively. Encoder-decoder models (used for summarization and translation—precisely the prefill-heavy workloads where disaggregation is claimed to help most) are not evaluated at all, yet differ fundamentally in having separate encoder and decoder components that would interact differently with prefill/decode separation.

• Hardware specificity: All results assume Blackwell GPUs with FP4 precision. Modern deployments span H100, H200, A100, and inference-specific accelerators with different memory bandwidth, interconnect speeds, and numerical format support. The NVLink domain sensitivity analysis (Section 4.4) demonstrates that hardware topology affects disaggregation benefits, but only by varying one parameter (domain size) within one platform family. The claim that KV cache transfer bandwidth is sufficient (Section 5.1) is evaluated only for DeepSeek-R1 on what is implicitly a high-bandwidth datacenter fabric. A deployment using older interconnects or lower per-node bandwidth would face proportionally higher KV cache transfer overhead, potentially making it a bottleneck.

• Traffic pattern coverage: Four canonical patterns span the ISL–OSL space, but real workloads have complex joint distributions (e.g., bimodal—some very short queries, some very long document processing—within the same deployment). The P50 validation (Figure 14) tests one real-world distribution, which is a start, but cannot establish that the four canonical patterns cover the space of possible workload characteristics.

What evidence exists in the paper. The model architecture comparison (Figure 6) covers MLA and GQA. The model scale sweep (Figure 7) covers 8B, 70B, 405B. The traffic sensitivity (Figure 8) covers four patterns. The hardware sensitivity (Figure 11) varies NVLink domain size. These are each one-dimensional sweeps that cannot capture interactions. There is no experiment that varies model architecture, model scale, and traffic pattern simultaneously to test whether the claimed benefits are robust to combinations of factors underrepresented in the evaluation (e.g., a 30B GQA model under balanced traffic on a small NVLink domain).

Mitigation status. Partially acknowledged, not systematically addressed. The paper's title and framing—"A Pragmatic Take"—suggests awareness that the findings are not universal. Section 7 lists future work including "model architecture evolution," implicitly acknowledging that architectural diversity is an open question. The P50 validation (Appendix C) partially addresses traffic pattern generalization for one real distribution. But the paper does not discuss the limited model coverage as a limitation, does not identify the 10B threshold as an interpolation rather than a measurement, and does not discuss how hardware platform differences (beyond NVLink domain size) could affect the conclusions. The reader is left to extrapolate from the evaluated configurations to their specific deployment scenario without guidance on which extrapolations are safe.

---

6.4 Disaggregation Benefits Are Evaluated in Isolation From Other Inference Optimizations

The assumption or constraint. The paper compares disaggregated serving against co-located serving with piggybacking—"the superposition of piggybacked and non-piggybacked configurations" (Section 4.1). It does not evaluate how disaggregation interacts with other inference optimizations that could independently shift the Pareto frontier for either serving mode.

The consequence. The paper cannot distinguish between two possible interpretations of its results: (a) disaggregation provides benefits that are additive with other optimizations, meaning a fully optimized disaggregated system would pull even further ahead of a fully optimized co-located system, or (b) disaggregation's benefits partially overlap with those of other optimizations, meaning the gap would narrow if both modes were augmented with the same complementary techniques. Specific techniques that could interact include:

• Prefix caching / KV cache reuse: In co-located serving, caching KV caches for common prompt prefixes reduces prefill computation. This directly reduces the prefill bottleneck that disaggregation addresses, potentially narrowing the gap between disaggregated and co-located serving for prefill-heavy workloads where prefix reuse is high (e.g., RAG with shared system prompts).

• Speculative decoding: Using a smaller draft model to generate candidate tokens that are verified in parallel by the main model changes the decode bottleneck—verification is more compute-intensive than autoregressive generation. In a co-located setting, this could change the optimal prefill/decode batch ratio in ways not captured by the paper's piggybacking analysis.

• KV cache quantization (beyond FP4): The paper assumes FP4 KV caches. More aggressive quantization (e.g., to 2-bit or binary KV caches) would reduce memory bandwidth pressure during decode, potentially making co-located serving more competitive by freeing memory capacity for larger decode batches.

• Attention kernel optimizations: FlashAttention variants, paged attention improvements, and MLA-specific fused kernels affect prefill and decode latency differently. Optimizations that particularly accelerate prefill (e.g., FlashAttention-3's improved prefill throughput) would reduce the penalty of co-located prefill processing, narrowing the disaggregation gap.

The paper's comparison is methodologically valid for what it tests—disaggregation vs. co-location with piggybacking—but provides incomplete information for a practitioner deciding between these serving modes in the context of a full optimization stack. The practitioner does not choose between "co-located vs. disaggregated" in a vacuum; they choose between specific system configurations that include multiple orthogonal optimizations. The paper's results speak to the marginal benefit of disaggregation assuming all other optimizations are applied equally, but does not test this assumption.

What evidence exists in the paper. None. The paper does not evaluate any optimization beyond parallelism strategies (TP, EP, PP, CPP, TEP) and batching (IFB, piggybacking). Prefix caching, speculative decoding, KV cache quantization, and attention kernel variants are not mentioned in the evaluation. Section 7 lists "KV cache reuse" and "speculation" as future work directions, suggesting the authors recognize this gap but have not explored it.

Mitigation status. Acknowledged as future work only. Section 7 states: "The optimization space for large-scale inference serving is expanding rapidly with new algorithmic techniques and hardware innovations. Of the many interesting directions to explore, the impacts of KV cache reuse, speculation, inference-time compute techniques, and model architecture evolution appear to be particularly promising directions to pursue." This acknowledges the existence of these optimizations but does not frame their omission as a limitation of the current study. The paper does not discuss whether the reported disaggregation benefits should be interpreted as upper bounds (if other optimizations would partially address the same bottlenecks) or lower bounds (if optimizations amplify the benefits of independent prefill/decode configuration).

---

6.5 No Analysis of Deployment Complexity, Failure Modes, or Operational Overhead

The assumption or constraint. The paper evaluates disaggregation purely in terms of the throughput–interactivity Pareto frontier, treating GPU count, parallelism configuration, and rate matching ratio as the only relevant dimensions. It does not consider the operational complexity of deploying and maintaining a disaggregated system compared to a co-located one.

The consequence. The paper's implicit recommendation—that practitioners with prefill-heavy workloads and large models should adopt disaggregation—ignores costs that may dominate the throughput improvements in practice:

• System complexity: A disaggregated deployment requires two separate serving pipelines (prefill and decode) with distinct parallelism configurations, an active rate matching controller that monitors load and adjusts GPU allocation, a KV cache transfer mechanism with reliable networking, and fault tolerance logic to handle GPU failures in either pool without losing in-flight requests. This is substantially more complex than a monolithic co-located deployment, requiring additional engineering effort to build, test, and maintain. The paper provides no framework for assessing whether the throughput improvement (which is presented only as qualitative frontier expansion, not a quantitative percentage) justifies this complexity.

• Failure mode diversity: In a co-located deployment, a GPU failure affects all phases of affected requests equally. In a disaggregated deployment, a prefill pool failure leaves decode GPUs idle with incomplete KV caches; a decode pool failure leaves prefill GPUs producing KV caches that cannot be consumed; a KV cache transfer failure can cause silent data corruption or unavailability of in-flight requests. Each failure mode requires specific recovery logic, and partial failures (some GPUs in one pool failing) interact with the rate matching ratio in non-obvious ways. The paper provides no analysis of reliability, fault tolerance, or availability.

• Load balancing and elasticity: The paper demonstrates that the optimal ctx-to-gen ratio varies with latency targets (Figure 9) and traffic patterns (Section 4.3), motivating dynamic rate matching. But it does not address how a system should implement this: elastic scaling requires the ability to reallocate GPUs between pools (which may involve reloading different model shards), potentially taking tens of seconds or minutes. During reallocation, the system must either serve with a suboptimal ratio (degrading throughput) or buffer requests (increasing latency). The paper provides no latency vs. utilization analysis for the rate matching controller itself.

• Cold start and configuration selection: Deploying a disaggregated system requires choosing initial parallelism strategies and GPU ratios before traffic characteristics are fully known. The paper provides guidance on which regimes benefit (prefill-heavy, large models) but no methodology for a practitioner to determine the specific parallelism configuration and rate matching ratio for their workload without exhaustively simulating their own design space.

What evidence exists in the paper. None. The paper does not discuss system complexity, reliability, elasticity mechanisms, or configuration methodology. These are entirely outside the scope of the simulation-based Pareto frontier analysis.

Mitigation status. Not addressed. The paper treats these concerns as out of scope, focusing exclusively on steady-state performance of correctly configured deployments. Section 7 lists future work directions that could partially address these gaps (e.g., "inference-time compute techniques" could relate to dynamic configuration), but does not connect these to operational concerns. For a paper whose stated goal is providing "practical guidance for at scale deployment" (Section 8), the absence of any discussion of deployment complexity is a significant gap: the paper answers "when does disaggregation help?" in terms of throughput and latency, but not "is it worth the engineering investment?" which is the question a practitioner actually needs answered.

---

6.6 The Pareto Frontier Quantification Is Qualitative, Not Quantitative

The assumption or constraint. Throughout the results, the paper presents throughput–interactivity Pareto frontiers as curves with axes labeled but specific numerical values omitted or normalized. The paper explicitly states in Figure 1's caption that "most results in this paper are presented in normalized form, as our primary objective is to convey trends rather than make specific performance claims." This normalization—removing absolute throughput numbers, latency values, and scale labels—is consistently applied across all result figures (Figures 1, 5–12, 14).

The consequence. The paper successfully conveys directional trends—disaggregation expands the frontier for large models, prefill-heavy traffic benefits most, larger NVLink domains help—but fails to convey magnitude. A practitioner reading the paper cannot answer the following questions:

• What is the approximate throughput improvement (in percentage terms) from adopting disaggregation for a 70B model under prefill-heavy traffic?
• At what interactivity target (tokens/s/user) does the benefit peak?
• How much does suboptimal rate matching degrade throughput—is it a 5% loss or a 50% loss?
• What is the absolute gap between disaggregated and co-located frontiers in tokens/s/GPU?

Without magnitude information, the paper cannot help a practitioner perform cost-benefit analysis. If disaggregation improves throughput by 5% at the cost of 2× engineering complexity, it may not be worth it. If it improves throughput by 40%, it almost certainly is. The paper provides no basis for making this judgment. The normalization is justified by the paper's stated objective of conveying trends, but trends without magnitudes are insufficient for "actionable insights for efficient disaggregated deployments" (Section 1) and "practical guidance for at scale deployment" (Section 8)—both phrases the paper uses to describe its contribution. An actionable insight requires knowing not just what direction the effect points but how large the effect is.

Additionally, the normalization prevents cross-paper comparison. A reader familiar with published throughput numbers for Llama-3.1-70B on H100 GPUs cannot assess whether the simulator's co-located baseline is realistic because the absolute numbers are hidden. The reader cannot determine whether the simulated throughputs are close to what existing systems achieve, making it impossible to calibrate the paper's claims against known performance baselines.

What evidence exists in the paper. The normalized presentation is consistent across all figures. The KV cache transfer bandwidth analysis (Section 5.1, Figure 12) represents a partial exception—bandwidth values are shown on the y-axis, but the axis label and specific values are sufficiently obfuscated that a reader cannot extract concrete numbers. The paper occasionally provides qualitative magnitude descriptions (e.g., "batch sizes are in the hundreds at the high-throughput end"), but these are few and imprecise.

Mitigation status. The normalization is deliberate policy, not an oversight: the paper states it explicitly. But the paper does not discuss the tension between this policy and its stated goal of providing practical guidance. It does not offer a supplementary appendix with absolute numbers for a reference configuration, nor does it discuss what information a practitioner would need to translate the trends into deployment decisions. The paper could have provided one or two fully specified operating points with absolute throughput, latency, and GPU count numbers—even under a disclaimer that these are simulation estimates rather than hardware-validated benchmarks—to anchor the normalized curves in concrete magnitudes. The choice not to do so limits the paper from "actionable insights" to "directional hypotheses" that require substantial independent validation before deployment.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around inference disaggregation from "is it promising?" to "when is it worth it?" — a transition from advocacy to engineering pragmatism. Prior to this work, the community had accumulated a wave of disaggregation system proposals (DistServe, Splitwise, Mooncake, DéjàVu, P/D-Serve, among others) and open-source implementations (TensorRT-LLM, vLLM), but lacked a systematic framework for determining which workloads and which model scales actually benefit. The paper's core methodological contribution is establishing the throughput–interactivity Pareto frontier as the correct evaluation surface for disaggregation — not peak throughput, not minimum latency, but the full tradeoff curve that characterizes how configurations perform under realistic SLO constraints. This reframes the analysis from point estimates to frontier expansion, which is a conceptual upgrade that future work in the area should adopt as standard.

The reconciliation of conflicting intuitions is a key service. The paper shows that both proponents and skeptics of disaggregation can be correct, depending on the operating regime. Proponents are right that disaggregation can meaningfully expand the Pareto frontier — but only under prefill-heavy traffic patterns where co-located serving forces genuine compromises between prefill throughput and decode interactivity (Figure 8). Skeptics are right that disaggregation adds complexity without commensurate benefit — but primarily for generation-heavy workloads and small models where the co-located baseline already performs near-optimally (Figures 7, 8). The paper's difficulty is not that it picks a side, but that it maps the boundary conditions precisely enough to convert a polarized debate into a decision framework.

The research directions that become more attractive following this work are those that address the specific bottlenecks the paper identifies as limiting disaggregation's reach. The finding that disaggregation benefits are largest for prefill-heavy workloads but absent for generation-heavy ones (Section 4.2) implies that techniques reducing prefill cost in co-located settings — prefix caching, KV cache reuse across requests, more efficient attention for long contexts — could narrow the gap and make disaggregation less necessary. Conversely, techniques that make disaggregation viable in more regimes — reducing KV cache transfer overhead for generation-heavy traffic, enabling fine-grained elastic scaling between pools — could expand disaggregation's applicability beyond its current sweet spot. The finding that larger models benefit disproportionately (Figure 7) suggests that as frontier models continue scaling, disaggregation becomes more important over time, not less — the trendline favors investment in disaggregation infrastructure even if current-generation models at smaller scales show marginal returns.

The research directions that become less attractive are those focused on ever-more-complex disaggregation system designs without first establishing that the target workload benefits. The paper's demonstration that a simple integer-solver-based rate matching algorithm (Appendix B) suffices to find Pareto-optimal configurations — and that the real challenge is knowing whether to deploy disaggregation at all, not how to implement it once deployed — suggests that algorithmic sophistication in the disaggregation controller is not the bottleneck. Research investment should shift from "how to disaggregate" (which is increasingly well-understood, with multiple open-source implementations available) to "when to disaggregate" (which requires workload characterization, traffic forecasting, and cost-benefit analysis frameworks that this paper begins to provide but does not complete).

The paper's identification of dynamic rate matching as essential but un-implemented (Section 4.3) also redirects attention. The optimal context-to-generation GPU ratio varies substantially across latency targets and traffic patterns (Figure 9), and fixed ratios degrade performance significantly (Figure 10). Yet the paper provides only a static analysis — it shows that the optimal ratio varies, but does not propose or evaluate a mechanism that adapts the ratio online. This makes dynamic rate matching the most obvious open problem exposed by the paper: a system that can monitor traffic characteristics, forecast near-term ISL/OSL distributions, and reallocate GPUs between prefill and decode pools with low transition overhead would directly address the paper's central finding that static configurations are inherently suboptimal.

Finally, the paper's KV cache transfer bandwidth analysis (Section 5.1) provides a quantitative counter-argument to what has been a persistent objection to disaggregation — that moving KV caches between pools consumes prohibitive interconnect bandwidth. By deriving the required bandwidth from first principles (Equations 1–2) and showing that it falls within provisioned datacenter bandwidth for DeepSeek-R1 under realistic latency constraints (Figure 12), the paper removes a perceived barrier to adoption. This shifts the burden of proof: critics of disaggregation must now argue about other bottlenecks (operational complexity, failure modes, cold start latency) rather than bandwidth, because the bandwidth objection has been analytically addressed for the evaluated configurations.

Follow-Up Research This Work Enables

Online traffic-adaptive rate matching with bounded transition cost. The paper demonstrates that the optimal context-to-generation GPU ratio varies with latency targets and traffic patterns (Figures 9, 10), but evaluates only static configurations. The natural follow-up is a system that monitors request arrival patterns and sequence length distributions in real time, forecasts the P50 ISL/OSL over a sliding window (validated in Appendix C as a sufficient proxy), and triggers GPU reallocation between prefill and decode pools when the forecast optimal ratio deviates from the current ratio by more than a threshold. The key research question is the tradeoff between reallocation frequency and throughput: every reallocation incurs a transition cost (idle GPUs during repartitioning, KV cache migration, model weight reloading), and a system that reallocates too aggressively will lose more to transition overhead than it gains from improved ratio alignment. A strong follow-up would characterize this tradeoff curve — throughput improvement vs. reallocation interval — under realistic bursty traffic traces, and determine whether a simple hysteresis controller (reallocate only when the optimal ratio has shifted by, say, >20% and remained there for >30 seconds) captures most of the benefit of continuous optimal ratio tracking while avoiding thrashing.

Disaggregation benefit quantification for the 10B–70B parameter range. The paper's model-scale analysis (Figure 7) has data points at 8B (minimal benefit), 70B (discernible benefit), and 405B (substantial benefit). The claim that disaggregation benefits become meaningful above 10B parameters is an interpolation, not a measurement. A focused parameter-scale sweep — evaluating Llama-3.1 or comparable dense models at 13B, 30B, and 65B under the same traffic patterns and hardware assumptions — would establish the functional form of the benefit-vs-scale curve. Does it exhibit a phase transition (sudden onset of benefits at some threshold scale where the parallelism search space becomes rich enough) or a smooth increase (benefits grow gradually with parameter count)? The answer has direct practical implications: if benefits emerge suddenly at ~30B, the ">10B" guidance in the paper is too optimistic for the 10B–30B range; if benefits grow smoothly, a 13B model might see modest but worthwhile gains. This experiment requires no new methodology — the paper's simulation framework can be applied to intermediate-scale models without modification — and would convert the paper's qualitative trend into a quantitative decision boundary.

Prefix caching interaction with disaggregation — substitution or complement? The paper evaluates disaggregation against co-located serving with piggybacking, but does not consider prefix caching (KV cache reuse across requests sharing common prompt prefixes). This omission matters because prefix caching directly reduces prefill computation — the same bottleneck that disaggregation addresses by offloading prefill to dedicated GPUs. The key question is whether prefix caching and disaggregation are substitutes (both address prefill cost, so the benefit of adding disaggregation to a system that already caches prefixes is diminished) or complements (prefix caching reduces prefill load, freeing prefill GPUs to serve more unique requests, while disaggregation prevents the cached-prefill hit path from interfering with decode interactivity). A strong experiment would simulate a RAG workload where a shared system prompt or document context creates high prefix reuse, measure the Pareto frontier for co-located-with-caching vs. disaggregated-with-caching, and determine whether the marginal benefit of disaggregation increases or decreases as the cache hit rate varies. If the marginal benefit decreases at high hit rates, practitioners with cacheable workloads may prefer investing in caching infrastructure over disaggregation. If the marginal benefit is additive, the two techniques should be deployed together.

KV cache transfer bandwidth analysis for GQA-based models at scale. The paper's bandwidth sufficiency analysis (Section 5.1, Figure 12) covers only DeepSeek-R1 with MLA, which the paper notes "may require less egress bandwidth than smaller models with less efficient attention architectures" because MLA compresses the KV representation. For Llama-3.1-70B or 405B using GQA, the KV cache size per token is larger (more KV heads, no low-rank compression), and the analytical formulas (Equations 1–2) predict correspondingly higher bandwidth requirements. The critical follow-up experiment is to apply the same analytical model to Llama-3.1-405B under the traffic patterns and latency constraints used in the paper's Pareto frontier analysis, compute the resulting egress and ingress bandwidth requirements, and determine whether they remain within provisioned datacenter bandwidth or exceed it. If bandwidth becomes a bottleneck for large dense models, this would establish a boundary condition the paper currently does not address: disaggregation may be bandwidth-feasible for MLA-based MoE models but not for large GQA-based dense models, narrowing the set of architectures for which KV cache transfer overhead is negligible.

End-to-end system prototype with dynamic rate matching and real traffic. The paper's analysis is entirely simulation-based, evaluating steady-state Pareto frontiers under constant ISL/OSL approximations. The P50 validation (Appendix C, Figure 14) suggests the approximation preserves trend fidelity, but cannot capture dynamic effects: bursty arrivals, queueing delays at the KV cache transfer layer, tail latency amplification under load, and the overhead of the rate matching controller itself. A strong follow-up would implement a minimal disaggregated serving prototype — building on one of the existing open-source implementations (vLLM's experimental disaggregated prefilling or TensorRT-LLM's disaggregated serving) — instrument it with the rate matching logic from Appendix B extended to support online ratio adjustment, and evaluate it on a replay of real production traffic traces. The key measurements are: (1) does the Pareto frontier measured on real hardware match the simulated frontier in shape and approximate magnitude? (2) does dynamic rate matching (reallocating GPUs in response to detected traffic shifts) improve throughput relative to a static best-guess ratio, and by how much? (3) what is the latency tail (P99 FTL and TTL) under bursty load, and does disaggregation amplify or dampen tail effects compared to co-located serving? This experiment would bridge the gap between the paper's simulation-based design principles and operational reality, either validating the trends or revealing dynamics the simulator misses.

Disaggregation under heterogeneous hardware — does the benefit scale with NVLink topology diversity? The paper's NVLink domain sensitivity analysis (Figure 11) shows that larger NVLink domains improve disaggregated serving performance, but evaluates only two domain sizes within a single hardware platform (Blackwell). A more ambitious experiment would evaluate disaggregation benefits across a heterogeneous cluster — mixing nodes with different NVLink domain sizes, GPU memory capacities, or interconnect bandwidths — and determine whether the ability to independently map prefill and decode to different hardware profiles (e.g., prefill on high-compute GPUs with large NVLink domains, decode on memory-bandwidth-optimized GPUs) provides additional gains beyond the homogeneous case. This connects to the HexGen-2 line of work on heterogeneous disaggregation (Jiang et al., 2024) but with the Pareto frontier methodology introduced by this paper. The hypothesis is that heterogeneity amplifies disaggregation benefits because the optimal hardware for prefill (high arithmetic throughput, large batch capacity) differs from the optimal hardware for decode (high memory bandwidth, low per-token latency), and disaggregation allows matching each phase to its ideal hardware rather than compromising on a single instance type.

Practical Applications and Downstream Use Cases

Long-context document processing pipelines (RAG, summarization, multi-document QA). These workloads are characterized by prefill-heavy traffic: input sequences of 32K–256K tokens with relatively short outputs (hundreds to low thousands of tokens). The paper identifies this as the regime where disaggregation provides the greatest Pareto frontier expansion (Section 4.2, Figure 8, prefill-heavy pattern). A deployment serving document summarization at scale — where each request involves ingesting a lengthy report and producing a paragraph-length summary — would benefit from dedicating a prefill pool optimized with Chunked Pipeline Parallelism (Figure 5) to process long contexts within FTL SLOs, while a smaller decode pool handles the modest generation phase. The paper's specific guidance: use CPP with deep pipeline parallelism for prefill (Figure 5 demonstrates FTL reduction with increasing PP while maintaining throughput for DeepSeek-R1 at 256K ISL), independent tensor parallelism for decode (Section 4 notes Llama-3.1-70B scales from 2× to 64× TP as TTL tightens), and a high ctx-to-gen GPU ratio (Figure 9 shows ratios vary with latency target, but prefill-heavy traffic implies more prefill than decode GPUs). The practical benefit is the ability to serve long-context requests without violating FTL SLOs, which in a co-located setup would require either unacceptably large decode stalls or unacceptably slow prefill processing.

Frontier-scale model serving where GPU count justifies specialization. The paper's model-scale analysis (Figure 7) shows that disaggregation benefits grow with model size: minimal for 8B, discernible for 70B, substantial for 405B. For organizations deploying models at the 70B+ scale — where the model must be partitioned across dozens or hundreds of GPUs regardless of serving mode — the incremental complexity of assigning some GPUs to a prefill pool and others to a decode pool is small relative to the baseline complexity of multi-node serving. In this regime, the paper's guidance is to adopt disaggregation as the default architecture: the richer parallelism search space at large scale (Section 4.1) means that the opportunity cost of using a single partitioning strategy for both phases is higher, and the relative engineering overhead of managing two pools is lower because any deployment at this scale already requires sophisticated orchestration. The benefit is higher throughput per GPU at a given interactivity target — the paper's Pareto frontier curves show substantial separation for 405B across much of the interactivity range — translating directly to lower infrastructure cost per request served.

Batch inference with heterogeneous sequence lengths (evaluation, data generation, scoring). In batch inference scenarios — running a fixed set of prompts through a model to generate completions for evaluation, training data synthesis, or RLHF reward modeling — the traffic pattern is often mixed, with wide variation in both input and output lengths. Unlike interactive serving, batch inference does not have per-request TTL SLOs; latency is measured end-to-end for the entire batch. However, the fundamental tension between prefill and decode still applies: a co-located batch with mixed ISL/OSL will have some GPUs idle during prefill-dominated periods and others bottlenecked during decode-dominated periods. Disaggregation allows the prefill pool to process all inputs as a single optimized batch (using the optimal parallelism for the P50 ISL) and stream KV caches to the decode pool, which processes generation for all requests with decode-optimized parallelism. The paper's rate matching methodology (Appendix B) can be adapted to batch settings by computing the ratio based on total prefill and decode work rather than per-request rates. The practical benefit is higher GPU utilization and faster batch completion, which matters for pipelines where batch inference latency gates downstream processing.

When to Prefer This Method

The paper articulates a clear tradeoff between disaggregated and co-located serving, with specific conditions determining which mode is preferable. The decision framework implicit in the paper's findings is:

• Prefer disaggregated serving when the workload is prefill-heavy (ISL substantially exceeds OSL, Figure 8), the model is large (roughly >70B parameters, with benefits increasing with scale per Figure 7), and the deployment operates at a scale where the parallelism search space is rich enough that independent prefill/decode mapping provides meaningful latitude (multi-node, large NVLink domains per Figure 11). Additionally, disaggregation is favored when dynamic rate matching infrastructure is available to adapt the ctx-to-gen ratio to the target latency regime (Figures 9–10), since fixed ratios degrade performance away from their design point.

• Prefer co-located serving with piggybacking when the workload is generation-heavy (OSL dominates ISL, Figure 8), the model is small (roughly ≤10B parameters, where Figure 7 shows minimal frontier separation), or piggybacking's chunking mechanism is efficient for the model's attention architecture (GQA models avoid the MLA-specific redundant projection overhead described in Section 4.1). Co-located serving is also preferred when operational simplicity is paramount — the paper does not quantify the engineering cost of disaggregation, but the absence of KV cache transfer infrastructure, rate matching logic, and dual-pool management is a legitimate consideration for small-scale or latency-insensitive deployments.

• The boundary is not sharp. For intermediate regimes — medium-scale models (13B–65B), balanced traffic patterns, or moderate NVLink domains — the paper does not provide sufficient evidence for a definitive recommendation. The qualitative frontier separation in Figures 6–8 suggests modest benefits that may or may not justify the engineering investment depending on organizational priorities. A practitioner in this regime should use the paper's simulation methodology (or a comparable in-house simulator) to estimate the specific throughput improvement for their model, traffic, and hardware before committing to disaggregation.