Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads

🎯 Pitch

Co-running LLM prefill and decode requests on the same GPU causes massive slowdowns—up to 10× for mixed-length prefills—because one phase is compute-bound and the other is memory-bound. TetriInfer eliminates this by physically separating prefill and decode instances and using a length-predictive scheduler, cutting first-token latency by 97% and job completion time by 47% while using 38% fewer resources.

1. Executive Summary

TetriInfer introduces a cloud-scale LLM inference serving system that analyzes and mitigates interference across mixed downstream workloads by carefully scheduling and grouping requests based on their distinct prefill and decode characteristics. Evaluated against vLLM using OPT-13B and the ShareGPT dataset, the system employs three named mechanisms: chunked prefill (partitioning prompts into fixed-size computation-saturated chunks to avoid prefill-prefill interference), disaggregated prefill and decode instances (separating the computation-heavy prefill phase from the memory-intensive, latency-critical decode phase onto independent, dynamically flippable instances), and a two-level scheduling algorithm augmented with predicted resource usage (an LLM-based length predictor that classifies generation length into buckets, enabling decentralized load-balancing across decode instances and working-set-aware intra-decode scheduling to avoid hotspots). On light-prefill-heavy-decode workloads, TetriInfer lowers average TTFT by 97% and average JCT by 47% while using 38% fewer total hardware resources, yielding a 2.4× improvement in perf/$. On mixed workloads, it reduces average TTFT by 85% and JCT by 50%, establishing that disaggregation with length-aware scheduling substantially outperforms coupled prefill-decode serving — but the gains reverse on heavy-prefill-heavy-decode workloads where the room for improvement is marginal and the introduced transfer overhead cannot be offset.

2. Context and Motivation

The Core Problem: LLM Inference Serves Radically Different Workloads on the Same Infrastructure

The paper addresses a fundamental tension in modern LLM serving: a single deployed model must simultaneously handle inference requests with dramatically different computational signatures, yet existing serving systems treat all requests as if they belong to a single, uniform workload. This is not a niche edge case — it is the default operational reality for any cloud LLM service.

The authors ground this in the two-phase structure of generative LLM inference. Every request goes through a prefill phase (processing the entire input prompt to produce the first token and the KV cache) followed by a decode phase (generating output tokens one-by-one auto-regressively, each step attending to the growing KV cache). These two phases have fundamentally different resource profiles, which the paper empirically confirms in Figure 2:

"the prefill phase is computation-bound, and the decode phase is memory-bound"

Specifically, prefill throughput plateaus once the accelerator reaches a computation-saturated threshold (adding more tokens to a prefill batch past this point yields no throughput gain but continues to increase latency). Decode throughput, by contrast, scales with batch size until memory bandwidth saturates — a different bottleneck entirely.

The problem becomes acute because real-world LLM services serve mixed downstream workloads whose requests differ by more than two orders of magnitude along both dimensions. The paper characterizes this concretely using Figure 1, which plots the cumulative distribution of prompt tokens and generated tokens across three workload types sampled from public datasets:

Conversation / chat (ShareGPT): short prompts (~18 tokens median), moderate-length generations (~128 tokens median), corresponding to "light prefill, light-to-moderate decode"
Summarization (PubMed summarization dataset): long prompts (often 2,000+ tokens), short generations (often <100 tokens), corresponding to "heavy prefill, light decode"
Content creation / writing (write_doc_sft_v1): short-to-moderate prompts, long generations (often 1,000+ tokens), corresponding to "light prefill, heavy decode"

When these requests land on the same inference cluster — as they inevitably do in any multi-tenant cloud service — the scheduling decisions made by the serving system determine whether they interfere destructively or execute efficiently.

Why This Problem Matters

The significance of this problem has multiple dimensions, both practical and architectural.

Practical: cost efficiency at scale. LLM inference serving is expensive. The paper notes that "numerous works were proposed to improve the cost efficiency of LLM inference" (Section 1), citing vLLM's PagedAttention and other optimizations. But these prior efforts focus on within-phase efficiency — better memory management, faster attention kernels, smarter batching — without addressing the cross-phase interference that arises when prefill and decode co-exist in the same serving engine. If interference causes a 5× slowdown in decode iteration time (as Figure 4b shows), then all the kernel-level optimizations in the world cannot compensate. For a cloud provider running thousands of GPUs, a 2.4× improvement in performance per dollar (as TetriInfer achieves on LPHD workloads in Figure 12) translates directly to millions in infrastructure savings or the ability to serve more customers with the same hardware.

Theoretical: a gap in the systems research landscape. The authors identify a specific missing piece in the LLM serving literature. The community has recognized that prefill is compute-bound and decode is memory-bound for some time (Pope et al., 2023, cited as reference [33]). But no prior system had built a serving architecture that fully extracts the implications of this observation. Existing systems either:

Couple prefill and decode in the same engine with continuous batching (vLLM, Orca), treating the phase distinction as a scheduling detail rather than an architectural boundary;
Disaggregate prefill and decode but do so statically, without mechanisms for dynamic role-flipping, length-aware load balancing, or fine-grained prefill chunking to prevent intra-phase interference (the paper positions Splitwise as a concurrent work that also disaggregates but does not address the full interference taxonomy).

The paper fills this gap by providing a systematic interference taxonomy (§2.2) and then building a system whose architecture is a direct response to each category of interference identified.

Architectural: setting the stage for elastic, workload-aware serving. By making prefill and decode instances virtual concepts that can be independently scaled and flipped (Section 3.5), TetriInfer enables an operational model where the cluster adapts to shifting workload mixes in real time — spinning up more decode instances during a content-creation surge, converting idle prefill capacity to decode during a summarization-heavy period, and so on. This is a qualitatively different operational capability than what static, coupled architectures provide, and it matters because real LLM workloads are anything but static.

Where Prior Approaches Fall Short

The paper identifies specific limitations in existing LLM serving systems, supported by the interference measurements in Section 2.2. The authors classify inference requests into four types based on the two axes (prefill length, decode length) and one binary property (heavy or light, where heavy means long tokens): heavy prefill, light prefill, heavy decode, light decode. They then systematically measure what happens when these types co-exist in a single serving engine.

Prefill-Prefill interference (Section 2.2.1): computation saturation causes cascading slowdowns. Using OPT-13B on V100 GPUs, the authors establish that the accelerator-saturate threshold for prefill is 512 tokens (Figure 2a). When multiple prefill requests are batched and their total token count exceeds this threshold, latency increases dramatically even though throughput stays flat. Figure 3 quantifies this: a light prefill request (18 tokens) experiences a 2× slowdown when co-running with 7 other light prefill requests and an 8× slowdown with 63 concurrent light prefills. Mixing light and heavy prefill is worse: a light prefill incurs >10× latency slowdown when running alongside even a single heavy prefill (512 tokens), and heavy prefills themselves suffer 3× slowdown when co-run with light prefills.

The root cause: "when the total number of tokens in a batch is larger than the accelerator-saturate threshold, the prefill latency dramatically increases." Existing systems that use fixed batch sizes (like vanilla vLLM's continuous batching) have no mechanism to prevent this — they keep adding prefill requests to a batch until a memory or batch-size limit is hit, even though doing so pushes the total token count deep into the saturated regime where every additional token is pure latency penalty with zero throughput benefit.

Prefill-Decode interference (Section 2.2.2): co-scheduling batch and latency-critical jobs causes mutual harm. This is perhaps the most damaging category of interference the paper identifies. When prefill and decode requests are interleaved in the same continuous batch (as vLLM and Orca do), both phases suffer. Figure 4a shows that a light decode request's per-iteration decoding time increases by 5× even when only a single heavy prefill request joins the batch. The mechanism: prefill's computation-heavy forward pass blocks the decode phase's latency-critical iterations, and the large KV cache generated by heavy prefill occupies accelerator memory that could otherwise be used to batch more decode requests.

The reverse direction is also harmful. Figure 4c shows that a light prefill's latency "increases once the number of co-running light decode requests is more than 7," and Figure 4d shows heavy prefill slowing by up to 2.5× when co-run with decode requests. The paper's diagnosis: "mixing prefill and decode requests hurts both because we co-run batch and latency-critical jobs at the same time."

This is a classic systems scheduling problem — batch and latency-critical tasks have fundamentally different optimization objectives — but prior LLM serving systems treated both phases as simply "the next forward pass" and interleaved them opportunistically without recognizing the tension.

Decode-Decode interference (Section 2.2.3): memory bandwidth contention from length heterogeneity. Even when only decode requests run together, mixing light decode (20–100 generated tokens) with heavy decode (>512 generated tokens) causes measurable degradation. Figure 5 shows that with a batch size of 128, compared to an all-light-decode batch, a batch that is half heavy-decode and half light-decode suffers a 16% throughput drop and a 23% latency increase.

The root cause: "we are unaware of the memory bandwidth and capacity usage, thus leading to contention and head-of-line blocking." Heavy decode requests consume more KV cache memory (growing with each generated token) and more memory bandwidth (each decode step must attend to the full accumulated KV cache). When these resource hogs share a batch with light decode requests, the light requests are blocked behind the heavy ones, and the available memory bandwidth is unequally consumed.

This problem is particularly insidious because it is invisible to systems that schedule based on request count rather than resource usage. Two decode requests might look identical to a count-based scheduler, but one that will generate 2000 tokens imposes vastly more memory pressure than one that will generate 20 tokens. Without predicting generation length, the scheduler cannot avoid co-locating resource-heavy and resource-light decode requests.

The naive solution is economically infeasible. The paper explicitly considers and rejects the obvious fix: "a naive solution to avoid interference is to provision resources for each downstream task statically." Given the high cost of LLM serving infrastructure and the wide variation in workload mixes, statically partitioning GPUs into "summarization-only," "chat-only," and "content-creation-only" pools would mean most accelerators are idle most of the time. The challenge is to achieve the isolation benefits of static partitioning without the cost — precisely what disaggregation with dynamic instance flipping aims to provide.

How This Paper Positions Itself

TetriInfer positions itself as the first system to comprehensively address all three categories of LLM inference interference through a unified architectural approach rather than point-fixes. The intellectual move is to go from the observation that "prefill resembles a computation-heavy batch job" while "decode resembles a memory-intensive, latency-critical task" (Section 2.3) to an architecture where these two job types are handled by separate, specialized instances with different scheduling policies, connected by an efficient KV cache transfer mechanism.

The paper explicitly differentiates its approach from the two closest related systems in Table 1 (Section 6):

Versus vLLM (Kwon et al., 2023). vLLM's contributions are in KV cache memory management (PagedAttention) and continuous batching. TetriInfer is implemented based on vLLM and inherits its paging mechanism, but adds chunked prefill, disaggregation, and two-level length-aware scheduling. In the comparison taxonomy of Table 1, vLLM has none of these four features, while TetriInfer has all four. The paper is careful to acknowledge vLLM's memory management contribution while showing that memory efficiency alone does not solve the interference problem.

Versus Splitwise (Patel et al., 2023). Splitwise is a concurrent work that also disaggregates prefill and decode. TetriInfer differs in three ways the paper highlights: (1) Splitwise does not use chunked prefill, leaving prefill-prefill interference unaddressed; (2) Splitwise does not address decode-decode interference through length-aware scheduling (the Interference column in Table 1 is marked with "×" for Splitwise); (3) TetriInfer adds the instance flip mechanism, which the paper treats as a practical necessity for cloud deployments where workload mixes shift.

Versus Sarathi (Agrawal et al., 2023). Sarathi proposed chunked prefill but runs prefill-decode-mixed chunks — it piggybacks decode steps onto prefill chunks rather than separating them. TetriInfer argues this is insufficient because "we observe non-negligible interference between prefill and decode, thus choose to disaggregate prefill from decode" (Section 6). So Sarathi has chunked prefill (✓ in Table 1) but lacks disaggregation (×) and length-aware distributed scheduling (×).

Versus FastServe (Wu et al., 2023). FastServe addresses JCT minimization through multi-level priority feedback queues for decode scheduling but does not disaggregate prefill and decode, does not use chunked prefill, and does not address the interference taxonomy the paper identifies. In Table 1, FastServe has only the Distributed-Scheduling feature.

The paper's unique synthesis. What distinguishes TetriInfer is not any single technique in isolation — chunked prefill appears in Sarathi, disaggregation appears in Splitwise — but rather the integration of all three pillars into a unified system with a specific design rationale for each:

Chunked prefill with prefill-only chunks addresses prefill-prefill interference (§2.2.1) by ensuring the accelerator always operates at its computation-saturated sweet spot without ever exceeding it.
Disaggregated prefill and decode instances with dedicated scheduling for each addresses prefill-decode interference (§2.2.2) by preventing these fundamentally different job types from ever sharing the same accelerator at the same time.
Length prediction + two-level resource-aware scheduling addresses decode-decode interference (§2.2.3) by spreading heavy decode requests evenly across instances and making intra-instance scheduling decisions based on predicted working set size rather than request count alone.

The paper's central intellectual claim is that these three interference categories are not independent problems that can be solved piecemeal — they interact. For example, disaggregation alone (without chunked prefill) would still leave prefill instances vulnerable to prefill-prefill interference. Chunked prefill alone (without disaggregation, as in Sarathi) would still leave prefill and decode interfering in the same batch. Length prediction alone (without disaggregation) would leave prefill-decode interference unsolved. The synthesis matters.

Finally, the paper is honest about its boundaries. It explicitly states that TetriInfer's design "is not ideal for HPHD workloads as the room for improvement is small, and the overhead we introduce cannot be offset" (Section 5.1 Takeaways). This is not presented as a failure but as a clear boundary condition — when both prefill and decode are heavy, the overhead of KV cache transfer (which scales with prefill length) dominates, and the interference that disaggregation eliminates was never the primary bottleneck. This self-awareness strengthens the paper's contribution because it tells practitioners exactly when to adopt TetriInfer and when not to.

3. Technical Approach

3.1 Reader Orientation

TetriInfer is a distributed LLM inference serving system that restructures how inference requests are processed across GPU instances to eliminate performance interference between different types of requests. It solves the problem that a single deployed LLM must simultaneously handle inference requests with dramatically different computational signatures — some are computation-heavy batch jobs (long prompts during prefill), others are memory-intensive, latency-sensitive streaming tasks (token-by-token decode), and still others are light chat requests — yet existing systems dump them all into the same GPU engine where they destructively interfere with each other. The shape of the solution is a three-pillar architecture: (1) partition all prefill work into fixed-size computation chunks so no single prefill can oversaturate the accelerator, (2) physically separate prefill processing from decode processing onto different GPU instances so batch and latency-critical work never share the same hardware, and (3) predict how many tokens each request will generate before scheduling its decode phase, then use that prediction to spread heavy decode work evenly across instances and to make memory-aware scheduling decisions within each instance.

3.2 Big-Picture Architecture (Diagram in Words)

The system consists of five major component types, organized into a centralized control layer and a distributed execution layer:

Centralized Control Plane — a distributed system (not a single bottleneck node) containing two modules. The global scheduler receives inference requests from external services, assigns each to a prefill instance based on current load, maintains a request status table tracking arrival time, current phase, and SLA requirements, and streams completed outputs back to clients. The cluster monitor collects load statistics (CPU, memory, queue depth) from every prefill and decode instance every 100ms, aggregates and broadcasts this information so that prefill instances can make informed downstream routing decisions, and manages the lifecycle of instances — adding, removing, or flipping them between roles.

Prefill Instances — GPU instances dedicated exclusively to running the prefill phase. Each contains four internal modules. The local prefill scheduler sorts incoming raw requests using one of three policies (FCFS, SJF, or LJF) with a configurable scheduling batch size to prevent starvation. The length predictor runs a small, fine-tuned LLM classification model (OPT-125M) in parallel with the main LLM to predict which length-range bucket the request's generation will fall into. The main LLM engine (OPT-13B in the paper's evaluation) runs chunked prefill: it slices and merges prompt tokens into fixed-size chunks set to the accelerator's computation-saturate threshold, then processes one chunk at a time, padding the final chunk with zeros if needed. The dispatcher runs after a request's first chunk is prefilled; it executes a decentralized load-balancing algorithm (power-of-two with length-aware tiebreaking) to select a decode instance, then transfers the prefilled KV cache to that instance.

Decode Instances — GPU instances dedicated exclusively to running the decode phase. Each contains a receiver module (the network stack endpoint that accepts incoming KV caches and request metadata), a local scheduler that groups requests using continuous batching with PagedAttention-based KV cache management, and the main LLM engine running auto-regressive token generation. The local scheduler implements two working-set-aware policies (reserve-static and reserve-dynamic) that use predicted generation length to decide whether to admit a new request to the current batch based on available and future memory, rather than greedily admitting requests until memory is full.

Length Prediction Model — a small OPT-125M classification model fine-tuned offline to predict the token-length bucket of responses that a specific target LLM (OPT-13B) would generate for a given prompt. It runs at every prefill instance, consumes the same prompts as the main LLM, and outputs a length-range label (e.g., bucket 0 for 0–200 tokens, bucket 1 for 200–400 tokens) that downstream modules use for resource-aware scheduling.

KV Cache Transfer Network — a unified network abstraction layer that transmits prefilled KV caches from prefill instances' accelerator memory to decode instances' accelerator memory. It abstracts over three physical link types (Direct, Direct-NIC, Indirect) and two software stack categories (one-sided, two-sided), exposing a uniform API (send, receive, read, write) so the dispatcher can use whatever high-performance fabric is available at deployment time.

Information flows as follows: an external request arrives at the global scheduler → the global scheduler routes it to the least-loaded prefill instance → the prefill instance's local scheduler sorts it into a scheduling batch → the length predictor classifies its expected generation length in parallel with chunked prefill execution → after the first prefill chunk completes, the dispatcher selects a decode instance using power-of-two selection biased toward instances where this request's predicted length profile causes the least interference → the prefilled KV cache transfers to the chosen decode instance → the decode instance's local scheduler admits the request to its continuous batching loop when memory conditions permit → generated tokens stream back through the global scheduler to the client.

3.3 Roadmap for the Deep Dive

This section walks through TetriInfer's architecture in the order that information and control flow through the system during request processing, because this order naturally builds understanding: each downstream component depends on decisions made or predictions generated upstream.

First: the Control Plane (Section 3.2 in the paper, expanded in §3.4.1 below), because every request enters through it and every instance reports to it. Understanding the global scheduler and cluster monitor establishes how load information propagates, how instance flipping decisions are made, and why certain scheduling decisions are centralized while others are decentralized.

Second: Prefill Instance internals — Scheduler, Length Predictor, and Chunked Prefill (Sections 3.3.1–3.3.3), because these are the first components that touch a request after global routing. The prefill scheduler determines request ordering, the length predictor generates the resource-usage estimate that governs all downstream decode scheduling, and chunked prefill is the mechanism that prevents prefill-prefill interference. These three modules run concurrently and their outputs feed the dispatcher.

Third: The Dispatcher and KV Cache Transfer (Section 3.3.4), because this is where the prefill instance makes the critical load-balancing decision that determines which decode instance will handle the request's decode phase. The dispatcher consumes the length prediction and decode-instance load information to execute the power-of-two algorithm, then initiates the network transfer that makes disaggregation physically possible.

Fourth: Decode Instance Scheduling (Section 3.4), because the decode instance is where the length prediction is actually used to prevent decode-decode interference through working-set-aware admission control. This section covers the two proposed policies (reserve-static and reserve-dynamic) and how they differ from vLLM's greedy baseline.

Fifth: Instance Flip (Section 3.5), because dynamic reallocation of instances between prefill and decode roles is the mechanism that makes the disaggregated architecture cost-effective under shifting workload mixes, closing the loop on why disaggregation does not simply double hardware requirements.

Sixth: Length Predictor Training (embedded in Section 3.3.2), because the prediction model is trained offline and deployed at all prefill instances, and its accuracy and granularity directly determine the effectiveness of every downstream scheduling decision.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems design paper whose core idea is that the three categories of LLM inference interference (prefill-prefill, prefill-decode, decode-decode) can be eliminated by restructuring the serving architecture around the distinct computational characteristics of the prefill and decode phases: fixed-size prefill chunking to cap compute saturation, full disaggregation to isolate batch from latency-critical work, and length prediction to enable memory-aware scheduling of heterogeneous decode requests. The paper does not propose new ML techniques; rather, it synthesizes and extends prior ideas (chunked prefill from Sarathi, disaggregation from Splitwise, PagedAttention from vLLM) into a unified system with novel scheduling algorithms and dynamic instance management.

3.4.1 Centralized Control Plane

The control plane is the coordination layer that manages the entire inference cluster. It is explicitly designed as a distributed system without single points of failure or processing bottlenecks, meaning its components can be replicated and sharded.

Global Scheduler. The global scheduler sits at the boundary between external services and the inference cluster. When a new inference request arrives, the global scheduler executes a three-step procedure. First, it selects a prefill instance using a least-load policy — it maintains load information for every prefill instance (received periodically from the cluster monitor) and routes the request to the instance with the smallest current load. The paper does not define "load" precisely at the global scheduler level, but the prefill instance reports queue depth and resource utilization, so the least-loaded instance is the one with the fewest queued requests or the lowest accelerator utilization. Second, it inserts the request into a request status table that stores each request's arrival time, current phase (prefill or decode), and any SLA requirements. This table is the source of truth for tracking request progress through the system. Third, as decode instances generate output tokens, the global scheduler receives these tokens in a streaming fashion and forwards them back to the external client that originated the request. The global scheduler is intentionally not involved in decode instance selection — that decision is decentralized to the prefill instances' dispatchers to avoid the global scheduler becoming a bottleneck.

Cluster Monitor. The cluster monitor manages the lifecycle of prefill and decode instances and serves as the information hub for load-aware scheduling. It performs two periodic functions. First, it collects load statistics from every prefill and decode instance at a regular interval — the paper specifies every 100ms. These statistics include metrics like queue depth, accelerator utilization, available memory, and the number of active requests. Second, it broadcasts aggregated decode-instance load information to all prefill instances, also on a regular cadence. This broadcast is critical because it enables the decentralized dispatcher at each prefill instance to make informed decode-instance selection decisions without querying a central service for every request. The cluster monitor also handles instance lifecycle operations: adding new instances when load increases, removing idle instances, and triggering instance flips (described in §3.4.5).

Design rationale. The paper centralizes global routing (which prefill instance gets each new request) but decentralizes decode-instance selection (each prefill instance chooses decode instances independently). This split is deliberate. Global routing is a lightweight operation — a single lookup per request — and centralizing it simplifies load balancing across prefill instances. Decode-instance selection, by contrast, requires detailed knowledge of each decode instance's current memory state and workload composition, which changes rapidly as tokens are generated. Decentralizing this decision to the prefill instances (which already hold the latest broadcast state) avoids the global scheduler becoming a serial bottleneck and eliminates an extra network hop per scheduling decision.

3.4.2 Prefill Scheduler

The prefill instance's local scheduler transforms the unordered stream of incoming requests into an ordered sequence for prefill processing. It maintains two queues: a raw request queue that holds requests as they arrive from the global scheduler, and a scheduled queue that holds sorted requests ready for chunked prefill.

Three scheduling policies. The paper implements and evaluates three non-preemptive policies:

FCFS (First-Come-First-Serve): Requests are scheduled in their original arrival order. This is the simplest policy, requires no sorting overhead beyond maintaining a FIFO queue, and works well when requests have similar prompt lengths. However, the paper notes that FCFS "can lead to head-of-line blocking and high average job completion time (JCT) when requests have long prompts," which is problematic because "the length differences among LLM inference requests are more than three orders of magnitude (see Figure 1)." A single 2000-token summarization prompt arriving slightly before twenty 18-token chat prompts would force all the chat prompts to wait behind the summarization prefill.

SJF (Shortest-Job-First): Requests are sorted by prompt token length in ascending order — the shortest prompts are prefilled first. By design, SJF minimizes average waiting time (a classic result from scheduling theory) because short jobs no longer wait behind long jobs. The paper reports that SJF "lowers average prefill waiting time by 7.8% compared to FCFS when the batch size is set to 16" (Figure 16). However, SJF introduces a starvation risk: if short prompts continuously arrive, long-prompt requests might never be scheduled. The paper mitigates this with a configurable PrefillSchedBatch parameter (described below).

LJF (Longest-Job-First): Requests are sorted by prompt token length in descending order — the longest prompts are prefilled first. The paper includes this policy for completeness but does not advocate for it strongly; the results in Figure 16 show that LJF performs worse than SJF in terms of TTFT CDF.

PrefillSchedBatch: starvation prevention. The paper introduces a batching mechanism specifically for the scheduling decision (distinct from the batching used during actual prefill execution). The variable PrefillSchedBatch controls how many requests from the raw queue are considered together in one scheduling round. For example, if PrefillSchedBatch = 16 and the raw queue has 20 requests, the scheduler takes the first 16, sorts them according to the chosen policy, and places them in the scheduled queue. The remaining 4 requests wait for the next scheduling round. This mechanism "prevents starvation during the prefill phase" because even under SJF, a long-prompt request will eventually be among the oldest requests in the raw queue and will be included in a scheduling batch where it is the shortest among the 16 considered. Increasing PrefillSchedBatch gives the scheduler a larger window to reorder requests — the paper shows in Figure 16b that increasing it from 16 to 128 under SJF "decreases average TTFT by 46.5%."

Relationship to chunked prefill. The scheduled requests are not fed directly to the LLM. Instead, they are first partitioned into fixed-size chunks by the chunked prefill mechanism (§3.4.4). The scheduler's job is only to determine the order of requests; the chunker determines how they are batched for GPU execution. This separation means the scheduler can optimize for JCT and fairness while the chunker independently optimizes for hardware utilization.

Sorting overhead. The paper explicitly measures the cost of request sorting, which uses Python's native sort API. The overhead "ranges from 10s to 100s of microseconds, which is negligible compared to millisecond-level or second-level TTFT latency." This justifies the use of SJF/LJF without concern for scheduler overhead.

3.4.3 Length Predictor

The length predictor is the component that enables all decode-phase interference mitigation. Without predicting how many tokens a request will generate, the system cannot know whether a request will be a light-decode (minimal memory pressure, fast completion) or heavy-decode (large KV cache growth, long occupancy). The prediction happens at the prefill instance, before the request reaches any decode instance, so that routing and scheduling decisions can be made proactively.

Design goal: predict length ranges, not exact lengths. The paper explicitly argues against predicting an exact token count. The primary reason is that "the latter is extremely difficult to predict" because LLM outputs are fundamentally non-deterministic — "various inference parameters, such as temperature and top-p, result in significant response variations from the same LLM model to the same question in practice." A secondary reason is that "an exact length estimation is unnecessary; a length range suffices" for the scheduling decisions the system needs to make. If the predictor can say "this request will generate roughly 200–400 tokens," the dispatcher and decode scheduler can estimate the lower and upper bounds of its memory usage, which is sufficient for admission control and load balancing.

Model architecture. The length predictor uses a small LLM-based classification model. In the paper's evaluation, this is OPT-125M (125 million parameters) configured as a sequence classification model — specifically, OPTForSequenceClassification from the HuggingFace Transformers library. The target model whose behavior it predicts is OPT-13B (13 billion parameters). The predictor is approximately 100× smaller than the target model, making it "roughly ten times faster than the larger one" in the paper's measurements. This size differential is critical: the predictor runs on the same GPU as the main LLM during prefill, so it must be cheap enough not to meaningfully steal compute resources from the primary workload.

Prediction granularity. The predictor classifies requests into fixed-size length buckets. The paper defines granularity as the bucket width in tokens. With granularity $G$ , responses with token lengths in $[0, G)$ are labeled 0, $[G, 2G)$ are labeled 1, $[2G, 3G)$ are labeled 2, and so on. The paper tests three granularities: 100, 200, and 400 tokens, achieving prediction accuracies of 58.9%, 74.9%, and 85% respectively. The default operational granularity is 200 tokens, balancing accuracy (roughly three-quarters of predictions correct) against scheduling utility (a 200-token range provides enough precision to meaningfully distinguish light from heavy decode). The paper notes that the granularity tradeoff is intuitive: "a smaller granularity means more accurate resource and performance estimation but lower accuracy in practice. A larger granularity means higher accuracy but essentially makes scheduling harder."

Training procedure. The predictor is trained offline in three steps (illustrated in Figure 8):

Dataset assembly. The authors take prompts from public datasets (specifically ShareGPT) and send them to the target LLM (OPT-13B). The target model generates responses using its standard inference configuration. For each prompt-response pair, the response's token length is measured and mapped to a bucket label based on the chosen granularity. The paper uses 75K training examples from ShareGPT.
Label pairing. Each prompt is paired with its bucket label, creating a supervised classification dataset where the input is the prompt text and the output is an integer bucket index.
Fine-tuning. The OPT-125M classification model is fine-tuned on this dataset using the HuggingFace Trainer API. The paper does not report training hyperparameters (learning rate, epochs, batch size), which is a notable omission. The output is a model that takes a prompt as input and predicts which length bucket the target LLM's response would fall into.

Execution mode: parallel vs. sequential. The paper evaluates two ways to run the predictor at the prefill instance. In sequential mode, the predictor runs first on a given prompt, and only after it produces a prediction does the main LLM begin prefill. This adds the predictor's full latency to the TTFT. In parallel mode, the main LLM and the predictor run concurrently on the same prompt — the main LLM begins chunked prefill immediately while the predictor simultaneously processes the prompt to generate the length estimate. The parallel mode may reduce the main LLM's throughput because both models compete for GPU compute and memory.

The paper chooses parallel mode as the default. Figure 17 quantifies the impact: when both models co-run with a padding limit of 512 tokens, "80% of large LLM's prefill requests remain unchanged compared to when it runs alone." The average prefill latency increases by 10%, and throughput drops by 12% under stress-test conditions. These numbers represent worst-case scenarios; the paper states that "the impact will be smaller in practice" and that "beefier hardware can further mitigate the drop."

Padding and batching for the prediction model. Unlike the main LLM, the small prediction model uses fixed-size batching rather than chunked prefill, because "this is due to the model's small size, which does not exhibit a clear compute-saturate threshold as seen in larger models." However, HuggingFace's sequence classification models require all inputs in a batch to be padded to the same length. To prevent short prompts from being padded to extreme lengths when batched with a very long prompt, the paper introduces a cutting limit: "requests higher than the limit will run alone." The default cutting limit is 512 tokens. The suffix in the notation "L+P512" in Figure 17 means the large model co-runs with the predictor, and any prompt longer than 512 tokens is run solo by the predictor rather than being batched. This prevents pathological cases where a single 2000-token prompt forces all other prompts in the batch to be padded to 2000 tokens.

Where the prediction is used. The length prediction flows to two downstream consumers. The prefill instance's dispatcher uses it during inter-decode instance scheduling to select a decode instance that has sufficient resources and where adding this request would cause minimal interference. The decode instance's local scheduler uses it during intra-decode instance scheduling to decide whether to admit the request to the current continuous batching loop based on predicted memory usage.

Alternative designs considered and rejected. The paper discusses two alternative placements for the length predictor. Running it at each decode instance would mean the prefill instance schedules blindly and only after transfer does the decode instance discover the request's length profile — by which point it is too late to route the request elsewhere, and interference (as measured in §2.2.3) is inevitable. Running it at the global scheduler would centralize all intelligence but "could make the global scheduler a bottleneck" because prediction requires a full forward pass through OPT-125M, which is far more expensive than the lightweight routing logic the global scheduler currently performs. The paper argues that the prefill-instance placement is "easier and simpler to reason about and deploy."

3.4.4 Chunked Prefill

Chunked prefill is TetriInfer's mechanism for eliminating prefill-prefill interference (§2.2.1) by ensuring that every prefill forward pass processes exactly the number of tokens that saturates the accelerator's compute capacity — no more, no less. The fundamental insight from Figure 2a is that prefill throughput stops increasing once the total number of tokens in a batch exceeds a model-and-hardware-specific threshold, but latency continues to rise linearly with additional tokens. Traditional fixed-batch-size systems (including vanilla vLLM) ignore this threshold and pack as many requests as possible into each batch, wasting latency on tokens that produce zero throughput benefit.

ChunkSize definition. The paper defines ChunkSize as the number of tokens at which the accelerator reaches its computation-saturated limit. This is an empirically measured property of a specific (model, hardware) pair. For OPT-13B on NVIDIA V100 GPUs in the paper's testbed, ChunkSize = 512 tokens. This value is visible in Figure 2a, where prefill throughput (tokens/s) plateaus at roughly 512 tokens while GPU utilization reaches near 100%. The paper states that "the accelerator and the LLM model architecture determine the ChunkSize," and that "models with larger hidden dimensions and accelerators with lower capabilities typically result in a smaller ChunkSize."

Slicing and merging algorithm. Given a set of scheduled requests (ordered by the prefill scheduler policy) and a target ChunkSize, the chunker performs two operations:

Slice: Each request's prompt is conceptually divided into ChunkSize-sized segments. A request with 1000 prompt tokens would be sliced into two chunks: a 512-token chunk and a 488-token chunk. A request with 150 prompt tokens remains as a single partial chunk.
Merge: The sliced segments from consecutive requests are concatenated to fill each chunk as close to ChunkSize as possible. Critically, "we first slice and then merge prompt tokens into fixed-size chunks without altering their order." The chunker does not reorder tokens — it respects the order established by the prefill scheduler — but it does pack tokens from multiple requests into the same chunk. For example, if the scheduled queue contains a 300-token request, a 200-token request, and a 400-token request, the chunker would produce: chunk 1 = 300 + 200 = 500 tokens (12 tokens short of ChunkSize, but the next request's first 12 tokens would not be placed here because chunk boundaries cannot split a token), chunk 2 = 400 tokens (still one request), and so on.
Pad: The final chunk in a scheduling batch may be partial. The paper specifies: "the final chunk in a batch could be partial, and we will pad it to ChunkSize with zeros." Zero-padding ensures the GPU's matrix multiplications operate on uniform-dimension tensors without adding any real computation (zeros multiplied by anything are zero, and modern GPU kernels can skip or efficiently handle zero-padded regions).

Execution model. After chunking, the main LLM engine "invoke[s] the main LLM model to execute prefill forward one chunk at a time." Each chunk triggers one forward pass through the model. The paper introduces a progress-tracking variable per request: "we maintain a simple variable per request that records the last prefilled token position." This variable is critical for the KV cache transfer described in §3.4.5 — it tells the dispatcher which tokens' KV cache entries have been computed and are ready for transfer.

Why ChunkSize = 512 and not larger. The choice of ChunkSize is an optimization decision. Setting it smaller than the saturate threshold would under-utilize the accelerator (throughput would be lower than achievable). Setting it larger would push into the oversaturated regime where latency grows without throughput benefit — exactly the interference the paper is trying to eliminate. The saturate threshold is the point of maximum throughput at minimum latency, making it the unique optimal operating point. The paper's contribution is not the observation that this threshold exists (that was established by prior work, including Pope et al. [33]) but rather the systematic use of this threshold as a fixed chunk size in a disaggregated serving architecture.

Contrast with Sarathi's mixed chunks. Sarathi [1] also proposed chunked prefill but ran prefill-decode-mixed chunks: a chunk might contain some prefill tokens and some decode tokens in the same forward pass. TetriInfer runs prefill-only chunks because the decode phase has been fully disaggregated onto separate instances. The paper argues this is a better design because the interference measurements in §2.2.2 show that co-running prefill and decode in the same batch (which Sarathi's mixed chunks inherently do) causes significant slowdowns. By using prefill-only chunks, TetriInfer guarantees that every forward pass on a prefill instance is purely computation-bound with predictable latency, and every forward pass on a decode instance is purely memory-bound with predictable latency.

Performance impact. The paper quantifies the benefit of chunked prefill in Figure 16. "Compared to vLLM's fixed batch mode, chunked prefill alone with FCFS improves latency by 86.4%." This is the improvement from switching from a batch-size-based prefill to a token-count-based prefill, even without changing the scheduling order. The additional benefit from SJF scheduling (7.8%) is on top of this. The 86.4% number is the paper's strongest single piece of evidence that prefill-prefill interference is both real and addressable.

3.4.5 Dispatcher and KV Cache Transfer

The dispatcher is the bridge between the disaggregated prefill and decode instances. It makes the critical routing decision that determines which decode instance will service each request's decode phase, and it orchestrates the physical transfer of the prefilled KV cache across the network.

Trigger condition. The dispatcher runs on an event-driven basis: "running whenever there are prefilled requests (or chunks)." Specifically, once a request's first prefill chunk has been processed by the main LLM, the dispatcher is invoked for that request. This is before all chunks of a multi-chunk request are complete — the first chunk's completion is sufficient because the dispatcher can begin making routing decisions and initiating transfer while subsequent chunks are still being prefilled. The paper discusses this as a transfer granularity tradeoff: chunk-level transfer enables overlapping prefill and network transfer, while request-level transfer (waiting for all chunks) minimizes the number of network round-trips. The implementation uses request-level transfer "for simplicity" and leaves chunk-level to future work, acknowledging that a concurrent work (Splitwise [32]) proposes layer-wise transfer that could be combined with chunk-level transfer.

Inter-decode load-balancing algorithm. The dispatcher selects a decode instance using a three-step decentralized algorithm:

Step 1: Filter by resource availability. The dispatcher partitions all known decode instances into two sets. Set $\alpha$ contains instances with sufficient resources to execute this request's decode phase. Set $\beta$ contains instances without sufficient resources. The resource calculation uses the predicted length range from the length predictor: given a predicted bucket (e.g., 200–400 tokens), the dispatcher estimates the request's KV cache memory footprint and checks whether each decode instance has at least that much available accelerator memory. The paper does not specify the exact memory estimation formula, but it would involve: $\text{required\_memory} \approx \text{upper\_bound\_tokens} \times \text{layers} \times \text{hidden\_dim} \times 2 \text{ (K and V)} \times \text{bytes\_per\_element}$ . The load information for all decode instances is available locally because the cluster monitor broadcasts it every 100ms.

Step 2: Power-of-two random selection. From set $\alpha$ , the dispatcher randomly selects two instances using the power-of-two choices algorithm (also known as the "power of two random choices" load-balancing technique). The paper cites NGINX's load-balancing documentation [25] for this technique. Power-of-two is a classic randomized algorithm that achieves exponential improvements in load balance over pure random selection while requiring only two random probes — it is asymptotically close to the optimal greedy (least-loaded) strategy while being far cheaper to implement.

Step 3: Interference-minimizing tiebreak. From the two randomly chosen instances, the dispatcher selects "the one that would encounter the least interference if the prefilled request is sent to it." The interference metric is based on the decode-decode interference patterns measured in §2.2.3, specifically the goal "to establish the lowest average ratio of heavy decode:light decode." In operational terms: if the current request is predicted to be a heavy decode, the dispatcher prefers the instance that currently has fewer heavy decode requests relative to light decode requests, so that heavy decode work is spread evenly. If the current request is a light decode, the dispatcher can route it to either instance without concern, or prefer the one with an existing light-decode majority to keep heavy-decode instances from becoming overloaded. The paper validates this algorithm in Figure 19, showing that it "achiev[es] the lowest total decoding time compared to other policies" and that it evenly balances the heavy-to-light request ratio across instances.

Network transfer abstraction. Once a decode instance is selected, the dispatcher transmits the request's metadata and prefilled KV cache. The paper designs a unified network transfer abstraction to handle the diversity of physical interconnects that may exist between prefill and decode instances in a real deployment. The abstraction exposes a uniform API — send, receive, read, write — and internally maps these calls to the appropriate low-level transport mechanism.

The paper classifies physical data links into three types (Figure 9):

Direct: Accelerators are connected by a high-speed link such as NVLink (900 GB/s bandwidth) or HCCS (Huawei's equivalent). Data can move directly between GPU/NPU memories without CPU involvement. The software stack uses low-level memory copy primitives (cudaMemcpy-style APIs [14, 26]) or collective communication libraries (NCCL [28]).
Direct-NIC: Accelerators communicate through their companion Network Interface Cards (NICs) without going through CPU DRAM. Technologies like NVIDIA GPUDirect RDMA [27] enable this. The software stack uses InfiniBand Verbs or similar RDMA APIs. Example bandwidth: 200 Gbps for ConnectX-6 adapters. This corresponds to the TS-RoCE configuration in the evaluation.
Indirect: No direct accelerator-to-accelerator or accelerator-to-NIC path exists. Data must be copied from accelerator memory to CPU DRAM, then transmitted over the network (e.g., TCP sockets), then copied from CPU DRAM to the remote accelerator memory. This incurs at least two extra memory copies. The paper's current implementation "only supports the Indirect type using sockets" due to limited access to high-end hardware.

The abstraction also categorizes software stacks as one-sided (the sender accelerator can directly write to the receiver accelerator's memory without involving the receiver's CPU, analogous to RDMA semantics) or two-sided (both sender and receiver CPUs are involved in the transfer, analogous to traditional socket-based communication). The paper notes that "accelerators like GPU or NPU can do one-sided memory access as they have low-level primitives such as direct memory copies between devices" [14, 26], but one-sided operation raises "typical challenges associated with building large-scale RDMA-based memory systems" [10, 12] that are left unexplored.

Mock mechanism for evaluation. Because the paper's testbed lacks high-speed interconnects, the evaluation uses an emulation layer: "for a given set of requests, we initially run their prefill phase offline to obtain their prefilled KV cache. Before testing, we load these prefilled KV caches into the decode instance's local memory. When testing starts, the prefill instance transmits only the request metadata to the decode instance, excluding the actual prefilled KV cache. Subsequently, the decode instance calculates the latency of the KV cache transfer and waits accordingly." The transfer latency is computed as $\text{latency} = \text{KV\_cache\_size\_bytes} / \text{emulated\_bandwidth}$ , where the KV cache size depends on the model architecture and the prompt length, and the bandwidth is set to the emulated hardware configuration (200 Gbps for RoCE, 300 GB/s for NVLink). This mechanism allows the paper to evaluate disaggregation performance under different interconnect assumptions without physically possessing those interconnects.

Transfer granularity discussion. The paper identifies a tradeoff between chunk-level and request-level transfer. Chunk-level transfer sends each prefilled chunk's KV cache as soon as it is generated, overlapping communication with the computation of subsequent chunks. This reduces end-to-end latency but increases the number of network messages. Request-level transfer aggregates all chunks and sends them in a single transfer, minimizing network overhead but adding serialization latency. The paper notes that combining chunk-level transfer with Splitwise's layer-wise transfer "could further optimize compute and network parallelization." The current implementation uses request-level transfer "for simplicity," which means the decode instance cannot begin decoding until all prefill chunks for a request are complete and transferred — a potential source of latency for very long prompts.

3.4.6 Decode Instance and Intra-Decode Scheduling

The decode instance is where the actual token generation happens. It receives prefilled KV caches and request metadata from any prefill instance, groups requests using continuous batching, and runs the auto-regressive decode loop. The key innovation at the decode instance is the local scheduler, which uses the predicted generation length to make working-set-aware admission decisions.

Receiver module and queuing. When a prefill instance's dispatcher initiates a KV cache transfer, the target decode instance's receiver module (part of the unified network transfer abstraction) accepts the connection and waits for the complete prefilled KV cache to arrive. Once received, the request — now consisting of its metadata (including the predicted length bucket) and its KV cache — is "added to the local scheduler's queue."

Continuous batching with PagedAttention. The decode instance uses vLLM's PagedAttention mechanism [21] for KV cache management. Instead of reserving a contiguous block of accelerator memory equal to the maximum possible context length for each request (which wastes enormous amounts of memory for short-generation requests), PagedAttention allocates KV cache in fixed-size pages. As the decode phase generates new tokens, new pages are allocated on demand. This enables much higher batch sizes because memory is not stranded in over-provisioned per-request reservations.

vLLM's greedy scheduling baseline. Vanilla vLLM's scheduler is greedy: "as long as the accelerator has spare memory, it will add requests to the current iteration." When a decode step completes and frees resources (either because a request finished generating or because a page was evicted), the scheduler immediately fills the freed memory with waiting requests. The problem, which the paper explicitly identifies, is that this greedy policy "may run out of memory in future iterations and cause thrashing. Fundamentally, it is oblivious to the working set size." A request that will generate 2000 tokens will continuously consume new KV cache pages throughout its lifetime. If the scheduler admits such a request when free memory is just barely sufficient for its current state, the system will inevitably run out of memory mid-generation, triggering expensive page evictions or even request preemption.

Reserve-static policy. The reserve-static policy addresses the memory obliviousness of greedy scheduling by incorporating the predicted generation length into admission decisions. The rule is:

"a request is scheduled only if its predicted memory usage is smaller than the available accelerator memory for the current iteration."

Here, "predicted memory usage" is estimated from the predicted length bucket's lower bound. If the predictor says a request will generate 200–400 tokens, the system reserves $\text{MemoryEstimate}(200)$ — the KV cache footprint for 200 tokens — as the minimum guaranteed memory the request will need. Using the lower bound is conservative: it guarantees that even if the predictor is wrong and the request generates fewer tokens than predicted, the reserved memory is sufficient for the near term. The request may exceed this reservation if the true generation length is longer, but by then other requests may have completed and freed memory. The rule ensures that at the moment of admission, there is enough headroom for the request's minimum plausible lifetime.

Reserve-dynamic policy. The reserve-dynamic policy is more sophisticated. Instead of only checking current available memory, it "takes a more proactive approach by considering the predicted number of remaining tokens." Specifically:

"a new request is added to the scheduled batch only if there is still spare memory when the shortest remaining job in the batch finishes."

This policy models the future memory state by looking at the currently running decode requests. Among all active requests, it identifies the one with the fewest remaining tokens to generate (estimated from the predicted length bucket's upper bound minus tokens already generated). It then checks: if that shortest request finishes (freeing its KV cache pages), will there be enough memory to accommodate this new request for the duration until that point? If yes, the request is admitted; if no, it waits. This policy "effectively mitigates memory thrashing while maximizing the advantages of paging."

Estimation using the predicted length range. Both policies use the predicted length range (not the raw bucket label) for resource estimation. Given a request predicted to fall in bucket $[L_{\text{lower}}, L_{\text{upper}})$ , the system can compute:

$\text{KV\_cache\_size\_lower} \approx L_{\text{lower}} \times \text{layers} \times \text{hidden\_dim} \times 2 \times \text{bytes\_per\_element}$

and similarly for the upper bound. The reserve-static policy uses the lower bound for the admission check. The reserve-dynamic policy uses the upper bound (minus tokens generated so far) to estimate the "remaining tokens" for the shortest-job calculation, because it wants to know the worst-case time until a running request finishes.

Performance of working-set-aware policies. The paper's evaluation in Figure 18 reveals a nuanced story. With the actual prediction accuracy of 74.9% (at granularity 200), both reserve-static and reserve-dynamic perform "on par with vLLM's greedy policy" — they do not hurt performance but do not improve it either. However, under an ideal prediction accuracy of 100%, "reserve-dynamic and reserve-static improve average JCT by 12% and 10%, respectively." This gap between actual and ideal performance indicates that the scheduling policies' effectiveness is gated by prediction accuracy. The paper treats this as motivation for improving the predictor rather than as a weakness of the scheduling algorithms: the algorithms are designed correctly, but they cannot outperform greedy scheduling without reliable input about which requests are memory-heavy.

Why two policies rather than one? The paper does not explicitly argue for having both reserve-static and reserve-dynamic, but the design implies complementary strengths. Reserve-static is simpler, has less computational overhead (one check per admission decision), and is sufficient when the workload's generation lengths are relatively uniform. Reserve-dynamic is more complex but better handles heterogeneous workloads where short-generation and long-generation requests are mixed — the "shortest remaining job" lookahead prevents long-generation requests from starving short ones by ensuring that new admissions do not block memory that short requests will need.

3.4.7 Instance Flip

Instance flipping is the mechanism that makes the disaggregated architecture economically viable under dynamically shifting workload mixes. Without flipping, provisioning dedicated prefill and decode instances would require over-provisioning both pools to handle peak load independently, roughly doubling hardware requirements. With flipping, the system can convert idle prefill instances to decode instances (or vice versa) as workload characteristics change, maintaining the isolation benefits of disaggregation while using resources efficiently.

Policy: when to flip. The centralized control plane runs a transition watcher module that "regularly checks load and decides whether certain instances should be flipped." The paper describes a pluggable policy framework and gives one example: "flipping an instance if its load has been under 10% for the past minute." This is a hysteresis-based approach: load must be persistently low (not a transient dip) before an instance is repurposed, preventing oscillation. The paper does not specify the checking interval, but given the cluster monitor's 100ms statistics collection cadence, it is likely on the order of seconds.

Mechanism: prefill-to-decode flip. The steps are illustrated in Figure 10 (left side):

S1: Notify. The global scheduler sends a flip request to the selected prefill instance.
S2: Drain. The global scheduler stops forwarding new requests to this prefill instance. The prefill instance continues processing all queued requests and completes any in-flight KV cache transfers to decode instances. It waits until all queued requests are fully prefilled and dispatched.
S3: Flip. The instance changes an internal variable that switches its role from prefill to decode — crucially, "without restarting the process or reloading models." Both prefill and decode instances run the same model weights (OPT-13B); only the execution mode differs. The flip time is roughly 5 to 7 ms (excluding the drain time, which depends on queue depth).
S4: Reply. The instance informs the global scheduler that it is now available as a decode instance.

Mechanism: decode-to-prefill flip. The steps are slightly more complex because multiple prefill instances may have in-flight transfers to the target decode instance (Figure 10, right side):

S1: Notify. The global scheduler notifies the selected decode instance of the impending flip and simultaneously notifies all prefill instances to stop forwarding new requests to this decode instance.
S2 (at prefill instances): Prefill instances mark this decode instance as unavailable in their local load tables, so future dispatcher decisions will route requests elsewhere.
S3 (at the decode instance): The decode instance finishes all currently running decode requests (the "drain" period) and then flips its internal role variable to prefill.
S4: Reply. The instance informs the global scheduler that it is now available as a prefill instance.

Cost model. The paper emphasizes that flipping is fast because it is a role change, not a model reload: "it involves changing an internal variable without restarting the process or reloading models." This is possible because both prefill and decode instances run the same model and the same codebase (TetriInfer instances are based on vLLM); only the scheduling policies and execution paths differ. The drain time — waiting for queued requests to complete — is the dominant cost, and it depends on the workload. For decode-to-prefill flips, this could be long if heavy-decode requests with many remaining tokens are in flight.

Why this matters for the interference argument. Instance flipping closes the economic argument for disaggregation. A critic could say: "disaggregating prefill and decode doubles your hardware, so any performance improvement is offset by 2× cost." The paper's response, supported by the resource usage time metrics in Figures 11–15, is that disaggregated instances complete work faster (by eliminating interference), so the total resource-time product is comparable to or better than coupled serving for most workloads. Instance flipping ensures that during periods when one phase dominates (e.g., a summarization-heavy hour with lots of heavy prefill but little decode), idle instances are repurposed rather than wasted.

Summary of Design Choices and Their Justifications

Prefill-only chunks (not mixed prefill-decode chunks): justified by the prefill-decode interference measurements in §2.2.2 showing that co-running these phases causes 5× slowdowns; mixed chunks (Sarathi's approach) would inherit this interference.

Disaggregation with KV cache transfer (not in-place phase switching): justified by the need for independent scaling and specialized scheduling per phase; the transfer overhead (which the paper acknowledges as the primary cost) is offset by the elimination of prefill-decode interference for all workloads except HPHD.

Length-range prediction (not exact token prediction): justified by the inherent non-determinism of LLM generation and the sufficiency of range estimates for memory-aware scheduling; the 74.9% accuracy at granularity 200 is sufficient for the scheduling policies to match greedy performance, with headroom to improve JCT as accuracy increases.

Power-of-two decentralized load balancing (not centralized or least-loaded): justified by the need to avoid the global scheduler becoming a bottleneck, combined with the asymptotic optimality of power-of-two choices for load balancing; the interference-minimizing tiebreak adds a workload-specific optimization layer.

Reserve-static and reserve-dynamic (not greedy admission): justified by the decode-decode interference measurements in §2.2.3; greedy admission is oblivious to future memory pressure, while the working-set-aware policies prevent the thrashing that occurs when memory-heavy and memory-light requests are blindly co-scheduled.

Instance flipping (not static provisioning): justified by the cost argument: disaggregation would be economically non-viable without the ability to dynamically rebalance prefill and decode capacity; the fast flip mechanism (5–7ms plus drain time) makes rebalancing practical at cloud scale.

Unified network abstraction (not point-to-point hardcoded transfers): justified by the diversity of deployment environments; the abstraction allows TetriInfer to use the best available interconnect (NVLink, RoCE, TCP) without application-level code changes, future-proofing the system for hardware evolution.

4. Key Insights and Innovations

Innovation 1: A Taxonomy of LLM Inference Interference as a Design Principle

The paper's most foundational contribution is not any single mechanism but the systematic interference taxonomy itself — the recognition that LLM inference performance degradation under mixed workloads decomposes into three causally distinct categories, each requiring a structurally different solution. Prior work had observed that performance suffers when different request types mix, but treated the phenomenon as an amorphous "resource contention" problem addressed through better memory management (vLLM), faster kernels (FlashAttention), or generic scheduling optimizations (Orca's continuous batching). What those approaches missed — and what TetriInfer's Section 2.2 makes explicit through careful controlled experiments — is that the cause of each interference category is different, and therefore the solution must be different.

The three categories are:

Prefill-prefill interference (Section 2.2.1, Figure 3): caused by adding computation-heavy jobs to an already computation-saturated accelerator. The throughput curve is flat past the saturate threshold, but latency continues to rise linearly with each additional token. This is a batching granularity problem — the system is packing too many tokens into a single forward pass.
Prefill-decode interference (Section 2.2.2, Figure 4): caused by co-scheduling batch (prefill) and latency-critical (decode) jobs on the same hardware. A single heavy prefill can delay decode iterations by 5×. This is a workload isolation problem — two job types with fundamentally different optimization objectives should not share an accelerator.
Decode-decode interference (Section 2.2.3, Figure 5): caused by memory bandwidth and capacity contention when heavy-decode requests (large, growing KV caches) share batch slots with light-decode requests. The 16% throughput drop when mixing heavy and light decode is a scheduling awareness problem — the scheduler is blind to per-request resource usage.

What makes this taxonomy intellectually distinctive is that it moves the field beyond "LLM inference is slow, let's optimize kernels" toward "LLM inference has multiple distinct resource pathologies, and each needs its own architectural response." The taxonomy is diagnostic: it identifies root causes, not symptoms. It is actionable: each category implies a specific design requirement (cap prefill batch sizes at the saturate threshold; isolate prefill from decode; make decode scheduling working-set-aware). And it is falsifiable: each category is supported by specific controlled experiments with measured magnitudes.

This is a conceptual advance rather than a mechanistic one. The paper does not invent the idea that prefill is compute-bound and decode is memory-bound — that was known from Pope et al. (2023, reference [33]) and others. The contribution is showing that because these phases have different resource profiles, mixing them causes specific, severe, and categorically distinct forms of interference, and that existing systems were implicitly suffering from all three without recognizing them as separate phenomena. This is analogous to how the database community benefited from Gray et al.'s taxonomy of transaction isolation anomalies: naming and distinguishing the categories enables principled solutions.

Innovation 2: Disaggregation as a Structural, Not Incremental, Solution to Phase Interference

The paper's second major insight is that prefill-decode interference cannot be adequately addressed by better scheduling within a coupled engine — it requires physical separation into independently scheduled instances. This is a structural claim, not an incremental optimization, and it represents a departure from the dominant deployment paradigm at the time of writing.

To understand why this is distinctive, consider what the field was doing before. The state-of-the-art serving systems — vLLM, Orca, and their derivatives — all used continuous batching to interleave prefill and decode forward passes within a single GPU engine. The intellectual premise of continuous batching was: "since both prefill and decode require forward passes through the same model, we should batch them together to maximize accelerator utilization and amortize the cost of loading model weights." This premise is correct for throughput in a simplified model where all operations are identical. It is wrong when prefill and decode have dramatically different resource signatures — prefill being compute-bound and latency-tolerant (a batch job), decode being memory-bound and latency-sensitive (an interactive task).

The interference measurements in Figure 4 make the case empirically. The 5× decode slowdown from co-running with a single heavy prefill is not a small scheduling inefficiency that could be tuned away with better priorities or time-slicing. It is a structural consequence of forcing a latency-critical decoding step to wait while a computation-heavy prefill consumes the accelerator. No amount of scheduling cleverness within a coupled engine can eliminate this wait — the prefill work must be done somewhere, and if it shares the same accelerator as decode, one of them will be blocked.

The disaggregation design in TetriInfer (Section 3.5) responds to this diagnosis with a structural separation: prefill instances run only prefill, decode instances run only decode, and the KV cache transfer over the network is the cost of maintaining this separation. The paper is honest that this introduces overhead — KV cache transfer latency and the need for more total instances — and that this overhead can dominate when both prefill and decode are heavy (the HPHD workload, Figure 14, where TetriInfer's perf/ $is only marginally better than vLLM's). But for most realistic workloads, the interference eliminated is larger than the transfer overhead introduced, yielding the 2.4× perf/$ improvement on LPHD (Figure 12).

The conceptual move here is architectural: TetriInfer treats prefill and decode not as two phases of one job but as two different jobs that happen to share a model. This reframing enables independent scaling, independent scheduling policies, and independent failure domains — exactly the benefits that microservices architectures brought to monolithic web applications. The concurrent Splitwise paper (Patel et al., 2023) made a similar structural choice, confirming that disaggregation was an idea whose time had come. TetriInfer's contribution to this line of thinking is showing that disaggregation is necessary to eliminate a specific category of interference (prefill-decode), not merely a potential optimization.

Innovation 3: Length Prediction as a Scheduling Primitive for Memory-Aware Decode Batching

The paper's third conceptual contribution is elevating generation length prediction from a curiosity or an auxiliary task to a first-class scheduling primitive that governs routing and admission decisions throughout the system. Prior work had explored length prediction (Zheng et al., 2023, reference [48], used a large LLM to predict response length for sequence scheduling), but TetriInfer integrates it systematically into a disaggregated architecture and shows that it addresses the previously undiagnosed decode-decode interference problem.

What makes this distinctive is the design decision to predict length ranges rather than exact lengths, and the justification for why ranges suffice. An exact-length predictor faces a fundamental accuracy ceiling because LLM generation is non-deterministic — temperature, top-p sampling, and the inherent stochasticity of the model mean that the same prompt produces different-length responses on different runs. Insisting on exact prediction would either yield low accuracy (making the prediction useless for scheduling) or require an impractically large model (making the prediction too expensive to run at inference time). By predicting buckets (e.g., 200–400 tokens), the paper achieves 74.9% accuracy with a model that is ~100× smaller than the target LLM, and this accuracy is sufficient to inform scheduling decisions: the lower bound of the predicted range provides a conservative memory reservation, while the upper bound provides a worst-case completion estimate.

The integration of length prediction into the system's scheduling fabric is what elevates this from "we built a length predictor" (incremental) to "length prediction is a scheduling primitive for heterogeneous decode workloads" (fundamental). The prediction feeds into three downstream decisions:

Inter-decode load balancing (Section 3.3.4): the dispatcher uses predicted length to spread heavy-decode requests evenly across instances, preventing the hotspots measured in Figure 19 where imbalanced heavy-decode concentrations cause disproportionate slowdown.
Intra-decode admission control (Section 3.4): the reserve-static and reserve-dynamic policies use predicted memory footprint to decide whether to admit a new request, avoiding the thrashing that occurs when greedy policies fill memory with requests whose future KV cache growth will exceed available headroom.
Instance provisioning (implicit): the cluster monitor's aggregate statistics on predicted decode lengths across the request mix inform instance flipping decisions — a surge of predicted-heavy-decode requests signals the need for more decode instances.

The paper is transparent about the current limitation of this approach: with 74.9% prediction accuracy, the working-set-aware scheduling policies only match (not beat) vLLM's greedy policy (Figure 18). But the ideal-accuracy results — 12% JCT improvement — demonstrate that the scheduling algorithms are correctly designed and that the bottleneck is prediction accuracy, not scheduling logic. This frames length prediction accuracy as the key gating factor for future improvements in decode-phase inference efficiency, redirecting research attention toward better prediction models rather than more sophisticated scheduling heuristics.

Innovation 4: Instance Flipping as the Economic Enabler of Disaggregation

The paper's fourth conceptual contribution is recognizing that disaggregation's primary objection — "it doubles your hardware" — can be addressed through dynamic role reassignment without model reloading, and demonstrating that this mechanism closes the cost-effectiveness argument for most workloads. This is not a deep technical innovation in mechanism (the flip itself is a 5–7ms variable change) but rather a pragmatic insight about system economics that makes disaggregation operationally viable.

The argument is implicit in the paper's structure but becomes clear when examining the resource usage time metrics in Figures 11–15. TetriInfer runs prefill and decode on separate instances, so naively it uses 2× the GPU-seconds of a coupled system. But the elimination of interference means each phase completes faster — often nearly 2× faster — so the total resource-time product (GPU count × wall-clock time) is comparable or better than the coupled baseline. For LPHD workloads (Figure 12), TetriInfer uses 38% fewer total hardware resources (in resource-time) while simultaneously delivering 97% lower TTFT and 47% lower JCT. Even for mixed workloads (Figure 15), resource usage drops by 21%.

Instance flipping (Section 3.5) amplifies this efficiency by preventing idle capacity. In a coupled system, imbalance between prefill and decode load is absorbed within each instance, which means GPU resources are shared but also means neither phase can be optimized independently. In TetriInfer, if the workload shifts toward heavy-decode, idle prefill instances flip to become decode instances, maintaining the isolation benefit without stranding hardware. The paper's chosen flip policy — "under 10% load for the past minute" — is a simple hysteresis rule, but the mechanism supports more sophisticated policies based on predicted workload mix.

What makes this intellectually interesting is that it reframes the disaggregation cost debate from "does disaggregation provide enough performance benefit to justify 2× hardware?" to "does disaggregation with dynamic flipping provide net cost savings after accounting for interference elimination?" The paper's data says yes for LPLD, LPHD, HPLD, and Mixed workloads, and says no for HPHD — and the HPHD negative result (Figure 14, perf/$ only 1.1×) actually strengthens the argument because it identifies the exact boundary condition where the economics flip. This is systems thinking at the level of total cost of ownership, not just performance metrics.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses the ShareGPT dataset [35], a public collection of conversational interactions with LLMs. The dataset provides the input prompt and generated response lengths for each request, which the paper uses to construct workloads following the length distributions illustrated in Figure 1. For the length predictor's training, 75K training examples are drawn from ShareGPT. For end-to-end workload construction, requests are sampled from ShareGPT and categorized along the prefill length and decode length dimensions: prefill requests with more than 512 prompt tokens are labeled heavy prefill, others light prefill; decode requests with more than 128 generated tokens are labeled heavy decode, others light decode — the 128-token threshold is chosen because it is "ShareGPT answers' median length." The paper also draws summarization and writing workload distributions from additional public datasets for characterization (Figure 1) but uses ShareGPT for all evaluation.
Base model(s). The main LLM under test is OPT-13B (Open Pre-trained Transformer, 13 billion parameters) [47], deployed with tensor parallelism (TP=2) across the testbed's GPUs. The paper states that OPT-13B is chosen as a representative large model for the available hardware. The length prediction model is OPT-125M, configured as a sequence classification model (OPTForSequenceClassification from HuggingFace Transformers [16]), fine-tuned to predict the length bucket of responses that OPT-13B would generate. The prediction model is roughly 100× smaller than the target model.
Metrics. The paper reports three primary metrics, each measured as both averages and cumulative distribution functions (CDFs) across requests:
- Time-to-First-Token (TTFT): the latency from when a request arrives until its first output token is generated. This captures prefill-phase performance (including queuing, scheduling, chunked prefill execution, and — for TetriInfer — KV cache transfer latency).
- Job Completion Time (JCT): the total latency from request arrival until the final output token is generated. This captures end-to-end performance including both prefill and decode phases, making it sensitive to decode-phase interference and scheduling.
- **Performance per Dollar (perf/ $):** a composite efficiency metric, computed as the inverse of **resource usage time** — the aggregated wall-clock time across all GPU instances used to process a workload. If a workload uses one prefill instance for 2 seconds and one decode instance for 3 seconds, the resource usage time is 5 seconds. For vLLM (which couples prefill and decode), resource usage time equals the total runtime since a single instance handles both phases. Perf/$ improvement is calculated as the ratio of vLLM's resource usage time to TetriInfer's resource usage time for the same workload. This metric captures whether TetriInfer's disaggregation genuinely reduces total hardware occupancy or merely shifts work between instances.
Baselines. The paper compares TetriInfer against a single baseline system: vanilla vLLM [21], the state-of-the-art LLM serving system at the time. vLLM uses continuous batching to interleave prefill and decode in a single engine, PagedAttention for efficient KV cache memory management, and a greedy scheduling policy that admits requests whenever spare accelerator memory is available. TetriInfer is implemented atop vLLM and inherits PagedAttention, meaning the comparison isolates the effects of chunked prefill, disaggregation, and two-level scheduling rather than confounding them with memory management differences. The paper does not compare against Orca [45], Sarathi [1], Splitwise [32], or FastServe [41] in experiments, though these systems are discussed in related work (Table 1). This is a notable omission: direct experimental comparison with Sarathi (which also uses chunked prefill but keeps prefill and decode coupled) would isolate the benefit of disaggregation specifically, and comparison with Splitwise (which disaggregates but without chunked prefill or length-aware scheduling) would isolate the benefit of those additional mechanisms.
Generation budget / compute accounting. The paper measures compute in terms of resource usage time (GPU-seconds across all instances), as described above. There is no explicit generation budget in the sense of "N tokens generated per request" — the workloads are defined by their length distributions, and all systems process the same set of requests to completion. For the KV cache transfer cost, the paper uses a mock mechanism (Section 4): "for a given set of requests, we initially run their prefill phase offline to obtain their prefilled KV cache. Before testing, we load these prefilled KV caches into the decode instance's local memory. When testing starts, the prefill instance transmits only the request metadata to the decode instance, excluding the actual prefilled KV cache. Subsequently, the decode instance calculates the latency of the KV cache transfer and waits accordingly." The transfer latency is computed as KV_cache_size_bytes / emulated_bandwidth. Two emulated hardware configurations are evaluated: TS-RoCE (200 Gbps RoCE, representing Direct-NIC) and TS-NVLink (300 GB/s NVLink, representing Direct). The base testbed uses Indirect (socket-based) transfer. This means the transfer latency is modeled, not measured on real high-speed interconnects, making it a parameterized simulation rather than an empirical measurement for the disaggregated configurations.
Cross-validation / statistical protocol. The paper does not describe any cross-validation or statistical significance testing. Results are presented as CDFs across requests (e.g., Figures 11–15), which implicitly show the distribution of per-request latencies, but no confidence intervals, error bars, or statistical tests are reported. The test set sizes are 128 requests for each workload type in the end-to-end evaluation (LPLD, LPHD, HPLD, HPHD, Mixed). For microbenchmarks, sizes vary: 256 requests for intra-decode scheduler evaluation (Figure 18), 32 requests per decode instance for inter-decode scheduling (Figure 19), and unspecified numbers for the prefill scheduler and length predictor microbenchmarks.

Main Quantitative Results

The evaluation is organized into two parts: end-to-end performance (Section 5.1), which compares TetriInfer against vLLM across five workload types, and microbenchmarks (Section 5.2), which isolate the performance of individual components.

End-to-End: Light Prefill, Light Decode (LPLD)

This workload represents conversational chat: short prompts (~18 tokens median), moderate-length responses (~128 tokens median). The test uses 128 requests.

Headline results (Figure 11): TetriInfer reduces average TTFT by 44% and average JCT by 40% compared to vLLM, for both emulated hardware configurations (TS-NVLink and TS-RoCE). Resource usage time is comparable to vLLM — the paper states that "despite using twice the number of hardware cards, TetriInfer completes tasks almost twice as fast, resulting in resource usage time that is comparable to the vanilla vLLM." The perf/$ improvement is 1.4×.

Interpretation: On chat workloads, which are the most common LLM serving scenario, TetriInfer's benefits come primarily from chunked prefill (reducing prefill-prefill interference when multiple small prompts arrive simultaneously) and disaggregation (preventing decode steps from being blocked by prefill batches). The TTFT CDF in Figure 11a shows that TetriInfer's distribution is shifted left and compressed — both median and tail latencies improve. The comparable resource usage time is significant because it addresses the intuitive objection that disaggregation should cost 2×: the speedup from interference elimination nearly offsets the extra hardware.

End-to-End: Light Prefill, Heavy Decode (LPHD)

This workload represents content creation: short prompts, long generated responses (>128 tokens, often 1000+ tokens). The test uses 128 requests. This is TetriInfer's strongest result.

Headline results (Figure 12): TetriInfer reduces average TTFT by 97% and average JCT by 47%. It uses 38% less total hardware resources (resource usage time). The perf/$ improvement is 2.4×.

Why TTFT improves by 97% despite short prompts: The paper explains: "this is because vLLM's prefill incurs serious interference while running prefill and decode requests in the same batch; in contrast, TetriInfer disaggregates them into separate instances." Even though the prefill itself is light (short prompts), in vLLM it must share the GPU with ongoing heavy decode requests from earlier submissions — the same interleaving that Figure 4 showed causes 5× decode slowdown also delays new prefill requests from executing. TetriInfer's prefill instances never run decode work, so a new light-prefill request is processed immediately by a dedicated prefill instance without waiting for in-progress decode steps.

Why JCT improves by 47%: The heavy decode phase benefits from TetriInfer's variable decode batch sizes (vLLM uses fixed batch sizes for decode), which allow more efficient packing, and from the elimination of prefill-decode interference. The 38% resource reduction is the paper's strongest evidence that TetriInfer's disaggregation can reduce total GPU occupancy, not just shift it between phases. The combination of 97% TTFT improvement, 47% JCT improvement, and 38% resource reduction producing a 2.4× perf/$ gain makes LPHD the workload where TetriInfer's architecture is most clearly justified.

End-to-End: Heavy Prefill, Light Decode (HPLD)

This workload represents summarization or prompt engineering: long prompts (>512 tokens), short generated responses (<128 tokens). The test uses 128 requests.

Headline results (Figure 13): TetriInfer improves average TTFT by 9% and average JCT by 23%. However, resource usage time increases by 43%, meaning TetriInfer uses more total GPU-seconds than vLLM. The perf/$ is 14% worse than vLLM (i.e., vLLM outperforms TetriInfer on this metric).

Why the gains are small and costs are high: Heavy prefill means the KV cache is large — for a 512+ token prompt, the prefilled KV cache for OPT-13B is substantial. TetriInfer must transfer this entire cache over the network to a decode instance, which adds latency and network bandwidth consumption. The paper identifies this as one of two challenges for HPLD: "(a) large prefilled KV caches and (b) the main LLM may be impacted by the prediction model (roughly 10% as shown in Figure 17)." The prefill-prefill interference that chunked prefill eliminates is still beneficial (hence the 23% JCT improvement), but the KV cache transfer overhead is large enough that total resource usage increases. This is the first workload where disaggregation's costs begin to outweigh its benefits.

End-to-End: Heavy Prefill, Heavy Decode (HPHD)

This workload represents the worst case for disaggregation: long prompts, long generated responses. Both phases are resource-intensive.

Headline results (Figure 14): TetriInfer improves average TTFT — the paper states "TetriInfer's TTFT improvement is more pronounced because we disaggregated heavy decode from prefill, akin to Figure 12" — and improves average JCT by 19%. Resource usage increases by 7%. Perf/$ improves by 1.1× — essentially breakeven.

Why this is TetriInfer's worst workload: The paper states explicitly in the Takeaways: "TetriInfer's design is not ideal for HPHD workloads as the room for improvement is small, and the overhead we introduce cannot be offset." Both phases are heavy: prefill must chunk and transfer large KV caches, decode must generate many tokens. The interference that disaggregation eliminates (prefill-decode co-scheduling) was never the dominant bottleneck for this workload because each request is already so resource-intensive that the GPU is saturated regardless of how phases are scheduled. The transfer overhead — proportional to the large prefill KV cache — is simply deadweight. The 1.1× perf/$ improvement is within measurement noise and does not justify the architectural complexity.

End-to-End: Mixed Workload

This workload randomly samples requests from ShareGPT, representing a realistic multi-tenant deployment where all request types arrive interleaved. The test uses 128 requests.

Headline results (Figure 15): TetriInfer reduces average TTFT by 85% and average JCT by 50%. Resource usage time decreases by 21%. Perf/$ improves by 1.9×.

Interpretation: The mixed workload is the most ecologically valid test because it closely approximates production conditions. The 85% TTFT improvement and 50% JCT improvement are substantial and demonstrate that TetriInfer's benefits are not limited to artificially constructed homogeneous workloads. The 21% resource reduction is particularly compelling: for a realistic mixed workload, disaggregation with instance flipping genuinely reduces total GPU occupancy, not just redistributes it. The 1.9× perf/$ improvement means that a cloud provider running TetriInfer could serve the same request volume with roughly half the hardware cost compared to vLLM.

Microbenchmark: Prefill Scheduler

The paper evaluates the prefill scheduler by comparing TetriInfer's chunked prefill with FCFS, SJF, and LJF policies against vanilla vLLM's fixed-batch-size prefill. Tests use OPT-13B with TP=2, ChunkSize = 512, and vLLM's batch size set to 16.

Headline results (Figure 16):

Chunked prefill alone (FCFS policy): "Compared to vLLM's fixed batch mode, chunked prefill alone with FCFS improves latency by 86.4%." This isolates the benefit of fixed-size token-count-based batching over fixed-batch-size batching, even without intelligent request ordering.
SJF over FCFS: "SJF lowers average prefill waiting time by 7.8% compared to FCFS when the batch size is set to 16." This is the marginal benefit of shortest-job-first ordering on top of chunked prefill.
Impact of PrefillSchedBatch: When the scheduling batch size is increased from 16 to 128, SJF's average TTFT decreases by 46.5%. Larger scheduling batches give the SJF policy more freedom to reorder requests, pushing short prompts ahead of long ones more aggressively.
Sorting overhead: "ranges from 10s to 100s of microseconds, which is negligible compared to millisecond-level or second-level TTFT latency."

Interpretation: The 86.4% improvement from chunked prefill alone is the dominant effect. The scheduling policy provides incremental gains on top, with SJF being clearly best. The PrefillSchedBatch parameter provides a tunable knob for the starvation-vs-JCT tradeoff — larger values reduce average TTFT at the cost of potentially delaying long-prompt requests longer. The CDFs in Figure 16 show that SJF compresses the tail of the distribution more than FCFS, which is consistent with SJF's known property of minimizing average waiting time.

Microbenchmark: Length Predictor

The paper evaluates the prediction model's accuracy and the performance impact of co-running the predictor with the main LLM.

Prediction accuracy (Section 5.2.2): The OPT-125M predictor achieves 58.9% accuracy at granularity 100, 74.9% at granularity 200, and 85% at granularity 400. Accuracy naturally increases with coarser granularity because classification into fewer, larger buckets is an easier task.

Co-running impact (Figure 17): When the prediction model runs in parallel with the main LLM (both consuming the same prompts simultaneously), and a padding limit of 512 tokens is used:

80% of the large LLM's prefill requests show unchanged latency compared to when the large LLM runs alone.
The large LLM's average prefill latency increases by 10%.
Throughput drops by 12%.

The paper notes that "these are stress tests. The impact will be smaller in practice" and that "beefier hardware can further mitigate the drop." The padding limit of 512 (the cutting limit for the predictor's batching) is the key configuration parameter: longer limits reduce the number of prompts that run solo, but increase the padding overhead for short prompts batched with long ones.

Interpretation: The prediction accuracy numbers establish the performance upper bound for the downstream scheduling policies. At 74.9% accuracy (granularity 200), the length predictor correctly identifies the bucket about three-quarters of the time. This is sufficient for the scheduling policies to avoid pathological cases but not high enough to improve over greedy scheduling (Figure 18). The co-running overhead of 10–12% on the main LLM is non-trivial but acceptable given that it enables all downstream decode-phase optimizations.

Microbenchmark: Intra-Decode Instance Scheduling

The paper compares three intra-decode scheduling policies — vLLM's greedy, TetriInfer's reserve-static (RS), and TetriInfer's reserve-dynamic (RD) — using 256 requests following ShareGPT distribution (Figure 18). The policies are evaluated at the actual prediction accuracy (74.9% at granularity 200) and an ideal 100% accuracy.

Headline results (Figure 18):

At actual accuracy (74.9%): reserve-dynamic "achieves the same JCT as vLLM's greedy algorithm." Neither reserve-static nor reserve-dynamic outperforms greedy.
At ideal accuracy (100%): reserve-dynamic improves average JCT by 12%, and reserve-static improves average JCT by 10%.

Interpretation: This is a nuanced result. The working-set-aware policies are correctly designed — when given perfect information about generation lengths, they outperform greedy admission by a meaningful margin (10–12%). But with the current predictor's accuracy, the policies' benefits are offset by mispredictions that cause suboptimal admission decisions. This establishes prediction accuracy as the gating factor for intra-decode scheduling improvements and suggests that future work should prioritize better length prediction rather than more sophisticated scheduling algorithms.

Microbenchmark: Inter-Decode Instance Scheduling (Load Balancing)

The paper compares three distributed load-balancing algorithms for the dispatcher's decode-instance selection (Figure 19): TetriInfer's decentralized power-of-two with interference-minimizing tiebreak, a random selection policy, and an "imbalance" policy that simulates worst-case behavior by consistently routing heavy decode requests to the same instances. Tests run 32 requests per decode instance, spanning 2 to 8 decode instances.

Headline results (Figure 19):

TetriInfer's algorithm "achiev[es] the lowest total decoding time compared to other policies."
The right panel of Figure 19 shows the heavy-to-light request ratio on the slowest decode instance. TetriInfer's algorithm produces the most even balance, while the imbalance policy concentrates heavy requests onto specific instances.
The performance gap between TetriInfer's algorithm and random selection increases with the number of decode instances — at 8 instances, the total decoding time advantage is more pronounced than at 2 instances, consistent with power-of-two's asymptotic optimality properties.

Interpretation: The decentralized load-balancing algorithm works as designed. The power-of-two selection provides probabilistic evenness, and the interference-minimizing tiebreak (preferring the instance with the lower heavy:light ratio for a heavy-decode request) adds a workload-specific optimization layer. The "imbalance" policy serves as an informative negative control, demonstrating that decode-decode interference (as measured in Section 2.2.3) translates directly to degraded system performance when scheduling is workload-oblivious.

Ablation Studies and Robustness Checks

The paper's evaluation structure does not include formal ablation studies in the usual sense (removing one component at a time to measure its marginal contribution). However, several comparisons in the experiments serve the function of ablations:

Chunked prefill with FCFS vs. vLLM's fixed-batch prefill (Figure 16a, left): This comparison isolates the effect of chunked prefill (partitioning prompts into fixed-size token-count-based chunks) from the effect of scheduling policy. The 86.4% latency improvement demonstrates that the batching granularity change alone is responsible for the vast majority of prefill-phase gains, with SJF ordering providing only 7.8% additional improvement. This effectively ablates the contribution of the prefill scheduler: chunked prefill is the essential mechanism, and the scheduler policies are fine-tuning.

SJF vs. FCFS vs. LJF at the same ChunkSize (Figure 16a, left): This comparison holds the chunked prefill mechanism constant and varies only the scheduling order. SJF outperforms FCFS, confirming that shortest-job-first ordering provides incremental benefit even when chunked prefill already caps per-iteration latency. LJF performs worse than FCFS for this workload distribution, which is expected since LJF prioritizes long prompts and therefore increases average waiting time for the more numerous short prompts. The paper does not explore whether LJF might be preferable for workloads dominated by long-prompt requests (e.g., a purely summarization-focused deployment).

Reserve-static and reserve-dynamic vs. greedy at actual vs. ideal accuracy (Figure 18): This comparison ablates the effect of prediction accuracy on scheduling policy performance. The finding — policies match greedy at 74.9% accuracy but beat greedy by 10–12% at 100% accuracy — is one of the paper's most important results because it quantifies the headroom available from better prediction and confirms that the scheduling algorithms are well-designed (they degrade gracefully rather than catastrophically under imperfect predictions).

Random vs. power-of-two vs. imbalance inter-decode scheduling (Figure 19): This comparison ablates the effect of length-aware load balancing on decode instance utilization. The imbalance policy shows what happens when decode-decode interference is uncontrolled (heavy requests cluster). Random selection shows the performance of a simple decentralized policy. TetriInfer's power-of-two with tiebreaking shows the marginal benefit of length awareness over random selection.

TS-NVLink vs. TS-RoCE emulated hardware: All end-to-end results (Figures 11–15) report both emulated hardware configurations. The CDFs for TS-NVLink and TS-RoCE largely overlap across all workloads, suggesting that for the tested request sizes and model scale, network bandwidth is not a primary bottleneck — the transfer latency for either 200 Gbps or 300 GB/s is small relative to prefill and decode computation time. The paper does not test lower-bandwidth configurations (e.g., 25 Gbps Ethernet), which would stress the network transfer component more aggressively and might reveal bandwidth sensitivity.

Negative result: HPHD workload (Figure 14). The paper includes the HPHD workload explicitly as a case where TetriInfer provides negligible benefit (perf/$ 1.1×, within margin of error). This serves as an important robustness check — it demonstrates that the authors are not cherry-picking favorable workloads and that the system's limitations are understood. The paper attributes the weak performance to two factors: the KV cache transfer overhead scaling with prefill length, and the fact that heavy-prefill-heavy-decode requests already saturate the GPU individually, leaving little interference to eliminate.

Negative result: HPLD workload (Figure 13). The perf/$ is 14% worse than vLLM, meaning TetriInfer's overhead exceeds its benefits for this workload. This negative result establishes a clear boundary condition: when prefill is heavy but decode is light, the large KV cache transfer cost is not offset by decode-phase improvements because the decode phase is too short to benefit meaningfully from disaggregation or length-aware scheduling.

Instance flip latency: The paper reports that "both instance flips take roughly 5 to 7 ms, excluding the dynamic draining time." This measurement is from Section 3.5 (reported in the design section, not a separate ablation), but it serves as a sanity check on the viability of dynamic instance repurposing. The 5–7 ms is negligible compared to second-level inference latencies, confirming that the mechanism's overhead is not a bottleneck.

Prediction model granularity sweep (Section 5.2.2): The three granularity configurations (100, 200, 400) show the accuracy-granularity tradeoff: 58.9% → 74.9% → 85%. The paper chooses 200 as the default, balancing accuracy with scheduling utility, but does not experimentally evaluate how the different granularities affect downstream scheduling performance. This is a missing ablation: does the 85% accuracy at granularity 400 produce better scheduling decisions than the 74.9% at granularity 200, or does the coarser information negate the accuracy gain?

Critical Assessment

Do the experiments support the paper's central claims?

The paper makes three main performance claims: (1) TetriInfer reduces TTFT and JCT across most workloads through its three-pillar architecture; (2) TetriInfer improves perf/$ (cost efficiency) for most workloads, with the notable exception of HPHD; (3) the improvements come from eliminating specific categories of interference identified in Section 2.2.

Claim 1 — TTFT and JCT improvements: The end-to-end results (Figures 11–15) consistently show TTFT and JCT reductions across all workloads, with the magnitude varying by workload type. The 97% TTFT reduction on LPHD (Figure 12) and the 85% TTFT reduction on Mixed (Figure 15) are the strongest numbers. The 9% TTFT reduction on HPLD (Figure 13) is modest. These results are directionally consistent with the paper's claims, but the absence of direct comparisons against Sarathi (which has chunked prefill without disaggregation) or Splitwise (which has disaggregation without chunked prefill) means the paper cannot attribute the improvements to each pillar independently. The reader must accept the inference that chunked prefill addresses prefill-prefill interference, disaggregation addresses prefill-decode interference, and length-aware scheduling addresses decode-decode interference, but the experiments only test the full system against a single baseline without any of these features. An ablation study that incrementally adds pillars (e.g., vLLM + chunked prefill → vLLM + chunked prefill + disaggregation → full TetriInfer) would have provided much stronger causal evidence for each mechanism's contribution.

**Claim 2 — Perf/ $improvements:** The perf/$ metric depends on resource usage time, which directly addresses the cost objection to disaggregation. The results are mixed in a way that actually strengthens the paper's credibility: the 2.4× improvement on LPHD and 1.9× on Mixed are strong, the 1.4× on LPLD is moderate, the 1.1× on HPHD is breakeven, and the 0.86× (14% worse) on HPLD is a clear negative. The paper is transparent about the HPLD and HPHD results, which establishes credibility. However, the mock mechanism for KV cache transfer is a significant limitation. The transfer latency is computed mathematically rather than measured on real hardware interconnects. If the real RoCE or NVLink stack introduces protocol overhead, contention, or CPU involvement that the model does not capture, the transfer costs could be higher than estimated, eroding the perf/$ advantages. Conversely, if real high-speed interconnects provide better performance than the emulated parameters, the benefits could be larger. The paper cannot distinguish these cases.

Claim 3 — Interference elimination as the causal mechanism: The interference measurements in Section 2.2 are controlled experiments that isolate each interference category. They convincingly demonstrate that prefill-prefill, prefill-decode, and decode-decode interference exist and are severe. However, the causal link between eliminating these specific interferences and the end-to-end performance improvements is indirect. The paper does not instrument TetriInfer to show, for example, that decode-step latency variance decreases because prefill-decode co-scheduling is eliminated, or that per-decode-instance memory usage is more balanced because length-aware scheduling spreads heavy requests. The microbenchmarks (Figures 16–19) partially address this by isolating components, but they test each component under controlled conditions, not under the mixed workload that the system is designed for. A decomposition of end-to-end TTFT into queuing time, prefill execution time, KV cache transfer time, and decode start time — comparing TetriInfer and vLLM — would make the causal chain explicit.

Genuine weaknesses:

Single model, single dataset. All experiments use OPT-13B on NVIDIA V100 GPUs with ShareGPT-derived workloads. The paper argues that OPT-13B is representative, but the specific ChunkSize of 512 tokens, the KV cache size per token, and the compute-memory ratio are all model- and hardware-specific. Results for larger models (e.g., 70B, 175B) on newer hardware (A100, H100) with different compute-memory balances could differ substantially. The paper provides no evidence that the findings generalize.
Limited request volume in end-to-end tests. The end-to-end workloads use only 128 requests. This is a small number for a cloud-scale serving system and may not produce the sustained queuing dynamics that characterize production deployments. The decode instances may never reach steady-state memory pressure, and the instance flip mechanism is never tested under realistic workload shifts. The paper acknowledges this implicitly by labeling the length predictor co-running tests as "stress tests" distinct from typical conditions, but does not apply similar stress-test methodology to the end-to-end evaluation.
No comparison against any system other than vLLM. As noted above, comparisons against Sarathi (chunked prefill, coupled prefill-decode) and Splitwise (disaggregation, no chunked prefill, no length-aware scheduling) would isolate the marginal contribution of each pillar. These systems are discussed in related work (Section 6, Table 1) but not experimentally evaluated. The paper is not required to implement competitors, but acknowledging this gap matters for assessing the strength of the evidence.
The mock transfer mechanism is not validated. The paper emulates KV cache transfer by computing latency from cache size and bandwidth and inserting artificial delays. Whether this accurately reflects real RDMA or NVLink behavior — including protocol overhead, contention, CPU involvement, and memory registration costs — is not validated against any real high-speed interconnect measurement. For a system whose central architectural innovation is physical disaggregation with network transfer, this is a significant evaluation gap.
No ablation that separates chunked prefill from disaggregation. The paper's prefill scheduler microbenchmark (Figure 16) compares chunked prefill against vLLM's fixed-batch prefill, showing an 86.4% latency improvement. This is an isolated test with only prefill requests. The end-to-end tests compare full TetriInfer (chunked prefill + disaggregation + length-aware scheduling) against vLLM (none of these). There is no experiment that runs coupled prefill-decode with chunked prefill (i.e., Sarathi's approach) to determine how much of the end-to-end benefit comes from chunking alone versus disaggregation. The strong LPHD results (97% TTFT reduction) are attributed to disaggregation eliminating prefill-decode interference, but chunked prefill alone might have provided a substantial fraction of that benefit without the KV cache transfer overhead.
Instance flip is described but not stress-tested. The paper reports that flip latency is 5–7 ms (plus drain time), but does not test flipping under load, measure the impact on in-flight requests during draining, evaluate the transition watcher's policy under realistic workload shifts, or measure how quickly the system converges to a new prefill:decode ratio after a workload change. This is an important mechanism for the perf/$ argument, but its evaluation is limited to the latency of the variable change itself.
No analysis of KV cache transfer as a fraction of end-to-end latency. The paper identifies KV cache transfer as the primary overhead of disaggregation but does not report transfer time as a fraction of TTFT or JCT for any workload. For HPLD (where resource usage increases by 43%), the reader cannot determine whether this is due to slow transfer, network contention, or other factors. A breakdown of TTFT into its components would illuminate the bottlenecks.

Experiments that would have strengthened the paper:

Incremental ablation of the three pillars. Running vLLM + chunked prefill only, vLLM + chunked prefill + disaggregation (no length-aware scheduling), and full TetriInfer on the same workloads would quantify each pillar's marginal contribution.
Comparison with Sarathi's mixed chunked prefill. Running Sarathi-style prefill-decode-mixed chunks against TetriInfer's prefill-only chunks + disaggregation on the LPHD workload would directly test whether disaggregation is necessary for eliminating prefill-decode interference or whether smarter chunk scheduling suffices.
Scalability test: varying model size and hardware. Running a subset of experiments with OPT-6.7B and OPT-30B (if hardware permits) or with simulated larger models would test whether the ChunkSize, transfer overhead, and interference patterns scale with model size.
Real interconnect measurement. If any high-speed interconnect is available (even within a single multi-GPU server using NVLink), measuring actual KV cache transfer latency and comparing against the mock mechanism would validate the emulation approach.
Sustained load test with workload variation. Running 1000+ requests with time-varying workload mixes (e.g., 10 minutes of chat, then a summarization burst, then mixed) would test the instance flip mechanism, the transition watcher policy, and the system's ability to maintain perf/$ under realistic dynamics.

Where the claims hold and where they need qualification:

Light-prefill workloads (LPLD, LPHD): TetriInfer's benefits are clearest and largest. The claims of substantial TTFT/JCT improvement and perf/$ gains are well-supported for these workloads, with the caveat that the mock transfer mechanism means absolute TTFT numbers are parameterized by the emulated bandwidth.
Heavy-prefill, light-decode (HPLD): The claims must be qualified. TetriInfer does improve latency (23% JCT reduction), but it does not improve cost efficiency — perf/$ is 14% worse. A practitioner with primarily summarization workloads should not adopt TetriInfer based on these results.
Heavy-prefill, heavy-decode (HPHD): TetriInfer provides essentially no benefit (perf/$ 1.1×). The paper is explicit about this, and the claim that "the room for improvement is marginal and the overhead cannot be offset" is directly supported.
Mixed workloads: The 1.9× perf/$ improvement is strong, but it depends on the specific mix of request types in the 128-request sample. The paper does not report the composition of the mixed workload or test multiple random seeds to establish variance. The mixed workload result should be interpreted as a single-point estimate rather than a robust characterization.
The causal claim that interference elimination drives improvements: This is supported by the controlled interference measurements (Section 2.2) and the correlation between workload characteristics and improvement magnitude, but the lack of component-wise ablation leaves open the possibility that some of the gain comes from mechanisms other than interference elimination (e.g., simply having separate queues for prefill and decode, or better batching dynamics from fixed-size prefill chunks).

6. Limitations and Trade-offs

Limitation 1: The KV Cache Transfer Cost Is Modeled, Not Measured, on Real High-Speed Interconnects

The assumption or constraint. TetriInfer's central architectural innovation—disaggregating prefill from decode—depends on transferring the prefilled KV cache from prefill instances to decode instances over a network. The paper evaluates this transfer using a mock mechanism that computes latency mathematically from the KV cache size and an emulated bandwidth parameter, rather than measuring actual transfer performance on real high-speed interconnects. The paper explicitly acknowledges this constraint in Section 4:

"Due to limited high-end hardware availability, our current implementation only supports the Indirect type using sockets... In order to evaluate TetriInfer's performance across different hardware configurations, we have implemented a mock mechanism to emulate varying network bandwidth."

The mock mechanism works by pre-computing the prefilled KV cache offline, loading it into the decode instance's local memory before the test begins, and then having the decode instance "calculate the latency of the KV cache transfer and wait accordingly" when only request metadata is actually transmitted.

The consequence. This approach cannot capture real-world network stack overheads that materially affect disaggregation viability. RDMA and NVLink transfers involve protocol overhead, memory registration costs, CPU involvement for connection management, and contention when multiple transfers share the same link—none of which a simple size / bandwidth calculation captures. If real transfer latency is higher than modeled (for example, due to PCIe bottlenecks between the GPU and NIC in the Direct-NIC case, or due to NVLink congestion when multiple GPU pairs communicate simultaneously), the TTFT improvements reported for disaggregation would shrink. Since the paper identifies KV cache transfer as the primary overhead of disaggregation, any error in the transfer cost model directly affects the headline perf/$ numbers. This is particularly concerning for the HPLD workload, where the paper acknowledges that TetriInfer's resource usage increases by 43% (Figure 13), but cannot isolate how much of this is due to transfer overhead versus other factors.

What evidence exists in the paper. The paper reports end-to-end results for two emulated hardware configurations (TS-RoCE at 200 Gbps and TS-NVLink at 300 GB/s) across all five workload types (Figures 11–15). The CDFs for TS-RoCE and TS-NVLink largely overlap across all workloads, which superficially suggests that bandwidth is not a bottleneck. However, this overlap is entirely determined by the mock mechanism's size / bandwidth formula—it cannot reveal whether real RoCE or NVLink stacks would produce different results. The paper provides no validation of the mock mechanism against any real high-speed interconnect measurement, even within a single multi-GPU server where NVLink is available.

Mitigation status. The paper does not attempt to mitigate this limitation experimentally. It acknowledges the constraint as a hardware access issue and designs the unified network abstraction (Section 3.3.4, Figure 9) to accommodate multiple physical link types, but does not validate that abstraction against real hardware. The paper identifies two unexplored research questions related to KV cache transfer—simultaneous use of multiple data links and one-sided remote memory access—but frames these as future exploration rather than as validation gaps. A practitioner considering deployment on real high-speed interconnects cannot take the paper's TTFT numbers at face value without independently benchmarking KV cache transfer on their specific hardware.

Limitation 2: Single Model, Single Hardware Configuration, Single Dataset

The assumption or constraint. All experiments in the paper use OPT-13B running on NVIDIA V100 GPUs (32 GB HBM, 4 GPUs per server) with workloads derived from the ShareGPT dataset. The authors state in Section 4 that they "believe this model is representative of the capabilities of many contemporary LLMs," but provide no evidence across different model scales, architectures, or hardware generations.

The consequence. Several of TetriInfer's key design parameters and performance characteristics are model- and hardware-specific. The ChunkSize of 512 tokens—the core parameter governing chunked prefill—is empirically measured for OPT-13B on V100 GPUs. The paper correctly notes that "the accelerator and the LLM model architecture determine the ChunkSize" and that "models with larger hidden dimensions and accelerators with lower capabilities typically result in a smaller ChunkSize." However, this means that deploying TetriInfer on an A100 or H100 with a 70B or 175B model would require re-measuring the saturate threshold, which might differ substantially due to different compute-to-memory ratios, larger hidden dimensions, and different attention implementations (FlashAttention, etc.).

More significantly, the interference magnitudes documented in Section 2.2—the 10× prefill slowdown when heavy and light prefill co-run, the 5× decode slowdown when prefill and decode share a batch, the 16% throughput drop from mixing heavy and light decode—are all measured on this specific (model, hardware) pair. On hardware with higher memory bandwidth (H100's HBM3 provides ~3.3 TB/s vs. V100's ~900 GB/s), the decode-phase memory bottleneck may be less severe, reducing decode-decode interference. On models with different KV cache sizes per token (determined by layers × hidden_dim × num_heads), the memory pressure per generated token differs, changing when and how the reserve-static and reserve-dynamic policies provide benefit.

The length predictor is trained to predict the generation behavior of OPT-13B specifically. If the target model changes—for example, to a model with different response length distributions, such as one fine-tuned for concise answers or one prone to verbose outputs—the predictor must be retrained. The paper's finding that 74.9% accuracy at granularity 200 is sufficient to match (but not beat) greedy scheduling (Figure 18) is specific to OPT-13B's generation length distribution on ShareGPT prompts.

What evidence exists in the paper. The paper presents no experiments varying the model size, model architecture, GPU generation, or dataset. The interference study (Section 2.2), the chunked prefill threshold measurement (Figure 2), the prediction model training (Section 3.3.2), and all end-to-end evaluations (Section 5.1) use OPT-13B on V100 with ShareGPT. The paper does not discuss how findings might generalize or provide guidance for practitioners using different configurations.

Mitigation status. Not addressed. The paper does not claim generalizability, but it also does not acknowledge the scope limitation as a threat to practical adoption. The design principles—chunked prefill at the saturate threshold, disaggregation to isolate batch from latency-critical work, length-aware scheduling—are argued conceptually and would likely apply across models and hardware, but the specific parameters and the quantitative magnitude of improvements would differ. A practitioner with a different serving stack must independently characterize their own saturate threshold, interference patterns, and prediction accuracy before adopting the architecture.

Limitation 3: Length Prediction Accuracy Gates the Benefit of Working-Set-Aware Decode Scheduling

The assumption or constraint. TetriInfer's decode-phase scheduling innovations—the reserve-static and reserve-dynamic policies that make admission decisions based on predicted memory usage—depend entirely on the length predictor's accuracy. The paper explicitly demonstrates that these policies provide benefit only when prediction accuracy is high, stating in the decode instance section (Section 3.4):

"when the prediction accuracy increases, these two policies can lower the average JCT by roughly 10%."

The consequence. At the paper's achieved prediction accuracy of 74.9% (granularity 200, Section 5.2.2), the reserve-dynamic policy merely matches vLLM's greedy admission policy—it does not improve upon it (Figure 18). The 12% JCT improvement that reserve-dynamic provides at ideal (100%) accuracy is entirely unrealized in the actual system. This means one of TetriInfer's three core mechanisms—the intra-decode working-set-aware scheduling—provides zero end-to-end benefit under the current prediction model.

The decode-decode interference that the paper documents in Section 2.2.3 (16% throughput drop when mixing heavy and light decode at batch size 128) is real, but the mechanism designed to address it (length-aware admission control) cannot actually mitigate it with the current predictor. The length prediction is used productively only by the inter-decode load balancer (the dispatcher's power-of-two algorithm), which spreads heavy requests evenly across instances and demonstrably reduces the heavy:light ratio on the slowest instance (Figure 19, right). The intra-decode policies are effectively dead weight.

This creates a dependency chain: the paper's central claim that length prediction is a "first-class scheduling primitive" (Innovation 3) is only partially validated. The prediction enables better load balancing (inter-decode scheduling) but does not yet enable better admission control (intra-decode scheduling). The architecture is forward-looking—it is designed to benefit from better predictors when they become available—but does not deliver the full benefit today.

What evidence exists in the paper. Figure 18 provides the direct ablation. The reserve-dynamic curve (dark blue, solid) overlaps nearly perfectly with the vLLM greedy curve (red, dashed) at the actual 74.9% accuracy. Only the "Reserve Dynamic Ideal" curve (light blue, dashed) shows the 12% improvement. The paper is candid about this result: "Our tests in Figure 18 suggest that with our current prediction accuracy, these two policies are on par with vLLM's greedy policy." The paper also reports prediction accuracy at three granularities (58.9%, 74.9%, 85%) but does not test how the different accuracies affect scheduling performance—another missing ablation.

Mitigation status. The paper explicitly identifies prediction accuracy improvement as future work (Section 5.2.2: "Since improving prediction accuracy is not the focus of this work, we leave it for future work"), but does not propose specific directions for improvement beyond this acknowledgment. The predictor's current architecture (OPT-125M classification model fine-tuned on 75K examples) is a straightforward baseline; approaches such as larger predictor models, prompt engineering for the predictor, training on more diverse data, or predicting at inference time using early LLM layer outputs are not explored. The paper also does not discuss whether the 10% co-running overhead of the prediction model (Section 5.2.2, Figure 17) would increase if a larger or more accurate predictor were used, creating a potential accuracy-vs-overhead tradeoff.

Limitation 4: The Instance Flip Mechanism Is Described but Not Stress-Tested Under Realistic Workload Dynamics

The assumption or constraint. TetriInfer's economic argument for disaggregation—that it does not simply double hardware costs—depends critically on the instance flip mechanism (Section 3.5), which dynamically converts idle prefill instances to decode instances and vice versa as workload mixes shift. The paper describes the mechanism: a transition watcher that checks load and flips instances "if its load has been under 10% for the past minute," a drain phase that waits for queued requests to complete, and a fast variable change (5–7 ms) that avoids model reloading.

The consequence. The instance flip mechanism is central to TetriInfer's cost model, but it is never tested under realistic conditions. The end-to-end evaluations (Section 5.1) use static workloads of 128 requests with no workload mix variation over time. The system is configured with a fixed number of prefill and decode instances, and the transition watcher is never triggered because there are no workload shifts. This means the following failure modes are unexplored:

Flip oscillation: If the workload fluctuates rapidly between prefill-heavy and decode-heavy (e.g., due to diurnal patterns or bursty traffic), the transition watcher might repeatedly flip instances back and forth. Each flip incurs a drain time that depends on queue depth—for a decode instance with many long-running heavy-decode requests, the drain could take tens of seconds or more. During drain, the instance is unavailable for either role, creating a temporary capacity reduction.
In-flight request disruption: When a decode instance is flipped to prefill, all in-progress decode requests must complete first (the drain phase). The paper does not specify whether new decode requests that were already dispatched to this instance (but whose KV cache transfer is still in flight) are rejected, queued, or redirected. The paper notes that the global scheduler notifies all prefill instances to stop forwarding to the selected decode instance, but there is a race condition between the notification and in-flight dispatcher decisions made before the notification was received.
Prediction accuracy during shifts: The length predictor is trained on OPT-13B's behavior, but when instances are flipped, the hardware resources available to each phase change. The paper does not discuss whether the predictor's accuracy degrades when the ratio of prefill to decode instances shifts significantly, potentially because the target model's generation behavior is not affected (it's the same model), but the queuing dynamics change.
Transition watcher policy sensitivity: The paper gives one example policy—"under 10% load for the past minute"—but does not evaluate alternatives, test sensitivity to the threshold or window, or demonstrate that this policy works for common workload shift patterns.

What evidence exists in the paper. The only measurement related to instance flips is the flip latency itself: "both instance flips take roughly 5 to 7 ms, excluding the dynamic draining time" (Section 3.5). This is the time to change an internal variable, which is trivially fast. The paper does not report drain times for any workload, does not measure flip frequency or oscillation under variable workloads, and does not test any workload scenario that triggers a flip. The resource usage time metrics (Figures 11–15) are for static workloads and do not reflect the cost of idle instances during workload transitions or the overhead of flips themselves.

Mitigation status. Not addressed. The paper frames instance flipping as a mechanism that makes disaggregation economically viable, but the evaluation does not demonstrate that the mechanism works under conditions that would require it. A practitioner deploying TetriInfer in a production environment with variable workloads cannot estimate from the paper's data how often flips would occur, what their cost would be, or whether the transition watcher's simple hysteresis policy is adequate. The paper identifies the transition watcher's policy as pluggable ("various policies can be plugged in"), suggesting that more sophisticated policies could be developed, but does not evaluate even the default policy.

Limitation 5: No Direct Ablation Separating the Contribution of Each Architectural Pillar

The assumption or constraint. TetriInfer's architecture comprises three named pillars: chunked prefill, disaggregated prefill/decode instances, and two-level length-aware scheduling. The paper attributes different interference categories to each pillar: chunked prefill addresses prefill-prefill interference, disaggregation addresses prefill-decode interference, and length-aware scheduling addresses decode-decode interference. The paper presents these as a unified design, stating in Section 1 that "Our designs are three-fold" and evaluating the full system against a single baseline (vanilla vLLM) that has none of these features.

The consequence. The paper cannot quantify the marginal contribution of each pillar to the end-to-end performance improvements. This matters for several reasons:

Adoption complexity: A practitioner might want to adopt only the easiest-to-implement pillar. If chunked prefill alone provides 80% of the benefit on a particular workload, the added complexity of disaggregation (KV cache transfer, dual instance management, instance flipping) might not be justified. The paper provides no decomposition that would guide such decisions.
Causal attribution: The paper claims that disaggregation eliminates prefill-decode interference, but the LPHD workload's 97% TTFT improvement (Figure 12)—attributed primarily to disaggregation—might benefit substantially from chunked prefill as well, since chunked prefill reduces the time each prefill occupies the GPU, indirectly reducing the blocking time for decode. Without an ablation, the causal chain is inferred rather than demonstrated.
Negative results interpretation: The HPLD workload's 14% worse perf/$ (Figure 13) is attributed to large KV cache transfer overhead. But if chunked prefill alone (without disaggregation) provides most of the latency improvement for this workload, the disaggregation overhead might be avoidable. The paper cannot make this determination.
Comparison with concurrent work: Sarathi uses chunked prefill with prefill-decode-mixed chunks (no disaggregation). Splitwise uses disaggregation without chunked prefill or length-aware scheduling. Without an ablation that tests TetriInfer's components against these partial baselines, the paper cannot demonstrate that its full combination is better than either partial approach, undermining the claim in Table 1 that TetriInfer uniquely has all four features.

What evidence exists in the paper. The paper provides two isolated microbenchmarks that partially serve as ablations:

Prefill scheduler microbenchmark (Figure 16): Compares vLLM's fixed-batch prefill against TetriInfer's chunked prefill with FCFS, showing an 86.4% latency improvement. This isolates chunked prefill's contribution to prefill-only workloads but does not show its effect in mixed prefill-decode scenarios.
Intra-decode scheduling microbenchmark (Figure 18): Compares vLLM's greedy admission against reserve-static and reserve-dynamic on decode-only workloads, showing that the working-set-aware policies match greedy at current accuracy. This isolates the intra-decode scheduling pillar but does not show its effect when prefill and decode are disaggregated.

Neither microbenchmark tests the combination of pillars in an end-to-end setting. There is no experiment that runs, for example, vLLM + chunked prefill + coupled prefill-decode (Sarathi's approach) against full TetriInfer on the LPHD workload to isolate the marginal benefit of disaggregation. There is no experiment that runs TetriInfer with disaggregation but without length-aware scheduling to isolate the marginal benefit of the two-level scheduler.

Mitigation status. The paper does not acknowledge this as a limitation. The evaluation structure—end-to-end full-system comparisons plus isolated component microbenchmarks—is common in systems papers, and the microbenchmarks do provide evidence that individual components work as designed. However, the absence of incremental ablations means the paper's strongest claim—that all three pillars are necessary and that their integration is the key contribution—is supported by architectural argument rather than direct experimental evidence. A defender of the paper would argue that the interference taxonomy (§2.2) provides the causal rationale for each pillar, and that testing partial configurations would require implementing multiple intermediate system versions. This is a reasonable practical constraint, but it does not change the fact that the marginal contribution of each pillar to end-to-end performance remains unevaluated.

Limitation 6: No Evaluation at Production Scale—Request Volumes, Sustained Load, and Workload Variance Are All Limited

The assumption or constraint. The paper evaluates TetriInfer on 128 requests per workload type for end-to-end tests (Section 5.1) and up to 256 requests for some microbenchmarks (Figure 18). These tests run to completion with fixed workload compositions and no time-varying arrival patterns. The paper does not simulate production conditions with hundreds or thousands of concurrent requests, sustained load over minutes or hours, or realistic arrival processes (e.g., Poisson arrivals with variable rates).

The consequence. Several of TetriInfer's design elements are motivated by cloud-scale dynamics that the evaluation does not exercise:

Continuous batching dynamics: vLLM's PagedAttention and TetriInfer's decode scheduler are designed to handle large numbers of concurrent decode requests, amortizing model weight loading across many requests. With only 128 requests total, the system may never reach the steady-state batch sizes where batching efficiency matters most. The decode-decode interference documented in Figure 5 shows a 16% throughput drop at batch size 128, but the end-to-end tests may not sustain batch sizes that large for meaningful durations.
Queuing and scheduling under load: The prefill scheduler's SJF policy with PrefillSchedBatch is designed to prevent starvation while minimizing average waiting time. With 128 requests, the raw request queue may never grow large enough for the scheduling policy to matter—the claims about 46.5% TTFT improvement when increasing PrefillSchedBatch from 16 to 128 (Figure 16) are from microbenchmarks, not from the end-to-end mixed-workload scenario.
Instance flip dynamics: As discussed in Limitation 4, instance flipping is never triggered because workloads are static.
Cluster monitor scalability: The cluster monitor collects and broadcasts load information from every instance every 100ms. With 4 GPUs (the testbed size), this is trivial. At cloud scale with hundreds of instances, the broadcast could become a bandwidth or processing bottleneck. The paper states the control plane is "a distributed system without a single point of failure or processing bottlenecks" but provides no evidence for this claim at scale.
KV cache transfer contention: With 128 requests and emulated transfer, there is no opportunity for multiple simultaneous KV cache transfers to contend for network bandwidth. In a real deployment with hundreds of concurrent requests, many prefill-to-decode transfers would be in flight simultaneously, potentially saturating the interconnect and increasing transfer latency beyond the simple size / bandwidth model.

What evidence exists in the paper. The paper's testbed consists of 4 NVIDIA V100 GPUs in a single server. The end-to-end workloads use 128 requests. The interference study (Section 2.2) uses controlled batches of specific sizes (2, 4, 8, ..., 128) to isolate individual interference effects, which is appropriate for a microbenchmark but does not represent production dynamics. No experiments exceed 256 requests. No experiments use time-varying arrival rates, bursty traffic patterns, or diurnal workload cycles. The prefill scheduler microbenchmark (Figure 16) varies PrefillSchedBatch up to 128, but this is a batch size for the scheduling decision, not the total number of requests in the system.

Mitigation status. The paper does not claim to evaluate at production scale, but it also does not acknowledge the gap between its test conditions and cloud-scale deployment. The system is described as "a cloud-scale LLM inference serving system" (Section 3.1), and the centralized control plane is justified with cloud-scale arguments (no single point of failure, distributed), but the evaluation does not test any cloud-scale property. The paper's contributions are primarily architectural, and controlled experiments on a small testbed are standard for systems papers introducing new architectures. However, the specific claims about TTFT, JCT, and perf/$ improvements are quantitative and may not hold under different load conditions—for example, under high sustained load, the relative benefit of disaggregation might increase (because interference is more severe when instances are saturated) or decrease (because the transfer network becomes a bottleneck). The paper provides no basis for predicting which direction the effects would go.

7. Implications and Future Directions

How This Work Changes the Landscape

TetriInfer does for LLM inference serving what microservices architectures did for monolithic web applications: it decomposes a single, coupled processing pipeline into independently scheduled, independently scaled components organized around the distinct resource profiles of the sub-tasks. This is not an incremental optimization layered on top of existing serving systems—it is an architectural reframing of what an LLM inference cluster looks like. Before TetriInfer, the dominant paradigm (vLLM, Orca) treated prefill and decode as two phases of a single job to be interleaved opportunistically on the same GPU. The intellectual premise was that since both phases require forward passes through the same model weights, co-locating them maximizes accelerator utilization and amortizes model loading costs. TetriInfer demonstrates that this premise, while correct in a simplified throughput model, is wrong when latency, interference, and workload heterogeneity enter the picture.

The paper's central conceptual shift is to treat prefill and decode not as phases of one job but as two different job types with fundamentally different optimization objectives—batch computation versus latency-critical streaming—that happen to share model weights. This reframing has consequences that cascade through the entire system design: if prefill and decode are different job types, they need different schedulers (SJF for prefill throughput, working-set-aware admission for decode memory pressure), different batching strategies (fixed-size token chunks for prefill, continuous batching with PagedAttention for decode), different scaling policies (add prefill instances during a summarization surge, add decode instances during a content-creation surge), and different failure domains (a stalled prefill does not block decode generation, and vice versa).

This reframing resolves a latent contradiction in the LLM serving literature. Prior work had documented that prefill is compute-bound and decode is memory-bound (Pope et al., 2023, reference [33]), but the dominant serving systems did not extract the architectural consequences of this observation. Instead, they focused on making the coupled engine more efficient—better memory management (vLLM's PagedAttention), better attention kernels (FlashAttention), priority-based scheduling (FastServe's multi-level feedback queue). Each of these is valuable, but none addresses the structural interference that arises when compute-bound batch work and memory-bound latency-critical work share a GPU. TetriInfer's interference taxonomy (Section 2.2) provides the diagnostic framework: it shows that the 5× decode slowdown from co-running with a heavy prefill (Figure 4a) is not a scheduling inefficiency that can be tuned away with better priorities or time-slicing within a coupled engine—it is a physical resource conflict that can only be resolved by spatial separation.

The paper also shifts the research agenda for LLM serving in two directions. First, it redirects attention from within-phase efficiency (faster kernels, better memory management) toward cross-phase isolation (disaggregation, network transfer, instance management). This is a higher-level architectural concern that had been largely absent from the LLM serving literature. Second, it elevates length prediction from an auxiliary ML task to a first-class scheduling primitive, showing that the ability to predict generation length—even at coarse granularity with modest accuracy—enables decentralized load balancing that demonstrably reduces decode-decode interference (Figure 19). The finding that intra-decode scheduling policies cannot outperform greedy admission at current prediction accuracy (Figure 18) but would provide 10–12% JCT improvement at ideal accuracy makes prediction quality the explicit gating factor for future decode-phase optimization. This reframes length prediction not as a "nice-to-have" but as the primary bottleneck that the community should prioritize.

A negative result that strengthens the contribution: the paper's honest documentation of where disaggregation fails—the HPHD workload where perf/ $is essentially breakeven (1.1×), and the HPLD workload where perf/$ is 14% worse than vLLM—establishes clear boundary conditions for when the architecture is appropriate. This prevents over-adoption and guides practitioners toward the workloads (LPLD, LPHD, Mixed) where the benefits are clearest. The fact that the worst-case workload for disaggregation (HPHD) is also the workload where interference was already least severe (because each request individually saturates the GPU) confirms the paper's causal model: disaggregation helps precisely when interference hurts, and it is neutral or negative when interference was never the dominant bottleneck.

Follow-Up Research This Work Enables

Validating the KV cache transfer model against real high-speed interconnects. The paper's central architectural innovation depends on network transfer of prefilled KV caches, yet all transfer latency numbers come from a mock mechanism (size / bandwidth) rather than real measurements on RDMA or NVLink hardware. A direct follow-up would deploy TetriInfer on a cluster with actual RoCE (200 Gbps) or NVLink (300 GB/s+) interconnects, measure end-to-end KV cache transfer latency as a function of prompt length and model size, and compare against the mock mechanism's predictions. The key question: does real RDMA protocol overhead (memory registration, connection management, PCIe bottlenecks between GPU and NIC in the Direct-NIC case) cause transfer latency to deviate from the simple bandwidth model, and if so, by how much? For OPT-13B with 512-token prompts, the KV cache size is roughly 2 × 13B_parameters × 2_bytes (FP16) × (512 / sequence_length_proportional_factor) — but the exact byte count per token depends on model architecture details the paper does not specify. A validation experiment would measure transfer time for prompt lengths from 128 to 2048 tokens, compare against the mock mechanism's predictions, and determine whether the HPLD workload's 43% resource usage increase would shrink or grow under real interconnect conditions. This experiment is essential for any practitioner considering production deployment, because the entire perf/$ argument for disaggregation rests on transfer overhead being smaller than the interference it eliminates.

Incremental ablation of the three architectural pillars. The paper evaluates only the full three-pillar system against vLLM, which has none of the pillars. A critical follow-up would measure the marginal contribution of each pillar by testing intermediate configurations: (a) vLLM + chunked prefill with prefill-decode coupling (i.e., Sarathi's approach, but with prefill-only chunks rather than mixed chunks), (b) vLLM + disaggregation without chunked prefill (prefill instances use fixed batch sizes), and (c) vLLM + disaggregation + chunked prefill without length-aware scheduling (random inter-decode routing and greedy intra-decode admission). Running these configurations on the LPHD workload (where TetriInfer shows 97% TTFT reduction and 2.4× perf/$) would answer: How much of the TTFT improvement comes from chunked prefill alone versus disaggregation alone? Does the combination provide more than the sum of individual benefits? Is length-aware scheduling essential for the decode-phase gains, or does simple disaggregation with random routing already capture most of the benefit? This ablation would directly test the paper's causal model—that each pillar addresses a specific interference category—and would identify which pillars are essential versus nice-to-have for different workload types. A practitioner could then adopt a subset of TetriInfer's mechanisms appropriate to their deployment constraints (e.g., chunked prefill alone for a single-GPU deployment that cannot afford dual instances).

Length prediction with larger predictor models and online adaptation. The paper demonstrates that intra-decode scheduling policies are gated by prediction accuracy: at 74.9% accuracy (granularity 200), reserve-dynamic matches greedy; at 100% accuracy, it improves JCT by 12% (Figure 18). A natural extension is to improve the predictor. Specific directions: (a) Use a larger prediction model—the paper uses OPT-125M, but OPT-350M or OPT-1.3B might achieve higher accuracy, and the paper's Figure 17 provides the framework for measuring the co-running overhead against the accuracy gain. The key tradeoff: does a 2× larger predictor that reduces main LLM throughput by 20% (versus the current 12%) produce enough scheduling improvement to justify the cost? (b) Train the predictor on the first few generated tokens rather than only the prompt—once decoding begins, the model's early output tokens may be highly predictive of total generation length, and this information could refine the initial bucket prediction into an exact-length estimate. (c) Investigate whether the predictor transfers across target models: train on OPT-13B's generation behavior and test prediction accuracy for OPT-6.7B, OPT-30B, or LLaMA-family models. If a single predictor works across related model families, the training cost is amortized.

Instance flip policies under realistic workload dynamics. The paper describes the flip mechanism but never triggers it in evaluation because all workloads are static. A follow-up would construct time-varying workload traces—for example, alternating between chat-heavy (LPLD) and summarization-heavy (HPLD) periods with realistic transition rates derived from production LLM serving logs—and evaluate: (a) the transition watcher's responsiveness (how quickly does it detect a workload shift and trigger flips?), (b) the overhead of drain phases (how many in-flight requests are delayed, and by how much?), (c) whether the simple hysteresis policy ("under 10% load for the past minute") causes oscillation under bursty workloads, and whether a more sophisticated policy (e.g., exponential smoothing of load with a minimum flip interval) improves stability, and (d) the overall resource-time product over a multi-hour trace with and without instance flipping enabled. This experiment would determine whether the flip mechanism delivers the cost savings it promises under realistic conditions or whether drain overhead and oscillation negate the benefits of dynamic repurposing.

Testing generalizability across model scales and hardware generations. All experiments use OPT-13B on V100 GPUs. The paper's key parameters—ChunkSize of 512, interference magnitudes (10× prefill slowdown, 5× decode slowdown, 16% throughput drop), KV cache transfer size—are specific to this (model, hardware) pair. A scaling study would measure how these parameters change when moving to (a) larger models (OPT-30B, LLaMA-70B) on the same V100 hardware (if memory permits via tensor parallelism), (b) the same OPT-13B on newer hardware (A100, H100) with higher memory bandwidth and compute, and (c) smaller models (OPT-1.3B, OPT-6.7B) to test the lower bound of where disaggregation's overhead dominates. The key predictions to test: on hardware with higher memory bandwidth (H100's ~3.3 TB/s vs. V100's ~900 GB/s), decode-phase memory pressure should decrease, reducing decode-decode interference and potentially making length-aware scheduling less beneficial. On larger models, the KV cache per token grows, increasing transfer overhead and potentially shifting the crossover point where HPLD and HPHD become unfavorable. This study would produce a design space map showing, for each (model size, hardware generation) combination, which workloads benefit from disaggregation and by how much—directly useful for practitioners making deployment decisions.

Integration with model parallelism and multi-node disaggregation. The paper runs OPT-13B with tensor parallelism (TP=2) on a single server's 4 GPUs. Production deployments of larger models (175B+) use pipeline parallelism across multiple nodes. This raises questions the paper does not address: (a) When a model is split across GPUs, the KV cache is distributed—each GPU holds the K and V tensors for its layers. Transferring the KV cache to a decode instance requires gathering these distributed tensors and transmitting them, potentially involving inter-GPU communication (all-gather) within the prefill instance before network transfer. How does this additional communication affect the transfer latency model? (b) Can prefill and decode instances use different parallelism strategies? For example, prefill might benefit from tensor parallelism (high compute per token, parallelized across GPUs), while decode might benefit from pipeline parallelism (lower per-step compute, amortized across more GPUs). (c) At what scale does the cluster monitor's 100ms broadcast become a bottleneck—how many instances can it manage before broadcast latency exceeds the scheduling decision horizon? A follow-up deploying TetriInfer on a multi-node cluster with model parallelism would test these scalability questions and determine whether the disaggregation architecture extends naturally to the distributed setting or requires fundamental redesign.

Practical Applications and Downstream Use Cases

Cost-efficient multi-tenant cloud LLM serving. The most direct application of TetriInfer is for cloud providers or enterprises running LLM inference services that handle diverse, unpredictable workloads from multiple tenants. The mixed workload result (Figure 15) is the most ecologically valid evaluation: with random requests sampled from ShareGPT, TetriInfer delivers 85% lower average TTFT, 50% lower average JCT, and 21% lower total resource usage, yielding a 1.9× improvement in performance per dollar compared to vLLM. For a cloud provider running 1,000 V100 GPUs for LLM inference, a 1.9× perf/ $improvement means serving the same request volume with approximately 525 GPUs—a reduction of 475 GPUs, corresponding to roughly$ 1–2 million annually in cloud rental costs (at ~$2–3/GPU-hour for V100-class instances). The workload mix in production is likely closer to "Mixed" than to any single homogeneous workload type, making this result the most relevant for adoption decisions. The key deployment consideration is that the instance flip mechanism is essential for realizing these cost savings—without it, the disaggregated architecture's extra instances would consume the savings. Practitioners should budget for implementing and tuning the transition watcher policy based on their specific workload patterns.

On-demand content creation platforms with heavy decode. Content creation workloads (LPHD) show TetriInfer's strongest results: 97% TTFT reduction, 47% JCT reduction, 38% fewer hardware resources, and 2.4× perf/$ improvement (Figure 12). This workload profile—short prompts (e.g., "Write a blog post about...") followed by long generations (1,000+ tokens)—is characteristic of AI writing assistants, code generation tools (where the "prompt" is a short natural language specification and the "generation" is a multi-hundred-line code file), and creative content platforms. In these applications, TTFT directly affects user experience: a 97% reduction means the first token appears nearly instantaneously rather than after a multi-second delay during which vLLM is busy processing other requests' decode steps or batching prefill work. For a code generation tool where users iteratively refine prompts based on initial output, this latency improvement changes the interaction from batch-like (submit, wait, review) to interactive (submit, see immediate response, refine). The JCT improvement of 47% means users get complete generations in roughly half the time, enabling faster iteration cycles. The 38% resource reduction means the platform can handle more concurrent users with the same GPU fleet.

Summarization and document processing pipelines with heavy prefill. The HPLD workload (Figure 13) is TetriInfer's weakest positive result—9% TTFT improvement and 23% JCT improvement, but 43% more resource usage and 14% worse perf/$. This workload profile—long prompts (document summarization, prompt engineering with extensive context) followed by short generations—is common in enterprise document processing, legal document review, and retrieval-augmented generation (RAG) pipelines where retrieved context is concatenated with the query. The key practical takeaway for these applications is conditional: adopt TetriInfer if TTFT or JCT are the binding constraints (e.g., a user-facing summarization tool where latency directly affects experience), but do not adopt it if cost efficiency is the primary concern—vLLM is cheaper for this workload. The paper's negative result here is practically useful because it prevents misguided adoption. Organizations with mixed workloads (some heavy-prefill-light-decode, some chat, some content creation) should consider whether the gains on other workload types justify the disaggregation overhead that partially affects the HPLD sub-workload, or whether hybrid deployment (vLLM for summarization, TetriInfer for chat and generation) is preferable. The instance flip mechanism partially addresses this by allowing instances to serve both roles, but the overhead of transferring large KV caches for heavy prefill remains regardless of which role an instance currently occupies.

Deployments where network bandwidth is abundant and latency is paramount. The paper's two emulated hardware configurations—200 Gbps RoCE and 300 GB/s NVLink—produce overlapping CDFs across all workloads (Figures 11–15), suggesting that even 200 Gbps is sufficient to make transfer overhead negligible relative to compute time for OPT-13B-scale models. This finding has direct implications for deployment architecture: in data centers where high-speed interconnects (100+ Gbps) are available between GPU nodes, disaggregation's network cost is not a bottleneck, and the latency and efficiency benefits are "free" from a networking perspective. This is particularly relevant for deployments using NVLink-connected GPU pairs within a single server (e.g., 2 GPUs for prefill, 2 GPUs for decode on the same NVSwitch fabric), where transfer bandwidth is in the hundreds of GB/s and latency is sub-microsecond. In this intra-server disaggregation scenario, the paper's architecture provides strong isolation benefits without any meaningful network penalty. The mock mechanism should be validated against real NVLink measurements (as discussed in Follow-Up Research), but if the mock is accurate, intra-server disaggregation is a low-risk, high-reward deployment pattern for any multi-GPU server. The paper also opens the possibility of cross-server disaggregation using RoCE or InfiniBand, where the network cost is higher but the ability to independently scale prefill and decode pools across many servers provides operational flexibility that coupled architectures cannot match.