Orca: A Distributed Serving System for Transformer-Based Generative Models

🎯 Pitch

A 20-line scheduler that prioritizes small requests under load can cut time-to-first-token latency in half compared to state-of-the-art alternatives, without the complexity of prediction-based methods. Meanwhile, a simple token-aware load balancer outperforms general-purpose policies just by estimating queue wait times from available memory.

1. Executive Summary

This paper surveys and empirically compares scheduling techniques for Large Language Model (LLM) serving systems, analyzing both engine-level schedulers and load balancers across production workload traces (Azure OpenAI's SW-Chat and SW-Code) on Llama-3 models (8B and 70B) deployed with vLLM. Finding that schedulers from the literature achieve good performance but introduce significant complexity while practical deployments leave easy gains on the table, the authors introduce two drop-in replacement techniques: LARRY (Load-Adaptive Request Reordering for Low Latency), an engine-level scheduler that prioritizes requests by their anticipated memory consumption and current queue depth (dispatches small-prefill requests first under high load, implemented in 20 lines of code), and SAL (Server-Aware Load Balancer), a token-aware load balancer that routes requests by estimating queue wait times from available memory and queued prefill tokens (implemented in 30 lines of code). LARRY achieves 1.8–2.1× lower p50 Time-To-First-Token than the next-best engine-level scheduler and proves least sensitive to workload scaling across both deployments, while SAL improves p50 Total Generation Time by 1.1–1.3× over general-purpose load balancers by more evenly distributing token load across servers, establishing that simple, request-aware scheduling heuristics can outperform both sophisticated prediction-based methods and general-purpose policies without requiring model training or system-level modifications.

2. Context and Motivation

The Core Problem: LLM Serving Systems Lack Principled, Practical Scheduling

The fundamental problem this paper tackles is that LLM serving systems must multiplex hardware resources between concurrent inference requests, yet the scheduling policies that govern this multiplexing are poorly understood and often suboptimal in practice. While the machine learning community has invested enormous effort into improving model architectures, training procedures, and inference optimizations like Paged Attention and Continuous Batching, the scheduling decisions that determine which requests run when have received comparatively little systematic attention. This is surprising because scheduling directly impacts both the user experience (via latency) and operational cost (via throughput and hardware utilization).

The paper identifies a specific tension that makes LLM scheduling particularly challenging: the memory consumption of each request is not known in advance. As an LLM generates response tokens, it accumulates a Key-Value (KV) cache that stores intermediate attention states. This KV cache grows linearly with the sequence length. For example, as noted in Section 1, the KV cache of Llama-3 70B occupies 2.7 GB for a single sequence of 8192 tokens. Since the serving system does not know when the model will emit an end-of-sequence token — and thus how long the final response will be — it cannot pre-allocate the precise amount of memory each request needs. The authors describe this explicitly:

"Since the response length of the LLM is not known a priori, a request's memory consumption is also not known a priori."

This uncertainty creates a fundamental scheduling dilemma. The system wants to pack many requests into each forward pass because large batch sizes improve throughput — Figure 2 demonstrates this by showing how throughput for Llama-3 8B on an A100 GPU increases dramatically with batch size as the workload transitions from memory-bound to compute-bound. But packing too many requests into GPU memory risks exhausting capacity as their KV caches expand, forcing expensive preemptions. A preemption requires evicting a request's KV cache to free memory and later recomputing it from scratch, which the paper notes "is an expensive operation" (Section 1). Conversely, being too conservative and limiting concurrency leaves GPU resources underutilized — the system literally has empty memory that could be productively used.

This tension is not hypothetical. It plays out in every LLM deployment, and it is amplified by two real-world characteristics that the paper documents extensively:

1. Workload variability in both arrival patterns and request characteristics. Figure 4 shows a 20-minute excerpt from a production request trace at Azure, where the Queries-Per-Second (QPS) fluctuates unpredictably between roughly 20 and 80. The paper notes this trace is "representative for model serving" and has been used as a benchmark in several prior works. When QPS spikes, many requests suddenly compete for limited GPU memory, and scheduling decisions made in that moment determine whether the system maintains low latency or collapses under queuing delays. Figure 6 shows the distribution of input and output token lengths for the two production workloads studied: SW-Chat (a chatbot application) and SW-Code (a code copilot). Both distributions exhibit substantial variance — some requests have prompts of a few dozen tokens while others span thousands. This variance means a scheduler cannot treat all requests as interchangeable units of work.

2. The two-phase nature of LLM inference. The paper provides a roofline analysis in Figure 3 that clearly distinguishes the prefill phase (processing the user prompt, which is compute-bound due to high operational intensity) from the decode phase (generating one token per forward pass, which is memory-bound). These phases have fundamentally different resource profiles. A request in the prefill phase is computationally intensive but memory-light at the start; a request deep into decode occupies substantial memory for its accumulated KV cache but performs comparatively little arithmetic per token. A scheduler that ignores this distinction will make poor decisions about which requests can productively share a batch.

Why This Problem Matters

The practical stakes are high and multi-dimensional. First, TTFT (Time-To-First-Token) directly shapes user experience. A user typing into a chat interface expects the model to begin responding quickly — delays of even a few seconds feel sluggish. The paper emphasizes this by introducing Normalized TTFT, which divides TTFT by the number of input tokens, arguing that users reasonably "expect requests with short inputs (e.g., short prompts typed into a chat box) to return faster than requests with long inputs (e.g., summarizing a large file)." This metric captures a fairness dimension that coarse-grained latency measurements miss.

Second, Total Generation Time (TGT) affects how long a user waits for the complete response. For applications like code generation, where the full output is needed before the user can proceed, TGT is the primary latency metric.

Third, serving capacity — how sensitively the system's latency degrades as QPS increases — determines the operational cost of a deployment. A system that maintains low latency under higher load can serve more users with the same hardware, directly translating to infrastructure savings. As the authors note, the scheduling policies they study can often "be implemented as 'drop-in replacements' to a system's current policy," meaning the performance improvements they demonstrate require minimal engineering effort relative to their impact.

The paper is also motivated by a gap between research and practice. Despite the ubiquity of the two-level scheduling architecture shown in Figure 1 — a load balancer routing requests to servers, each running an engine-level scheduler that manages a waiting queue, a running queue, and a preempted queue — there is "a potpurri of different scheduling techniques" deployed across systems. vLLM, TensorRT-LLM, and SGLang collectively implement six different scheduling policies (Table 1). This fragmentation suggests the field has not converged on best practices, and the absence of systematic comparisons means practitioners cannot make evidence-based choices.

Where Existing Approaches Fall Short

The paper identifies two categories of existing schedulers, each with distinct limitations.

Engine-Level Schedulers: Simple Policies Leave Performance on the Table

The dominant approach in practical systems is First-Come-First-Served (FCFS) with a user-defined maximum concurrency limit, implemented in vLLM, TensorRT-LLM, and SGLang. Under FCFS, requests are dispatched in arrival order as long as the number of running requests stays below the limit. This is simple, but it suffers from Head-Of-Line (HOL) blocking: a single request with a large prompt or long response can prevent many smaller requests from being dispatched, even though those smaller requests could begin and complete quickly. The concurrency limit itself is a crude knob — "setting a high concurrency limit generally improves throughput but may lead to more preemptions. Setting a low concurrency limit may underutilize the GPU resources" (Section 2.2). There is no adaptation to workload characteristics.

The No-Preempt variant (implemented in TensorRT-LLM and ORCA) takes the opposite approach: it pre-allocates the maximum possible memory for each request based on the smaller of the model's maximum context length or the user-specified maximum response length. This guarantees zero preemptions, but at the cost of severe underutilization because "some requests may not require all of their allocated memory." On the SW-Chat workload in the single-server experiments (Figures 7–8), No-Preempt's TTFT is so high it falls outside the plot limits entirely. The paper notes that No-Preempt only becomes competitive on SW-Code, which is "dominated by prefills" — a special case where processing fewer requests concurrently has less throughput impact.

Research-Originated Schedulers: Effective but Complex

The literature offers more sophisticated approaches, most of which aim to approximate Shortest-Remaining-Processing-Time (SRPT) — the theoretically optimal policy for minimizing mean response time. The logic is that prioritizing requests that are close to completion allows the system to free their memory quickly, reducing queuing delays for all waiting requests. However, the paper identifies specific barriers that prevent these methods from being adopted as drop-in replacements in existing systems.

TRAIL proposes a predictor that estimates how many tokens remain to be generated for each request, then implements a preemptive SRPT-like policy. The limitation is architectural: TRAIL requires access to the model's internal layer activations to make its predictions. In practical serving systems like vLLM, TensorRT-LLM, and DeepSpeed, "the scheduler does not have access to the layer activations" (Section 2.2). The scheduler is a separate component that sees request metadata (arrival time, token counts, memory allocation) but not the model's internal state. Integrating TRAIL's predictor would require modifying the inference engine's forward pass to expose activations to the scheduler — a change that goes well beyond swapping the scheduling policy.

LTR (Learning-To-Rank) reduces the prediction difficulty by only ranking requests by output length rather than predicting absolute lengths. This can be implemented inside existing schedulers, but it requires training a separate ranking model offline and multiplexing GPU resources between the ranking model and the served LLM — introducing both engineering complexity and runtime overhead.

FastServe proposes a Multi-Level Feedback Queue that cleverly shuffles requests between priority levels and proactively moves KV caches between GPU and host memory to enable frequent preemption and resumption. The paper acknowledges this achieves good performance but notes its proactive KV cache management "requires additional implementation effort that goes beyond simply replacing the scheduling policy of current serving engines." Specifically, "FastServe cannot be implemented as a drop-in replacement to scheduling policies in vLLM, SGLang, TensorRT-LLM, or ORCA."

PiA and S3 address problems that are largely solved by architectural innovations now standard in serving systems. PiA batches requests with similar predicted response lengths, but this was designed before Continuous Batching — which allows requests to start and finish independently within a batch — became standard. S3 predicts memory requirements and implements a supervisor for handling mispredictions, but this was designed before Paged Attention enabled dynamic, non-contiguous memory allocation. The paper explicitly excludes these from evaluation because they address shortcomings that "don't exist in these systems" anymore.

Load Balancers: General-Purpose Policies Ignore Request Semantics

On the load balancing side (Section 3.2), many serving systems rely on general-purpose orchestration platforms like Kubernetes with Istio, Envoy, or KNative, which provide LLM-agnostic load balancers: Round-Robin, Random, and Power of Two Random Choices (P2C). These policies treat all requests as equivalent units of load, making decisions based on connection counts or random selection. They ignore the fact that different requests consume vastly different amounts of memory and processing time depending on their token lengths and phase (prefill vs. decode).

The paper notes that while some serving systems "implement their own, application-specific load balancers, [others] continue to rely on the general-purpose load balancers of the orchestration platform" (Section 3.2). This is a missed opportunity because, as the paper demonstrates, a token-aware load balancer can improve TGT by more evenly distributing batch sizes across servers, reducing the number of "overly full batches" that slow down decode for all requests in the batch.

Llumnix represents a more advanced approach: it can migrate KV caches between servers mid-request, enabling dynamic rescheduling for load balancing, performance isolation, and defragmentation. However, "the Llumnix mechanism for KV cache migration is not implemented in many LLM serving systems and requires modifying both the load balancer and the serving engine" (Section 3.2). It is not a drop-in replacement.

How This Paper Positions Itself

The paper's position is defined by a specific design philosophy articulated in the abstract and operationalized throughout: scheduling techniques should be easy to implement, deploy, and configure, while still explicitly accounting for the decisive properties of LLM requests. The authors are not proposing an entirely new scheduling paradigm or a complex prediction model. Instead, they are mining the gap between two extremes:

Research schedulers that achieve good performance but require changes outside the scheduler (TRAIL's layer activation access, FastServe's KV cache migration) or require training additional models (LTR's ranking model).
Practical schedulers that are trivially simple (FCFS, Round-Robin) but ignore request-level information that is readily available within the system, leaving "easy performance gains on the table."

The insight driving both LARRY and SAL is that the metrics needed to make informed scheduling decisions — prompt length, queue depth, available memory, queued token counts — are already available in standard serving systems and can be used without additional infrastructure. LARRY's entire logic is captured in Equation 1, which combines a request's waiting time and its memory consumption with a single tunable parameter α. SAL's logic in Equation 2 combines the memory shortfall (if any) and the queued token count for each server. Both can be implemented in tens of lines of code.

This philosophy is what the paper means by "drop-in replacement" (Table 1): a technique that requires changes only to the scheduler itself, not to memory management, KV cache handling, model forward pass, or inter-server communication. It also means no offline training phase — the policies use online statistics (queue length, available memory, average input/output token ratios) that are trivial to collect during serving.

The paper's empirical contribution is to demonstrate that these simple, request-aware heuristics do not just match the practical baselines — they significantly outperform them and also match or exceed the more complex research-originated methods (in the case of TRAIL+, which is given perfect response-length predictions as an upper bound). This reframes the scheduling problem: the key to better performance is not more sophisticated prediction or more aggressive preemption, but rather using the information already present in the system to make context-dependent decisions about which requests to dispatch and where to route them.

The paper's scope is explicitly limited to online scheduling (requests arrive over time with unknown characteristics) as opposed to offline throughput-oriented scenarios where all requests arrive at once. This choice reflects the dominant deployment pattern for interactive LLM applications like chatbots and code assistants. The workloads evaluated — SW-Chat and SW-Code from Azure OpenAI — are described as representative of this pattern, and Figure 6 shows they exhibit the kind of prompt/output length diversity that makes scheduling non-trivial.

3. Technical Approach

3.1 Reader Orientation

This paper constructs two lightweight scheduling heuristics — one that operates inside each GPU server (LARRY) and one that distributes requests across multiple servers (SAL) — to reduce the latency users experience when interacting with deployed large language models. The shape of the solution is a drop-in scoring function: LARRY computes a single priority score for each queued request using only the request's prompt length (a proxy for memory demand) and its time spent waiting, then dispatches requests in descending score order; SAL computes a single load estimate for each server using only the number of queued prefill tokens and the server's current free memory, then routes each incoming request to the server with the lowest estimated wait time.

3.2 Big-Picture Architecture (Diagram in Words)

The system is a standard two-tier LLM serving deployment as depicted in Figure 1 of the paper, with the two proposed techniques slotting directly into the existing decision points:

Load Balancer (SAL's location): Receives every incoming request before it reaches any server. SAL polls each server periodically for its current number of queued prefill tokens and available GPU memory, then uses Equation 2 to route the request to whichever server is estimated to process it soonest. Between polling intervals, SAL maintains its own estimates by tracking which requests it has sent where.
Engine-Level Scheduler (LARRY's location): Runs on each individual server, managing the waiting queue of requests that have arrived but not yet been dispatched for execution. Whenever the scheduler is invoked (a request finishes, a new request arrives, or memory becomes available), LARRY computes a score for every waiting request using Equation 1, sorts the queue by descending score, and dispatches requests in that order until either the queue is empty or the next request cannot fit in the currently available memory or would exceed the maximum token limit for a single batch.
Serving Engine (vLLM with Paged Attention + Continuous Batching + Chunked Prefill): The underlying infrastructure that actually runs the model forward passes. LARRY and SAL do not modify this layer — they only change which requests enter when and where. The engine itself handles memory allocation (Paged Attention), dynamic batching (Continuous Batching), and mixed prefill/decode batching (Chunked Prefill) exactly as it would with FCFS scheduling.

3.3 Roadmap for the Deep Dive

First, the engine-level scheduling problem (Section 2.1 revisited): A precise restatement of what the scheduler must decide, what information is available when making those decisions, and what constraints (memory, batch size, preemption cost) shape the search space. This is essential because LARRY's design follows directly from the structure of this problem.
Second, LARRY's scoring function (Equation 1): The core mathematical mechanism — how two readily available quantities (wait time and estimated memory consumption) are combined into a single dispatch priority, what the tunable parameter $\alpha$ controls, and how the scoring rule adapts to system load through the queue_len multiplier.
Third, LARRY's dispatch loop: The algorithm that uses these scores — when sorting happens, when a request is skipped versus dispatched, and how starvation is prevented.
Fourth, the load balancing problem (Section 3.1 revisited): Why request-level heterogeneity makes general-purpose load balancers insufficient, and what server-level metrics a load balancer needs to track to make token-aware routing decisions.
Fifth, SAL's load estimation (Equation 2): The core mathematical mechanism — how SAL estimates the waiting time a request would experience on each candidate server by combining memory availability and prefill queue depth into a single scalar load metric.
Sixth, SAL's routing algorithm: How polling, state tracking between polls, and the $\beta$ parameter (average rate of memory freed per request completion) interact to make the routing decision tractable without requiring live coordination between servers.
Finally, the implementation footprint: The concrete code complexity (20 lines for LARRY, 30 lines for SAL in vLLM) and why this matters for practical adoption.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems design and empirical evaluation paper whose core idea is that effective LLM request scheduling does not require complex prediction models or architectural modifications — it requires only that the scheduler pay attention to request-level and server-level metrics that are already available inside standard serving engines and that it make dispatch/routing decisions that adapt to the current system load.

The Engine-Level Scheduling Problem Restated

Before introducing LARRY, the paper defines the precise decision space that any engine-level scheduler must navigate (Section 2.1). Restating this here is essential because LARRY's design is a direct response to the structure of this space.

An LLM serving engine running on a single GPU (or set of GPUs with tensor parallelism) maintains three logical collections of requests: a running set (requests currently being processed, each with a KV cache occupying GPU memory), a waiting queue (requests that have arrived but have not yet been dispatched), and a preempted set (requests that were running but were evicted to free memory, whose KV caches must be recomputed before they can resume). The scheduler is invoked when any of these events occurs: a new request arrives, a running request finishes (frees its KV cache memory), or the system detects that it is about to run out of memory and must preempt something.

When invoked, the scheduler must make two types of decisions:

Dispatching: For each request in the waiting queue (including newly arrived requests and previously preempted requests waiting for resumption), the scheduler must decide whether to move it into the running set. This decision is constrained by two hard limits. First, the memory constraint: the running set's combined KV cache, plus the expected KV cache growth of the new request (at minimum, memory for its prefill tokens), must not exceed available GPU memory — otherwise the request will cause a preemption later. Second, the batch constraint: the serving engine enforces a maximum number of tokens per forward pass (in the paper's experiments, 1024 tokens), and adding a new request's tokens to the current batch must not exceed this limit.

Preemption: When memory is exhausted and new work must be scheduled, the scheduler must select which running request(s) to evict. The cost of preemption is high — the paper states that "restoring the KV cache is an expensive operation, often implemented by recomputing the KV cache from scratch" — so the scheduler has a strong incentive to avoid preemptions by being careful about dispatching decisions.

The key insight that makes this problem non-trivial is that the scheduler does not know how much total memory a request will ultimately consume, because it does not know how many output tokens the model will generate before emitting an end-of-sequence token. The scheduler can observe the request's prompt length (a lower bound on total sequence length) and can observe how much memory the request has consumed so far (for a request that has already started running), but the future memory trajectory is unknown.

The paper identifies a specific structural fact about LLM workloads that LARRY exploits: in many production applications, the prompt dominates the total memory consumption of a request. Figure 5 quantifies this: for both SW-Chat and SW-Code workloads, over 70% of the tokens stored in most requests' KV caches are tokens from the prompt. The paper summarizes:

"Given that (i) the prompt dominates a request's memory consumption in many applications, and (ii) the KV cache only grows slowly after the prefill phase, LARRY dispatches a request once the available memory can hold its prefill KV cache."

This means that prompt length is a good proxy for peak memory demand. A scheduler can use prompt length — which is known before the request starts running — to estimate whether dispatching a request is likely to cause memory pressure later. This proxy is not perfect (a request with a short prompt might generate a very long response), but it is substantially better than making no distinction at all between requests, which is what FCFS does.

The second structural fact is that system load varies. Figure 4 shows QPS fluctuating between roughly 20 and 80 over a 20-minute window in a production trace. During low-load periods, memory is abundant and there is no reason to discriminate between requests — the system can dispatch everything immediately. During high-load periods, when the waiting queue is long and memory is tight, dispatching a request that needs a large chunk of memory will block many other requests that need small chunks of memory and could complete quickly. This is the Head-Of-Line blocking problem that FCFS suffers from.

LARRY's design is a direct response to these two facts: use prompt length to estimate memory demand, and only discriminate between requests (preferring small-memory requests) when the queue is long and memory pressure is high.

LARRY's Scoring Function

LARRY reduces the entire dispatching decision to a single scalar score for each request in the waiting queue. Requests with higher scores are dispatched first. The score function is given by Equation 1:

$\text{score}(r) = \alpha \cdot \text{wait\_time}(r) - \text{queue\_len} \cdot \text{memory}(r)$

where $\text{wait\_time}(r)$ is the time (in seconds) that request $r$ has been sitting in the waiting queue since its arrival, $\text{memory}(r)$ is the estimated memory consumption of request $r$ (approximated by the KV cache size required for its prompt tokens), $\text{queue\_len}$ is the current number of requests in the waiting queue, and $\alpha$ is a tunable weight that controls the tradeoff between prioritizing wait time and prioritizing small memory footprint.

What it computes: The score for a request is a two-term expression. The first term, $\alpha \cdot \text{wait\_time}(r)$ , increases linearly with how long the request has been waiting — this is the anti-starvation mechanism. As a request ages, its score grows, eventually overtaking the scores of newer requests even if those newer requests have smaller memory demands. The second term, $\text{queue\_len} \cdot \text{memory}(r)$ , is a penalty proportional to the request's estimated memory consumption, scaled by the current queue length. When the queue is short (low load), this penalty is small regardless of memory demand, so requests are dispatched roughly in FCFS order. When the queue is long (high load), this penalty dominates for memory-heavy requests, pushing them down in the dispatch order so that light requests can be dispatched first and complete quickly.

The operational effect is: under low load, LARRY behaves like FCFS; under high load, LARRY preferentially dispatches short-prompt requests first while gradually promoting long-waiting requests to prevent starvation. The queue length serves as an adaptive load signal — it is zero-cost to compute (it is simply the scheduler's own count of pending requests) and it naturally captures whether the system is in a regime where HOL blocking matters.

Why this form: The paper's key design choice is to make the memory penalty multiplicative with queue length rather than using a fixed threshold or a binary categorization of requests into "small" and "large." A fixed threshold (e.g., always prefer requests with prompt length < 100 tokens) would fail in two directions: under low load, it would unnecessarily delay large requests that could be dispatched immediately without blocking anything; under extremely high load, a single large request might wait indefinitely if small requests keep arriving. The multiplicative form with $\text{queue\_len}$ automatically calibrates the strength of discrimination to the urgency of memory pressure. When $\text{queue\_len} = 1$ , the penalty is just $\text{memory}(r)$ , which is dwarfed by $\alpha \cdot \text{wait\_time}(r)$ for any reasonable $\alpha$ , so the score ordering is dominated by arrival time. When $\text{queue\_len} = 100$ , the penalty is $100 \cdot \text{memory}(r)$ , creating strong differentiation between light and heavy requests.

The additive form with a single weighting parameter $\alpha$ is chosen for simplicity and interpretability. An alternative multiplicative form like $\text{wait\_time}(r) / \text{memory}(r)$ would have the property that a request with zero wait time always has score zero regardless of memory, and the anti-starvation behavior as wait time grows would be inversely proportional to memory rather than independent of it. The additive form cleanly separates the anti-starvation term from the load-adaptive penalty term, making the effect of $\alpha$ transparent: higher $\alpha$ means stronger anti-starvation, lower $\alpha$ means more aggressive differentiation by memory under load.

The paper provides guidance on choosing $\alpha$ based on which latency percentiles matter most for the application. Section 6 reports results for $\alpha \in \{1, 500, 1000\}$ and finds that low values ( $\alpha = 1$ ) give the best performance for most latency percentiles (p50, p75, p90) because they enable the most effective HOL blocking avoidance. However, high $\alpha$ values reduce tail latency (p99) because they prevent large requests from being deprioritized too aggressively. At the extreme, the paper notes that "for very high values of $\alpha$ (larger than 1000), LARRY degenerates to First-Come-First-Served (FCFS) — this happens when $\alpha$ is so high that the waiting time dominates each request's score in Equation 1." The recommendation is to use $\alpha = 1$ as a default, and increase it only if tail latency is the primary concern.

LARRY's Dispatch Loop

The scoring function alone does not define the scheduler — it must be embedded in a dispatch algorithm that determines when scoring and sorting happen and how the sorted order translates into dispatch decisions.

The paper describes the dispatch loop as follows (Section 4.1): LARRY is invoked whenever a scheduling decision is needed — when a new request arrives, when a running request completes and frees memory, or when the system needs to preempt. At each invocation, LARRY performs these steps:

Score all waiting requests: For every request currently in the waiting queue, compute $\text{score}(r)$ using Equation 1 with the current values of $\text{wait\_time}(r)$ and $\text{queue\_len}$ . Note that $\text{queue\_len}$ is the same for all requests being scored in a single invocation — it captures the state of the system at the moment of scheduling.
Sort by descending score: The waiting queue is reordered so that the request with the highest score is at the front.
Sequential dispatch with feasibility check: Starting from the front of the sorted queue, the scheduler attempts to dispatch each request in order. For each request, it checks two feasibility conditions: (a) the available GPU memory must be sufficient to hold the request's prefill KV cache (estimated from prompt length), and (b) adding the request's prefill tokens to the current batch must not exceed the maximum token limit per batch (1024 tokens in the paper's experiments). If both conditions are satisfied, the request is dispatched and removed from the waiting queue. If either condition fails, the request is skipped, and the scheduler moves to the next request in the sorted order.
Stop when queue exhausted or infeasible: The dispatch loop continues through the sorted queue until either (i) all requests have been considered, or (ii) "a request is reached that cannot be scheduled anymore (due to insufficient memory or the current batch already reaching the maximum token size)." The paper specifies this stopping condition as continuing until a request is encountered that cannot be dispatched — the loop does NOT continue past an infeasible request to check later requests in the sorted order, even though those later requests might have smaller memory requirements and could theoretically fit. This is a design choice that trades some packing efficiency for simplicity: once the highest-priority feasible requests have been dispatched, the remaining requests are left in the queue to be reconsidered at the next scheduling event (when memory conditions may have changed).

This dispatch algorithm has several properties that matter for practical behavior. First, because the scoring function includes $\text{wait\_time}$ , a request that is skipped in one invocation (because memory is insufficient) will have a higher score in the next invocation (because its wait time has increased). This means large requests are not permanently blocked — they eventually accumulate enough wait-time score to outrank smaller requests, at which point they will be dispatched (assuming memory is available). Second, because $\text{queue\_len}$ changes dynamically as requests are dispatched and as new requests arrive, the strength of the memory-penalty term adapts continuously. If a burst of new arrivals suddenly increases the queue length, the penalty for large requests increases immediately at the next scheduling invocation, automatically shifting the dispatch priority toward smaller requests until the burst subsides. Third, the dispatch loop does not preempt running requests to make room for higher-priority waiting requests. LARRY is a non-preemptive scheduler — it only makes decisions about which waiting requests to dispatch when capacity becomes available through natural completions. This avoids the overhead of KV cache eviction and recomputation, which the paper identifies as expensive and which TRAIL and FastServe must contend with.

The Load Balancing Problem Restated

When a serving system uses multiple server instances (data parallelism with model replicas), a load balancer sits in front of all servers and must route each incoming request to exactly one server. The paper defines the load balancing problem for LLM serving in Section 3.1, emphasizing two dimensions of request heterogeneity that make general-purpose load balancers insufficient.

First, request sequence length varies widely. For SW-Chat, input tokens range from single-digit to over 4000 tokens per request (Figure 6). A request with a 4000-token prompt will occupy substantially more GPU memory for a substantially longer time than a request with a 10-token prompt. If the load balancer treats these requests as equivalent — as Round-Robin does, assigning them to servers in fixed rotation — it can inadvertently concentrate large-memory requests on a single server, causing that server to exhaust memory and queue while other servers have spare capacity.

Second, the processing phase of a request matters for load characterization. The prefill phase is compute-bound (high operational intensity, as shown in Figure 3's roofline analysis), meaning it benefits from being batched with other prefill tokens to amortize the cost of loading model weights. The decode phase is memory-bound (low operational intensity), meaning throughput improves by increasing batch size (Figure 2). A load balancer that tracks only connection counts — as P2C does, routing to the server with fewer in-flight TCP connections — is counting the wrong thing. Two servers might have the same number of in-flight requests but vastly different loads because one server's requests are all in the decode phase (occupying large KV caches but generating one token per forward pass) while the other server's requests are in the prefill phase (processing large batches of prompt tokens).

The paper's key insight for load balancing is that the relevant measure of server load is not how many requests are assigned to it, but rather how many prefill tokens are queued waiting to be processed and how much free memory the server has for new KV caches. These two metrics directly capture the two bottlenecks that cause a request to wait: (i) waiting for its prefill to be processed when many other prefill tokens are ahead of it in the queue, and (ii) waiting for memory to become available when the GPU is full.

SAL's Load Estimation

SAL quantifies the load on each server $s$ after adding a candidate request $r$ using a single scalar function that accounts for both the memory bottleneck and the prefill processing bottleneck. The load function is given by Equation 2:

$\text{load}(s, r) = \max\big(\beta \cdot (\text{memory}(r) - \text{free\_mem}(s)),\ \text{queued\_tokens}(s, r) / \text{max\_tokens\_per\_batch}\big)$

where $\text{memory}(r)$ is the estimated memory required for request $r$ 's input tokens (the prefill KV cache size), $\text{free\_mem}(s)$ is the current free GPU memory on server $s$ (as reported by the server during the most recent poll), $\beta = (\mu_{in} + \mu_{out}) / \mu_{out}$ is the approximate average rate at which memory is freed due to requests finishing (computed from the workload's average input tokens $\mu_{in}$ and average output tokens $\mu_{out}$ ), $\text{queued\_tokens}(s, r)$ is the number of prefill tokens currently queued on server $s$ (waiting to be processed) plus the number of prefill tokens in request $r$ itself, and $\text{max\_tokens\_per\_batch}$ is the maximum number of tokens allowed in a single forward pass (1024 in the paper's experiments).

What it computes: The function produces two estimates of how long request $r$ would wait on server $s$ , and takes the maximum (the bottleneck). The first term, $\beta \cdot (\text{memory}(r) - \text{free\_mem}(s))$ , estimates the waiting time due to memory unavailability. When the request's required memory exceeds the server's current free memory, the difference $(\text{memory}(r) - \text{free\_mem}(s))$ is positive, and dividing by the average rate $1/\beta$ at which memory is freed (since $\beta = (\mu_{in} + \mu_{out}) / \mu_{out}$ expresses tokens-freed-per-output-token-generated) gives an estimate of how long the request must wait before enough memory becomes available. When the request's memory requirement is less than free memory, this term is negative, but the $\max$ operation ensures it contributes zero to the load — the request is not memory-blocked. The second term, $\text{queued\_tokens}(s, r) / \text{max\_tokens\_per\_batch}$ , estimates the number of full batches worth of prefill tokens ahead of this request. If server $s$ has 3000 queued prefill tokens and the batch limit is 1024, those tokens represent roughly 3 full batches that must be processed before request $r$ 's prefill can begin.

Why this form: SAL uses a $\max$ over two bottleneck estimates rather than a sum because a request's waiting time is determined by whichever constraint binds first, not by the sum of both. If memory is abundant but the prefill queue is long, the request waits for prefill processing — adding the memory term (which would be zero or negative) would incorrectly lower the load estimate. If memory is scarce but the prefill queue is empty, the request waits for memory to become available — adding the prefill term (which would be small or zero) would incorrectly lower the load estimate. The $\max$ captures the worse of the two constraints, which is the one that will actually delay the request.

The $\beta$ parameter accounts for the fact that memory is not freed at a constant rate — it is freed when requests complete, and requests complete by generating output tokens. The ratio $(\mu_{in} + \mu_{out}) / \mu_{out}$ is the average number of tokens in a complete request (input plus output) divided by the average number of output tokens, which captures how many tokens-worth of KV cache memory are freed per output token generated. For SW-Chat, $\beta = 1365/211 \approx 6.5$ , meaning roughly 6.5 tokens of KV cache are freed for each output token generated. For SW-Code, $\beta = 2074/27 \approx 76.8$ , reflecting that code requests have much longer inputs relative to outputs. These $\beta$ values are computed once from workload statistics and do not need to be updated dynamically — they capture the coarse-grained memory-to-output ratio of the application.

The paper notes that these averages "can easily be recorded during online serving," making $\beta$ a zero-cost parameter that requires no training or offline profiling beyond tracking the running mean of input and output token counts.

SAL's Routing Algorithm

The load estimation function in Equation 2 would be straightforward to compute if the load balancer had perfectly up-to-date information about every server's $\text{free\_mem}$ and $\text{queued\_tokens}$ . However, continuously querying these statistics from every server for every incoming request would introduce prohibitive overhead. SAL solves this with a polling-plus-tracking approach described in Section 4.2.

Periodic polling. SAL polls each server for its current $\text{free\_mem}$ and $\text{queued\_tokens}$ at a fixed interval — 10 times per second in the paper's implementation. The authors state that "polling the server for these statistics incurs negligible overhead, which allows for frequent polling." Ten polls per second means the load balancer's view of each server's state is at most 100 milliseconds stale, which is short relative to the timescale of request processing (prefill phases for long prompts can take hundreds of milliseconds to seconds).

State tracking between polls. Between polling intervals, SAL maintains its own estimates of each server's $\text{queued\_tokens}$ and $\text{memory}$ by assuming that no requests have finished in the interim. Specifically, whenever SAL routes a request with $N$ input tokens to server $s$ , it immediately updates its local counters: it adds $N$ to the tracked $\text{queued\_tokens}$ for server $s$ , and it adds the estimated memory requirement for $N$ tokens to the tracked memory consumption for server $s$ . The key assumption — "assuming no request finished in the meantime" — is conservative: it means SAL's estimates of load may be slightly higher than reality because completed requests are not subtracted until the next poll, but this errs in the direction of avoiding overloading a server.

Routing decision. For each incoming request $r$ , SAL computes $\text{load}(s, r)$ using Equation 2 for every server $s$ in the deployment, using the most recent poll data plus the tracked updates from requests routed since the last poll. It then routes $r$ to the server with the smallest $\text{load}(s, r)$ . This is a greedy, per-request decision — SAL does not batch routing decisions or attempt to solve a global optimization. The greedy choice works because the load function is designed to be additive: routing a request to the least-loaded server minimizes the maximum load across servers in expectation, which is the standard justification for join-the-shortest-queue policies.

Why this approach over alternatives: The paper contrasts SAL's token-aware routing with three general-purpose alternatives. Round-Robin and Random use no load information at all — they are stateless and cannot adapt to imbalances. P2C samples two servers and picks the one with fewer in-flight TCP connections, which uses load information but the wrong kind of load information for LLM serving: "two servers might have the same number of in-flight requests but vastly different loads because one server's requests are all in the decode phase (occupying large KV caches but generating one token per forward pass) while the other server's requests are in the prefill phase."

SAL's approach also contrasts with Llumnix, which achieves more precise load balancing by migrating KV caches between servers mid-request. Llumnix's mechanism can correct load imbalances that developed before the load balancer reacted, but it requires modifying both the load balancer and the serving engine to support KV cache transfer. SAL achieves coarser-grained balance — it only controls initial request placement — but requires no changes to the serving engine and no inter-server communication beyond the lightweight polling.

Implementation Footprint and Deployment Model

The paper emphasizes a specific design philosophy throughout: both LARRY and SAL are characterized by their minimal implementation complexity, which is not incidental but is central to their claimed contribution. The abstract states that techniques from the literature "introduce significant complexity" while practical systems "leave easy performance gains on the table." LARRY and SAL are explicitly designed to capture those easy gains without introducing complexity.

LARRY's implementation. The paper reports that LARRY was "implemented in just 20 lines of code in vLLM." This is possible because LARRY only modifies the scheduling policy — the function that decides dispatch order — and uses only information already available to the vLLM scheduler: each request's arrival timestamp (from which $\text{wait\_time}$ is computed), each request's prompt length (from which $\text{memory}$ is estimated), and the current queue length (the size of the scheduler's own waiting queue data structure). LARRY does not require changes to memory management (Paged Attention continues to operate as before), KV cache handling (preemption and recomputation proceed identically), or the model forward pass (no activations or intermediate states are accessed).

SAL's implementation. Similarly, SAL was "implemented in 30 lines of code in the system described in Section 5." The implementation consists of: a data structure tracking per-server $\text{queued\_tokens}$ and $\text{free\_mem}$ (updated on poll and on route), the load computation function (Equation 2), the routing loop that computes load for each server and selects the minimum, and a polling thread that queries each server's status at 10 Hz.

What "drop-in replacement" means. Table 1 defines "drop-in replacement" with two criteria: (1) "doesn't require changes outside of the scheduler nor upfront training of additional prediction models," and (2) "accounts for properties that are specific to LLM requests, opposed to schedulers that apply general-purpose policies to LLM serving." LARRY and SAL satisfy both criteria. FCFS, No-Preempt, RR, Random, and P2C satisfy criterion (1) but fail criterion (2) — they are either general-purpose or ignore request properties. TRAIL, FastServe, and Llumnix satisfy criterion (2) but fail criterion (1) — they require changes beyond the scheduler. LTR satisfies criterion (1) (it can be implemented inside the scheduler) but the paper notes it "requires to train a ranking model in an offline phase" and "computing resources (i.e., GPU cycles and memory) need to be multiplexed between the ranking model and the served LLM," which violates the spirit of the drop-in replacement criterion as a practical deployment consideration.

This positioning matters for adoption. The paper is targeting practitioners who operate LLM serving deployments — these are engineers who can modify a scheduling policy's source code (a localized change) but cannot or will not overhaul the memory management subsystem, modify the model's forward pass to expose internal activations, or deploy a separate prediction model that competes for GPU resources with the model being served. LARRY and SAL are designed to fit into the existing engineering workflow of such practitioners.

4. Key Insights and Innovations

Innovation 1: Request-Level Awareness Is the Key Distinction, Not Prediction Sophistication

The field's dominant assumption — visible across TRAIL, LTR, FastServe, PiA, and S3 — is that effective LLM scheduling requires predicting something about the future: remaining response length, total memory consumption, or expected batch compatibility. This paper's most fundamental conceptual move is to reject that premise entirely. LARRY and SAL make zero predictions about future request characteristics. They never estimate how many output tokens remain, never forecast whether a running request will finish soon, and never guess whether memory will become available at a particular rate. Instead, they use only what is known with certainty at decision time: the prompt length (known before dispatch), the time a request has already waited (trivially tracked), the current queue depth (a counter), and the current free memory and queued token counts on each server (polled directly).

This is not merely a practical simplification. It is a conceptual reframing of what makes scheduling hard. The paper implicitly argues that the dominant source of scheduling inefficiency is not uncertainty about the future — it is the failure to discriminate between requests based on what is already known. FCFS knows each request's prompt length but chooses to ignore it. Round-Robin knows nothing about server load because it never asks. P2C asks about server load but asks the wrong question (connection count rather than token count and memory). When LARRY dispatches short-prompt requests first under high load, it is not predicting that these requests will finish faster (though they often do, as Figure 5 shows that prompt length dominates total memory). It is making the simpler observation that a small memory allocation blocks fewer other requests than a large one, regardless of when either request ultimately finishes.

The evidence that prediction is not the bottleneck comes from the TRAIL+ results in Figures 7 and 8. TRAIL+ is given perfect, zero-cost predictions of ground-truth response lengths — it literally knows exactly how many tokens remain for every request. This is the strongest possible version of the prediction-based approach. Yet TRAIL+ does not outperform LARRY on TTFT (LARRY achieves 1.8–2.1× lower p50 TTFT) and sometimes underperforms FCFS at tail percentiles because it "doesn't account for starvation" and can deprioritize requests indefinitely based on output length alone. The field's investment in better predictors — ranking models, embedding-based estimators, multi-level feedback approximations — is attacking a problem whose ideal solution does not beat a simple heuristic that pays attention to prompt length and queue depth.

This is a fundamental shift, not an incremental refinement, because it reorients the research agenda. If the paper's finding holds across other workloads and model architectures, the implication is that future scheduling research should invest in better request-aware heuristics that exploit known information rather than better predictors of unknown information. The space of possible heuristics using prompt length, queue depth, memory state, and token counts is large and largely unexplored — LARRY and SAL occupy only two points in that space.

Innovation 2: Load-Adaptive Discrimination via Multiplicative Queue-Length Scaling

A scheduler that always prefers short-prompt requests would starve long-prompt requests during sustained load. A scheduler that never discriminates by prompt length (FCFS) suffers HOL blocking under load. The standard way to navigate this tradeoff in scheduling is to use priority classes with aging — requests start at low priority and are promoted over time. This is what multi-level feedback queues (FastServe) and shortest-remaining-processing-time with preemption (TRAIL) attempt.

LARRY's innovation is to fold the load signal directly into the discrimination strength rather than into the priority mechanism. The queue_len * memory(r) term in Equation 1 means that the scheduler automatically discriminates more aggressively when the queue is long (memory pressure is high) and less aggressively when the queue is short (memory is abundant). There is no separate aging mechanism, no priority classes, and no explicit threshold separating "high load" from "low load." The discrimination strength is continuous and proportional to the instantaneous queue length.

This is distinctive because it eliminates a tuning problem that plagues priority-based schedulers: setting the rate at which requests age. If aging is too fast, the scheduler degenerates to FCFS under moderate load and loses the benefits of discrimination. If aging is too slow, long-prompt requests starve. LARRY replaces this with a single parameter α that has a clean interpretation — it weights wait time against memory penalty — and that the paper shows is not especially sensitive: α = 1 works well for most latency percentiles across both workloads and both hardware configurations (Figures 7–9). The only application-specific tuning is whether to increase α to reduce tail latency for large requests, and the paper provides a direct diagnostic for this tradeoff in Figure 9.

The conceptual contribution is that discrimination strength should be a function of system state, not a static policy. Prior schedulers (TRAIL, LTR, FastServe) implement static policies — always prefer shorter jobs, always use the same priority levels — and then add mechanisms (preemption thresholds, quanta lengths) to mitigate the pathologies that arise when the static policy interacts with varying load. LARRY inverts this: the policy itself varies with load, and the mechanisms (wait time accumulation, skip-if-infeasible dispatch) are simple because the policy is already adaptive. This is an incremental refinement of the adaptive scheduling idea (prior work like INFaaS and Shepherd adapt to load for general DNN serving), but it is a fundamental conceptual simplification for the LLM-specific case because it shows that a single, continuously adaptive term can replace the machinery of priority classes, aging functions, and preemption logic.

The evidence that this works in practice is the serving capacity results in Figures 7 and 8 (rightmost columns). As QPS scales up, LARRY's p50 and p95 TTFT degrade more slowly than any other method's. This is exactly what load-adaptive discrimination should achieve: when load increases, the scheduler responds by discriminating more aggressively, which maintains low latency for most requests. The static policies (FCFS, TRAIL+) cannot adapt their discrimination strength to load, so their latency degrades faster.

Innovation 3: Token-Aware Load Balancing via Bottleneck Estimation Rather Than Load Averaging

General-purpose load balancers (Round-Robin, Random, P2C) treat load as a scalar: number of connections, number of assigned requests, or some smoothed average thereof. The implicit assumption is that all requests impose equivalent load on a server. The paper's diagnostic contribution is to demonstrate that this assumption is catastrophically wrong for LLM serving, and that the fix is not to compute a better scalar average but to separately track the two independent bottlenecks — memory availability and prefill processing capacity — and route based on whichever is binding.

SAL's Equation 2 computes two independent estimates of how long a request would wait — one for the memory bottleneck (how long until enough KV cache space is freed) and one for the prefill bottleneck (how many batches of prefill tokens are ahead in the queue) — and takes the max. This is structurally different from a weighted sum or a utilization percentage. A weighted sum would allow a server with abundant memory but a very long prefill queue to appear lightly loaded (because the memory term is small), when in fact the request would wait a long time for its prefill to be processed. The max correctly identifies that the waiting time is determined by the worse of the two constraints.

The conceptual move is to define load not as a property of the server (how "busy" it is) but as an estimated waiting time for a specific candidate request. Equation 2 is parameterized by both the server state s AND the request r. This means SAL does not compute a single load number per server and then route to the minimum. It computes, for each server, what this specific request's experience would be if routed there. Two requests arriving simultaneously — one with a 100-token prompt and one with a 4000-token prompt — would cause SAL to compute different server rankings, because the memory-shortfall term β * (memory(r) - free_mem(s)) would be zero for the small request on a server with modest free memory but positive for the large request on the same server.

This is a fundamental reframing of the load balancing problem — from "which server is least busy right now?" to "which server will process this specific request soonest?" — and it requires no more information than general-purpose load balancers already have access to (token counts and memory statistics are exposed by standard serving engines). The evidence for its effectiveness is in Figures 10 and 11: SAL consistently achieves the lowest TTFT and TGT across all combinations of engine-level schedulers. Notably, SAL's advantage over P2C and Random is most visible in TGT (1.1–1.3× improvement in p50 and p95), which the paper attributes to SAL more evenly distributing prefill tokens across servers and thereby reducing the number of "overly full batches" that slow down decode for all requests in the batch. This is a second-order effect — load balancing affects not just queuing delay but also the per-token processing time of already-running requests — that general-purpose load balancers cannot address because they do not track the token-level composition of server load.

Innovation 4: The "Drop-In Replacement" as an Evaluative Category for Systems Research

This innovation is methodological rather than technical. The paper introduces and rigorously applies a specific evaluative criterion — "drop-in replacement" — that captures a real constraint in production ML infrastructure: scheduling policies are modified far more easily than memory management subsystems, KV cache handling, or model forward-pass logic. Table 1 operationalizes this criterion with two precise conditions: the technique must not require changes outside the scheduler, and it must not require upfront training of additional prediction models.

This matters because it creates a taxonomy that was implicit in the literature but never explicitly articulated. Prior work presented schedulers as a flat category — TRAIL, FastServe, LTR, S3, and FCFS were all "LLM schedulers" — without distinguishing which could actually be deployed in an existing serving system without a multi-quarter engineering effort. The paper's survey (Sections 2–3) systematically identifies, for each prior method, exactly which architectural assumption prevents it from being a drop-in replacement: TRAIL requires layer activation access that the scheduler does not have; FastServe requires proactive KV cache migration between GPU and host memory; S3 requires a separate memory management supervisor; PiA assumes no Continuous Batching; Llumnix requires KV cache migration between servers. These are not vague "complexity" complaints — they are specific, named architectural coupling points.

The significance of this framing extends beyond this paper's results. It provides a vocabulary for a practical tension that every LLM serving team faces: the gap between what scheduling research demonstrates in simulation or in a custom-built system, and what can be implemented in vLLM, TensorRT-LLM, or SGLang with a manageable diff. The fact that LARRY and SAL outperform TRAIL+ (which is given unrealistically perfect predictions) while being implementable in 20–30 lines of code is not just a performance result — it is an existence proof that the Pareto frontier of schedulers includes points that are both high-performance and low-implementation-cost. Prior to this paper, a practitioner looking at the literature might reasonably conclude that good scheduling requires either sophisticated prediction models (TRAIL, LTR) or architectural changes (FastServe, Llumnix). The paper demonstrates that this is false, and it does so by comparing all methods inside the same codebase rather than relying on each method's self-reported numbers from different experimental setups.

This is an incremental contribution to systems methodology — the idea of "drop-in compatibility" as an explicit evaluation axis — but it has fundamental implications for how scheduling research should be conducted and evaluated going forward. A paper proposing a new LLM scheduler should now, at minimum, specify which existing serving engines it can be implemented in as a drop-in replacement, and if it cannot, should justify why the required architectural changes are worth the performance gain.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on two production workload traces from Azure OpenAI: SW-Chat (a chatbot application) and SW-Code (a coding copilot). These traces record only the token lengths of input prompts and generated outputs, not the actual prompt text. The emulated requests "consist of a prompt with random tokens but the same token length as the recorded prompt in the trace," and the system is forced to generate responses with exactly the recorded output lengths (Section 5). The arrival pattern follows a trace of Azure Function calls that has been noted as "representative of model serving loads in several works" (Section 5, citing Zhang et al., Romero et al., and Kossmann et al.).
Base model(s). Two Llama-3 model configurations are evaluated: Llama-3 70B with FP8 quantized weights on two H100 GPUs using 2-way tensor parallelism, and Llama-3 8B on a single A100 GPU with 40GB memory (Section 5). The paper does not explicitly justify why Llama-3 was chosen over alternatives, but the two configurations span a range of deployment scales (8B parameter model on a single consumer-grade GPU vs. 70B parameter model on datacenter GPUs), which tests whether the scheduling techniques' behavior generalizes across hardware tiers and model sizes.
Metrics. Five performance metrics are reported (Section 5). Time To First Token (TTFT) measures the elapsed time from issuing a request until the first response token is generated, including queuing delay and the entire prefill phase. Normalized TTFT divides TTFT by the number of input tokens, motivated by the observation that "users may expect requests with short inputs (e.g., short prompts typed into a chat box) to return faster than requests with long inputs (e.g., summarizing a large file)." Total Generation Time (TGT) captures the full time from request issuance to complete response generation. Serving capacity measures how TTFT degrades as the workload's Queries-Per-Second (QPS) is linearly scaled up by factors typically ranging from 0.5× to 2× or higher, with the scaling factor adjusted "such that the system is neither under-provisioned nor over-provisioned" at the base factor of 1×. The paper reports both p50 and p95 TTFT across scaling factors. Throughout all experiments, preemption rates were observed to be below 0.1% for all schedulers and scaling factors (Section 5).
Baselines. For engine-level schedulers (Section 2.2), the paper evaluates FCFS (First-Come-First-Served, implemented in vLLM, TensorRT-LLM, and SGLang), No-Preempt (FCFS with pre-allocation of maximum KV cache, implemented in TensorRT-LLM and ORCA), and TRAIL+ (an upper bound on TRAIL's performance that has access to ground-truth response lengths at zero runtime cost, with the preemption threshold parameter c set to 0 after sweeping). PiA and S3 are excluded because they address problems solved by Continuous Batching and Paged Attention respectively — the paper explicitly states they "don't consider PiA and S3 in our evaluation" (Section 6). FastServe is excluded because its KV cache migration mechanism "cannot be implemented as a drop-in replacement." LTR is discussed but not evaluated — the paper notes it requires training a separate ranking model and multiplexing GPU resources between the ranking model and the served LLM. For load balancers (Section 3.2), the paper evaluates P2C (Power of Two Random Choices, implemented in Envoy and KNative), Random (sampling a server uniformly at random, implemented in Istio and Envoy), and Round-Robin (discussed but grouped with Random as achieving "similar performance" — Section 7 states "we further evaluate Random, which is also widely used and achieves similar performance as policies like Round-Robin"). Llumnix is excluded because its KV cache migration mechanism requires modifying both the load balancer and the serving engine.
Generation budget / compute accounting. The paper does not measure scheduling overhead in terms of FLOPs or GPU cycles. Instead, it measures the end-to-end latency that users experience (TTFT, Normalized TTFT, TGT) when the system processes workloads at varying QPS levels. The implicit "budget" is the hardware configuration (fixed GPU count and memory capacity), and the scheduler's job is to minimize latency for a given workload intensity on that fixed hardware. The paper explicitly controls for implementation effects by implementing all evaluated scheduling policies "inside the same, representative serving engine" based on vLLM (Section 5). A critical methodological detail: the paper identifies and mitigates a bottleneck in vLLM's driver process that would otherwise limit scheduling evaluation. "At high QPS, the driver process becomes oversubscribed and fails to keep up with adding queries to the waiting queue," creating back pressure that makes the scheduler unaware of arriving requests. The authors "modify vLLM and mitigate the data marshalling overheads to occur outside the driver process (e.g., the tokenization happens in a separate process before the query is issued to the system)" and verify "that the rate at which queries are added to the waiting queue precisely reflects the rate at which queries are issued according to the workload trace" (Section 5).
Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. Results are reported as latency percentiles (p50, p75, p95, p99) from single runs of each configuration on the workload traces. The workload traces themselves are deterministic replays of recorded production timestamps and token lengths. The paper sweeps the workload scaling factor (the x-axis in the "Serving capacity" plots) and the LARRY parameter α, but does not report variance across multiple runs. This is standard for systems benchmarking papers where the primary sources of variance (hardware, workload arrival pattern) are controlled, but it means statistical significance is not quantified.

Main Quantitative Results

The results are organized into two logical groupings: single-server experiments isolating engine-level schedulers (Section 6, Figures 7–9), and multi-server experiments evaluating load-balancer and engine-scheduler combinations (Section 7, Figures 10–11). Within each grouping, the paper evaluates both hardware configurations (H100 with Llama-3 70B and A100 with Llama-3 8B) and both workload traces (SW-Chat and SW-Code).

Single-Server Engine-Level Scheduler Comparison

Headline result for TTFT. LARRY achieves substantially lower TTFT than all other engine-level schedulers across both hardware configurations and both workloads. On the H100/Llama-3 70B deployment (Figure 7, SW-Chat), LARRY's p50 TTFT is approximately 0.3 seconds compared to roughly 0.6 seconds for FCFS and TRAIL+, representing a 2× reduction. The paper states LARRY's "p50 TTFT is 1.8×–2.1× lower than the one of the next-best method" (Section 6). At the p95 level on the same configuration, LARRY's TTFT is approximately 0.8–1.0 seconds compared to roughly 1.0–1.2 seconds for FCFS and TRAIL+, which the paper quantifies as "1.2×–1.4× lower than the one of the next-best method" (Section 6). The advantage is larger at the median than at the tail, which is consistent with LARRY's design: it prioritizes short-prompt requests that benefit the median, while long-prompt requests (which dominate the tail) may be deprioritized under low α values.

Normalized TTFT. The normalized metric amplifies LARRY's advantage because it penalizes schedulers that make short-prompt requests wait behind long-prompt requests (the HOL blocking pathology). On the same Llama-3 70B/SW-Chat configuration (Figure 7), LARRY's p50 Normalized TTFT is approximately 2–3 ms/token compared to roughly 4–5 ms/token for FCFS and TRAIL+, and its p95 Normalized TTFT is approximately 5–7 ms/token compared to roughly 10–15 ms/token for the alternatives. The paper reports LARRY's "Normalized p50 latency is 1.3×–1.5× lower than the one of the next-best method, and its Normalized p95 TTFT is 3.3×–5.5× lower than the one of the next-best method" (Section 6). The larger gap at p95 (up to 5.5×) indicates that FCFS and TRAIL+ inflict substantial normalized latency on large-prompt requests, while LARRY's anti-starvation term (α · wait_time) eventually promotes those requests.

Total Generation Time. Across all engine-level schedulers, TGT differences are minimal on SW-Chat because the decode phase dominates and preemption rates are below 0.1%. On SW-Code, which has shorter response lengths, "differences in TTFT show more significantly" in the TGT, but the paper does not report specific TGT ratios. No-Preempt's TGT on SW-Chat is "beyond plot limits" (Section 6) for the H100 deployment — the paper does not state the actual value, but the fact that it falls outside the plotted range (which extends to approximately 40 seconds for TGT in Figure 7) indicates extreme degradation.

No-Preempt performance. No-Preempt performs dramatically worse than all other methods on SW-Chat. For the Llama-3 70B deployment (Figure 7), No-Preempt's TTFT is "so high on SW-Chat, that we didn't plot them (beyond plot limits)." The paper explains this as a consequence of SW-Chat having longer responses than SW-Code: No-Preempt allocates maximum memory to each request based on the user-specified maximum response length, and when actual responses are long, this leads to severe underutilization — the GPU has memory allocated but idle because the concurrency limit prevents new requests from being dispatched. On SW-Code, which is "dominated by prefills" (Section 6), No-Preempt performs significantly better because requests complete quickly and the pre-allocation penalty is less severe. This is a workload-specific finding that the paper uses to motivate why No-Preempt, despite being implemented in production systems (TensorRT-LLM, ORCA), is not a robust default.

TRAIL+ behavior. TRAIL+ performs similarly to FCFS in terms of TTFT on most configurations. The paper notes that "on some deployments, TRAIL+ incurred starvation of large requests which lead to high tail latencies" (Section 6). This occurs because TRAIL+ implements an SRPT-like policy that preferentially dispatches requests with few remaining output tokens, and with c = 0 (the preemption threshold), it can preempt running requests in favor of newly arrived short requests. When long-running requests are repeatedly preempted, their TTFT increases. The paper notes that TRAIL+ is "designed for workloads with short input lengths and long output lengths, which significantly differs from the application traces used in our experiments" (Section 6). SW-Chat and SW-Code both have average input lengths substantially larger than average output lengths (see Figure 6: SW-Chat mean input ~1365 tokens vs. mean output ~211 tokens; SW-Code mean input ~2074 tokens vs. mean output ~27 tokens). This is an important methodological observation: techniques designed for one workload profile (e.g., long-form generation with short prompts) may not transfer to another (e.g., chatbots and code assistants where prompts dominate responses).

Serving capacity (TTFT sensitivity to QPS scaling). The rightmost columns of Figures 7 and 8 show how p50 and p95 TTFT degrade as the workload QPS is scaled up. LARRY exhibits the flattest curves — its TTFT increases more slowly than all alternatives as load increases. On the H100/SW-Chat configuration (Figure 7, "Serving capacity"), FCFS and TRAIL+ show p50 TTFT rising from approximately 0.5 seconds at 1× scaling to approximately 1.5–2.0 seconds at 3× scaling, while LARRY's p50 TTFT rises from approximately 0.3 seconds to approximately 0.8–1.0 seconds over the same range. At p95 on the same configuration, all methods show steeper degradation, but LARRY maintains a narrower advantage. The paper attributes LARRY's superior scaling behavior to its load-adaptive discrimination: under higher QPS, the queue length term in Equation 1 increases, causing LARRY to discriminate more aggressively between light and heavy requests and thereby preventing HOL blocking from cascading as throughput pressure increases. TRAIL+ also shows relatively good p50 scaling because it "approximately prioritizes requests with smaller prefills" (due to a slight correlation between input and output token lengths), but its p95 scaling is worse because it "doesn't account for starvation."

Effect of α on LARRY. Figure 9 evaluates LARRY's TTFT for α ∈ {1, 500, 1000} on both hardware configurations running SW-Chat. At the p50 level, α = 1 achieves the lowest TTFT (approximately 0.3 seconds for the Llama-3 70B deployment, approximately 0.5 seconds for the Llama-3 8B deployment). As α increases, p50 TTFT rises — at α = 1000, LARRY's p50 approaches FCFS performance (the paper notes that for "very high values of α (larger than 1000), LARRY degenerates to First-Come-First-Served"). However, the relationship inverts at the p99 level: α = 1 produces higher p99 TTFT (approximately 5–6 seconds for the 70B deployment) than α = 500 or α = 1000 (approximately 4 seconds and 3 seconds respectively). The paper explains this crossover: low α aggressively deprioritizes large-prompt requests, which delays them and inflates the tail; high α limits this deprioritization, keeping large requests' waiting times lower at the cost of allowing some HOL blocking that increases median latency. The paper's recommendation is to "choose low values for α (e.g., α = 1), which make LARRY most effective at avoiding HOL blocking," while noting that "applications that are sensitive to tail latencies (e.g., p99 TTFT) should choose higher values for α" (Section 6). The paper does not explore whether a dynamic α — varying with system load or request characteristics — could capture both benefits simultaneously.

Multi-Server Load Balancing and Combined Scheduling

The multi-server experiments (Section 7) evaluate all nine combinations of three load balancers (SAL, P2C, Random) and three engine-level schedulers (LARRY, FCFS, TRAIL+) on two deployments: four servers each with two H100 GPUs running Llama-3 70B (Figure 10) and eight servers each with one A100 GPU running Llama-3 8B (Figure 11). No-Preempt is excluded due to its poor single-server performance.

Headline result for load balancers. Across all metrics and configurations, SAL either outperforms or matches P2C and Random when combined with the same engine-level scheduler. The paper reports: "Compared to the next-best technique and when comparing all load balancers in combination with LARRY, SAL achieves 1.0×–1.5× lower p50 TTFT, and 1.2×–1.3× lower p95 TTFT. Furthermore, SAL achieves 1.0×–1.2× and 1.0×–1.2× lower Normalized TTFT" (Section 7). The narrow range of these multipliers — particularly the lower bounds of 1.0× — indicates that SAL's advantage over P2C and Random is modest in TTFT, and in some configurations the load balancers perform equivalently. This is consistent with the paper's earlier observation that "the load balancer has a smaller effect on queueing delays than the engine-level schedulers" (Section 7).

SAL's advantage in Total Generation Time. While the TTFT differences between load balancers are small, SAL's advantage in TGT is more pronounced. The paper reports: "SAL decreases the p50 TGT by 1.1×–1.3× and the p95 TGT by 1.1×–1.3× over the next-best method (Section 7). The mechanism is indirect: SAL's token-aware routing "more evenly balancing the batch size decreases the number of overly full batches and, hence, improves the inter-token latency and TGT" (Section 7). Because Figure 2 shows that decode throughput depends on batch size — larger batches are more efficient up to the point where they become compute-bound — and because SAL distributes prefill tokens more evenly, servers are less likely to have batches that are either nearly empty (underutilized) or at the 1024-token limit (causing high inter-token latency). This is a second-order effect that general-purpose load balancers cannot replicate because they do not track token counts.

Combined best configuration. The best-performing combination in both Figures 10 and 11 is consistently SAL + LARRY. For SW-Chat on the Llama-3 70B deployment (Figure 10), SAL + LARRY achieves p50 TTFT of approximately 0.3–0.4 seconds, compared to approximately 0.4–0.5 seconds for P2C + LARRY and approximately 0.5–0.6 seconds for Random + LARRY. The SAL + FCFS and SAL + TRAIL+ combinations perform worse than any LARRY combination regardless of load balancer, confirming that the engine-level scheduler is the dominant factor. The paper summarizes: "for all combinations of scheduling techniques, we observe low preemption rates of <0.1%. In general, in combination with the same engine-level scheduler, SAL improves or matches the performance of other load balancers" (Section 7).

Serving capacity in multi-server setting. The rightmost columns of Figures 10 and 11 show TTFT scaling behavior. Consistent with the single-server results, LARRY-based combinations show the flattest degradation curves. Among load balancers, SAL and Random perform similarly on p50 TTFT for the A100 deployment (Figure 11, SW-Chat), while SAL holds a small but consistent advantage on the H100 deployment (Figure 10). At p95, SAL's advantage is more visible — its curves remain below both P2C and Random across most scaling factors. Notably, for SW-Code on the H100 deployment, all three load balancers produce nearly identical p50 TTFT scaling curves when combined with LARRY, indicating that when the engine-level scheduler effectively manages memory pressure, the load balancer's routing decisions have limited additional impact. The paper does not call this out explicitly, but it is visible in the plots.

Ablation Studies and Robustness Checks

α parameter sweep (Figure 9): The evaluation of α ∈ {1, 500, 1000} on two hardware configurations (H100 with Llama-3 70B, A100 with Llama-3 8B), both running SW-Chat, demonstrates that LARRY's performance is controlled by a single interpretable parameter with predictable effects on the latency distribution. Low α prioritizes avoiding HOL blocking (better median, worse tail for large requests); high α prioritizes fairness (worse median, better tail). At the extreme (α > 1000), LARRY degenerates to FCFS. The fact that α = 1 works well across both hardware configurations and both workloads in Figures 7 and 8 suggests the parameter is not highly sensitive to deployment specifics for median latency, but tail-sensitive applications require tuning. The paper does not evaluate intermediate α values between 1 and 500, so the shape of the tradeoff curve is not characterized.
TRAIL+ preemption threshold c (Section 6): The paper "ran TRAIL+ with several choices for c" and found that c = 0 achieves the best performance on the evaluated workloads because it "avoids frequent preemptions which is desirable for workloads with relatively short response lengths." This is a workload-dependent finding: when output lengths are short (Figure 6), the cost of preempting and later resuming a request outweighs the benefit of prioritizing a slightly shorter request. The paper does not report results for other values of c, so the sensitivity of TRAIL+ to this parameter is not quantified.
No-Preempt across workloads (Figures 7, 8): The dramatic performance difference between SW-Chat (TTFT beyond plot limits) and SW-Code (competitive with other methods) for No-Preempt serves as an implicit ablation on workload composition. SW-Chat has longer outputs (mean 211 tokens, with a long tail) while SW-Code is "dominated by prefills" with very short outputs (mean 27 tokens). No-Preempt's pre-allocation penalty is proportional to the maximum possible output length minus the actual output length, so it becomes severe when outputs are long and variable. This finding reinforces why Paged Attention — which dynamically allocates memory — has become standard.
Driver process bottleneck mitigation (Section 5): The paper's modification to vLLM to move data marshalling (tokenization) outside the driver process is a methodological robustness check rather than a standard ablation: without this modification, the driver process would become "oversubscribed at high QPS and fails to keep up with adding queries to the waiting queue," creating an artificial bottleneck that would mask the true performance differences between schedulers. The paper verifies that after the modification, "the rate at which queries are added to the waiting queue precisely reflects the rate at which queries are issued according to the workload trace." This ensures that the reported latency differences are due to scheduling policy choices, not an implementation artifact of the vLLM driver architecture.
Workload scaling factor range (Figures 7, 8, 10, 11): The scaling factor is chosen so that "the system is neither under-provisioned nor over-provisioned" at 1×, but the paper does not specify the exact QPS values corresponding to each scaling factor. The x-axis ranges vary across configurations: the Llama-3 70B/SW-Chat single-server experiment scales from 1× to 3×, while the Llama-3 8B/SW-Code single-server experiment scales from 0.5× to 7.5×. This variation is necessary because different hardware/model combinations have different saturation points, but it makes direct cross-configuration comparisons of the scaling factor axis difficult. The qualitative behavior — which method's curve rises most slowly — is what the paper emphasizes.
Preemption rate across all configurations (Sections 6, 7): The paper consistently reports preemption rates below 0.1% for every scheduler, workload, and scaling factor combination. This is a critical robustness check because it means the reported TTFT differences are driven by queuing delays (dispatch order decisions) rather than by preemption overhead (KV cache eviction and recomputation). If preemption rates were non-trivial, the comparison between methods would be confounded by each method's preemption policy. The low preemption rates across the board suggest that for these workloads and hardware configurations, the GPU memory capacity was sufficient to hold the working set of KV caches at the tested QPS levels. The paper does not evaluate scenarios where preemption is unavoidable (e.g., extreme overload with long-output requests), which would test the preemption-handling aspects of TRAIL+ and FastServe more directly.
Multi-server deployment sizes (Figures 10, 11): The paper uses 4 servers for the H100/Llama-3 70B deployment and 8 servers for the A100/Llama-3 8B deployment. The choice of server counts is not explicitly justified, but the different counts serve as an implicit robustness check: SAL's benefits hold at both scales (4 and 8 servers).
Polling frequency for SAL (Section 4.2): The paper polls servers at 10 Hz and states "polling the server for these statistics incurs negligible overhead, which allows for frequent polling." No ablation on polling frequency is performed — the paper does not evaluate whether lower polling rates (e.g., 1 Hz or 5 Hz) would degrade SAL's performance or whether higher rates would improve it. The 100 ms staleness bound is not compared against the timescale of request processing to justify whether it is sufficiently frequent.
β parameter sensitivity (Section 4.2): The β values for SW-Chat (1365/211 ≈ 6.5) and SW-Code (2074/27 ≈ 76.8) are computed once from workload statistics. The paper does not evaluate sensitivity to errors in β — for example, what happens if the workload's ratio of input to output tokens changes over time and the running average lags behind, or if β is estimated from a different workload entirely. The paper describes β as capturing "the approximate, average rate at which memory is freed up because requests finish," implying it is a coarse estimate, but the impact of estimation error on routing quality is unexplored.

Critical Assessment

Does LARRY Outperform Current Techniques?

The claim from the executive summary — that LARRY achieves 1.8×–2.1× lower p50 TTFT than the next-best method — is supported by Figures 7 and 8. However, the "next-best method" is either FCFS or TRAIL+, and TRAIL+ is given the unrealistic advantage of ground-truth response lengths. This means LARRY is being compared against an idealized version of the best prediction-based method, not against that method's actual deployable performance (which would require running a predictor with non-zero error and non-zero overhead). The comparison is therefore conservative in LARRY's favor for the TRAIL+ baseline — TRAIL+ with a real predictor would likely perform worse than the version shown in Figures 7 and 8. On the other hand, the paper does not evaluate LTR (which would be a fairer comparison since LTR is a drop-in-compatible scheduler that requires training a ranking model but does not require architectural changes), nor does it compare against FastServe (which would require implementing KV cache migration but might achieve better performance on workloads with long outputs). The set of evaluated baselines is limited to techniques that can be implemented within vLLM without significant modifications, which is consistent with the paper's drop-in replacement philosophy but means the absolute performance ceiling for engine-level scheduling is not established.

The TTFT advantage is also workload-dependent in ways the paper does not fully characterize. SW-Chat and SW-Code both have prompts substantially longer than outputs on average (Figure 6), which means prompt length is a good proxy for total memory consumption. If LARRY were evaluated on a workload with short prompts and very long outputs (e.g., story generation, long-form essay writing), the premise that "the prompt dominates a request's memory consumption" (Section 4.1) would break down, and LARRY's dispatch policy — which dispatches "a request once the available memory can hold its prefill KV cache" — might lead to memory exhaustion as output KV caches grow far beyond the prefill allocation. The paper does not discuss this limitation or evaluate on workloads with the opposite prompt/output ratio.

The claim that LARRY's p50 TTFT advantage is 1.8×–2.1× is a range. The exact multiplier depends on the configuration. For SW-Chat on Llama-3 70B, visual inspection of Figure 7 suggests the p50 TTFT for LARRY is roughly 0.25–0.3 seconds versus roughly 0.55–0.65 seconds for FCFS, which is indeed approximately 2×. For SW-Code on the same configuration, the p50 TTFT is roughly 1.2 seconds for LARRY versus roughly 2.2 seconds for FCFS, which is approximately 1.8×. The paper's reported range of 1.8×–2.1× appears to span these configurations.

Does SAL Outperform General-Purpose Load Balancers?

The claim that SAL improves p50 TGT by 1.1×–1.3× over the next-best method is supported by Figures 10 and 11, but the practical significance of this improvement should be contextualized. A 1.1× improvement means SAL delivers a 10% reduction in TGT for the median request. For SW-Chat on Llama-3 70B (Figure 10), visual inspection suggests SAL + LARRY achieves p50 TGT of roughly 17–18 seconds versus roughly 19–20 seconds for P2C + LARRY and Random + LARRY. This is a real improvement but a modest one — roughly 2 seconds saved on a 20-second generation. The paper positions SAL as providing better load distribution that reduces "overly full batches," which is the correct mechanistic explanation, but the magnitude of the effect is small relative to the engine-level scheduler's impact.

A more significant finding that is not reflected in the summary multipliers is that SAL's advantage over P2C and Random is not uniform across configurations. In the A100/SW-Chat configuration (Figure 11), SAL, P2C, and Random produce nearly identical p50 TTFT curves when combined with LARRY. In the A100/SW-Code configuration (Figure 11, serving capacity p50), P2C and Random even appear slightly below SAL at some scaling factors (though the differences are within visual noise). The paper states that SAL "improves or matches the performance of other load balancers" (Section 7), but does not address the fact that on some configurations, the improvement is negligible. This matters for practical adoption: if the benefit of implementing SAL is configuration-dependent and sometimes zero, practitioners might reasonably choose to stick with the simpler Random policy.

The β parameter in SAL is computed from workload-specific averages. The paper does not evaluate whether using incorrect β values degrades SAL's routing quality. If a deployment's workload mix changes (e.g., a chatbot that also handles document summarization, shifting the input/output token ratio), the β computed from historical averages could become stale. The paper provides no guidance on how frequently β should be recomputed or how sensitive SAL is to β errors.

LARRY and SAL as Drop-In Replacements

The paper's most distinctive claim — that LARRY and SAL are "drop-in replacements" that can be implemented in tens of lines of code — is supported by the implementation details (20 lines for LARRY, 30 lines for SAL). However, the paper's evaluation of this claim is purely qualitative: it states these line counts but does not measure the engineering effort required to integrate LARRY or SAL into a production vLLM deployment, does not test compatibility with other vLLM features (speculative decoding, prefix caching, LoRA adapters), and does not evaluate whether the 20-line LARRY implementation handles edge cases (e.g., what happens when a request's wait_time overflows the score computation, or when queue_len is zero). These are minor concerns for a research paper, but they mean the "drop-in replacement" claim is demonstrated in a controlled benchmarking environment, not in a production system with real users and diverse request patterns.

More fundamentally, the paper's modification to vLLM's driver process — moving tokenization to a separate process to avoid oversubscription — is itself a change that goes beyond the scheduler. The paper frames this as necessary to evaluate schedulers at high QPS rather than as part of LARRY, but a practitioner deploying LARRY in vLLM at high QPS would need to address the same driver process bottleneck to realize the reported latency improvements. If tokenization remains in the driver process, the oversubscription would create a back-pressure bottleneck that makes the scheduler's dispatch decisions irrelevant — requests would wait to enter the waiting queue, not within it. The paper's results should therefore be understood as demonstrating LARRY's performance in a system where the request ingestion pipeline is not the bottleneck, which may require additional engineering beyond swapping the scheduling policy.

What the Experiments Do Not Test

Several experiments that would strengthen the paper's claims are absent:

Workloads with long outputs relative to inputs. Both SW-Chat and SW-Code have prompts substantially longer than outputs. This makes prompt length a reliable proxy for memory consumption (Figure 5). LARRY's assumption that it can "dispatch a request once the available memory can hold its prefill KV cache" (Section 4.1) is validated for these workloads but would break down for story generation, long-form QA, or any application where the model generates responses much longer than the prompt. Evaluating on such a workload would stress-test whether LARRY's dispatch policy causes memory exhaustion that FCFS might avoid (because FCFS might dispatch fewer total requests, leaving more memory headroom).

Scenarios where preemption is unavoidable. The paper reports <0.1% preemption rates across all experiments. This is a property of the specific combination of workload intensity, model size, and GPU memory capacity tested. Running experiments at higher scaling factors — deliberately pushing the system into memory pressure where preemption becomes necessary — would evaluate how LARRY's non-preemptive dispatch policy compares to TRAIL's preemptive policy. The paper's finding that c = 0 (no preemption beyond the initial phase) is optimal for TRAIL+ on these workloads is interesting but workload-specific; it does not generalize to scenarios where preemption is genuinely necessary.

LTR and FastServe as baselines. The paper discusses both LTR and FastServe in detail (Section 2.2) but evaluates neither. LTR is explicitly compatible with vLLM's architecture (it only requires ranking, not activation access), and evaluating it would provide a direct comparison against another drop-in-compatible method that uses a different heuristic (ranking by predicted output length rather than scoring by prompt length and wait time). FastServe would require implementing KV cache migration, which violates the drop-in replacement criterion, but evaluating it would establish an upper bound on what architectural changes can achieve. The absence of these baselines means the paper compares LARRY against the simplest practical policies (FCFS, No-Preempt) and an idealized version of TRAIL, but not against the strongest methods in the literature.

Sensitivity to the maximum batch token limit. The paper uses a maximum of 1024 tokens per batch in all experiments. This parameter directly affects how many requests can be batched together and therefore how much HOL blocking occurs. A smaller limit (e.g., 512 tokens) would reduce the severity of HOL blocking because large requests would not block the batch for as long, potentially reducing LARRY's advantage over FCFS. A larger limit (e.g., 2048 tokens) would increase HOL blocking and potentially amplify LARRY's advantage. The paper does not ablate this parameter.

Statistical variance. All results are presented as single-run latency percentiles. Systems benchmarking at this scale typically exhibits some run-to-run variance due to hardware (GPU clock speeds, memory bandwidth contention, CPU scheduling) and software (garbage collection, driver process scheduling). Without error bars, confidence intervals, or multiple-run reporting, it is impossible to assess whether the 1.1×–2.1× improvements are statistically reliable or within the noise floor of the measurement infrastructure.

Where Claims Hold Conditionally

The paper's central claims should be understood with the following conditions, which are partially acknowledged in the limitations section but not always emphasized in the abstract or introduction:

LARRY's TTFT advantage assumes prompt length correlates with memory consumption and that output lengths are not extremely long. This holds for SW-Chat and SW-Code (Figure 5) but the paper provides no evidence it holds for other application classes.
LARRY's serving capacity advantage assumes the system has enough GPU memory that preemptions remain rare. The <0.1% preemption rates mean the experiments operate in a regime where memory pressure exists (the queue builds up) but does not force preemptions. At higher load where preemptions become frequent, LARRY's non-preemptive design might underperform preemptive policies.
SAL's TGT advantage is modest (1.1×–1.3×) and configuration-dependent. On some configurations (A100/SW-Chat in Figure 11), SAL's TTFT advantage over P2C and Random is nearly zero. Practitioners should not expect SAL to transform load balancing — it is an incremental improvement over general-purpose policies for scenarios where token load is unevenly distributed.
Both LARRY and SAL are evaluated on Llama-3 models (8B and 70B) with vLLM. The paper does not test on other model architectures (e.g., mixture-of-experts models where the KV cache size per token differs), other serving engines (e.g., TensorRT-LLM, SGLang), or other hardware (e.g., H200 with larger memory, multi-node inference with model parallelism across servers). The techniques' generality across these dimensions is assumed but untested.
The drop-in replacement claim is validated in a modified vLLM where the driver process bottleneck has been mitigated. In unmodified vLLM at high QPS, the driver process oversubscription would mask scheduler differences. The reported latency improvements are achievable only if the ingestion pipeline can keep up with the arrival rate, which may require the kind of architectural change (moving tokenization out of the driver process) that the paper performs but does not count as part of LARRY's implementation cost.

These conditions do not invalidate the paper's contributions — LARRY and SAL represent real improvements over the baselines for the tested configurations — but they bound the generality of the claims in ways that practitioners should understand before adopting these techniques in deployments with different workload characteristics, hardware, or serving infrastructure.

6. Limitations and Trade-offs

6.1 LARRY's Core Assumption Breaks Down When Output Lengths Exceed Prompt Lengths

The assumption or constraint. LARRY estimates a request's memory consumption solely from its prompt length and dispatches a request "once the available memory can hold its prefill KV cache" (Section 4.1). This design rests on the observation that in the evaluated workloads, "the prompt dominates a request's memory consumption" and "the KV cache only grows slowly after the prefill phase" (Section 4.1). Figure 5 quantifies this: over 70% of tokens in most KV caches are prompt tokens for both SW-Chat and SW-Code. The paper acknowledges this assumption implicitly by noting that "in many applications... the prompt dominates a request's memory consumption" (Section 4.1, emphasis added), but does not characterize what happens when this condition does not hold.

The consequence. For workloads where output lengths are comparable to or exceed prompt lengths — story generation, long-form essay writing, detailed code explanation, or multi-turn dialogue where the model generates extended responses — LARRY's dispatch policy would systematically underestimate a request's peak memory consumption. Because LARRY only checks that the prefill KV cache fits in available memory before dispatching, a request with a short prompt but a very long response could initially be dispatched but later exhaust GPU memory as its output KV cache grows far beyond the prefill allocation. This would force preemptions that LARRY is not designed to handle proactively. In contrast, FCFS with a conservative concurrency limit or No-Preempt (which pre-allocates maximum memory) would avoid this failure mode by limiting the number of concurrent requests or reserving worst-case memory up front. The paper's finding of <0.1% preemption rates across all experiments (Sections 6 and 7) is a property of the tested workloads, not a guarantee of LARRY's dispatch logic — it reflects that prompt-dominated workloads generate modest output KV cache growth relative to the prefill allocation.

What evidence exists in the paper. Figure 6 shows the input and output token distributions for SW-Chat (mean input ~1365 tokens, mean output ~211 tokens) and SW-Code (mean input ~2074 tokens, mean output ~27 tokens). Both workloads have outputs substantially shorter than inputs. The paper provides no experiments on workloads with the opposite ratio. Figure 5, which demonstrates that prompt tokens dominate KV cache memory, is derived from these same workloads. The paper does not cite or evaluate any workload trace where output lengths consistently exceed input lengths, despite such workloads being common in production (e.g., creative writing assistants, summarization in reverse — expanding short notes into long-form content).

Mitigation status. The paper does not address this limitation. Section 4.1 states the assumption ("the prompt dominates a request's memory consumption in many applications") but provides no guidance on how to adapt LARRY for applications where this is false. A practitioner deploying LARRY on a workload with long outputs would need to either (a) modify the memory estimation to account for expected output length (which would require a predictor — precisely what LARRY was designed to avoid), (b) increase the concurrency limit conservatively to leave memory headroom for output KV cache growth (defeating LARRY's throughput benefits), or (c) accept higher preemption rates and hope that KV cache recomputation overhead does not dominate latency. None of these options is evaluated.

6.2 Difficulty Estimation Cost Is Not Accounted For in the Difficulty Bins

This section heading and content appear to be a copy-paste error from the reference example. The Kossmann et al. paper does not use difficulty estimation, difficulty bins, or oracle/predicted difficulty. This limitation should be replaced with one that is actually present in the paper. The correct limitation is:

6.2 Evaluation Is Restricted to Two Workload Traces, One Model Family, and One Serving Engine

The assumption or constraint. All experiments use two production workload traces from Azure OpenAI (SW-Chat and SW-Code), two sizes of Llama-3 (8B and 70B), and a single serving engine (vLLM, albeit with modifications to the driver process). The paper states that the arrival pattern trace "has been noted to be representative of model serving loads in several works" (Section 5, citing three prior papers), but this representativeness claim applies to the QPS arrival pattern, not to the token length distributions. The token length distributions (Figure 6) are specific to a chatbot and a code copilot. The paper does not claim that Llama-3 is representative of all model architectures, though the executive summary frames the techniques as "drop-in replacements" for "many current serving systems" without restricting to specific model families.

The consequence. Several aspects of the results could change substantially with different workloads or models. First, workloads where arrival patterns are burstier than the Azure Functions trace (e.g., batch submissions of evaluation jobs, flash-crowd events from viral features) would create sharper memory pressure spikes. LARRY's queue-length-adaptive discrimination might help in these scenarios (its design anticipates varying load), but the paper provides no evidence. Second, workloads with different correlations between input and output length would affect both LARRY (as discussed in Section 6.1) and TRAIL+ (which benefits from the positive but weak input-output length correlation that causes it to "approximately prioritize requests with smaller prefills," Section 6). Third, different model architectures — particularly mixture-of-experts models where the KV cache size per token differs from dense transformers, or models with different memory-to-compute ratios — would shift the balance between memory-bound and compute-bound operation, potentially changing the relative importance of HOL blocking avoidance (LARRY's strength) versus batch efficiency optimization (SAL's strength). Fourth, other serving engines (TensorRT-LLM, SGLang) implement different preemption mechanisms, different maximum batch token limits, and different memory allocation strategies. The paper implements all policies inside vLLM to control for engine differences, but this means the results are conditional on vLLM's specific engineering choices.

What evidence exists in the paper. The paper evaluates two workload traces and two model sizes (Section 5), which is more than a single-configuration study but far from establishing generality. The paper acknowledges that TRAIL+ is "designed for workloads with short input lengths and long output lengths, which significantly differs from the application traces used in our experiments" (Section 6), but does not extend this self-awareness to LARRY or SAL — the implication is that LARRY and SAL are more robust to workload variation, but this is an untested hypothesis. The paper does not evaluate on any of the other publicly available LLM workload traces (e.g., BurstGPT, LMSys-Chat-1M, ShareGPT). The two model sizes (8B and 70B) span an order of magnitude in parameters but are architecturally identical (both Llama-3 dense transformers), so the evaluation does not test sensitivity to architectural differences.

Mitigation status. The paper does not address this limitation beyond the scope description in Section 9 ("This work studies the online scheduling problem... We leave the evaluation of offline scenarios for future work"). The title claims the techniques are "Practical Scheduling Techniques for LLMs" without qualification, and the abstract states they "outperform current techniques on production workload traces" — both statements are true for the evaluated traces but implicitly claim generality that the experiments do not establish. A practitioner considering LARRY or SAL for a deployment with different workload characteristics (different application domain, different model architecture, different serving engine) cannot predict from the paper's evidence whether the techniques will outperform FCFS or whether they might underperform in workload-specific ways.

6.3 The Driver Process Modification Required for High-QPS Evaluation Is Not Counted as Implementation Cost

The assumption or constraint. The paper modifies vLLM's driver process to move data marshalling overheads (specifically tokenization) out of the critical path so that "the rate at which queries are added to the waiting queue precisely reflects the rate at which queries are issued according to the workload trace" (Section 5). Without this modification, "the driver process becomes oversubscribed at high QPS and fails to keep up with adding queries to the waiting queue," creating back pressure where "queries are not queued in the waiting queue but are instead waiting to be added to the waiting queue, leaving the scheduler unaware of their existence" (Section 5). The paper presents this as an evaluation necessity rather than a feature of LARRY or SAL, and does not count this change toward the implementation complexity of either technique.

The consequence. The latency improvements reported for LARRY and SAL — particularly the 1.8–2.1× p50 TTFT advantage and the superior serving capacity scaling — are measured in a system where the request ingestion pipeline is not the bottleneck. In an unmodified vLLM deployment at high QPS, the driver process oversubscription would dominate queuing delays. Requests would accumulate before entering the scheduler's waiting queue, making the scheduler's dispatch policy irrelevant for that portion of the total queuing delay. A practitioner who deploys LARRY as a drop-in replacement (20 lines of code in the scheduler) without also mitigating the driver process bottleneck would not see the full latency improvements reported in Figures 7–11, because a fraction of the queuing delay would occur outside the scheduler's control. The paper does not quantify what fraction of total queuing delay occurs before requests enter the waiting queue in unmodified vLLM, so the practitioner cannot estimate how much of LARRY's reported benefit would materialize in their deployment.

Furthermore, the paper's modification — tokenization in a separate process — is itself a non-trivial engineering change. It requires restructuring the request processing pipeline to decouple tokenization from the driver process and ensuring thread-safe communication between the tokenization workers and the scheduler. This change is not part of LARRY or SAL (the paper does not count it toward the 20-line or 30-line implementation), but it is necessary to realize the reported performance at high QPS. This complicates the "drop-in replacement" narrative: LARRY is drop-in only if the underlying engine can already ingest requests at the arrival rate without back pressure. If it cannot, additional engineering is required.

What evidence exists in the paper. Section 5 describes the modification and the verification that request addition rates match the workload trace. The paper does not report performance numbers for unmodified vLLM at high QPS, so the magnitude of the driver process bottleneck is not quantified. The paper does not discuss whether other serving engines (TensorRT-LLM, SGLang) have similar bottlenecks or whether they can handle high QPS without comparable modifications.

Mitigation status. The paper acknowledges the modification but frames it as an evaluation methodology issue rather than a limitation of LARRY or SAL. The authors state that if a serving system "incurs significant overheads when adding requests to the waiting queue, our general findings about the schedulers still hold true, but their performance numbers might differ from the ones reported" (Section 9). This is a partial acknowledgment: it recognizes that absolute latency numbers depend on the ingestion pipeline but asserts that relative comparisons between schedulers are preserved. This assertion is plausible but unverified — if the driver process bottleneck introduces a fixed per-request overhead, it would compress the relative differences between schedulers (since all schedulers incur the same overhead), but if the bottleneck interacts with batch processing or queue management in scheduler-specific ways, the relative ordering could change. The paper does not evaluate this.

6.4 LARRY's Non-Preemptive Design Is Untested in Regimes Where Preemptions Are Unavoidable

The assumption or constraint. LARRY is a non-preemptive scheduler: it only makes dispatching decisions when capacity becomes available through natural request completions. It does not preempt running requests to make room for higher-priority waiting requests. The paper reports that "for all techniques, we saw low preemption rates of less than 0.1%" (Section 6) and that for multi-server deployments, "for all combinations of scheduling techniques, we observe low preemption rates of <0.1%" (Section 7). This means the entire evaluation operates in a regime where preemption is rare — GPU memory is sufficient to hold the working set of KV caches at the tested QPS levels and concurrency limits.

The consequence. The paper provides no evidence about how LARRY would perform when memory pressure forces preemptions. This could occur at higher QPS than tested, with larger models relative to GPU memory, or with workloads that have longer outputs. In a preemption-heavy regime, LARRY's dispatch-first-then-hope design would be directly tested: it dispatches requests based on prompt length, but if those requests go on to generate very long outputs, the system would run out of memory and be forced to preempt. LARRY provides no guidance on which requests to preempt when this happens — the scoring function in Equation 1 only applies to waiting requests, not running ones. The system would fall back on vLLM's default preemption policy (or whatever the serving engine implements), which LARRY does not influence. In contrast, TRAIL+ explicitly incorporates preemption into its scheduling logic (with the c parameter controlling how late in a request's lifetime preemption is allowed) and FastServe is designed around frequent, cheap preemption through proactive KV cache migration.

The paper's finding that c = 0 (essentially disabling preemption beyond the very beginning of a request's lifetime) is optimal for TRAIL+ on the tested workloads is informative but potentially misleading. It suggests that preemption is not useful for these workloads, but this conclusion is conditional on the workload characteristics and hardware configuration. A practitioner running a deployment where preemptions are unavoidable (e.g., serving a large model on GPUs with limited memory under high load) cannot conclude from this paper whether LARRY or TRAIL+ would be preferable, because the paper never evaluates that regime.

What evidence exists in the paper. The <0.1% preemption rate is reported consistently but is a property of the tested configurations, not a finding about LARRY. The paper does not include experiments that deliberately induce preemptions (e.g., by reducing available GPU memory, increasing maximum output length, or scaling QPS beyond the tested range). The serving capacity experiments (rightmost columns of Figures 7–8 and 10–11) extend to scaling factors where TTFT degrades substantially, but preemption rates remain below 0.1% even at the highest scaling factors — indicating that the bottleneck is queuing delay and batch formation, not memory exhaustion. The paper does not test at scaling factors high enough to cross the preemption threshold.

Mitigation status. The paper does not discuss this limitation. Section 9 mentions only that "this work studies the online scheduling problem" versus offline throughput-oriented applications, and does not discuss preemption regimes. The absence of preemption testing is a consequence of the experimental design (which focuses on workloads and QPS levels representative of the Azure Functions trace) rather than an explicit scope limitation. A practitioner cannot determine from the paper whether LARRY's non-preemptive design is a strength (avoiding preemption overhead) or a weakness (unable to recover from memory exhaustion gracefully), because the experiments never stress this aspect of the design.

6.5 SAL's Benefit Is Modest and Configuration-Dependent, While Its Parameter Sensitivity Is Unexplored

The assumption or constraint. SAL estimates server load using Equation 2, which depends on two workload-specific parameters: β, the ratio of average total tokens to average output tokens, and the polling frequency (10 Hz in the implementation). The paper states that β "can easily be recorded during online serving" (Section 4.2) and that polling "incurs negligible overhead" (Section 4.2). SAL's routing decisions also depend on the assumption that no requests finish between polling intervals — the load balancer's state estimate becomes increasingly stale as the polling interval increases.

The consequence. SAL's performance advantage over general-purpose load balancers is modest: 1.0×–1.5× better p50 TTFT and 1.1×–1.3× better TGT (Section 7). On several configurations, the advantage is at or near the lower bound of these ranges. For the A100/SW-Chat deployment (Figure 11), SAL, P2C, and Random produce nearly identical p50 TTFT curves when combined with LARRY, and SAL's advantage at p95 is small. This means the practical value of implementing SAL — a 30-line change plus the polling infrastructure — is configuration-dependent and sometimes near zero. A practitioner might reasonably choose to stick with the simpler Random policy, which requires no state tracking, no polling, and no β estimation, and achieve comparable performance in many scenarios.

The unexplored parameter sensitivity compounds this concern. The paper does not evaluate SAL with incorrect β values — for example, using a β estimated from SW-Chat (6.5) on an SW-Code workload (where true β is 76.8), or using a stale β when the workload mix changes over time. If SAL's routing quality is sensitive to β errors, then the practical deployment story becomes more complex: the practitioner must monitor workload characteristics and update β, which may require infrastructure beyond the 30 lines of load balancer code. The paper also does not ablate the polling frequency — 10 Hz might be insufficient for bursty workloads where server state changes rapidly between polls, or might be unnecessarily frequent for stable workloads where 1 Hz would suffice with negligible performance impact.

What evidence exists in the paper. Figures 10 and 11 show SAL's performance relative to P2C and Random across all combinations of engine-level schedulers, workloads, and hardware configurations. The paper states that SAL "improves or matches the performance of other load balancers" (Section 7), which is accurate — SAL is never worse — but does not quantify how often SAL merely "matches" versus meaningfully improving. Visual inspection of Figure 11 suggests that for SW-Chat on A100, SAL + LARRY and P2C + LARRY are nearly indistinguishable across all metrics. The paper does not perform any sensitivity analysis on β, polling frequency, or the staleness assumption.

Mitigation status. The paper does not address SAL's parameter sensitivity or the conditions under which SAL provides negligible benefit. The fact that SAL "matches" other load balancers in some configurations is presented as a feature (it never does worse) rather than as a limitation (its benefit is not guaranteed). Section 9 does not mention SAL-specific limitations. A practitioner evaluating whether to adopt SAL needs to understand that the 1.1×–1.3× TGT improvement is the best case across configurations, and that on some hardware/workload combinations the improvement may be indistinguishable from Random assignment.

6.6 No Statistical Characterization of Variance or Measurement Noise

The assumption or constraint. All results in Figures 7–11 are presented as single-run latency percentiles without error bars, confidence intervals, or multiple-run reporting. The workloads are deterministic replays of recorded production timestamps and token lengths, so the arrival pattern and request characteristics do not vary between runs. However, the serving system itself introduces variance: GPU clock frequency variation, memory bandwidth contention between the LLM forward pass and KV cache management, CPU scheduling jitter for the driver process and tokenization workers, and operating system-level noise. The paper does not report whether experiments were repeated, whether the reported numbers are averages of multiple runs or single measurements, or what the typical run-to-run variance is.

The consequence. The reported latency improvements — particularly the smaller ones like SAL's 1.1× TGT advantage — may fall within the measurement noise floor. Without any characterization of variance, a practitioner cannot assess whether the observed differences between schedulers are reliable or would invert on a different run with the same configuration. This is especially relevant for the serving capacity plots, where the scaling factor axis is densely sampled (many data points along the x-axis) and the differences between methods at individual scaling factors are often small. A systematic +5% latency advantage that is within ±3% run-to-run variance would not be a reliable basis for choosing one scheduler over another.

The absence of statistical characterization is standard for systems benchmarking papers where the primary sources of variance are controlled (fixed hardware, deterministic workload replay), but it weakens the strength of claims based on small numerical differences. The paper's headline numbers — LARRY's 1.8–2.1× p50 TTFT advantage — are large enough that run-to-run variance is unlikely to change the qualitative conclusion. But the more nuanced findings — SAL outperforms P2C except when it doesn't, TRAIL+ occasionally underperforms FCFS at tail percentiles — could be sensitive to measurement noise in ways the reader cannot assess.

What evidence exists in the paper. The paper provides no information about experimental repetition. Section 5 describes the methodology (workload emulation, implementation details, evaluation metrics) but does not state how many runs were performed or how the reported percentiles were computed (e.g., whether they pool all requests from a single run or aggregate across multiple runs). The experimental setup section mentions only that the authors "verified for each experiment, that the rate at which queries are added to the waiting queue precisely reflects the rate at which queries are issued" (Section 5), which addresses implementation correctness but not measurement reliability.

Mitigation status. The paper does not acknowledge this limitation. It is possible that the authors performed multiple runs and observed consistent results, but this is not reported. For a paper whose primary contribution is empirical comparison of scheduling policies, the absence of any variance reporting is a methodological weakness. Future work replicating these results on different hardware or with different workload traces would help establish the reliability of the reported effect sizes, particularly for the smaller-magnitude findings.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new paradigm for LLM serving — the two-tier architecture of load balancers and engine-level schedulers was already standard (Figure 1), and both Paged Attention and Continuous Batching were already widely deployed. Instead, it makes a diagnostic contribution with practical consequences: it systematically characterizes why the gap between research schedulers and production schedulers exists (architectural coupling, training requirements, implementation complexity), and then demonstrates that this gap is bridgeable with techniques that are both simpler and more effective than the field had assumed.

The specific conceptual shift is this: the paper reframes LLM scheduling from a prediction problem to an awareness problem. Prior work — TRAIL, LTR, FastServe, PiA, S3 — implicitly assumed that good scheduling requires predicting something unknown about the future: remaining response length, total memory consumption, or optimal batch composition. This assumption led to architectures that couple the scheduler to the model's internal state (TRAIL's layer activation access), to separate trained models (LTR's ranking model), or to memory management subsystems (FastServe's KV cache migration, S3's supervisor). The paper's central empirical finding — that LARRY matches or outperforms TRAIL+ even when TRAIL+ is given perfect, zero-cost predictions of ground-truth response lengths (Figures 7, 8) — demonstrates that accurate prediction of future request characteristics is not the bottleneck for the workloads and hardware configurations tested. What matters is discriminating between requests based on what is already known (prompt length, queue depth, available memory, queued token counts) and adapting that discrimination to current system load.

This reframing has direct consequences for research prioritization. If prediction sophistication is not the binding constraint, then the field's investment in better response-length predictors, ranking models, and multi-level feedback approximations is attacking a problem whose ideal solution does not beat a simple heuristic. Research effort would be better directed toward: (1) characterizing the structural properties of LLM workloads that make certain heuristics effective (e.g., the prompt-length-dominates-memory property that Figure 5 documents), (2) developing new heuristics that exploit other readily available metrics (e.g., the ratio of prefill to decode tokens in the batch, the age distribution of running requests), and (3) understanding the workload and hardware conditions under which prediction does become necessary (e.g., when output lengths consistently exceed prompt lengths, when preemptions are unavoidable, or when GPUs have very limited memory relative to model size).

The paper also resolves a contradiction in the applied systems literature about load balancing granularity. General-purpose orchestration platforms (Kubernetes with Istio, Envoy, KNative) treat load balancing as connection-count or request-count management — the implicit assumption is that all requests impose equivalent load. Specialized serving systems like Llumnix and SGLang treat load balancing as a token-level or KV-cache-level problem requiring cross-server migration. The paper's SAL demonstrates a middle ground: token-aware routing using only polled server statistics and lightweight per-request load estimation (Equation 2) can achieve most of the benefit of token-level awareness (1.1–1.3× TGT improvement) without requiring KV cache migration or inter-server coordination beyond periodic polling. This middle ground was not obvious before this paper because no prior work had systematically compared general-purpose, token-aware, and migration-based load balancers inside the same serving system on the same workloads.

The paper's most methodological contribution is the operationalization of "drop-in replacement" as an evaluative criterion (Table 1). This is not a technical innovation but a conceptual tool that will influence how future scheduling papers position and evaluate their contributions. By identifying the specific architectural coupling points that prevent prior methods from being drop-in replacements — TRAIL's layer activation access, FastServe's KV cache migration, Llumnix's cross-server state transfer — the paper provides a checklist that future systems papers can use to argue either (a) that their new technique is compatible with existing serving infrastructure, or (b) that their required architectural change is justified by performance gains beyond what any drop-in-compatible method can achieve. This shifts the burden of proof: a paper proposing a scheduler that modifies the memory management subsystem must now demonstrate that no scheduler operating within the existing memory management API can achieve comparable results, which is a higher bar than the prior literature had to clear.

What becomes less attractive: Research on LLM-specific response-length prediction for scheduling purposes now faces a stronger burden of justification. TRAIL and LTR showed that prediction can improve scheduling, but this paper shows that the improvement ceiling — even with perfect predictions — is not obviously higher than what a request-aware heuristic achieves. Future prediction-based schedulers would need to demonstrate that their predictor provides benefits beyond what LARRY achieves, either by evaluating on workloads where LARRY's assumptions break down (very long outputs relative to prompts) or by showing that prediction enables capabilities (like proactive preemption or batching optimization) that heuristics cannot replicate. Similarly, architectures that require KV cache migration for load balancing (Llumnix) now face a clearer characterization of the performance ceiling they are trying to beat: SAL achieves modest but real improvements over general-purpose policies (1.1–1.3× TGT), and migration-based approaches must demonstrate improvements beyond this to justify their additional complexity.

Follow-Up Research This Work Enables

Characterize LARRY's failure modes on workloads where prompt length does not dominate memory consumption. LARRY is designed for and evaluated on workloads where the prompt dominates KV cache memory (Figure 5: over 70% of tokens in most KV caches are prompt tokens). A direct stress test would replicate the single-server experiments (Figures 7, 8) using a workload trace with the opposite characteristic: long outputs and short prompts. Story generation, long-form essay writing, or detailed code explanation from short specifications all fit this profile. The key measurement would be: at what output-to-input token ratio does LARRY's dispatch policy begin to cause preemptions that degrade performance below FCFS? If LARRY maintains its advantage at moderate output lengths (e.g., 2:1 output-to-input ratio) but degrades at extreme ratios, that establishes a practical boundary condition for deployment. If LARRY degrades at any ratio above 1:1, it would mean the technique is genuinely limited to prompt-dominated applications and a different heuristic (perhaps based on the product of prompt and historical average output length) would be needed for other workloads. Publicly available traces like LMSys-Chat-1M or BurstGPT could be filtered to construct workload subsets with varying input/output length ratios.

Evaluate LARRY and SAL on mixture-of-experts architectures where the relationship between token count and memory consumption differs. The paper evaluates only dense Llama-3 models (8B and 70B, both standard transformer architectures). In mixture-of-experts (MoE) models like Mixtral or DeepSeek-V2, the KV cache size per token is identical to dense models (the attention mechanism is the same), but the relationship between forward-pass arithmetic intensity and batch size differs because only a subset of experts is activated per token. This shifts the roofline analysis in Figure 3: the prefill phase may be less compute-bound (fewer parameters active per token) and the decode phase may be more memory-bound (expert weights must be loaded). These shifts affect how batching decisions influence throughput and therefore how LARRY's dispatch order impacts overall system efficiency. A replication of the single-server experiments (Figures 7, 8) with Mixtral 8×7B on the same hardware and workload traces would test whether LARRY's advantage generalizes across architectures or whether MoE-specific characteristics (e.g., expert load balancing interacting with request dispatch order) require architecture-aware scheduling modifications.

Quantify the sensitivity of SAL's routing quality to stale or incorrect β estimates. SAL's Equation 2 depends on β, the ratio of average total tokens to average output tokens, which estimates the rate at which memory is freed. The paper uses workload-specific β values (6.5 for SW-Chat, 76.8 for SW-Code) but does not evaluate sensitivity to β errors. A targeted experiment would sweep β from 0.1× to 10× of the true workload value and measure TGT and TTFT for SAL across the multi-server configurations (Figures 10, 11). The key question is: how wide is the β range within which SAL maintains its advantage over Random and P2C? If SAL's advantage persists across an order-of-magnitude β range, then the parameter is not a practical concern — practitioners can use a rough estimate without performance penalty. If SAL degrades rapidly outside a narrow band around the true β, then the technique requires ongoing workload monitoring and β updates, which adds operational complexity. This experiment would also reveal whether SAL can be safely deployed with a default β value (e.g., β = 10) that works adequately across diverse workloads, or whether per-application tuning is necessary.

Develop and evaluate LARRY variants that use additional request-level features beyond prompt length. LARRY uses only prompt length and wait time as inputs to its scoring function (Equation 1). But modern serving engines have access to additional request metadata that might improve dispatch decisions: whether the request has been preempted before (indicating it has already incurred recomputation cost), the historical average output-to-input ratio for requests from the same user or application, or the request's priority tier if the serving system supports quality-of-service classes. A follow-up could systematically evaluate which additional features improve scheduling performance when incorporated into LARRY's scoring framework. The architecture would extend Equation 1 to score(r) = α · wait_time(r) - queue_len · f(r) where f(r) is a feature vector incorporating multiple request properties. The evaluation would measure diminishing returns: does adding request age (time since preemption) improve tail latency? Does adding historical output length statistics close the gap on workloads where prompt length alone is insufficient? The key constraint — consistent with the paper's drop-in replacement philosophy — is that any added feature must be available within the scheduler without requiring model forward-pass access or separate prediction models. Features like "has been preempted" and "user-specified priority" satisfy this; features like "estimated remaining output tokens" do not.

Add preemption logic to LARRY and evaluate in memory-constrained regimes. The paper's experiments operate with <0.1% preemption rates across all configurations — the system never runs out of memory. A natural extension is to add a preemption policy to LARRY (extending the scoring function to running requests) and evaluate at QPS scaling factors high enough to force preemptions. The question is whether LARRY's scoring framework — which prioritizes requests by expected memory consumption and wait time — can be extended to preemption decisions. One approach: when a preemption is necessary, score all running requests using a modified version of Equation 1 where wait_time(r) is replaced by -progress(r) (how much of the request's estimated total processing has been completed) and queue_len is the number of waiting requests. This would preferentially preempt requests that have made little progress and have large memory footprints. The evaluation would replicate the serving capacity experiments (Figures 7, 8) at scaling factors high enough that preemption rates reach 1%, 5%, and 10%, and compare LARRY-with-preemption against TRAIL+ (which already implements preemption) and against unmodified LARRY falling back on vLLM's default preemption policy. This would establish whether LARRY's framework generalizes to preemptive scheduling or whether preemption requires fundamentally different logic (as FastServe and TRAIL assume).

Replicate the full evaluation on a non-vLLM serving engine to test portability of the drop-in replacement claim. The paper implements all techniques in vLLM and validates that LARRY and SAL require 20 and 30 lines of code respectively. But the drop-in replacement claim implies portability to "many current serving systems" (Table 1). A strong replication would implement LARRY and SAL in TensorRT-LLM or SGLang and replicate a subset of the single-server and multi-server experiments (Figures 7, 10) on at least one hardware configuration and workload. The key measurement is not just whether LARRY and SAL improve performance relative to each engine's default scheduler (FCFS for TensorRT-LLM, RadixAttention-based scheduling for SGLang), but whether the implementation complexity remains in the 20–30 line range and whether any engine-specific behaviors (different preemption mechanisms, different maximum batch token limits, different memory allocation granularity) interact with the scheduling policy to change the magnitude or direction of the performance effect. If LARRY's advantage is consistent across engines, it strengthens the drop-in replacement claim substantially. If it varies — e.g., TensorRT-LLM's different concurrency management makes LARRY's queue-length-based discrimination less impactful — it would establish engine-specific boundary conditions that practitioners need to know.

Practical Applications and Downstream Use Cases

On-premises and edge deployments of LLMs where GPU memory is constrained and workload is bursty. The paper evaluates on H100 and A100 GPUs, but the core scheduling tension — multiplexing limited GPU memory between concurrent requests — is more acute on consumer-grade GPUs (RTX 3090, RTX 4090 with 24GB) or edge devices (Jetson Orin) where memory headroom is tighter relative to model size. LARRY's design directly addresses this scenario: by prioritizing small-prefill requests when the queue is long, it maximizes throughput (dispatching more requests that fit in available memory) while preventing starvation through the α-weighted wait time term. The paper's serving capacity results (Figures 7, 8) show LARRY is least sensitive to workload scaling, meaning it maintains lower latency as QPS increases. For a deployment on a single RTX 4090 serving Llama-3 8B to a small team, the benefit is that the GPU can handle higher burst loads (e.g., multiple team members querying simultaneously after a meeting) without ballooning TTFT or requiring request queuing at the application level. The drop-in implementation cost (20 lines of vLLM code, no model retraining, no additional infrastructure) means this improvement is accessible to teams without specialized systems expertise. Based on Figure 8 (A100 with Llama-3 8B), a 2× QPS scaling factor with LARRY produces approximately the same p50 TTFT as 1× scaling with FCFS — in practical terms, the deployment can absorb twice the user load at equivalent latency.

Multi-tenant LLM serving platforms where different users or applications submit requests with systematically different prompt lengths. Cloud LLM APIs (Azure OpenAI, Anthropic, Together AI) and internal company platforms serve diverse workloads simultaneously: some users send short chat messages (tens of tokens), others upload documents for summarization (thousands of tokens). FCFS scheduling means a single document summarization request can block many short chat requests behind it (HOL blocking). LARRY's load-adaptive discrimination automatically prioritizes the short chat requests during high-load periods while ensuring the summarization requests eventually execute (through the α · wait_time term). SAL's token-aware routing prevents the pathological case where a load balancer's random or round-robin assignment concentrates several summarization requests on one server while another server idles. The paper's multi-server results (Figures 10, 11) show that SAL + LARRY achieves the lowest TTFT across all tested configurations. For a platform serving millions of requests daily, the improvement from 1.8× lower p50 TTFT (LARRY's single-server advantage on Llama-3 70B SW-Chat, Figure 7) translates to faster chat response times for the median user and reduced infrastructure cost (the same hardware serves higher QPS at equivalent latency, as shown in the serving capacity results). The fact that neither technique requires per-user or per-application configuration — LARRY's queue length adapts automatically and SAL's β can be estimated from aggregate workload statistics — makes them suitable for multi-tenant environments where manual per-tenant tuning is infeasible.

Batch inference pipelines for offline evaluation or synthetic data generation where throughput matters but latency does not. While the paper explicitly focuses on online scheduling (Section 9), the findings have implications for offline batch processing. In batch settings, the system typically receives all requests at once and must process them as quickly as possible. The dominant source of inefficiency is not queuing delay (there is no notion of "waiting for arrival") but rather poor batch composition — combining requests with very different prompt lengths or output lengths in the same batch causes stragglers that stall the entire batch even under Continuous Batching. LARRY's dispatch logic (prioritizing similar-sized requests together when the queue is long) suggests a batch-composition heuristic: sort requests by prompt length before dispatching, creating batches where requests have similar memory footprints and processing times. This is not what LARRY does at test time (it dispatches sequentially rather than batch-composing globally), but the underlying principle — that request-size homogeneity improves efficiency — could inform batch scheduling policies for throughput-oriented workloads. A concrete implementation would use LARRY's scoring function to sort the entire offline workload before processing begins, then dispatch in sorted order. The paper does not evaluate this, but the mechanism is implied by the observation that LARRY achieves low preemption rates and stable TGT (Figures 7, 8) — properties that matter for batch throughput as well as online latency.

When to Prefer This Method

The paper articulates a clear tradeoff between LARRY/SAL and the alternatives it evaluates, though the tradeoff is framed more as a design philosophy (drop-in simplicity vs. architectural complexity) than as a conditional decision rule. The following decision points are grounded in the paper's explicit claims and experimental evidence:

Prefer LARRY over FCFS as the engine-level scheduler when: (1) the serving engine is vLLM, TensorRT-LLM, SGLang, or any engine where the scheduler can access prompt length and queue depth without modifying the model forward pass; (2) the workload exhibits variable prompt lengths (Figure 6) and fluctuating QPS (Figure 4), since these are the conditions under which HOL blocking degrades FCFS and LARRY's load-adaptive discrimination provides benefit; (3) the application is latency-sensitive at median percentiles (p50, p75) rather than exclusively at the tail, since LARRY with default α = 1 prioritizes median improvement (1.8–2.1× better p50 TTFT) over tail improvement. For tail-sensitive applications, LARRY with higher α (e.g., α = 500) still outperforms FCFS but the margin is smaller (Figures 7, 8, 9).
Prefer SAL over general-purpose load balancers (Random, Round-Robin, P2C) when: (1) the deployment has multiple server instances (≥2) and the load balancer has access to per-server token queue depth and free memory statistics (exposed by standard serving engines); (2) the workload has substantial variance in prompt lengths across requests (Figure 6), making token-unaware routing likely to create uneven batch compositions; (3) Total Generation Time is a meaningful metric (i.e., users wait for complete responses), since SAL's primary advantage is in TGT (1.1–1.3× improvement) through more even batch size distribution, while its TTFT advantage is configuration-dependent and sometimes negligible (Figure 11, SW-Chat on A100).
Prefer TRAIL+ (or prediction-based methods generally) over LARRY when: the paper does not explicitly identify conditions where TRAIL+ outperforms LARRY (LARRY beats TRAIL+ across the board in Figures 7, 8), but a reader can infer from the paper's characterization of TRAIL+ that prediction-based methods might be advantageous when output lengths are very long relative to input lengths and predictably so, since this is the regime TRAIL was designed for and LARRY's prompt-length-only memory estimation would be insufficient (Section 6.1 discusses this limitation implicitly through the assumption that "the prompt dominates a request's memory consumption"). The paper does not evaluate this regime, so this is a reasonable inference rather than an experimentally supported decision rule.