Llumnix: Dynamic Scheduling for Large Language Model Serving

🎯 Pitch

Existing LLM serving systems are broken because they dispatch requests once and never move them—but this paper shows you can literally migrate in-flight LLM requests across GPUs with near-zero downtime by pipelining KV cache copies. Doing this at runtime delivers an order-of-magnitude tail latency improvement and 36% cost savings by fixing fragmentation and load imbalance that no existing system can handle.

1. Executive Summary

Llumnix introduces a scheduling system for LLM inference serving that dynamically reschedules requests across multiple model instances at runtime via an efficient live migration mechanism (pipelining KV cache copying with token generation to achieve near-zero, constant downtime). Evaluated on a 16-GPU cluster serving LLaMA models under realistic workloads against state-of-the-art baselines, Llumnix improves P99 first-token latency by up to 15× and P99 per-token generation latency by up to 2×, accelerates high-priority requests by up to 1.5×, and delivers up to 36% cost savings while preserving similar tail latencies—establishing that runtime rescheduling unifies load balancing, de-fragmentation, and priority differentiation elegantly through a single virtual usage abstraction that reconciles the classic isolation-versus-fragmentation tradeoff even under the unpredictable memory demands and heterogeneous sequence lengths inherent to autoregressive LLM serving.

2. Context and Motivation

The Core Problem: Scheduling Is Broken for Multi-Instance LLM Serving

The fundamental problem this paper addresses is deceptively simple: how should inference requests be distributed and managed across multiple instances of an LLM to achieve acceptable latency, isolation, and utilization? Existing systems answer this question with a one-shot dispatching decision—route each incoming request to an instance, then leave it there for the entire autoregressive generation process (which may involve hundreds or thousands of sequential decoding steps). The paper argues that this static approach is fundamentally broken for the unique characteristics of LLM workloads, and that we need runtime rescheduling of in-flight requests across instances instead.

This problem arises from a gap between two separate bodies of work. On one side, the LLM inference engine community has made remarkable progress on per-instance efficiency—techniques like continuous batching (Orca, Yu et al., 2022), PagedAttention (vLLM, Kwon et al., 2023), and optimized attention kernels (FlashAttention, Dao et al., 2022) have dramatically increased the throughput a single GPU can deliver. These systems, however, focus exclusively on what happens inside one model instance. On the other side, cluster-level scheduling for DNN serving (INFaaS, Romero et al., 2021; Clipper, Crankshaw et al., 2017; TritonServer, NVIDIA) has been designed for traditional models where inference is one-shot, stateless, and deterministic—a request arrives, runs once through the model (e.g., a single forward pass for an image classifier), and terminates. The state of the art for multi-instance LLM serving essentially uses one of these generic DNN schedulers or a round-robin dispatching policy (DeepSpeed-MII), treating LLM requests as if they were traditional DNN requests. The paper's central contention is that this mismatch produces systematic failures that existing systems cannot address through better dispatching alone.

Why This Problem Matters

The paper identifies four concrete consequences of this scheduling gap, each with significant practical impact:

1. Request preemptions cause severe latency spikes. With dynamic memory allocation for KV cache (as in vLLM's PagedAttention), memory demand grows unpredictably during generation because the output length is unknown a priori. When an instance runs out of memory, it must preempt running requests—evicting them from GPU memory and later recomputing their KV cache—which can add tens of seconds of service stall. Section 3 demonstrates this empirically: even at a moderate cluster load (~62% average memory utilization), 8% of requests were preempted, and the P99 per-token decode latency was 3.8× worse than the median, with preemption loss accounting for 70% of that tail latency spike. The worst-case request experienced 50 seconds of total preemption loss across two preemptions. For interactive applications like chatbots, where users expect sub-second first-token responses, this represents a catastrophic user experience degradation.

This is not a problem that can be solved by simply over-provisioning more GPUs per instance, because the memory demand of KV cache per request scales linearly with sequence length. A LLaMA-2-13B model with 4k context requires 3.2 GB of KV cache per request, while model weights themselves consume 26 GB on an A100, and GPU memory is capped at 40–80 GB. Even with careful batching, unpredictable output lengths mean that memory exhaustion and preemption are fundamental characteristics of LLM serving, not transient bugs.

2. Co-located requests interfere with each other's performance. Figure 4 shows that the compute time for a single decode step degrades by up to 2.6× as more requests are batched together, due to competition for GPU compute units and memory bandwidth. The more requests packed onto an instance (to maximize throughput and utilization), the slower each individual request runs. This is the classic packing-vs-spreading tradeoff known from datacenter scheduling (discussed extensively in work like Paragon, Quasar, and Borg), but it takes on new dimensions in LLM serving because:

The interference is dynamic: a request's memory consumption and computation cost change throughout its lifetime as more output tokens are generated.
The interference is asymmetric: a long running request (outputting 1,000+ tokens) imposes sustained degradation on many short requests that come and go during its lifespan.

3. Memory fragmentation causes unnecessary queuing, even when cluster-wide memory is abundant. This is one of the paper's most striking findings. Spreading requests across instances (to reduce per-instance memory pressure and thus reduce preemptions) creates a different pathology: the free memory of the cluster becomes fragmented across instances such that no single instance has enough contiguous space to accommodate a new request with a long input, even though the total free memory across the cluster is sufficient. Figure 5 provides a concrete illustration: over a 700-second window, five different queuing requests were blocked on their respective instances, even though in each case the cluster-wide free memory could have accommodated the request on a different instance. The queues existed purely because of fragmentation, not because of absolute capacity shortage.

This fragmentation is a direct consequence of the prefill phase's memory requirement. While PagedAttention eliminates internal fragmentation during decode (blocks are allocated one at a time), the prefill phase requires allocating all the KV cache blocks for the input tokens simultaneously—essentially requiring a large contiguous allocation of blocks on a single instance. Having free blocks scattered across four instances is useless to a request that needs them all in one place.

4. Existing systems treat all requests equally, but real workloads demand priority differentiation. This point is both a technical and a commercial concern. Technically, different applications have different latency sensitivity: an interactive chatbot assistant needs sub-second first-token latency, while a batch evaluation job (scoring thousands of prompts) is insensitive to latency within reason. Commercially, the paper cites ChatGPT Plus, where users pay for faster responses—creating a natural requirement for the serving system to deliver differentiated service levels to different request classes. Yet "existing LLM inference systems [vLLM, Orca] often treat all requests for a model equally and cannot differentiate their priorities" (Section 1). Without priority support, a burst of latency-tolerant batch requests could saturate an instance and starve interactive requests that share the same deployment.

Why Prior Approaches Fall Short

The paper identifies specific limitations in how existing systems approach multi-instance LLM serving:

Round-robin or load-balancing dispatching (DeepSpeed-MII, AlpaServe, Ray Serve, TritonServer) cannot react to post-dispatch dynamics. A dispatching decision is made once, at request arrival, based on the information available at that moment—typically the current memory load or queue length of each instance. However, the critical information needed for good scheduling is future information: how many output tokens will each running request generate? How will each request's memory consumption grow? Since the autoregressive generation process is inherently unpredictable (the EOS token could appear at step 5 or step 500), the initial dispatching decision is essentially made with incomplete information. A request dispatched to what appeared to be a lightly loaded instance may end up growing to maximum sequence length, while a request on a different instance may complete immediately after the next iteration.

The paper frames this as a fundamental limitation: "dispatching can also consider load balancing of memory usage, [but] it could be sub-optimal as the final memory usages of requests are unknown at the arrivals, due to the unpredictability of output lengths" (Section 4.1). The only way to react to actual outcomes rather than predictions is to move requests between instances after they have started executing—hence the need for migration-based rescheduling.

Load-aware dispatching (INFaaS++) improves over round-robin but creates a conflicting optimization. INFaaS (Romero et al., 2021) introduced load-aware dispatching for multi-model serving, and the paper implements an optimized version (INFaaS++) that tracks GPU memory load including queuing requests. This helps with initial load balancing but intensifies the fragmentation problem: consistently dispatching to the least-loaded instance spreads requests evenly across instances, which is good for avoiding local overload but terrible for maintaining contiguous free space on any one instance. The paper's core insight is that you cannot simultaneously optimize for load balancing and low fragmentation through dispatching alone because these goals work in opposite directions: load balancing spreads work, fragmentation reduction packs work.

Standard datacenter schedulers have faced this packing-spreading tradeoff for decades, but in traditional settings (batch jobs, VMs), the workloads are much more predictable—a job's resource requirements are typically declared upfront (CPU cores, RAM, GPU). In LLM serving, the unknown output lengths make the tradeoff both more acute (the consequences of getting it wrong include severe preemption stalls) and harder to pre-compute (you don't know the right packing level at dispatch time).

Per-instance techniques (continuous batching, PagedAttention, preemptive schedulers) cannot see across the cluster boundary. This is the other half of the gap. vLLM, Orca, FastServe, and related systems do an excellent job of managing resources within a single instance, but they have no mechanism to coordinate requests across instances. If a request on instance A is queuing due to memory fragmentation while instance B has free space, this is invisible to the per-instance engine. The global scheduler (which sees all instances) could identify this mismatch, but without a mechanism to move in-flight requests, it cannot act on it—it can only adjust future dispatching, which leaves the already-queuing request stranded.

How This Paper Positions Itself

The paper's positioning can be understood through its central analogy: LLM serving should take inspiration from operating system process management, not traditional DNN serving. The authors explicitly state that they find "LLMs more similar to modern operating systems hosting processes with dynamic working sets and different priorities on multiple cores" (Section 1) than to traditional one-shot DNN models. This analogy shapes every design decision:

Context switching: Just as an OS can preempt a process, save its state to memory, and resume it on a different CPU core, Llumnix can migrate an in-flight request with its KV cache from one GPU instance to another, enabling runtime load balancing across the cluster.
Virtual memory / page migration: Just as an OS uses virtual memory to give processes the illusion of contiguous memory even when physical pages are scattered, Llumnix's live migration mechanism uses the append-only property of KV cache to pipeline state transfer with computation, making the migration downtime constant rather than proportional to sequence length.
Priority scheduling: Just as an OS can boost the priority of an interactive process while deprioritizing background computation, Llumnix can reserve headroom on instances for high-priority requests and migrate co-located normal requests away to maintain low interference.
Working set migration: Just as VM live migration (Clark et al., 2005) transfers dirty pages iteratively while the VM continues running, Llumnix copies KV cache blocks while the request continues decoding, reducing downtime to a single iteration's worth of state transfer.

This OS analogy is not just a rhetorical device—it directly motivates the paper's key technical contribution: that rescheduling at runtime (via migration) is the missing primitive that enables a dynamic scheduler to reconcile the conflicting goals of load balancing, de-fragmentation, and priority differentiation. In the OS world, a scheduler that could only make one initial processor assignment per process and never migrate would be considered primitive; in an OS context, process migration for load balancing is a well-understood technique going back decades. The paper's contribution is to recognize that LLM serving has reached the complexity point where this OS lesson applies—the workloads are dynamic, multi-tenant, and have differentiated quality-of-service requirements—and to provide the first practical mechanism for "context switching" LLM requests across instances.

The paper also positions itself relative to Gandiva (Xiao et al., 2018), which introduced introspective migration for deep learning training jobs. Gandiva migrates training jobs between GPUs by checkpointing at mini-batch boundaries, which is possible because training is iterative and the working set (model weights) can be saved and restored. However, LLM inference migration is significantly harder because the latency SLO is measured in milliseconds (not hours), and the per-request state (KV cache) is proportional to sequence length—making naive checkpoint-and-restore approaches far too slow. Llumnix's live migration mechanism addresses this head-on, making the migration downtime constant with respect to sequence length by leveraging the same append-only KV cache property that makes preemption so costly in the first place.

Finally, the paper explicitly identifies itself as filling a gap in the LLM serving landscape: while inference engines (vLLM, Orca, FasterTransformer) and scheduling systems (INFaaS, AlpaServe) have each advanced one side of the problem, "the common practice today is still to use generic scheduling systems or policies inherited from the era of traditional DNNs" for multi-instance coordination. Llumnix aims to close this gap by introducing runtime migration as a new dimension of scheduling flexibility, one that enables the system to react to workload dynamics after the initial dispatching decision, much as an OS reacts to changing process demands after initial process creation.

The Unifying Challenge: All Problems Stem from the Same Root Cause

A key insight the paper conveys—implicitly through its architecture but explicitly through the virtual usage abstraction—is that the four problems above (preemptions, interference, fragmentation, lack of priorities) all stem from a single underlying cause: the inability to move requests between instances after execution begins. If requests could be freely migrated, then:

Preemptions could be avoided by moving requests off an instance before memory exhaustion and onto an instance with free space.
Fragmentation could be corrected by moving short requests to pack free space on a target instance, creating room for long-input requests.
Priority isolation could be created by migrating normal requests away from instances serving high-priority requests, dynamically adjusting per-instance load.
Auto-scaling could proceed faster by draining or saturating instances on demand rather than waiting for natural request completions.

The paper therefore frames request migration as the single enabling mechanism that transforms LLM serving from a stateless dispatching problem into a dynamic resource management problem, analogous to the transformation that preemptive multitasking brought to operating systems. The rest of the paper is devoted to making this mechanism efficient (Section 4.2), scaling it across many instances (Section 4.3), and designing policies that exploit it to achieve multiple scheduling goals simultaneously (Section 4.4).

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

Llumnix is a scheduling layer that sits on top of existing LLM inference engines and dynamically migrates in-flight requests between multiple model instances at runtime. The core problem it solves is that one-shot dispatching of LLM requests to instances cannot adapt to the unpredictable memory demands and heterogeneous sequence lengths of autoregressive generation—causing preemption stalls, memory fragmentation, and interference—so Llumnix provides the missing primitive of request migration, analogous to OS context switching, enabling a global scheduler to continuously rebalance and reorganize the workload across instances as conditions change.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components, organized in a two-level hierarchy:

Backend Inference Engines — per-instance GPU processes (vLLM in the current implementation) that perform the actual autoregressive token generation, manage KV cache memory via PagedAttention, and expose interfaces for batching, preemption, and memory block operations. Each engine runs on one or more GPUs (tensor parallelism).
Llumlets — per-instance distributed actors that serve as the bridge between cluster-level scheduling and per-instance execution. Each llumlet contains a local scheduler (manages request queuing, batching, and block allocation inside its instance) and a migration coordinator (orchestrates the handshake protocol for moving requests in/out while coordinating with the local scheduler and the remote llumlet). The llumlet is responsible for computing the instance's memory load under the virtual usage abstraction and reporting it to the global scheduler.
Global Scheduler — a single cluster-level actor that makes all cross-instance scheduling decisions. It does NOT track individual requests; instead, it operates on instance-level load metrics (freeness values) reported periodically by llumlets. It dispatches new incoming requests to instances, triggers migration by pairing source and destination instances, and controls auto-scaling by adding or removing instances. The global scheduler's decisions are coarse-grained (which instances to migrate between), while the llumlets handle fine-grained decisions (which specific requests to migrate).
Request Frontends — distributed actors that expose an OpenAI-compatible API endpoint to external clients. When a request is migrated between backend instances, the frontend remains the stable endpoint: generated tokens are forwarded from whichever backend instance currently hosts the request to the frontend, which streams them back to the client without interruption.

Information flows as follows: an external client submits a request to a frontend → the frontend forwards it to the global scheduler for dispatching → the global scheduler routes it to the llumlet of the chosen instance → that llumlet queues the request and, when batching conditions allow, admits it to the inference engine for execution → during execution, the llumlet periodically reports its instance's freeness (a composite load metric) to the global scheduler → the global scheduler evaluates migration triggers based on these reports and, when a migration is needed, marks specific instances as source/destination → the source llumlet selects specific requests to migrate and initiates the live migration handshake with the destination llumlet → KV cache blocks are copied between instances while token generation continues → after migration completes, the destination llumlet resumes the request on its local engine → the frontend streams tokens from the new backend instance, transparently to the client.

3.3 Roadmap for the Deep Dive

First, the live migration mechanism (Section 4.2) — because request migration is the fundamental primitive that enables everything else, and its efficiency determines whether runtime rescheduling is practically viable. We must understand how the append-only KV cache property is exploited to pipeline copying with computation, achieving near-zero downtime.
Second, the distributed scheduling architecture (Section 4.3) — because the two-level design (global scheduler + llumlets) is what makes continuous rescheduling scalable. This section explains the clean separation of concerns and the narrow interface (instance-level load reports) that decouples the global scheduler from individual request tracking.
Third, the virtual usage abstraction and dynamic scheduling policy (Section 4.4) — because this is the intellectual core that unifies load balancing, de-fragmentation, prioritization, and auto-scaling into a single load metric. We must understand how virtual usage is calculated differently for different request types and scenarios, and how the freeness metric derived from it drives all scheduling decisions.
Fourth, the dispatching, migration, and auto-scaling policies (Section 4.4.3) — because these are the concrete algorithms that use the virtual usage abstraction to make decisions. This builds directly on the previous subsection.
Fifth, implementation details (Section 5) — covering the specific engineering choices (Ray actors, Gloo communication, block fusion, fault tolerance) that make the architecture practical on real GPU clusters.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems design paper whose core idea is that LLM serving requires a new scheduling primitive—request migration across instances—and that this primitive can be implemented efficiently enough to enable continuous, dynamic rescheduling that unifies multiple scheduling goals through a single abstraction called virtual usage.

The Root Enabling Insight: KV Cache Is Append-Only

Before diving into the migration mechanism itself, we must understand the one property of LLM inference that makes efficient migration possible. During autoregressive generation, the model iteratively produces one output token at a time. At each iteration, the model computes attention over all previous tokens (input tokens plus all output tokens generated so far). The intermediate results of this attention computation—the key and value tensors for each token in each transformer layer—are stored in GPU memory as the KV cache. Critically, the KV cache entries for previously processed tokens are never modified by subsequent iterations; they are only read. When iteration $t+1$ executes, it reads the KV cache entries from iterations $1$ through $t$ (which are immutable) and appends new entries for token $t+1$ . This append-only property is the linchpin: it means that KV cache blocks from earlier iterations can be safely copied to another GPU while later iterations execute, because no in-flight computation will write to those blocks.

This property distinguishes LLM inference from many other stateful computations. In a database transaction or a virtual machine, the working set (memory pages) can be modified by ongoing computation, requiring dirty-page tracking and iterative re-copying (as in VM live migration). The KV cache's immutability of historical entries eliminates this complexity entirely, enabling a simpler multi-stage pipeline.

Live Migration Mechanism (Section 4.2)

The naive alternatives and why they fail. Before presenting the live migration design, the paper identifies two straightforward approaches and explains why both are unacceptable for LLM serving:

Recomputing: Kill the request on the source instance and restart it from scratch on the destination instance, recomputing the entire KV cache by running the model over the full input prefix plus all output tokens generated so far. This introduces downtime proportional to sequence length—recomputing an 8k-token sequence takes 3.5 seconds for LLaMA-30B (Figure 10), equivalent to roughly 54 decode steps. For an interactive application expecting sub-second responses, this is catastrophic.
Blocking copy: Pause the request's execution on the source instance, copy all KV cache blocks to the destination via GPU-to-GPU transfer (using NCCL or Gloo), then resume on the destination. The downtime equals the total data transfer time, which scales linearly with sequence length (Figure 10 shows this reaching over 3 seconds for 8k sequences on LLaMA-30B). While faster than recomputing, this still introduces an unacceptable service stall.

Both approaches make migration downtime proportional to the state size, and since LLM state grows with sequence length, they become increasingly impractical as models support longer contexts (the trend from 32k to 128k tokens).

The multi-stage pipelined approach. Llumnix's key insight is to decompose the KV cache into two categories: already-generated blocks (from previous iterations, immutable) and in-progress blocks (from the current and very recent iterations, still being written or just completed). The mechanism proceeds in stages:

Stage 0 (initial copy + computation): When migration is initiated, the source instance begins copying all KV cache blocks that are already finalized (those from iterations before the current one). While this copy is in progress, the source instance continues decoding—producing new output tokens and appending new KV cache blocks. The copy and the computation execute concurrently: the GPU runs inference kernels on one CUDA stream while a separate stream handles the memory transfer to CPU (for Gloo-based transfer) or directly to the destination GPU.

Stage 1 (copy the delta): Once the initial block copy completes, there will be a small number of new KV cache blocks that were generated during Stage 0's computation. Stage 1 copies these delta blocks while again allowing computation to continue. Because the data transfer rate is generally much faster than the token generation rate (each token generation requires a full forward pass through all transformer layers, while a block is just a few hundred KB of data), the delta accumulated during one stage is typically small—often just a handful of blocks.

Stage N (final synchronization): The process repeats until the number of uncopied blocks is small enough to copy in negligible time. In the final stage, the source instance drains the request from its current batch (stops including it in computation for one iteration) and copies the remaining blocks. This introduces the downtime of the request—but it is only the time to copy the blocks generated in the most recent iteration (one decode step's worth of KV cache), not the entire sequence. Once this final copy completes, the destination instance resumes the request by adding it to its own running batch, and the source instance releases its local blocks.

Empirical behavior. The paper reports (Section 6.2, Figure 10) that for all tested sequence lengths (256 to 8k tokens) and both model sizes (7B and 30B), the migration completes in exactly two stages (Stage 0 and Stage N). This is because the GPU's KV cache copying speed (via Gloo with CPU staging) is fast enough relative to the decode computation speed that the number of new blocks generated during Stage 0 is always small enough to copy in the final stage without needing intermediate deltas. The downtime is roughly 20–30 ms across all sequence lengths—less than a single decode step's latency for typical configurations—and is constant with respect to sequence length, contrasting with the linear growth of blocking copy and recompute (which reach up to 111× the migration downtime at 8k tokens).

Why this works for LLMs but not for general stateful services. The append-only KV cache property means that the "working set" of mutable state is only the most recent iteration's output. This is fundamentally different from VM live migration, where any memory page can be dirtied at any time, requiring iterative re-copying of pages that were modified during the previous copy phase. In LLM inference, the working set is trivially bounded (one iteration's output) and known in advance (the KV cache never shrinks or modifies). This makes the migration protocol both simpler and more efficient than general VM migration.

The handshake protocol and exception handling. Because both source and destination instances are continuously processing other requests, and because the request being migrated might complete (generate an EOS token) or run out of memory during migration, Llumnix needs a coordination protocol to handle these cases. The mechanism is a per-stage handshake between the source and destination llumlets (Figure 7):

Pre-allocation: Before each stage, the source instance sends a pre-allocate request to the destination specifying the number of KV cache blocks to be transferred in that stage. The destination attempts to allocate and reserve these blocks from its free block pool. If allocation succeeds, the destination sends an ACK; if it fails (due to insufficient free memory), it sends an ABORT. The pre-allocation ensures that the migration never starts a transfer only to fail mid-way, which would require rolling back partial state.
Post-stage check: After each stage (before proceeding to the next), the source instance checks whether the request being migrated has either completed (generated an EOS token) or been preempted (evicted by the local scheduler due to memory pressure). If either condition is true, the source notifies the destination to ABORT the migration and release any reserved blocks. This prevents wasted work: if the request finished, there is nothing to migrate; if it was preempted, the request's state is already lost on the source and cannot be meaningfully transferred.
Final commit: After the last stage, the source releases its local blocks for the request (freeing memory on the source instance) and notifies the destination to COMMIT the migration. The destination then adds the request to its local scheduler's queue (or directly to the running batch) with the received KV cache blocks, and the request resumes execution.
Failure handling: If either instance fails at any point during the handshake (detected via timeout or explicit error), both sides clean up their reserved blocks and the migration is aborted. The request continues running on the source instance as if the migration had never been initiated.

This handshake protocol is essential for correctness because the migration spans multiple asynchronous operations (computation, memory copy, network transfer) whose durations are not known in advance, and because the request's own lifecycle (completion, preemption) proceeds independently of the migration.

Distributed Scheduling Architecture (Section 4.3)

Why a two-level design? The paper identifies a scalability challenge that would arise from a naive centralized design. If a single global scheduler tracked every running request across all instances, it would need to maintain and update the status (current sequence length, memory consumption, queue position, priority) of potentially thousands of requests, synchronized with every iteration of every instance. This would create a communication bottleneck: on each iteration, each instance would need to report updates (which requests completed, which grew in memory, which were preempted) to the global scheduler, and the global scheduler would need to process this state and potentially make rescheduling decisions. With 16+ instances each processing batches of dozens of requests, this fine-grained synchronization would overwhelm the scheduler.

Llumnix's architecture avoids this by defining a clean separation of concerns: the global scheduler operates exclusively on instance-level metrics, not request-level state, while the per-instance llumlets handle all request-level tracking and make the fine-grained decisions (which requests to migrate, which to preempt) autonomously.

The llumlet: local scheduler + migration coordinator. Each model instance has a co-located llumlet (implemented as a Ray actor) that serves three functions:

Local scheduling (traditional): Manages the request queue for its instance, decides which queuing requests to admit to the running batch based on available memory and scheduling priority, and interacts with the inference engine for batching and block management. This is similar to what vLLM's built-in scheduler does, but with additional hooks for migration and virtual usage accounting.
Load reporting: Periodically computes the instance's freeness metric (defined in Section 4.4.3) based on the virtual usages of all requests (running and queuing) on the instance, and reports this single scalar to the global scheduler. This is the critical interface: the global scheduler never asks "what requests are on this instance?" — it only asks "how loaded is this instance?" The abstraction collapses all the complexity of request-level state into one number.
Migration execution: When the global scheduler marks the instance as a source or destination for migration, the llumlet's migration coordinator takes over. As source, it selects which specific requests to migrate (preferring lower-priority, shorter-sequence requests), initiates the handshake with the destination llumlet, and instructs the inference engine to perform the KV cache transfer. As destination, it responds to pre-allocation requests, reserves blocks, and integrates the migrated request into its local queue.

The global scheduler: instance-oriented decisions. The global scheduler is a single actor that:

Receives periodic freeness reports from all llumlets.
Maintains a view of which instances are in which states (normal, source, destination, terminating).
Dispatches new incoming requests to the instance with the highest freeness (Section 4.4.3).
Evaluates migration triggers: if any instance's freeness falls below a lower threshold or rises above an upper threshold, it becomes a candidate for migration. The scheduler pairs the most overloaded instance (lowest freeness) with the least loaded instance (highest freeness), marks them as source and destination respectively, and leaves the llumlets to execute the actual migration.
Controls auto-scaling: monitors the average freeness across instances and adds or removes instances to keep it within a target range.

Why this separation scales. The global scheduler's workload is $O(\text{\# instances})$ , not $O(\text{\# requests})$ . The llumlets handle request-level operations in parallel across instances, and the migration handshake is a point-to-point protocol between two llumlets that requires no global scheduler involvement after the initial pairing. This means the global scheduler can manage large clusters without becoming a bottleneck—the stress test in Section 6.6 shows that with 64 instances, a centralized scheduler (tracking all requests) experiences up to 40 ms of scheduling stall per iteration under high load (1.7× slowdown), while Llumnix's distributed design shows near-zero scheduling overhead at the same request rates.

Interface between levels. The narrow interface—llumlets report a single freeness value per reporting interval; the global scheduler responds with instance state changes (source/destination flags, scaling commands)—is deliberately simple. This makes the global scheduler's logic independent of the specific inference engine (vLLM, Orca, etc.) and the specific model architecture, enabling Llumnix to work as a non-intrusive layer on top of different backends.

Virtual Usage: The Unifying Abstraction (Section 4.4.1–4.4.2)

This is the paper's intellectual core—the abstraction that makes all the different rescheduling scenarios (load balancing, de-fragmentation, prioritization, auto-scaling) expressible as a single load-balancing problem.

The fundamental observation: all rescheduling scenarios reduce to creating or eliminating load imbalance. The paper identifies that its rescheduling cases fall into two categories: (1) genuine load balancing—moving requests away from overloaded instances to underloaded ones to reduce preemptions and interference—and (2) creating free space on a specific instance for a specific purpose (accommodating a queuing request, reserving headroom for high-priority requests, draining an instance for termination). The insight is that category (2) can be simulated as load imbalance by artifically inflating the reported load of the instance that needs free space. If the load appears high enough, the load-balancing policy will automatically migrate requests away, creating the desired free space without needing a separate mechanism.

The virtual usage function. Each request on an instance is assigned a virtual usage—a value that may differ from its actual physical memory consumption. The physical usage is the number of KV cache blocks currently allocated to that request. The virtual usage is computed by a function CalcVirtualUsage(req, instance) (Algorithm 1) that applies different rules depending on the request's type and the instance's state:

For a queuing request (not yet admitted to the running batch):

If the request is the head-of-line (first in the queue), its virtual usage is set to its demand—the number of KV cache blocks it would require if admitted. The demand equals the number of tokens in its input (since prefill requires allocating blocks for all input tokens at once).
If the request is not head-of-line, its virtual usage is 0.

What this achieves: By inflating the head-of-line queuing request's virtual usage to its full demand, Llumnix makes the instance appear more loaded than its physical memory usage would suggest—potentially overloaded. This triggers load-balancing migration, which moves running requests off the instance, freeing physical blocks and thereby creating space for the queuing request. In effect, the load-balancing policy is being repurposed as a de-fragmentation mechanism: instead of explicitly reasoning about fragmentation (which requires knowing the free block layout across instances), Llumnix simply makes instances with blocked queuing requests appear overloaded and lets the load-balancer fix the imbalance. Figure 9(b) illustrates this: the yellow request (queuing, head-of-line) has a virtual usage bar equal to its demand, making the total virtual usage of instance 1 exceed the memory capacity, so the load-balancing policy migrates the green and blue running requests to instances 2 and 3.

For a running request:

If the request has normal execution priority, its virtual usage equals its physical usage (the actual number of allocated KV cache blocks).
If the request has high execution priority, its virtual usage equals its physical usage plus a headroom term: GetHeadroom(p, instance) returns headroomForPriority[p] / instance.numRequests[p], where headroomForPriority[p] is a fixed memory reservation configured for priority level $p$ , and instance.numRequests[p] is the number of requests of that priority currently running on the instance. In the current implementation, there are two priority classes: high and normal. The headroom for normal requests is 0. The headroom for high priority is empirically set to the memory that would maintain "near-ideal decode speed" (i.e., no visible interference), determined through offline profiling.

What this achieves: By inflating the virtual usage of high-priority requests, Llumnix makes the instance appear more loaded than it physically is. This triggers migration of normal-priority requests away from the instance, reducing the total number of co-located requests. With fewer requests competing for GPU resources, the high-priority requests experience lower interference and faster decode times. The headroom is divided among all high-priority requests on the instance (line 10 of Algorithm 1: headroomForPriority[p] / instance.numRequests[p]) so that if multiple high-priority requests share an instance, the total reserved headroom remains bounded.

For a fake request (used during instance termination):

Virtual usage is set to $\infty$ (infinity). This is an artificial request added to the instance's request list when the instance is marked for termination.

What this achieves: An infinite virtual usage makes the instance's total virtual usage infinite, ensuring it will always be selected as the most overloaded instance. The load-balancing policy then migrates all real requests off the instance as quickly as possible, draining it for clean shutdown. This replaces the slow process of waiting for natural request completions (which could take minutes for long-running requests).

The freeness metric. From the virtual usages, the llumlet computes its instance's freeness:

$F = \frac{M - \sum V}{B}$

where $M$ is the total memory capacity of the instance (in KV cache blocks), $\sum V$ is the sum of virtual usages of all requests (running and queuing) on the instance, and $B$ is the current batch size (number of running requests). The numerator $M - \sum V$ is the virtual free space—the number of free blocks adjusted for the inflated usages. The denominator $B$ normalizes by batch size.

What it computes: Freeness estimates how many more decode iterations the current batch can sustain before the instance runs out of memory, under the conservative assumption that virtual usages reflect true resource pressure. If the batch size $B$ is large, then each iteration consumes blocks for $B$ new tokens (one per request in the batch), so a given free space is exhausted faster. Dividing by $B$ converts an absolute free space into a time-to-exhaustion estimate.

Why this form: A purely memory-based metric (just $M - \sum V$ ) would misrepresent the urgency of load imbalance. Two instances with the same free space but different batch sizes have different consumption rates—the one with the larger batch will exhaust its memory sooner. Normalizing by batch size captures this dynamic component. Furthermore, because virtual usage can exceed physical usage (for queuing requests, high-priority requests, or fake requests), freeness can be negative. A negative freeness signals that the instance is "overloaded" in the virtual sense—it has more demand (or reserved headroom) than capacity—and should not receive new requests. This is crucial for the scheduling policies: dispatching a new request to an instance with negative freeness would be counterproductive, as that instance is already trying to shed load.

Unification of scenarios. The power of virtual usage is that all the scheduling scenarios described in Figure 1 are handled by the same underlying mechanism:

Load balancing (Figure 1a): Normal running requests have virtual usage = physical usage. Freeness reflects true memory load. Migration is triggered when freeness imbalance exceeds thresholds.
De-fragmentation (Figure 1b): Head-of-line queuing requests get virtual usage = demand. This inflates the instance's total virtual usage, lowering freeness (potentially below zero), triggering migration of running requests off the instance to free blocks for the queuing request. This works without any explicit fragmentation detection—the system simply treats blocked queues as overload.
Prioritization (Figure 1c): High-priority running requests get virtual usage = physical + headroom. This inflates the instance's virtual usage when high-priority requests are present, lowering freeness, triggering migration of normal requests away, reducing interference.
Auto-scaling drain (Figure 1d): A terminating instance gets a fake request with infinite virtual usage, driving its freeness to $-\infty$ , ensuring continuous outbound migration until all real requests are drained.
Auto-scaling saturation: A newly launched instance starts with zero virtual usage (no requests), so its freeness is at maximum. This naturally attracts dispatching of new requests and inbound migration from overloaded instances, saturating the new instance quickly.

The paper summarizes this elegantly: "Llumnix just needs to define a set of rules for setting the virtual usages of requests in different scenarios, and then use a simple load-balancing policy based on the virtual usages" (Section 4.4.2).

Scheduling Policies (Section 4.4.3)

With virtual usage and freeness defined, the paper specifies three concrete policies for dispatching, migration, and auto-scaling.

Dispatching policy. When a new request arrives at the global scheduler:

Requests are ordered by scheduling priority (higher first), and within the same priority, by arrival time (FCFS). Scheduling priority is a separate dimension from execution priority—it controls queue ordering, while execution priority controls the headroom and interference protection.
The request is dispatched to the instance with the highest freeness $F$ . This is a greedy load-balancing decision, but because freeness incorporates virtual usages, it automatically avoids instances that are overloaded (negative freeness), have queuing requests with large demands, or are serving high-priority requests with reserved headroom.
If all instances have equal or very similar freeness, the policy effectively performs round-robin-like spreading, but with the crucial difference that virtual usage biases the decision toward instances with actual capacity to absorb the new request.

Migration policy. The global scheduler evaluates migration periodically (every reporting interval):

It selects candidate source instances as those with freeness below a configurable lower threshold, and candidate destination instances as those with freeness above a configurable upper threshold. The thresholds control how aggressively the system rebalances—tighter thresholds (smaller gap) lead to more frequent migration but more balanced load.
It pairs sources and destinations greedily: pick the instance with the lowest freeness (most overloaded) as source and the instance with the highest freeness (least loaded) as destination, mark them accordingly, and repeat until no more pairs can be formed with freeness values on opposite sides of the thresholds.
Once marked as a source, the llumlet continuously migrates requests to the paired destination. The llumlet selects which specific requests to migrate by preferring: (a) lower execution priority requests first, and (b) among requests of the same priority, shorter sequence lengths (lower migration cost, faster completion of the migration round). The migration continues until the source llumlet determines that its freeness has risen above the threshold, at which point it stops selecting new requests for migration (already-in-flight migrations complete).
In the next scheduling round, if a previously marked instance is no longer beyond the thresholds, its migration state is cleared and the migration stops. This prevents over-correction: the system continuously adjusts as load changes, rather than making one large migration decision and then having to reverse it.

Auto-scaling policy. Llumnix adjusts the number of active instances based on the average freeness across all instances, computed using only the virtual usages of normal-priority requests (excluding the inflation for headroom, queuing requests, and fake requests). The policy maintains the average freeness within a target range $[x, y]$ :

If the average freeness is below $x$ for a sustained period (to avoid reacting to transient spikes), the global scheduler launches a new instance. The new instance starts with zero load (maximum freeness) and naturally attracts dispatching and migration, raising the cluster-wide average.
If the average freeness is above $y$ for a sustained period, the global scheduler selects an instance for termination. It chooses the instance with the fewest running requests to minimize the amount of migration needed. The selected instance is marked as terminating, a fake request with infinite virtual usage is added (making its freeness $-\infty$ ), and the migration policy automatically drains all real requests off it to other instances. Once the instance has no running requests, it is safely decommissioned.

Why the range $[x, y]$ matters. If the auto-scaling only added instances at a single threshold (e.g., freeness < 0) and removed at another, the system would oscillate—adding an instance immediately raises the average freeness, potentially triggering removal, creating instability. The target range provides hysteresis: the system tolerates some variation without scaling, adding capacity only when persistently overloaded and removing only when persistently underloaded. The paper uses $[10, 60]$ as the default range (Section 6.5), meaning: if the average instance can sustain only 10 or fewer decode iterations before memory exhaustion, scale up; if it can sustain 60 or more, scale down. The specific values are chosen based on the desired balance between cost (fewer instances) and latency (more headroom).

Implementation Details (Section 5)

The paper describes several engineering choices that make the architecture practical on real GPU clusters:

Ray actors for distributed components. Llumnix implements the global scheduler, each llumlet, each backend inference engine instance, and each request frontend as separate Ray actors (Moritz et al., 2018). Ray is a Python-native distributed runtime that provides actor-based concurrency with asynchronous message passing. Using Ray allows fine-grained coordination between these components (e.g., the migration handshake between two llumlets) without building custom RPC infrastructure. Each GPU hosts one Ray actor for the inference engine and one for the llumlet; frontends and the global scheduler can run on CPU-only nodes.

Gloo for KV cache transfer. The KV cache blocks are transferred between instances using the Gloo collective communication library (a Facebook-developed library for CPU-based collective operations), specifically its Send/Recv primitives. A natural alternative would be NCCL (NVIDIA's GPU-optimized communication library), which is generally faster for GPU-to-GPU transfers and is already used for tensor parallelism within a single multi-GPU instance. However, Llumnix needs to perform KV cache transfers concurrently with GPU computation (the inference kernels running on the main CUDA stream), and NCCL is documented as unsafe for concurrent invocations from the same process (NVIDIA documentation states that "using multiple NCCL communicators concurrently" can lead to deadlocks). Since the pipelined migration design hides transfer latency behind computation anyway (only the final stage's small delta contributes to downtime), using the slower but concurrency-safe Gloo is a pragmatic choice that avoids interfering with inference.

CPU staging for Gloo. Because Gloo operates on CPU memory, Llumnix must copy KV cache blocks from GPU memory to a CPU buffer before sending, and from CPU to GPU on the destination side. This copy is performed on a separate CUDA stream from the main inference stream, ensuring it does not block the token generation computation. The tradeoff is higher total data movement (GPU → CPU → network → CPU → GPU, rather than GPU → network → GPU with GPUDirect RDMA), but the pipelining hides this cost for all but the final stage.

Block fusion to amortize message overhead. vLLM's PagedAttention stores KV cache in small, non-contiguous blocks (e.g., for a 16-bit LLaMA-7B model, each block is 128 KB, and a 1k-token sequence with 32 layers translates to 4,000 blocks). Sending each block as a separate Gloo message would incur prohibitive per-message overhead. Llumnix fuses the blocks by copying them from GPU memory into a contiguous CPU buffer (aggregating all blocks for a given request), then sends the entire buffer as a single Gloo message. This dramatically reduces the number of messages and the associated overhead.

Fault tolerance. Llumnix handles two failure modes:

Global scheduler failure: The frontends detect that the global scheduler is unreachable and enter a scheduler-bypassing mode: they dispatch new requests directly to instances using simple rules (e.g., round-robin or random), and migration is disabled. This ensures that the service remains available for new requests, albeit with degraded scheduling quality. When the global scheduler restarts (Ray automatically restarts failed actors), the system transitions back to normal operation.
Instance (or llumlet) failure: If a backend instance or its co-located llumlet fails, all requests running on that instance are aborted (their state in GPU memory is lost). Ongoing migrations involving the failed instance are also aborted, handled by the handshake protocol's timeout mechanism—the healthy side detects the failure, cleans up reserved blocks, and the request being migrated either continues on the source (if the source is healthy) or is aborted (if the destination failed after the source released its blocks). Ray automatically restarts failed actors, and the restarted instance re-joins the cluster with zero load.

Extensibility to other backends. Llumnix is implemented as 3,300 lines of Python code, structured as a standalone library that interfaces with backend inference engines through a defined API. The current implementation supports vLLM, but the paper notes that the architecture is "non-intrusive and extensible to different backends" because the interface is narrow: the backend must support queuing, batching, block allocation/deallocation, and the ability to copy KV cache blocks to/from a provided buffer. Any inference engine that provides these primitives (e.g., Orca, TensorRT-LLM) could be integrated.

Summary of Design Choices and Their Justifications

Live migration over recomputation or blocking copy: The pipelined approach exploits the append-only KV cache property to achieve downtime that is constant ( $\sim$ 20–30 ms) with respect to sequence length, versus linear growth ( $\sim$ 3.5 seconds at 8k tokens) for alternatives. This makes migration viable for long-context requests, which are the ones most likely to need rescheduling (they consume the most memory and cause the most fragmentation).
Two-level architecture (global scheduler + llumlets) over centralized tracking: The instance-level interface (llumlets report freeness, not request lists) keeps the global scheduler's complexity $O(\text{\# instances})$ rather than $O(\text{\# requests})$ , enabling scalability to 64+ instances without scheduling stalls (Section 6.6). The llumlets parallelize request-level work and execute migrations asynchronously.
Virtual usage over explicit multi-objective optimization: Rather than designing separate mechanisms for load balancing, de-fragmentation, prioritization, and auto-scaling—each with its own metrics, thresholds, and conflict-resolution logic—Llumnix reduces all of them to a single load-balancing objective by inflating the reported load of requests in specific scenarios. This is philosophically similar to how OS schedulers use "nice" values to express priority as a load offset, rather than implementing priority as a separate scheduling dimension.
Freeness normalized by batch size over raw free space: Dividing free space by batch size converts a static capacity metric into a dynamic time-to-exhaustion metric, capturing the fact that memory consumption rate depends on how many requests are running. This makes the metric self-correcting: when an instance has many small requests, its batch size is large and freeness is low (accurately reflecting high consumption rate), even if the absolute free space is moderate.
Gloo with CPU staging over NCCL for KV cache transfer: NCCL's concurrency unsafety makes it incompatible with overlapping transfer and computation on the same GPU. The slower but safer Gloo path is acceptable because the pipelining hides transfer latency, and only the final stage's delta contributes to downtime. The block fusion optimization mitigates Gloo's per-message overhead.
Head-of-line queuing request virtual usage = full demand (not gradual): The paper chooses to inflate the queuing request's virtual usage to its full demand immediately, "which favours reducing queuing delays" (Section 4.4.2). A gradual approach (slowly increasing virtual usage as the request waits) would trade off between de-fragmentation urgency and load balancing—a request that just started waiting might not justify disrupting the current load distribution. The paper argues that because "queuing delay can dominate the end-to-end latency," immediate full inflation is preferable, and "the high flexibility of migration" ensures that load balancing benefits are preserved despite this aggressive de-fragmentation bias.
Ray actors over custom distributed framework: Using Ray provides battle-tested actor management (creation, failure detection, restart), asynchronous message passing, and Python-native programming. This reduces implementation complexity significantly versus building a custom RPC layer. The tradeoff is dependence on the Ray runtime, but given Ray's widespread adoption in ML infrastructure, this is a reasonable engineering choice.

4. Key Insights and Innovations

Innovation 1: Request Migration as the Missing Scheduling Primitive for Stateful LLM Inference

The dominant assumption across all prior LLM serving systems—from inference engines (vLLM, Orca) to cluster-level schedulers (INFaaS, AlpaServe, DeepSpeed-MII, TritonServer)—is that requests are immobile once dispatched. A request is routed to an instance at arrival time and remains there for the entire autoregressive generation, which may span hundreds or thousands of decoding steps. This immobility is not an explicit design choice but an inherited assumption from traditional DNN serving, where inference is one-shot and stateless: a request runs through the model once (a single forward pass) and terminates, making migration both unnecessary (there is no ongoing state to manage) and trivial (there is no state to transfer).

The paper identifies this assumption as the root cause of the four scheduling pathologies documented in Section 3. Preemptions occur because a request cannot be moved when its instance runs out of memory—it must be evicted and recomputed. Fragmentation causes queuing because a long-input request cannot be relocated to an instance with contiguous free space—it must wait on its originally assigned instance. Priorities cannot be enforced because a high-priority request cannot shed co-located interference—it is stuck with whatever requests happen to share its instance. Auto-scaling is slow because requests cannot be drained from a terminating instance—they must complete naturally.

The intellectual contribution is not the migration mechanism itself (live migration is well-known from VM management, Clark et al. 2005, and was applied to DL training by Gandiva, Xiao et al. 2018), but the recognition that stateful, unpredictable, long-running inference constitutes a new class of workload that demands mobile requests, and that the append-only KV cache property makes migration tractable where it would be prohibitive for general stateful services. This is a fundamental reframing of the scheduling problem: from a one-shot dispatching optimization (how to best assign arriving requests to instances given current load information) to a continuous dynamic resource management problem (how to reposition in-flight requests as conditions evolve). The analogy to OS process migration is precise—just as an OS scheduler would be considered primitive if it could never move a process between cores after creation, Llumnix argues that an LLM serving scheduler is primitive if it can never move a request between instances after dispatch.

This reframing has implications beyond the specific mechanisms in this paper. It establishes migration as a first-class scheduling action in the LLM serving design space, analogous to how preemptive multitasking transformed OS design from cooperatively-scheduled to interactively-responsive. Any future system that grapples with load imbalance, fragmentation, or priority in multi-instance LLM deployment must now either incorporate migration or justify why it can achieve equivalent results without it.

The evidence supporting the significance of this reframing is the breadth of improvements from a single mechanism: P99 first-token latency improved by up to 15× through de-fragmentation (Figure 11, L-L trace), P99 decode latency improved by up to 2× through preemption reduction (Figure 11), high-priority request latency improved by 1.5× through dynamic isolation (Figure 13), and 36% cost savings through improved auto-scaling efficiency (Figure 15). No prior system demonstrated comparable multi-objective improvements because no prior system had the primitive that makes them possible.

Innovation 2: Virtual Usage as a Unified Optimization Abstraction for Conflicting Scheduling Goals

The second fundamental contribution is an abstraction that collapses multiple, apparently conflicting scheduling objectives into a single load-balancing problem. Before Llumnix, a scheduler that wanted to balance load across instances (to reduce preemptions), de-fragment memory (to reduce queuing), isolate high-priority requests (to meet SLOs), and drain terminating instances (for auto-scaling) would need separate mechanisms, each with its own metrics, thresholds, and decision logic—and would face the problem that these mechanisms work against each other. Load balancing spreads requests across instances, which is good for reducing local memory pressure but bad for fragmentation (free space gets scattered). De-fragmentation packs requests onto fewer instances, which is good for accommodating long inputs but bad for preemption risk. Priority isolation reserves capacity, which is good for latency guarantees but bad for overall utilization. The packing-versus-spreading tradeoff is a classic and notoriously difficult scheduling problem, studied extensively in datacenter resource management (Borg, Paragon, Quasar, Gandiva) and typically addressed through complex multi-dimensional optimization or heuristics that are tuned for specific workload characteristics.

Llumnix's insight is that all of these goals can be expressed as biased load reporting: make the instance appear more loaded than it physically is in proportion to the urgency of the need to free space on that instance. A head-of-line queuing request inflates the instance's reported load by its memory demand, triggering load-balancing migration that frees space—de-fragmentation without an explicit de-fragmentation policy. A high-priority request inflates the instance's reported load by a fixed headroom, triggering migration of normal-priority requests away—isolation without explicit resource reservation. A terminating instance gets an infinite virtual load, triggering complete evacuation—draining without a separate evacuation mechanism.

This is conceptually novel in the scheduling literature for LLM serving, and draws a deep parallel to how operating systems handle priority: a Unix nice value is not a separate scheduling dimension but a bias on the CPU scheduler's notion of how deserving a process is of time. Llumnix extends this idea to memory load: instead of scheduling with explicit multi-dimensional objectives (balance memory, reduce fragmentation, enforce priority), it schedules with a single objective (balance virtual memory load) and encodes all other objectives as biases on the load signal. The elegance is that the conflict between objectives is resolved implicitly by the load-balancing policy's own optimization: when de-fragmentation and load balancing conflict, the virtual usage inflation makes de-fragmentation look like load imbalance, so the load balancer's "fix" to the imbalance naturally de-fragments while preserving acceptable load distribution.

The empirical validation of this abstraction's power is that the same scheduling policy (greedy load balancing on freeness), with no scenario-specific logic, produces qualitatively correct behavior across all four scenarios: it spreads requests under normal load to reduce preemptions (Figure 11, preemption loss reduction), it packs requests when queuing occurs to reduce fragmentation (Figure 12, 92% fragmentation reduction), it isolates high-priority requests (Figure 13, 1.5× acceleration with minimal normal-request degradation), and it drains instances for scaling (Figure 15, 36% cost savings). The fact that a single policy parameterized only by the virtual usage rules achieves all of these is strong evidence that the abstraction captures something fundamental about the structure of the scheduling problem, rather than being a coincidental fit to the evaluated workloads.

This abstraction also has generative power for future work: any new scheduling concern (e.g., deadlines, fairness across tenants, energy efficiency) can potentially be expressed as a new virtual usage rule, inheriting the existing load-balancing infrastructure rather than requiring a new scheduling mechanism. This mirrors how OS schedulers have evolved—new scheduling classes (real-time, deadline, proportional-share) were added by extending the priority calculation, not by replacing the core scheduler.

Innovation 3: Live Migration Downtime Decoupled from State Size via Append-Only Property Exploitation

While live migration of stateful services is a well-established technique (VM migration, database replica migration, DL training job migration in Gandiva), all prior approaches suffer from downtime that scales with the size of the mutable state. In VM live migration, the iterative pre-copy phase must recopy pages that were dirtied during the previous copy iteration; the downtime is bounded by the rate of dirtying, and pathological workloads with high write rates can prevent convergence. In Gandiva's DL training migration, the model weights are the state, and migration involves checkpointing at a mini-batch boundary—downtime scales with model size (tens of GB for large models) and is acceptable because training jobs run for hours or days.

The paper's technical contribution here is the recognition that LLM inference state has a structural property—append-only immutability of historical KV cache—that eliminates the fundamental tension between state size and migration downtime. Because previously computed KV cache entries are never modified by subsequent iterations, they can be copied without any risk of the source mutating them during the copy. This means the working set requiring synchronization is bounded to exactly one iteration's output (the KV cache generated since the last copy stage completed), regardless of total sequence length. The downtime is therefore constant with respect to sequence length, unlike VM migration (where downtime depends on dirtying rate, which often correlates with VM memory size) or Gandiva migration (where downtime scales with model size and does not benefit from append-only properties).

This is not merely an engineering optimization—it is a structural property exploitation that makes migration viable for a workload class where it would otherwise be impractical. At 8k tokens, LLaMA-30B recompute migration takes 3.5 seconds (~54 decode steps of stall), blocking copy takes ~3 seconds, and Llumnix migration takes ~20–30 ms (less than one decode step). If migration downtime scaled linearly with sequence length—as it does for all prior stateful migration approaches applied to this domain—then as models move toward 128k or 1M token contexts, migration would become prohibitively expensive, and the entire rescheduling vision would collapse. The constant-downtime property means that migration cost is independent of the very trend (longer contexts) that makes rescheduling increasingly necessary (longer contexts mean larger per-request memory, more severe fragmentation, and higher preemption cost).

The evidence is Figure 10: downtime at 256 tokens is ~20 ms, at 8k tokens is still ~20 ms, while blocking copy grows from ~100 ms to ~3,200 ms and recompute from ~100 ms to ~3,500 ms. The migration completes in exactly two stages for all tested lengths, confirming empirically that the GPU's KV cache copy bandwidth exceeds the decode token generation rate enough to always catch up in one round.

This insight has broader implications for the design of stateful serving systems beyond LLMs. Any iterative inference workload where intermediate state is append-only (or more generally, where state mutation is localized to a bounded recent window) can adopt this pipelined migration pattern. The paper identifies the append-only property of attention KV cache as the enabling characteristic; other transformer-based architectures that share this property (e.g., vision transformers with cached patch features, multi-modal models with cached cross-attention states) could potentially benefit from the same approach.

Innovation 4: De-fragmentation via Load Inflation Rather Than Explicit Bin-Packing

Memory fragmentation in LLM serving arises specifically from the prefill phase's requirement to allocate KV cache for all input tokens simultaneously—a large contiguous block allocation on a single instance—while the decode phase's block-by-block allocation is naturally fragmentation-free under PagedAttention. Traditional approaches to fragmentation in resource management involve explicit packing algorithms: the scheduler identifies fragmented resources, computes a re-packing plan (which requests to move where to create the largest contiguous free space), and executes that plan. This is computationally expensive (bin-packing is NP-hard, so heuristics like first-fit, best-fit, or worst-fit are used), requires detailed knowledge of free block layouts across instances, and must be re-evaluated as conditions change.

Llumnix's approach is radically simpler: a queuing request inflates the virtual load of its instance, and the load-balancing policy's natural response (migrating requests away) automatically creates the needed contiguous space without any explicit fragmentation analysis. The system never computes a packing plan, never inspects free block layouts, and never makes a decision explicitly labeled "de-fragment." Instead, it converts the fragmentation problem into an overload problem and reuses the existing load-balancing solution.

This is intellectually distinctive because it inverts the standard approach to the packing-spreading tradeoff. Rather than oscillating between a packing policy and a spreading policy (or trying to find some optimal intermediate point), Llumnix runs a single spreading policy (load balancing) and lets the load metric itself encode when local packing is needed. The head-of-line queuing request's virtual usage acts as a "fragmentation tax" on the instance: the longer a request waits due to fragmentation, the higher the virtual load, the more aggressively the load balancer moves other requests off—which directly reduces fragmentation. The system is self-correcting without explicit feedback control.

The evidence for the effectiveness of this approach is Figure 12: Llumnix reduces the proportion of fragmented cluster memory from an average of 7.9% (INFaaS++) to 0.7%—a 92% reduction—and achieves this with the same policy that also balances load. The de-fragmentation benefit is most pronounced in traces with long-tail length distributions (L-L, S-L in Figure 11), where fragmentation would otherwise cause severe queuing delays for long-input requests. INFaaS++ with load-aware dispatching still shows up to 10% fragmentation because dispatching alone cannot re-pack already-running requests; Llumnix's migration-based de-fragmentation recovers this wasted capacity.

This innovation also has a negative result embedded in its success: the fact that inflating head-of-line virtual usage to full demand immediately (rather than gradually) works well empirically (the paper notes this choice "favours reducing queuing delays") suggests that the cost of an unnecessary migration (moving a request when fragmentation wasn't actually severe) is low relative to the cost of queuing delay. This is a non-obvious finding about the relative costs of different scheduling actions in LLM serving, and it justifies the aggressive de-fragmentation stance.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses synthetic traces generated from statistical distributions and two public ChatGPT-4 conversation datasets, ShareGPT (GPT4) and BurstGPT (GPT4-Conversation). The synthetic traces use Poisson or Gamma distributions for request arrivals and power-law distributions for input/output lengths, with multiple configurations (Short, Medium, Long distributions; S-S, M-M, L-L, S-L, L-S pairings) to emulate both frequent short requests and rare long-tail sequences. Each trace contains 10,000 requests. Maximum sequence length is constrained to fit within GPU memory (13,616 tokens for LLaMA-7B on A10). The two real datasets capture realistic conversation workloads: ShareGPT has mean input length 306 tokens (P95 1,484) and mean output length 500 tokens (P95 988); BurstGPT has mean input length 830 tokens (P95 2,345) and mean output length 271 tokens (P95 669), as shown in Table 1.
Base model(s). Experiments use the LLaMA model family in two configurations: LLaMA-7B (runs on a single A10 GPU) and LLaMA-30B (runs on 4 A10 GPUs using tensor parallelism). Both use 16-bit precision. The paper notes that the underlying vLLM version supports the original LLaMA with a maximum sequence length of 2k, but argues that recent LLaMA variants supporting longer contexts (4k–256k tokens) have "mostly similar" architectures and inference performance, so the results are representative of broader model types. The models were chosen because they are popular open-weight models representative of the scale at which multi-instance serving is practical.
Metrics. The primary metrics are request latencies:
- End-to-end latency: total time from request arrival to the generation of the final token.
- Prefill latency: time from request arrival until the first output token is generated (including any queuing delay).
- Decode latency: the average latency per output token after the first token, computed as the total time from first token to last token divided by the number of generated tokens. The paper uses per-token decode latency rather than total decode latency to normalize for different output lengths.
- Preemption loss: the total extra time (queuing + recomputation) caused by preemption events during a request's lifetime, reported as a mean across all requests. All latency metrics are reported as mean and P99 (99th percentile) values. For auto-scaling experiments, resource cost is measured as the average number of active instances over the experiment duration.
Baselines. Three schedulers are compared against:
- Round-robin dispatching: a simple policy that distributes requests evenly across instances, described as "a typical behavior of production-grade serving systems" (DeepSpeed-MII, Ray Serve, TritonServer).
- INFaaS++: an optimized version of INFaaS (Romero et al., 2021) that uses load-balancing dispatching and load-aware auto-scaling. The paper improves it by focusing on GPU memory load (the dominant resource in LLM serving) and incorporating the memory demands of queuing requests into the per-instance load metric to reflect queue pressure.
- Llumnix-base: a priority-agnostic version of Llumnix that enables all features (migration, de-fragmentation, auto-scaling) but treats all requests as the same priority. This serves as a within-system ablation for the priority support.
All baselines use vLLM as the underlying inference engine, so the comparison isolates the scheduling policy differences rather than per-instance engine optimizations.
Generation budget / compute accounting. The paper does not measure a "generation budget" in the conventional sense (e.g., FLOPs or number of samples). Instead, the relevant resource is GPU memory and instance count. The "compute" being scheduled is the ongoing autoregressive generation of requests on GPU instances. The experiments fix hardware (16 A10 GPUs for performance experiments, with LLaMA-7B using one GPU per instance for 16 total instances) and control workload characteristics (request rate, arrival distribution, length distribution) to create varying load conditions. The key comparison unit is latency at a given request rate and hardware configuration. For auto-scaling experiments, instances are dynamically added or removed, and resource cost is measured as the average number of instances used.
Cross-validation / statistical protocol. The paper does not describe cross-validation or statistical significance testing. Each trace is run once at each request rate or configuration point. The traces contain 10,000 requests each, providing a large sample size for percentile metrics, but the paper does not report confidence intervals or error bars on latency measurements. The difficulty of running large-scale GPU cluster experiments likely precludes multiple random seeds, but this means the variance of the reported metrics is unknown.

Main Quantitative Results

Migration Efficiency (Section 6.2)

The paper first establishes that the live migration mechanism itself is practical by measuring its cost. Using two instances each of LLaMA-7B (1 GPU) and LLaMA-30B (4 GPUs), with both instances running batches totaling 8k tokens of load, a single request is migrated between instances while measuring the migrating request's downtime and the overhead on co-located requests.

Migration downtime is constant and negligible. Across sequence lengths from 256 to 8k tokens, Llumnix migration downtime is approximately 20–30 ms for both model sizes, less than a single decode step's latency at typical batch configurations (Figure 10, left). By contrast:

Blocking copy downtime grows linearly with sequence length, reaching ~3,200 ms at 8k tokens for LLaMA-30B (~111× Llumnix).
Recompute downtime also grows linearly, reaching ~3,500 ms at 8k tokens for LLaMA-30B (~117× Llumnix).

The migration completes in exactly two stages for all tested sequence lengths, confirming that the GPU's KV cache copy speed exceeds the token generation rate sufficiently to always catch up in one round. This is the minimum possible number of stages, since every migration requires at least one initial copy and one final synchronization.

Migration overhead on co-located requests is negligible. Comparing per-step decode times on the source instance during active migration versus normal execution, the paper reports "up to 1% performance differences" for both LLaMA-7B and LLaMA-30B (Figure 10, right). Furthermore, across all serving experiments in later sections, the average fraction of time with ongoing migration per instance is "only roughly 10%," making the effective overhead (1% slowdown × 10% of time) essentially invisible. This demonstrates that the Gloo-based KV cache transfer on a separate CUDA stream successfully avoids interfering with the main inference computation.

Why this matters for scheduling: The near-zero cost of individual migrations justifies frequent, aggressive rescheduling. If migration were expensive (e.g., recompute with 3.5-second stalls), the scheduler would need to be very conservative about when to migrate, limiting the benefits of runtime adaptation. Constant-downtime migration means the scheduler can treat migration as a cheap action, enabling the continuous rebalancing that the virtual usage policy relies on.

Serving Performance: Real Datasets (Section 6.3, Figure 11, top two rows)

Using 16 LLaMA-7B instances with auto-scaling disabled, the paper evaluates Llumnix against round-robin and INFaaS++ on the ShareGPT and BurstGPT real conversation traces.

End-to-end latency. Llumnix outperforms both baselines in end-to-end request latency by up to 2× for mean and 2.9× for P99 across both traces (Figure 11, first column). Round-robin performs substantially worse than both INFaaS++ and Llumnix, demonstrating that even simple load-aware dispatching is valuable given the high variance in sequence lengths in real conversation data.

Prefill latency. This is where Llumnix shows the largest absolute gains:

Over round-robin: Llumnix improves prefill latency by up to 26.6× for mean and 34.4× for P99 (ShareGPT trace). This is because round-robin can dispatch new requests to already-overloaded instances, causing extremely long queuing delays, especially for requests with long inputs that require substantial memory allocation.
Over INFaaS++: Llumnix improves prefill latency by up to 2.2× for mean and 5.5× for P99 (BurstGPT trace). This improvement comes from de-fragmentation via migration: INFaaS++ dispatches to the least loaded instance but cannot re-pack already-running requests, so fragmentation still causes queuing delays. Llumnix's migration resolves this by freeing contiguous space on the instance where the queuing request is waiting.

Decode latency. Llumnix improves P99 decode latency by up to 2× over round-robin and 1.3× over INFaaS++. The P99 improvement is driven by reduced preemptions—Llumnix's load balancing via migration prevents the memory exhaustion that triggers preemption in the first place. The mean decode latency improvements are more modest because most requests are not preempted, but the tail (P99) is where preemption losses concentrate.

Preemption loss. Llumnix reduces preemption loss (mean extra time due to preemptions across all requests) by 84% on average compared to round-robin (Figure 11, rightmost column). Absolute values are small (sub-second) but represent the elimination of catastrophic service stalls for individual requests. The paper notes that preemption loss appears small when averaged over all requests because it is diluted by the many requests that are never preempted, but for the affected requests, the stall can be tens of seconds (as shown in Figure 3).

Takeaway: The real-dataset results establish that dispatch-time load balancing (INFaaS++) is necessary but not sufficient—migration provides additional gains by reacting to the actual memory usage growth of requests, which is unknown at dispatch time. The gains are most pronounced in the tail (P99), which is the metric most relevant to user experience for interactive applications.

Serving Performance: Generated Distributions (Section 6.3, Figure 11, bottom five rows)

To systematically explore workload characteristics, the paper evaluates Llumnix against INFaaS++ using five synthetic trace configurations: S-S (short inputs, short outputs), M-M (medium-medium), L-L (long-long), S-L (short inputs, long outputs), and L-S (long inputs, short outputs). These span different degrees of long-tail behavior, with the paper omitting round-robin because it "showed up to two orders of magnitude worse latencies" on the higher-variance synthetic traces.

End-to-end latency improvements. Llumnix outperforms INFaaS++ by up to 1.5× for mean end-to-end latency and 1.6× for P99 end-to-end latency across the five traces.

Prefill latency improvements. This is again the largest gain category:

Mean prefill latency: up to 7.7× improvement (L-S trace, high request rate).
P99 prefill latency: up to 14.8× improvement (M-M trace, high request rate).

The pattern across traces reveals that gains are larger when long inputs are common (L-L, S-L, L-S traces), because long inputs require more contiguous memory for prefill allocation, making fragmentation a more severe bottleneck. Traces with short inputs (S-S) show more modest prefill gains (still ~1.2–1.5×) because memory allocation is less constrained.

Decode latency improvements. P99 decode latency improves by up to 2× (M-M trace). The improvement comes from reduced preemptions, as confirmed by the preemption loss metric.

Preemption loss reduction. Across all generated-distribution experiments, Llumnix reduces mean preemption loss by an average of 70.4% relative to INFaaS++ (Figure 11, rightmost column). This translates to an average reduction of 1.3 seconds in end-to-end request latency caused by preemption events. In many cases (especially lower request rates), preemption loss is reduced to near zero.

Memory fragmentation case study (Figure 12). The paper zooms into a specific experiment (M-M trace, request rate 7.5) to quantify fragmentation. Over a busy 1,000-second period, INFaaS++ shows fragmentation proportions "often higher than 10%," with an average of 7.9% of cluster memory wasted by fragmentation (free blocks that could satisfy queuing requests on other instances but are scattered). Llumnix reduces this to an average of 0.7% — a 92% reduction — with fragmentation "often 0." This is direct evidence that the virtual usage mechanism successfully converts queuing pressure into migration actions that de-fragment memory, without any explicit fragmentation detection.

Difficulty-dependent pattern. While the paper does not explicitly bin requests by difficulty as in the LLM test-time compute paper, the per-trace analysis reveals an analogous pattern: the more "difficult" the serving scenario (higher variance, longer sequences, more long-tail), the larger the relative gain from Llumnix over INFaaS++. This suggests that migration-based rescheduling is most valuable precisely when traditional dispatching-based schedulers struggle most—under high variance and unpredictable workloads.

Support for Priorities (Section 6.4, Figure 13)

To evaluate priority support, the paper randomly selects 10% of requests in the S-S trace with Gamma arrivals (varying coefficients of variance, CV) and assigns them both high scheduling priority (earlier queuing) and high execution priority (lower interference). The comparison is between full Llumnix and Llumnix-base (which treats all requests identically).

High-priority request latency improvements. Llumnix improves mean request latency for high-priority requests by 1.2× to 1.5×, with larger gains at higher CV values (Figure 13, top row, first column). Higher CV means burstier arrivals and more load spikes, creating more situations where high-priority requests would otherwise suffer interference from co-located normal requests. Crucially, Llumnix maintains "similar latencies of high-priority requests" across CV values, demonstrating effective isolation even under fluctuating load.

The components of the improvement are:

Prefill latency: Mean improvements of 2.9× to 8.6×, P99 improvements of 3.6× to 10× (Figure 13, top row, third and fourth columns). The primary mechanism is scheduling priority: high-priority requests skip ahead in the queue, reducing their queuing delay.
Decode latency: Mean improvements of 1.2× to 1.5×, P99 improvements of 1.3× to 2.2× (Figure 13, top row, fifth and sixth columns). This comes from execution priority: the virtual usage headroom triggers migration of normal requests away from instances hosting high-priority requests, reducing the batch size and thus interference. The paper confirms this by showing similar gains in the raw decode computation time (the rightmost column, "Decode Execution Time"): 1.2× to 1.5× improvement, indicating that the decode speed itself is faster (less interference), not just that queuing is reduced.
Request-level mean latency: The overall 1.2–1.5× improvement for high-priority requests (Figure 13, top row, second column) combines the prefill and decode gains.

Normal request impact. Llumnix preserves "similar performance of the normal requests" (Figure 13, bottom row): mean request latency increases by at most 4.5%, mean prefill latency by at most 13%, and mean decode latency by at most 2%. This is impressive given that 10% of requests are being prioritized—the cost is not evenly distributed at 11% degradation but is efficiently absorbed by the migration mechanism, which spreads the displaced normal requests across multiple instances rather than concentrating them on a single penalized instance.

Why this works without static reservation. The key insight is that the headroom for high-priority requests (a fixed memory reservation of 1,600 tokens per high-priority request, empirically chosen for near-ideal decode speed) is enforced dynamically: when a high-priority request is present, the instance's virtual load increases by the headroom amount, making it appear overloaded and triggering migration of normal requests off. When no high-priority request is on the instance, no headroom is reserved, and the memory is available for normal requests. This avoids the utilization penalty of static reservation (where reserved but unused memory sits idle) while still providing isolation when needed. The experiment with varying CV demonstrates that this dynamic headroom adapts correctly: as load spikes from burstiness, headroom is maintained on instances with high-priority requests even as the overall cluster load increases.

Auto-Scaling (Section 6.5, Figures 14 and 15)

The auto-scaling experiments use L-L traces (long inputs and outputs, the most challenging configuration for memory management) with Poisson arrivals (varying request rates) and Gamma arrivals (varying CV at fixed request rate 2). Both Llumnix and INFaaS++ use the same scaling strategy (scale up when average freeness < 10, scale down when > 60, maximum 16 instances). The comparison isolates the scheduling efficiency, not the scaling policy itself.

Latency improvements with auto-scaling enabled. Across varying request rates (Poisson, Figure 14, top row), Llumnix "consistently achieves latency improvements" over INFaaS++:

P99 prefill latency: up to 12.2× improvement (mean prefill: up to ~2×).
P99 request latency: up to ~3× improvement.
P99 decode latency: up to ~2× improvement.

Across varying burstiness (Gamma, Figure 14, bottom row):

P99 prefill latency: up to 11× improvement.
Similar patterns for other latency metrics.

Resource cost savings. Llumnix saves up to 16% of resource cost (Poisson experiments) and 18% (Gamma experiments) compared to INFaaS++, measured as the average number of instances used (Figure 14, rightmost column). This comes from improved auto-scaling efficiency: Llumnix saturates new instances more quickly (via migration) and drains terminating instances faster, meaning the system spends less time in transitional states where instances are underutilized (or where the cluster has excess capacity that is not yet fully absorbing load).

Cost efficiency at equivalent latency (Figure 15). The paper's most striking auto-scaling result examines how many instances are needed to achieve a given P99 prefill latency target. By varying the scale-up threshold $t$ (with the threshold range $[t, t+50]$ ), Llumnix achieves roughly 5 seconds P99 prefill latency (the red dashed line in Figure 15) while using ~12 instances on average, compared to INFaaS++ requiring ~16 instances to achieve the same latency. This translates to a 36.49% cost saving while delivering "similar" tail latencies. The cost saving combines two effects: (1) Llumnix reduces queuing delays through de-fragmentation, allowing it to achieve the target latency with fewer instances, and (2) Llumnix saturates/drains instances more efficiently, so the average utilization of each instance is higher.

Why auto-scaling efficiency matters. Traditional auto-scaling in LLM serving is slow because instances can only be drained by waiting for running requests to complete naturally—a request generating 500 output tokens at 50 ms per token takes 25 seconds to finish. During this drain period, the instance consumes resources (GPU memory, power) but is not accepting new work, lowering overall cluster utilization. Llumnix's migration-based draining evacuates all requests from a terminating instance in a fraction of that time (the time to migrate the requests, not the time for them to complete), sharply reducing the transitional period and thus increasing the effective utilization of the scaled-down cluster.

Scheduling Scalability (Section 6.6, Figure 16)

To validate the distributed scheduling architecture's scalability, the paper simulates 64 LLaMA-7B instances (using sleep-based timing from offline GPU measurements rather than real GPUs, since the cluster exceeds the testbed size) and compares Llumnix against a centralized scheduler baseline that extends vLLM's per-instance scheduler to manage all requests across all instances.

Centralized scheduler bottleneck. As request rate increases from 100 to 500 req/s, the centralized scheduler experiences growing scheduling stalls—time spent waiting for the scheduler to make decisions and communicate with instances—of up to 40 ms per iteration at the highest rate (Figure 16). This translates to a 1.7× slowdown in per-token decode latency relative to the computation-only time. The bottleneck is the communication between instances and the centralized scheduler, which must synchronize request status and scheduling decisions for every iteration.

Llumnix scalability. By contrast, Llumnix exhibits "near-zero scheduling stalls" even at 500 req/s. The two-level architecture achieves this by (1) parallelizing intra-instance scheduling decisions across the 64 llumlets, which operate asynchronously from the global scheduler, and (2) restricting communication to instance-level metrics (freeness values) rather than per-request updates. The global scheduler's workload scales with the number of instances (64), not the number of active requests (potentially thousands at 500 req/s with long-running generations).

This result is important for validating the paper's architectural choice: the distributed design is not just a theoretical clean separation of concerns but a necessary practical optimization for large-scale deployments. A single centralized scheduler tracking all requests would become the bottleneck well before cluster sizes reach practical cloud deployment scales.

Ablation Studies and Robustness Checks

Impact of real vs. synthetic workload distributions (Figure 11, all rows): The paper runs the same scheduler comparison across two real traces (ShareGPT, BurstGPT) and five synthetic traces (S-S, M-M, L-L, S-L, L-S), showing that the qualitative pattern (Llumnix > INFaaS++ ≫ round-robin) is consistent across all workload types. The magnitude of improvement varies: Llumnix's advantage over INFaaS++ is largest on traces with long-tail length distributions (L-L, L-S) and smallest on traces with only short sequences (S-S), which is expected because fragmentation and preemption are worse with longer sequences. This across-trace consistency supports the claim that the improvements are not artifacts of a particular workload distribution.

Migration overhead isolation (Section 6.2, Figure 10): The migration efficiency experiment directly measures both the downtime to the migrated request and the decode speed slowdown on co-located requests during active migration. The finding that overhead is ≤1% per-step and active migration occupies only ~10% of instance time removes migration cost as a confound in the scheduling experiments—the latency improvements can be attributed to better scheduling decisions, not to an unmeasured migration overhead that is amortized over the experiment duration.

Priority treatment without virtual usage (Llumnix-base comparison, Figure 13): The priority experiment's comparison against Llumnix-base isolates the effect of the virtual usage headroom mechanism. Without virtual usage (Llumnix-base), high-priority and normal requests receive identical treatment, and the high-priority requests suffer the same interference and queuing as normal requests. The degradation of high-priority performance in Llumnix-base at high CV (bursty load) demonstrates the need for explicit priority support, and the comparison shows that Llumnix's approach (virtual usage headroom + migration) provides this without static resource partitioning.

Scaling strategy controlled comparison (Section 6.5, Figures 14–15): Both Llumnix and INFaaS++ use the same auto-scaling strategy (same threshold range, same max instances), so the cost and latency differences are attributable to scheduling efficiency, not to different scaling aggressiveness. However, the paper does not provide an ablation that disables migration during auto-scaling (e.g., a version of Llumnix that scales instances but does not migrate to saturate/drain). Such an ablation would isolate the contribution of migration to the 36% cost savings figure more cleanly.

Virtual usage rule sensitivity (implicit in priority and de-fragmentation results): The paper's virtual usage rules are fixed (head-of-line queuing request = full demand; high-priority headroom = fixed profiled value) and are not swept. The only parametric variation explored is the scaling threshold range in Figure 15, which varies the aggressiveness of auto-scaling to show the cost-latency tradeoff. The virtual usage rules themselves are not ablated—we do not know, for example, how sensitive the results are to the choice of headroom size, or whether gradually inflating queuing request virtual usage (vs. immediate full demand) would change the load-balancing/de-fragmentation tradeoff.

Request selection for migration (implicit in priority results): When the llumlet selects which requests to migrate, it prefers lower-priority and shorter-sequence requests. The paper does not ablate this selection policy (e.g., longest-first, random, priority-agnostic). The priority experiments implicitly test the selection: high-priority requests are not migrated away (they are the ones being protected), and normal requests are the ones being migrated, consistent with the stated policy. However, the impact of the sequence-length preference is not explicitly quantified.

Fault tolerance (Section 5): The paper describes fault-tolerance mechanisms (scheduler bypass mode on global scheduler failure, migration abort on instance failure) but does not evaluate them experimentally. There are no experiments with injected failures (e.g., killing the global scheduler mid-experiment, killing an instance during migration) to measure recovery time or request loss. This is a notable gap, as the robustness claims are unvalidated.

Single engine backend (vLLM only): All experiments use vLLM as the inference engine. While the architecture is described as backend-agnostic, there is no evaluation with alternative engines (e.g., Orca, TensorRT-LLM) to demonstrate portability. The KV cache block size, memory allocation granularity, and batching behavior of vLLM's PagedAttention may interact with the migration mechanism in ways that differ from other engines—for instance, the block fusion optimization assumes small, non-contiguous blocks, which is specific to PagedAttention.

Critical Assessment

Claim 1: "Llumnix improves P99 first-token latency by up to 15× and P99 per-token generation latency by up to 2× over state-of-the-art baselines."

This claim is supported but requires careful contextualization. The 15× figure appears in the synthetic L-S trace comparison against INFaaS++ (Figure 11, L-S row, Prefill P99 column at the highest request rate), where INFaaS++ degrades severely due to fragmentation while Llumnix continues to perform well. However:

The comparison against the stronger baseline (INFaaS++) shows a range of 1.2–14.8× improvement across traces, with the largest gains on traces specifically designed to stress fragmentation (long inputs). On traces without long inputs (S-S), the improvement is ~1.5× for P99 prefill.
On real datasets (ShareGPT, BurstGPT), the P99 prefill improvement is up to 5.5×, not 15×. The 15× figure is the maximum across a sweep of synthetic traces, not a typical improvement.
The 2× P99 decode improvement is robust across multiple traces but is inherently limited because decode latency is dominated by computation time (each token requires a full forward pass), which migration cannot change; it can only prevent preemption stalls, which affect a minority of requests. The 2× upper bound reflects this ceiling.

The paper would be strengthened by reporting the geometric mean improvement across traces rather than only the maximum, to give a more representative picture of expected gains.

Claim 2: "Llumnix accelerates high-priority requests by up to 1.5× while preserving similar performance of normal requests."

This claim is supported with qualifications. Figure 13 shows that high-priority mean request latency improves by 1.2–1.5× across CV values, with minimal impact on normal requests (≤4.5% increase in mean request latency). However:

The experiment uses a fixed 10% high-priority proportion. The behavior at different proportions (e.g., 50% high-priority) is unexplored. If half the requests are high-priority, the virtual usage headroom might trigger excessive migration or leave insufficient capacity for adequate separation.
The priority scheme is binary (high vs. normal). Real-world priority requirements may require more granularity (e.g., three tiers in ChatGPT: free, Plus, Team). The paper claims generalization to "more priorities" but does not demonstrate it.
The decode computation time improvement (1.2–1.5×) suggests the headroom of 1,600 tokens empirically achieves near-ideal decode speed, but this is a single profiled value for LLaMA-7B on A10 GPUs. The headroom would need recalibration for different models, hardware, or precision settings, and the sensitivity to this choice is unexplored.

Claim 3: "Llumnix delivers up to 36% cost savings while achieving similar tail latencies."

This claim is supported by Figure 15, but the "similar tail latencies" is an eyeball judgment (the red dashed line at ~5 seconds P99 prefill latency). The paper does not specify what "similar" means quantitatively (e.g., within 10%, within confidence intervals). The 36.49% figure is precise, suggesting it was measured at a specific operating point, but it is unclear how sensitive this figure is to the chosen latency target. If the target were stricter (e.g., 3 seconds), the cost savings might shrink; if looser (e.g., 10 seconds), they might grow.

Additionally, the 36% savings is the maximum achieved by varying the scaling threshold; the paper does not report the savings at other latency targets or provide a Pareto frontier of cost vs. latency for Llumnix vs. INFaaS++. A more complete analysis would sweep latency targets and show the cost savings curve, as the paper does in Figure 15 for a single dimension but without explicitly labeling all tradeoff points.

Claim 4: "Runtime rescheduling unifies load balancing, de-fragmentation, and priority differentiation through a single virtual usage abstraction."

This is the paper's central architectural claim, and the evidence is best described as a consistency argument, not a direct validation. The four scenarios (load balancing, de-fragmentation, prioritization, auto-scaling) are tested separately: load balancing is tested via the general serving performance experiments (Figure 11), de-fragmentation via Figure 12, prioritization via Figure 13, and auto-scaling via Figures 14–15. All scenarios show improvement. However:

The scenarios are tested under different workload configurations (different traces, different request rates), making it difficult to assess whether the virtual usage rules correctly handle simultaneous activation of multiple scenarios. What happens when a high-priority request is on an instance with a head-of-line queuing request? Do the virtual usage inflations add constructively, or do they create over-inflation that triggers excessive migration? The paper never tests combined scenarios.
The virtual usage rules are hand-designed and not compared against any alternative abstraction (e.g., multi-objective optimization, rule-based priority with separate mechanisms). The claim that virtual usage "unifies" the scenarios is supported only by showing that it works for each scenario in isolation, not by showing that simpler decompositions of the problem would fail to achieve comparable results.
The paper does not compare against a variant of Llumnix that uses separate mechanisms for each scenario (e.g., an explicit de-fragmentation routine, explicit priority-based preemption, explicit draining protocol) but lacks virtual usage. Such a comparison would validate that unification is genuinely simpler or more efficient, not just that each mechanism works.

Missing evaluations that would strengthen the paper:

Simultaneous scenario testing. A trace with both priority-class requests and long-tail length distributions, under auto-scaling, would show whether the virtual usage rules compose correctly. This is the most important missing experiment for validating the unification claim.
Virtual usage rule sensitivity sweeps. Sweeping the headroom size for high-priority requests, the queuing request virtual usage inflation (gradual vs. immediate), and the scaling thresholds would characterize how sensitive the results are to these parameters and whether the fixed values used are near-optimal or merely adequate.
Failure injection experiments. Injecting global scheduler failures, instance failures during migration, and network partitions would validate the fault-tolerance mechanisms described in Section 5 but never tested.
Multi-model experiments. The paper envisions extending to multiple model types/variants (Section 7), but no experiments with multiple model versions (e.g., a base LLaMA-7B and a fine-tuned coding variant) are provided. This limits the generalizability claim.
Comparison against explicit bin-packing de-fragmentation. A scheduler that periodically runs a bin-packing rebalancing pass (identifying fragmented memory and issuing explicit move commands) versus Llumnix's implicit de-fragmentation via virtual usage would clarify whether the virtual usage approach is merely simpler or actually more effective.
Longer context lengths. The maximum tested sequence length is 8k tokens (blocking copy experiment). With models now supporting 32k–128k contexts, the constant-downtime migration property should be validated at these scales. At 128k, the KV cache is ~16× larger than at 8k; does the migration still complete in two stages, or does the increased data volume eventually require more stages?

Genuine weaknesses that are not addressed:

No variance reporting. All latency and cost figures are reported as point estimates without confidence intervals. With 10,000 requests per trace, the variance in P99 metrics could still be substantial (P99 is estimated from the tail 100 requests in a 10k trace). Without error bars or multiple random seeds, readers cannot assess whether differences between scheduler configurations are statistically significant or within noise.
Single hardware configuration. All experiments use A10 GPUs (24 GB), which are mid-range inference GPUs. The memory pressure, batching dynamics, and migration speed characteristics might differ on high-memory GPUs (A100 80 GB, H100 80 GB), where per-GPU capacity is higher and fragmentation might be less severe. Conversely, on even smaller edge GPUs, fragmentation might be more limiting. The paper does not discuss hardware sensitivity.
No analysis of migration churn or instability. The virtual usage mechanism creates feedback loops: inflating virtual usage triggers migration, which changes virtual usage, which might trigger reverse migration. The paper does not analyze whether the scheduling policy is stable (e.g., whether it converges or oscillates) or measure the rate of "unnecessary" migrations (requests migrated multiple times without net benefit). The migration overhead measurement (≤1% per-instance, 10% of time) suggests churn is not catastrophic, but this is an indirect inference rather than a direct measurement.
Auto-scaling evaluation limited to a single trace type (L-L). The auto-scaling experiments (Figures 14–15) use only the long-long length distribution. This is the most challenging case for memory management, but it does not characterize how auto-scaling performs under more typical mixed workloads. If fragmentation is less severe under shorter sequences, the relative advantage of Llumnix's auto-scaling might be smaller.
The "24% cost savings" in auto-scaling vs. "36% cost savings" at equivalent latency are different operating regimes. The 16–18% savings in Figure 14 (rightmost column) are at variable latency—Llumnix uses fewer instances AND achieves lower latency. The 36% savings in Figure 15 is at fixed latency—Llumnix matches INFaaS++'s latency with fewer instances. The paper does not clearly distinguish these regimes or explain why the savings percentage differs.

Conditionality of claims:

The paper's claims about Llumnix's benefits hold most strongly when:

Workloads exhibit high variance in sequence lengths (long-tail distributions), because this is when fragmentation and preemption are most severe, creating the largest opportunity for migration to help.
Cluster load is moderate rather than extreme (the paper's experiments target "nearly no queuing delays and preemptions for P50 requests, and queuing delays within a few tens of seconds for P99 requests"). Under very low load, there is little to gain from rescheduling because instances are rarely overloaded. Under very high load (all instances saturated), migration cannot help because there is no underloaded destination to migrate to—the only remedy is auto-scaling, which the paper does test.
Instances have heterogeneous load at fine time scales, which is the natural consequence of unpredictable output lengths. If output lengths were highly predictable, one-shot dispatching might suffice, and Llumnix's advantages would shrink.

The claims are weaker or untested when:

Workloads are homogeneous (all requests similar length) — the paper does not evaluate this regime, but the mechanism would likely still work with reduced benefit.
All requests require tight latency SLOs — Llumnix's priority support is tested with only 10% high-priority requests; it is unclear whether it can provide adequate isolation when a majority of requests are high-priority.
The system needs to handle multiple model versions simultaneously — the infrastructure supports it architecturally but the experiments do not test it.
Instances span heterogeneous hardware — all GPUs are identical A10s, and the freeness metric assumes uniform memory capacity and decode speed.

6. Limitations and Trade-offs

Limitation 1: Single Benchmark, Single Model Family, Single Hardware Configuration

The assumption or constraint. All experiments use the LLaMA model family (7B and 30B) on a single hardware configuration (NVIDIA A10 GPUs, 24 GB VRAM each, PCIe 4.0 interconnect, 64 Gb/s network). The paper states that "since the model architectures and inference performance of [recent LLaMA variants] are mostly similar to those of LLaMA, we believe that our results are representative of more model types and larger sequence length ranges from a systems perspective" (Section 6.1). This is an assertion, not a demonstrated fact.

The consequence. The migration mechanism's efficiency depends critically on the relationship between KV cache copy bandwidth and token generation speed. This relationship is hardware- and model-specific. On GPUs with higher memory bandwidth (A100 80 GB with 2 TB/s vs. A10 with 600 GB/s), KV cache copying might be proportionally faster than computation, making migration even cheaper—or the bottleneck might shift to network bandwidth between instances. On smaller edge GPUs with less memory bandwidth, the KV cache copy might not keep pace with token generation, requiring more migration stages and potentially higher downtime than the measured ~20–30 ms. More importantly, the fragmentation and preemption dynamics are functions of per-GPU memory capacity: a GPU with 80 GB can host many more concurrent requests than one with 24 GB, which changes the relative severity of fragmentation (more requests per GPU means more opportunity for internal fragmentation via batching but also more absolute free space to absorb bursts). The paper provides no sensitivity analysis across GPU tiers, models sizes beyond 7B/30B, or model architectures (e.g., encoder-decoder vs. decoder-only, models with different KV cache per token ratios).

What evidence exists in the paper. The migration efficiency experiment (Figure 10) compares LLaMA-7B and LLaMA-30B—two different model sizes but on the same GPU architecture, and the paper notes that "the downtime of migration is nearly constant with increasing sequence lengths (roughly 20–30 ms)" for both. This provides some evidence within the A10 hardware, but no cross-hardware comparison. All serving experiments use LLaMA-7B exclusively (the 30B is only used for the migration efficiency microbenchmark). The paper does not report any experiments on alternative model families (e.g., Falcon, Mistral, GPT-derived architectures) or alternative GPU hardware.

Mitigation status. The paper does not attempt to address this limitation experimentally. It acknowledges the model scope only in passing and asserts generalizability without evidence. Future work would need to validate the approach on higher-end GPUs (A100, H100) with larger memory capacity, lower-end edge GPUs (T4, L4) where memory pressure is more acute, and model architectures with different KV cache characteristics (e.g., multi-query attention vs. multi-head attention, which changes the KV cache size per token). Given that KV cache size per token varies across architectures and precision settings (e.g., 8-bit KV cache quantization), the constant-downtime property may be sensitive to these factors in ways the paper does not characterize.

Limitation 2: All Scenarios Tested in Isolation—No Validation of Virtual Usage Under Combined Objectives

The assumption or constraint. The paper's central architectural claim is that virtual usage "unifies" four distinct scheduling goals (load balancing, de-fragmentation, prioritization, auto-scaling) under a single load-balancing policy. However, the evaluation tests each scenario in isolation with separate experiments: serving performance (load balancing + de-fragmentation) in Figure 11, priorities in Figure 13, and auto-scaling in Figures 14–15. The paper never runs an experiment where multiple virtual usage inflations are simultaneously active—for instance, a trace with priority-class requests, long-tail length distributions, AND auto-scaling enabled.

The consequence. Virtual usage works by inflating an instance's reported load to trigger migration. Multiple simultaneous inflations—a high-priority request adding headroom, a head-of-line queuing request adding its full demand, and a terminating instance adding infinite load—could interact in ways that the current evaluation cannot reveal:

Over-inflation and excessive migration. If an instance hosts a high-priority request (virtual usage inflated by headroom) AND has a long-input queuing request (virtual usage inflated by its full demand), the total virtual usage could far exceed physical memory, triggering aggressive migration that pushes requests off the instance even when the physical memory situation does not warrant it. This would increase migration churn without corresponding benefits.
Priority inversion under fragmentation pressure. The de-fragmentation rule (head-of-line queuing request = full demand) is designed to aggressively free space. The prioritization rule adds headroom to high-priority running requests. If a queuing high-priority request arrives at an instance that already hosts high-priority running requests, the queuing request's virtual usage inflation PLUS the running requests' headroom inflation might make the instance appear so overloaded that the load-balancer migrates the running high-priority requests away—the opposite of the intended isolation behavior.
Auto-scaling oscillation with priority workloads. The auto-scaling policy uses average freeness computed from normal-priority virtual usages only. But if high-priority requests with inflated virtual usage constitute a significant fraction of load, the auto-scaling controller's view of cluster load diverges from the physical reality, potentially scaling too aggressively or too conservatively.

What evidence exists in the paper. None. The paper never reports a combined-scenario experiment. The closest is the priority experiment (Figure 13), which uses the S-S trace (short sequences) with Gamma arrivals—a workload that minimizes fragmentation, so the de-fragmentation virtual usage rule is rarely activated. The auto-scaling experiments (Figures 14–15) use the L-L trace only, without priority classes.

Mitigation status. The paper does not acknowledge this as a limitation or propose experiments to address it. The Section 4.4.2 claim that virtual usage "unif[ies] these different, sometimes conflicting goals into a simple load metric" is supported only by showing that each goal can be achieved separately, not that they compose correctly. A combined evaluation is the most important missing experiment for validating the paper's central architectural thesis, and its absence is a significant gap given the paper's claim to have solved the multi-objective scheduling problem.

Limitation 3: Difficulty Estimation Cost Is Unaccounted for in the Virtual Usage Framework

The assumption or constraint. The virtual usage abstraction replaces explicit scheduling logic (de-fragmentation algorithms, priority enforcement mechanisms, draining protocols) with a load-reporting bias. However, this requires that the llumlet correctly determine the demand of a head-of-line queuing request (the number of KV cache blocks it needs for prefill) and the appropriate headroom for high-priority requests. The paper handles these as follows:

Queuing request demand: The llumlet knows the input token count of the queuing request, so the demand is simply ceil(input_tokens / block_size) × num_layers blocks. This is straightforward to compute.
High-priority headroom: The paper states that "the headroom for high-priority requests is currently defined as that required to preserve the ideal decode speed (i.e., no visible interference), which is obtained through profiling" (Section 4.4.2). The specific value (1,600 tokens of target memory load for LLaMA-7B on A10) comes from offline profiling (Figure 4). The profiling involves measuring decode step latency as a function of total batched tokens (sequence length × batch size) and determining the inflection point where interference becomes visible.

The consequence. The headroom profiling is model-specific, hardware-specific, and workload-specific in subtle ways:

It must be re-profiled for each model-GPU combination. Deploying LLaMA-7B on an A100 (vs. A10) would require re-profiling because the compute-to-memory bandwidth ratio differs, changing the batch size at which interference begins.
It assumes interference is a function of total batched tokens, not the distribution of sequence lengths. Figure 4 shows decode latency as a function of total batched tokens, but the interference might differ between a batch of many short requests versus few long requests, even at the same total token count, due to differences in attention computation patterns.
It is static. The headroom is profiled offline and remains fixed during serving. However, the "ideal decode speed" is not a universal constant—it depends on the level of interference that is acceptable. An application with tight latency SLOs might need more headroom (lower utilization), while a cost-sensitive batch application might tolerate more interference (higher utilization). The paper's binary priority model (high vs. normal) does not support tunable headroom as an SLO parameter.

What evidence exists in the paper. The paper implicitly addresses the correctness of the profiled headroom through the priority experiment (Figure 13), which shows that high-priority decode computation time improves by 1.2–1.5×, indicating that the 1,600-token headroom achieves near-ideal decode speed. But this is a single validation point. The paper does not sweep headroom values to show robustness, nor does it demonstrate that the profiling is transferable across hardware.

Mitigation status. The paper does not discuss the cost or methodology of headroom profiling as a limitation. It treats the profiled value as a fixed system parameter. For production deployments, this profiling step adds an operational burden: every model-hardware combination must be benchmarked before Llumnix can be deployed with priority support. The paper's claim of "simple" priority differentiation via virtual usage elides this profiling cost.

Limitation 4: No Analysis of Migration Stability, Oscillation, or "Unnecessary" Migration Churn

The assumption or constraint. The virtual usage mechanism creates closed-loop feedback: instances report inflated load → global scheduler triggers migration → load changes → virtual usage changes → migration may reverse. The paper's architecture provides hysteresis (instances are marked as source/destination only when freeness exceeds thresholds, and the marking is cleared when freeness returns to normal), but there is no analysis of whether this feedback is stable under realistic workload dynamics.

The consequence. Three potential failure modes are unexplored:

Oscillation. If migration is too aggressive (low thresholds), requests might "ping-pong" between instances: migration from instance A to B increases B's load enough to trigger migration back to A. While the freeness threshold provides some damping, the paper does not measure the rate of "unnecessary" migrations (requests migrated multiple times without net benefit) or characterize whether Llumnix converges to a stable load distribution or perpetually rebalances.
Short-sequence migration bias. The llumlet prefers migrating shorter-sequence requests (Section 4.4.3) because they are cheaper to migrate and complete faster. In a workload with a mix of short and long requests, this could lead to a situation where short requests are repeatedly migrated to balance load while long requests remain stationary and accumulate on certain instances. This might create a form of load imbalance that the freeness metric cannot express (since long requests' memory consumption grows over time, their eventual impact is not captured by their current virtual usage).
Migration during request bursts. Under bursty arrivals (high CV Gamma distributions, as tested in Figures 13–14), many requests might be dispatched to or migrated to the least-loaded instance simultaneously. If the burst is large enough, the destination instance could become overloaded before the next freeness report, triggering reverse migration. The paper's periodic migration evaluation (every reporting interval) might not react quickly enough to prevent transient overload during bursts.

What evidence exists in the paper. The migration overhead measurement (Section 6.2) provides indirect evidence that churn is not catastrophic: the average fraction of time an instance spends with active migration is "only roughly 10%," and the per-step decode overhead during migration is ≤1%, for an effective overhead of ≤0.1%. This suggests that migration frequency is moderate. However, this is an average—it does not characterize worst-case migration rates during load spikes or whether some requests are migrated many times while others are migrated zero times.

The priority experiment (Figure 13) provides some evidence that the feedback loop is at least directionally correct: high-priority requests receive protection (they are not migrated away, and co-located normal requests are), and normal request performance degrades only slightly. But this is a one-directional test (migration triggered by priority, not reversed by subsequent load changes).

Mitigation status. The paper does not analyze migration stability or quantify migration churn. The freeness thresholds provide hysteresis, but whether the current threshold values (which are not specified for migration—only the auto-scaling threshold range [10, 60] is reported) provide adequate damping is unknown. Future work should include metrics like "fraction of requests migrated more than K times" and "standard deviation of per-instance freeness over time" to characterize convergence.

Limitation 5: Fault Tolerance Mechanisms Are Described but Not Evaluated

The assumption or constraint. Section 5 describes two fault-tolerance mechanisms: (1) scheduler-bypassing mode when the global scheduler fails—frontends dispatch requests directly to instances using simple rules and migration is disabled; and (2) migration abort when an instance fails—the handshake protocol detects the failure via timeout and cleans up reserved blocks. The paper states that "these failed actors will be automatically restarted by Ray, after which the service could go back to normal state."

The consequence. The paper makes implicit claims about service availability ("Llumnix provides fault tolerance for each component to ensure high service availability") but provides no experimental validation of this claim. Unanswered questions include:

Recovery time after global scheduler failure. When the scheduler restarts, it has no state about current instance loads, running requests, or in-progress migrations. How long does it take to re-converge to a good load distribution? What happens to requests that were in the middle of migration when the scheduler failed—does the scheduler-bypassing mode route new requests independently, potentially creating load imbalance that the restarted scheduler must then fix?
Impact of scheduler-bypassing mode on latency. When the global scheduler is unavailable and frontends dispatch directly (using "simple rules" that are not specified further), the scheduling quality degrades—especially de-fragmentation and priority differentiation, which depend on virtual usage and migration. The paper does not measure the latency degradation during the bypass period, which could be substantial under high load.
Frequency and impact of instance failures. The paper does not report the failure rate of instances or llumlets in its experiments. Without this data, the claim of "high service availability" is ungrounded. In a production deployment with hundreds of instances, even a low per-instance failure rate could mean that some instance is failing at any given time.
Request loss during instance failure. When an instance fails, "the requests running on it will be aborted" (Section 5). The paper does not quantify the fraction of requests that would be lost under realistic failure rates or whether the system provides any mechanism for clients to retry failed requests.

What evidence exists in the paper. None. There are no experiments with injected failures of any component. All serving experiments (Figures 11–15) are run with all components healthy throughout.

Mitigation status. The paper does not acknowledge this as a gap. The fault-tolerance mechanisms are described architecturally but are entirely unvalidated. For a systems paper targeting production LLM serving (and claiming applicability to "people's daily lives" in the abstract), this is a significant omission. Even a simple experiment—killing the global scheduler mid-experiment, measuring the latency spike during the bypass period, and showing recovery time—would substantially strengthen the paper's deployment claims.

Limitation 6: The 36% Cost Savings Figure Overstates Practical Savings—Difficulty Estimation and Migration Overhead Are Externalized

The assumption or constraint. The 36% cost savings claim (Figure 15) compares Llumnix against INFaaS++ at equivalent P99 prefill latency (~5 seconds) and finds that Llumnix achieves this with ~12 instances on average versus ~16 for INFaaS++—a 36.49% reduction in instance-hours. However, this accounting assumes that:

Llumnix's migration mechanism has zero additional infrastructure cost (no additional CPU/memory for llumlets, no additional network bandwidth consumption).
The migration overhead on decode latency (≤0.1% effective, as noted above) does not materially affect the latency target being matched.
The auto-scaling policy's decision to add or remove instances is instantaneous and costless.

The consequence. In a practical deployment, these costs do not vanish:

Llumlet and global scheduler overhead. The global scheduler and per-instance llumlets consume CPU and memory resources on the cluster. For a 16-GPU deployment, these are negligible. At scale (hundreds or thousands of GPUs), the global scheduler's single-actor design could become a CPU bottleneck (the scalability stress test in Figure 16 only tests up to 64 instances). The paper does not report the CPU/memory footprint of the scheduling infrastructure.
Network bandwidth consumption during migration. Each migration transfers the entire KV cache of a request between instances. For long-context requests (thousands of tokens, each generating MBs of KV cache), this consumes inter-instance network bandwidth. In the evaluated setup (4 VMs, 64 Gb/s each), this bandwidth is abundant relative to the migration rate (~10% of instance time with active migration). In a shared cloud environment where network bandwidth is metered or contended, this cost is non-zero.
Auto-scaling cold start latency. The paper's auto-scaling evaluation assumes that newly launched instances become available immediately. In practice, launching a GPU VM with model weights loaded takes minutes (model loading, CUDA initialization, KV cache block pool allocation). During this cold start period, the cluster operates with fewer effective instances, and latency may spike. The paper's auto-scaling experiments (Figures 14–15) do not model cold start delays.

What evidence exists in the paper. The paper provides partial mitigation for these concerns: the migration overhead is quantified as ≤0.1% (Section 6.2), the scheduling stress test (Figure 16) shows scalability to 64 instances without stalls, and the auto-scaling experiments (Figures 14–15) implicitly include migration costs because they measure end-to-end latency. However, the paper never provides a holistic TCO (total cost of ownership) analysis that includes all infrastructure costs, nor does it characterize the sensitivity of the 36% savings figure to realistic cold start delays or network bandwidth constraints.

Mitigation status. The paper does not address these as limitations. The 36.49% figure is presented as a headline result without caveats about externalized costs. A more accurate comparison would: (1) account for the CPU/memory resources consumed by Llumnix's scheduling components; (2) model realistic VM cold start times (2–5 minutes) in the auto-scaling simulation; (3) measure the actual network bandwidth consumed by migration and subtract its cost (if metered); and (4) report cost savings at different latency targets to show the Pareto frontier rather than a single operating point. Without these, the 36% figure should be understood as an upper bound on achievable savings in an idealized deployment.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a fundamentally new scheduling primitive—runtime request migration across instances—into LLM serving, which has been dominated by one-shot dispatching since its inception. The shift is not an incremental improvement to existing load-balancing heuristics but a reconceptualization of the scheduling problem itself: from a stateless dispatching optimization (assign arriving requests to instances based on current load) to a continuous, stateful resource management problem (reposition in-flight requests as conditions evolve). This is analogous to the transformation that preemptive multitasking brought to operating systems, where processes became mobile across cores rather than being pinned at creation time. The paper's central analogy—"LLM serving should take inspiration from OS process management"—is not rhetorical but architectural: it identifies that the same reasons OSes needed context switching (dynamic working sets, multi-tenancy, differentiated priorities, load imbalance) now apply to LLM inference, and that the same solution primitive (migration) is both necessary and newly tractable.

The magnitude of this shift should be understood as a new capability rather than a better heuristic. Prior to this work, a multi-instance LLM deployment facing memory fragmentation had no recourse: the fragmented free space was simply stranded, and queuing requests waited despite abundant cluster-wide capacity. A deployment needing to prioritize certain requests had to resort to static over-provisioning—dedicating separate instances or GPUs to high-priority traffic, paying the full cost of reserved capacity. A deployment scaling down had to wait for natural request completions, which for long-running generations (hundreds of output tokens) could take tens of seconds per draining instance. Llumnix provides a single mechanism—live migration—that addresses all three scenarios without additional infrastructure, and a single policy abstraction—virtual usage—that expresses all three as variations of load imbalance. This is not a point solution for any one scheduling pathology; it is a platform capability that changes what schedulers can express.

The paper also reconciles a latent contradiction in prior LLM serving work. Inference engines like vLLM and Orca optimize for maximum per-instance throughput by packing requests aggressively into large batches—the "packing" side of the classic packing-spreading tradeoff. Cluster-level schedulers like INFaaS and AlpaServe lean toward spreading requests across instances for load balancing and isolation. These two directions are individually well-motivated but mutually opposing: a per-instance engine incentivized to pack will resist the cluster scheduler's attempts to spread, and vice versa. The paper identifies this tension as a fundamental structural problem—not an implementation deficiency that better heuristics could resolve—because the optimal packing level for a given request depends on its future memory growth, which is unknown at dispatch time. Llumnix breaks the impasse by making packing/spreading a runtime-adjustable parameter: the system can pack aggressively initially (for throughput), then spread via migration when memory pressure or fragmentation demands it (for latency), dynamically adapting to realized request lengths rather than pre-committing based on predictions. The 92% reduction in fragmentation (from 7.9% to 0.7%, Figure 12) while simultaneously reducing preemption loss by 70.4% (Figure 11) empirically demonstrates that this dynamic approach achieves what static packing-spreading tradeoffs could not.

The work also redirects research attention from per-instance optimizations (attention kernels, quantization, speculative decoding) toward cross-instance coordination. The LLM serving community has invested heavily in making individual GPUs more efficient; this paper shows that even with perfect per-instance engines, the cluster-level scheduling gap leaves substantial performance on the table (15× P99 prefill latency degradation under fragmentation, Figure 11). The implication is that future LLM serving research should treat cross-instance scheduling as a first-class design dimension, not an afterthought layered onto inference engines. Systems that co-design the inference engine's memory management with the cluster scheduler's migration capabilities—for instance, by exposing fine-grained KV cache block states to the scheduler or by aligning block sizes with migration granularity—could achieve tighter integration than Llumnix's backend-agnostic layering.

Perhaps most significantly, the paper establishes that LLM serving has crossed a complexity threshold where OS design principles become directly applicable. The success of the virtual usage abstraction—collapsing four different scheduling scenarios into a single load-balancing policy by simply biasing the load signal—is not an LLM-specific trick but a direct application of the OS principle that scheduling policy should be separated from scheduling mechanism. In Unix, the scheduler mechanism (the runqueue) is fixed, and policy variations (priority, fairness, real-time) are expressed through different nice values that bias the same mechanism. Llumnix applies the same pattern: the mechanism (load-balancing migration) is fixed, and policy variations (de-fragmentation, prioritization, draining) are expressed through different virtual usage rules that bias the same freeness metric. This is not an analogy but an architectural isomorphism, suggesting that the decades of OS scheduling research—on topics like proportional-share scheduling, hierarchical scheduling, deadline-based scheduling, and scheduler activations—may now be directly applicable to LLM serving with minimal translation. This opens a rich vein of prior art that the ML systems community has not yet tapped.

Follow-Up Research This Work Enables

1. Dynamic headroom calibration for interference-aware SLO enforcement. The paper uses a static headroom value (1,600 tokens for LLaMA-7B on A10) profiled offline to determine the memory reservation that preserves "near-ideal decode speed." This headroom is a single number for a single model-hardware pair, and it is binary (high priority gets headroom, normal does not). A natural extension is to learn a continuous interference model that predicts decode latency as a function of (batch size, total batched tokens, sequence length distribution, model architecture, GPU type) and then dynamically sets per-request headroom to achieve a specified SLO (e.g., "decode latency ≤ 50 ms per token with 99% probability"). The profiling data from Figure 4 provides the starting point for such a model. A strong follow-up would: (a) profile decode latency across a grid of (batch size × sequence length × total tokens) for 3-4 model families and 3-4 GPU tiers, (b) fit a predictor with uncertainty estimates, (c) integrate it into Llumnix's virtual usage calculation so that headroom is computed per-request and dynamically updated as batch composition changes, and (d) evaluate whether dynamic headroom achieves better utilization than static headroom while maintaining the same tail latency SLO. The key question is whether simple static headroom is "good enough" (as the paper's Figure 13 results suggest) or whether dynamic headroom can squeeze out meaningful additional cost savings.

2. Stability analysis and oscillation suppression in closed-loop migration control. The virtual usage mechanism creates a feedback loop: inflated load triggers migration → load balance shifts → virtual usage recalculated → migration may reverse. The paper provides no analysis of whether this loop converges to a stable distribution or oscillates, and the freeness thresholds provide only crude hysteresis. This is a control theory problem disguised as a scheduling policy. A strong follow-up would: (a) formally model the migration control loop as a dynamical system with state vector = [freeness per instance], control input = [migration pairing decisions], and disturbance = [workload arrivals and completions]; (b) analyze stability conditions—what combination of reporting intervals, freeness thresholds, and migration rates guarantees convergence to a balanced state without oscillation; (c) instrument Llumnix to measure "migration churn" as the fraction of migration volume that is later reversed (request A migrated from instance 1 to 2, then later back to 1); (d) compare the current threshold-based policy against a PID controller or a Lyapunov-optimized policy that explicitly minimizes both load imbalance and migration cost. The paper's measurement that average per-instance migration time is ~10% of total time suggests churn is moderate, but without measuring reverse migrations, we cannot distinguish between productive rebalancing and wasteful oscillation. This direction is practically important because churn directly eats into the net benefit of migration—if 20% of migrations are later reversed, the effective scheduling gain is lower than the headline latency improvements suggest.

3. Multi-model, multi-SLO scheduling with cross-model migration. The paper mentions (Section 7) that Llumnix could extend to scheduling across multiple model types or variants (e.g., a base LLaMA-7B and a fine-tuned coding variant, or models at different quantization levels). This is a substantially harder problem because the migration mechanism assumes identical model architecture on source and destination—the KV cache format must match. For cross-model migration, the KV cache from one model variant would need to be transformed or partially reused. This opens several research questions: (a) Can the KV cache from a base model be partially reused as a "warm start" for a fine-tuned variant, reducing the prefill cost? (b) For models at different quantization levels (e.g., FP16 vs. INT8), is there a lossy KV cache conversion that preserves enough information to avoid full recomputation? (c) How should the virtual usage abstraction represent instances with different capabilities (memory, compute, accuracy) when the "capacity" of an instance is not a single number? A strong follow-up would implement a prototype of cross-model migration with KV cache adaptation for a specific model family (e.g., LLaMA-7B base vs. LLaMA-7B-chat, which share architecture but differ in fine-tuning), measure the accuracy impact (does decoding from a partially-reused KV cache from a different model variant produce lower-quality outputs?), and compare the cost-latency tradeoff against the alternative of running separate instance pools per model variant. The practical motivation is strong: production LLM services often serve dozens of fine-tuned variants, and static partitioning wastes capacity when demand for different variants fluctuates.

4. Combined request migration with prefill-decode disaggregation. A recent trend in LLM serving is disaggregating prefill and decode phases across different instances (SplitWise, Sarathi-Serve, DistServe), where one set of instances handles the compute-heavy prefill and another handles the memory-heavy decode. Llumnix's migration mechanism currently assumes that a request's entire lifecycle (prefill and decode) occurs on a single instance. Integrating migration with disaggregation introduces new scheduling dimensions: when should a request be migrated from a prefill instance to a decode instance? Can a request that has started decode be migrated back to a prefill instance if its decode instance becomes overloaded? Does the append-only KV cache property still hold across the prefill-decode boundary, enabling the same pipelined migration? A strong follow-up would extend Llumnix to support heterogeneous instance pools (prefill-specialized with high compute, decode-specialized with high memory) and design a joint scheduling policy that decides (a) which instance type to dispatch to, (b) when to trigger the prefill→decode migration, and (c) whether to migrate decode requests between decode instances, all under the virtual usage abstraction (with separate capacity models for prefill and decode memory). The evaluation would compare against a baseline where prefill and decode instances are scheduled independently with no cross-pool migration, measuring both latency and utilization. The hypothesis is that coordinated scheduling with migration could absorb mismatches between prefill and decode load more gracefully than fixed capacity allocation, which is analogous to how Llumnix absorbs fragmentation mismatches in the current homogeneous setting.

5. Stress-testing the virtual usage abstraction under combined, adversarial workloads. The paper's central architectural claim—that virtual usage unifies multiple scheduling objectives—is validated only for scenarios tested in isolation. The most important missing experiment is a combined stress test: a workload that simultaneously activates de-fragmentation (long-tail length distribution), prioritization (mixed priority classes), and auto-scaling (varying load levels), with bursty arrivals (high CV) to stress transient behavior. This experiment would reveal whether the virtual usage inflations compose additively or interact destructively. Concretely, a strong stress test would: (a) construct a trace with 3 priority classes, a length distribution with 10% of requests having 5× the mean input length, and a sinusoidal request rate that forces auto-scaling to repeatedly add and remove instances; (b) measure several diagnostic metrics beyond latency: migration churn (fraction of requests migrated >3 times), worst-case instance overload duration (time an instance spends with negative freeness despite migration being active), and SLO violation rate disaggregated by priority class; (c) vary the virtual usage rules (e.g., gradual vs. immediate inflation for queuing requests, additive vs. max-based composition of multiple inflations on the same instance) to identify which design choices are robust to adversarial workload combinations. This direction would refine the paper's contribution from "virtual usage works for each scenario" to "virtual usage is a compositional abstraction"—a much stronger claim that would justify its use as a scheduling foundation for future systems.

6. Fault injection and recovery time characterization for production readiness. The paper describes fault-tolerance mechanisms (scheduler bypass mode, migration abort on instance failure) but does not evaluate them experimentally. For Llumnix to be credibly deployable in production, three failure modes need systematic characterization: (a) Global scheduler failure and recovery. Kill the scheduler mid-experiment under moderate load (with ongoing migrations). Measure: time to detect failure (frontend timeout), latency degradation during bypass mode (since migration and virtual-usage-aware dispatching are disabled), and time to recover to pre-failure latency after the scheduler restarts and re-establishes the load distribution. (b) Instance failure during migration. Kill the source instance while a request is mid-migration. Measure: whether the destination correctly aborts and releases reserved blocks, whether the request is lost or can be retried, and the latency impact on other requests on the destination that had blocks reserved for the aborted migration. (c) Network partition. Simulate a connectivity loss between a pair of instances with active migration. Measure: whether the handshake timeout correctly aborts the migration without resource leaks, and whether the freeness reporting from the partitioned instances causes the global scheduler to make incorrect dispatching or migration decisions. A strong fault-tolerance evaluation would report, for each scenario, the 99th percentile latency spike, the fraction of requests lost (if any), and the time to recovery. These experiments are tedious but essential for a systems paper that claims "high service availability."

Practical Applications and Downstream Use Cases

1. Multi-tier LLM API services with differentiated pricing. The paper's priority support is directly applicable to commercial LLM API products that offer tiered service levels (e.g., free tier with best-effort latency, paid tier with SLO guarantees, premium tier with dedicated capacity). Llumnix's virtual usage mechanism enables this without static resource partitioning: high-priority (premium) requests are assigned headroom that dynamically reserves memory and reduces interference on whatever instances they happen to land on, while normal-priority requests fill the remaining capacity. The advantage over static partitioning (dedicating separate GPU pools per tier) is that headroom is only "paid for" when high-priority requests are actually present—during periods with no premium traffic, the full cluster capacity serves standard-tier requests, improving overall utilization. The paper's result that high-priority requests are accelerated by 1.5× with only 4.5% degradation to normal requests (Figure 13, mean request latency) translates directly to a service that can offer a "2× faster" premium tier at roughly 5% infrastructure overhead, rather than the 100% overhead of a dedicated premium GPU pool. For a service processing millions of requests per day, this utilization improvement could represent substantial cost savings while enabling new revenue from differentiated pricing.

2. Burst handling for interactive LLM applications. Many LLM-powered applications experience sharp load spikes—a viral chatbot, a live event summarization service, a coding assistant during a hackathon. During a burst, dispatching-based schedulers face a dilemma: spread new requests across all instances for load balancing (which risks fragmentation as many short-lived requests scatter) or pack them onto a subset of instances (which risks local overload and preemptions). Llumnix's migration-based approach sidesteps this dilemma: requests can be dispatched aggressively (even to seemingly overloaded instances) to minimize initial queuing, then migrated to rebalance as actual memory usage materializes. This is particularly valuable for applications where prefill latency (time-to-first-token) is the critical user experience metric—the 15× P99 prefill latency improvement under fragmentation (Figure 11, L-S trace) means that during a burst, Llumnix can deliver first tokens to users 10-15× faster than a dispatching-only scheduler, even if the cluster is operating near capacity. For a customer-facing chatbot, where users abandon requests that take more than a few seconds to start responding, this directly translates to higher engagement and retention.

3. Cost-efficient batch inference for LLM evaluation and data generation. Organizations that run large-scale batch inference—evaluating thousands of prompts against benchmark datasets, generating synthetic training data via LLMs, or scoring candidate outputs—typically provision a fixed GPU cluster and submit jobs as offline batch workloads. In this setting, utilization and throughput are the primary metrics, and tail latency is less critical (within reason). Llumnix's auto-scaling with efficient instance draining (Section 6.5) is directly applicable: when the batch workload completes, instances can be drained rapidly via migration (rather than waiting for the longest-running requests to finish naturally), allowing the cloud instances to be released sooner. The 36% cost savings at equivalent tail latency (Figure 15) was measured in an online serving context, but the underlying mechanism—faster draining and saturation—translates to batch settings as well. For a team spending $10,000/month on GPU instances for LLM evaluation, a 36% reduction through more efficient auto-scaling represents$ 3,600/month in savings, with no change to the model, prompts, or evaluation pipeline—purely a scheduling improvement.

4. Edge-cloud hybrid LLM deployments. A emerging deployment pattern for LLM applications places small models (or quantized variants) on edge devices for low-latency interactive use, with larger cloud-based models as fallback for complex queries. Llumnix's migration mechanism could enable a more fluid boundary between edge and cloud: a request might begin execution on an edge GPU, and if its memory consumption grows beyond the edge device's capacity (due to a long output), it could be live-migrated to a cloud instance—without recomputing its KV cache and without the user perceiving a stall (the constant ~20-30 ms downtime is imperceptible). Conversely, a cloud instance under heavy load could migrate short, latency-sensitive requests to edge instances. The paper's migration efficiency results (Figure 10) are directly applicable: migration downtime is constant regardless of sequence length, so even long-context requests (which are the most likely to overflow edge memory) can be migrated without service interruption. While the paper's experiments use homogeneous hardware within a single cluster, the mechanism itself is hardware-agnostic (KV cache blocks are transferred via Gloo, which operates over standard TCP/IP), making cross-datacenter migration feasible over WAN links, though latency would increase accordingly. This use case would require addressing new challenges (heterogeneous GPU memory capacity, different KV cache formats across model variants, authentication/security for cross-boundary data transfer), but the core live migration primitive is already demonstrated and provides the foundation.

(Conditional) When to Prefer This Method

The paper positions Llumnix against existing dispatching-based schedulers (round-robin, INFaaS++, AlpaServe-style load-aware dispatching) and argues that migration-based rescheduling is strictly more capable—the question is not whether to prefer Llumnix but when the additional complexity of enabling migration is worth the benefit. Based on the paper's experimental results, the decision hinges on three workload characteristics:

Prefer Llumnix (enable migration) when: (1) the workload exhibits high variance in input and/or output sequence lengths—especially long-tail distributions where a minority of requests are much longer than the median (because these create the fragmentation and preemption pathologies that migration resolves; the L-L and L-S traces show the largest gains in Figure 11); (2) tail latency (P99) matters for user experience or SLO compliance—because migration's largest improvements are in the tail (up to 15× P99 prefill, 2× P99 decode), not the median; (3) the deployment serves requests with differentiated priorities or SLOs—because without migration, priority isolation requires static resource reservation, which has a direct utilization cost. In these regimes, the paper's evidence suggests Llumnix can deliver 4-15× tail latency improvements and 16-36% cost savings (at iso-performance) with negligible migration overhead (~0.1% effective decode slowdown).
Stick with dispatching-only scheduling when: (1) the workload is highly homogeneous (all requests have similar, predictable sequence lengths)—in this regime, one-shot dispatching can achieve near-optimal load balance, and fragmentation is minimal because all requests have similar memory footprints; (2) the deployment has minimal tail latency sensitivity (e.g., pure offline batch processing where only throughput matters)—the marginal value of reducing P99 latency is low; (3) the infrastructure constraints prevent migration—for instance, if instances are on isolated networks with no inter-GPU communication path, or if the backend inference engine does not expose the block-level KV cache access that migration requires. The paper does not evaluate a homogeneous-workload regime, so the threshold for "how much variance justifies migration" is not precisely characterized, but the S-S trace results (Figure 11, third row) show that even with short, relatively homogeneous sequences, Llumnix still improves P99 prefill latency by 1.5×—suggesting the bar for migration to be beneficial may be fairly low.

The paper does not frame a tradeoff against alternative migration mechanisms (e.g., recompute-based migration, blocking-copy migration) because those are strictly dominated by live migration in performance (Figure 10). The only scenario where blocking-copy or recompute might be considered is if the deployment cannot tolerate the implementation complexity of a pipelined migration protocol with the handshake process—but given that Llumnix is available as open-source software, this is more a build-vs-buy decision than a technical tradeoff.