PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications

ArXiv: 2505.07203

🎯 Pitch

Storing KV caches across all layers is wasteful when LLM applications output just a single token—a fast-growing pattern in real-world discriminative tasks. By keeping only the final layer's KV cache and chunking exclusively non-attention layers, PrefillOnly cuts GPU memory pressure so sharply that it handles inputs 5× longer than standard engines without parallelization, while a JCT-aware scheduler that re-estimates waiting times before each decision spikes throughput by up to 4× with no latency penalty.

---

1. Executive Summary

This paper introduces PrefillOnly, the first LLM inference engine tailored for prefill-only workloads — an emerging class of LLM applications in discriminative tasks such as recommendation, credit verification, and data labeling where the model generates only a single output token per request. The system is evaluated across four hardware setups, three LLM models (Llama-3.1-8B, Qwen-32B, Llama-3.3-70B), and two application traces — post recommendation and credit verification — against baselines including PagedAttention, chunked prefill, tensor parallelism, and pipeline parallelism. PrefillOnly contributes hybrid prefilling (processing non-attention layers chunk-by-chunk while forwarding attention layers normally, reducing peak GPU memory by shrinking intermediate tensors from linear layers that are up to 14× larger than one-layer KV caches) and continuous JCT calibration (re-estimating each waiting request's job completion time before every scheduling decision to prioritize requests that can hit newly arrived prefix caches, with a fairness offset proportional to queueing time). The engine achieves up to 4× higher query-per-second without inflating average or P99 latency compared to baselines, and expands the maximum input length by up to 5× without requiring inference parallelization, establishing that discarding or offloading KV caches for suffix tokens — combined with chunking only non-attention layers — is sufficient to serve prefill-only workloads at substantially higher throughput than traditional engines, but only when the scheduling logic dynamically accounts for how prefix cache availability alters job completion times.

2. Context and Motivation

The Core Problem: LLM Inference Engines Are Designed for Generative Workloads, Not Discriminative Ones

The fundamental problem this paper addresses is a mismatch between how LLM inference engines are engineered and how an increasingly important class of applications actually uses them. Existing inference engines — vLLM, Orca, Sarathi-Serve, and their contemporaries — are architected around a core assumption: that each request will generate an arbitrarily long sequence of output tokens. This assumption is baked into every layer of their design, from memory management (storing full KV caches across all layers to accelerate future decoding steps) to scheduling logic (treating job completion time as fundamentally unpredictable because output length can vary).

But as the paper documents, a substantial and growing fraction of LLM use in production systems doesn't follow this generative pattern at all. Instead, LLMs are increasingly deployed as discriminative classifiers and rankers — replacing traditional deep learning pipelines in recommendation, credit verification, data labeling, and similar tasks. In these applications, the LLM generates exactly one output token per request: a "Yes" or "No," a label, a preference score. The paper calls these prefill-only workloads because the LLM engine only needs to execute the prefilling phase — the initial forward pass that processes the full input and produces the first token — without any of the subsequent decoding steps that dominate traditional LLM inference.

This mismatch creates a genuine technical gap. Engines optimized for multi-token generation carry unnecessary overhead when serving single-token requests: they allocate and retain GPU memory for KV caches that will never be reused, they employ scheduling policies that assume job lengths are unknown, and they rely on throughput optimization strategies (like large-batch decoding) that don't apply to compute-bound prefill operations. The paper's core claim is that by re-architecting the inference engine to exploit the specific properties of prefill-only workloads, one can achieve substantial throughput and latency improvements that are invisible to general-purpose engines.

Why This Problem Matters: The Shift from Generative to Discriminative LLM Use

The paper's motivation is not purely a systems optimization exercise — it reflects a genuine structural shift in how industry deploys LLMs. The authors cite concrete evidence: Meta's 360Brew (Firooz et al., 2025) demonstrates that a single LLM can serve over 30 tasks across 8 different domains for recommendation and ranking, and multiple recent publications show LLMs matching or exceeding production models in credit verification (Feng et al., 2023; Son et al., 2023) and data labeling (He et al., 2023; Lan et al., 2024; Zhang et al., 2023). These are not speculative use cases — they represent production pipelines where LLMs are replacing teams of engineers who previously had to co-optimize data preprocessing, feature engineering, and model tuning in domain-specific pipelines.

This shift has two important properties that make the systems problem urgent:

The volume is enormous. The paper notes that recommendation workloads can easily reach tens of thousands of queries per second, requiring hundreds or thousands of GPUs to serve. At this scale, even modest per-request efficiency improvements translate to massive cost savings. A $4\times$ throughput improvement on a deployment of 1,000 GPUs effectively saves 750 GPUs' worth of compute cost.

The input characteristics are qualitatively different. In these discriminative applications, the inputs tend to be very long — often tens of thousands of tokens. A recommendation request might include months of user browsing history; a credit verification request might include ten months of transaction records (40,000–60,000 tokens, as the paper's dataset shows). This creates a specific memory pressure: the KV cache size scales linearly with input length, and at these lengths, it can easily exhaust GPU memory. For example, the paper measures that the maximum input length on an NVIDIA A100 40GB GPU with a Qwen-32B model (FP8-quantized) is only 11,000 tokens under standard memory management — far short of what these applications need.

Sharing GPUs with generative workloads is impractical. The paper makes an important architectural point here: prefill-only and traditional generative workloads should not share GPU resources. When they do, inter-workload interference significantly degrades the performance of generative use cases, because decoding jobs (which are memory-bandwidth-bound) get batched with prefill jobs (which are compute-bound), disrupting the batching patterns that decoding throughput depends on. This means organizations running both types of workloads need dedicated GPU clusters for each, making the efficiency of the prefill-only cluster a first-order cost concern.

Where Existing Approaches Fall Short

The paper identifies three categories of existing solutions and explains why each fails to fully address the prefill-only opportunity. This is not just a list of shortcomings — the authors methodically show that the very mechanisms these approaches use to handle long inputs (their primary value proposition) degrade throughput in ways that are acceptable for generative workloads but fatally suboptimal for prefill-only workloads.

Chunked Prefill: The Performance Penalty

Chunked prefill, introduced in Sarathi-Serve (Agrawal et al., 2024), processes long inputs by splitting them into smaller chunks that fit in GPU memory, processing each chunk sequentially. This works — it allows handling longer inputs than monolithic prefilling — but at a cost. The attention kernels operate less efficiently on smaller chunks (the paper measures a 14% end-to-end throughput reduction when chunking a 20,000-token input with a chunk size of 512). More critically, chunked prefill only extends maximum input length by less than $2\times$ (Table 2: on A100, chunked prefill reaches 17,000 tokens vs. 11,000 for PagedAttention — a $1.55\times$ improvement). This is because chunked prefilling still requires storing the full KV cache of all previous chunks across all layers, so the memory savings are limited to whatever temporary buffers can be amortized across chunks. For credit verification workloads needing 40,000–60,000 token inputs, this is insufficient.

The deeper issue the paper identifies — and this is where the contribution becomes clearer — is that chunked prefill's approach to the attention layers is the bottleneck. By chunking everything uniformly, it forces the attention operation to run in a suboptimal regime without fully addressing the memory problem. The paper's hybrid prefilling (§4) specifically breaks this coupling: chunk the non-attention layers (which are linear and can be chunked without performance penalty) while running attention normally (maintaining full kernel efficiency), and then discard the attention layer's KV caches after use since they won't be needed for decoding. This is only possible because the workload is prefill-only — in a generative setting, those KV caches must persist for decoding.

Tensor Parallelism: Communication Overhead Kills Throughput

Tensor parallelism distributes model weights across GPUs, splitting each layer's computation so that each GPU holds a fraction of the parameters. It directly addresses the maximum input length problem — the paper shows that with tensor parallelism (degree 2), maximum input length expands to 77,000 tokens on A100 (Table 2), comfortably handling both workloads. However, tensor parallelism requires all-reduce communication between GPUs after every layer, and this communication time is pure overhead — GPUs sit idle during the synchronization. Even with NVLink acceleration, the paper shows (Figure 8) that tensor parallelism achieves lower throughput than PrefillOnly because this communication cost is fundamental to the approach.

The trade-off is particularly stark because prefill-only workloads are compute-bound rather than memory-bandwidth-bound. In traditional generative workloads, the decoding phase is memory-bound (reading model weights and KV caches dominates runtime), so batching many requests together amortizes the weight reads and makes communication overhead acceptable. But in prefill-only workloads, where the GPU's compute units are the bottleneck, any time spent on communication is time not spent computing — and that directly reduces throughput. Tensor parallelism is solving a memory problem (KV caches don't fit on one GPU) by introducing a communication problem that the workload is particularly sensitive to.

Pipeline Parallelism: Bubbles and Chunked Prefill Dependency

Pipeline parallelism partitions the model's layers across GPUs, with each GPU responsible for a contiguous segment of layers. The paper notes that pipeline parallelism can theoretically achieve the same latency-throughput trade-off as single-GPU inference scaled across multiple GPUs — but only when all requests have identical lengths. When request lengths vary (which they do in practice — the paper's post recommendation workload has user profile lengths varying from 11,000 to 17,000 tokens), pipeline bubbles emerge: some GPUs wait idle while others finish processing their assigned layers, because the pipeline's stages are unbalanced.

Existing implementations mitigate this by using chunked prefill to equalize chunk sizes across the pipeline stages, but this introduces the same attention kernel performance penalties discussed above. Table 2 shows pipeline parallelism achieves 38,000 tokens on A100 — better than chunked prefill alone, but still insufficient for the 40,000–60,000 token credit verification workload, and at reduced throughput due to the chunking penalty.

The Deeper Issue: JCT Uncertainty in Scheduling

Beyond the memory management problem, the paper identifies a second limitation in existing engines: their schedulers are JCT-agnostic. Standard LLM engines use first-come-first-serve (FCFS) scheduling because, in generative workloads, the job completion time is fundamentally unpredictable — the model might generate 10 tokens or 1,000, and there's no reliable way to know in advance. But in prefill-only workloads, JCT is deterministic and predictable: the output length is always 1, so the total work is proportional to the number of input tokens that need processing (minus any tokens that hit the prefix cache).

This opens up a classic systems optimization that existing engines cannot exploit: shortest-remaining-job-first (SRJF) scheduling. By processing shorter requests first, average latency decreases because long requests don't block the queue. But as the paper shows with an illustrative example (Figure 5), naively applying SRJF in a prefix-caching context backfires: it doesn't account for how JCT changes dynamically when new prefix caches become available. A request that initially looks long might become much shorter if a preceding request populates its prefix cache — but only if the scheduler re-evaluates priorities after each scheduling decision.

The paper's continuous JCT calibration (§6.3) addresses this by re-estimating every waiting request's JCT before each scheduling step, accounting for which prefix caches are currently available. This transforms the scheduling problem from a static optimization (sort requests by length at arrival time) to a dynamic one (re-sort after each completion to exploit newly available caches). The example in Figure 5 demonstrates this: with naive SRJF, request D (which shares a prefix with A but is long) gets scheduled fourth and misses A's cache because C evicted it; with continuous calibration, D gets prioritized second because the scheduler notices after A completes that D can now hit A's cache, yielding one additional cache hit and lower average latency.

How PrefillOnly Positions Itself

The paper positions PrefillOnly not as a general-purpose LLM engine, but as a specialized engine for a specific workload class that is growing rapidly. This is a deliberate design philosophy: rather than trying to make one engine efficient for all workloads (which forces compromises), build an engine that fully exploits the properties of its target workload.

This positioning is analogous to how database systems evolved specialized engines for OLAP vs. OLTP workloads, or how deep learning serving systems developed separate optimizations for throughput-oriented batch inference vs. latency-oriented online serving. The paper's opening argument is essentially: prefill-only workloads are sufficiently distinct from generative workloads, and sufficiently common in production, that they deserve their own inference engine architecture.

The paper draws a clear contrast in Figure 1: traditional LLM inference reuses KV caches across multiple decoding steps to amortize the prefilling cost. Prefill-only inference generates one token and stops — those KV caches will never be reused for decoding, so retaining them is pure waste. This single observation cascades into the paper's two main technical contributions:

1. Hybrid prefilling enables discarding KV caches (suffix discarding) without slowing down inference, because non-attention layers can be chunked independently while attention layers run at full efficiency — and the attention KV caches can be discarded immediately after the forward pass completes, since no decoding will follow.

2. Continuous JCT calibration enables latency-optimal scheduling in the presence of prefix caching, because the scheduler can precisely determine how long each request will take given current cache state, and can dynamically reprioritize when new caches become available.

These contributions are not independent: hybrid prefilling enables suffix KV cache discarding, which frees GPU memory for prefix caches; continuous JCT calibration ensures that freed memory is used optimally by prioritizing requests that can reuse existing caches. The paper also explicitly notes that hybrid prefilling does not only benefit prefill-only requests — it's a general technique for reducing peak memory during prefilling — but it is the enabler for the prefill-only-specific optimization of discarding suffix KV caches without performance penalty.

Relationship to Prior Work: What's Genuinely New

The paper situates itself in three research threads and clarifies what it inherits vs. what it contributes:

LLM inference engines (vLLM, Orca, Sarathi-Serve, DistServe): Existing engines optimize for the decode phase — continuous batching (Orca), paged memory management (vLLM), chunked prefill (Sarathi-Serve), and prefill-decode disaggregation (DistServe). PrefillOnly inherits the memory management abstractions (it's built on vLLM) and the concept of chunked processing, but it inverts the optimization target: instead of making decoding efficient, it makes single-pass prefilling efficient by eliminating the memory overhead of persistent KV caches.

LLM caching systems (prefix caching, KV cache compression, CacheBlend, CacheGen): These systems focus on improving cache hit rates and reducing cache size. PrefillOnly is complementary — it doesn't change how KV caches work, it changes which KV caches are retained and when they're evicted. The paper explicitly notes compatibility with KV cache compression and blending techniques.

Traditional deep learning serving systems: The idea of eliminating unnecessary intermediate tensors and using JCT-aware scheduling is not new — it's table stakes in traditional DL serving. However, traditional systems face different bottlenecks (convolution layers create memory pressure, not linear layers; JCT is roughly constant, not prefix-cache-dependent). PrefillOnly's contributions are specifically tailored to LLM architecture: hybrid prefilling exploits the fact that almost all non-attention layers in LLMs are linear (and thus chunkable without cross-chunk dependencies), and continuous JCT calibration exploits the prefix-caching property unique to autoregressive transformer inference.

The paper's edge over prior work is not any single technique in isolation, but the integration of memory management (hybrid prefilling + suffix discarding) and scheduling (continuous JCT calibration) into a unified engine that treats prefill-only workloads as a first-class citizen rather than a degenerate case of generative inference.

3. Technical Approach

3.1 Reader Orientation

PrefillOnly is a modified LLM inference engine — built on top of vLLM — that is specialized for serving requests where the model generates exactly one output token. The problem it solves is that standard LLM engines waste GPU memory and make suboptimal scheduling decisions because they are designed for multi-token generative workloads, while prefill-only workloads have fundamentally different properties: KV caches don't need to persist for decoding, and job completion time is deterministic. The solution has two interdependent halves: a hybrid prefilling mechanism that reduces peak GPU memory by chunking only the memory-intensive non-attention layers (enabling suffix KV cache discarding without performance penalty), and a continuous JCT calibration scheduler that re-estimates each waiting request's completion time before every scheduling decision to exploit dynamically changing prefix cache availability.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components operating in a processing pipeline:

1. HTTP API Server (OpenAI-compatible): Accepts incoming prefill-only requests, tokenizes them, and forwards them to the scheduler via ZeroMQ-based RPC. Returns the single-token probability score to the client after execution.

2. Scheduler Process with Continuous JCT Calibration: Maintains a waiting queue of tokenized requests. At each scheduling step, enumerates all waiting requests, re-computes each one's JCT based on current prefix cache state (number of cached vs. uncached tokens) and queueing time (with a fairness offset), selects the request with minimum adjusted JCT, and dispatches it to an executor. This runs as a single-step granularity loop — one request scheduled per step.

3. Hybrid Prefilling Executor: Executes the scheduled request's forward pass through the LLM. Processes non-attention layers (MLP blocks, linear projections) chunk-by-chunk to minimize peak memory, processes attention layers normally in a single pass to maintain kernel efficiency, and optionally discards or offloads the KV caches of suffix tokens after the forward pass completes.

4. Prefix Cache Store: Retains KV caches of prefix tokens in GPU memory (subject to available capacity) for reuse by subsequent requests sharing common prefixes. Suffix KV caches that cannot fit are either discarded or offloaded to CPU memory. The store's contents are queried by the scheduler during JCT calibration to determine which tokens of each waiting request would be cache hits.

At a offline profiling stage, the system forwards a fake request of user-specified maximum length through the LLM to measure peak GPU memory usage, then reserves the remaining GPU memory for the prefix cache store. During online operation, the flow is: request arrives → tokenized → enters waiting queue → scheduler picks minimum-adjusted-JCT request → executor processes via hybrid prefilling → suffix KV caches discarded or offloaded as needed → single-token probability score returned to client → scheduler re-calibrates JCTs for remaining waiting requests → next request dispatched.

3.3 Roadmap for the Deep Dive

• First, hybrid prefilling — the memory optimization technique that enables everything else. I'll explain why naively discarding KV caches doesn't work (intermediate tensors from linear layers dominate peak memory), how chunking non-attention layers while running attention normally resolves this, and the implementation details (torch.compile, output preallocation, in-place computation).

• Second, suffix KV cache discarding and offloading — how PrefillOnly leverages hybrid prefilling's single-pass property to selectively retain prefix caches while dropping suffix caches, and why this is superior to parallelization-based approaches for throughput.

• Third, continuous JCT calibration — the scheduling algorithm that dynamically re-estimates job completion times. I'll explain why naive shortest-remaining-job-first fails under prefix caching, how continuous recalibration captures dynamically available caches, the JCT proxy (cache-miss tokens), and the fairness mechanism.

• Fourth, the JCT estimation model — how PrefillOnly profiles and predicts request completion time, including the linear regression approach and the empirical justification for using cache-miss token count as a proxy.

• Fifth, the starvation prevention mechanism — the fairness offset parameter $\lambda$ and how it trades off average latency against P99 latency.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems paper whose core idea is that prefill-only workloads have two exploitable properties — KV caches don't need to persist for decoding, and job completion time is deterministic — and that building an inference engine around these properties yields substantial throughput and latency improvements over general-purpose engines that treat prefill-only as a degenerate case of generative inference.

---

Hybrid Prefilling: Reducing Peak GPU Memory Without Hurting Attention Kernel Performance

Hybrid prefilling is the memory optimization that makes suffix KV cache discarding practical. To understand why it's necessary, we need to first understand why the obvious approach — simply discarding KV caches after use since no decoding follows — doesn't work as well as expected.

The problem: intermediate tensors from linear layers dominate peak GPU memory. When an LLM processes an input during prefilling, the GPU memory allocator's peak usage comes not from the KV caches (which are relatively small per layer) but from the intermediate tensors allocated inside the MLP (multi-layer perceptron) modules. Figure 3a shows the GPU memory trace over time when prefilling a 32,768-token input through Llama-3.1-8B: periodic spikes correspond to the allocation of these intermediate tensors. Figure 4 quantifies this for the Llama-3.1-8B MLP module: with bfloat16 precision and 32,768 tokens, the input tensor to the MLP's upward projection is $32768 \times 4096$ elements, but the intermediate activation after the first projection is $32768 \times 28672$ elements — 14× larger than the one-layer KV cache size. The second intermediate tensor (after the SwiGLU gate and before the downward projection) is $32768 \times 14336$ elements — 7× larger than the one-layer KV cache. These tensors exist simultaneously in GPU memory during the MLP forward pass, and their combined footprint dwarfs the KV cache savings from discarding.

The paper notes that this ratio — intermediate tensors being much larger than per-layer KV caches — is a general property of modern LLM architectures, not specific to Llama models. The design rationale across model families is: to increase decoding throughput, you want to minimize KV cache size (so larger batches fit in GPU memory) while keeping the total parameter count high (for model quality). The solution is to inflate the MLP intermediate dimension, packing parameters into linear layers whose activations are temporary rather than persistent. This design choice creates exactly the memory pressure that makes naive KV cache discarding insufficient.

The partial solution: chunked prefilling and its limitation. Chunked prefilling (from Sarathi-Serve) addresses the intermediate tensor problem by splitting the input into smaller chunks and processing them sequentially through the entire model — including attention layers. For a chunk size of $C$ tokens, each intermediate tensor is $C \times D$ rather than $N \times D$ (where $N$ is total input length and $D$ is the hidden dimension), proportionally reducing peak memory. However, chunking the attention layers has two costs:

• Reduced kernel efficiency: Attention kernels are optimized for large sequence lengths; small chunks underutilize the GPU's parallel compute units. The paper measures a 14% end-to-end throughput reduction when chunking a 20,000-token input with chunk size 512.

• Limited memory reduction: Chunked prefilling still requires storing the full KV cache of all previous chunks across all layers, because the attention operation for each new chunk must attend to all preceding tokens. So the memory savings are limited to the intermediate tensors — the KV caches still grow linearly with total input length. This is why chunked prefill only extends maximum input length by less than 2× (Table 2: 17,000 vs. 11,000 tokens on A100 for Qwen-32B).

The hybrid prefilling insight: chunk non-attention layers, run attention normally. The key observation is that the non-attention layers in LLMs — MLP blocks, layer norms, projection layers — are all linear operations (or compositions of linear operations with element-wise nonlinearities). Linear layers have no cross-token dependencies: the computation for token $i$ depends only on token $i$'s hidden state, not on any other token. This means they can be trivially chunked — process a subset of tokens through the linear layers, produce partial outputs, and concatenate — without any communication between chunks and without any change to the mathematical result.

Attention layers, in contrast, have cross-token dependencies: the output for token $i$ depends on all other tokens (in a causal attention mask, all tokens $j \leq i$). Chunking attention is possible but requires storing intermediate KV caches and re-attending across chunks, which is exactly what degrades kernel performance.

Hybrid prefilling exploits this asymmetry: process non-attention layers chunk-by-chunk to reduce peak intermediate tensor memory, but process attention layers in a single forward pass over the full sequence to maintain kernel efficiency. After the attention forward pass completes, the KV caches for that layer can be discarded (or offloaded) immediately — they won't be needed again because there is no decoding phase.

How hybrid prefilling changes the memory trace. Figure 3 contrasts the GPU memory trace without and with hybrid prefilling. Without it (Figure 3a), the trace shows large periodic spikes where the MLP modules allocate their full intermediate tensors for all 32,768 tokens simultaneously. With hybrid prefilling (Figure 3b), these spikes are reduced by approximately 2 GB — the peak memory is lower because each chunk's intermediate tensors are allocated, used, and freed before the next chunk processes the same layer. The attention layers still allocate their full KV caches (since they run on the complete sequence), but these are much smaller than the MLP intermediate tensors and can be discarded immediately after the layer's forward pass completes.

Critically, hybrid prefilling does not change the output of the LLM at all — it's mathematically identical to a standard forward pass. Chunking linear operations produces exactly the same tensor as processing them monolithically because linearity guarantees $f(x_1 \oplus x_2) = f(x_1) \oplus f(x_2)$ (where $\oplus$ is concatenation). The attention layers run on the complete sequence, so their outputs are unchanged. No approximation or compression is involved.

Implementation: torch.compile and three optimizations. PrefillOnly implements hybrid prefilling using torch.compile, PyTorch 2's graph compilation framework. The approach works at the computation graph level rather than modifying the model code directly, which provides two benefits: it works across different model architectures without model-specific code changes, and it operates on the optimized graph that PyTorch's compiler produces (capturing fused operations, memory planning, etc.).

The implementation proceeds in three stages, with each stage addressing a specific inefficiency:

Stage 1 — Graph rewriting: The torch.compile-produced computation graph is traversed to identify consecutive linear operations — sequences of matrix multiplications, element-wise activations (like SwiGLU), and other operations that have no cross-token dependencies. These consecutive operations are grouped into a single "virtual layer." The virtual layer's forward pass is then modified to iterate over chunks of the input along the sequence dimension: for each chunk, compute the virtual layer's output for those tokens, and concatenate the chunk outputs at the end.

The reason for grouping consecutive linear operations rather than chunking them individually is to minimize the number of concatenation operations (which require memory allocation) and to amortize the Python interpreter overhead of the chunking loop. If every individual linear layer were chunked separately, the overhead of splitting, processing, and reassembling would accumulate across dozens of layers, potentially negating the memory savings.

Stage 2 — Output preallocation: The naive approach to chunked processing produces a list of output tensors (one per chunk) and then calls torch.cat to concatenate them, which allocates a new tensor of the full output size and copies each chunk into it. This doubles the peak GPU memory for the output tensor: the chunk tensors exist simultaneously with the concatenated output during the copy operation. To avoid this, PrefillOnly preallocates the full output tensor before processing any chunks, using shape information inferred from the computation graph. Each chunk's output is then written directly into the appropriate slice of this preallocated tensor (e.g., output[chunk_start:chunk_end, :] = chunk_result). This eliminates the temporary list of chunk tensors and the copy operation, reducing peak memory to just the preallocated output tensor plus one chunk's intermediate tensors at any time.

Stage 3 — In-place computation: PrefillOnly additionally reuses GPU memory when the input and output tensors have identical shapes. The key observation is that for many linear layers, the output tensor for chunk $i$ occupies the same relative position within the full output tensor as chunk $i$'s input occupies within the full input tensor. This means the output tensor slice can be written directly into the memory that previously held the input tensor slice — the input has been fully consumed by that point in the computation and is no longer needed. This further reduces peak memory by avoiding the simultaneous allocation of input and output tensors for the same chunk.

The paper evaluates the cumulative effect of these optimizations in Figure 10 (though this figure appears in the Evaluation section, the techniques are part of the technical approach): chunking alone provides a $7.9\times$ improvement in maximum input length, output preallocation adds a further improvement, and in-place computation adds yet more. The final result is that hybrid prefilling achieves over $8.7\times$ the maximum input length of vanilla prefilling on Qwen-2.5-32B with FP8 on an A100 GPU, without any throughput degradation — in contrast to chunked prefill, which trades throughput for memory.

Why hybrid prefilling enables suffix KV cache discarding. An important architectural point: hybrid prefilling ensures that the entire prefilling process completes in one LLM forward pass — the output is produced, the single token is sampled, and the request is done. This is in contrast to chunked prefill, which runs multiple forward passes (one per chunk) and therefore requires persisting KV caches between passes so that each chunk can attend to preceding chunks. Hybrid prefilling's single-pass property means that after the attention layer's forward pass, its KV caches have served their purpose — they were used for the current token's self-attention computation, and no future passes will need them. They can be safely discarded or offloaded to CPU memory. The paper explicitly states this enabling relationship: "hybrid prefilling is the enabler of this technique, as hybrid prefilling only prefills each request within one LLM inference, allowing one to potentially discard part of the KV cache without worrying about slowing down the inference" (Section 5.1).

A crucial distinction: hybrid prefilling enables discarding; it doesn't mandate it. The paper emphasizes that hybrid prefilling merely creates the possibility of not keeping all KV caches in GPU memory. The engine can choose among three strategies per request: keep the KV cache entirely in GPU (if memory is abundant and prefix caching is valuable), offload part to CPU (if memory is tight but the KV cache might be reused), or discard part (if memory is tight and reuse is unlikely). This flexibility means PrefillOnly can adapt its memory strategy based on available GPU memory and the prefix-caching characteristics of the workload, rather than being locked into a one-size-fits-all policy.

---

Suffix KV Cache Discarding and Offloading

Once hybrid prefilling ensures that the forward pass is self-contained, PrefillOnly can selectively retain or discard KV caches to balance prefix caching benefits against GPU memory constraints.

What gets retained vs. discarded. PrefillOnly retains the KV caches of prefix tokens — tokens at the beginning of the input that are shared across multiple requests (as identified by the prefix caching mechanism). It discards or offloads the KV caches of suffix tokens — tokens unique to a specific request that are unlikely to be reused by future requests. The intuition is that in many prefill-only applications, requests share long common prefixes. For example, in the post recommendation workload, all 50 requests for a given user share the same user profile and browsing history (11,000–17,000 tokens) as a prefix, differing only in the final 150-token article description. Retaining the prefix KV cache across all 50 requests eliminates the need to recompute the attention for those shared tokens each time. The suffix (article-specific) tokens are unique per request and have no reuse value, so their KV caches are discarded immediately.

Implementation via vLLM's sliding window abstraction. The paper implements suffix KV cache discarding by reusing vLLM's existing abstractions for sliding window attention — a mechanism originally designed to limit attention to a fixed-size window of recent tokens. By configuring the "window" to cover only the prefix portion, PrefillOnly achieves the desired retention behavior without modifying low-level GPU kernels. This design choice — reusing existing abstractions rather than writing custom CUDA kernels — is deliberate: it maintains compatibility across GPU architectures and avoids introducing hardware-specific code paths.

Why this is superior to parallelization for throughput. The paper argues that parallelization-based approaches to increasing maximum input length — tensor parallelism and pipeline parallelism — solve a memory problem by adding communication overhead. Tensor parallelism requires all-reduce operations after every layer, which consumes GPU time that could otherwise be spent on computation. In a compute-bound workload like prefill-only inference, this communication directly reduces throughput. Pipeline parallelism introduces pipeline bubbles when request lengths vary, which similarly wastes GPU compute cycles. PrefillOnly's approach — keep the inference on a single GPU and manage memory by discarding unnecessary KV caches — avoids both communication overhead and pipeline bubbles entirely, which is why it achieves higher throughput than either parallelization approach at equivalent maximum input lengths (as demonstrated in Figure 8 and Table 2).

The offloading alternative. The paper notes that the current implementation performs suffix KV cache discarding (the KV caches are simply freed), but acknowledges that this prevents future requests from potentially reusing the discarded computation — for instance, if two users happen to share part of their browsing history beyond the common system prompt. Offloading to CPU memory (via solutions like LMCache) would preserve the data for potential reuse while freeing GPU memory, and the paper flags this as future work. The core architecture — hybrid prefilling making it possible to not keep all KV caches in GPU during inference — is compatible with either strategy.

---

Continuous JCT Calibration: Dynamic Scheduling Under Prefix Caching

The second major technical contribution is the scheduling algorithm. The problem it addresses is subtle: while prefill-only workloads make JCT deterministic and predictable (since output length is always 1), the JCT is not static — it changes dynamically as prefix caches are populated and evicted.

Why naive SRJF fails under prefix caching. The shortest-remaining-job-first (SRJF) algorithm — always process the request with the lowest expected completion time — is optimal for minimizing average latency when JCTs are known and static. But in a prefix-caching context, JCTs are dynamic: when request A completes, it may leave behind KV caches in GPU memory. If request B shares a prefix with A, B's JCT suddenly drops (because part of its input is now cached). If request C (which does not share a prefix with A) is scheduled instead, C may evict A's cache before B gets a chance to use it, and the opportunity for a cache hit is lost.

The paper illustrates this with the concrete example in Figure 5, using four requests A, B, C, D:

• Length: $A < C < B < D$ (request A is shortest, D is longest)
• Shared prefixes: A and D share one prefix; B and C share another
• GPU cache capacity: can hold KV cache of only one request at a time

Under FIFO scheduling (order: A, B, C, D): B hits C's cache. Total cache hits: 1. Under naive SRJF (order: A, C, B, D): B still hits C's cache, but D (which could have hit A's cache) misses because C evicted A's KV cache. Total cache hits: 1. Under SRJF with continuous calibration (order: A, D, C, B): after A completes, the scheduler re-evaluates all waiting requests and discovers that D can now hit A's cache, making D's calibrated JCT lower than C's (despite D being longer overall, the cached portion eliminates work). D is scheduled second and gets a cache hit. C and B follow, with B hitting C's cache. Total cache hits: 2.

The additional cache hit comes from the scheduler's ability to re-evaluate priorities after each completion, noticing that a previously deprioritized long request has become short due to newly available cache. Naive SRJF, which sorts once at arrival time, misses this transient opportunity.

The continuous calibration algorithm. Algorithm 1 in the paper formalizes this. The scheduler runs in a loop, one request per step:

1. For each waiting request $r$ in the queue:
   • Compute $n_{\text{input}}$: the total number of input tokens in $r$
   • Compute $n_{\text{cached}}$: the number of tokens in $r$ that hit the current prefix cache (determined by querying which KV caches are currently resident in GPU memory)
   • Compute $T_{\text{queue}}$: the time request $r$ has been waiting in the queue
   • Compute a score: $\text{score} = \text{get\_jct}(n_{\text{input}}, n_{\text{cached}}) - \lambda \cdot T_{\text{queue}}$
2. Select the request with the minimum score.
3. Dispatch it to the executor.
4. After completion, return to step 1 (which now sees updated cache state and re-computes $n_{\text{cached}}$ for all remaining requests).

The key operational detail is when calibration happens: before every scheduling decision, not at request arrival time. This is what allows the scheduler to exploit transient cache states — the $n_{\text{cached}}$ value for each request is recomputed based on the current GPU cache contents at the moment of scheduling, which may be different from what was available 200ms ago when the previous request was dispatched.

---

The JCT Estimation Model

The get_jct function called in the scheduling algorithm needs to predict how long a request will take to execute given its total length and how many of its tokens are already cached. The paper takes an empirical, profiling-based approach rather than an analytical one.

Profiling procedure. Offline (before serving begins), PrefillOnly profiles the LLM by running synthetic requests across a grid of $(n_{\text{input}}, n_{\text{cached}})$ pairs, covering the full range of expected input lengths (up to the user-specified maximum) with a granularity of 1,000 tokens. For each pair, it measures the actual wall-clock time from request dispatch to completion, producing a lookup table of JCT values.

Linear regression model. Rather than storing and querying the full lookup table (which would be large and sparse), PrefillOnly fits a small linear regression model to the profiling data. The model takes $n_{\text{input}}$ and $n_{\text{cached}}$ as features and predicts JCT. This is compact (a few floating-point weights), fast to evaluate (a dot product), and smooths out measurement noise from the profiling runs.

Empirical proxy: cache-miss token count. The paper reports an important empirical finding: the number of cache-miss tokens — that is, $n_{\text{input}} - n_{\text{cached}}$ — is an excellent proxy for JCT. On the Qwen-32B model with FP8 quantization on an A100 GPU, the Pearson correlation coefficient between actual JCT and $n_{\text{input}} - n_{\text{cached}}$ is 0.987 (where 1.0 is perfect correlation). This means the total length matters much less than how much new computation (uncached tokens) is required — which is intuitively reasonable, since processing cached tokens is essentially free (they just need to be loaded from the KV cache, which is a memory operation that is fast relative to computing attention from scratch).

Because of this strong correlation, PrefillOnly uses $n_{\text{input}} - n_{\text{cached}}$ directly as the JCT proxy by default, rather than using the full linear regression model. This simplifies the scheduling score to:

$\text{score} = (n_{\text{input}} - n_{\text{cached}}) - \lambda \cdot T_{\text{queue}}$

where $n_{\text{input}} - n_{\text{cached}}$ is the number of tokens that must be computed from scratch (the effective work), $T_{\text{queue}}$ is the time the request has been waiting, and $\lambda$ is the fairness parameter.

Why this proxy works and what it assumes. The near-perfect correlation implies that the per-token computation cost for uncached tokens is approximately constant, regardless of total sequence length. This is a reasonable assumption for prefill-only workloads on compute-bound hardware: the GPU's compute units process each token through the same sequence of linear and attention operations, and the attention computation scales linearly with sequence length for the prefilling phase (each token attends to all preceding tokens, so total attention FLOPs scale as $O(N^2)$, but the per-token cost at a given position is $O(N)$ — in aggregate, the prefilling time is well-approximated by a linear function of token count for the sequence lengths considered). If this assumption broke down — for instance, if attention became a superlinear bottleneck at extremely long sequences — the proxy would underestimate JCT for long requests, and the full regression model would be needed.

---

Starvation Prevention: The Fairness Offset

A pure SRJF scheduler (always pick the request with the fewest uncached tokens) risks starvation: long requests may be indefinitely postponed if there's a continuous stream of short requests arriving. PrefillOnly prevents this by incorporating a fairness offset into the scheduling score.

Mechanism. The score used for scheduling is:

$\text{score} = (n_{\text{input}} - n_{\text{cached}}) - \lambda \cdot T_{\text{queue}}$

where $\lambda$ is a tunable hyperparameter. As a request waits in the queue ($T_{\text{queue}}$ increases), its score decreases (because $-\lambda \cdot T_{\text{queue}}$ is subtracted), making it progressively more likely to be selected — regardless of its length.

What $\lambda$ controls. The parameter $\lambda$ sets the trade-off between average latency and worst-case latency:

• $\lambda = 0$: pure SRJF — always pick the request with fewest uncached tokens. Best average latency, but unbounded worst-case latency for long requests.
• $\lambda \to \infty$: approximate FIFO — queueing time dominates the score, so requests are processed roughly in arrival order. Bounded worst-case latency, but higher average latency.
• Intermediate $\lambda$: a blend where short requests are preferred, but long requests eventually get prioritized if they've waited long enough.

The paper uses $\lambda = 500$ as the default. Figure 11 shows how the latency distribution shifts with $\lambda$: higher values improve P99 latency (the tail of the distribution) at the cost of inflating the mean, while $\lambda = 0$ achieves the lowest mean latency but with a long tail.

Operational interpretation. The fairness offset $\lambda \cdot T_{\text{queue}}$ can be understood as converting waiting time into "effective tokens": each second of waiting is equivalent to reducing the request's effective length by $\lambda$ tokens. With $\lambda = 500$, a request that has waited 2 seconds gets a score reduction of 1,000 — meaning it will be scheduled ahead of a newly arrived request with up to 1,000 more uncached tokens. This provides a principled way to balance the competing objectives of minimizing average latency (favor short requests) and providing fairness (don't starve long requests).

---

Why Not Batching Prefill-Only Requests?

The paper makes a deliberate architectural choice to schedule prefill-only requests one at a time rather than batching them together. This runs counter to the dominant paradigm in LLM inference, where continuous batching is the primary throughput optimization, so it requires justification.

The compute-bound vs. memory-bound distinction. In traditional generative LLM workloads, the decoding phase is memory-bandwidth-bound: the GPU's compute units are underutilized because they spend most of their time waiting for model weights and KV caches to be read from GPU memory. Batching multiple requests together amortizes these memory reads — reading the model weights once and applying them to many tokens simultaneously increases throughput with only a marginal increase in latency per request. The decoding batch size is typically limited only by GPU memory capacity.

Prefill-only workloads are different: the prefilling phase is compute-bound. The attention operation's quadratic complexity and the MLP's large matrix multiplications keep the GPU's compute units fully utilized even for a single request. Adding more requests to a batch doesn't increase throughput — it just increases the latency for all requests in the batch, because they now share compute resources that were already saturated. As the paper states: "batching prefill-only requests increases the average latency compared to processing the requests one by one, and does not improve the throughput."

Implications for the architecture. This compute-bound property simplifies the scheduler: there's no need for the complex continuous batching logic that dominates generative LLM engines (managing a dynamic batch of requests at different stages of decoding, adding and removing requests as they start and finish). Instead, PrefillOnly can use a simple sequential dispatch model: pick one request, execute it to completion, pick the next. This sequential execution also makes the JCT calibration more predictable — there's no interference between concurrently executing requests that could perturb the profiled JCT values.

Potential nuance. The paper doesn't explore whether there are edge cases where batching might help — for example, if multiple very short requests arrive simultaneously, the overhead of dispatching them individually (CUDA kernel launch overhead, Python scheduling loop overhead) might exceed the benefit of sequential processing. However, given that prefill-only requests in the target applications are typically thousands to tens of thousands of tokens, the per-request computation dwarfs any dispatch overhead, and the compute-bound assumption is safe.

---

System-Wide Integration: The Profile Run and Memory Budgeting

Before any online serving, PrefillOnly performs a one-time profile run to determine how much GPU memory is available for prefix caching. This works as follows:

1. The user specifies the maximum request length (in tokens) that PrefillOnly needs to handle.
2. PrefillOnly constructs a synthetic "fake" request of exactly that length (e.g., padding tokens).
3. It forwards this fake request through the LLM using hybrid prefilling and measures the peak GPU memory usage during this forward pass — this is the memory required for the LLM inference itself (model weights, intermediate tensors from hybrid prefilling, KV caches for the attention forward pass).
4. The remaining GPU memory — total GPU memory minus peak inference usage — is allocated as the prefix cache store.

This memory budgeting is static: the division between inference memory and cache memory is fixed for the lifetime of the serving process. The prefix cache store then manages this fixed budget using an eviction policy (inherited from vLLM's prefix caching implementation) to decide which KV caches to retain when the budget is full.

Why this works. The peak memory during hybrid prefilling is deterministic for a given maximum request length — it doesn't depend on the actual content of the request, only on the tensor shapes. The profile run captures this deterministically. Since PrefillOnly processes requests sequentially (no batching), only one request's inference tensors are in GPU memory at a time, so the peak from the profile run is the true peak for any request at or below the maximum length.

Limitation. The static division doesn't adapt if the actual workload has requests shorter than the maximum — in that case, some of the "inference" memory budget is unused, and it could have been allocated to the prefix cache for better cache hit rates. The paper doesn't address this dynamic rebalancing opportunity.

---

Summary of Design Choices and Their Justifications

• Hybrid prefilling (chunk non-attention, full attention) over uniform chunked prefilling: maintains attention kernel efficiency (compute-bound workload is sensitive to this) while reducing peak memory from linear layers (where most memory pressure comes from). The asymmetry exploits the fact that non-attention layers have no cross-token dependencies.

• Suffix KV cache discarding over retaining all KV caches: for prefill-only workloads, suffix caches have no decoding reuse value and only consume GPU memory. Retaining them provides no benefit while limiting maximum input length.

• Suffix KV cache discarding over parallelization (tensor/pipeline): avoids all-reduce communication overhead and pipeline bubbles, both of which reduce throughput on compute-bound workloads. The paper explicitly shows this in Figure 8 and Table 2.

• Continuous JCT calibration over one-time JCT estimation at arrival: captures dynamic changes in JCT as prefix caches are populated and evicted by preceding requests. The paper's example (Figure 5) demonstrates an additional cache hit from this dynamic re-evaluation.

• Cache-miss token count as JCT proxy over full regression model: the empirical 0.987 Pearson correlation with actual JCT makes the simpler proxy sufficient. It's computationally cheaper (one subtraction per request per scheduling step) and doesn't require storing regression weights.

• Sequential request processing (no batching) over continuous batching: prefill-only workloads are compute-bound, so batching increases latency without improving throughput. Sequential processing simplifies the scheduler and makes JCT predictions more reliable.

• Fairness offset $\lambda \cdot T_{\text{queue}}$ in the scheduling score: prevents indefinite starvation of long requests under SRJF, providing a tunable trade-off between average and tail latency. The paper shows this works empirically in Figure 11.

• torch.compile-based implementation of hybrid prefilling: provides model-architecture-agnostic graph rewriting (works for Llama, Qwen, etc. without model-specific code) and benefits from PyTorch's compiler optimizations (kernel fusion, memory planning). The paper's 4,600 lines of Python are primarily scheduling logic and integration code — the core memory optimization is a compiler pass.

• Reuse of vLLM's sliding window abstraction for suffix discarding: avoids writing custom CUDA kernels, maintaining hardware portability across GPU architectures. The abstraction already exists in vLLM's codebase and provides the necessary functionality (limiting which KV caches are retained based on token position).

• Static memory budgeting via profile run over dynamic rebalancing: simpler to implement and sufficient when the maximum request length is known in advance. The paper acknowledges this as a potential area for improvement (dynamic rebalancing when actual requests are shorter than the maximum).

4. Key Insights and Innovations

Innovation 1: Prefill-Only as a First-Class Workload Category

The paper's most distinctive intellectual move is not a single optimization technique — it's the identification and characterization of prefill-only workloads as a distinct, economically significant category that deserves its own inference engine architecture rather than being treated as a degenerate case of generative inference. Before this work, the systems community implicitly assumed that LLM serving is synonymous with multi-token generation. Every major inference engine — vLLM (Kwon et al., 2023), Orca (Yu et al., 2022), Sarathi-Serve (Agrawal et al., 2024), DistServe (Zhong et al., 2024) — is architected around the decode phase, with continuous batching, paged KV cache management, and prefill-decode disaggregation all targeting the memory-bandwidth-bound regime of autoregressive token generation.

What makes this framing distinctive is that the authors don't just observe that some applications happen to generate one token — they argue that this is a structurally different workload with its own optimization landscape. The key conceptual shift is recognizing that when output length is fixed at 1, two properties invert the standard assumptions: (a) KV caches transition from essential assets (reused across many decoding steps) to disposable liabilities (used once in attention, then never needed again), and (b) job completion time transitions from fundamentally unpredictable (since output length varies) to deterministic and predictable (proportional to uncached input tokens). Neither property holds in generative settings, so existing engines are blind to both opportunities.

This reframing matters beyond the paper's specific techniques because it opens a design space that was previously invisible. The authors aren't just making existing engines more efficient for a corner case — they're arguing that prefill-only serving has its own architectural sweet spot (single-pass prefilling, selective KV cache retention, JCT-aware scheduling, sequential execution) that is qualitatively different from the decode-optimized sweet spot (large-batch decoding, full KV cache persistence, JCT-agnostic scheduling, continuous batching). This is analogous to how database systems evolved separate OLTP and OLAP engines rather than trying to make one engine serve both workloads optimally — a recognition that workload characteristics dictate architecture, not just parameter tuning.

The paper supports this framing with evidence that the problem is not niche: they cite concrete production deployments (Meta's 360Brew serving 30+ tasks across 8 domains, credit verification systems processing months of transaction history, data labeling pipelines) and quantify the scale (recommendation workloads at tens of thousands of queries per second requiring hundreds of GPUs). Table 2 demonstrates that existing engines cannot even fit representative prefill-only inputs (11,000-token maximum on A100 for Qwen-32B with standard memory management vs. 40,000–60,000 token credit verification requests). This isn't a performance gap — it's a capability gap: standard engines literally cannot run the workload without parallelization workarounds that sacrifice throughput.

Innovation 2: Asymmetric Chunking as a Memory-Throughput Decoupling Mechanism

Hybrid prefilling — chunking non-attention layers while running attention layers at full sequence length — represents a conceptual departure from how the field has approached the memory-throughput tension in LLM prefilling. Prior work treated this as a uniform trade-off: chunk everything (Sarathi-Serve's chunked prefill, which reduces peak memory at the cost of attention kernel efficiency) or chunk nothing (vanilla prefilling, which maintains kernel efficiency at the cost of high peak memory from intermediate tensors). Both approaches assume that all layers must be treated identically.

The paper's insight is that this uniformity is unnecessary because attention and non-attention layers have fundamentally different chunkability properties. Linear layers (MLP blocks, projections, layer norms) have no cross-token dependencies — they compute f(x_i) independently for each token i. Attention layers have cross-token dependencies — computing the output for token i requires attending to all tokens j ≤ i. Chunking attention is expensive (it requires storing intermediate KV caches across chunks and re-attending, degrading kernel performance), but chunking linear layers is essentially free (it's mathematically equivalent to processing all tokens at once, with no communication between chunks).

The intellectual contribution is not the technique itself (chunking linear operations is trivial) but the diagnosis of where peak memory actually comes from and the corresponding insight that decoupling the chunking strategy across layer types breaks the trade-off that prior work accepted as fundamental. Figure 4 quantifies this diagnosis: intermediate tensors in the Llama-3.1-8B MLP module are 7–14× larger than one-layer KV caches. This ratio is not an accident — it's a direct consequence of how modern LLMs are architected (inflated MLP intermediate dimensions compensate for reduced KV cache sizes, which are minimized to increase decoding batch size). The memory pressure in prefilling comes primarily from the layers that are easiest to chunk without penalty, and the layers that are expensive to chunk (attention) contribute relatively little to peak memory. Prior work missed this asymmetry because it didn't analyze which tensors dominate the memory trace — it treated peak memory as an aggregate property of the model rather than a layer-by-layer phenomenon.

The empirical evidence in Figure 10 validates the significance: hybrid prefilling achieves 7.9× the maximum input length of chunked prefill on a Qwen-2.5-32B model with FP8 on A100, without the throughput degradation that chunked prefill imposes (14% in the paper's measurement). This is a fundamental improvement, not an incremental one: it demonstrates that the attention-nonattention asymmetry is large enough that exploiting it changes the qualitative scaling behavior (maximum input length grows by nearly an order of magnitude rather than the ~2× from chunked prefill).

This innovation also has a systemic implication that the paper touches on but could explore further: since this memory-throughput decoupling is enabled by the MLP-to-attention memory ratio, and this ratio is a design choice in LLM architecture, future models could be co-designed with inference engines to exaggerate this ratio (even larger MLP intermediate dimensions, even smaller KV caches) to make hybrid prefilling even more effective. The paper doesn't make this argument explicitly, but it's a natural consequence of the analysis.

Innovation 3: Dynamic JCT Re-Estimation as a Cache-Aware Scheduling Primitive

The paper's scheduling contribution — continuous JCT calibration — addresses a problem that the systems community didn't recognize existed: that prefix caching makes job completion times dynamic rather than static, and that scheduling algorithms must account for this dynamism to exploit caching opportunities. Before this work, JCT-aware scheduling for LLM inference wasn't a meaningful concept because output length uncertainty made JCT unpredictable. By establishing that prefill-only workloads make JCT predictable, the paper opens the scheduling design space — but then immediately identifies that naive predictable-JCT scheduling (SRJF based on arrival-time estimates) fails under prefix caching.

The conceptual contribution is the recognition that the scheduler itself can create caching opportunities that it must then exploit. When request A completes, it populates the prefix cache with KV caches that reduce the JCT of request D (if D shares A's prefix). But this reduction is transient — if request C is scheduled next and evicts A's cache, D's JCT reverts to its original value before D gets a chance to benefit. The scheduler's decision about what to run next determines whether this caching opportunity is realized or wasted. This is a form of scheduling-induced cache lifecycle management that doesn't arise in simpler caching contexts (where the cache is populated by whatever the scheduler runs, and the scheduler just needs to pick good candidates — not actively re-evaluate them after each cache state change).

The Figure 5 example crystallizes this: three different scheduling policies (FIFO, naive SRJF, SRJF with continuous calibration) produce different cache hit counts (1, 1, 2) from the same workload and cache capacity. The additional cache hit comes entirely from the scheduler's willingness to re-evaluate priorities after each completion, detecting that a previously long request has become short due to newly available cache. This is not a performance optimization — it's a correctness property of the scheduling algorithm under dynamic cache state. Without continuous re-evaluation, SRJF makes suboptimal decisions because it's optimizing against stale JCT estimates.

What makes this a fundamental insight rather than an incremental refinement is that it changes the abstraction boundary between the scheduler and the cache. In traditional LLM engines, the scheduler selects requests and the cache passively provides hits or misses — they're loosely coupled. PrefillOnly's continuous calibration makes the scheduler cache-state-aware: it queries the current cache contents before every decision and uses that information to compute JCTs that reflect the true cost of executing each request given what's currently cached. This tight coupling between scheduling and caching is the architectural novelty, not the specific SRJF policy.

The empirical correlation finding (0.987 Pearson between actual JCT and cache-miss token count, making JCT predictable from cache state alone) validates that this coupling is practical: the scheduler can accurately estimate how cache state affects execution time, which is what makes cache-state-aware scheduling possible. Without this predictability, the scheduler would be guessing at JCT and the calibration wouldn't help.

Innovation 4: The Compute-Bound vs. Memory-Bound Distinction as a Batching Policy Driver

The paper's decision to not batch prefill-only requests — and its justification for this choice — represents a conceptual clarification that applies beyond prefill-only workloads. The field has internalized that continuous batching is the key to LLM serving throughput, to the point where "LLM inference engine" is almost synonymous with "dynamic batching system." Orca, vLLM, and their successors all optimize for the memory-bandwidth-bound regime where adding requests to a batch amortizes weight reads and improves throughput.

PrefillOnly's contribution here is to explicitly distinguish when batching helps vs. when it doesn't, based on whether the workload is memory-bandwidth-bound or compute-bound. In the decoding phase, the GPU's compute units are underutilized because they're waiting on memory — batching amortizes the memory reads and thus increases throughput. In the prefilling phase (and especially in prefill-only workloads where the entire job is prefilling), the GPU's compute units are saturated by the attention and MLP operations on a single request — batching doesn't increase total throughput because the compute resources are already fully utilized, and it does increase latency because requests now share those saturated resources.

This distinction is not new in computer architecture (compute-bound vs. memory-bound is a standard taxonomy), but it had not been applied to LLM inference engine design as a first-order architectural consideration. Prior work treated the prefill phase as an annoyance to be managed (chunked prefill, prefill-decode disaggregation) rather than recognizing that when prefill is the entire workload, the optimization landscape inverts. The specific implication — that sequential request processing is optimal for prefill-only workloads — runs counter to the continuous batching orthodoxy but follows directly from the compute-bound diagnosis.

The significance of this insight extends beyond the paper's immediate results. As LLM applications diversify, the assumption that all workloads are decode-dominated will break down in more contexts. Prefill-only is the extreme case, but there may be intermediate regimes (workloads with very short typical output lengths, or prefill-heavy workloads like long-document summarization) where the optimal batching strategy differs from the decode-optimized default. The paper provides a diagnostic framework (identify whether the workload is compute-bound or memory-bound, then choose batching accordingly) that generalizes beyond prefill-only.

The evidence for this insight is indirect but consistent: PrefillOnly's throughput advantage over parallelization baselines (Figures 6, 8, 9) shows that avoiding unnecessary work (all-reduce communication, pipeline bubbles, attention chunking overhead) matters more than batching optimization in this regime. If batching were beneficial, a system that batches (like tensor-parallel vLLM) would close the throughput gap despite its communication overhead, but it doesn't — confirming that compute resources are saturated without batching.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses two simulated datasets — a post recommendation workload and a credit verification workload — rather than existing LLM benchmarks. The post recommendation dataset simulates 20 users, each with a browsing history profile of 11,000–17,000 tokens (normally distributed, mean 14,000, standard deviation 3,000), and 50 candidate posts per user (each ~150 tokens), totaling ~14,000,000 tokens across all requests. The credit verification dataset simulates 60 users, each with 10 months of credit history (40,000–60,000 tokens per request), totaling ~3,000,000 tokens. These are not publicly available datasets — they are synthetic traces generated to match the characteristics of real production workloads described in prior literature. Each request in both datasets corresponds to one prefill-only inference call (one post evaluation or one credit decision). The authors justify this simulation approach by noting that "existing LLM datasets mainly focus on evaluating the LLM accuracy instead of the performance of the LLM engine" (Section 7.1). The paper does not report whether the synthetic traces were validated against real production traces beyond length distributions.

• Base model(s). Three LLMs are evaluated across different hardware tiers (Table 3): meta-llama/Llama-3.1-8B (low-end GPU, 2× NVIDIA L4 PCIe 24GB), RedHatAI/DeepSeek-R1-Distill-Qwen-32B-FP8-dynamic (middle-end GPU, 2× NVIDIA A100 PCIe 40GB), and Infermatic/Llama-3.3-70B-Instruct-FP8-Dynamic (high-end GPU, 2× NVIDIA H100 PCIe 80GB, and high-end w/ NVLink, 2× NVIDIA H100 NVLink 80GB). The models span an order of magnitude in parameter count (8B to 70B) and represent both dense (Llama) and mixture-of-experts distilled (DeepSeek-R1-Distill-Qwen) architectures. The FP8 quantization on the 32B and 70B models is a practical choice — these models would not fit on the target GPUs at full precision with reasonable batch sizes. The paper does not evaluate a single model across all hardware configurations, which means model architecture and hardware platform are partially confounded (e.g., Qwen-32B on A100 vs. Llama-8B on L4 vs. Llama-70B on H100 — the performance differences cannot be cleanly attributed to hardware alone or model alone).

• Metrics. The primary metric is query-per-second (QPS) vs. latency trade-off curves (Figure 6 for mean latency, Figure 7 for P99 latency). Latency is the end-to-end time from request arrival to response return, measured in seconds. QPS is controlled by varying the arrival rate $\lambda$ in a Poisson process — the paper sweeps QPS at multiples of the saturation throughput (1/4, 1/2, 1, 2, 3, 4 times the throughput achieved when all requests arrive simultaneously). Throughput is reported as requests per second (Figures 8, 9). Maximum input length (MIL) — the longest request an engine can process without running out of GPU memory — is reported in tokens (Table 2, Figure 10). The paper reports hitting these MIL thresholds rather than memory utilization percentages, which is appropriate for a capacity-planning metric but doesn't capture how close to the edge the engine operates under typical loads. All latency measurements include queueing time (since the scheduler dispatches requests one at a time from a waiting queue), not just execution time.

• Baselines. Four baselines are evaluated, all built on vLLM (Kwon et al., 2023) to control for implementation quality:

  • PagedAttention (the default vLLM memory management with FCFS scheduling, no chunking, no parallelism — the standard production baseline for LLM serving)
  • Chunked prefill (from Sarathi-Serve, Agrawal et al., 2024 — processes input chunk-by-chunk through all layers, with FCFS scheduling)
  • Tensor parallel (degree 2, using vLLM's built-in implementation — distributes each layer's computation across 2 GPUs with all-reduce communication after each layer)
  • Pipeline parallel (degree 2, using vLLM's built-in implementation — partitions model layers across 2 GPUs, with FCFS scheduling)

  All baselines have prefix caching enabled. For non-parallelized baselines (PagedAttention, chunked prefill) deployed on 2 GPUs, the paper launches one independent instance per GPU and uses user-ID-based routing (same user's requests go to the same instance, users round-robin assigned to instances). This is a fair deployment strategy — it doesn't artificially penalize single-GPU baselines by forcing them to use only one GPU. However, it means the comparison is not purely per-GPU; it's per-deployment (2 GPUs in all cases). The four hardware configurations (Table 3) produce 4 × 2 datasets × 2 latency metrics = 16 latency-throughput trade-off curves (Figures 6 and 7), plus throughput-only comparisons (Figures 8, 9) and MIL comparisons (Table 2, Figure 10). Some baselines cannot run certain workloads because their MIL is too short — Table 2 indicates this with checkmark/× notation (e.g., on A100, PagedAttention and Chunked Prefill cannot handle credit verification's 40,000+ token inputs).

• Generation budget / compute accounting. PrefillOnly measures "compute" implicitly through latency at a given QPS — there is no explicit FLOP counting or token-level cost model. The comparison is fair in the sense that all systems run the same requests on the same hardware, but the paper does not report normalized throughput (e.g., tokens processed per second per GPU) that would enable cross-hardware comparisons. The hardware configurations differ in both GPU count (always 2) and GPU capability (L4 vs. A100 vs. H100 with/without NVLink), so absolute latency numbers are not directly comparable across configurations — only the relative ordering of PrefillOnly vs. baselines within each configuration matters. For MIL comparisons (Table 2, Figure 10), "compute" is measured as the maximum sequence length that fits in GPU memory — a capacity metric rather than a throughput metric.

• Cross-validation / statistical protocol. The paper does not report any statistical protocol — no error bars, no confidence intervals, no multiple runs with different random seeds. The datasets are simulated with specific random seeds (for user profile lengths and request arrival patterns), but the paper does not discuss variability across seeds or across different instantiations of the Poisson arrival process. The lack of error bars on the QPS-latency curves (Figures 6, 7) makes it impossible to assess whether the observed differences between PrefillOnly and baselines are statistically significant or within measurement noise. For systems papers of this type, measurement variability from GPU scheduling, CUDA kernel launch timing, and OS-level interference can be non-trivial, especially at the millisecond scale relevant for low-latency regimes.

Main Quantitative Results

Throughput-Latency Trade-Offs (Figures 6 and 7)

Headline finding: PrefillOnly achieves 1.4–4.0× higher QPS than baselines at equivalent mean latency. The paper states in the evaluation overview that "PrefillOnly handles 1.4 − 4.0× larger query-per-second without inflating the average latency and P99 latency compared to baselines." This range comes from comparing the x-axis (QPS) value at which PrefillOnly's latency curve crosses a given latency threshold against where each baseline crosses the same threshold, aggregated across the 16 latency curves in Figures 6 and 7. The specific 4× figure appears to correspond to the largest gaps — for example, in Figure 6(c) (post recommendation, H100 w/o NVLink, mean latency), PrefillOnly sustains roughly 40 QPS at ~20s mean latency while pipeline parallel and tensor parallel reach similar latency at ~10 QPS (a ~4× difference). In Figure 6(e) (credit verification, L4, mean latency), PrefillOnly reaches ~0.4 QPS at ~50–100s mean latency while baselines are in the 0.1–0.2 QPS range at comparable latency — roughly a 2–4× gap.

Low-QPS regime: PrefillOnly can have higher latency than parallelized baselines. In the low-QPS regime (left side of each subfigure in Figure 6), tensor parallel and pipeline parallel sometimes achieve lower mean latency than PrefillOnly. For example, in Figure 6(d) (post recommendation, H100 w/ NVLink), the tensor parallel baseline has lower latency than PrefillOnly at QPS below ~10, though PrefillOnly overtakes it at higher QPS. The paper acknowledges this: "though tensor parallelism sometimes have lower latency than PrefillOnly under low QPS, it has much lower throughput than PrefillOnly due to extra communication cost and thus scales much worse than PrefillOnly in high QPS" (Section 7.2, Figure 6 caption). This is consistent with the architectural explanation: parallelization reduces per-request execution time (latency) by dividing work across GPUs, at the cost of communication overhead that limits total throughput — PrefillOnly avoids the overhead but runs on a single GPU, so single-request latency is bounded by single-GPU compute.

P99 latency follows the same pattern (Figure 7). The P99 latency curves largely mirror the mean latency curves: PrefillOnly achieves lower P99 latency at high QPS, while parallelized baselines may have lower P99 at low QPS. The paper uses the P99 results to argue that JCT-based scheduling with the fairness offset does not inflate tail latency — the fairness parameter $\lambda = 500$ (Section 7.1) is the default, and Figure 11 shows that varying $\lambda$ trades off mean vs. tail latency. The specific P99 values are harder to read precisely from the log-scale axes, but the relative ordering is consistent: PrefillOnly's curves are below (better) the baselines' curves at high QPS in all eight P99 subfigures.

Source of improvement differs by workload. The paper analyzes the two datasets separately (Section 7.2):

Post recommendation (short context, frequent prefix reuse): Figure 9 shows the throughput of PrefillOnly and baselines as a function of QPS on 2× H100 without NVLink. PrefillOnly maintains high throughput (~35 req/s) even as QPS increases to 50, while the chunked prefill baseline's throughput drops under high QPS — the paper attributes this to "prefix cache throttling" (the prefix cache fills up and starts evicting entries needed by subsequent requests, reducing cache hit rates and thus increasing per-request work). PrefillOnly avoids this throttling because continuous JCT calibration prioritizes cache-hit requests (which have lower JCT) before irrelevant requests can evict their prefixes. Parallelized baselines (tensor/pipeline parallel) avoid throttling because they distribute the prefix cache across GPUs, giving more total cache capacity — but they suffer lower throughput due to communication and synchronization overhead.

Credit verification (long context, minimal prefix reuse between users): Figure 8 contrasts throughput on 2× H100 with and without NVLink. PrefillOnly achieves the highest throughput in both configurations — ~0.13 req/s without NVLink, ~0.17 req/s with NVLink — because it handles long requests (40,000–60,000 tokens) on a single GPU via suffix KV cache discarding, avoiding parallelization overhead. Tensor parallelism's throughput improves substantially with NVLink (from ~0.06 to ~0.12 req/s) because the all-reduce communication is accelerated, but still trails PrefillOnly. Pipeline parallelism shows a smaller improvement, likely because pipeline bubbles (from variable request lengths) limit throughput regardless of communication speed. The paper notes that some baselines cannot run this workload at all: on A100 (Table 2), PagedAttention and Chunked Prefill lack sufficient MIL to process 40,000+ token requests, and pipeline parallelism (38,000 tokens MIL) also falls short. Only tensor parallelism (77,000 tokens) and PrefillOnly (87,000 tokens) can serve the credit verification workload on A100, and PrefillOnly does so at higher throughput (Figures 6(f), 7(f)).

Maximum Input Length (Table 2, Figure 10)

Headline finding: PrefillOnly expands MIL by up to 5× compared to non-parallelized baselines, and surpasses parallelized baselines on some configurations. Table 2 reports MIL in tokens for each configuration and workload combination:

On L4, PrefillOnly (130,000) outperforms chunked prefill (46,000) by 2.8× and PagedAttention (24,000) by 5.4×, but is surpassed by tensor parallel (195,000). On A100, PrefillOnly (87,000) achieves the highest MIL, beating tensor parallel (77,000) by 1.13× and PagedAttention (11,000) by 7.9×. On H100, PrefillOnly (97,000) trails both pipeline parallel (183,000) and tensor parallel (238,000). The paper's "up to 5×" claim refers to the comparison against PagedAttention on L4 and A100 — against parallelized baselines, the advantage is smaller or reversed (H100) since parallelism directly addresses the memory bottleneck by distributing KV caches across GPUs. The key point is that PrefillOnly achieves this MIL without parallelization overhead, so at a given MIL that all systems can reach, PrefillOnly has higher throughput (Figures 6–9).

Figure 10: Contribution of individual optimizations to MIL. On a Qwen-2.5-32B-FP8 model on A100: vanilla vLLM achieves some baseline MIL (the figure reports 7.9 on an unlabeled axis — likely 7,900 tokens); adding hybrid prefilling (chunking alone, without output preallocation or in-place computation) substantially increases this; adding output preallocation further increases MIL; adding in-place computation brings the total to over 8.7× the vanilla baseline. The paper claims this as evidence that the three implementation optimizations (chunking, preallocation, in-place) each contribute meaningfully. The exact MIL values are difficult to read from Figure 10's bar chart since only relative multipliers are labelled.

Workload coverage assessment (Table 2 checkmarks): For post recommendation (WL1, 11,000–17,000 tokens), all configurations and baselines can handle the workload except PagedAttention on A100 and H100. For credit verification (WL2, 40,000–60,000 tokens), PagedAttention and Chunked Prefill fail on all three GPU types; pipeline parallel fails on A100; only tensor parallel and PrefillOnly can serve credit verification on A100. On L4, pipeline parallel, tensor parallel, and PrefillOnly can all serve credit verification, but PrefillOnly does so at the highest throughput (Figure 6(e), 7(e)). On H100, all parallelized baselines and PrefillOnly can serve credit verification.

Latency Distribution and Fairness (Figure 11)

Headline finding: The fairness parameter $\lambda$ provides a tunable trade-off between mean and tail latency. Figure 11 shows the CDF of request latency for PrefillOnly under three values of $\lambda$ (0, 200, 2000) on an unspecified hardware/model configuration (likely the post recommendation workload, but the paper does not state this explicitly — a notable omission). With $\lambda = 0$ (pure SRJF, no fairness offset), the CDF rises fastest at low latencies (best average case) but has a long tail extending beyond 60 seconds. With $\lambda = 2000$, the CDF is shifted right (higher average latency, roughly 20–30s vs. ~10s for $\lambda = 0$ at the median) but the tail is compressed — nearly all requests complete by ~55s vs. ~65s for $\lambda = 0$. The default $\lambda = 500$ (used in all other experiments) falls between these extremes. The paper does not report exact numerical values for mean and P99 latency at each $\lambda$, only the CDF curves.

This result is important for validating that the fairness mechanism works as designed — higher $\lambda$ shifts the scheduler toward FIFO-like behavior, bounding worst-case latency at the expense of average latency. However, the paper does not report how $\lambda = 500$ was chosen or whether this parameter requires tuning for different workloads or hardware configurations. The choice of 500 as a default appears to be an empirical selection rather than a principled one.

Throughput Under Extreme Load (Figure 9)

Headline finding: PrefillOnly maintains throughput under high QPS while chunked prefill degrades due to cache thrashing. Figure 9 (post recommendation, 2× H100 w/o NVLink) shows that PrefillOnly's throughput remains roughly constant at ~35 req/s as offered QPS increases from near zero to 50. The chunked prefill baseline initially matches this throughput at low QPS but drops to approximately 10–15 req/s at the highest QPS. The paper's explanation — that chunked prefill suffers from "prefix cache throttling" where the limited GPU cache capacity causes evictions that reduce cache hit rates — is plausible but not directly measured. The paper does not report cache hit rates as a function of QPS for any system, which would directly validate this explanation. The parallelized baselines maintain constant throughput (no throttling, since their cache capacity is larger) but at lower absolute levels (~20–25 req/s for pipeline parallel, ~10 req/s for tensor parallel) due to communication/synchronization overhead.

Ablation Studies and Robustness Checks

Hybrid prefilling implementation optimizations (Figure 10): Chunking alone provides a 7.9× MIL improvement over vanilla vLLM on Qwen-2.5-32B-FP8 on A100. Adding output preallocation (to avoid doubling memory during concatenation) further increases MIL. Adding in-place computation (reusing input memory for output when shapes match) brings the total improvement to over 8.7×. The contribution of each optimization is modest in relative terms (7.9× → 8.0+× → 8.7×) but the absolute MIL gain from 7.9× to 8.7× represents thousands of tokens and can be the difference between fitting a workload and not (e.g., the credit verification workload requires >40,000 tokens — if 7.9× baseline MIL is 62,410 tokens on a hypothetical 7,900-token vanilla baseline, 8.7× would be 68,730 tokens). The paper does not report absolute MIL numbers, only multipliers, which makes it difficult to assess practical significance.

JCT proxy: cache-miss token count vs. full regression model: The paper reports (Section 6.3) that on 1× A100 with Qwen-32B-FP8, the Pearson correlation coefficient between actual JCT and $n_{\text{input}} - n_{\text{cached}}$ is 0.987. This single measurement is used to justify using the cache-miss token count as the JCT proxy for all scheduling decisions, rather than the profiled linear regression model. The paper does not report correlation coefficients for other hardware/model combinations, does not evaluate whether the scheduler using the full regression model outperforms the proxy-based scheduler, and does not discuss scenarios where the proxy might break down (e.g., very long sequences where attention's quadratic scaling becomes the bottleneck rather than linear computation per token). The 0.987 correlation is high, but the residual 1.3% unexplained variance could translate to systematic errors for certain request lengths that affect scheduling decisions.

Fairness parameter $\lambda$ (Figure 11): As discussed above, varying $\lambda$ from 0 to 2000 shifts the latency CDF from SRJF-like (low mean, high tail) to FIFO-like (higher mean, bounded tail). The paper uses $\lambda = 500$ as default. No sweep across workloads or hardware configurations is reported — it is unclear whether $\lambda = 500$ generalizes or requires per-deployment tuning.

Parallelization with and without NVLink (Figure 8): On credit verification workload with 2× H100, NVLink improves tensor parallel throughput from ~0.06 to ~0.12 req/s (~2× improvement), pipeline parallel throughput from ~0.05 to ~0.07 req/s (~1.4× improvement), and PrefillOnly throughput from ~0.13 to ~0.17 req/s (~1.3× improvement). NVLink most helps tensor parallel (where all-reduce communication is the bottleneck), moderately helps pipeline parallel (where communication is less frequent but still present), and provides a modest benefit to PrefillOnly (likely from faster weight-loading or P2P transfers, though PrefillOnly runs on a single GPU per request so NVLink between GPUs should not directly accelerate its inference — the improvement may come from the round-robin routing across instances benefiting from faster inter-GPU communication for prefix cache sharing, though the paper does not discuss this mechanism).

Workload coverage (Table 2): The checkmark/× notation shows which baselines can even run which workloads on which hardware. This is not an ablation per se but a capability table that contextualizes the throughput results — when a baseline's MIL is insufficient, it simply cannot serve that workload, making throughput undefined. The fact that non-parallelized baselines (PagedAttention, chunked prefill) fail on credit verification for most hardware configurations highlights the practical significance of PrefillOnly's MIL improvement.

Negative result: PrefillOnly does not always achieve the highest MIL (Table 2). On H100, tensor parallel (238,000 tokens) and pipeline parallel (183,000 tokens) both exceed PrefillOnly's 97,000 tokens. This is because parallelization directly addresses the KV cache memory bottleneck by distributing it across GPUs, while PrefillOnly's suffix discarding can only eliminate the KV caches that are not prefix caches — the prefix caches of long requests still consume GPU memory. This limitation is acknowledged implicitly: PrefillOnly's MIL advantage is largest when comparing against non-parallelized baselines (5× on L4, 7.9× on A100), not against parallelized ones.

Missing ablation: hybrid prefilling without suffix KV cache discarding. The paper does not isolate the contribution of hybrid prefilling to throughput separately from suffix KV cache discarding. Both are enabled together in PrefillOnly, so the throughput improvements (Figures 6–9) reflect the combined effect. An ablation comparing PrefillOnly with hybrid prefilling + suffix discarding against PrefillOnly with hybrid prefilling but retaining all KV caches (or against chunked prefill with suffix discarding) would decompose the contributions of the two techniques, but this is not reported.

Missing ablation: continuous JCT calibration vs. static SRJF. The paper provides an illustrative example (Figure 5) and Algorithm 1, but does not report empirical comparisons between PrefillOnly's scheduler (SRJF with continuous calibration) and a static SRJF scheduler (sort by arrival-time JCT, never re-evaluate) on the actual datasets. This is the most direct ablation for the scheduling contribution, and its absence is notable — the paper argues that continuous calibration improves prefix cache hit rates (the Figure 5 example shows 2 cache hits vs. 1), but this is not validated experimentally. The paper attributes PrefillOnly's advantage on the post recommendation workload to avoiding "prefix cache throttling" that affects chunked prefill (Figure 9), implying that the scheduler is responsible, but does not directly compare scheduling policies.

Critical Assessment

The experiments demonstrate that PrefillOnly achieves higher throughput than vLLM-based baselines on prefill-only workloads across diverse hardware configurations, and that hybrid prefilling substantially increases maximum input length compared to non-parallelized baselines. However, the experimental design has several limitations that affect how broadly the paper's claims should be interpreted.

Claim: PrefillOnly handles 1.4–4.0× higher QPS without inflating latency. The throughput-latency curves (Figures 6, 7) support this claim for the tested configurations. The 4× figure is aggressive — it corresponds to the largest observed gaps, not the typical improvement — but even the lower bound of 1.4× represents a meaningful throughput gain. However, the lack of error bars or multiple measurement runs means we cannot assess whether the 1.4× figure, for instance, is reliably above 1.0× or within measurement noise. GPU inference latency measurements are subject to variability from CUDA kernel scheduling, power throttling, and OS-level interference, and without statistics, the reader cannot determine which differences are systematic vs. transient. Systems papers in venues like OSDI and EuroSys typically report multiple runs with error bars for latency-throughput curves; their absence here weakens the quantitative claims.

Claim: PrefillOnly expands maximum input length by up to 5× without requiring inference parallelization. Table 2 and Figure 10 support this, but with important caveats. The 5× figure compares against PagedAttention, not against the strongest non-parallelized baseline (chunked prefill). Against chunked prefill, the improvement is more modest: 130,000 vs. 46,000 on L4 (2.8×), 87,000 vs. 17,000 on A100 (5.1×), and 97,000 vs. 25,000 on H100 (3.9×). Furthermore, on H100, both parallelized baselines exceed PrefillOnly's MIL, so the "without requiring inference parallelization" qualifier matters — the paper is not claiming PrefillOnly achieves the highest possible MIL across all configurations, only that it achieves high MIL while maintaining throughput. This is a correct claim given the evidence, but the reader should understand that parallelization remains the right choice if MIL is the sole objective and throughput is not a concern.

Claim: Continuous JCT calibration avoids prefix cache throttling. This is the weakest link in the experimental evidence. The paper attributes PrefillOnly's throughput maintenance under high QPS (Figure 9) to the scheduler avoiding cache thrashing, but this attribution is inferential rather than directly demonstrated. Figure 9 shows that chunked prefill's throughput drops at high QPS while PrefillOnly's does not, and the paper posits that chunked prefill suffers from prefix cache throttling while PrefillOnly avoids it through better scheduling. However, chunked prefill differs from PrefillOnly along multiple dimensions simultaneously: it uses chunked attention (reduced kernel efficiency), retains full KV caches (different memory pressure), and uses FCFS scheduling (different scheduling policy). The throughput degradation could be caused by any of these differences or their interaction. To isolate the scheduling contribution, one would need to compare PrefillOnly's scheduler against an FCFS scheduler within the same PrefillOnly engine — but PrefillOnly always uses continuous JCT calibration, so this comparison is not reported. The illustrative example in Figure 5 demonstrates the potential for an additional cache hit, but whether this potential materializes in the actual workload traces and accounts for the throughput difference is unmeasured.

Similarly, the paper claims that the source of improvement on the post recommendation workload is scheduling (avoiding cache thrashing) while on the credit verification workload it's memory management (handling long inputs without parallelization). This decomposition is plausible given the workload characteristics (frequent prefix reuse in post recommendation, long unique inputs in credit verification), but it's an interpretation of the aggregate results, not an experimentally isolated finding.

Missing experiments that would strengthen the paper:

• Scheduling ablation: Compare PrefillOnly with continuous JCT calibration against PrefillOnly with FCFS scheduling (keeping all other components identical — hybrid prefilling, suffix discarding) on both workloads. This would isolate the scheduling contribution.
• Cache hit rate reporting: Report prefix cache hit rates as a function of QPS for all systems. This would directly validate the "cache throttling" explanation for chunked prefill's throughput degradation and the "continuous calibration improves cache hits" claim.
• Single-model cross-hardware comparison: Evaluate one model (ideally the Qwen-32B, which fits on all three GPU tiers with appropriate quantization) across L4, A100, and H100 to cleanly separate hardware effects from model effects.
• Multiple runs with error bars: Run each QPS-latency data point multiple times (e.g., 5–10 trials) and report mean ± standard deviation or confidence intervals.
• Real workload trace validation: The paper uses synthetic traces with Poisson arrivals and simulated content lengths. Validating against production traces from deployed recommendation or credit verification systems would strengthen the external validity claim, though the paper acknowledges the difficulty of obtaining such traces.
• Sensitivity to chunk size in hybrid prefilling: The paper does not report the chunk size used for non-attention layers in hybrid prefilling, nor does it sweep this parameter. Chunk size is a critical hyperparameter — too small and iteration overhead dominates, too large and peak memory is not sufficiently reduced. An ablation over chunk sizes would characterize this trade-off.

Conditional nature of the claims. The paper's central claim — that PrefillOnly outperforms general-purpose engines on prefill-only workloads — holds most strongly when:

• The workload consists of long inputs (tens of thousands of tokens) that stress GPU memory, where suffix KV cache discarding provides substantial relief.
• There is frequent prefix reuse across requests (as in the post recommendation workload), where continuous JCT calibration can exploit dynamically available caches.
• The deployment uses GPUs without high-speed interconnects (NVLink), where parallelization overhead is most severe.
• The workload is throughput-sensitive rather than latency-sensitive at low QPS, where PrefillOnly's single-GPU execution has higher per-request latency than parallelized alternatives.

Conversely, the advantages diminish when:

• Inputs are short (fits comfortably in GPU memory), where memory optimization provides no benefit.
• There is no prefix reuse across requests, where scheduling optimization provides no benefit and FCFS would be equivalent.
• NVLink is available (Figure 8 shows tensor parallel with NVLink approaches PrefillOnly's throughput on H100), reducing the communication overhead penalty.
• Low-latency serving at low QPS is the primary requirement (Figures 6(d), 7(d) show tensor parallel with NVLink beating PrefillOnly's latency in this regime).

The paper is generally forthright about these conditions — the discussion of when parallelization may be preferable for latency (§5.2, §7.2) and the acknowledgment that some baselines outperform PrefillOnly at low QPS show awareness of the boundaries. However, the abstract and introduction emphasize the 4× figure without these qualifiers, which could mislead a reader who does not examine the per-configuration results.

Single-implementation concern. All baselines and PrefillOnly are built on vLLM, which is an appropriate choice for controlling implementation quality. However, this means the results may not generalize to other serving frameworks (e.g., SGLang, TensorRT-LLM) that may have different performance characteristics for the same techniques. The paper also implements hybrid prefilling via torch.compile — an elegant approach for portability, but one that may not achieve the same performance as hand-written CUDA kernels for chunked processing. A comparison against an optimized CUDA implementation of chunked prefill (rather than vLLM's Python-level implementation) would test whether the hybrid prefilling advantage persists at the performance frontier.

Scale of evaluation. The paper evaluates 2 GPUs per configuration, which is reasonable for demonstrating the concept but small relative to production deployments that the motivation discusses (hundreds or thousands of GPUs). The scaling behavior of PrefillOnly's round-robin routing across many instances, the interaction between prefix caching and the routing policy at scale, and the throughput implications of many independent schedulers operating without coordination are not evaluated. This is not a flaw per se — the paper is presenting a first system for a new workload class — but the reader should not extrapolate the 4× throughput improvement to thousand-GPU clusters without evidence that the gains are preserved under distributed serving.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Not Included in the Headline Efficiency Numbers

The assumption or constraint. PrefillOnly's core scheduling mechanism — continuous JCT calibration — requires knowing, before each scheduling decision, how many tokens of each waiting request are already cached (to compute n_cached and thus the JCT proxy n_input − n_cached). This is not free: the scheduler must query the prefix cache store to determine cache intersections for every waiting request at every scheduling step. The paper does not account for this calibration overhead in any of its latency or throughput measurements. Section 6.3 describes the calibration procedure as computing n_cached by checking "how many tokens in request r that hits the prefix cache," but the paper never discusses the computational cost of performing this check for potentially hundreds of waiting requests at each scheduling iteration.

The consequence. As the waiting queue grows (which happens under high QPS), the calibration overhead scales linearly with the number of waiting requests. For each waiting request, the scheduler must compare its token sequence against all cached prefixes — an operation whose cost depends on the prefix cache data structure and the number of cached entries. If this overhead is non-trivial relative to the request execution time, the scheduler could become a bottleneck, particularly when request execution is fast (short inputs, powerful GPUs). The paper's throughput-latency curves implicitly include whatever calibration overhead exists in their implementation, but since the overhead is not measured or reported separately, a practitioner cannot estimate how the scheduler scales with queue depth or cache size. In the worst case, a naive implementation of prefix-cache intersection checks (e.g., linear scan over all waiting requests against all cached prefixes) could consume more GPU time than it saves, and the paper provides no analysis of this risk.

What evidence exists in the paper. The paper provides no direct evidence on calibration overhead — no profiling of scheduler CPU time, no measurement of scheduling latency as a function of queue depth, and no analysis of the prefix cache data structure's query complexity. The throughput results (Figures 6, 8, 9) demonstrate that the system as a whole performs well, but they cannot isolate whether the scheduler is adding latency that partially offsets the gains from better cache hit rates. The paper does report (Section 6.3) that JCT can be well-approximated by cache-miss token count with 0.987 Pearson correlation, which simplifies the per-request computation (a subtraction rather than a regression model evaluation), but this addresses only the JCT estimation step, not the cost of determining n_cached itself.

Mitigation status. The paper does not measure, discuss, or attempt to mitigate calibration overhead. There is no mention of the computational complexity of prefix-cache intersection queries, no profiling of scheduler execution time, and no analysis of how the scheduling algorithm scales with queue depth. The implementation detail that calibration happens "every time before running the scheduling algorithm" (Section 6.3) suggests that the authors treat the cost as negligible, but this is an untested assumption. A practitioner deploying PrefillOnly would need to independently benchmark the scheduler under their expected workload characteristics to determine whether calibration overhead is material.

---

The Scheduler's JCT Model Assumes Compute-Bound Linearity, Which Breaks at Extreme Sequence Lengths

The assumption or constraint. The JCT proxy used by the scheduler — that job completion time is proportional to n_input − n_cached (the number of uncached tokens) — relies on the implicit assumption that per-token computation cost is roughly constant regardless of total sequence length. Section 6.3 justifies this with a single empirical measurement: on Qwen-32B-FP8 with an A100 GPU, the Pearson correlation between actual JCT and cache-miss token count is 0.987. However, the attention operation in transformer models has O(N^2) complexity for prefilling — each new token attends to all preceding tokens, so the total FLOPs for processing N tokens is proportional to N^2, not N. For the sequence lengths tested in the paper's evaluation workloads (up to 60,000 tokens for credit verification), the quadratic term may be small enough relative to the linear terms (MLP computation, memory movement) that the linear proxy remains accurate. But as sequence lengths grow beyond the evaluated range, attention computation will eventually dominate, and the linear proxy will systematically underestimate the JCT of very long requests — causing the scheduler to deprioritize them relative to their true cost and potentially worsening tail latency for the longest inputs.

The consequence. A scheduler that underestimates the JCT of long requests will treat them as "shorter" than they actually are, scheduling them ahead of medium-length requests that would actually complete faster. This misprioritization would increase average latency and could interact badly with the fairness offset: a long request whose true JCT is underestimated would accumulate queueing time offset λ · T_queue more slowly than its true cost warrants, delaying the point at which the fairness mechanism correctly prioritizes it. The result would be systematically higher latency for very long requests — exactly the regime where the paper's MIL improvements are most valuable (credit verification at 40,000–60,000 tokens). If attention's quadratic cost becomes dominant at, say, 70,000+ tokens, the linear JCT proxy would be inaccurate for sequences near the MIL ceiling that PrefillOnly enables.

What evidence exists in the paper. The paper provides the 0.987 correlation measurement only for the Qwen-32B-FP8 model on A100, and does not specify the sequence length range over which this correlation was computed. The evaluation workloads top out at 60,000 tokens (credit verification), so the proxy has been validated only up to this length. The paper does not measure how the correlation degrades as sequence length increases, does not report an equivalent correlation for the Llama-3.1-8B or Llama-3.3-70B models on other hardware, and does not discuss the O(N^2) attention term as a potential confound. The scheduler would continue to use the linear proxy at lengths beyond those tested, with no mechanism to detect or correct the inaccuracy. Figure 11 (latency CDF under different λ values) does not break out latency by request length, so it's impossible to determine whether very long requests experience systematically higher latency than the JCT proxy predicts.

Mitigation status. The paper does not address this limitation. The full linear regression model mentioned in Section 6.3 (trained on profiled (n_input, n_cached) pairs) could potentially capture the quadratic attention term if the profiling grid included sufficiently long sequences and the regression included nonlinear features — but the paper switches to the simpler cache-miss proxy for deployment and does not compare the proxy's accuracy against the regression model on long sequences. A mitigation would involve either (a) validating the linear proxy across the full range of supported sequence lengths for each model and hardware configuration, or (b) incorporating a nonlinear JCT model that accounts for attention's O(N^2) scaling at extreme lengths.

---

Suffix KV Cache Discarding Permanently Sacrifices Potential Cache Reuse

The assumption or constraint. PrefillOnly's memory management strategy discards the KV caches of suffix tokens when they cannot fit in GPU memory (Section 5.1). This is framed as an optimization for prefill-only workloads: since no decoding follows, suffix KV caches have no decoding reuse value. However, they might still have prefix reuse value for future requests. In the post recommendation workload, the paper's example shows that all 50 requests for a given user share a long common prefix (the user profile) — which is retained — while the suffix (the 150-token article description) is unique per request and discarded. But in real workloads, suffix sharing may occur in more complex patterns. For instance, two users with partially overlapping browsing histories might share some suffix tokens (e.g., both read the same popular article) even if their full profiles differ. By discarding suffix caches, PrefillOnly permanently loses the opportunity to accelerate future requests that partially share those suffixes — a opportunity that standard prefix caching (which retains all KV caches) could exploit. The paper explicitly acknowledges this trade-off in Section 9: "Current implementation of PrefillOnly performs suffix KV caches discarding, which prevents future requests to potentially reuse the computation of the discarded part."

The consequence. In workloads with non-trivial suffix overlap across requests, PrefillOnly's suffix discarding would achieve lower cache hit rates than an engine that retains all KV caches (memory permitting). This would manifest as higher average latency and lower effective throughput compared to a hypothetical version of PrefillOnly that selectively retains suffix caches when reuse is likely. The severity depends on the workload: in the paper's post recommendation dataset, suffix sharing is minimal by construction (each article description is unique to its post), so discarding costs nothing. But in other prefill-only applications — e.g., document classification where many documents share boilerplate text, or multi-turn dialogue where later turns share content with earlier turns — suffix overlap could be substantial, and discarding would be a genuine performance regression. The paper provides no characterization of how much suffix overlap exists in real prefill-only workloads or how sensitive throughput is to this overlap.

What evidence exists in the paper. The paper provides no direct evidence on suffix cache reuse patterns. The evaluation datasets are constructed such that suffix sharing is minimal (unique posts per request in recommendation, unique credit histories per user in credit verification), so the throughput results reflect a best-case scenario for suffix discarding. Table 2 shows that suffix discarding enables higher MIL — which is the benefit — but does not measure the cost in terms of lost cache hits. There is no ablation comparing suffix discarding against suffix retention (with some alternative memory management, such as offloading to CPU) in a workload with suffix overlap.

Mitigation status. The paper partially acknowledges this limitation (Section 9: "This limitation can be alleviated by offloading the KV caches to CPU instead via solutions like LMCache") but does not implement or evaluate suffix offloading. The suggestion is that CPU offloading would preserve suffix caches for potential reuse while freeing GPU memory, trading off CPU-to-GPU transfer latency against the recomputation cost that discarding would incur. However, CPU offloading introduces its own trade-offs: PCIe bandwidth is limited (tens of GB/s vs. hundreds of GB/s for GPU memory), and prefetching suffix caches from CPU to GPU at scheduling time would add latency to cache-hit requests, potentially negating the benefit. The paper leaves this as future work, so a practitioner cannot currently deploy PrefillOnly with suffix offloading — the only memory management option is discarding, with its attendant loss of suffix reuse potential.

---

Evaluation Uses Synthetic Traces Without Validation Against Production Workloads

The assumption or constraint. All evaluation results are based on two simulated datasets — post recommendation and credit verification — whose characteristics (request lengths, prefix sharing patterns, arrival distributions) are specified by the authors based on "rough estimation" rather than measured from production systems. Section 7.1 details the construction: post recommendation uses a normal distribution for user profile length (mean 14,000, standard deviation 3,000) based on assumptions about user engagement frequency (four times per week for four weeks, five to six posts per session), and article length estimated from "X as an example" (presumably Twitter/X, though the paper does not state this explicitly). Credit verification uses ten months of credit history at 4,000–6,000 tokens per month. The request arrival pattern is assumed to be a Poisson process — a standard but simplifying assumption that may not capture bursty or time-correlated arrival patterns common in production serving.

The consequence. The throughput improvements reported (up to 4×) may not generalize to production workloads whose characteristics differ from the synthetic traces along dimensions that affect PrefillOnly's optimization mechanisms. Specifically:

• Prefix sharing patterns: The paper's post recommendation workload constructs requests so that all 50 posts for a given user share exactly the same long prefix (user profile) followed by a unique short suffix. This is an ideal case for prefix caching and for continuous JCT calibration (which exploits transient cache availability). If real workloads have more complex sharing patterns — multiple levels of partial prefix overlap, prefix reuse across users rather than within users, or prefix caches that are evicted before all sharing requests arrive — the scheduler's benefit would differ, potentially substantially.

• Request length distribution: The paper's workloads have relatively narrow length distributions (11,000–17,000 tokens for post recommendation, 40,000–60,000 for credit verification). In production, length distributions could be heavier-tailed (a few extremely long requests mixed with many short ones), which would stress the hybrid prefilling memory management (extreme lengths risk OOM even with suffix discarding) and the fairness mechanism (very long requests may still starve despite the fairness offset if their effective work n_input − n_cached is enormous).

• Arrival patterns: Poisson arrivals assume independent, exponentially distributed inter-arrival times. Production serving systems often see bursty arrivals (many requests arriving nearly simultaneously, followed by idle periods) due to batch processing, user activity cycles, or upstream system behavior. Bursty arrivals would create transient deep queues that stress the scheduler's calibration overhead (as discussed in the first limitation) and potentially cause the fairness offset to behave differently than under steady Poisson load.

What evidence exists in the paper. The paper offers no validation of the synthetic traces against production data. The justification in Section 7.1 — "existing LLM datasets mainly focus on evaluating the LLM accuracy instead of the performance of the LLM engine" — explains why public benchmarks aren't used, but doesn't establish that the synthetic traces are representative. There is no sensitivity analysis varying the key workload parameters (prefix sharing degree, length distribution shape, arrival burstiness) to show how the throughput gains change. The paper evaluates exactly two workload scenarios, which leaves open the question of where in the workload space PrefillOnly's advantages are largest or smallest.

Mitigation status. The paper does not address this limitation. The authors do not claim that the synthetic traces are validated against production data, nor do they discuss the sensitivity of results to workload assumptions. A practitioner considering PrefillOnly for their own prefill-only workload would need to independently benchmark the system under their specific traffic patterns and content characteristics — the paper's results provide a proof of concept but not a predictive model of throughput improvement.

---

Fairness Parameter λ Requires Per-Deployment Tuning with Unclear Generalization

The assumption or constraint. The fairness offset mechanism (λ · T_queue subtracted from the JCT estimate) has a tunable parameter λ that controls the trade-off between average latency (favored by lower λ, approaching pure SRJF) and tail latency (favored by higher λ, approaching FIFO). The paper uses λ = 500 as the default for all experiments (Section 7.1) and shows (Figure 11) that varying λ shifts the latency CDF as expected. However, the paper provides no guidance on how to choose λ for a new deployment: no relationship between λ and workload characteristics (request length distribution, arrival rate, prefix sharing degree), no analysis of whether λ = 500 generalizes across the four hardware configurations and three models tested, and no method for automatically tuning λ based on latency SLOs.

The consequence. A practitioner deploying PrefillOnly would need to empirically tune λ for their specific workload and hardware, potentially through trial-and-error under production load — a process that risks degrading user experience during tuning (if λ is too low, tail latency spikes; if too high, average latency increases unnecessarily). Without understanding how λ interacts with workload parameters, the practitioner cannot predict whether λ = 500 is a reasonable starting point or off by orders of magnitude. The parameter's physical interpretation is unclear: λ has units of "tokens per second" (since it converts queueing time into equivalent token count), but the paper doesn't discuss what value of λ corresponds to what latency guarantee (e.g., "a request with 10,000 uncached tokens will be scheduled within X seconds even under heavy load").

The four hardware configurations (L4, A100, H100 w/o NVLink, H100 w/ NVLink) have substantially different per-token processing speeds, so the conversion between queueing time and effective token reduction (λ · T_queue) should scale with hardware throughput. A λ = 500 that provides reasonable fairness on an A100 (processing perhaps thousands of tokens per second) might be far too low on an L4 (processing hundreds of tokens per second), causing long requests to accumulate insufficient fairness offset and starve. Conversely, on an H100, λ = 500 might be too high, causing the scheduler to behave essentially as FIFO and losing the average-latency benefits of SRJF. The paper does not discuss this hardware-dependence.

What evidence exists in the paper. Figure 11 shows latency CDFs for three values of λ (0, 200, 2000) on an unspecified hardware/model configuration. This demonstrates that λ has the expected qualitative effect and that λ = 500 is a plausible intermediate value. However, the paper does not report whether λ = 500 was used for all four hardware configurations in Figures 6 and 7, or whether it was tuned per configuration. If λ = 500 was used uniformly, the throughput-latency results already incorporate any hardware-λ mismatch — but this means the reported 4× improvement might be suboptimal for some configurations (if λ is mis-tuned) or might rely on fortuitously appropriate λ values that wouldn't hold in other settings.

Mitigation status. The paper does not discuss λ tuning methodology, does not propose an automatic tuning approach, and does not analyze the sensitivity of throughput-latency results to λ. The parameter is introduced as a mechanism for starvation prevention (Section 6.3) and demonstrated to work (Figure 11), but its deployment implications are not addressed. A practitioner would need to develop their own tuning strategy — for example, profiling the latency CDF at multiple λ values under representative load and selecting based on P99 latency SLOs.

---

No Demonstration That Hybrid Prefilling + Suffix Discarding Works Under Concurrent Multi-Tenant GPU Execution

The assumption or constraint. All evaluation uses dedicated GPUs running only PrefillOnly (or a single baseline) with requests routed by user ID to independent instances. Section 2.4 explicitly argues that prefill-only workloads should use dedicated GPUs rather than sharing with generative workloads because "inter-workload interference is significant" and "the volume of prefill-only workload is large enough that it deserves dedicated GPU resources." However, even dedicated PrefillOnly instances may experience internal concurrency: if multiple requests from the same user arrive nearly simultaneously, the round-robin routing (Section 7.1) may assign them to different GPU instances, each running independent PrefillOnly processes. But within a single instance, the paper's design explicitly sequentializes execution — "PrefillOnly chooses to schedule the requests one by one instead of batching them" (Section 6.1) — so there is no intra-instance concurrency. The assumption is that this sequential execution is sufficient and that per-instance GPU utilization remains high without batching.

The consequence. Sequential execution means that at low QPS, the GPU sits idle between requests. The paper's throughput-latency curves (Figures 6, 7) show latency at various QPS levels, but do not report GPU utilization. If utilization is low at typical operating QPS (say, 50% utilization at half the saturation throughput), then the throughput-per-GPU numbers may understate what could be achieved by a system that could overlap execution of independent requests — for example, by processing one request's non-attention chunks while another request's attention layer runs on a separate CUDA stream, or by batching very short requests together. The paper argues that prefill-only is compute-bound (Section 6.1), implying high utilization even for single requests, but this argument is theoretical — the paper does not empirically verify that single-request execution saturates the GPU's compute units across the range of input lengths in the evaluation workloads. For relatively short inputs (e.g., the 11,000-token profiles in post recommendation), the GPU may not be fully saturated by a single request, and sequential execution would leave throughput on the table.

What evidence exists in the paper. The paper provides no GPU utilization measurements, no profiling of compute unit occupancy during single-request execution, and no analysis of whether request execution time is dominated by compute vs. memory vs. kernel launch overhead for the evaluated input lengths and models. The theoretical claim that prefill-only is compute-bound (Section 2.4, 6.1) is stated without empirical support — it is an architectural assumption, not a measured fact. The throughput advantage over baselines (Figures 6, 8, 9) is consistent with PrefillOnly making good use of GPU resources, but does not rule out the possibility that both PrefillOnly and the baselines are underutilizing the GPU, and that a system with better concurrency could achieve even higher throughput.

Mitigation status. The paper does not address GPU utilization or the potential for intra-GPU concurrency beyond the brief theoretical argument in Section 6.1. The sequential execution model is presented as optimal given the assumption of compute-boundedness, but this assumption is not validated. A practitioner deploying PrefillOnly on expensive GPU hardware would want to know whether they're achieving good utilization of their investment — the paper provides throughput numbers but not the efficiency context (utilization, FLOPs/second relative to theoretical peak) that would allow this assessment.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a case for workload-specific LLM serving architectures. The dominant paradigm in LLM inference engines — exemplified by vLLM, Orca, Sarathi-Serve — assumes all workloads are generative and optimizes accordingly. PrefillOnly demonstrates that a different workload class (discriminative, single-token, long-input) has an entirely different optimization landscape: KV caches are liabilities rather than assets, compute-bound execution makes batching counterproductive, and deterministic job completion times enable scheduling algorithms that are impossible under output-length uncertainty. The paper's key conceptual contribution is demonstrating that recognizing a workload as "prefill-only" unlocks architectural choices that are invisible to general-purpose engines.

The magnitude of the shift matters. This is not an incremental optimization — like a 10% throughput improvement from a better attention kernel — but a re-architecting of the inference engine around workload properties. The 4× throughput improvement (Section 7, Figures 6–9) and 5× maximum input length expansion (Table 2) come from structural changes to memory management and scheduling, not from tuning existing knobs. The paper is arguing, in effect, that prefill-only workloads deserve their own engine architecture in the same way that OLAP workloads deserve columnar databases rather than a few extra indexes on a row-store — the workload characteristics are different enough that the optimal design qualitatively differs.

More broadly, the paper reframes how the systems community should think about LLM serving diversity. As LLMs displace traditional ML pipelines in recommendation, verification, and labeling, the assumption that all serving is multi-token generation will break down. PrefillOnly is the first system to treat this as a first-class design consideration, but the pattern — identify workload-specific properties, invert the optimization target — generalizes. Code completion workloads (short, repetitive prefixes; latency-sensitive), summarization (very long inputs, moderate output lengths), and agentic tool-use (structured output, stateful interactions) may each have their own architectural sweet spots that general-purpose engines miss. The paper opens the door to a family of specialized LLM inference engines rather than a single one-size-fits-all design.

The paper also clarifies the compute-bound vs. memory-bound distinction in a way that should influence how new LLM serving systems are designed. Section 6.1's argument — that batching helps only when weight memory reads dominate, and that prefill-only is compute-bound so batching is counterproductive — is simple but has been largely absent from the LLM serving literature, which treats continuous batching as universally beneficial. This distinction provides a diagnostic framework: before designing an inference engine, first determine whether the target workload is compute-bound or memory-bandwidth-bound, then choose batching, memory management, and scheduling strategies accordingly. For prefill-heavy or prefill-only workloads, this means the entire continuous batching machinery of modern engines may be unnecessary complexity.

The paper does not reconcile prior contradictions in the same way that a scientific paper resolves conflicting empirical findings — it's a systems paper, not an empirical ML paper. However, it does explain why existing engines underperform on prefill-only workloads in terms that reframe the problem: it's not that vLLM or chunked prefill are "bad" systems — they're optimized for a different workload regime. The contradiction isn't between conflicting results but between implicit assumptions (all LLM serving is multi-token generation) and emerging reality (production systems increasingly use LLMs for single-token discriminative tasks). The paper resolves this by making the assumption explicit and showing what changes when you drop it.

Research directions that become more attractive: (a) specialized engines for other LLM workload classes (code completion, summarization, agentic tool-calling), following PrefillOnly's template of identifying workload-specific properties and re-architecting around them; (b) co-design of LLM architectures and inference engines, since the paper shows that architectural choices (KV cache size, MLP intermediate dimension) directly affect the memory-throughput trade-off that hybrid prefilling exploits; (c) dynamic difficulty estimation or workload classification that routes requests to the appropriate specialized engine (generative vs. prefill-only) at serving time; (d) suffix cache offloading strategies (to CPU or remote memory) that preserve reuse potential while freeing GPU memory.

Research directions that become less attractive: (a) further optimizing general-purpose engines for prefill-only workloads through incremental tuning — the paper shows that structural changes (hybrid prefilling, suffix discarding, JCT-aware scheduling) provide order-of-magnitude improvements that incremental optimizations within the old architecture cannot match; (b) uniform chunked prefilling as a solution to long-input memory pressure — the paper demonstrates that asymmetric chunking (hybrid prefilling) is strictly better, since it avoids attention kernel degradation while achieving the same memory reduction; (c) treating continuous batching as universally necessary — the compute-bound diagnosis provides a clear criterion for when batching helps vs. hurts.

Follow-Up Research This Work Enables

Suffix cache offloading with predictive prefetching. PrefillOnly currently discards suffix KV caches, permanently losing reuse potential for future requests that partially share suffixes. The paper acknowledges (Section 9) that offloading to CPU via LMCache could preserve these caches, but offloading alone doesn't solve the scheduling problem: when should suffix caches be fetched back to GPU, and which ones? A concrete follow-up would implement a predictive prefetching scheduler that, upon scheduling a request, checks whether any of its tokens match CPU-offloaded suffix caches from recently completed requests, issues asynchronous PCIe transfers for matching caches, and overlaps the transfer with the ongoing request's execution. The experiment would measure cache hit rates and throughput on a workload with non-trivial suffix overlap (e.g., multi-turn dialogue where later turns share content with earlier turns, or document classification where boilerplate text appears across many documents) and compare against both suffix discarding (current PrefillOnly) and full KV cache retention (standard vLLM). The key question is whether the PCIe transfer latency (tens to hundreds of microseconds for large KV cache tensors) is consistently less than the recomputation cost — the paper provides no data on this trade-off.

Empirical validation of the JCT proxy across sequence lengths, models, and hardware. The paper uses cache-miss token count as a JCT proxy based on a single correlation measurement (0.987 Pearson on Qwen-32B-FP8, A100). A rigorous follow-up would characterize how this correlation degrades as sequence length increases, where attention's O(N²) cost becomes dominant over the linear MLP cost. The experiment would profile JCT at sequence lengths ranging from 1,000 to 200,000 tokens (spanning the MIL limits that PrefillOnly enables) across the three evaluated model/hardware combinations (Llama-8B/L4, Qwen-32B/A100, Llama-70B/H100), compute the correlation between JCT and cache-miss tokens at each length, and identify the crossover point where the linear proxy's error exceeds some threshold (e.g., 5% MAPE). If the crossover occurs within the evaluated workload lengths (60,000 tokens), the scheduler's correctness is directly impacted — the paper doesn't check this. If it occurs well beyond, the proxy is safe for current workloads. The follow-up would also evaluate whether a simple quadratic feature (n²_uncached) added to the regression model captures the attention term and improves prediction accuracy at long lengths.

Prefill-decode disaggregation with PrefillOnly on the prefill node. The paper notes (Section 9) that PrefillOnly could serve as the prefill node in a disaggregated serving architecture (DistServe, Zhong et al., 2024), since the prefill node's workload is also prefill-only. A concrete follow-up would integrate PrefillOnly as the prefill component in a disaggregated deployment, comparing end-to-end throughput and latency against a standard disaggregated setup (using vLLM's prefill instances) on mixed generative workloads (some requests generate many tokens, some generate few). The key question is whether PrefillOnly's memory management (hybrid prefilling, suffix discarding) provides enough prefill throughput improvement to increase the overall system's goodput under TTFT/TPOT SLOs, or whether the decode phase bottlenecks make the prefill improvement irrelevant. The experiment would use DistServe's goodput metric and SLO framework, sweeping the ratio of prefill-heavy to decode-heavy requests.

Hardware-conscious auto-tuning of the fairness parameter λ. The paper uses λ = 500 as the default fairness offset without discussing how it should scale with hardware throughput, workload characteristics, or latency SLOs. A concrete follow-up would develop an auto-tuning procedure: during an offline profiling phase, sweep λ across multiple orders of magnitude, run representative workload traces, measure the resulting mean and P99 latency, and select λ that minimizes mean latency subject to a P99 latency SLO constraint (e.g., P99 < 2× mean). The experiment would test whether the optimal λ is consistent across workload traces (post recommendation vs. credit verification) and hardware configurations (L4, A100, H100), or whether it must be tuned per deployment. If optimal λ varies substantially, the paper's uniform λ = 500 may be suboptimal for many deployments; if it's stable, the default is well-justified. The follow-up would also analyze the relationship between optimal λ and the ratio of input length variance to mean — intuitively, workloads with high length variance need stronger fairness offsets to prevent long requests from starving, and this relationship could be quantified.

Scheduling ablation with identical memory management, different scheduling policies. The paper attributes part of PrefillOnly's throughput advantage to continuous JCT calibration (avoiding prefix cache throttling, Figure 9) but never isolates the scheduling contribution from the memory management contribution. A concrete follow-up would implement multiple scheduling policies (FCFS, static SRJF sorted at arrival, SRJF with continuous calibration) within the same PrefillOnly engine (identical hybrid prefilling + suffix discarding), and measure throughput-latency curves and prefix cache hit rates for each policy on both workloads. This would decompose the total improvement into "how much comes from better memory management" (hybrid prefilling + suffix discarding vs. vLLM baselines) and "how much comes from better scheduling" (continuous calibration vs. FCFS within PrefillOnly). The experiment would also measure calibration overhead explicitly — CPU time per scheduling decision as a function of queue depth — to determine whether the scheduler itself becomes a bottleneck under high load.

FlashAttention-compatible hybrid prefilling kernel implementation. The paper implements hybrid prefilling at the torch.compile graph level, which provides portability but may not achieve the performance of hand-optimized CUDA kernels. A concrete follow-up would implement hybrid prefilling as a custom CUDA kernel (or integrate it into an existing attention framework like FlashAttention or FlashInfer) that fuses the chunked non-attention computation with the attention operation, minimizing kernel launch overhead and maximizing cache locality. The experiment would compare the throughput and peak memory of the kernel-level implementation against the torch.compile version across the three model/hardware combinations, measuring whether the compilation overhead is negligible (justifying the paper's portable approach) or substantial (motivating kernel-level optimization). This is a stress-test of the paper's design choice to prioritize portability over peak performance.

Practical Applications and Downstream Use Cases

High-volume recommendation and ranking pipelines. The most direct application is serving LLM-based recommendation systems like Meta's 360Brew (Firooz et al., 2025), where a single LLM evaluates hundreds or thousands of candidate items per user, each requiring a prefill-only forward pass over a long user profile (tens of thousands of tokens of browsing history). With PrefillOnly, an organization deploying such a system could serve the same QPS with roughly 4× fewer GPUs (Section 7, Figure 6), or serve 4× more users with the same GPU budget, because the engine avoids the memory-parallelization overhead that dominates standard LLM serving on long inputs. The prefix caching benefit is particularly strong here: all candidates for a given user share the same profile prefix, so continuous JCT calibration ensures these cache-hit requests are prioritized and the prefix cache remains warm. The paper's post recommendation dataset (50 candidates × 20 users × ~14,000-token profiles) directly models this scenario, and the throughput improvement at high QPS (Figure 9, ~35 req/s for PrefillOnly vs. ~10 req/s for tensor parallel) translates to concrete GPU cost savings.

Batch credit verification and fraud detection. Financial institutions processing credit applications or fraud checks in batch mode (e.g., nightly batch processing of thousands of applications) face the long-input, single-token pattern that PrefillOnly targets: each application might include 10 months of transaction history (40,000–60,000 tokens, as in the paper's credit verification dataset), and the LLM outputs a single approval/denial or fraud score token. The paper's results on the credit verification workload (Figures 6(e)–(h), 7(e)–(h)) show that PrefillOnly achieves higher throughput than any baseline on this workload — and on mid-range GPUs like the A100, it is the only non-parallelized solution that can even fit the requests in GPU memory (Table 2: PagedAttention and chunked prefill both fail at 40,000+ tokens). For a financial institution with compliance requirements that mandate on-premise GPU clusters (rather than cloud), PrefillOnly's ability to serve these workloads on fewer GPUs without NVLink — since NVLink-equipped servers are substantially more expensive — directly reduces capital expenditure. The ~1.5× throughput advantage over tensor parallel on H100 without NVLink (Figure 8, ~0.13 req/s vs. ~0.06 req/s) means fewer H100s need to be purchased.

Data labeling and annotation pipelines at scale. Organizations using LLMs to label large datasets (He et al., 2023; Zhang et al., 2023) — classifying millions of documents, annotating training data for smaller models, or filtering web-scale corpora — operate prefill-only workloads at enormous volume. Each labeling decision is a single token output (a class label, a binary flag, a quality score) given a potentially long input (the full document or example). PrefillOnly's throughput advantage over standard engines means the labeling pipeline can process the same dataset in a fraction of the time, or with a fraction of the GPU-hours. The sequential execution model (no batching, Section 6.1) is particularly well-suited to batch labeling: the arrival pattern is essentially "process this fixed set of N requests as fast as possible" with no latency SLOs or queueing dynamics, so the scheduler's continuous JCT calibration provides maximal benefit (ensuring prefix cache hits across requests with shared prefixes, like documents from the same source or with the same system prompt) without the fairness offset being needed (λ can be set to 0, maximizing throughput). The paper's results on post recommendation at high QPS (Figure 9) demonstrate this "saturated throughput" regime where PrefillOnly achieves ~35 req/s to the baselines' 10–25 req/s.

When to Prefer This Method

Prefer PrefillOnly over general-purpose LLM engines (vLLM, TensorRT-LLM, etc.) when:

• The workload is prefill-only — every request generates exactly one output token, and there is no decoding phase (Section 2.3). If requests sometimes generate more than one token, PrefillOnly's KV cache discarding will break decoding.
• Inputs are long (tens of thousands of tokens) and stress GPU memory, making suffix KV cache discarding valuable. At short input lengths that fit comfortably in GPU memory, the memory optimization provides no benefit.
• Throughput matters more than single-request latency at low load — PrefillOnly's single-GPU sequential execution has higher per-request latency than tensor-parallel alternatives when QPS is low (Figures 6(d), 7(d) show tensor parallel + NVLink beating PrefillOnly on latency at low QPS), but achieves higher maximum throughput.
• GPUs lack NVLink or other high-speed interconnects — the throughput penalty of tensor/pipeline parallelism is most severe without fast all-reduce communication (Figure 8), making PrefillOnly's avoidance of parallelism most valuable.
• There is prefix reuse across requests (e.g., all candidates for one user share a profile prefix) — continuous JCT calibration provides scheduling benefits that are absent in workloads with fully unique inputs per request.

Prefer tensor-parallel general-purpose engines when:

• Single-request latency at low QPS is the binding constraint, and NVLink is available. Figures 6(d) and 7(d) show tensor parallelism on H100 with NVLink achieving lower latency than PrefillOnly when QPS is below ~10.
• Maximum input length exceeds PrefillOnly's suffix-discarding MIL and parallelism is the only way to fit the KV caches. Table 2 shows tensor parallelism achieves 238,000 tokens on H100 vs. PrefillOnly's 97,000 — if the workload requires >100,000 token inputs on H100, PrefillOnly cannot serve it alone.
• The workload is not strictly prefill-only — if requests sometimes generate variable-length outputs, PrefillOnly's engine assumptions break. Use a general-purpose engine with continuous batching.

Prefer chunked prefill when:

• The workload is generative (multi-token output) but has long inputs that need memory management — chunked prefill is the appropriate technique for generative workloads (as in Sarathi-Serve), while PrefillOnly is specialized for single-token output. The paper's hybrid prefilling technique could be adapted for generative settings, but the suffix KV cache discarding and JCT-aware scheduling assume prefill-only workloads.