Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference

ArXiv: 2508.19559

🎯 Pitch

Conventional autoscaling fails catastrophically in disaggregated LLM serving, where independently scaling prefill and decode pools creates imbalance that a single, well-chosen metric—decode Tokens-Per-Second—can actually resolve. By coordinating scaling around this one signal and enforcing network-aware placement, HeteroScale boosts GPU utilization by 26.6 percentage points across tens of thousands of GPUs, proving that simplicity can tame the chaos when the metric truly reflects system-wide pressure.

1. Executive Summary

This paper introduces HeteroScale, a coordinated autoscaling framework purpose-built for Prefill-Decode (P/D) disaggregated LLM inference serving in large-scale, heterogeneous GPU clusters. Operating on ByteDance’s Seed Serving Platform across tens of thousands of GPUs and processing trillions of tokens daily, HeteroScale combines a topology-aware scheduler (enforcing network affinity via Deployment Group and RDMA Subgroup abstractions) with a metrics-driven scaling policy (using decode Tokens-Per-Second as the unified scaling signal for both prefill and decode pools to maintain architectural balance). The deployed system increases average GPU utilization by 26.6 percentage points and SM activity by 9.2 percentage points, while saving hundreds of thousands of GPU-hours daily—establishing that coordinated, single-metric autoscaling can robustly outperform both static provisioning and naive independent-per-pool approaches in production, but only when the chosen metric (here, decode TPS) is validated to reflect true workload pressure across both compute-bound and memory-bound serving stages.

2. Context and Motivation

The Core Problem: Autoscaling Breaks When You Disaggregate LLM Inference

The paper tackles a problem that emerges at the intersection of two major trends in LLM serving infrastructure. First, Prefill-Decode (P/D) disaggregation — splitting the two phases of transformer inference onto separate GPU pools — has become a dominant architectural pattern for improving throughput and cost efficiency. Second, production LLM traffic exhibits strong diurnal patterns, swinging between peak and off-peak demand by large multiples, which makes static provisioning economically untenable. The paper's central observation is that these two realities are in direct tension: P/D disaggregation introduces complex coordination dependencies that break traditional autoscaling approaches, and no existing system addresses the full set of challenges that arise when you try to autoscale a disaggregated LLM serving deployment at production scale.

This gap matters enormously in practical terms. The paper cites prior work showing that naive homogeneous GPU provisioning inflates cost per generated token by 41% compared to phase-aware heterogeneous deployment [58]. When you multiply this inefficiency across tens of thousands of GPUs processing trillions of tokens daily, the financial and environmental waste is staggering. Yet the alternative — running heterogeneous hardware with P/D disaggregation under traditional autoscalers — creates operational chaos rather than cost savings. The paper's own experience on ByteDance's Seed Serving Platform revealed that conventional approaches produce exactly the wrong behavior: decode pools appear saturated even at low load, independent scaling decisions on each pool create crippling architectural imbalance, and topology-oblivious placement strangles KV cache transfer bandwidth.

Why This Problem Is Hard: Three Interlocking Challenges

The paper identifies three specific challenges that make autoscaling P/D disaggregated services fundamentally different from autoscaling traditional microservices (Sections 1, 2.2):

Challenge 1: Heterogeneous Hardware Inefficiency. The prefill and decode phases have fundamentally different computational profiles: prefill is compute-intensive and benefits from high arithmetic throughput, while decode is memory-bandwidth-bound (its throughput is limited by how fast the GPU can read KV cache data from HBM). These distinct bottlenecks mean that no single GPU type is simultaneously optimal for both phases. A homogeneous approach — running both phases on general-purpose GPUs — essentially "over-provisions compute for decode or HBM bandwidth for prefill," as described in Section 1. The paper flags this as the root cause of the 41% cost inflation cited from prior work [58].

The scheduling implication is subtle but severe: when you do deploy heterogeneous hardware (different GPU types optimized for prefill vs. decode), you create a constrained placement problem where prefill instances must land on compute-optimized GPUs, decode instances must land on memory-optimized GPUs, and both must still satisfy the network proximity requirements essential for KV cache transfer. A homogeneous pool avoids the placement complexity but incurs the 41% cost penalty; a heterogeneous pool saves cost but creates a combinatorial scheduling problem that conventional autoscalers cannot solve.

Challenge 2: Network Bottlenecks. P/D disaggregation requires transferring the KV cache — which can be tens to hundreds of megabytes for long contexts — from prefill instances to decode instances over the cluster network. This is a bandwidth-intensive operation on a critical latency path: every millisecond spent on KV cache transfer adds directly to Time-To-First-Token (TTFT). The paper reports a concrete empirical finding that underscores why topology-oblivious scheduling is catastrophic: placing prefill and decode instances across different network switches reduces available KV cache transfer bandwidth by approximately 20% compared to co-locating them under the same switch.

This 20% bandwidth penalty translates directly to higher latency, and the degradation is not something that adding more instances can fix — it's a placement quality problem, not a capacity problem. Traditional autoscalers like Kubernetes HPA treat all nodes in a cluster as a flat resource pool. When scaling out, they place pods wherever capacity exists, with no awareness of whether the chosen nodes share a low-latency network domain with the pods they need to communicate with. In a disaggregated LLM deployment, this naivety creates a silent performance regression: the system scales out successfully from a resource-counting perspective, but SLOs degrade because the new instances are poorly placed.

Challenge 3: Architectural Imbalance. The prefill and decode pools are interdependent: if you have too few prefill instances relative to demand, the prefill queue backs up and TTFT skyrockets; if you have too few decode instances, the KV cache accumulates in memory on decode nodes while the prefill pool sits idle. The ratio of prefill-to-decode capacity (P/D ratio) is a critical architectural parameter that depends on model architecture, prompt length distribution, generation length distribution, and hardware characteristics.

The paper emphasizes a particularly insidious failure mode when scaling pools independently: decode GPU utilization is a misleading metric. Because the decode phase is memory-bound and must maintain KV cache in GPU memory, its GPU utilization (as reported by standard tools) stays high even under low workload — the memory pressure from stored KV caches inflates utilization readings. An independent autoscaler for the decode pool, using GPU utilization as its signal, would see "high utilization" and either refuse to scale in during low demand (wasting resources) or, worse, scale out unnecessarily because it cannot distinguish compute saturation from memory residency. Meanwhile, a prefill pool autoscaler using the same metric might correctly scale in during a lull, creating a P/D ratio skew where prefill becomes the bottleneck despite overall light traffic.

This is why "naive solutions cannot address" these challenges (Section 1): applying standard HPA to each pool independently solves none of the three problems. It doesn't help with heterogeneous hardware placement, it ignores network topology, and it actively creates architectural imbalance because it operates on metrics that mean different things in different pools.

Where Prior Approaches Fall Short

The paper identifies several categories of existing work and explains why each fails to address the full problem:

Traditional Kubernetes Autoscalers (HPA, VPA, KEDA). These are designed for stateless microservices where replicas are interchangeable, metrics (CPU, memory, request rate) scale roughly linearly with load, and there are no inter-replica network affinity constraints [29, 30, 28]. In the P/D disaggregated setting, replicas are not interchangeable (prefill and decode have different resource requirements and must be placed on appropriate hardware), metrics are not linear (decode GPU utilization shows the polarization documented in the paper), and KV cache traffic imposes strong network proximity requirements. The conceptual mismatch is fundamental — these autoscalers operate on a model of the world that simply does not apply to disaggregated LLM serving.

LLM-Specific Serving Systems (vLLM, TensorRT-LLM, SGLang, DistServe, SplitWise, Mooncake). These systems focus on single-node or cluster-level inference optimization — continuous batching, kernel fusion, memory management, and the mechanics of P/D disaggregation itself (Section 5.1). They answer the question "how do we serve LLM requests efficiently on a given set of GPUs?" but not "how many GPUs should we allocate to which phase as workload changes?" They provide the serving substrate but leave resource provisioning to external mechanisms. DistServe [57] and SplitWise [39] optimize the P/D split for throughput and cost but assume fixed resource pools, not dynamic scaling. Mooncake [41] addresses KV cache transfer but does not autoscale the pools. These systems are complementary to HeteroScale — they could serve as the inference engine underneath HeteroScale's scaling decisions — but they do not themselves solve the autoscaling problem.

Heterogeneous Resource Schedulers (Tiresias, Gandiva, HexGen, Mélange). These systems optimize GPU allocation across mixed workloads and hardware types, but they target training workloads (where job duration is measured in hours to days and placement decisions are made at job-submission time) rather than serving workloads (where demand changes in minutes and scaling decisions must be continuous). The time scales and decision semantics are fundamentally different. Moreover, these schedulers lack the P/D ratio maintenance logic and network affinity constraints that are specific to disaggregated inference — they can place jobs on appropriate hardware, but they don't understand that prefill and decode instances form a coupled pair that must scale in lockstep and reside within the same network domain.

Network-Aware Schedulers (Firmament, Paragon, Quincy, Sinbad). These systems incorporate network topology into placement decisions, which is a key requirement for HeteroScale. However, they were designed for data-intensive workloads (MapReduce, distributed storage) where the optimization target is data locality and inter-job bandwidth fairness. They lack the concept of P/D disaggregation, the specific KV cache transfer bottleneck, and the coordinated scaling requirement between coupled pools. The paper draws on their topology-awareness ideas but notes that these must be combined with LLM-specific scheduling constraints to be useful.

Learning-Based and Predictive Autoscalers (AutoScale, DeepScaling, Resource Central). These systems use ML models to predict workload and proactively adjust capacity. While sophisticated, they still rely on per-pool metrics and lack coordination mechanisms, meaning they would suffer the same architectural imbalance problems as HPA when applied independently to prefill and decode pools. They also introduce training and maintenance complexity that the paper explicitly seeks to avoid in its production design philosophy.

The Crucial Gap: No Metric Validation for P/D Disaggregated Autoscaling

Perhaps the most important gap the paper identifies — and the one that most distinguishes its contribution from prior work — is the absence of any systematic, data-driven analysis of what metrics actually work for autoscaling P/D disaggregated services. The paper frames this explicitly as the first large-scale empirical study of its kind (Section 1, Challenge 3):

"This calls for deeper investigation into metrics that better capture the true workload and performance characteristics of each pool."

Prior autoscaling systems, both general-purpose and LLM-specific, have either defaulted to hardware utilization metrics (CPU, GPU, memory) without questioning their validity, or adopted application-level metrics (request rate, latency) without analyzing their signal characteristics in the disaggregated setting. The paper's empirical investigation (Section 3.3.2, Figures 2, 6) reveals that common assumptions about metrics are systematically wrong in this context:

• GPU utilization and SM activity on decode nodes are persistently high regardless of load, making them useless as autoscaling triggers. The paper's traces show decode GPU utilization hovering at elevated levels even during deep workload valleys, because KV cache residency — not compute demand — dominates the utilization reading.
• Prefill GPU utilization is responsive but less sensitive than throughput-based metrics, making it a second-tier choice that would produce slower, less efficient scaling.
• Latency metrics (TTFT, TBT) exhibit a cliff-like non-linearity: they remain flat across a wide range of load levels and then spike abruptly near saturation. This makes them unsuitable for proportional control — you can't compute "we need 30% more instances because latency increased 30%" when latency doesn't move at all until you're already violating SLOs.
• Throughput metrics (prefill TPS, decode TPS) show high signal-to-noise ratios, track load proportionally, and respond quickly to traffic changes — but even here, there are nuances: KV cache hits make raw prefill TPS unreliable (cached prompts don't go through the full prefill pipeline), and different hardware configurations complicate per-pool normalization.

Without this analysis, any autoscaling system — no matter how sophisticated its control algorithm — would be built on a foundation of misleading signals. The paper's key empirical contribution is demonstrating that decode TPS is the only metric among eight candidates that simultaneously satisfies reliability, responsiveness, and deployability requirements across diverse services and modalities.

How This Paper Positions Itself

HeteroScale is positioned not as a fundamentally new algorithmic contribution to autoscaling theory, but as a systems contribution that identifies and solves the specific challenges that make existing approaches fail in production P/D disaggregated deployments. This is a deliberate framing choice grounded in operational reality.

The paper's claim to novelty rests on three pillars, each addressing a gap left by prior work:

1. The coordinated, single-metric scaling policy is novel not because proportional control is new, but because the paper provides the first rigorous empirical validation showing why it must be applied in a coordinated fashion using decode TPS as the unified signal, and why the alternatives (per-pool GPU utilization, latency-based control) are dangerous in this setting. The policy design is simple, but the metric selection process is new and transferable.

2. The Deployment Group and RDMA Subgroup abstractions are novel scheduling primitives that encode the specific constraints of P/D disaggregated serving — network affinity for KV cache transfer, heterogeneous hardware matching for prefill vs. decode computational profiles, and priority-based resource conservation — into a topology-aware scheduler. These abstractions fill the gap between generic topology-aware scheduling (which exists) and LLM-specific inference engines (which exist but don't schedule).

3. The production deployment and scale (tens of thousands of GPUs, trillions of tokens daily, cross-modality services) provides evidence that the approach works under real operational constraints — heterogeneous hardware, diurnal traffic, multiple service types — not just in simulation or controlled experiments. The 26.6 percentage point GPU utilization improvement and hundreds of thousands of GPU-hours saved daily are presented as the primary validation, not as aspirational projections.

The paper explicitly acknowledges the limitations of its scope: the P/D ratio is fixed (derived from offline pressure testing and historical data), the difficulty estimation for more dynamic ratio adaptation is left to future work (Section 6), and the system does not currently incorporate KV cache hit rates or inference engine internal statistics into scaling decisions. These are presented as natural extensions, not as gaps in the current contribution. The key insight is that doing coordination well with a simple, validated metric already yields substantial production gains — the marginal improvement from more sophisticated policies (online ratio adaptation, multi-metric fusion) is real but incremental compared to the step-change from fixing the fundamental coordination and metric selection problems.

3. Technical Approach

3.1 Reader Orientation

HeteroScale is a production autoscaling system that acts as the control plane for how many GPUs are allocated to the prefill and decode stages of an LLM serving deployment at any moment, and exactly where in the physical cluster those GPUs should come from. It solves the core problem that traditional autoscalers fail catastrophically on P/D disaggregated services because they (a) use metrics that are misleading in disaggregated settings, (b) scale prefill and decode pools independently despite their deep interdependence, and (c) ignore network topology despite KV cache transfers being bandwidth-critical — and the shape of HeteroScale's solution is exactly a coordinated, topology-aware policy engine that uses a single validated metric (decode TPS) to scale both pools in lockstep while satisfying hardware heterogeneity and network affinity constraints through novel scheduling abstractions (Deployment Groups and RDMA Subgroups).

3.2 Big-Picture Architecture (Diagram in Words)

HeteroScale is a three-layer system (Figure 1) that forms a closed control loop over a Kubernetes-based GPU cluster:

1. Autoscaling Layer with Policy Engine — the brain. It collects real-time metrics from running services, evaluates configured scaling policies, and emits scaling decisions: "Service S needs N more prefill instances and M more decode instances, maintaining a defined P/D ratio." This layer implements both a periodic (time-based) policy and a metrics-driven policy, with the latter being the primary innovation. The metrics-driven policy uses a proportional control algorithm driven by decode Tokens-Per-Second (TPS) as its unified signal, with a negative-feedback latency safety net.

2. Federated Pre-Scheduling Layer — the translator from "what to scale" to "where to place it." It takes abstract scaling decisions and maps them onto the physical topology of the cluster. It assembles a fresh topological resource tree (Figure 3) on each scheduling cycle, evaluates candidate placements against the service's network affinity requirements and hardware type constraints, and selects the optimal placement using a priority system encoded in Deployment Groups and RDMA Subgroups. This layer is where the P/D ratio maintenance logic, heterogeneous hardware matching, and network co-location enforcement actually execute.

3. Sub-cluster Scheduling Layer — the interface to Kubernetes. It translates the pre-scheduler's placement decisions into CRD (Custom Resource Definition) updates that the Kubernetes API server understands, and it exposes node information upward for topology assembly. This layer is deliberately thin; the paper treats it as infrastructure plumbing.

Information flows in a cycle: metrics flow upward from running pods to the policy engine → the policy engine computes desired instance counts → the pre-scheduler maps those counts to specific physical placements → the sub-cluster scheduler enacts those placements through Kubernetes → the new pod instances begin serving traffic and emitting metrics → the cycle repeats.

The monitoring component provides feedback to the policy engine, creating what the paper describes as "a closed-loop control system that continuously adapts to changing workload conditions" (Section 3.1).

3.3 Roadmap for the Deep Dive

• First, the Autoscaling Layer architecture and configuration management (Section 3.2): how the system is configured, how policies are bound to services, and how the evaluation loop works — the structural scaffolding that the metrics and algorithms plug into.

• Second, the metrics-driven scaling policy and its metric selection (Section 3.3.2): the empirical analysis that validates decode TPS as the primary signal, including the properties of each candidate metric class (throughput, hardware, latency) and why alternatives fail. This section also covers the two control algorithms (proportional control for linear metrics, negative feedback for non-linear metrics) and their complementary roles.

• Third, the periodic scaling policy (Section 3.3.1): the simpler, time-based alternative used for services with predictable patterns or experimental configurations that cannot yet use metrics-driven scaling.

• Fourth, the workload-centric policy curation pipeline (Section 3.3.3, Algorithm 1): how the system selects which policy to apply to a given service, including the pressure-testing step that determines the optimal P/D ratio.

• Fifth, the Federated Pre-Scheduling Layer (Section 3.4): the heterogeneous resource management framework, the Deployment Group and RDMA Subgroup abstractions, the P/D ratio maintenance mechanism, and the affinity-aware scheduling algorithm — the component that translates scaling decisions into actual pod placements while respecting hardware, topology, and balance constraints.

• Sixth, the system stability mechanisms (Section 3.6): the anti-flapping protections (cooling periods, hysteresis thresholds, dampening factors) and disaster recovery measures (soft scaling-in, state preservation, graceful degradation) that make the control loop safe for production.

• Seventh, the extension to disaggregated Mixture-of-Experts (MoE) (Section 3.4): how the Deployment Group abstraction adapts to services where the prefill stage itself is split into attention and feed-forward sub-components, requiring dual-ratio control.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a production systems paper whose core idea is that autoscaling P/D disaggregated LLM services requires three things that no existing system provides: (1) a single, validated metric that accurately reflects load pressure across both the compute-bound prefill and memory-bound decode stages, (2) a coordinated scaling mechanism that uses this metric to adjust both pools simultaneously while preserving their ratio, and (3) a topology-aware scheduler that encodes hardware heterogeneity, network affinity, and resource conservation into every placement decision. The paper's technical contribution is the concrete design and production validation of a system that delivers all three.

Autoscaling Layer and Configuration Management

The autoscaling layer (Section 3.2) provides the configuration interface and policy evaluation loop that form the system's control plane. It is divided into two sub-components: configuration management and the policy engine.

Configuration management is the operator-facing surface. It "empowers backend engineers and operators to define production service parameters aligned with desired SLOs and performance targets" (Section 3.2). For a given service, an operator specifies:

• The target metric to use for scaling (decode TPS is the default and recommended choice).
• The target per-instance value of that metric — for example, "each decode instance should handle X tokens per second at steady state."
• Scaling thresholds — how far above or below the target the observed metric can deviate before triggering a scale-out or scale-in action.
• Cooling periods — minimum time windows between successive scaling actions to prevent flapping.
• The P/D ratio — how many prefill instances should exist for every decode instance (derived offline through pressure testing).
• Service priority — a numerical priority that determines scheduling precedence when resources are constrained.
• Network affinity requirements — whether the service requires S1-level, S2-level, or cluster-level co-location of prefill and decode instances.
• Hardware type preferences — which GPU models are acceptable for prefill instances and for decode instances, potentially different for each phase.

These parameters are stored in the platform's configuration store and versioned, allowing operators to update scaling behavior without redeploying services.

The policy engine is the runtime evaluator. "The policy engine periodically evaluates these configurations across services, leveraging real-time metric observations to execute scaling actions" (Section 3.2). The paper specifies that evaluation is periodic — the engine runs on a fixed interval for each service, collecting the current value of the configured metric, comparing it against targets and thresholds, and emitting one of three decisions per component: ScaleOut, ScaleIn, or NoChange. The period is not explicitly specified in the paper but is implicitly tuned per service to balance responsiveness against stability (the cooling periods mentioned in Algorithm 2 provide a lower bound).

A key architectural choice: the policy engine does not make placement decisions. It only computes desired instance counts. The federated pre-scheduling layer handles the "where" — this separation means the policy logic can remain simple and the placement complexity can evolve independently.

The Metrics-Driven Scaling Policy: Why Decode TPS Wins

The paper's most important technical contribution is not an algorithm but an empirical validation of what metric to use as the scaling trigger. This analysis (Section 3.3.2) is conducted on production traces from an open-domain dialogue service using Doubao-Seed-1.6-thinking on hundreds of GPUs, and is replicated on a vision-language search service (Appendix 8) with the same qualitative findings.

The candidate metrics and their characteristics. The paper groups candidate metrics into three classes (Section 3.3.2):

Throughput metrics: Prefill TPS and decode TPS. These measure tokens processed or generated per second across all instances in the respective pool.

Hardware metrics: GPU utilization and SM (Streaming Multiprocessor) activity, each measured separately for prefill and decode instances. GPU utilization is the standard metric reported by NVIDIA drivers, representing the fraction of time the GPU was busy over a sampling interval. SM activity (measured via sm__cycles_active from NVIDIA profiling tools) is a finer-grained metric representing the fraction of cycles where at least one warp was active on any SM.

Latency metrics: Time-To-First-Token (TTFT) and Time-Between-Tokens (TBT). TTFT measures the delay from request arrival to the production of the first output token; TBT measures the average (or percentile) delay between successive output tokens during decode.

The paper analyzes the signal characteristics of each metric class using traces without autoscaling (Figure 2), revealing three distinct behavioral patterns:

Throughput metrics: high signal-to-noise, proportional response. The paper states that throughput metrics "show significant differences between peak and off-peak periods, with high signal-to-noise ratios, accurately reflecting service load conditions" (Section 3.3.2) and that they "respond quickly to traffic changes, enabling timely scaling operations." Figure 2(a) shows normalized TPS values tracking the load pattern closely. However, there is a critical nuance: "Due to the interference of the KV cache hit rate, prefill TPS measurements under caching are unreliable for autoscaling, thus only 'KV cache missed prefill TPS' and decode TPS are considered." When a request's KV cache is already resident (a cache hit), the prefill phase is partially or fully skipped — the tokens still appear in prefill TPS counters but don't represent real prefill compute demand. This makes raw prefill TPS unreliable as a scaling signal because the same reported throughput could correspond to very different actual GPU workloads depending on the cache hit rate.

Hardware metrics: polarized behavior between prefill and decode. Figure 2(c) and 2(d) reveal the critical finding that drives much of the paper's design. Prefill hardware metrics "respond sensitively to load changes with high signal-to-noise ratios" — prefill GPU utilization and SM activity rise and fall in correlation with demand. However, decode hardware metrics "maintain high values even under slight pressure, showing low sensitivity to load changes." The paper explains the mechanism: "the decode stage is memory-bound, with a large portion of its utilization coming from KV cache storage and data transfer operations. These memory operations keep hardware-level metrics at consistently high levels regardless of modest drops in workload."

This means that if you use decode GPU utilization as an autoscaling signal, the system will conclude the decode pool is saturated even during deep traffic valleys — it will never scale in, wasting resources. If you use it as the basis for a proportional control policy, you'll get nonsensical behavior because the metric barely moves when load changes.

Latency metrics: non-linear, cliff-like response. Figure 2(b) shows the most dangerous behavior for control purposes. TTFT and TBT remain "nearly flat" across a wide range of load levels and then "shoot upward abruptly" near saturation. The paper characterizes this as a "cliff-like transition" that "makes it impossible to scale resources proportionally from latency alone and instead demands a negative-feedback controller."

The fundamental problem with latency as a primary metric is the non-linear control surface: moderate increases in latency don't correspond to moderate capacity shortfalls — they either mean you're fine (latency is flat) or you're already in trouble (latency is spiking). A proportional controller trying to use latency would either do nothing (when latency is stable but the system is actually approaching saturation) or overreact violently (when latency spikes, producing wild oscillations). The paper also notes that among the two, "TBT provides a clearer signal, whereas TTFT suffers from a lower signal-to-noise ratio."

The hybrid policy design. From this analysis, a "clear pattern emerges: throughput and hardware-level metrics vary proportionally and predictably with changes in resource allocation, whereas latency metrics exhibit a non-linear, threshold-driven behavior" (Section 3.3.2). The paper's design response is a hybrid policy with two complementary algorithms:

1. Proportional control for linear metrics (throughput, prefill hardware utilization) — the primary scaling driver.
2. Negative feedback for non-linear metrics (latency) — a safety mechanism, not a primary driver.

This hybrid approach is the key architectural insight: don't try to make one algorithm handle fundamentally different signal types. Let the proportional controller handle the day-to-day scaling, and let the negative feedback controller act as an emergency brake that triggers only when SLOs are at risk.

Why decode TPS specifically? The paper selects decode TPS (rather than prefill TPS or prefill SM activity) as the primary metric for three practical reasons grounded in production constraints:

• Strong alignment with business objectives: tokens generated is the primary value metric for LLM services.
• Straightforward configurability: "decode TPS is preferred, as it can be uniformly distributed across instances. In contrast, prefill instances often differ in hardware configurations, introducing additional complexity in metric normalization" (Section 4.2.2). When prefill runs on heterogeneous hardware, a single "prefill TPS per instance" target is harder to define.
• Universal applicability: decode TPS is well-defined across all models and modalities, since every LLM has a decode phase that produces tokens.

An important implicit assumption: "within a given service, the input-output length distribution remains relatively stable, leading to consistent behavior in both decode TPS and prefill TPS. This correlation allows the two metrics to be used interchangeably in principle" (Section 4.2.2). This stability is what permits the single-metric approach to work — if prefill TPS and decode TPS were to diverge systematically (e.g., if prompt lengths suddenly doubled while generation lengths stayed constant), the P/D ratio would need adjustment, but the single-metric assumption would still hold for detecting overall load.

Proportional Control Algorithm (Algorithm 2)

The proportional control algorithm is the primary scaling driver for metrics that vary linearly with load. The paper presents it as Algorithm 2 (Appendix 9), and it operates as follows.

Inputs and state. The algorithm takes:

• $I_{curr}$: the current number of instances (for either prefill or decode — though in HeteroScale it is applied in coordinated fashion).
• $M_{curr}$: the currently observed value of the target metric (decode TPS).
• $M_{target}$: the target per-instance value of that metric (e.g., "each decode instance should sustain X tokens/second").
• $\theta_{out}$ and $\theta_{in}$: scaling thresholds for scale-out and scale-in, expressed as fractions above/below 1.0 (e.g., $\theta_{out} = 0.2$ means "scale out when observed metric is 20% above target").
• $C_{out}$ and $C_{in}$: cooling periods (minimum time between consecutive scale-out and scale-in actions).
• $T_{last}$: the timestamp of the last scaling action.

Step 1: Compute expected instance count. The algorithm computes:

$I_{expected} = I_{curr} \times \frac{M_{curr}}{M_{target}}$

where $I_{expected}$ is the number of instances that would be needed to bring the per-instance metric to exactly $M_{target}$, assuming perfect linear scaling, $I_{curr}$ is the current instance count, $M_{curr}$ is the observed aggregate metric value, and $M_{target}$ is the desired per-instance metric value.

What it computes: the product of current instances and the ratio of observed-to-target metric. If $M_{curr} > M_{target}$ (the system is overloaded — each instance is handling more than the target), the ratio exceeds 1 and $I_{expected} > I_{curr}$ (scale out). If $M_{curr} < M_{target}$ (the system is underloaded), the ratio is below 1 and $I_{expected} < I_{curr}$ (scale in). The result is a real-valued number that is rounded in the implementation.

Why this form: the computation assumes throughput scales linearly with instance count, which is a reasonable approximation for LLM inference when instances are independent and load-balanced. The ratio form normalizes the metric so the same scaling thresholds work across services with different $M_{target}$ values. A simpler approach — e.g., "if metric > threshold, add N instances" — would require tuning N per service and would not naturally produce larger adjustments for larger imbalances.

Step 2: Compute the scaling ratio. The algorithm computes:

$R = \frac{I_{expected}}{I_{curr}}$

where $R$ is the scaling ratio — the factor by which the current instance count should be multiplied.

What it computes: a scalar that is 1.0 at perfect equilibrium (observed matches target), > 1.0 when scaling out is needed, and < 1.0 when scaling in is needed. The magnitude of the deviation from 1.0 indicates how far the system is from target.

Why this form: $R$ is directly comparable to the thresholds $(1 + \theta_{out})$ and $(1 - \theta_{in})$, enabling a simple rule: if $R$ is more than $\theta_{out}$ above 1, scale out; if $R$ is more than $\theta_{in}$ below 1, scale in. This creates a deadband $[1 - \theta_{in}, 1 + \theta_{out}]$ where no action is taken, preventing the system from oscillating around the target.

Step 3: Check cooling periods and emit decision. The algorithm checks whether sufficient time has elapsed since the last scaling action:

if R > 1 + θ_out and cooling ≥ C_out then
    return (ScaleOut, I_expected)
else if R < 1 - θ_in and cooling ≥ C_in then
    return (ScaleIn, I_expected)
else
    return (NoChange, I_curr)

The cooling period is computed as $cooling = CurrentTime() - T_{last}$. If a scaling decision is warranted but the cooling period has not elapsed, the decision is deferred to the next evaluation cycle.

The crucial coordination step. The paper emphasizes that "the scaling signal from one component (e.g., decode TPS) is used to calculate the required capacity for both the prefill and decode pools, strictly enforcing the target P/D ratio" (Section 3.3.2). This means the algorithm does not run independently on each pool. Instead:

1. The observed decode TPS is used to compute $I_{expected}^{decode}$ — the total desired decode instances.
2. The P/D ratio $r_{opt}$ (derived from pressure testing) is then applied: $I_{expected}^{prefill} = I_{expected}^{decode} \times r_{opt}$.
3. Both pools are scaled simultaneously to their respective targets.

This coordinated application is what "transforms a simple algorithm into a powerful mechanism for maintaining architectural integrity" (Section 3.3.2). If instead each pool were scaled independently — using, say, prefill TPS for the prefill pool and decode TPS for the decode pool — any divergence in the two metrics (caused by changing prompt/generation length distributions or different hardware configurations) would create ratio drift, where one pool gradually becomes over- or under-provisioned relative to the other.

Negative Feedback Algorithm (Algorithm 3)

The negative feedback algorithm serves as a safety mechanism for latency metrics that exhibit non-linear, cliff-like behavior. The paper presents it as Algorithm 3 (Appendix 9).

Inputs and state. The algorithm takes:

• $I_{curr}$: current instance count.
• $L_{curr}$: current observed latency (TTFT or TBT).
• $L_{target}$: target latency (the SLO).
• $\alpha_{out}$, $\beta_{out}$, $\gamma_{in}$: threshold multipliers for different severity levels.
• $C_{out}$, $C_{in}$: cooling periods.
• $T_{last}$: last scaling timestamp.

Step 1: Tiered threshold evaluation. Unlike the proportional controller, the negative feedback controller does not compute a desired instance count from a continuous function. Instead, it evaluates whether the observed latency has crossed predetermined severity thresholds and applies a fixed-percentage adjustment:

• Severe breach: If $L_{curr} \geq L_{target} \times \alpha_{out}$ (where $\alpha_{out}$ is a multiplier greater than 1, e.g., 1.5 or 2.0), the system triggers a large, urgent scale-out — $I_{expected} = I_{curr} \times 1.2$ (a 20% increase).
• Moderate breach: If $L_{curr} \geq L_{target} \times \beta_{out}$ (where $\beta_{out} < \alpha_{out}$, e.g., 1.2 or 1.3), the system triggers a small, cautious scale-out — $I_{expected} = I_{curr} \times 1.1$ (a 10% increase).
• Under-utilization: If $L_{curr} \leq L_{target} \times \gamma_{in}$ (where $\gamma_{in}$ is a multiplier less than 1, e.g., 0.5 or 0.7), the system triggers a conservative scale-in — $I_{expected} = I_{curr} \times 0.95$ (a 5% decrease).

What it computes: a fixed-step adjustment rather than a proportional one. The adjustment magnitude is predetermined (1.2, 1.1, or 0.95 of current instances) rather than being computed from the ratio of observed to target. The tiered thresholds create a "deadband" between $\gamma_{in}$ and $\beta_{out}$ where latency is acceptable and no action is taken.

Why this form: the fixed-step design directly addresses the fundamental problem with latency-based control — the cliff-like non-linearity. A proportional controller would compute $I_{expected} = I_{curr} \times L_{curr} / L_{target}$, but because latency is near-flat across a wide load range, small increases in $L_{curr}$ would produce negligible scaling responses until suddenly, as the system hits saturation, a tiny load increase would produce a massive latency spike and a correspondingly massive (over-reactive) scaling decision. The fixed-step approach avoids this: it only acts when latency is already in a dangerous range, and it acts with a predetermined, conservative magnitude that prevents the system from overshooting the target.

Step 2: Cooling period check. As with the proportional controller, the negative feedback controller checks whether sufficient time has elapsed since the last scaling action before executing a decision. The cooling period check is performed after computing $I_{expected}$ (not before deciding whether action is needed), ensuring that the decision logic always reflects current conditions even if execution is deferred.

The step-based safety net philosophy. The paper explicitly frames this algorithm as "a safety mechanism rather than a primary scaling driver" (Section 3.3.2). Its purpose is not to optimize efficiency but to prevent SLO violations that the proportional controller (driven by decode TPS) might miss — for instance, if the P/D ratio drifts temporarily and prefill becomes a bottleneck before TPS reflects it, or if an unexpected workload shift causes latency to degrade before the proportional controller can respond to the throughput change.

Practical complexity concern. The paper is candid about the operational difficulty of tuning this controller: "the nonlinear relationship between latency and resource allocation necessitates a multi-tier feedback-driven scaling mechanism. This design inevitably introduces numerous hyperparameters... achieving a balance between meeting SLOs and maximizing throughput demands that the system operates within a narrow and highly sensitive configuration range, which further complicates the parameter space. In practice, tuning so many interdependent parameters is prohibitively difficult in large-scale production environments" (Section 4.2.2). This practical difficulty is cited as the primary reason latency-based control is used only as a safety mechanism, not as the primary scaling driver.

Periodic Scaling Policy

The periodic scaling policy (Section 3.3.1) is the simpler, time-based alternative to the metrics-driven policy. It "adjusts resources based on time-of-day patterns, enabling proactive scaling based on expected workload patterns."

The policy is defined as a schedule — a mapping from time windows to static target instance counts and P/D ratios. An operator might specify: "from 00:00 to 06:00, maintain 20 prefill and 100 decode instances (ratio 1:5); from 06:00 to 10:00, scale up to 60 prefill and 300 decode; from 10:00 to 14:00, maintain 40 prefill and 200 decode; etc." The system automatically adjusts capacity at the scheduled transition times, scaling out before the expected load increase and scaling in after the expected decrease.

The paper states that "in production environments, periodic scaling is employed for services that operate under specific constraints or involve experimental configurations, which are not amenable to metrics-driven scaling policies" (Section 3.3.1). This includes services where:

• The optimal metric target has not yet been determined (e.g., new model deployments still undergoing pressure testing).
• The service has unusual traffic patterns that don't fit the standard diurnal model (and the metrics-driven policy's thresholds haven't been tuned yet).
• The service operates under resource allocation agreements that require predictable, fixed-schedule provisioning.

The periodic policy manages a minority of the GPU fleet — the paper reports (Section 4.3) that the TPS-based metrics-driven policy manages 64% of the total GPU fleet under HeteroScale's control, implying the periodic policy and other configurations cover the remaining 36%.

The key advantage of periodic scaling is predictability and simplicity: no metrics need to be collected, no thresholds tuned, no worry about signal quality. The key disadvantage is inefficiency: static schedules cannot adapt to day-to-day variations in demand, special events, or gradual workload drift. The paper quantifies this gap in production: "The TPS-based policy delivered a 10.0 percentage points higher GPU utilization and an 11.1 percentage points higher SM activity compared to the periodic policy" (Section 4.3).

Workload-Centric Policy Curation (Algorithm 1)

The paper introduces a systematic process for selecting and configuring scaling policies per service, formalized as Algorithm 1 (Section 3.3.3). This addresses the practical question: given a new service with a new model and workload profile, which policy should we use and what P/D ratio should we set?

The pipeline has three stages:

Stage 1: Pressure testing to determine optimal P/D ratio. The pressure test takes a service $\mathcal{S}$ and its workload profile $\mathcal{W}$ (including input length distribution, output length distribution, and SLO constraints) and empirically determines two outputs:

$(r_{opt}, \hat{m}) = \text{PressureTest}(\mathcal{S}, \mathcal{W})$

where $r_{opt}$ is the optimal P/D ratio that maximizes throughput while maintaining SLO compliance, and $\hat{m}$ is the expected per-instance metric value ($M_{target}$) observed during the test.

The paper does not specify the exact pressure testing methodology in detail, but the results in Figure 4 (showing throughput vs. P/D ratio curves for two services with distinct input-output length distributions) indicate that the test involves sweeping across P/D ratios under controlled load and measuring the maximum sustainable throughput before SLO violations occur. The experimental setup described in Section 4.1 uses "16 nodes, each equipped with eight H20 GPUs" and tests services with different input-output length ratios (8.5 for Service A, 11 for Service B) and different SLO targets.

Why this is necessary: the optimal P/D ratio is not a fixed property of the model — it depends on the workload distribution. The paper reports that in production experience, the optimal ratio "spans a considerable range—from 1P/5D to 9P/1D—depending on the input-output length distribution, hardware configurations, and SLO priorities" (Section 4.1). A service with short prompts and long generations will be decode-bound and need proportionally more decode instances; a service with very long prompts and short completions will be prefill-bound and need the opposite. No single ratio works for all services.

Stage 2: Simulate candidate policies. For each candidate policy $p$ from the set of available policies $\mathcal{P}$ (which might include TPS-based proportional control, SM-activity-based control, latency-based negative feedback, and periodic scheduling), the system simulates its behavior under the baseline conditions determined by the pressure test. The paper states this is a "simulation" (Algorithm 1 uses SimulatePolicy), though the exact simulation framework is not described — it likely involves replaying workload traces against a model of the service's performance characteristics.

The simulation produces a score for each candidate policy, and the optimal policy is selected as:

$p_{opt} = \arg\max_{p \in \mathcal{P}} score[p]$

Stage 3: Output the configuration. The pipeline returns a tuple $(p_{opt}, r_{opt}, \hat{m})$ — the selected policy, the optimal P/D ratio, and the expected metric target. These become the configuration parameters plugged into the autoscaling layer's configuration management system.

Why this exists. The policy curation pipeline recognizes that the space of possible policies and configurations is large, and manual tuning would be error-prone and inconsistent across services. By standardizing the pressure-test-and-simulate pipeline, the platform can onboard new services with minimal operator intervention — the operator provides the workload profile and SLOs, the pipeline discovers the optimal ratio and selects the best policy, and the resulting configuration is deployed automatically. This is particularly important at ByteDance's scale, where "tens of thousands of GPUs across numerous services" (Section 4.3) means manual per-service tuning is operationally infeasible.

Heterogeneous Resource Management Framework

The federated pre-scheduling layer (Section 3.4) is responsible for translating abstract scaling decisions into concrete pod placements on physical hardware. The first component is the heterogeneous resource management framework, which encapsulates the logic for matching prefill and decode instance requirements to available GPU types.

The resource matching problem. Not all GPUs are suitable for all roles. The prefill phase is compute-intensive and performs best on GPUs with high arithmetic throughput (high TFLOPS, many SMs). The decode phase is memory-bandwidth-bound and performs best on GPUs with high HBM bandwidth. In a heterogeneous cluster containing multiple GPU types (the paper mentions NVIDIA H20 and L20 as examples), the scheduler must:

• Place prefill instances on compute-optimized GPUs.
• Place decode instances on memory-bandwidth-optimized GPUs.
• Respect the service's affinity constraints (prefill and decode must be within the same network domain for KV cache transfer).
• Avoid consuming scarce, high-priority resources for services that don't need them.

Algorithm 4 (the scheduling algorithm) handles the resource matching through a priority-based allocation loop, described below. The framework tracks available resources by GPU type, organized into the topological resource tree (Figure 3), and evaluates each placement candidate against the service's hardware type preferences.

The pressure test's role in hardware selection. The paper implies but does not explicitly detail that the pressure testing step (Algorithm 1) determines not only the optimal P/D ratio but also the hardware configuration — which GPU types produce acceptable performance for each phase. This information feeds into the service's configuration as hardware type preferences, which the scheduler then enforces.

Deployment Group Abstraction for Network Affinity

The Deployment Group is HeteroScale's central scheduling abstraction for managing the coupling between prefill and decode instances. It is a "logical container for the prefill and decode roles of a single service" (Section 3.4) with two defining characteristics.

Shared scheduling domain. All instances within a Deployment Group "are bound by a common network affinity constraint" (Section 3.4). This means the scheduler guarantees that every prefill and decode instance belonging to the same Deployment Group resides within the same network domain. The specificity of this constraint is configurable per service:

• S2-level affinity: For high-performance services, all instances in a group must be placed under the same S2 aggregation switch. This provides the lowest-latency, highest-bandwidth interconnect for KV cache transfers.
• S1-level affinity: For the most demanding heterogeneous configurations, the affinity can be tightened to the S1 level — all instances must be under the same S1 switch, which connects machines within a single rack.
• Cluster-level affinity: For services with less stringent networking needs, the constraint can be relaxed to the physical cluster level, allowing instances to be placed anywhere within the same cluster.

The paper emphasizes that this flexibility is not just about performance — it's about resource efficiency. Services that don't need tight affinity should not consume the limited capacity within a single switch, because that capacity is scarce and should be reserved for services that genuinely require it.

Independent scaling roles. Within a Deployment Group, "the prefill and decode roles function as independent deployment units that can be scaled out or in separately, subject to the system's P/D ratio maintenance logic" (Section 3.4). This means the autoscaling policy can decide to add more prefill instances without necessarily adding decode instances, as long as the resulting ratio stays within acceptable bounds. The Deployment Group abstraction does not constrain the number of instances of each role — only their placement domain.

The Deployment Group as a scheduling decision unit. When the policy engine requests a scale-out, the pre-scheduler must decide whether to expand an existing Deployment Group (placing new instances within the same network domain as the service's existing instances) or create a new Deployment Group in a different domain. This decision is guided by resource availability and priority:

• Expand existing group: If the service's current network domain has sufficient capacity (of the required GPU types), new instances are placed there. This preserves network proximity and avoids fragmenting the service across multiple domains.
• Create new group: If the current domain is resource-constrained (no more suitable GPUs available under that switch), the scheduler provisions a new Deployment Group in a different, compatible domain. The service then has instances in two (or more) network domains, with KV cache traffic localized within each group.

This two-tier scaling strategy prevents a service from being "bottlenecked by local resource exhaustion" (Section 3.4).

Co-location as a first-class constraint. The paper's emphasis on Deployment Groups as a scheduling primitive reflects a key production insight: in P/D disaggregated serving, where you place instances is almost as important as how many you place. A system that scales out successfully but places the new prefill instance on a different switch from its decode counterpart has degraded performance despite increased capacity — a silent failure mode that resource-counting autoscalers cannot detect.

RDMA Subgroup Priority System

The RDMA Subgroup is a complementary abstraction that encodes hardware priority into the scheduling decision. While Deployment Groups define what network proximity a service needs, RDMA Subgroups define which available resources to prefer when satisfying that need.

What an RDMA Subgroup represents. An RDMA Subgroup is "a logical collection of one or more S1/S2 switches, classified into a distinct priority tier based on the hardware they contain" (Section 3.4). It is a way of labeling portions of the cluster topology according to their value.

The three priority tiers are defined as follows (from lowest to highest priority):

• Low Priority: S2 Homogeneous GPU Subgroups. These contain S2 switches where every underlying GPU is of the same type. All machines under that S2 switch have, for example, only H20 GPUs, or only L20 GPUs. These are the most common configurations and are suitable for the widest range of services. They have no special hardware diversity value.

• Medium Priority: S2 Heterogeneous GPU Subgroups. In these, an S2 switch manages a mix of GPU types, but each underlying S1 switch is homogeneous. For example, the S2 switch connects to one S1 switch with H20 GPUs and another S1 switch with L20 GPUs, but no single S1 switch mixes types. This allows prefill instances to be placed on one GPU type and decode instances on another, all within the same S2 domain — enabling heterogeneous optimization while maintaining S2-level network affinity.

• High Priority: S1 Heterogeneous GPU Subgroups. These are the most valuable pools. An S1 switch directly connects machines with different GPU types — for instance, a rack containing 4 nodes with H20 GPUs and 4 nodes with L20 GPUs, all under the same S1 top-of-rack switch. This enables the tightest possible affinity (S1-level) while still allowing prefill and decode to use different, specialized hardware.

How priority guides scheduling. When evaluating candidate placements for a scale-out request, the scheduler sorts compatible RDMA Subgroups by priority and selects the one that "satisfies the service's affinity constraints while minimizing the consumption of high-priority resources" (Section 3.4). The specific logic:

• If a service has low-affinity requirements (cluster-level co-location suffices), the scheduler strongly prefers low-priority subgroups. This conserves high-priority S1 and S2 heterogeneous subgroups for services that genuinely require tight proximity.
• If a service requires heterogeneous hardware under tight affinity (e.g., different GPUs under one S1 switch), the scheduler filters for high-priority subgroups that can satisfy this constraint.

The priority system is the mechanism that prevents a tragedy of the commons: without it, a service that only needs cluster-level co-location might accidentally consume scarce S1 heterogeneous capacity (simply because that was the first available resource found), leaving a high-affinity service with no suitable placement.

Conservation as a scheduling objective. The paper frames this as a conservation strategy: "scarce, high-performance resource pools are reserved for the workloads that need them most, optimizing global cluster efficiency" (Section 3.4). This is a departure from simple first-fit or best-fit scheduling, which would optimize for immediate placement speed rather than long-term resource preservation.

P/D Ratio Maintenance

Maintaining the configured P/D ratio during scaling operations is one of the core responsibilities of the pre-scheduling layer. The paper describes both the proactive mechanism that prevents ratio drift during scaling and the reactive mechanism that handles transient imbalances.

Proactive: simultaneous scaling. The primary mechanism is simple: "the prefill and decode instances are always scaled in or out simultaneously" (Section 3.4). When a scale-out decision is made, the system immediately creates both the required prefill and decode instances as a single atomic operation. The paper explains the rationale:

"This approach is intended to prevent a scenario where, after either prefill or decode instances are successfully scaled out individually, the other one fails to scale due to insufficient resources, thereby avoiding the issue of an imbalanced P/D ratio."

The failure scenario this prevents is: the policy engine requests 10 new prefill and 5 new decode instances (maintaining a 2:1 ratio). If these were independent operations, the prefill scale-out might succeed (acquiring 10 GPUs from available capacity) while the decode scale-out fails (no suitable GPUs available), leaving the system with 10 extra prefill instances and a severely skewed ratio. Simultaneous scaling ensures that either both succeed or neither does.

Reactive: service discovery gating. Even with simultaneous scaling, transient imbalances can occur because "prefill and decode instances may start out of order due to differences in configurations and startup strategies" (Section 3.4). For example, if prefill instances boot faster than decode instances (because they have lighter initialization requirements), the prefill pool temporarily has more ready instances than the decode pool.

To handle this, the system implements a soft P/D ratio maintenance mechanism at the service discovery level (Section 3.4). The logic:

1. After a scale-out, new instances register with service discovery as they become ready.
2. If the ratio of ready prefill instances to ready decode instances deviates significantly from the configured ratio, "service discovery for the role with a larger quantity will be suspended."
3. Registration resumes "only after the other instances from the other role complete their service discovery registration and instances in ready state recover to a tolerable P/D ratio."

The paper provides an explicit example of why this matters: "the occasional spikes observed in TTFT are caused by temporary P/D ratio imbalances that can occur during scaling operations" (Section 4.3). By gating service discovery, the system prevents traffic from reaching an imbalanced configuration, avoiding the TTFT degradation that would otherwise occur.

Ratio calculation and smoothing. The paper mentions but does not detail the ratio calculation logic. It states that the system "takes into account the current number of prefill and decode instances, the target P/D ratio, a scaling threshold, and historical workload data. It first calculates the current P/D ratio and checks if an adjustment is needed based on the threshold. If an adjustment is required, it calculates the optimal instance counts based on workload data and applies a smooth transition to avoid abrupt changes" (Section 3.4). The smoothing mechanism is not specified, but its purpose — preventing abrupt changes — suggests it likely involves gradual ramping rather than immediate jumps to new instance counts.

Fixed vs. dynamic ratios. The paper acknowledges that the current system uses a fixed P/D ratio derived from pressure testing, and identifies dynamic ratio adaptation as future work (Section 6). The fixed ratio is "derived from service pressure tests and historical empirical data" (Section 3.4). The paper does not claim that the fixed ratio is optimal under all workload conditions — it claims that maintaining some consistent ratio (even if suboptimal) is far better than independent scaling, and that the ratio determined by pressure testing is good enough for the observed workload stability.

The Affinity-Aware Scheduling Algorithm (Algorithm 4)

The core scheduling loop is described in Algorithm 4 (Appendix 9). It is executed by the federated pre-scheduler on each scheduling cycle and translates pending scaling requests into specific pod-to-node assignments.

Step 1: Topology discovery. At the start of each cycle, the scheduler "builds a fresh topological resource tree" (Section 3.4, Figure 3). This tree provides a hierarchical view of all available GPUs and their network locations:

• VDCs (Virtual Data Centers) — the highest-level organizational unit, representing a logical partition of the physical data center.
• Physical clusters — Kubernetes clusters within a VDC.
• S2 switches — high-level aggregation switches within a physical cluster.
• S1 switches — top-of-rack or mini-pod switches within an S2 domain.
• Nodes — individual machines, each with a specific GPU type and count.

The tree encodes available capacity at each level, organized by GPU type. Building it fresh each cycle ensures the scheduler operates on an up-to-date view of cluster state, incorporating any changes from recently completed pod creations, deletions, or failures.

Step 2: Request sorting. All pending scaling requests generated by the autoscaling policy engine are "sorted, primarily by service priority" (Section 3.4). Higher-priority services are allocated resources first. The paper does not detail secondary sorting criteria (e.g., request size, service age, resource type) but the primary sort ensures that critical workloads receive resources before lower-priority ones during resource contention.

Step 3: Candidate evaluation (for scale-out). For each scale-out request in priority order, the scheduler identifies all valid placement options. A placement option is a specific Deployment Group (existing or new) and a specific allocation of pods within that group's network domain. The evaluation involves:

• Filtering compatible RDMA Subgroups based on the service's hardware type preferences (which GPU types are acceptable for prefill and decode) and its affinity constraints (what level of network proximity is required).
• Checking available capacity within each compatible subgroup — does the subgroup have enough unallocated GPUs of the required types to satisfy the request?
• Considering both existing and new Deployment Groups — can the request be satisfied by expanding one of the service's existing groups, or must a new group be provisioned?

Step 4: Priority-based selection. Each valid candidate placement is scored based on the priority of its associated RDMA Subgroup. The scheduler selects the candidate that "satisfies the service's affinity constraints while minimizing the consumption of high-priority resources" (Section 3.4). The selection logic implements a least-valuable-resource-first policy:

• For a service with low-affinity requirements, the scheduler prefers low-priority subgroups, even if this means creating a new Deployment Group in a less-optimal location rather than expanding an existing group that happens to reside in high-priority hardware.
• For a service requiring high-affinity and heterogeneous hardware, the scheduler filters to high-priority subgroups and places there.

This is non-trivial: the scheduler might choose to place a service's instances in a "worse" network location (e.g., a different S2 switch) to preserve scarce high-priority capacity, even though the service's existing instances are already in a "better" location. The trade-off is between per-service optimality and global cluster efficiency.

Step 5: Virtual allocation. Once a placement decision is made, the selected resources are "virtually deducted from the topological tree for the remainder of the cycle" (Section 3.4). This is an in-memory reservation within the scheduling cycle — the resources are not yet physically allocated, but they are marked as consumed so that lower-priority requests within the same cycle cannot claim them. This prevents the scheduler from over-committing resources within a single scheduling run.

Scale-in handling. Scale-in requests are simpler. The scheduler "selects one or more of the service's Deployment Groups to scale in, typically targeting those occupying high-priority resource pools to free them up" (Section 3.4). The priority-biased scale-in ensures that high-value hardware is returned to the available pool first. Released resources are not immediately available for other requests within the same cycle — "the entire resource view is rebuilt from the underlying cluster state at the beginning of the next scheduling cycle" (Section 3.4) — preventing double-allocation.

Why a fresh tree each cycle? The paper does not use incremental updates to the resource view. Instead, each scheduling cycle starts from scratch by querying the cluster state. This design choice trades some overhead (the cost of reconstructing the topology) for strong correctness guarantees — the scheduler never operates on stale state that might have been invalidated by pod completions, failures, or manual interventions between cycles.

Extending to Disaggregated Mixture-of-Experts (MoE)

The paper briefly describes how HeteroScale's abstractions generalize to the more complex case of disaggregated Mixture-of-Experts (MoE) serving, where the prefill phase itself is split into attention (attn) and feed-forward network (ffn) sub-components [58].

Dual-ratio control. In the MoE setting, the Deployment Group abstraction is adapted to contain three roles: prefill-attention, prefill-ffn, and decode. These are subject to two ratio constraints:

• An attn-to-ffn ratio within the prefill replicas — ensuring that the attention and expert computation are balanced.
• A prefill-to-decode ratio between the aggregate prefill capacity and decode capacity — the same P/D ratio constraint as in the standard case.

The scheduler co-locates the prefill-attention and prefill-ffn instances under a high-affinity S1 switch (since they must communicate frequently during prefill), and places the entire prefill-decode pair under a common S2 switch. This "hierarchical scheduling enables dual-ratio control" (Section 3.4).

The paper does not provide experimental results for the MoE extension but frames it as a demonstration that the Deployment Group and RDMA Subgroup abstractions are general enough to capture more complex disaggregation patterns.

System Stability Mechanisms

The paper devotes Section 3.6 to the stability mechanisms that prevent the control loop from causing harm in production. These are not afterthoughts — they are essential to making autoscaling safe at scale, and the paper treats them as first-class design elements.

Anti-flapping mechanisms. Rapid oscillation between scaling in and scaling out — "flapping" — wastes resources (instances that are continuously created and destroyed) and destabilizes the system (each scale action causes a transient disturbance). HeteroScale mitigates this through three complementary mechanisms:

• Cooling periods: "enforcing a minimum interval between scaling actions to prevent rapid oscillations" (Section 3.6). After any scale-out action, the system enters a cooling period $C_{out}$ during which no further scale-out is allowed. Similarly, after scale-in, a cooling period $C_{in}$ applies. Importantly, these are separate — a scale-in cooling period does not prevent a scale-out, and vice versa — allowing the system to reverse direction immediately if the situation changes, while preventing repeated actions in the same direction.

• Hysteresis thresholds: "using different trigger points for scaling out and scaling in, creating a buffer zone that promotes stability" (Section 3.6). The proportional controller's thresholds $\theta_{out}$ and $\theta_{in}$ (Algorithm 2) create a deadband where no action is taken even though the metric is not exactly at target. For example, if $\theta_{out} = 0.2$ and $\theta_{in} = 0.1$, the system will scale out only when the metric exceeds 120% of target, but will scale in only when the metric falls below 90% of target — the range [90%, 120%] is a stability zone where no action occurs.

• Dampening factors: "applied to moderate the scale of adjustments, further smoothing the system's response to changing conditions" (Section 3.6). This likely means that the computed $I_{expected}$ is not applied directly — it might be averaged with the previous decision, limited to a maximum percentage change, or adjusted by a dampening coefficient. The paper does not specify the exact dampening mechanism but mentions it as part of the anti-flapping toolkit.

Disaster recovery measures. Beyond normal stability, the system includes mechanisms for handling failures and abnormal events:

• Soft scaling in: "instances identified for removal are withdrawn from service discovery but kept running. During this observation period, the system monitors SLOs such as latency in real time. If performance remains within targets, the instances are then terminated; however, if degradation is detected, they are reinstated immediately, avoiding the startup delay associated with provisioning new instances" (Section 3.6). This is a crucial safety mechanism: scale-in decisions are reversible without cost (the instances were still running, just undiscoverable), eliminating the risk of a scale-in that proves premature. The observation period is not specified but is implied to be long enough to detect SLO impact — likely on the order of minutes.

• State preservation: "the platform also preserves critical state information to enable fast resumption of normal operations after a failure" (Section 3.6). This ensures that if the autoscaling control plane itself crashes, it can recover its view of the world (desired instance counts, pending scaling actions, cooling period timers) without having to rebuild from scratch or, worse, take incorrect actions based on stale state.

• Graceful degradation: "when resources become constrained, it applies graceful degradation strategies, maintaining essential functionality while temporarily reducing non-critical services" (Section 3.6). This uses the service priority system: if the cluster cannot satisfy all pending scale-out requests, lower-priority services are denied resources first, preserving capacity for critical workloads.

Why these mechanisms are production-critical. The paper's framing is instructive: these are not optional optimizations but baseline requirements for any autoscaling system operating at production scale. Without cooling periods and hysteresis, even a correctly-designed controller will oscillate due to measurement noise, load balancer lag, and the inherent delay between scaling action and metric response. Without soft scaling in, a single misjudged scale-in (triggered by a transient load dip) could cause SLO violations that take minutes to recover from (due to instance startup time). Without priority-based graceful degradation, a resource crunch could take down critical services rather than gracefully shedding non-critical load. The paper's emphasis on these mechanisms reflects hard-won operational experience rather than theoretical design.

Summary of Design Choices and Their Justifications

• Single-metric coordinated scaling rather than multi-metric per-pool scaling: avoids the decode GPU utilization trap and prevents ratio drift; backed by empirical validation showing decode TPS is the only metric that works reliably across both pools.
• Proportional control for throughput, negative feedback for latency rather than a unified control law: acknowledges the fundamental difference between linear and non-linear metric behaviors; proportional control would be dangerously unstable on latency signals, and negative feedback would be unnecessarily conservative on throughput signals.
• Simultaneous scaling with P/D ratio enforcement rather than independent per-pool scaling: prevents the failure mode where one pool scales successfully and the other fails, creating architectural imbalance that cripples overall throughput.
• Deployment Group as a scheduling unit rather than individual pod placement: encodes the network affinity constraint (prefill and decode must co-reside in the same switch domain) as a first-class scheduling constraint; this constraint has no analog in traditional autoscalers but is critical for KV cache transfer performance.
• RDMA Subgroup priority system rather than simple first-fit or best-fit: prevents low-affinity services from consuming scarce high-priority hardware (S1/S2 heterogeneous subgroups) needed by high-affinity services; optimizes global cluster efficiency rather than per-service placement speed.
• Fresh topology tree each cycle rather than incremental state tracking: trades overhead for correctness; avoids bugs from stale state and provides a natural recovery mechanism if the scheduler crashes mid-cycle.
• Soft scaling in with observation period rather than immediate termination: makes scale-in decisions reversible; eliminates the risk that a premature scale-in causes an SLO violation that takes minutes to recover from.
• Fixed P/D ratio from offline pressure testing rather than online ratio adaptation: a deliberate simplification; the paper acknowledges this as a limitation and identifies dynamic adaptation as future work, but the production results demonstrate that maintaining even a fixed ratio is sufficient for substantial gains over independent scaling.
• Workload-centric policy curation pipeline rather than per-service manual tuning: standardizes the onboarding process and makes policy selection reproducible; essential for managing "tens of thousands of GPUs across numerous services" without operator burnout.

4. Key Insights and Innovations

Innovation 1: Autoscaling Signal Selection as an Empirical, Production-Grade Contribution

The paper's most distinctive intellectual contribution is not the control algorithm itself—proportional control is textbook—but rather the rigorous, data-driven refutation of default metric assumptions that underpin virtually all prior autoscaling systems. Before HeteroScale, the field operated on an implicit consensus that hardware utilization (GPU, CPU, memory) or simple request-rate metrics were sufficient signals for scaling decisions. Kubernetes HPA, KEDA, and research systems like DeepScaling and AutoScale all inherit this assumption. The paper demonstrates, with production traces and systematic comparison across eight candidate metrics (Figure 2, Figure 6), that this consensus is actively dangerous in P/D disaggregated settings because it produces decisions that are not merely suboptimal but qualitatively wrong.

The diagnostic move that makes this innovative is the polarization analysis. The paper shows that prefill and decode hardware metrics behave in opposite ways under the same workload conditions (Section 3.3.2, Figure 2), yet traditional autoscalers treat them as interchangeable signals. Decode GPU utilization remains high during load valleys because KV cache residency—not compute demand—dominates the utilization reading. An autoscaler using this signal would conclude the decode pool is saturated and refuse to scale in, wasting resources. Meanwhile, prefill GPU utilization tracks load proportionally, creating an asymmetry where the same metric class means fundamentally different things in different pools. No prior autoscaling work for LLM serving had characterized this polarization, and its discovery explains why naive per-pool scaling fails in ways that are not obvious from resource counting alone.

The latency cliff characterization is a second diagnostic contribution. The paper demonstrates that TTFT and TBT exhibit a near-flat response across a wide load range followed by an abrupt saturation spike (Figure 2b). This is not a minor non-linearity—it fundamentally breaks the assumptions of proportional-integral-derivative (PID) control, which requires that the controlled variable respond approximately linearly to control inputs in the operating regime. The paper's identification of this cliff-like behavior explains why latency-based autoscaling in LLM serving is "prohibitively difficult" to tune in production (Section 4.2.2), and provides a principled justification for relegating latency to a safety-net role rather than a primary driver.

What elevates this from "metric comparison" to a genuine innovation is its transferability. The paper replicates the analysis across an open-domain dialogue service (Figure 2) and a vision-language search service (Appendix 8, Figure 8), finding the same qualitative patterns. The authors state that "the same qualitative patterns hold for other workloads such as web search, long-form content understanding, real-time audio conversation, real-time video processing, and code generation" (Section 3.3.2). This cross-modality consistency suggests the polarization and latency-cliff phenomena are structural properties of the P/D disaggregation architecture itself—stemming from the compute-bound vs. memory-bound dichotomy—rather than artifacts of a particular model or service. The metric selection methodology (pressure test, trace replay with candidate metrics, comparative evaluation) is exportable to any P/D deployment, making it a methodological contribution as much as an empirical one.

The significance extends beyond raw performance gains. This innovation reframes autoscaling from a control-theory problem (tune the algorithm) to a measurement-theory problem (choose the right signal). The 10 percentage point GPU utilization advantage of TPS-based over periodic policies (Section 4.3) is impressive, but the deeper insight is that no control algorithm—no matter how sophisticated—can compensate for a misleading measurement. This has implications for the broader ML serving community: the paper implicitly argues that any new LLM serving system should begin by empirically validating its autoscaling signals against production traces, not by assuming what works for microservices will transfer.

Innovation 2: Coordinated Single-Metric Scaling as an Architectural Principle

The conceptual move that separates HeteroScale from prior work is the elevation of the P/D ratio from a configuration parameter to a first-class scheduling invariant enforced through coordinated, single-metric scaling. This is a fundamental shift in how to think about autoscaling disaggregated systems, not merely a different algorithm choice.

Prior work on P/D disaggregation (DistServe [57], SplitWise [39], Mooncake [41], P/D-Serve [25]) focused on optimizing the serving substrate—how to split the model, how to batch efficiently, how to transfer KV caches—but treated resource provisioning as an external concern. The implicit assumption was that a separate autoscaler (like HPA) could manage each pool independently, or that static provisioning was sufficient. The paper identifies why this assumption fails: the prefill and decode pools are coupled through a ratio that depends on workload characteristics (prompt length, generation length, hardware type), and independent scaling inevitably drifts from that ratio because the per-pool metrics that drive independent decisions have different noise characteristics, different signal-to-noise ratios, and different relationships to true load pressure.

The innovation is not that the ratio matters—that was known from prior work like Mooncake [41] and BurstGPT [46]—but that maintaining it requires a single scaling signal applied to both pools simultaneously. The paper's design choice to use decode TPS for both prefill and decode scaling decisions (Section 3.3.2) is counterintuitive on its face: why should the prefill pool be scaled based on a decode-phase metric? The answer lies in the empirical finding that throughput metrics are approximately proportional between phases when the input-output length distribution is stable (Section 4.2.2), making either pool's TPS sufficient to infer overall load. But the deeper insight is that using a single metric eliminates a coordination failure mode: if prefill TPS and decode TPS diverge slightly (due to measurement noise, cache effects, or minor workload shifts), independent controllers would make opposing decisions, causing the ratio to drift. A single signal, applied in lockstep, cannot drift because both pools always move together.

This is a control-theoretic insight masquerading as a systems design choice. In distributed control theory, independent controllers acting on partial observations of a coupled system are known to produce suboptimal and potentially unstable behavior—this is the classic "decentralized control" problem. The paper's solution—use a single observation to drive all actuators—is the control-theoretic equivalent of centralized control, and its superiority over decentralized (per-pool) control is predictable from theory. What the paper contributes is the empirical demonstration that this theoretical advantage is practically decisive in production LLM serving: the 26.6 percentage point GPU utilization gain (Section 4.3) is not from better tuning of per-pool controllers but from eliminating the coordination problem entirely.

The significance of this innovation is that it provides a design principle for any future system that serves disaggregated, coupled components: identify a single metric that reflects aggregate load across all components, validate it empirically against the per-component alternatives, and use it as the unified scaling signal. The paper's specific choice (decode TPS) is an instance of this principle, not the principle itself. The generalizability is what makes it an innovation rather than a configuration trick.

Innovation 3: Network Topology as a First-Class Scheduling Resource with Priority-Based Conservation

The Deployment Group and RDMA Subgroup abstractions (Section 3.4) represent a conceptual advance in how to think about resource scheduling for network-coupled distributed inference. Prior to this work, network-aware scheduling existed (Firmament [15], Paragon [12], Quincy [21], Sinbad [8]) but treated network topology as a performance optimization—place data and compute close together to reduce latency. HeteroScale reframes topology as a scarce resource that must be conserved through priority-based allocation, on par with GPU memory or compute capacity.

The key conceptual move is the recognition that co-location under a single network switch is a finite, allocatable resource with a clear quality hierarchy. Not all switch domains are equal: an S1 switch with heterogeneous GPUs (high priority) is fundamentally more valuable than an S2 switch with homogeneous GPUs (low priority) because it enables the tightest network proximity while still allowing hardware specialization. If a service that only needs cluster-level co-location consumes the last available S1 heterogeneous capacity, a high-affinity service is blocked—not because GPUs are exhausted, but because the right combination of GPU types within a single switch domain is exhausted.

This is a combinatorial resource constraint that has no analog in traditional autoscaling. Traditional schedulers track scalar quantities (CPU cores available, GPU count available, memory GB available). HeteroScale tracks a structured, hierarchical resource where the combinability of different GPU types within the same network domain is itself a resource. The RDMA Subgroup priority system encodes this structured resource as a tiered allocation policy, ensuring that scarce, high-value switch domains are consumed only by services that genuinely require their specific properties.

What makes this innovative is the conservation objective. Prior schedulers (Gandiva [50], Tiresias [17], Gavel) optimize for fairness, throughput, or job completion time. HeteroScale optimizes for preserving scarce combinability: it will deliberately place a low-affinity service in a "worse" network location to save the "better" location for a service that needs it, even if the better location has available capacity. This is a form of reservation scheduling where future demand (from high-priority services not yet requesting resources) influences current placement decisions. The priority system is the mechanism that makes this future-aware placement tractable without explicit demand prediction.

The evidence for this innovation's practical impact is indirect but compelling: the paper reports no network-related SLO violations in production despite managing tens of thousands of GPUs across multiple data centers, and the 20% bandwidth penalty for cross-switch placement cited in Section 1 implies that failing to enforce network affinity would produce measurable performance degradation. The innovation's significance extends beyond P/D disaggregation to any distributed inference architecture where inter-component communication bandwidth is a bottleneck—disaggregated MoE, model-parallel serving, and multi-node inference pipelines all share the property that placement quality (not just placement quantity) determines performance.

Innovation 4: The Production Validation Methodology as a Contribution in Itself

While production deployment papers are common in the systems community, HeteroScale's evaluation (Section 4) makes a distinctive methodological contribution by combining controlled trace-replay experiments with large-scale production A/B comparison in a way that isolates the effect of specific design choices. This dual evaluation strategy addresses a common weakness in production systems papers: they can report impressive aggregate numbers (X% utilization improvement) but cannot attribute the gain to specific mechanisms because too many variables changed simultaneously in production.

The paper solves this attribution problem through its two-stage evaluation design. First, the trace-replay experiments (Section 4.2) hold all variables constant except the autoscaling metric, directly comparing eight candidate signals on the same workload trace, same initial conditions, same resource quotas, and same scaling thresholds. This isolates the metric selection effect and produces the key finding that decode TPS outperforms all alternatives, with specific quantitative characterizations of why each alternative fails (decode GPU utilization's insensitivity, latency's cliff behavior, TTFT's noise). Second, the production deployment analysis (Section 4.3) validates that these controlled-experiment findings translate to real-world gains, reporting both aggregate fleet-wide improvements (26.6 pp GPU utilization) and per-service breakdowns (TPS-based vs. periodic policy, pre-autoscaling vs. post-autoscaling metrics).

What elevates this from "good experimental practice" to an innovation is the explicit comparison between policy types in production. The paper doesn't just report "autoscaling improves utilization"—it reports that the metrics-driven TPS policy delivers 10.0 pp higher GPU utilization and 11.1 pp higher SM activity than the periodic policy (Section 4.3), and that this gap is consistent across services. This is a within-HeteroScale ablation conducted at production scale: it demonstrates that the specific design choice of metrics-driven over periodic scaling matters, not just that HeteroScale as a whole beats no autoscaling. Most production systems papers cannot make this claim because they deploy only one configuration.

The cross-modality replication (chat service in Figure 7, vision-language service in Appendix 8, Figure 9) further strengthens the methodology. By showing that the same metric selection and policy design work across modalities without re-tuning, the paper provides evidence that the findings are not overfit to a particular service's workload characteristics. This is a form of external validity testing that is rare in production systems papers but critical for establishing generalizability.

The significance of this innovation is that it provides a reproducible evaluation template for future autoscaling systems. The combination of controlled trace replay (to isolate mechanism effects), production A/B comparison (to validate real-world impact), and cross-service replication (to test generalizability) is a methodology that the broader ML serving community can adopt. The specific numbers (26.6 pp, 9.2 pp, hundreds of thousands of GPU-hours) are impressive but the methodological contribution—how to rigorously evaluate an autoscaling system in production—is more durable.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The experiments use production workload traces and live traffic from services running on ByteDance’s Seed Serving Platform. The primary trace for metrics evaluation comes from an open-domain dialogue service using Doubao-Seed-1.6-thinking (Section 4.2.1, Figure 2). A vision-language search service is used for cross-modality replication (Appendix 8, Figure 8). The authors state that the same qualitative patterns hold for "web search, long-form content understanding, real-time audio conversation, real-time video processing, and code generation" (Section 3.3.2). The paper does not use a public benchmark dataset — all evaluations are on proprietary production workloads, which is appropriate for a production systems paper but limits external reproducibility.

• Base model(s). The primary model is Doubao-Seed-1.6-thinking, served in a P/D disaggregated configuration. The paper does not specify the model's parameter count, architecture details, or pretraining corpus; it is referred to as a production model at ByteDance. For the P/D ratio experiments (Section 4.1), two services with different workload profiles are tested: Service A (average input ~3k tokens, output ~350 tokens, I/O ratio 8.5; SLOs TTFT ≤ 1s, TBT ≤ 40ms) and Service B (average input ~7.8k tokens, output ~700 tokens, I/O ratio 11; SLOs TTFT ≤ 1s, TBT ≤ 20ms). For the metrics evaluation trace replay (Section 4.2), an eight-hour segment from an open-domain dialogue service workload is used.

• Metrics. The paper uses multiple evaluation metrics depending on the experiment:

  • GPU Utilization (%): the standard NVIDIA-reported fraction of time the GPU was busy over a sampling interval.
  • SM Activity (%): measured via sm__cycles_active, representing the fraction of cycles where at least one warp was active on any SM — a finer-grained measure of compute engine utilization.
  • Throughput (TPS): tokens per second, measured separately for prefill and decode stages. Decode TPS is the primary autoscaling signal.
  • Latency (TTFT and TBT): Time-To-First-Token (initial response delay) and Time-Between-Tokens (inter-token generation delay), used both as SLO constraints and as candidate autoscaling signals.
  • GPU-hours saved: the primary operational cost metric, computed as the difference in total GPU allocation between baseline and HeteroScale configurations.

  For the metrics-driven autoscaling comparison (Section 4.2.2, Figure 6), the evaluation examines responsiveness — whether the instance count tracks workload changes appropriately, scaling out during peaks and in during valleys.

• Baselines. The paper compares against two main alternatives, both measured in the same production environment (Section 4.3):

  • No autoscaling: Services running without any automated scaling mechanism — the pre-HeteroScale state for many services on the platform. This is the baseline for the primary production utilization claims (26.6 pp GPU utilization gain) and for the per-service before/after comparisons (Figures 7 vs. 2).
  • Periodic (time-based) scaling policy: The simpler policy within HeteroScale itself, which adjusts capacity based on predetermined time-of-day schedules rather than real-time metrics. This baseline isolates the value of metrics-driven scaling over schedule-based scaling — the paper reports a 10.0 pp GPU utilization advantage for TPS-based over periodic (Section 4.3).

  The paper does not compare against Kubernetes HPA [29], KEDA [28], or other prior autoscaling systems directly, though it argues in Section 2.2 and Section 5.2 that these are categorically unsuitable for P/D disaggregated serving due to their use of misleading hardware metrics and lack of coordination mechanisms.

• Generation budget / compute accounting. The paper measures compute in terms of GPU allocation over time — how many GPUs are assigned to prefill and decode pools, and for how long. The primary efficiency metric is GPU-hours saved daily. For the trace-replay experiments (Section 4.2.2), all comparisons are conducted under "standardized conditions: all experiments began with identical numbers of prefill and decode instances, shared the same resource quotas, and applied uniform scaling thresholds calibrated to induce scaling behavior under equivalent load conditions." The eight metrics are compared on the same 8-hour workload trace. The field experiments (Section 4.1) use 16 nodes, each with 8 H20 GPUs (128 GPUs total).

• Cross-validation / statistical protocol. The paper does not employ formal cross-validation or statistical significance testing. For the metrics comparison (Section 4.2.2), each metric is evaluated on the same trace under identical initial conditions, with scaling events recorded and visualized. The cross-modality replication (chat service + vision-language search service) serves as a form of external validation, though no quantitative reproducibility metric is reported. For the production deployment analysis (Section 4.3), the comparison is a retrospective observational study: "a comparative analysis on a representative day" (Section 4.3) and "an analysis comparing performance on a recent date with a date prior to the scaled deployment." This is standard for production systems papers but does not control for confounding factors like traffic pattern changes, model updates, or other infrastructure changes between the comparison dates.

Main Quantitative Results

P/D Ratio Determination

The paper first establishes that the optimal P/D ratio is workload-dependent and must be empirically determined per service (Section 4.1, Figure 4).

Headline finding: Both test services exhibit a clear midrange peak in throughput as the P/D ratio varies, with performance degradation on both sides due to different SLO violations. Service A achieves its maximum throughput at a P/D ratio within the tested range (1P/5D to 9P/1D), while Service B shows a similar peaked pattern.

Service A (I/O ratio 8.5, TTFT ≤ 1s, TBT ≤ 40ms): At low P/D ratios (too few prefill instances), "scarce prefill instances cause TTFT to exceed preset threshold, capping throughput despite idle decode capacity" (Section 4.1). At high P/D ratios (too many prefill instances), "excess prefill instances that overwhelm decode resources, pushing TBT beyond its limit, and dampening maximum throughput." The peak is visible in Figure 4(a) as an inverted-U shape, with the exact optimal ratio not specified numerically in the text but readable from the figure.

Service B (I/O ratio 11, TTFT ≤ 1s, TBT ≤ 20ms): The same qualitative pattern holds but at a different optimal ratio, reflecting the longer input-output length distribution and tighter TBT SLO. The peaked shape is shown in Figure 4(b).

Production variability: The paper states that in production experience, the optimal P/D ratio "spans a considerable range—from 1P/5D to 9P/1D—depending on the input-output length distribution, hardware configurations, and SLO priorities" (Section 4.1). This variability underscores why the workload-centric policy curation pipeline (Algorithm 1) with pressure testing is necessary — no single ratio works across services.

Metrics-Based Autoscaling: Trace-Replay Comparison

This is the central controlled experiment (Section 4.2, Figure 6) that compares eight candidate metrics as scaling signals on an identical workload trace.

Headline finding: TPS-based metrics (both prefill TPS and decode TPS) demonstrate the most reliable responsiveness to workload dynamics, while hardware-level metrics show polarized behavior (prefill hardware metrics are usable but less sensitive; decode hardware metrics are actively misleading), and latency-based metrics produce delayed, oscillatory scaling with prohibitively difficult hyperparameter tuning.

TPS-based autoscaling (Figures 6a, 6b):

• "During peak periods, the system promptly scales out resource instances to match the surge, ensuring that request demands are met without performance degradation; conversely, during valley periods, it efficiently scales in to avoid resource waste" (Section 4.2.2).
• Both prefill TPS and decode TPS show close tracking of the workload pattern, with instance counts rising and falling in correspondence with load.
• The paper confirms that "within a given service, the input-output length distribution remains relatively stable, leading to consistent behavior in both decode TPS and prefill TPS. This correlation allows the two metrics to be used interchangeably in principle."

GPU utilization-based autoscaling (Figures 6c, 6d):

• Prefill GPU utilization "exhibits reasonable validity as a scaling signal... though it is less sensitive to workload changes compared with TPS-based metrics" (Section 4.2.2, Figure 6c). The instance count tracks workload but with less amplitude and slower response than TPS-based scaling.
• Decode GPU utilization is "ineffective for guiding scaling decisions" (Figure 6d). The figure shows decode GPU utilization remaining at a high level throughout the trace, producing minimal or no scaling actions despite clear workload variation. This confirms the polarization finding from the metric analysis (Figure 2c): "decode GPU utilization remains at a high level even when the workload decreases."

SM activity-based autoscaling (Figures 6e, 6f):

• Prefill SM activity "correlates well with workload fluctuations and may be a strong candidate for further investigation" (Figure 6e).
• Decode SM activity "exhibits the same limitation as decode GPU utilization—persistently high values even during low request volumes—making it unsuitable as a reliable autoscaling indicator" (Figure 6f).

Latency-based autoscaling (Figures 6g, 6h):

• TTFT-based scaling (Figure 6g): "scale-out actions tend to overshoot the required capacity, followed by frequent corrective adjustments." The figure shows wild oscillations in instance count, with the system repeatedly scaling out aggressively then scaling in rapidly — the hallmark of instability on a non-linear signal.
• TBT-based scaling (Figure 6h): "demonstrates relatively smoother responses and fewer fluctuations compared to TTFT, reacting reasonably to workload variations." However, the paper identifies a practical barrier: "tuning the associated hyperparameters poses a significant challenge. The nonlinear relationship between latency and resource allocation necessitates a multi-tier feedback-driven scaling mechanism. This design inevitably introduces numerous hyperparameters... In practice, tuning so many interdependent parameters is prohibitively difficult in large-scale production environments, making this approach challenging to deploy reliably" (Section 4.2.2).

Selection rationale for decode TPS: Based on the comparative analysis, TPS-based and prefill SM activity-based strategies "exhibit the most reliable responsiveness." The paper selects decode TPS specifically because: (1) it aligns with business objectives (tokens generated is the primary value metric), (2) it "can be uniformly distributed across instances" whereas prefill instances "often differ in hardware configurations, introducing additional complexity in metric normalization," and (3) it is universal across models and modalities (Section 4.2.2).

Production Deployment: Aggregate Fleet-Wide Gains

This is the primary production validation (Section 4.3), comparing HeteroScale-enabled services against non-autoscaled services at ByteDance's production scale.

Headline finding: HeteroScale delivers a 26.6 percentage point increase in GPU utilization and a 9.2 percentage point increase in SM activity on services with HeteroScale enabled compared to services without autoscaling. Overall GPU utilization across the fleet increased by 8.6 pp and SM activity by 6.5 pp. Hundreds of thousands of GPU-hours are saved daily.

Fleet-wide comparison (Section 4.3):

• "A comparative analysis on a representative day revealed substantial utilization gains. Services with HeteroScale enabled showed a 26.6 percentage points increase in GPU utilization and a 9.2 percentage point increase in SM activity compared to services without autoscaling."
• "An analysis comparing performance on a recent date with a date prior to the scaled deployment showed that overall GPU utilization increased by 8.6 percentage points and SM activity rose by 6.5 percentage points."
• "Hundreds of thousands of GPU-hours are saved each day."
• HeteroScale manages "tens of thousands of GPUs across numerous services" that "collectively process trillions of prefill tokens and generate hundreds of billions of decode tokens" daily.

Policy-type comparison within HeteroScale:

• The TPS-based metrics-driven policy manages 64% of the total GPU fleet under HeteroScale's control.
• "The TPS-based policy delivered a 10.0 percentage points higher GPU utilization and an 11.1 percentage points higher SM activity compared to the periodic policy" (Section 4.3).
• This within-HeteroScale ablation demonstrates that the specific design choice of metrics-driven over time-based scaling is responsible for a substantial fraction of the total gain — not just that autoscaling in general beats no autoscaling.

Production Deployment: Per-Service Deep Dive

The paper provides a detailed before/after comparison for the open-domain dialogue service (Section 4.3, Figure 7 vs. Figure 2).

Headline finding: With HeteroScale enabled on the dialogue service, overall GPU usage is reduced by 41.3%, with prefill GPU utilization increasing from 46.8% to 76.2% and prefill SM activity from 36.6% to 62.5%, while decode metrics remain in high but stable ranges (GPU utilization 86.0% → 82.2%, SM activity 53.0% → 61.6%).

Instance count tracking (Figure 7a vs. Figure 2a):

• "The number of both prefill and decode instances closely tracks the TPS variations, confirming that the essential P/D ratio is well-maintained throughout the workload fluctuations" (Section 4.3).
• Overall GPU usage is reduced by 41.3% — a direct cost savings metric.

Prefill metrics (Figure 7 vs. Figure 2):

• Prefill GPU utilization: 46.8% → 76.2% (a 29.4 pp increase).
• Prefill SM activity: 36.6% → 62.5% (a 25.9 pp increase).
• These large increases reflect the fact that without autoscaling, prefill instances sat idle during valleys; with HeteroScale, they are scaled in aggressively during low demand, keeping the remaining instances at high utilization.

Decode metrics (Figure 7 vs. Figure 2):

• Decode GPU utilization: 86.0% → 82.2% (a 3.8 pp decrease).
• Decode SM activity: 53.0% → 61.6% (an 8.6 pp increase).
• The paper explains that decode GPU utilization remains high in both conditions — "a characteristic of the memory-bound decode stage" — while SM activity increases because fewer instances are handling the same load, forcing higher compute utilization on the remaining instances.

Latency stability (Figure 7d vs. Figure 2b):

• "The latency metrics, TTFT and TBT, both vary within a smaller, more stable range compared to the non-autoscaled service."
• The paper notes that "the apparent large fluctuations in Figure 7(d) are an artifact of normalization" — since the Y-axis is normalized, small absolute variations appear amplified.
• "The occasional spikes observed in TTFT are caused by temporary P/D ratio imbalances that can occur during scaling operations" (Section 4.3). This is the phenomenon addressed by the service discovery gating mechanism described in Section 3.4.

Cross-Modality Validation

Headline finding: The vision-language search service (Appendix 8, Figure 9 vs. Figure 8) shows "a similar pattern of improved performance and stability, underscoring the general applicability across modalities of HeteroScale" (Section 4.3).

The paper provides the vision-language metrics in Figures 8 and 9 of the appendix, showing the same qualitative TPS tracking, GPU utilization patterns, and latency behavior as the dialogue service. A notable caveat: "The metrics for the autoscaled service may appear similar to the non-autoscaled service at times because the policy's minimum instance count is set to a relatively high threshold. This configuration is a deliberate choice to maintain a higher number of instances during off-peak hours, prioritizing service stability and readiness for sudden traffic surges" (Section 4.3). This reveals that HeteroScale's configuration supports a floor on scaling in — services can specify a minimum instance count below which the autoscaler will not go, even if metrics suggest further reduction is possible.

Ablation Studies and Robustness Checks

The paper does not present traditional ablation studies (varying one component while holding others fixed) in the style of ML papers. However, several implicit ablations are present in the experimental design:

Metric choice ablation (Figures 6a–6h): The trace-replay comparison across eight candidate metrics is effectively an ablation of the metric selection component. By holding the workload, initial conditions, resource quotas, and scaling thresholds constant while varying only the metric, the experiment isolates the effect of metric choice. The finding that decode TPS outperforms decode GPU utilization (which produces essentially no scaling actions) demonstrates that the specific signal choice is critical — not just that "autoscaling works."

Policy type ablation (Section 4.3): The comparison between TPS-based (64% of HeteroScale fleet) and periodic (remaining 36%) policies is a production-scale ablation of the policy type. The 10.0 pp GPU utilization advantage for TPS-based over periodic demonstrates that real-time metrics-driven scaling outperforms schedule-based scaling in this environment — a finding that is not obvious a priori, since periodic scaling with well-chosen schedules can be competitive when traffic is highly regular.

Cross-modality robustness (Appendix 8, Figures 8–9): Replicating the metric analysis and production deployment on a vision-language search service (a different modality with potentially different workload characteristics) tests whether the findings are specific to text dialogue services. The "similar pattern of improved performance and stability" provides evidence that the decode TPS metric selection and coordinated scaling policy generalize across modalities. However, the paper does not quantify the similarity — no correlation coefficient, no statistical test — leaving the cross-modality claim qualitative.

The minimum-instance-count floor (Section 4.3, vision-language service): The observation that the vision-language service shows similar autoscaled and non-autoscaled metrics at times due to a high minimum instance threshold reveals a practical design trade-off: stability (readiness for surges) vs. efficiency (aggressive scale-in). The paper does not explore the sensitivity of utilization gains to this threshold, which is a missing ablation — how much of the 26.6 pp gain depends on setting the minimum close to true off-peak demand versus a more conservative floor?

No ablation on P/D ratio sensitivity: The paper uses a fixed P/D ratio throughout production deployment (Section 3.4, Section 4.1). It does not test how sensitive the performance gains are to small deviations from the optimal ratio. Would a 10% ratio error erase the 26.6 pp utilization gain, or is the system robust to moderate ratio mismatches? This is a significant unanswered question, especially given the paper's acknowledgment that the optimal ratio "spans a considerable range" across services.

No ablation on cooling periods and hysteresis: The stability mechanisms (Section 3.6) — cooling periods, hysteresis thresholds, dampening factors — are presented as essential but are never experimentally varied. The paper does not show what happens if cooling periods are too short (oscillation), too long (sluggish response), or if hysteresis thresholds are set to zero (flapping). These are operational parameters that a practitioner would need to tune, and the lack of sensitivity analysis is a gap.

Negative result: Latency-based scaling instability: The failure of latency-based metrics (TTFT oscillation in Figure 6g, TBT hyperparameter fragility described in Section 4.2.2) is an informative negative result. The paper does not merely claim that TPS is better — it demonstrates that the natural alternative (scale based on latency, since latency is what SLOs care about) produces dangerously unstable behavior. This negative finding validates the hybrid policy design (proportional control for TPS, negative feedback safety net for latency) more convincingly than a positive-only finding would.

Critical Assessment

Claim 1: HeteroScale increases average GPU utilization by 26.6 percentage points and saves hundreds of thousands of GPU-hours daily.

This claim is the paper's headline production result (Section 4.3). The evidence is a retrospective observational comparison between services with HeteroScale enabled and services without autoscaling on "a representative day." Two issues limit the strength of this evidence:

First, selection bias is not controlled. Services with HeteroScale enabled may differ systematically from services without it — they may have been chosen for HeteroScale deployment precisely because they had the most room for improvement, or they may have more regular diurnal patterns that make autoscaling more effective. The 26.6 pp figure is not from a randomized experiment but from comparing a treated group to a non-random control group. The paper acknowledges this implicitly by providing a second comparison: "an analysis comparing performance on a recent date with a date prior to the scaled deployment showed that overall GPU utilization increased by 8.6 percentage points" (Section 4.3). The 8.6 pp fleet-wide improvement (which includes services that transitioned from no-autoscaling to HeteroScale) is a more conservative and arguably more credible estimate of HeteroScale's impact than the 26.6 pp cross-sectional comparison. The 26.6 pp figure likely overstates the gain for an average service.

Second, the baseline "without autoscaling" is ambiguous. Are these services that were manually provisioned (operators adjusting capacity by hand), statically provisioned (fixed GPU count 24/7), or using some other mechanism? Manual provisioning at ByteDance's scale would be reactive and conservative (over-provision to avoid SLO violations), inflating the baseline waste and therefore the HeteroScale gain. If the baseline is worst-case static provisioning at peak capacity, the 26.6 pp gain is partially attributable to replacing a straw-man baseline rather than to HeteroScale's specific design. A stronger comparison would be against a reasonable manual provisioning policy (e.g., an operator adjusting capacity twice daily based on known diurnal patterns), which would narrow the gap.

The GPU-hours saved figure ("hundreds of thousands") is not broken down by service, hardware type, or mechanism. There's no decomposition of savings into "scale-in during valleys" vs. "better P/D ratio" vs. "heterogeneous hardware matching," making it impossible to attribute the gain to specific components.

Claim 2: Decode TPS is the most robust autoscaling signal, and hardware metrics (especially decode GPU utilization) are actively misleading in P/D disaggregated settings.

This claim is strongly supported by the trace-replay experiments (Figure 6). The evidence is direct, controlled, and replicated across two modalities. The eight-metric comparison under identical conditions eliminates the confounding factors present in the production comparison. The finding that decode GPU utilization produces essentially no scaling actions (Figure 6d) while TPS tracks load accurately (Figure 6b) is a clear, qualitative result that does not depend on precise numerical thresholds.

The limitation is that the trace is a single 8-hour segment from one service type (open-domain dialogue), though the cross-modality replication and the authors' claim that other workloads show the same patterns partially address this. The precise thresholds and scaling magnitudes are tuned for this trace and may not transfer directly, but the qualitative superiority of TPS over GPU utilization for decode pools is likely structural — it follows from the memory-bound nature of decode, which is a property of transformer architectures generally, not of specific datasets.

The KV cache interference on prefill TPS — "Due to the interference of the KV cache hit rate, prefill TPS measurements under caching are unreliable for autoscaling" (Section 3.3.2) — is an important subtlety that the trace-replay experiments do not isolate. The paper recommends using "KV cache missed prefill TPS" but does not show how the autoscaling performance degrades if raw prefill TPS (including cache hits) is used instead.

Claim 3: The TPS-based metrics-driven policy outperforms the periodic (time-based) policy by 10.0 pp GPU utilization and 11.1 pp SM activity.

This is a within-HeteroScale comparison that isolates the value of real-time metrics over static schedules. It is well-supported by the production data (64% of fleet on TPS-based vs. 36% on periodic, with the utilization gap measured between these groups). However, the same selection bias concern applies: services on TPS-based policies may be those with more variable traffic patterns (where periodic scaling would perform worst) or those operated by teams more sophisticated about resource optimization.

A missing experiment that would strengthen this claim: a head-to-head comparison of TPS-based vs. periodic on the same service over the same time period. This is difficult in production (you can't run two autoscalers on the same service simultaneously), but a time-split design (periodic for one week, TPS-based for another week, same weekday patterns compared) would be feasible and would eliminate between-service confounding.

Claim 4: The Deployment Group and RDMA Subgroup scheduling abstractions enable network-affinity-aware placement that avoids the ~20% KV cache transfer bandwidth penalty from cross-switch placement.

This claim is not directly experimentally validated. The 20% bandwidth penalty is cited in Section 1 as an "empirical observation," but the paper provides no experiment or measurement showing that HeteroScale's topology-aware scheduler actually avoids this penalty or that this avoidance translates to measurable latency improvements. The production deployment analysis (Section 4.3) shows that overall latency remains stable and within SLOs, but does not isolate the contribution of network-affinity-aware placement vs. other factors (adequate total capacity, P/D ratio maintenance, etc.).

A missing experiment: compare TTFT distributions for services scheduled with HeteroScale's affinity-aware placement vs. the same services scheduled with a topology-oblivious but resource-count-equivalent baseline. Alternatively, measure KV cache transfer latency distributions with and without HeteroScale's placement constraints. Without such experiments, the network affinity innovation remains a compelling design argument but not an empirically validated mechanism.

Overall strengths of the experimental design:

• Production scale with real financial impact: tens of thousands of GPUs, trillions of tokens daily, hundreds of thousands of GPU-hours saved — these are rare and valuable measurements that few academic systems papers can match.
• Clear within-system ablation (TPS vs. periodic): demonstrates that the specific design choices (metrics-driven, coordinated) matter, not just that autoscaling in general helps.
• Cross-modality replication: the vision-language service validation increases confidence in generalizability.
• Well-controlled metrics comparison: the trace-replay experiments directly answer the question "which metric should we use?" with standardized conditions.

Overall weaknesses of the experimental design:

• No comparison against prior autoscaling systems (HPA, KEDA, predictive autoscalers): while the paper argues these are categorically unsuitable, a direct comparison — even one showing catastrophic failure — would strengthen the claim that HeteroScale solves a problem no existing system addresses. The absence of such a comparison leaves open the possibility that a well-tuned HPA (with custom metrics, not default GPU utilization) might perform comparably.
• Selection bias in production comparisons: the 26.6 pp and 10.0 pp gains are from non-randomized comparisons. The 8.6 pp fleet-wide before/after is more credible but still subject to temporal confounding (model updates, traffic pattern changes, other infrastructure improvements).
• No sensitivity analysis: How sensitive are the gains to cooling period choice, hysteresis thresholds, P/D ratio error, or minimum instance floor? Without this, a practitioner replicating HeteroScale has no guidance on which parameters are critical vs. robust.
• Fixed P/D ratio throughout: while the paper acknowledges this as a limitation and identifies dynamic ratio adaptation as future work, the experiments don't quantify how much additional gain dynamic adaptation might provide, nor whether the fixed ratio assumption caused any SLO violations or inefficiencies during the evaluation period.
• Single model family (Doubao-Seed-1.6-thinking): the paper does not test with other model architectures, sizes, or serving frameworks, though the cross-modality replication partially addresses the concern that findings are model-specific.
• No measurement of the 20% KV cache bandwidth penalty in HeteroScale deployments: the network affinity scheduling is evaluated only indirectly through overall latency stability, not through direct KV cache transfer measurements.

6. Limitations and Trade-offs

Fixed P/D Ratio Under Workload Shifts

The assumption or constraint. HeteroScale uses a fixed P/D ratio derived from offline pressure testing (Algorithm 1, Section 3.4). The paper explicitly acknowledges this boundary:

"In the online environment, we use a fixed P/D ratio for scaling. This P/D ratio is derived from service pressure tests and historical empirical data." (Section 3.4)

The ratio is held constant regardless of runtime workload changes, with the stability justification that "within a given service, the input-output length distribution remains relatively stable" (Section 4.2.2).

The consequence. If the input-output length distribution shifts at runtime — e.g., users suddenly submit much longer prompts (increasing prefill load relative to decode) or request much longer generations (increasing decode load relative to prefill) — the fixed ratio becomes misaligned with actual demand. This produces the exact bottleneck pattern the paper identifies as catastrophic in Section 1: too few prefill instances cause TTFT to spike, while too few decode instances cause TBT violations and prefill idle. The paper's own P/D ratio experiments (Section 4.1, Figure 4) demonstrate that throughput degrades sharply on both sides of the optimal ratio. At low ratios, "scarce prefill instances cause TTFT to exceed preset threshold, capping throughput despite idle decode capacity." At high ratios, "excess prefill instances that overwhelm decode resources, pushing TBT beyond its limit." A fixed-ratio system experiencing a workload shift would drift into one of these degraded regimes.

The paper identifies "workload drift" as a real phenomenon in Section 6: "subtle changes in user behavior, prompt complexity, or generation length" constitute a recognized operational challenge. But the current system offers no automated defense against it — the pressure test would need to be re-run and the ratio manually updated, which is an offline, reactive process.

What evidence exists in the paper. The paper's evidence that the fixed ratio works in practice is the production deployment data (Section 4.3, Figure 7a), which shows that "the number of both prefill and decode instances closely tracks the TPS variations, confirming that the essential P/D ratio is well-maintained throughout the workload fluctuations." However, this evidence is from a service whose workload characteristics were presumably stable during the observation period — it demonstrates that the ratio doesn't drift under constant workload distribution, not that it would survive a distribution shift. The paper provides no stress test where the input-output length distribution is deliberately perturbed to measure the resulting SLO degradation.

The paper also documents "occasional spikes observed in TTFT" caused by "temporary P/D ratio imbalances" during scaling operations (Section 4.3), indicating that even transient ratio deviations produce measurable latency impact. A sustained misalignment would be significantly worse.

Mitigation status. The paper identifies dynamic P/D ratio adaptation as the second item in its future work section (Section 6): "Building on our current fixed-ratio mechanism, we will explore methods for making minor, dynamic adjustments to the P/D ratio in real-time." No mitigation exists in the current system beyond manual re-pressure-testing. The authors characterize the dynamic adaptation goal as making "minor" adjustments, suggesting they view the ratio as approximately stable and only needing small corrections — but the paper provides no evidence for how large the corrections would need to be under realistic workload drift scenarios.

Difficulty Estimation Cost Is Excluded from Headline Efficiency Numbers

The assumption or constraint. The compute savings reported by HeteroScale assume that the autoscaling infrastructure itself adds negligible overhead to total GPU consumption. This assumption is implicit — the paper never explicitly accounts for the computational cost of metric collection, policy evaluation, topology discovery, and scheduling execution. The production evaluation compares GPU utilization with HeteroScale enabled versus disabled (Section 4.3), treating HeteroScale's own resource consumption as part of the "enabled" total but not separately measured.

The consequence. The 26.6 percentage point GPU utilization improvement and the "hundreds of thousands of GPU-hours saved daily" are gross savings that do not net out HeteroScale's own resource footprint. In practice, the control plane consumes resources for:

• Metric collection: gathering decode TPS, prefill TPS, GPU utilization, SM activity, TTFT, and TBT at a granularity sufficient for near-real-time decision-making across tens of thousands of GPUs. The paper does not specify the collection interval, but the cooling periods mentioned in Section 3.6 imply it must be on the order of minutes or less.
• Policy evaluation: the periodic evaluation loop described in Section 3.2, which runs across all managed services.
• Topology discovery: "at the start of each cycle, the controller builds a fresh topological resource tree" (Section 3.4) by querying the entire cluster state — an operation whose cost scales with cluster size.
• Scaling action execution: creating, deleting, and monitoring pods through the Kubernetes API server.

For an individual service, these overheads are likely negligible relative to the GPU fleet being managed. But the paper presents aggregate fleet-wide numbers without isolating HeteroScale's own cost, making it impossible to determine the net GPU-hours saved. A reader cannot tell whether HeteroScale consumes 0.1%, 1%, or 5% of the total fleet's compute budget to operate. If the control plane requires, say, a dedicated set of GPU nodes for metric aggregation and scheduling (the paper does not specify whether the control plane runs on GPUs or CPU-only nodes), those resources are part of the total cost of the system.

What evidence exists in the paper. None. The paper provides no measurement of HeteroScale's resource consumption, no specification of the control plane's hardware requirements, and no breakdown of the hundreds of thousands of GPU-hours saved into gross savings minus control plane cost. The architecture diagrams (Figures 1, 3) show the components but not their resource footprints.

Mitigation status. Not addressed. This is a common omission in production systems papers — the control plane overhead is assumed small enough to ignore — but at the scale HetereoScale operates (tens of thousands of GPUs), even a 1% overhead represents hundreds of GPUs continuously. The paper's claims of cost savings would be stronger with an explicit accounting of control plane resource consumption and a statement about net savings.

Single Metric, Single Model Family, Single Infrastructure

The assumption or constraint. All experimental validation uses one base model family (Doubao-Seed-1.6-thinking) running on one inference infrastructure (ByteDance's Seed Serving Platform, a Kubernetes-based system with specific hardware — NVIDIA H20 and L20 GPUs with RDMA interconnects). The paper states that the model is "representative" and that "the same qualitative patterns hold for other workloads such as web search, long-form content understanding, real-time audio conversation, real-time video processing, and code generation" (Section 3.3.2), but provides quantitative evidence only for two modalities: open-domain dialogue (Section 4) and vision-language search (Appendix 8, Figures 8–9).

The consequence. Three distinct generalizability questions are conflated:

1. Model generalizability: Does the decode-TPS-as-primary-signal finding hold for models with different architectures (dense vs. MoE, different parameter counts, different attention mechanisms), different kv-cache implementations, or different batching strategies? The decode-phase's memory-bound property (which causes GPU utilization polarization) is structural to autoregressive transformer inference, so this finding likely generalizes broadly — but the specific thresholds, the optimal P/D ratio, and the sensitivity of TPS to load may differ substantially across model scales and architectures.

2. Hardware generalizability: Does the decode GPU utilization trap (Section 3.3.2) manifest identically on GPUs with different HBM bandwidth-to-compute ratios? The paper tests on H20 and L20, but higher-bandwidth GPUs (H100, H200) or lower-bandwidth inference-specific accelerators might show different polarization behavior. The RDMA Subgroup priority system depends on the specific network topology hierarchy (S1/S2 switches) present in ByteDance's data centers; clusters with flat network topologies, InfiniBand fabrics, or different disaggregation patterns would need to redesign this abstraction.

3. Infrastructure generalizability: The Deployment Group abstraction assumes Kubernetes with CRDs and service discovery. Organizations using different orchestration layers (Slurm, custom schedulers, cloud-specific container services) would need to re-implement the coordinated scheduling logic on a different substrate. The paper does not specify which capabilities are tightly coupled to Kubernetes and which are portable.

The cross-modality replication (dialogue + vision-language) provides evidence that the metric selection finding generalizes across input modalities — but both services run on the same model family, same hardware, and same infrastructure. The paper has effectively shown robustness to one axis of variation (workload modality) while leaving three others (model, hardware, infrastructure) untested.

What evidence exists in the paper. The primary evidence for generalizability is the claim about "other workloads" in Section 3.3.2, which is stated without supporting data. The vision-language replication (Appendix 8) is the only concrete evidence beyond the dialogue service, and it is presented qualitatively ("a similar pattern of improved performance and stability") without quantitative comparison. The paper does not report, for example, the utilization gain for the vision-language service separately or compare its optimal P/D ratio to the dialogue service's.

Mitigation status. Not addressed as a limitation. The paper frames the cross-modality replication as validating generalizability rather than as a first step that needs extension to other axes. The future work section (Section 6) identifies "model-agnostic, hardware-agnostic, and workload-agnostic" metrics as a goal, implicitly acknowledging that the current validation is insufficient to claim full agnosticism — but this framing treats the limitation as a future extension rather than a current constraint on the paper's claims.

The ~14× Larger Model Baseline Is Not Compute-Optimally Trained

Note: This limitation is structurally similar to the critique in the reference example (Section 6, Limitation on the $14\times$ larger model baseline), applied to HeteroScale's context. The paper's production comparison is between HeteroScale-enabled services and non-autoscaled services, not between different model sizes — but the generalization gap works differently here.

The assumption or constraint. The production evaluation (Section 4.3) compares HeteroScale-enabled services against two baselines: services without any autoscaling, and services using HeteroScale's own periodic (time-based) policy. There is no comparison against a well-tuned alternative autoscaling system — e.g., Kubernetes HPA with custom metrics, KEDA with a Prometheus-based TPS scaler, or a manually-provisioned capacity schedule designed by an experienced operator.

The consequence. The 26.6 percentage point GPU utilization gain and the "hundreds of thousands of GPU-hours saved daily" are measured against a baseline that the paper characterizes as essentially broken: "traditional autoscalers like Kubernetes's Horizontal Pod Autoscaler (HPA) struggle to meet the demands of modern LLM serving architectures" (Section 1). But the paper never quantifies how much better HeteroScale is than the best available alternative — it only demonstrates that HeteroScale beats no-autoscaling and beats its own simpler variant. A practitioner evaluating HeteroScale needs to know: if I already have a well-configured HPA using custom decode TPS metrics (which Kubernetes supports), does HeteroScale still provide a 26.6 pp improvement, or would the gap narrow to a few percentage points?

The absence of an external baseline is particularly important because several of HeteroScale's architectural choices — coordinated scaling with decode TPS, P/D ratio enforcement — could potentially be replicated on existing infrastructure with custom metrics and coordinated scaling scripts. The paper does not establish that the full integration (Deployment Groups, RDMA Subgroups, federated pre-scheduler) is necessary to achieve the reported gains, versus the simpler policy-level innovations (decode TPS selection, simultaneous scaling) being sufficient.

What evidence exists in the paper. The paper makes a theoretical argument about why HPA and similar systems would fail: "decode GPU utilization is a misleading metric" (Section 1, Challenge 3) and "independent scaling thus leads to architectural imbalance" (Section 1, Challenge 3). But it provides no experimental demonstration of this failure. The latency-based autoscaling failure (Figure 6g) is the closest thing to a negative baseline result, and it demonstrates instability — but this is a policy evaluation, not a comparison against a specific prior system.

The paper also notes that "conventional autoscalers like the Kubernetes HPA fall short for the unique demands of LLM services" (Section 2.2), but this is presented as motivation, not as a hypothesis tested in the evaluation.

Mitigation status. Partially addressed by the within-HeteroScale comparison (TPS-based vs. periodic policy, Section 4.3), which shows that the specific policy choice matters — a 10.0 pp GPU utilization gap between the two policies implies that "just use any autoscaler" is not sufficient. But this doesn't close the gap to a well-tuned external baseline. The paper's deployment context (ByteDance's Seed Serving Platform) may make such external comparisons infeasible — the platform was built around HeteroScale from the start — but this is a constraint the paper should acknowledge explicitly when making comparative claims.

Production Comparisons Are Retrospective and Non-Randomized

The assumption or constraint. All production deployment results (Section 4.3) are based on retrospective observational comparisons: "a comparative analysis on a representative day" and "an analysis comparing performance on a recent date with a date prior to the scaled deployment." The paper does not describe a controlled A/B test, randomized rollout, or time-split design. The "services with HeteroScale enabled" group was not randomly assigned; services were presumably selected for HeteroScale deployment based on operational priorities, readiness, or expected benefit.

The consequence. Three types of confounding threaten the validity of the headline numbers:

1. Selection bias: Services chosen for HeteroScale deployment may have had systematically lower utilization to begin with (making the improvement larger), more regular diurnal patterns (making autoscaling more effective), or more sophisticated operational teams (who would have achieved some improvement through manual tuning anyway). The 26.6 pp cross-sectional comparison may overstate the gain for an average service.

2. Temporal confounding: The before/after comparison ("a recent date with a date prior to the scaled deployment") is vulnerable to any changes that occurred between the two dates beyond HeteroScale deployment — model updates, traffic pattern shifts, other infrastructure improvements, seasonal effects, or changes in the user base. A 8.6 pp fleet-wide GPU utilization increase could be partially attributable to, say, a model optimization that increased throughput per GPU, reducing the number of GPUs needed for the same load.

3. No parallel comparison: There is no experiment where the same service runs simultaneously with and without HeteroScale (e.g., splitting traffic between two identically-provisioned pools, one autoscaled and one static). Such an experiment would eliminate temporal and selection confounds but is operationally complex in production.

What evidence exists in the paper. The paper reports three numbers that would ideally come from three different types of comparisons:

• 26.6 pp: cross-sectional (HeteroScale services vs. non-autoscaled services, same day)
• 8.6 pp: before/after (same fleet, different dates)
• 10.0 pp: within-system (TPS-based vs. periodic, cross-sectional)

The gap between 26.6 pp and 8.6 pp is substantial and implicitly supports the concern about selection bias — the cross-sectional comparison overstates the gain compared to the temporal comparison that controls for between-service differences (since it compares the same fleet). The paper does not discuss this gap or its implications for interpreting the headline number.

Mitigation status. Not discussed as a limitation. The paper reports all three numbers without caveats about their methodological differences or the strength of causal inference they support. The trace-replay experiments (Section 4.2) provide stronger causal evidence for the specific question of metric selection, but they don't validate the full system in production conditions with all components (scheduling, stability mechanisms, heterogeneous hardware) active. A practitioner should treat the 26.6 pp figure as an upper bound and the 8.6 pp fleet-wide improvement as the more conservative estimate — but the paper does not guide this interpretation.

No Direct Measurement of Network Affinity Scheduling Impact

The assumption or constraint. HeteroScale's Deployment Group and RDMA Subgroup abstractions (Section 3.4) are presented as key innovations that address the "Network Bottlenecks" challenge (Section 1, Challenge 2): "Our empirical observations show that such placements across different network switches can reduce the available bandwidth for KV cache transfer by approximately 20%." The scheduler is designed to prevent these cross-switch placements. The expected benefit is lower KV cache transfer latency and therefore lower TTFT.

The consequence. The paper never measures whether HeteroScale's topology-aware placement actually produces lower KV cache transfer latency than a topology-oblivious baseline, nor whether any latency improvement translates to end-to-end SLO improvements. The production evaluation (Section 4.3) reports that "the latency metrics, TTFT and TBT, both vary within a smaller, more stable range compared to the non-autoscaled service" — but this comparison confounds the effect of better placement with the effect of having adequate total capacity (since the non-autoscaled baseline may have been under-provisioned at peaks, causing latency degradation regardless of placement quality). The stability improvement could be entirely attributable to the autoscaling policy ensuring capacity adequacy, with the topology-aware placement contributing nothing measurable.

More fundamentally, the paper does not establish that the 20% bandwidth penalty from cross-switch placement — even if real — is a binding constraint in production. If KV cache transfers are not the bottleneck (e.g., if compute dominates TTFT, or if the available bandwidth even after a 20% penalty is still sufficient), then the entire network affinity scheduling machinery may be solving a non-problem.

What evidence exists in the paper. The 20% bandwidth penalty is cited as an "empirical observation" in Section 1, without details on how it was measured (what switch configurations were compared, what workload, what KV cache sizes). It is not reproduced or validated in the evaluation section. The network affinity scheduling algorithm (Algorithm 4) is described in detail, but its impact is never isolated experimentally. There is no experiment comparing:

• TTFT distributions under HeteroScale's affinity-aware placement vs. a version of HeteroScale with the same scaling policy but topology-oblivious placement.
• KV cache transfer latency distributions under co-located vs. cross-switch placement in the production environment.
• The fraction of placements that would have been cross-switch without HeteroScale vs. with it.

The paper's evidence for network affinity scheduling is entirely architectural: the problem is real, the mechanism is designed to solve it, and the overall system works — but the causal link from mechanism to improvement is unmeasured.

Mitigation status. Not addressed as a limitation. The Deployment Group and RDMA Subgroup abstractions occupy substantial space in the system design (Section 3.4) and are listed as key innovations in the introduction, but they are evaluated only indirectly through overall system performance. A practitioner considering whether to implement network-affinity-aware scheduling would find no experimental evidence in the paper quantifying its marginal benefit. The cost is clear (added scheduling complexity, the priority system, the topology discovery overhead), but the benefit is assumed.

7. Implications and Future Directions

How This Work Changes the Landscape

HeteroScale shifts the autoscaling conversation for LLM serving from a control-theory problem (tune the algorithm) to a measurement-theory problem (validate the signal). This is a conceptual reframing, not a paradigm shift—the proportional control algorithm is textbook, the topology-aware scheduling draws on prior work in data-intensive clusters—but the reframing is practically decisive. Before this paper, the dominant assumption was that hardware utilization metrics, perhaps augmented with custom application metrics, were sufficient scaffolding for autoscaling disaggregated inference. The field's default posture was: "Kubernetes HPA works for microservices; LLM serving is just another microservice with GPUs." HeteroScale demonstrates that this posture is not merely suboptimal but actively destructive in P/D disaggregated settings, because the metrics that work for stateless web services (CPU, memory, request rate) have systematically different signal characteristics in compute-bound vs. memory-bound inference stages.

The paper resolves a latent contradiction in the LLM serving literature. On one side, systems like DistServe [57], SplitWise [39], and Mooncake [41] demonstrated that P/D disaggregation improves throughput and cost efficiency—making it an increasingly attractive architectural pattern. On the other side, operational experience at ByteDance (and, anecdotally, at other large-scale LLM providers) revealed that disaggregated services were harder to manage than monolithic ones: resource utilization was paradoxically lower, SLO violations were harder to diagnose, and traditional autoscaling produced erratic behavior. The paper provides the diagnostic framework that reconciles these observations. P/D disaggregation is genuinely more efficient, but only when the control plane understands the coupling it introduces. Without coordinated scaling and metric validation, the disaggregation advantage is consumed—and then some—by autoscaling-induced inefficiency. The 26.6 percentage point GPU utilization improvement (Section 4.3) is not a claim that HeteroScale's algorithms are genius; it is evidence of how badly the baseline was broken. The implication is that disaggregation without coordinated autoscaling is an incomplete optimization—you get the structural throughput benefit but lose it to provisioning waste.

The paper also redirects research attention within the LLM serving community away from increasingly sophisticated single-node optimizations and toward the cluster-level control plane as the binding constraint on efficiency. The past several years have produced remarkable advances in inference kernels, batching strategies, and memory management (vLLM [32], TensorRT-LLM [37], SGLang [56]). These systems squeeze more throughput from a fixed set of GPUs. HeteroScale addresses the complementary problem: ensuring that the right number of GPUs are allocated at the right time, in the right network locations, with the right prefill-decode balance. The paper's implicit argument is that at production scale, the provisioning multiplier (how many GPUs you use vs. how many you actually need) is often larger than the kernel efficiency multiplier (how much throughput you get per GPU vs. a naive implementation). A 10% kernel improvement on a fleet that is 50% over-provisioned yields less total efficiency than a 0% kernel improvement on a fleet that is right-provisioned. This is not an argument against kernel optimization—it is an argument that the two optimizations are complementary and that the provisioning side has been relatively neglected.

The methodological contribution is equally important. The paper establishes a reproducible metric validation protocol for autoscaling disaggregated systems: (1) collect production traces without autoscaling, (2) characterize candidate metrics' signal properties (linearity, noise, polarization between components), (3) replay traces under each candidate with identical conditions, (4) compare scaling behavior qualitatively and quantitatively, and (5) replicate across modalities. This protocol is transferable to any P/D deployment and, with adaptation, to other disaggregated architectures (MoE, model-parallel serving, retrieval-augmented pipelines). It demystifies metric selection from an art to a procedure—and the procedure's first conclusion (decode GPU utilization is useless as a scaling signal) is likely robust across most hardware configurations because it follows from the architectural property that decode is memory-bandwidth-bound.

The paper's negative findings are arguably as valuable as its positive ones. By demonstrating that latency-based autoscaling produces dangerous oscillations (Figure 6g) and that decode hardware metrics are actively misleading (Figures 6d, 6f), the paper provides a map of where not to invest effort. A research team considering a sophisticated reinforcement-learning autoscaler driven by TTFT signals would be warned off by this paper's evidence—the signal is too non-linear for stable control, regardless of how advanced the policy network is. This is a form of negative knowledge that prevents the community from re-discovering the same failure modes independently.

Finally, the paper shifts the Overton window for what constitutes a credible production systems evaluation in the LLM serving space. The combination of controlled trace-replay experiments (Section 4.2) with large-scale production A/B comparison (Section 4.3) establishes a standard that future papers will be measured against. It is no longer sufficient to report simulation results or small-scale lab experiments; the field now has an existence proof that production-scale evaluation with real financial impact (hundreds of thousands of GPU-hours saved daily) is publishable and provides evidence that no amount of simulation can replicate. This raises the bar but also provides a template for how to meet it.

Follow-Up Research This Work Enables

Dynamic P/D ratio adaptation under workload distribution shifts. The paper identifies workload drift—subtle changes in prompt length, generation length, or user behavior—as a recognized operational challenge but leaves the ratio fixed. A concrete follow-up: deploy HeteroScale on a service with known diurnal variations in input-output length ratio (e.g., a coding assistant where morning users write short queries and evening users paste long debugging sessions). Instrument the system to detect ratio drift by tracking the relationship between prefill TPS and decode TPS over sliding windows. Implement a Kalman filter or simple exponential smoother that adjusts the target ratio within bounded limits (say, ±30% from the pressure-test optimum) when the observed workload composition shifts. The key metric: does dynamic adaptation reduce TTFT spikes and TBT violations during ratio-transition periods compared to the fixed-ratio baseline, and at what cost in GPU utilization (since ratio changes may temporarily over-provision one pool)? The paper's own data (Figure 4, showing throughput peaks at different ratios for services with different I/O distributions) provides the motivating evidence; the question is whether online adaptation can find and track these peaks without the instability that plagues latency-based control.

Cheap difficulty estimation via inference engine internal metrics. The paper's metric selection validated eight external metrics (TPS, GPU utilization, SM activity, TTFT, TBT) but explicitly identifies internal inference engine statistics as a future direction. A concrete study: instrument a serving engine (vLLM, SGLang, or TensorRT-LLM) to expose internal queue depths, batch sizes, KV cache hit rates, and per-request prefill/decode time breakdowns. Run the same trace-replay protocol used in Section 4.2 but include these internal metrics as additional candidates. The hypothesis to test is that KV cache hit rate in particular should be a leading indicator of prefill load change—when hit rate drops, it signals that new, uncached prompts are arriving, which will increase prefill demand before it manifests in TPS or latency. If true, hit rate could serve as an early-warning signal that triggers pre-emptive scaling, reducing the reaction lag inherent in throughput-based control. The paper's observation that "KV cache missed prefill TPS" is the reliable variant but raw prefill TPS is confounded by hits (Section 3.3.2) already hints at this direction. A strong negative result (internal metrics add no predictive power beyond decode TPS at practical collection granularities) would also be valuable, as it would simplify the design space and validate the paper's minimal-signal philosophy.

Stress-testing the single-metric assumption under adversarial workload shifts. The paper's coordinated scaling design rests on the empirical finding that "within a given service, the input-output length distribution remains relatively stable" (Section 4.2.2). A rigorous stress test: construct a synthetic workload trace where the input-output length ratio deliberately varies on timescales faster than the autoscaling evaluation period. For example, alternate hourly between short-prompt/long-generation workloads (I/O ratio 2) and long-prompt/short-generation workloads (I/O ratio 20). Run HeteroScale with its fixed P/D ratio set to the average optimum. Measure two outcomes: (1) what fraction of time is the system operating with a P/D ratio more than 20% from the instantaneous optimum, and (2) what is the resulting SLO violation rate compared to a hypothetical oracle that knows the instantaneous optimal ratio? This experiment would quantify the robustness envelope of the fixed-ratio assumption—the paper's production data shows it works under natural workload stability, but the community needs to know under what conditions it breaks. The pressure test methodology from Section 4.1 provides the tooling to determine instantaneous optimal ratios at each workload composition point.

Isolating the marginal value of network-affinity-aware placement. The Deployment Group and RDMA Subgroup abstractions are central to HeteroScale's design but unevaluated in isolation. A concrete experiment: run two instances of the same service on identical hardware pools, both using HeteroScale's TPS-based proportional control policy. In the treatment group, the scheduler enforces S2-level affinity (prefill and decode co-located under the same switch). In the control group, the scheduler is modified to ignore affinity constraints and places instances wherever capacity is available (the topology-oblivious baseline). Compare the full distributions of TTFT and KV cache transfer latency (measured at the inference engine level) between the two groups over a production workload trace. The 20% bandwidth penalty cited in Section 1 predicts a measurable TTFT degradation in the control group. If the degradation is small (<5% at p99), the network affinity machinery may be over-engineered for this workload/hardware combination. If it is large (>15%), the paper's design priority is validated. A null result would be highly informative—it would suggest that KV cache transfer is not the bottleneck (perhaps compute dominates TTFT, or available inter-switch bandwidth is sufficient even after the penalty) and that simpler, topology-oblivious scheduling is adequate, substantially reducing the complexity of adopting HeteroScale-like systems.

Behavioral characterization of the decode GPU utilization trap across GPU architectures. The paper's finding that decode GPU utilization remains persistently high regardless of load is attributed to "KV cache storage and data transfer operations" (Section 3.3.2) but is only demonstrated on H20 and L20 GPUs. A systematic study across GPU architectures: reproduce the metric collection from Figure 2 on H100, H200, A100, and inference-specific accelerators (Google TPU v5, AWS Inferentia, etc.) under controlled decode-only workloads at varying batch sizes and KV cache sizes. Measure the relationship between true decode throughput (tokens generated per second) and reported GPU utilization. The hypothesis: the polarization is caused by memory bandwidth saturation from KV cache reads dominating the utilization counter, and this saturation point depends on the GPU's HBM bandwidth-to-compute ratio. Architectures with proportionally higher memory bandwidth (H200's HBM3e vs. H100's HBM3) may show less polarization—the decode phase may not saturate memory bandwidth, allowing utilization to track load more proportionally. Such a finding would mean that TPS-based scaling becomes less critically important on higher-bandwidth hardware, while remaining essential on cost-optimized inference GPUs (like L20). This would provide hardware-specific guidance for metric selection rather than a one-size-fits-all recommendation.

Combining HeteroScale with predictive workload forecasting. The paper's policies are purely reactive—they respond to current metric values. The workload traces (Figure 5) show strong diurnal patterns that are predictable. A natural extension: train a lightweight temporal model (e.g., a seasonal ARIMA model or a small transformer) on historical decode TPS traces to forecast load 10–30 minutes ahead. Use the forecast to pre-warm the cluster (scale out in advance of predicted peaks) rather than waiting for the metric to cross the threshold. The key metric: does proactive scaling reduce the transient SLO violations that occur during rapid load ramps (morning surge in Figure 5), and at what cost in standby GPU waste if the forecast overshoots? The paper's periodic policy (Section 3.3.1) already demonstrates that time-based pre-provisioning works for predictable patterns, but it requires manual schedule specification. A learned forecaster would combine the responsiveness of metrics-driven scaling with the proactivity of periodic scaling, potentially closing the 10.0 pp GPU utilization gap between the two policies (Section 4.3) by avoiding the reactive overshoot and undershoot that cause waste.

Practical Applications and Downstream Use Cases

Cost-efficient multi-tenant LLM API platforms. A cloud provider or internal platform team offering LLM inference as a service to multiple downstream applications faces the core tension HeteroScale solves: each tenant has different workload characteristics (prompt lengths, generation lengths, diurnal patterns, SLO requirements), but they share a common GPU fleet. Deploying HeteroScale with per-tenant Deployment Groups enables hardware specialization—each tenant's prefill instances land on compute-optimized GPUs, decode instances on memory-bandwidth-optimized GPUs—while the P/D ratio maintenance ensures that no single tenant's workload shift starves others of prefill or decode capacity. The 41.3% GPU usage reduction observed for the dialogue service (Section 4.3) translates directly to cost savings at the platform level: serving the same aggregate token throughput with 41% fewer GPU-hours. For a platform processing trillions of tokens daily across tens of tenants, this represents millions of dollars in annual infrastructure savings. The heterogeneous hardware matching further compounds these savings: the paper cites prior work [58] showing 41% cost inflation from homogeneous provisioning, and HeteroScale's scheduler directly addresses this by placing workloads on appropriate GPU types without sacrificing network affinity.

On-demand scaling for bursty enterprise LLM deployments. An enterprise deploying an internal LLM-powered application (e.g., document summarization for a legal team, code generation for a development team) experiences highly bursty traffic—intense usage during business hours, near-zero usage overnight and on weekends. Static provisioning for peak load means GPUs sit idle ~70% of the time. HeteroScale's decode-TPS-driven proportional control (Algorithm 2) with soft scaling in (Section 3.6) enables aggressive scale-in during valleys while providing safety: if a few late-night users trigger a scale-out need, the system responds within one evaluation cycle; if a scale-in proves premature, the soft-scaling observation period catches SLO degradation and reinstates instances without cold-start delay. The paper's per-service before/after data (prefill GPU utilization from 46.8% to 76.2%, Section 4.3) quantifies the utilization improvement for a single service. For an enterprise with 100 GPUs, this represents roughly 30 GPUs that can be repurposed or not purchased. The anti-flapping mechanisms (Section 3.6) are critical in this setting: bursty traffic with deep valleys is precisely the regime where naive autoscalers oscillate, and HeteroScale's cooling periods and hysteresis thresholds are designed to prevent this.

Model update rollouts with automated capacity transition. When a model provider upgrades a serving model (e.g., Doubao-Seed-1.6 → a new version), the new model may have different computational characteristics—different optimal P/D ratio, different per-GPU throughput, different memory requirements. Manually re-provisioning the fleet for the new model is error-prone and slow. HeteroScale's workload-centric policy curation pipeline (Algorithm 1) automates this transition: operators run a pressure test on the new model, the pipeline determines the new optimal P/D ratio and expected per-instance metrics, and the configuration is deployed. HeteroScale then scales the new model's Deployment Groups to the appropriate size as traffic is gradually shifted. The paper's demonstration that optimal P/D ratios vary from 1P/5D to 9P/1D across services (Section 4.1) implies that manual ratio selection for a new model would be guesswork. The automated pressure-test-and-deploy pipeline eliminates this guesswork and the associated risk of provisioning the wrong ratio—which, as Figure 4 shows, can cripple throughput by creating prefill or decode bottlenecks. For a provider updating models monthly, this represents dozens of engineer-hours saved per update cycle.

Heterogeneous hardware procurement and capacity planning. The paper's characterization of how different GPU types map to prefill vs. decode efficiency has direct implications for hardware purchasing decisions. An organization planning a GPU cluster for LLM serving can use HeteroScale's pressure testing methodology to determine: for our workload mix (distribution of prompt lengths, generation lengths, concurrent users), what ratio of compute-optimized to memory-bandwidth-optimized GPUs minimizes total cost while meeting SLOs? Rather than buying a homogeneous fleet of general-purpose GPUs, the organization buys a heterogeneous mix informed by empirical P/D ratio measurements. The paper's finding that decode is the bottleneck for most services (since decode TPS is the limiting scaling signal) suggests that memory-bandwidth-optimized GPUs should constitute the majority of the fleet, with compute-optimized GPUs for prefill. The RDMA Subgroup priority system then informs the physical cluster layout: place heterogeneous GPU types under the same S1 switch where possible (high-priority subgroups) to enable mixed-hardware Deployment Groups with minimal KV cache transfer latency. This is a direct operational translation of the paper's architectural insights into hardware procurement strategy.