PowerInfer-2: Fast Large Language Model Inference on a Smartphone

🎯 Pitch

A 47-billion-parameter language model runs on a regular smartphone at 11.68 tokens per second—the first time an LLM of this scale has been served on a mobile device. PowerInfer-2 makes this possible by splitting the model into neuron clusters that are dynamically routed to the phone’s NPU or CPU based on their sparsity, while a fine-grained I/O pipeline overlaps flash reads with computation to overcome severe mobile storage bottlenecks.

1. Executive Summary

This paper introduces PowerInfer-2, a smartphone-based LLM inference framework that decomposes matrix operations into fine-grained neuron clusters (groups of neurons with similar activation patterns, dynamically sized at runtime per hardware target) to enable models exceeding device memory capacity. Evaluated on two OnePlus smartphones with Qualcomm Snapdragon processors running models from 7B to 47B parameters, PowerInfer-2 combines Sparsity-Aware Adaptation (routing dense-activated neuron clusters to the NPU while sparse clusters run on CPU, with dynamic ratio adjustment as batch sizes shift during decoding) with I/O-Aware Orchestration (a neuron-cluster-level pipeline that overlaps computation with UFS storage reads, plus a segmented neuron cache and differentiated I/O strategies for hot vs. cold neurons). The system achieves up to a 27.8× speedup over llama.cpp and a 3.84× average speedup over LLMFlash during offloaded decoding, delivers 11.68 tokens/s on a 47B model — the first system to serve an LLM of that scale on a smartphone — and reduces memory usage by 40% for 7B models while matching in-memory baselines, establishing that neuron-cluster-grained decomposition enables efficient mobile LLM inference only when both the heterogeneous XPU compute landscape and UFS storage bottlenecks are jointly addressed through fine-grained, difficulty-adaptive scheduling.

2. Context and Motivation

The Core Gap: LLM Deployment on Smartphones Is Stuck at Small Models

The fundamental problem this paper tackles is straightforward but technically demanding: modern smartphones cannot run large language models fast enough, and the models they can run are too small to be useful. The paper documents a specific, quantified gap. Google's Gemini Nano, the flagship on-device LLM at the time of writing, contains only 3.25B parameters and fits in under 2GB of memory (Section 1). Meanwhile, the models that power capable AI experiences on servers and high-end PCs routinely reach tens or hundreds of billions of parameters — Llama-13B, Mixtral-47B, and beyond. These larger models consistently outperform smaller ones on reasoning, code generation, and complex instruction-following tasks due to the well-established neural scaling laws (Kaplan et al., 2020).

This is not just a matter of slower responses or slightly worse answers. The capability gap between a 3.25B model and a 47B model is qualitative — tasks that a larger model handles correctly may be impossible for the smaller one regardless of how much inference-time patience a user has. The paper is therefore addressing a deployment barrier that prevents smartphone users from accessing the full capabilities that modern LLMs offer. The three motivations the authors list in Section 1 are all practical and user-facing: real-time AI assistance without network latency, privacy preservation through on-device processing, and reliable operation regardless of connectivity.

Why This Problem Resists Easy Solutions

Smartphones differ from the PC and server environments where most LLM inference systems were developed in two specific, hardware-rooted ways. These differences are not matters of degree — they change the structure of what solutions are viable.

1. The Sparse Computing Gap. High-end PCs accelerate sparse matrix operations using discrete GPUs that handle both dense and irregular sparse patterns efficiently relative to CPUs. PowerInfer (Song et al., 2024) and LLMFlash (Alizadeh et al., 2024) exploit this: they predict which neurons will activate for a given input, then route "hot" (frequently activated) neurons to the GPU for dense computation and "cold" (rarely activated) neurons to the CPU for sparse computation, achieving substantial speedups on PC hardware.

Smartphones break this design. Their Neural Processing Units (NPUs) — the analog of discrete GPUs in the mobile system-on-chip — deliver excellent throughput for dense matrix operations but have no dedicated hardware support for unstructured sparse computation. The paper reports (Section 2.3.1, Figure 3a) that sparse computation primitives implemented on Qualcomm's NPU actually run slower than equivalent CPU implementations. This is the inverse of the PC situation. Consequently, porting PowerInfer or LLMFlash directly to a smartphone forces all sparse computation onto the CPU, leaving the NPU underutilized and failing to extract the full memory bandwidth from the smartphone's unified memory architecture (UMA). The paper measures this (Section 2.4, Table 2): on an in-memory 7B model with no offloading, PowerInfer achieves only 41.1 GB/s of memory bandwidth utilization on the CPU, versus 59.6 GB/s when NPU and CPU are engaged together. The system leaves nearly a third of available bandwidth on the table because its architecture assumes GPU-grade sparse compute that mobile NPUs don't provide.

2. The Storage Performance Gap. When a model exceeds available DRAM — which is the norm for models larger than ~3B parameters on smartphones whose available application memory ranges from 11GB to 19GB (Table 3) — weights must be dynamically loaded from flash storage during inference. Here, the gap is quantitative but severe. The paper measures UFS 4.0 (the flash storage in the OnePlus 12) against NVMe SSDs found in PCs (Section 2.3.2): sequential read bandwidth is roughly half (4 GB/s vs. 7.5 GB/s), but random read performance diverges by a factor of 12× (100K IOPS vs. 1,200K IOPS).

This random read penalty is critical because sparse activation patterns — the very phenomenon that PowerInfer and LLMFlash exploit to reduce computation — produce random, non-contiguous memory access patterns on the weights that must be loaded. When a predictor identifies a specific set of cold neurons to activate, those neurons' weights are scattered across flash storage. Loading them requires random reads at the flash's IOPS limit, which on UFS is catastrophically low. LLMFlash introduced bundling strategies to co-load correlated neurons and reduce random I/O operations, but as the paper's measurements show (Section 2.4, Table 2), even with these optimizations, I/O overhead accounts for 76.7% of total latency when 50% of FFN weights are offloaded on mobile. The computation units sit idle 59–69% of the time waiting for storage. On a PC, LLMFlash achieves acceptable speeds because NVMe's far higher random-read throughput can supply weights faster than the compute units process them. On a smartphone, the storage pipe is too narrow, and the compute units starve.

Where Prior Approaches Fall Short

The paper evaluates the two most directly comparable systems: PowerInfer (designed for consumer-grade GPUs, extended by the authors to support flash offloading) and LLMFlash (designed for Apple Silicon Macs with unified memory and NVMe storage). Both were state-of-the-art at the time for memory-constrained LLM inference. The paper ports both to a OnePlus 12 and measures their performance with 50% of FFN weights offloaded (Section 2.4, Table 2).

The results are stark: PowerInfer drops from 12.4 tokens/s (in-memory) to 1.4 tokens/s (89% reduction). LLMFlash drops from 12.9 to 2.3 tokens/s (82% reduction). Neither system reaches interactive speeds (typically considered ~10 tokens/s for reading-comfortable text generation). The underlying causes are directly traceable to the two gaps described above:

PowerInfer's design assumes GPU-accelerated sparse compute. On a smartphone, sparse operations fall to the CPU, which cannot saturate memory bandwidth alone. The NPU, which could help, sits idle because PowerInfer has no mechanism to split computation between NPU and CPU — its architecture is binary: hot neurons go to GPU, cold to CPU. When the "GPU" (NPU) can't handle sparse patterns, cold neurons dominate the CPU, creating a bottleneck.
LLMFlash's bundling and caching strategies assume NVMe-grade random I/O. On UFS, even with bundling, the random reads required for cold neurons overwhelm the storage subsystem. The paper reports (Table 4) that LLMFlash spends 76.7% of its time on I/O and only 23.3% on computation. The compute units are idle because the storage cannot feed them fast enough.

Beyond these two systems, the paper surveys the broader landscape:

llama.cpp (Gerganov, 2024) is the most widely used CPU-based framework and supports offloading via memory-mapped files (mmap), but it has no neuron-level sparsity awareness, no predictor, and no XPU coordination. The paper treats it as a lower-bound baseline: it achieves 0.4 tokens/s on Bamboo-7B with 50% offloading (Figure 14), roughly 28× slower than PowerInfer-2's optimized configuration.
QNN (Qualcomm's proprietary inference engine) leverages the NPU effectively for dense inference but cannot handle models that exceed memory, lacks CPU-NPU co-processing, and supports only coarse per-channel quantization that degrades accuracy (Table 7).
MLC-LLM (MLC team, 2024) accelerates inference via mobile GPU, but GPU performance on matrix-vector operations is actually worse than both NPU and CPU on the tested hardware (Figure 3a), and GPU usage competes with rendering tasks, which manufacturers avoid in production (Section 2.3.1).
STI (Guo et al., 2023) introduced I/O-computation pipelining for BERT-scale models (~0.33B parameters) on embedded devices, but its mechanisms don't scale to the multiple-gigabyte weight matrices of modern LLMs.

A Deeper Challenge: Dynamic Sparsity Under Advanced Decoding

The paper identifies a problem that neither PowerInfer nor LLMFlash anticipated: sparsity patterns change with batch size, and advanced decoding strategies like Best-of-N sampling cause batch sizes to vary during a single inference session (Section 2.2, Figure 2).

In basic autoregressive sampling (Figure 1a), the model processes one token at a time, and the batch size is always 1. In Best-of-N sampling (Figure 1b), the model generates N candidate sequences simultaneously, processes tokens for all active sequences in parallel, and selects the best final response. During such a session, the effective batch size starts at N and gradually decreases as individual sequences reach their end tokens and terminate — from N down to 1 over the course of generation.

This has a dramatic effect on activation sparsity. The paper's analysis of Bamboo-7B's layer 10 (Figure 2) reveals that at batch size 1, fewer than 1% of neurons are activated for any given token — the pattern is highly sparse and irregular. At batch size 32, approximately 75% of neurons activate because different tokens in the batch trigger different neurons, and the union of all activations covers most of the network. Sparsity collapses as batch size grows.

For an inference system that partitions neurons into hot and cold sets, this means the optimal partition changes during generation. A neuron cluster that is sparse and CPU-suitable at batch size 1 may become part of the dense majority at batch size 4. A static partition — the approach taken by PowerInfer, which computes hot/cold assignments offline and never changes them — will be wrong for some portion of the decoding process. PowerInfer-2 is designed to handle this dynamic explicitly: its Adaptive Neuron Engine monitors batch size in real time and adjusts the CPU-NPU workload ratio accordingly (Section 4.1.3).

Flash Storage I/O Characteristics That Shape the Design

The paper's analysis of UFS 4.0 storage (Section 2.3.2) uncovers four performance characteristics that directly inform PowerInfer-2's design choices. These are not ancillary observations — each one maps to a specific mechanism in the system:

Block size impact. Sequential reads range from 450 MB/s at 4KB blocks to 4 GB/s at 512KB blocks. This enormous 8.9× range means that how data is grouped for loading matters as much as how much data is loaded. PowerInfer-2 exploits this through its Flexible Neuron Loading mechanism (Section 4.4): hot neurons, which are known in advance and activated densely, are loaded in large sequential blocks. Cold neurons, activated sparsely, require random reads — but by bundling corresponding neurons across the Gate, Up, and Down matrices (which have 80% co-activation probability) into a single 24KB read, the system moves from the punishing low end of the bandwidth curve to a more favorable operating point.

Data range sensitivity. Random read throughput within a 128MB range is 1 GB/s; across 512MB, it drops below 850 MB/s (Figure 3b). The penalty for accessing a wider range is ~15%, and it's most severe for small block sizes. This motivates PowerInfer-2's segmented neuron cache design (Section 4.2): rather than treating all neurons uniformly, the system maintains separate hot and cold regions with different eviction granularities, and the planner considers data locality when assigning neurons to these regions.

CPU core dependency. 4KB random reads achieve 1,076 MB/s on a big core (3.3 GHz) but only 762 MB/s on a little core (2.2 GHz) — a 29% gap (Table 1). I/O performance is not just a property of the flash chip but of the CPU core handling driver operations (interrupt servicing, command queue management). This finding directly motivates PowerInfer-2's decision to dedicate a specific CPU core (chosen by the offline planner based on hardware profiling) to I/O operations, rather than spreading I/O across available cores.

Limited concurrency. UFS uses a single command queue, unlike NVMe's multiple queues. The paper reports that using multiple cores for I/O degrades performance by up to 40% due to queue contention. This rules out the straightforward approach of parallelizing I/O across cores and reinforces the need for a single dedicated I/O thread with carefully scheduled operations — exactly the architecture PowerInfer-2 adopts in its neuron-cluster pipeline (Section 4.3).

How PowerInfer-2 Positions Itself

The paper frames PowerInfer-2 not as an incremental improvement over prior systems but as a fundamentally different decomposition granularity that enables solutions previously impossible on mobile hardware. The core abstraction shift is from matrix-level operations (the default in PowerInfer, LLMFlash, and essentially all prior LLM serving systems) to neuron-cluster-level operations (Section 3.1).

Under a matrix-level view, inference processes entire weight matrices: load the matrix, multiply by the input vector, produce the output. If part of the matrix is in flash and part is in memory, the system must wait until all required portions are available before computing. Even with pipelining (as LLMFlash does), the pipeline stages are coarse — entire matrices or large sub-blocks — leaving substantial idle time when some portions are slow to load.

Under a neuron-cluster-level view, the matrix is decomposed into independently processable neuron clusters. As soon as one cluster's weights are available (whether from cache or flash), computation on that cluster can begin, and it can overlap with I/O operations for other clusters — including clusters from different matrices. This finer granularity is what enables PowerInfer-2's neuron-cluster pipeline (Figure 6b) to substantially reduce idle bubbles compared to matrix-level pipelining (Figure 6a).

Crucially, this decomposition is not just about pipelining. It also enables heterogeneous scheduling: a neuron cluster can be assigned to the NPU or CPU based on its specific characteristics (size, activation density, storage location), rather than the all-or-nothing assignment that matrix-level partitioning forces. The NPU handles large, dense clusters; the CPU handles small, sparse clusters — and the system adjusts this assignment dynamically as sparsity patterns shift with batch size.

The paper's positioning is that neither the PC-derived solutions (PowerInfer, LLMFlash) nor the mobile-optimized but memory-bound solutions (QNN, MLC-LLM, llama.cpp) address the full stack of mobile constraints simultaneously. PowerInfer-2 claims to be the first system that jointly addresses: (1) mobile NPU's inability to handle sparse compute, (2) UFS's severe random-read penalty, (3) dynamic sparsity under changing batch sizes, and (4) the need for models larger than physical memory — all within a single, integrated design organized around the neuron cluster abstraction.

3. Technical Approach

3.1 Reader Orientation

PowerInfer-2 is a smartphone-based inference engine that runs large language models — including models too big to fit in the phone's memory — by splitting the computation into small, independently processable chunks called neuron clusters and dynamically assigning each chunk to either the NPU or CPU based on how densely activated it is, while simultaneously overlapping computation with flash storage reads so that the processing units rarely wait for data. The core shape of the solution is a fine-grained decomposition + adaptive scheduling + I/O hiding triple: decompose monolithic matrix multiplications into neuron clusters that can be independently loaded and computed, schedule each cluster on the most appropriate processing unit given current sparsity conditions, and pipeline I/O operations for cold neurons with computation for previously loaded clusters to hide the smartphone's extremely slow random-read flash performance.

3.2 Big-Picture Architecture (Diagram in Words)

PowerInfer-2 operates in two phases: an offline planning phase and an online inference phase (Figure 4 in the paper).

Offline Planner (executed once per model+device combination):

Takes as input the LLM weights, a calibration dataset (~10M tokens from Wikipedia and RefinedWeb), and hardware specifications of the target smartphone (NPU performance, CPU core configuration, UFS I/O characteristics, memory bandwidth).
Runs the model on the calibration data to collect neuron activation frequencies under various batch sizes.
Classifies each neuron into a hot cluster (frequently activated, will be processed by NPU) or a cold cluster (rarely activated, will be processed by CPU), with cluster sizes determined jointly by activation patterns and hardware I/O capabilities.
Produces an execution plan containing: (1) a neuron plan specifying which neurons belong to which clusters, (2) a hardware plan specifying NPU computation graph configurations for each batch size, and (3) a storage plan specifying how neuron weights are organized on flash.

Online Inference Engine (runs for every query):

Adaptive Neuron Engine (Section 4.1): Receives the input prompt. During prefill, routes all computation to the NPU in dense mode while a dedicated CPU core asynchronously preloads weights from flash. During decoding, splits FFN computation between NPU (dense hot clusters) and CPU (sparse cold clusters with predictor-based selective computation), dynamically adjusting the split ratio as batch size changes.
Segmented Neuron Cache (Section 4.2): Maintains three in-memory regions — attention weights (fixed, always resident), hot region (NPU-destined clusters, evicted cluster-level via LRU), and cold region (CPU-destined individual neurons, evicted neuron-level via LRU). Serves weight requests from compute units, triggering flash I/O only for cache misses.
Neuron-Cluster Pipeline (Section 4.3): Coordinates the five sequential stages of processing each neuron cluster — prediction, Gate weight I/O, Gate compute, Up/Down weight I/O, Up/Down compute — across multiple clusters from potentially multiple matrices, using one dedicated I/O thread and multiple compute threads to keep all units busy.
Flexible Neuron Loading (Section 4.4): Handles I/O for cache misses using differentiated strategies: large sequential reads for attention weights and hot neuron clusters, bundled random reads (aggregating corresponding neurons across Gate, Up, and Down matrices) for cold neurons, with model-specific optimizations for quantized weights (two-phase loading to avoid unnecessary I/O when Gate neuron outputs are zero).

Information flows as follows: prompt enters → Adaptive Neuron Engine determines prefill strategy → NPU processes prompt while CPU preloads subsequent layer weights → decoding begins → per FFN layer, the engine looks up the neuron plan for current batch size → hot cluster weights requested from cache (or preloaded from flash), cold neuron indices fed to predictor → predictor outputs which cold neurons will activate → activated cold neuron weights requested from cache or flash → neuron-cluster pipeline orchestrates compute and I/O across both XPU types → results merged → next layer.

3.3 Roadmap for the Deep Dive

First, the neuron cluster abstraction itself (Section 3.1) — what it is, how cluster boundaries are determined, and why it is the enabling concept for everything that follows.
Second, the offline planning process (Section 5) — how the system analyzes a model and device to produce the neuron plan, hardware plan, and storage plan, since all online mechanisms depend on these plans.
Third, the Adaptive Neuron Engine (Section 4.1) — the NPU-centric prefill strategy and the hybrid CPU-NPU decoding strategy with dynamic ratio adjustment, since this is where the execution plan meets runtime sparsity.
Fourth, the Segmented Neuron Cache (Section 4.2) — the temperature-based hot/cold region design, LRU policies at different granularities, and dynamic region resizing, since this determines which weights hit memory vs. trigger I/O.
Fifth, the Neuron-Cluster-Level Pipeline (Section 4.3) — the five-stage decomposition of cluster processing and how it enables I/O-computation overlap across matrix boundaries, since this is the mechanism that hides UFS latency.
Sixth, the Flexible Neuron Loading (Section 4.4) — the differentiated I/O strategies for attention weights, hot neurons, and cold neurons, including the Gate-Up-Down bundling and two-phase loading for quantized models, since this maximizes the usable bandwidth from UFS.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems design paper whose core idea is that decomposing FFN matrix multiplications into neuron clusters — groups of neurons with similar activation patterns, sized dynamically to match hardware characteristics — enables a smartphone to jointly solve three problems that previously had to be traded off against each other: utilizing the NPU for dense computation despite its inability to handle sparse patterns, hiding the catastrophic random-read latency of UFS flash storage, and adapting to shifting sparsity patterns under advanced decoding strategies.

The Neuron Cluster Abstraction

A neuron cluster is a contiguous group of neurons from a single FFN layer that share the same hardware target (NPU or CPU) and are processed as a single unit for scheduling, I/O, and computation purposes. The key property that makes neuron clusters different from prior decompositions (matrix-level blocks in LLMFlash, hot/cold splits in PowerInfer) is that cluster boundaries are not fixed by matrix structure — they are determined jointly by activation statistics and hardware characteristics during offline planning.

The paper defines two categories of neuron clusters (Section 3.1):

Hot clusters: large groups of frequently activated neurons (high activation frequency across the calibration dataset). These clusters are processed by the NPU because their high activation probability means they will almost always be needed, making dense matrix multiplication (the NPU's strength) the right computational primitive. The cluster size for hot neurons is set large because the NPU benefits from operating on substantial contiguous memory regions, and the dense computation pattern means there is no waste from computing neurons that end up inactive.
Cold clusters: small groups (down to individual neurons) of rarely activated neurons. These clusters are processed by the CPU because their low and irregular activation patterns make sparse computation (the CPU's strength relative to the NPU) the right primitive. The cluster size for cold neurons is set small because the CPU's predictor-based selective computation means only predicted-active neurons are computed — bundling inactive neurons into a cluster would waste computation.

The size of a cluster is adaptively determined at runtime based on "both hardware characteristics and activation frequencies" (Section 3.1). This means that the same neuron might belong to a differently-sized cluster depending on the target device's NPU tile size, CPU vector width, cache line size, and I/O characteristics. The offline planner (Section 5) profiles the target device and sizes clusters to maximize throughput on each compute unit.

Why this abstraction matters: Prior systems partitioned at the matrix level. PowerInfer's split between GPU and CPU was binary and static — an entire row/column range went to one processor. LLMFlash bundled co-activated neurons into row-column groups but still operated within the matrix structure. Under a matrix-level view, if any neuron in a block needs to be computed, the entire block must be loaded. Under the neuron-cluster view, clusters can be individually loaded, individually scheduled, and individually assigned to processors. This enables:

Heterogeneous scheduling: The NPU gets large dense clusters that match its throughput profile; the CPU gets small sparse clusters that match its low-overhead random access profile. Neither processor handles patterns it is bad at.
Fine-grained I/O pipelining: As soon as one cluster's I/O completes, it can start computing immediately, without waiting for other clusters from the same matrix. Computation for cluster A (matrix i) can overlap with I/O for cluster B (matrix i+1), breaking the matrix barrier that caused idle bubbles in prior systems.
Dynamic repartitioning: As batch size changes and sparsity patterns shift, neuron clusters can be reassigned between NPU and CPU by adjusting which clusters are loaded into which processor's computation graph, without changing the underlying weight storage layout.

Offline Planning: Neuron Classification and Execution Plan Generation

The offline planner (Section 5) is responsible for producing the static configuration that the online inference engine consumes. It runs once per (model, device) pair and produces three artifacts: a neuron plan, a hardware plan, and a storage plan.

Step 1: Activation Profiling. The planner runs the target model on a calibration dataset containing 10M+ tokens from Wikipedia and RefinedWeb. For each neuron in each FFN layer, it records the activation frequency — the fraction of tokens for which that neuron's ReLU-family activation function produces a non-zero output. This profiling is done at multiple batch sizes (1 through the maximum expected, typically up to 32 or more) because activation patterns change with batch size (Section 2.2, Figure 2). A neuron that is rarely activated at batch size 1 (individual tokens are sparse) may become frequently activated at batch size 32 (different tokens in the batch activate different neurons, and the union covers most of the network).

Step 2: Hardware Profiling. The planner profiles the target device's specific hardware characteristics:

NPU performance for matrix multiplications at various matrix dimensions (to determine optimal cluster sizes for the NPU).
CPU performance for sparse matrix-vector products at various sparsity levels (to determine when a cluster is better on CPU vs. NPU).
Memory bandwidth for CPU-only, NPU-only, and combined CPU+NPU access patterns (to determine whether parallel XPU usage saturates available bandwidth or causes contention).
UFS I/O performance for sequential reads at different block sizes and random reads across different data ranges (to determine prefetch sizes and I/O thread assignment).

Step 3: Hot/Cold Classification. For each neuron in each FFN layer, the planner decides whether it belongs to a hot cluster or a cold cluster. The decision rule (paraphrased from Section 5) is:

A neuron is classified as hot if its aggregate activation frequency across the expected batch size range is high enough that it will almost always be computed, making dense NPU processing efficient.
A neuron is classified as cold if its activation is sparse and irregular enough that CPU-based selective computation (only computing predicted-active neurons) saves more work than the NPU's dense throughput advantage would provide.

The exact threshold depends on two factors that the planner jointly optimizes:

The inherent activation pattern of the neuron under different batch sizes. Neurons with consistently high activation across batch sizes are hot; neurons with low or highly variable activation are cold.
The hardware I/O budget for prefetching. Hot neurons are asynchronously prefetched from flash during the attention block computation of the previous layer. The total size of hot neuron weights must be small enough that the prefetch I/O (using sequential reads) can complete within the attention computation time. The planner "carefully balances the number of hot neurons based on the available I/O bandwidth and attention block computation time" (Section 5) — if too many neurons are classified as hot, the attention block will finish before the prefetch completes, and the NPU will stall waiting for weights. This creates a hard upper bound on hot neuron count that is device-specific.

The planner also generates batch-size-specific neuron plans. For batch sizes 1–2 (high sparsity), more neurons are classified as cold because the predictor can skip most of them. For batch sizes 3–4 (lower sparsity), more neurons are reclassified as hot because the fraction of activated neurons is high enough that dense processing with the NPU's throughput advantage outweighs the overhead of computing some inactive neurons. This produces a set of plans: plan_batch1, plan_batch2, plan_batch4, etc., each with different hot/cold assignments and cluster sizes.

Step 4: Neuron Cluster Formation. Within the hot and cold categories, the planner groups neurons into clusters. The cluster size is determined by hardware characteristics:

Hot clusters are sized to match the NPU's preferred matrix dimensions. The Qualcomm NPU has specific tile sizes at which matrix multiplication throughput peaks; the planner aligns cluster boundaries to these dimensions to avoid padding overhead.
Cold clusters are sized small (down to individual neurons) because the CPU processes them with predictor-based sparsity: only predicted-active neurons within the cluster are computed. Larger cold clusters would include more inactive neurons and waste computation.

The clustering also respects the Gate-Up-Down correspondence (Section 4.4). In a standard FFN block with gated activation (using SiLU or ReGLU variants), there are three weight matrices: Gate ( $W_{\text{gate}}$ ), Up ( $W_{\text{up}}$ ), and Down ( $W_{\text{down}}$ ). The paper finds that "corresponding neurons at the i-th position across these matrices show high correlation, with an 80% co-activation probability" (Section 4.4). Therefore, the i-th neuron in the Gate matrix, the i-th neuron in the Up matrix, and the i-th neuron in the Down matrix are bundled into a single neuron cluster for I/O purposes — even though their computations happen at different stages of the FFN block. This bundling means that a single I/O operation loads all three corresponding weights, maximizing read size and amortizing the random-read seek penalty.

Step 5: Storage Plan Generation. The planner determines the physical layout of neuron weights on flash storage. The storage plan specifies:

Hot neuron clusters are stored contiguously in the order they will be prefetched (matching the attention block computation order) to enable efficient sequential reads.
Cold neuron clusters are stored as individual Gate-Up-Down bundles. The physical layout prioritizes data locality: neurons that tend to co-activate (based on profiling) are placed in nearby flash regions, exploiting the observation that random read throughput degrades with range size (Section 2.3.2, Figure 3b).
For 4-bit quantized models, each bundle occupies 7.5KB raw (2KB per INT4 matrix × 3 matrices + 0.5KB per FP16 scale factor × 3 matrices). The planner aligns bundle size to 8KB for storage efficiency (matching flash page boundaries) but marks them for two-phase loading at runtime.

Step 6: Hardware Plan Generation. The planner pre-generates NPU computation graphs for each expected batch size and corresponding hot neuron ratio. These are static graphs (Qualcomm's NPU uses a graph-execution model) that differ in their operator shapes based on the number of participating hot neurons. The planner also determines:

The number of CPU compute threads and their core assignments (which cores handle Gate computation, which handle Up/Down, etc.).
The dedicated I/O thread's core assignment (must be a big or mid core, since Table 1 shows little-core I/O throughput is 29% lower).
The memory budget allocation between the attention region, hot cache region, and cold cache region for each batch size configuration.

The planner outputs all of this as a single execution plan package that the online inference engine loads at startup.

Online Inference: Adaptive Neuron Engine

The Adaptive Neuron Engine (Section 4.1) is the runtime component that executes the inference computation according to the offline-generated plan, with dynamic adaptation to changing conditions. It implements two distinct strategies for the two inference phases and a monitoring mechanism that adjusts strategy parameters in real time.

NPU-Centric Prefill Strategy (Section 4.1.1, Figure 5a). During the prefill phase, the model processes the entire input prompt (potentially hundreds of tokens) in parallel. This means:

Sparsity collapses. Even though individual prompt tokens activate sparse, non-overlapping sets of neurons, the union of activations across all tokens in the batch covers nearly all neurons. The paper reports (Section 4.1.1) that "neuron activation probability approaches 99.99% in TurboSparse-Mixtral-47B with 128 batch size." There is essentially no sparsity to exploit.
NPU throughput dominates. At large batch sizes (128–512), the NPU's dense matrix multiplication throughput dramatically exceeds the CPU's (Figure 3a: NPU processes matrix-vector at batch size ≥4 faster than CPU, and the gap widens with batch size). The paper reports (Section 2.3.1) that for a 7B INT4 model, the NPU achieves 770 tokens/s for the prefill phase versus 37 tokens/s on GPU and 8 tokens/s on CPU.
Weight loading is sequential and overlapped. Since all neurons are needed, weights are loaded sequentially from flash (exploiting the 4 GB/s sequential read bandwidth) rather than randomly. The system dedicates a single big CPU core to asynchronously preload the next layer's weights into the neuron cache (a shared memory region accessible by the NPU) while the NPU computes the current layer. This is an instance of double-buffering: layer $i$ 's weights are loaded during layer $i-1$ 's computation, so the NPU never waits for I/O. Figure 9 in the paper demonstrates this: for Bamboo-7B with 512-token prompts, the I/O time (820–962ms) is completely hidden within the computation time (916–962ms), with the longer of the two determining the layer latency.
CPU contributions are I/O-only. The CPU cores do not participate in prefill matrix multiplications. They are used exclusively for weight prefetching. The attention computation (which is also dense during prefill) is handled by the NPU as well.

Hybrid CPU-NPU Decoding Strategy (Section 4.1.2, Figure 5b). During autoregressive decoding, each new token is generated sequentially, and the effective batch size equals the number of concurrently active generation sequences. This is typically small (1 for basic sampling, 4–16 for Best-of-N sampling). At these small batch sizes, activation sparsity is high, and the optimal strategy changes:

Attention computation is handled by splitting individual attention heads between NPU and CPU. Each XPU processes its assigned heads in parallel. The paper notes that attention blocks are "inherently dense but relatively small" (Section 4.1.2), so the splitting overhead is negligible relative to the computation savings.
FFN computation is split along the neuron dimension. The neuron engine divides an FFN weight matrix into two sub-matrices:
- The NPU sub-matrix contains all hot neuron clusters. The NPU performs dense matrix multiplication on this entire sub-matrix, computing every neuron regardless of whether it activates — but since these are hot neurons, the waste is minimal.
- The CPU sub-matrix contains all cold neuron clusters. The CPU uses a predictor (trained offline, similar to PowerInfer's approach) to predict which specific neurons within this sub-matrix will activate for the current input token. Only those predicted-active neurons are computed, using sparse matrix-vector multiplication with ARM Neon SIMD extensions for vectorized computation.
The split ratio is specified by the neuron plan for the current batch size. For example, at batch size 1, "it divides a typical FFN matrix (14336×4096) into two equal sub-matrices (7168×4096) for NPU and CPU respectively" (Section 4.1.2). As batch size increases and sparsity decreases, the NPU portion grows.
Result merging. After both XPUs complete their portions, the CPU merges the partial results (which are partial output vectors) to form the complete FFN output. The merge is a simple concatenation/addition operation with negligible overhead.

Dynamic CPU-NPU Adjustment (Section 4.1.3). The ratio of neurons assigned to NPU versus CPU is not static — it changes during a single inference session in response to shifting batch sizes. This is necessary because of advanced decoding strategies like Best-of-N sampling.

The mechanism works as follows:

Monitoring. CPU cores track the batch size by monitoring the creation and completion of decoding sequences. When a Best-of-N session starts with N=4, the batch size is 4. As individual candidate sequences reach their end-of-sequence tokens and terminate, the batch size drops to 3, then 2, then 1.
Ratio adjustment. Each batch size has a corresponding neuron plan with a specific NPU:CPU split. At batch size 4 (relatively dense), the NPU handles "about 70%" of neurons while the CPU handles "about 30%." At batch size 1 (highly sparse), the ratio shifts to 50:50. These are the paper's approximate figures; the exact ratios are device-specific and determined by the offline planner.
Graph swapping. The NPU's computation graph must change when the split ratio changes (since the number of neurons assigned to the NPU changes, altering the matrix dimensions). Because the NPU uses static graph execution, loading a new graph takes time. The engine hides this latency through asynchronous graph loading: when it detects that the batch size will change after the current token (because a sequence just terminated), it initiates loading the new NPU computation graph (typically ~10KB in size) into NPU memory while the NPU is computing the attention block. The attention computation is dense and relatively long, providing a window to complete the graph load. By the time the FFN computation begins, the new graph is active. The paper states that "this asynchronous loading mechanism completely overlaps with NPU computation, ensuring seamless transitions between different computational configurations without introducing additional overhead" (Section 4.1.3).

Why this dynamic mechanism matters: Without it, a system would either (a) use the batch-size-1 plan throughout, underutilizing the NPU during the early, dense portion of Best-of-N generation, or (b) use the batch-size-N plan throughout, misallocating neurons when sparsity increases later. PowerInfer-2's dynamic adjustment keeps the split close to optimal throughout the session. The paper quantifies this in Figure 13: PowerInfer-2 with dynamic adjustment achieves 1.42× speedup over a hypothetical static-configuration baseline by maintaining optimal XPU utilization as N decreases during Best-of-4 sampling.

Online Inference: Segmented Neuron Cache

The neuron cache (Section 4.2) is an in-memory data structure that holds recently used neuron weights, reducing the frequency of flash storage accesses. Its design is driven by the observation that "a small set of hot neurons dominates the connections" (Section 4.2) — a skewed activation distribution where a tiny fraction of neurons accounts for most activations — and by the UFS storage characteristics documented in Section 2.3.2.

Three-Region Design. The cache is divided into three regions with distinct purposes and eviction policies:

Attention region (fixed, preloaded at startup): Holds all attention weights and the KV cache. These weights are always needed for every token (attention is dense) and are relatively small compared to FFN weights. They are loaded once during initialization and never evicted. This region's size is determined by the model architecture — it simply holds everything attention-related.
Hot region (dynamic, cluster-level LRU eviction): Holds hot neuron clusters destined for the NPU. Neurons are grouped into the same clusters defined by the offline planner, forming contiguous dense matrices that the NPU can process efficiently. The eviction unit is the entire cluster — when the LRU policy selects a victim, the whole cluster is discarded from memory (without writing back to flash, since flash holds the master copy). The eviction granularity being cluster-level rather than neuron-level is crucial: evicting individual neurons from within a hot cluster would fragment the dense matrix, destroying the NPU's throughput advantage.
Cold region (dynamic, neuron-level LRU eviction): Holds individual cold neurons (or small cold clusters) destined for the CPU. Since cold neurons are processed sparsely and individually, the eviction unit can be the individual neuron. This finer granularity allows the cold region to retain precisely the most recently activated cold neurons without wasting cache space on inactive neighbors within the same cluster.

The cache uses an LRU (Least Recently Used) replacement policy for both dynamic regions, but at different granularities: cluster-level for hot, neuron-level for cold.

Why two separate regions? The paper argues that prior work like LLMFlash's bundling approach "overlooks the skewed activation patterns where a small set of hot neurons dominates the connections, leading to redundant loading of these frequently activated neurons across different bundles" (Section 4.2). By explicitly separating hot and cold neurons, the cache avoids this redundancy: a hot neuron is loaded once into the hot region and retained there across many tokens, rather than being repeatedly loaded and evicted as part of different co-activation bundles. The paper further notes that "after excluding hot neurons, the co-activation probability among remaining neurons drops below 20%" (Section 4.2), meaning that bundling cold neurons together (as LLMFlash does) provides little benefit — most neurons in a bundle won't activate together anyway, so the I/O cost of loading the bundle is largely wasted.

Dynamic Region Resizing. The boundary between hot and cold regions shifts during inference as batch size changes:

When batch size increases (more concurrent sequences, lower sparsity): The hot region expands by taking memory from the cold region. More neurons are classified as hot in the larger-batch plan, and the hot region needs space to hold them. The cold region shrinks by evicting the least recently activated neurons via LRU.
When batch size decreases (fewer concurrent sequences, higher sparsity): The hot region contracts, discarding the least recently activated clusters. The cold region expands to accommodate more individually-cached neurons.

This resizing is triggered by the same batch size monitoring mechanism described in Section 4.1.3. The memory transfer between regions is lightweight: evictions simply discard weights (no write-back), and newly allocated space is filled on-demand as weights are loaded from flash.

Cache Effectiveness. The paper reports (Section 7.2.3) that with 19GB of available memory running TurboSparse-Mixtral-47B, the cache achieves a 96.5% hit rate on average, with the P99 miss rate being 18.9%. This means that for the vast majority of tokens, nearly all needed weights are already in memory. When a miss does occur, it disproportionately affects tail latency: the slowest 1% of tokens experience 18.9% cache misses, which is why the neuron-cluster pipeline (next section) is essential for hiding the latency of those occasional flash reads.

In the ablation study (Figure 14), the Neuron Cache alone — added on top of the baseline with bundling — delivers a 2.82× speedup (from 1.09 to 4.18 tokens/s), making it the single most impactful optimization component in the system.

Online Inference: Neuron-Cluster-Level Pipeline

The neuron-cluster pipeline (Section 4.3) is the mechanism that overlaps computation with flash I/O to hide the latency of cache misses. It is the component that directly addresses the I/O bottleneck that causes prior systems to spend 77–82% of their time waiting for storage (Tables 2 and 4).

The Problem with Matrix-Level Pipelining. The paper first describes why a straightforward matrix-level pipeline (the "straightforward approach," Section 4.3) is insufficient. In a matrix-level pipeline (Figure 6a), the system processes one FFN matrix at a time:

Compute the Gate matrix, loading uncached neurons from flash as needed.
Wait for all Gate I/O to complete before proceeding.
Compute the Up and Down matrices, loading uncached neurons as needed.
Wait for all Up/Down I/O to complete before proceeding.
Move to the next FFN layer.

If a single neuron cluster within the Gate matrix requires a slow random read from flash (because it was a cache miss), all subsequent computations — including computations for neurons that are in cache — must wait. The pipeline stalls at the matrix barrier. The paper illustrates this in Figure 6a with an example where 4 of 8 neuron clusters per matrix are in memory and 4 are in flash. With matrix-level pipelining, the compute threads experience idle "bubbles" while waiting for the flash-resident clusters to load, despite there being in-memory work available in the next matrix.

The Neuron-Cluster Solution (Figure 6b). PowerInfer-2 breaks the matrix barrier by treating neuron clusters from different matrices as interchangeable work items. As soon as one neuron cluster finishes computation, the compute thread picks up the next available cluster — which may belong to a different FFN matrix (even a different Gate/Up/Down sub-matrix) — as long as its weights are already in memory.

Five-Stage Cluster Processing. The processing of each neuron cluster is decomposed into five sequential stages (Section 4.3):

Prediction (Pred): The predictor runs on the input vector to determine which neurons within the cluster will activate. For hot clusters, this stage is trivial (all neurons are predicted active). For cold clusters, the predictor outputs a binary mask indicating which specific neurons to compute.
Gate I/O (GIO): If the Gate matrix weights for the activated neurons are not in cache, they are read from flash. This is typically a random read for cold neurons or a sequential read for hot neurons.
Gate Computation (GC): The Gate matrix multiplication is performed using only the predicted-active neurons (for cold clusters) or all neurons (for hot clusters). The result is the Gate activation vector, which feeds into the element-wise gating operation.
Up/Down I/O (UDIO): If the Up and Down matrix weights for the activated neurons are not in cache, they are read from flash. The paper bundles Up and Down I/O into a single stage because these weights are stored together (as Gate-Up-Down bundles, Section 4.4).
Up/Down Computation (UDC): The Up and Down matrix multiplications are performed, and the gated activation is applied to produce the FFN output for this cluster.

These five stages are pipelined: while one cluster is in the GIO stage (waiting for flash), another cluster can be in the GC stage (computing), and a third can be in the Pred stage (preparing the next work item). The critical design choice is that a compute thread, upon finishing GC for cluster A (from matrix i), can immediately begin UDIO for cluster A (if Up/Down weights need loading) and simultaneously begin GC for cluster B (from matrix i+1, if its Gate weights are already in memory). This interleaving of work across matrices eliminates the idle bubbles visible in Figure 6a.

Thread Architecture. The pipeline uses multiple compute threads and one dedicated I/O thread (Section 4.3):

One I/O thread is pinned to a big or mid CPU core. It handles all flash storage operations: issuing read commands, managing the UFS command queue, and signaling compute threads when data is available. The paper dedicates exactly one core to I/O because "using multiple cores for I/O operations can degrade performance by up to 40% due to command queue contention" (Section 2.3.2). The core type matters: Table 1 shows that a big core achieves 1,076 MB/s for random reads versus 762 MB/s for a little core. The offline planner selects the specific core assignment based on hardware profiling.
Multiple compute threads (typically 4, one per available mid/big core after reserving one for I/O and others for system tasks) execute the Pred, GC, and UDC stages. Each compute thread can work on any cluster from any matrix, with work distribution managed through a shared work queue.
The paper's experimental configuration uses 4 compute threads and 1 I/O thread on the OnePlus 12's 1+5+2 core configuration, with the big core handling I/O and the mid cores handling computation.

Effectiveness. The ablation study (Figure 14) shows that adding the neuron-cluster pipeline on top of the neuron cache delivers a 1.29× additional speedup (from 4.18 to 9.60 tokens/s). The pipeline specifically addresses the remaining latency after caching: even with a 95% cache hit rate, 5% of neurons still require flash reads, and without pipelining those reads would stall the entire computation. The pipeline hides most of this residual I/O latency.

The paper's critical path breakdown in Table 4 quantifies the end-to-end effect: with PowerInfer-2, only 13.7% of time is spent on I/O versus 76.7% for LLMFlash. The compute fraction rises from 23.3% to 86.3% — the pipeline has nearly inverted the compute-I/O ratio.

Online Inference: Flexible Neuron Loading

Flexible Neuron Loading (Section 4.4) is the I/O strategy layer that determines how weights are read from flash when cache misses occur (or, for hot neurons, when they are prefetched). It implements three distinct strategies for three types of weights, each optimized for the specific access pattern and UFS performance characteristics.

Strategy 1: Sequential Preloading for Attention Weights. Attention weights are dense and always needed. They are loaded once at inference startup using large sequential reads (maximum block size) and retained in the attention cache region throughout the session. This strategy is straightforward but critical: if attention weights were loaded on-demand with random reads, the paper's UFS measurements (Figure 3b) show the bandwidth penalty would be severe.

Strategy 2: Sequential Asynchronous Prefetching for Hot Neurons. Hot neuron clusters are loaded from flash using sequential reads during the attention block computation of the previous layer. The mechanism (Section 4.1.1, Figure 5a):

While the NPU computes attention for layer $i$ , a dedicated CPU core issues sequential read commands for the hot neuron clusters of layer $i$ 's FFN block.
The data is loaded into the neuron cache's hot region.
By the time the NPU finishes attention and is ready for FFN computation, the hot neuron weights are already in memory.

This prefetching is possible specifically because hot neurons are, by definition, almost always activated — there is no uncertainty about whether they will be needed. The offline planner (Section 5) sizes the hot cluster set so that the sequential read time fits within the attention computation time, ensuring the prefetch never causes the NPU to stall. This is the mechanism that achieves the complete I/O-computation overlap shown in Figure 9.

Strategy 3: Bundled On-Demand Random Reads for Cold Neurons. Cold neurons present the hardest I/O challenge: they are activated sparsely and unpredictably (the predictor determines which ones at runtime), their weights are scattered across flash, and UFS random read performance is terrible (100K IOPS, 12× slower than NVMe). The paper develops two sub-strategies to maximize throughput under these constraints:

Gate-Up-Down Bundling. Rather than storing the Gate, Up, and Down matrices as separate files (the natural layout for matrix-level access), PowerInfer-2 stores corresponding neurons from all three matrices as a single physical unit on flash. Specifically, the i-th neuron's Gate weights, i-th neuron's Up weights, and i-th neuron's Down weights are stored contiguously as one bundle.

The justification (Section 4.4): "While neurons within a single FFN matrix rarely co-activate after removing hot neurons, corresponding neurons at the i-th position across these matrices show high correlation, with an 80% co-activation probability." This means that if the i-th Gate neuron is predicted to activate, there's an 80% chance that the i-th Up and Down neurons will also be needed — and if they are, loading them in the same I/O operation avoids a second random read with its associated seek penalty.

The bundle sizes and loading strategies are model-specific:

For unquantized models (e.g., Mistral-7B-FP16): Each neuron occupies 2 bytes per weight × 4096 columns = 8KB per matrix. A Gate-Up-Down bundle is 24KB. PowerInfer-2 issues a single 24KB random read, which operates at a higher point on the UFS bandwidth curve than smaller reads (Section 2.3.2, Figure 3b: bandwidth increases with block size for random reads).
For 4-bit quantized models (the practical default for mobile deployment): Each neuron bundle occupies 7.5KB raw (2KB INT4 weights + 0.5KB FP16 scales, per matrix, ×3 matrices). The planner aligns bundles to 8KB for storage efficiency but splits the load into two 4KB operations because "empirical measurements demonstrate superior bandwidth utilization compared to a single 8KB random read" (Section 4.4). This is a UFS-specific quirk: on the tested hardware, two 4KB random reads complete faster than one 8KB random read, likely due to the flash controller's internal parallelism characteristics.

Two-Phase Loading for Quantized Models. For 4-bit quantized models, PowerInfer-2 employs an additional optimization: it loads the Gate matrix weights first, computes the Gate activation, and only loads the Up/Down weights if the Gate neuron's output is non-zero. The reasoning: even with 80% co-activation, there is a 20% chance that the bundled Up/Down weights will not be needed. In a ReLU-family FFN block, if the Gate neuron output is zero, the gating operation zeros out the Up neuron's contribution regardless of what the Up matrix produces. Loading the Up/Down weights in that case would waste I/O bandwidth on weights that won't affect the output.

The two-phase process (Section 4.4):

After the predictor confirms activation, load only the Gate matrix weights (4KB) for the bundle.
Compute the Gate neuron output.
If the output is non-zero, load the Up/Down weights (4KB).
If the output is zero, skip the Up/Down load entirely.

This saves approximately 20% of cold-neuron I/O bandwidth on quantized models, at the cost of a small additional latency for the second-phase load when it is needed. Since UFS random I/O is the primary bottleneck, the bandwidth savings outweigh the latency penalty.

Why these strategies are necessary: The paper's UFS analysis (Section 2.3.2) shows that random read throughput is highly sensitive to both block size and access range. A naive approach — issuing a separate 2KB read for each matrix of each cold neuron — would operate at the worst possible point on the bandwidth curve and leave the compute units idle. The bundling and two-phase loading strategies move the operating point to a region where UFS can deliver enough throughput to keep the neuron-cluster pipeline fed, which is what enables the 13.7% I/O overhead reported in Table 4.

Summary of Design Choices and Their Justifications

Neuron cluster as the processing unit over matrix-level blocks: enables heterogeneous scheduling across NPU/CPU, fine-grained I/O pipelining across matrix boundaries, and dynamic repartitioning under changing batch sizes — none of which are possible at matrix granularity.
Offline planner with joint activation+hardware profiling over static heuristics or online-only adaptation: the NPU's graph execution model requires pre-compiled graphs; the UFS I/O characteristics require device-specific parameter tuning; and batch-size-dependent sparsity patterns require analyzing the model's behavior across the full range of expected conditions.
NPU-centric prefill + hybrid CPU-NPU decoding over uniform NPU or CPU processing: the NPU dominates at large batch sizes (prefill) but loses its advantage at small batch sizes (decoding) due to lack of sparse compute support; the CPU excels at sparse irregular access but cannot match NPU dense throughput. Using each for what it's best at maximizes total throughput.
Dynamic NPU-CPU ratio adjustment over static partitioning: advanced decoding strategies cause batch size to vary during a single session, and the optimal split changes with batch size. Asynchronous graph loading hides the configuration switch latency.
Three-region segmented cache over unified LRU: the skewed activation distribution means a small hot set dominates; mixing hot and cold neurons in a single cache would cause frequent eviction and reloading of hot neurons. Separate regions with cluster-level (hot) and neuron-level (cold) eviction granularities prevent this.
Neuron-cluster pipeline breaking matrix barriers over matrix-level pipelining: UFS random reads are so slow that even occasional cache misses create idle bubbles if the pipeline is bounded by matrix completion. Allowing computation from matrix $i+1$ to proceed while I/O for matrix $i$ completes keeps the compute units utilized.
Gate-Up-Down bundling with two-phase loading over per-matrix I/O: the 80% co-activation correlation makes bundling efficient; the two-phase optimization for quantized models eliminates 20% of wasted I/O by checking Gate output before loading Up/Down weights. Both strategies maximize the effective bandwidth extracted from UFS's limited random-read throughput.

4. Key Insights and Innovations

Innovation 1: Neuron-Cluster Granularity as the Universal Knob for Mobile LLM Inference

The paper's most foundational contribution is a shift in decomposition granularity — from matrix-level operations (the universal default in prior LLM serving systems) to neuron-cluster-level operations — and the recognition that this single abstraction enables solutions to three previously separate problems: heterogeneous scheduling, I/O hiding, and dynamic adaptation. This is not merely a finer-grained version of existing approaches; it changes the structure of what solutions are expressible.

What the field did before. Prior systems operated at matrix granularity. PowerInfer (Song et al., 2024) partitioned entire weight matrices into hot and cold sections, but the partitioning was binary and static — a contiguous range of rows went to the GPU, another to the CPU. LLMFlash (Alizadeh et al., 2024) introduced row-column bundling to co-load correlated neurons, but bundling was a storage optimization layered on top of matrix-level computation — the system still processed matrices as monolithic units, waiting for all required portions to be available before computing. Matrix-level granularity creates a fundamental coupling: if any part of a matrix needs slow I/O, all computation on that matrix stalls. This coupling is tolerable on PCs (where NVMe random reads are fast enough that stalls are short) but catastrophic on smartphones (where UFS random reads are 12× slower).

What PowerInfer-2 does differently. The neuron cluster abstraction decouples three concerns that matrix-level processing binds together: (1) which processing unit handles which computation (NPU for dense clusters, CPU for sparse clusters), (2) when each cluster is loaded from storage (independent I/O scheduling per cluster), and (3) how workload shifts when conditions change (clusters can be reassigned between NPU and CPU without restructuring the storage layout). The key insight is that these three concerns are orthogonal at cluster granularity but coupled at matrix granularity. You cannot heterogeneously schedule parts of a matrix to different processors if you must wait for the entire matrix before computing anything. You cannot pipeline I/O across matrix boundaries if the boundaries are synchronization points. You cannot dynamically adjust processor assignments without redistributing weights if the assignment is defined by matrix partitioning.

This is a fundamental architecture shift, not an incremental refinement. It is analogous to the difference between coarse-grained task parallelism (where the unit of work is a function or loop) and fine-grained data parallelism (where the unit of work is an individual element). The neuron cluster is to LLM inference what the thread is to operating systems: a unit of schedulable work with its own state (weights, activation prediction, I/O status) that can be independently assigned, preempted, and pipelined.

Evidence anchoring the claim. The paper's ablation study (Figure 14) shows that the neuron-cluster pipeline — which is only possible because the system decomposes into clusters — provides a 1.29× speedup on top of an already-optimized configuration with caching and bundling. More tellingly, the critical path breakdown (Table 4) shows that PowerInfer-2 inverts the compute-to-I/O ratio from 23:77 (LLMFlash, matrix-level) to 86:14 (PowerInfer-2, cluster-level). This inversion is not achievable by tuning matrix-level parameters; it requires the ability to interleave computation from one matrix with I/O for another, which is structurally impossible under matrix-level synchronization.

Innovation 2: Dynamic XPU Ratio Adjustment as a First-Class Response to Sparsity Shift

PowerInfer-2 identifies and solves a problem that no prior LLM inference system had even formulated: sparsity patterns change during a single inference session under advanced decoding strategies, and the optimal processor allocation changes with them. This is a diagnostic contribution — the identification of a previously invisible bottleneck — as much as it is a technical solution.

What the field did before. Prior sparsity-aware inference systems (PowerInfer, LLMFlash, DejaVu [Liu et al., 2023]) assumed static activation patterns. They profiled neuron activation frequencies offline, partitioned neurons into hot and cold sets once, and used that partition for all tokens regardless of the decoding context. This assumption holds for basic autoregressive sampling (batch size always 1, sparsity patterns stable) but breaks under techniques like Best-of-N sampling, Monte Carlo Tree Search, or map-reduce decoding, where multiple candidate sequences are generated in parallel and terminate at different times. As the paper's Figure 2 demonstrates, the difference is not subtle: at batch size 1, fewer than 1% of neurons are activated; at batch size 32, approximately 75% are activated. A partition optimized for batch size 1 (CPU-heavy, exploiting high sparsity) will be catastrophically wrong at batch size 32 (where the NPU should handle most work), and vice versa.

What PowerInfer-2 does differently. The system treats the CPU-NPU split ratio as a dynamic variable indexed by batch size, not a static configuration. The offline planner precomputes optimal ratios for each expected batch size and packages them as separate NPU computation graphs. At runtime, a lightweight monitor tracks batch size changes (by counting active and completed sequences), and when the batch size changes — which happens as individual Best-of-N candidates reach their end tokens — the system asynchronously swaps the NPU's computation graph for the new ratio. The graph swap is hidden within the attention computation, so the transition is seamless.

This is more than an optimization; it is a conceptual reframing. It says that sparsity is not a property of the model alone, nor of the input alone, but of the interaction between the model and the decoding algorithm's parallelism. A model has no single "sparsity level"; it has a sparsity surface over batch size. An inference system that does not adapt to this surface is leaving performance on the table for the entire duration of any decoding session that spans multiple batch sizes — which, in the era of Best-of-N and tree-search decoding, is increasingly all of them.

Significance beyond raw numbers. The paper's Figure 13 quantifies the gain — 1.42× over a static-configuration baseline during Best-of-4 sampling — but the deeper contribution is the diagnostic framework. It tells system designers: when you deploy an LLM with advanced decoding, do not profile at batch size 1 and call it done. Profile across the batch size range, and build adaptation into the runtime. This finding will only grow in importance as decoding strategies become more sophisticated (speculative decoding with dynamic tree sizes, multi-agent debates with varying numbers of active participants, etc.).

Innovation 3: UFS Performance Characteristics as First-Order Design Inputs, Not Afterthoughts

The paper treats smartphone flash storage not as a generic "slower SSD" but as a system with specific, idiosyncratic performance characteristics that must be designed against from the architecture level. This is a methodological contribution: it demonstrates that on mobile devices, storage I/O characteristics are not something you optimize for after building the compute pipeline — they are inputs to the fundamental decomposition and scheduling decisions.

What the field did before. PC-centric systems like LLMFlash treat storage as a uniform bandwidth pipe. LLMFlash's bundling and caching strategies assume that reducing the number of I/O operations is the primary goal, and that the storage subsystem will deliver reasonable throughput regardless of access pattern. On NVMe SSDs, this assumption roughly holds: random reads are slower than sequential reads, but the gap is manageable (12× is bad but not catastrophic at 1.2M IOPS). Porting these strategies to UFS without reconsidering the assumptions produces the results in Table 4: 76.7% of time spent on I/O because the storage simply cannot keep up with the random-access pattern that the compute pipeline demands.

What PowerInfer-2 does differently. Four UFS-specific characteristics become first-order inputs to the system design:

Block size sensitivity (450 MB/s at 4KB vs. 4 GB/s at 512KB sequential) directly determines the Gate-Up-Down bundling strategy. The paper sizes bundles to 24KB for FP16 models (one large random read instead of three small ones) and splits 4-bit bundles into two 4KB reads (because empirically, 2×4KB outperforms 1×8KB on UFS 4.0). These are not generic "use larger reads" optimizations; they are specific to the tested flash controller's bandwidth curve.
Data range sensitivity (1 GB/s within 128MB, 850 MB/s across 512MB for random reads) motivates the storage plan's data locality decisions. The offline planner arranges co-activated cold neurons in nearby flash regions to keep the active working set compact, exploiting the ~15% throughput difference.
CPU core dependency (1,076 MB/s on big core vs. 762 MB/s on little core for random reads) forces the system to dedicate a specific big/mid core to I/O rather than treating I/O as a background task that can run anywhere. This is a non-obvious constraint: on most systems, I/O performance is a property of the storage device, not the CPU configuration. On UFS, the CPU core matters.
Single command queue (using multiple I/O cores degrades performance by 40%) rules out the straightforward approach of parallelizing I/O across cores, forcing a single-threaded I/O architecture with carefully scheduled operations — which in turn requires the neuron-cluster pipeline to keep that single I/O thread fed with useful work while compute threads process previously loaded clusters.

Why this is significant beyond PowerInfer-2. This is not just a "we tuned I/O" result. It establishes a methodology for mobile systems design: characterize the storage's performance surface (block size × range × core dependency × concurrency), make these characteristics explicit inputs to the architecture, and let them drive decisions at every level from data layout through scheduling through thread assignment. Any future system that runs LLMs on smartphones — or, more broadly, any I/O-intensive workload on UFS-backed devices — will need to engage with these same characteristics. The paper provides both the characterization methodology and a worked example of how characterization results translate into system mechanisms.

Evidence. The most direct evidence for this claim's importance is the ablation in Figure 14. The "Bundle" optimization (which implements the UFS-aware I/O strategies) provides a 2.73× speedup on its own (from 0.40 to 1.09 tokens/s). This is the foundation on which all subsequent optimizations build. Without UFS-specific I/O strategies, caching and pipelining would be rearranging deck chairs — the storage pipe would still starve the compute units.

Innovation 4: The Verifier Over-Optimization Analogy — Sparsity Is Exploitable Only Within Hardware Limits

While not explicitly framed this way in the paper, PowerInfer-2 reveals an insight with parallels to the verifier over-optimization findings in the first paper summarized in the reference example: sparsity is a resource that can be over-exploited. There is an optimal degree of sparsity exploitation for a given hardware configuration, and pushing beyond it — by routing too much computation to a processor ill-suited for sparse patterns, or by fragmenting I/O into accesses too small for the storage subsystem — produces diminishing or negative returns.

The parallel. In the reference paper on test-time compute, beam search "over-optimizes" the PRM verifier signal: aggressive search finds solutions that score highly under the verifier but are actually incorrect, degrading performance on easy problems. The correct response is not to eliminate search but to deploy it selectively — using it only on problems where the verifier signal is reliable enough to provide genuine guidance.

PowerInfer-2 faces an analogous situation with sparsity on mobile NPUs. The NPU is the most powerful compute unit on the smartphone — it can deliver 770 tokens/s for dense prefill versus the GPU's 37 tokens/s and CPU's 8 tokens/s (Section 2.3.1). The natural instinct, following the PC playbook, is to route as much computation as possible to the NPU. But the NPU cannot handle sparse patterns efficiently — its sparse computation primitives run slower than the CPU's. Routing sparse (cold) neurons to the NPU would be "over-optimizing" for the NPU's raw throughput while ignoring its structural weakness, analogous to over-optimizing for PRM scores while ignoring verifier reliability. The correct response is selective deployment: NPU for dense (hot) neurons where its throughput advantage is real, CPU for sparse (cold) neurons where its flexibility advantage dominates.

What makes this a conceptual contribution. The paper does not just implement this selective scheduling; it provides the diagnostic framework for determining the boundary. The offline planner (Section 5) does not use a fixed threshold for hot vs. cold classification. It jointly considers activation frequency (how often is this neuron needed?) and hardware I/O budget (can we prefetch this neuron's weights within the attention computation window?). This joint optimization recognizes that "hotness" is not an intrinsic property of a neuron — it is a function of both the neuron's activation pattern and the device's ability to supply its weights in time. A neuron that is frequently activated might still be classified as cold if the device's I/O bandwidth cannot support prefetching it alongside other hot neurons, because misclassifying it as hot would cause NPU stalls when the prefetch doesn't complete in time.

Significance for future mobile systems. This insight generalizes: on heterogeneous mobile hardware, the optimal assignment of work to processors depends on the interaction between the work's characteristics (density, predictability) and the processor's strengths (throughput for regular patterns, flexibility for irregular patterns). The sweet spot is not "maximize NPU utilization" (which prior mobile ML frameworks like QNN target) nor "maximize sparsity exploitation" (which PC systems like PowerInfer target). It is balance them against each other given the device's specific hardware profile. The offline planner's joint optimization makes this balance explicit and automatable.

Evidence. Figure 13 shows the consequence of getting this balance wrong: QNN (NPU-only, optimized for throughput) achieves lower decoding speed than PowerInfer-2 at batch size 1 because it processes all neurons densely, wasting computation on the ~99% that are inactive. PowerInfer-2-CPUOnly (CPU-only, optimized for sparsity) achieves lower speed than PowerInfer-2 at batch size 4 because it cannot leverage the NPU's throughput when sparsity collapses. PowerInfer-2's dynamic balancing outperforms both extremes across the full batch size range, validating that the optimal assignment is context-dependent and that static "maximize X" strategies leave performance on the table.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on four representative real-world LLM tasks: multi-turn dialogue (Tunstall et al., 2023), code generation (Chen et al., 2021), math problem solving (Cobbe et al., 2021), and role play (Wang et al., 2023). These were chosen as "top representatives of real-world LLM tasks on the HuggingFace platform" (Section 7.1). Prompts are drawn from lmsys/lmsys-chat-1m (Zheng et al., 2023) for the energy consumption experiment. For accuracy evaluation, the paper uses OpenCompass (Contributors, 2023) with four benchmarks: Arc-Challenge, Arc-Easy, MMLU, and GSM8K. Prefill experiments use 128 and 512 token prompts; decoding experiments use up to 64 token prompts and 1,024 token outputs.
Base model(s). Five LLMs spanning different architectures, sizes, and activation functions: Mistral-7B (SiLU-based, 7B parameters), sparse Qwen2-7B (ReLU-based, 7B), Bamboo-7B (ReLU-based, 7B, explicitly trained for activation sparsity via TurboSparse), sparse Llama-13B (ReLU-based, 13B), and TurboSparse-Mixtral-47B (mixture-of-experts with ReLU-based sparsity, 47B total parameters but only ~3B activated per token). This range covers the dominant LLM architectures (dense SiLU, dense ReLU, sparse ReLU, sparse MoE) and spans roughly 7× in parameter count (7B to 47B), testing whether the system's benefits generalize across these axes. The models represent realistic deployment targets: 7B models are the current mobile frontier, while 47B represents a model previously impossible to serve on smartphones.
Metrics. Four primary metrics: (1) Decoding speed (tokens/s) — number of output tokens generated per second during autoregressive generation, the standard metric for interactive LLM performance; (2) Prefill speed (tokens/s) — number of prompt tokens processed per second during the prefill phase; (3) Memory usage — DRAM consumed for model weights, reported as absolute size (GB) and as reduction percentage versus baselines; (4) Accuracy — benchmark scores (Arc-Challenge, Arc-Easy, MMLU, GSM8K) under INT4 quantization, the de facto mobile deployment standard. Energy consumption uses Joules per token (J/token), averaged over 100 randomly sampled prompts from lmsys-chat-1m. All speed measurements are averaged over 10 runs to account for variance (Section 7.1).
Baselines. Four state-of-the-art frameworks: (1) llama.cpp (Gerganov, 2024) — the most popular CPU-based framework supporting flash offloading via mmap, serving as the CPU-only baseline; (2) QNN (Qualcomm, 2023) — Qualcomm's proprietary commercial inference engine with NPU acceleration but no offloading support; (3) MLC-LLM (MLC team, 2024) — a framework leveraging mobile GPU acceleration but requiring models to fit entirely in memory; (4) LLMFlash (Alizadeh et al., 2024) — designed for high-end PC contexts, reimplemented by the authors in llama.cpp based on the paper's description including sparsity prediction, row-column bundling, and neuron data caching. PowerInfer (Song et al., 2024) is also evaluated as an implicit baseline in Section 2.4's motivation experiments, extended by the authors to support flash offloading since the original lacked this capability.
Generation budget / compute accounting. Both generations and FLOPs are tracked. For speed comparisons, the budget is measured in wall-clock time with identical hardware and model configurations — all systems process the same inputs on the same device, and tokens/s is directly compared. For offloading experiments, the memory budget is explicitly controlled: FFN weight placement in DRAM is limited to 50% across all models except TurboSparse-Mixtral-47B on OnePlus Ace 2, which requires 75% offloading due to its 11GB available memory (Table 3). This ensures fair comparison: all systems operate under identical memory constraints and must dynamically load the same fraction of weights from flash. I/O overhead is measured as fraction of total latency (Table 4) by instrumenting the critical path.
Cross-validation / statistical protocol. All experiments are averaged over 10 runs "to account for variance" (Section 7.1). For the Best-of-N experiment (Figure 13), the batch size decreases by one every four iterations to simulate realistic sequence termination patterns. The offline planning calibration uses 10M+ tokens from Wikipedia and RefinedWeb — a substantial dataset that provides broad coverage of activation patterns. The paper does not describe a held-out validation split for the offline planner; the calibration data is used directly to determine neuron classification thresholds and cluster boundaries. For the accuracy evaluation (Table 7), standard benchmark protocols are followed (OpenCompass with the four specified benchmarks).

Main Quantitative Results

Offloading-Based Decoding Performance (Figures 7, 10, 11; Tables 4, 5)

The headline result: PowerInfer-2 achieves average decoding speedups of 3.84× (up to 4.63×) over LLMFlash and 24.6× (up to 27.8×) over llama.cpp on OnePlus 12, and 2.93× and 14.1× respectively on OnePlus Ace 2, when 50% of FFN weights are offloaded to flash storage across models ranging from 7B to 47B parameters (Figure 7).

On the flagship OnePlus 12 (Snapdragon 8 Gen 3, 24GB DRAM / 19GB available), the per-model decoding speeds with 50% FFN offloading show:

Bamboo-7B: PowerInfer-2 achieves 11.1 tokens/s versus LLMFlash at 2.4 tokens/s (4.63× speedup) and llama.cpp at 0.4 tokens/s (27.8× speedup).
Qwen2-7B: 10.0 tokens/s versus 2.4 tokens/s (4.17×) and 0.5 tokens/s (20.0×).
Llama-13B: 5.4 tokens/s versus 1.7 tokens/s (3.18×) and 0.2 tokens/s (27.0×). The lower absolute speed versus the 7B models is attributed to "nearly 2× more activated parameters than Bamboo-7B" due to lower activation sparsity (Section 7.2.1).
TurboSparse-Mixtral-47B: 9.5 tokens/s versus 2.8 tokens/s (3.39×) and 0.4 tokens/s (23.75×). Despite having 47B total parameters, its mixture-of-experts architecture and high sparsity result in only ~3B activated parameters per token, comparable to Bamboo-7B, explaining their similar performance. The paper notes this model "still has room for improvement through enlarged neuron cache" (Section 7.2.1).

On the mid-range OnePlus Ace 2 (Snapdragon 8+ Gen 1, 16GB DRAM / 11GB available), speeds are lower due to weaker compute and UFS 3.1 storage (vs. UFS 4.0 on the OnePlus 12), but the speedup ratios remain substantial: 4.9 versus 1.3 tokens/s (3.77× over LLMFlash) for Llama-13B, 7.0 versus 1.9 tokens/s (3.68×) for Bamboo-7B.

The critical path breakdown (Table 4) reveals why these speedups are achieved: PowerInfer-2 spends only 13.7% of time on I/O versus 76.7% for LLMFlash, with 86.3% of time spent on actual computation versus LLMFlash's 23.3%. This is the direct consequence of the neuron-cluster pipeline and flexible neuron loading mechanisms — the system has fundamentally inverted the compute-to-I/O ratio that bottlenecked prior approaches.

Memory scaling behavior (Figure 10). With varying available memory from 7GB to 19GB on OnePlus 12 running TurboSparse-Mixtral-47B:

At 7GB (extremely constrained, neuron cache holding only 1.8% of FFN weights): PowerInfer-2 achieves 2.13 tokens/s, still 1.84× faster than LLMFlash at the same configuration.
Performance scales linearly with memory: each additional GB increases the neuron cache hit rate, proportionally reducing flash I/O. At 19GB (maximum available), speed reaches 11.68 tokens/s — 3.12× over LLMFlash and 21.2× over llama.cpp. The paper notes this is the "first system to serve a 47B LLM on a smartphone" (Section 7.2.3).

Decoding consistency (Figure 11, Table 5). Across four downstream tasks (role-play, multi-turn dialogue, math problem solving, code generation), TurboSparse-Mixtral-47B maintains at least 11.4 tokens/s with a narrow range (11.4–11.8 tokens/s), demonstrating robust performance independent of task-specific activation pattern variations. The paper attributes minor variations to "task-dependent differences in model activation sparsity" (Section 7.2.4).

Token-level latency analysis (Table 5) reveals a meaningful tail latency issue. For TurboSparse-Mixtral-47B: mean latency is 99.76ms per token (roughly 10 tokens/s), P50 is 97.42ms, P90 is 116.16ms (16.5% above mean), and P99 reaches 140.56ms (40.9% above mean). The paper explains this variance through cache miss behavior: "while TurboSparse-Mixtral-47B maintains a low 3.5% average cache miss rate, the P99 miss rate reaches 18.9%." Tokens activating rarely-used neurons trigger flash reads, and P99 tokens hit these rare activation patterns disproportionately. For Bamboo-7B, the P99 latency is even higher relative to the mean (162.02ms vs. 90.32ms mean, a 79.4% increase). This tail latency is a practical concern for user experience (occasional perceptible pauses during generation) that the paper acknowledges but does not fully solve.

Offloading-Based Prefill Performance (Figures 8, 9)

PowerInfer-2 achieves 48.97× speedup over LLMFlash, 44.23× over llama.cpp, and 1.99× over QNN during prefill with 512-token prompts on OnePlus 12 (Figure 8, bottom left panel).

Per-model prefill speeds at 512-token prompts on OnePlus 12:

Bamboo-7B: PowerInfer-2 reaches 405 tokens/s versus QNN at 195 tokens/s (2.08×), LLMFlash at 7 tokens/s (57.9×), llama.cpp at 9 tokens/s (45.0×).
Qwen2-7B: 429 tokens/s versus QNN at 194 tokens/s (2.21×), LLMFlash at 7 tokens/s (61.3×), llama.cpp at 4 tokens/s (107.3×).
Llama-13B: 224 tokens/s versus QNN at 118 tokens/s (1.90×), LLMFlash at 4 tokens/s (56.0×), llama.cpp at 4 tokens/s (56.0×).
TurboSparse-Mixtral-47B: 79 tokens/s versus QNN at 42 tokens/s (1.88×), LLMFlash at 3 tokens/s (26.3×), llama.cpp at 4 tokens/s (19.8×). The lower absolute speed reflects the larger model size and correspondingly higher I/O volume.

At 128-token prompts, the pattern holds with proportionally smaller absolute speeds but similar speedup ratios: Bamboo-7B reaches 176 tokens/s (22.47× over LLMFlash), Qwen2-7B reaches 161 tokens/s.

The prefill advantage stems from two mechanisms whose effectiveness is demonstrated in Figure 9: (1) the NPU's dense matrix throughput at large batch sizes, which dwarfs CPU and GPU performance (Section 2.3.1, Figure 3a), and (2) complete overlap of sequential I/O with computation via prefetching. Figure 9 shows this overlap visually: for Bamboo-7B with 512-token prompts, sequential I/O time (820ms) is fully contained within computation time (916ms). For Qwen2-7B, I/O (778ms) fits within computation (962ms). "I/O operations are completely overlapped with computation time through our pipelining strategy, effectively hiding I/O latency" (Section 7.2.2).

The comparison with QNN is particularly informative. QNN also uses the NPU for prefill and achieves comparable performance for in-memory configurations (721 vs. 772 tokens/s, Figure 12 left), but when weights must be offloaded (50% FFN), QNN drops to 193 tokens/s versus PowerInfer-2's 404 tokens/s — a 2.09× gap. The reason: QNN cannot overlap weight loading with computation (it lacks PowerInfer-2's prefetching mechanism), so the I/O appears on the critical path. This validates that NPU acceleration alone is insufficient for offloaded inference; I/O-computation pipelining is equally critical.

SiLU-Based LLM Performance (Table 6)

While PowerInfer-2's largest gains come from ReLU-based models (which exhibit ~99% activation sparsity at batch size 1), the paper also evaluates a SiLU-based model to test generality. On Mistral-7B (SiLU activation) with 50% FFN offloading, PowerInfer-2 achieves 5.3 tokens/s versus LLMFlash's 2.18 tokens/s — a 2.4× speedup (Table 6).

This is substantially lower than the 4.63× speedup on the comparably-sized Bamboo-7B (ReLU-based), and the paper explains why: SiLU activation functions produce approximately 50% activation sparsity (citing CATS [Lee et al., 2024] and CHESS [He et al., 2024]), versus ReLU-based models' much higher sparsity. Lower sparsity means fewer cold neurons can be skipped by the CPU predictor, more neurons must be loaded from flash, and the neuron-cluster pipeline has less computation-skipping headroom to hide I/O latency. The 2.4× speedup validates that the system's mechanisms generalize beyond ReLU-specific sparsity, but also quantifies the performance penalty for denser activation patterns.

In-Memory Performance (Figure 12)

Under sufficient-memory conditions (all weights fit in DRAM), PowerInfer-2 is evaluated against QNN, llama.cpp, and MLC-LLM on Bamboo-7B.

Prefill: PowerInfer-2 achieves 721 tokens/s, comparable to QNN at 772 tokens/s (0.93×, a 7% gap), while llama.cpp reaches 8 tokens/s (90× gap) and MLC-LLM reaches 37 tokens/s (19.5× gap). The paper attributes the slight gap versus QNN to PowerInfer-2's mixed-precision quantization (handling outlier weights in INT8, remaining weights in INT4 per-channel), which is a deliberate accuracy-preserving trade-off versus QNN's pure INT4 per-channel quantization that causes accuracy degradation (Section 7.6). QNN does not support this mixed-precision approach, so its speed advantage comes at an accuracy cost.

Decoding: PowerInfer-2 achieves 20 tokens/s versus llama.cpp at 11 tokens/s (1.82×), MLC-LLM at 6 tokens/s (3.33×), and QNN at 8 tokens/s (2.50×). The speedups over CPU-based (llama.cpp) and GPU-based (MLC-LLM) baselines come from two sources identified in Section 7.3: (1) concurrent CPU-NPU computation improving memory bandwidth utilization from 43.9GB/s (CPU-only) to 59.6GB/s (CPU+NPU), and (2) leveraging activation sparsity to reduce FFN computations that baselines process densely.

Memory reduction with offloading: When 50% of FFN weights are offloaded, PowerInfer-2 reduces memory usage by 1.5GB (40%) while maintaining 10 tokens/s decoding speed, comparable to the in-memory baselines (llama.cpp at 11 tokens/s, MLC-LLM at 6 tokens/s). This demonstrates the memory-speed trade-off: PowerInfer-2 can match in-memory baseline speeds while using substantially less memory, freeing DRAM for other applications.

Best-of-N Sampling Performance (Figure 13)

PowerInfer-2 achieves 1.84× speedup over QNN and 1.28× over PowerInfer-2-CPUOnly during the initial, high-batch-size phase of Best-of-4 sampling on Bamboo-7B, with the advantage persisting as batch size decreases (Figure 13). All model parameters are resident in memory (no offloading).

The experiment simulates Best-of-4 sampling over 16 iterations, with batch size decreasing by one every 4 iterations as individual candidate sequences terminate. The speed curves reveal:

Iterations 1–4 (batch size = 4): PowerInfer-2 achieves approximately 45 tokens/s versus QNN at roughly 24 tokens/s (1.88×) and PowerInfer-2-CPUOnly at roughly 35 tokens/s (1.29×). At this larger batch size, sparsity is relatively low (many neurons activate across the 4 sequences), favoring the NPU-heavy configuration. QNN processes all neurons densely on the NPU, wasting compute on inactive neurons. PowerInfer-2-CPUOnly cannot leverage the NPU at all, leaving throughput on the table.
Iterations 5–8 (batch size = 3): All three configurations drop in speed as batch size decreases and sparsity increases. PowerInfer-2 maintains roughly 38 tokens/s versus QNN at 18 tokens/s and CPUOnly at 30 tokens/s. The dynamic adjustment mechanism has shifted more workload to the CPU, exploiting the higher sparsity.
Iterations 13–16 (batch size = 1): Maximum sparsity. PowerInfer-2 achieves roughly 28 tokens/s versus QNN at 16 tokens/s (1.75×) and CPUOnly at 23 tokens/s (1.22×). QNN's performance drops below CPUOnly because at batch size 1 with high sparsity, the NPU's dense computation processes a large matrix where most neurons are inactive — the CPU's sparse computation is more efficient. PowerInfer-2, by splitting work between NPU and CPU with dynamic ratio adjustment, outperforms both extremes.

The paper frames this as validation of the dynamic CPU-NPU adjustment: "When N reduces to 1, increased sparsity causes QNN's performance to drop below PowerInfer-2-CPUOnly. However, PowerInfer-2 still achieves 1.1× and 1.77× speedups over PowerInfer-2-CPUOnly and QNN respectively, benefiting from its hybrid CPU/NPU computation strategy" (Section 7.4).

Performance Breakdown (Ablation Path, Figure 14)

The paper quantifies each optimization's marginal contribution through an incremental ablation on Bamboo-7B with 50% FFN offloading, starting from a CPU-only baseline with no optimizations and adding components one at a time:

Baseline (CPU-only, no optimizations): 0.40 tokens/s. This represents llama.cpp-style processing with mmap for offloaded weights, no sparsity awareness, no caching, no pipelining.
+ Bundle (Gate-Up-Down bundling for I/O): 1.09 tokens/s (2.73× speedup). The bundling optimization alone more than doubles performance by replacing multiple small random reads with fewer larger reads, moving I/O to a more favorable point on UFS's bandwidth curve.
+ Neuron Cache: 4.18 tokens/s (additional 2.82× over previous, cumulative 10.45× over baseline). The cache achieves a 95% hit rate (as reported in Section 7.5), dramatically reducing the frequency of flash I/O. This is the single largest marginal improvement, confirming that flash I/O is the dominant bottleneck in offloaded mobile inference.
+ Neuron-Cluster-Level Pipeline: 9.60 tokens/s (additional 1.29× over previous, cumulative 24.0× over baseline). The pipeline overlaps the remaining I/O (from the 5% cache misses) with computation across matrix boundaries, hiding most of the residual flash latency.
+ XPU (NPU + CPU co-processing): 11.07 tokens/s (additional 1.15× over previous, cumulative 27.68× over baseline). Adding NPU participation for hot neuron clusters provides a modest but real gain on top of the already-efficient CPU-only pipeline, consistent with the observation that CPU-only memory bandwidth (43.9 GB/s) is substantially below combined CPU+NPU bandwidth (59.6 GB/s).

The cumulative 27.68× speedup matches the paper's claim of "up to 27.8×" over llama.cpp (Figure 7 shows 27.8× for Bamboo-7B on OnePlus 12), with the slight difference attributable to measurement variance across runs.

The breakdown reveals an important architecture insight: the biggest gains come from addressing I/O, not from adding compute. The sum of I/O-focused optimizations (Bundle + Cache + Pipeline) accounts for 24.0× of the 27.68× total speedup — roughly 87% of the improvement. Adding NPU computation provides the remaining 13%. This validates the paper's central thesis that UFS storage I/O, not raw compute throughput, is the binding constraint on mobile offloaded LLM inference, and that the neuron-cluster abstraction's primary value is in enabling fine-grained I/O hiding rather than (just) heterogeneous scheduling.

Accuracy Evaluation (Table 7)

PowerInfer-2 maintains comparable accuracy to llama.cpp under INT4 quantization while significantly outperforming QNN. On Qwen2-7B, the average across four benchmarks (Arc-Challenge, Arc-Easy, MMLU, GSM8K) is 78.38% for PowerInfer-2 versus 79.25% for llama.cpp (1.1% gap) and 56.93% for QNN (27.3% gap). On Bamboo-7B, the averages are 68.35%, 70.12%, and 63.26% respectively — a 1.8% gap versus llama.cpp and 7.7% advantage over QNN.

The paper explains the accuracy gap between frameworks through their quantization approaches (Section 7.6):

QNN uses per-channel quantization (one scale factor per weight matrix row), which "poorly handles weights with outlier values" — channels containing outlier weights get distorted because a single scale factor cannot represent both the outlier and normal weight ranges.
llama.cpp uses group-wise quantization (weights quantized in groups of 32 with per-group scales), which better handles outliers but is not supported by the NPU's hardware.
PowerInfer-2 adopts a hybrid quantization approach: outlier weights (identified offline during planning) are kept in INT8 precision, while remaining weights use INT4 per-channel quantization. This preserves accuracy comparable to group-wise quantization while maintaining NPU compatibility (the NPU can process per-channel INT4 with a separate INT8 path for outliers). The paper positions this as "a deliberate trade-off for accuracy preservation" (Section 7.3) that costs some prefill speed versus QNN's pure INT4 approach (721 vs. 772 tokens/s in Figure 12) but prevents the catastrophic accuracy degradation seen in QNN's GSM8K score (22.26% vs. PowerInfer-2's 75.24% on Qwen2-7B).

The accuracy results are critical validation because they demonstrate that PowerInfer-2's performance gains are not achieved through lossy compression that degrades model quality. The hybrid quantization preserves model capability, meaning the speedups translate to real user-facing improvements rather than benchmark artifacts.

Energy Consumption (Table 8)

PowerInfer-2 consumes 0.257 J/token, representing 31.1% and 61.8% reductions versus QNN (0.373 J/token) and llama.cpp (0.672 J/token) respectively, measured on Bamboo-7B across 100 randomly sampled prompts from real-world chatbot interactions (lmsys-chat-1m, Zheng et al., 2023).

Peak power draw is similar across frameworks: 5.095W (PowerInfer-2), 5.133W (QNN), 4.065W (llama.cpp). The energy-per-token advantage comes primarily from faster computation: PowerInfer-2 finishes processing each token in less time, so while instantaneous power is comparable to QNN, the energy integrated over time is lower. llama.cpp's lower peak power (due to CPU-only operation) is more than offset by its much slower speed, resulting in the highest energy per token despite the lowest instantaneous power.

This is a practically important metric for mobile deployment: even if a framework achieves high throughput, excessive energy consumption would drain the battery and create thermal throttling issues. PowerInfer-2's 0.257 J/token means that generating 1,000 tokens (a moderately long response) consumes approximately 257 Joules — roughly 1.5% of a typical 5,000 mAh smartphone battery, which is acceptable for interactive use.

Ablation Studies and Robustness Checks

SiLU vs. ReLU model performance (Table 6): The 2.4× speedup on SiLU-based Mistral-7B versus 4.63× on ReLU-based Bamboo-7B quantifies the sparsity-dependence of the gains. The paper attributes the difference to SiLU's approximately 50% activation sparsity versus ReLU's much higher baseline sparsity, citing CATS and CHESS. This is an important robustness check showing the system works on non-sparse models but delivers proportionally smaller benefits. The experiment implicitly tests whether the neuron-cluster pipeline and caching mechanisms are effective when the hot/cold distinction is less sharp — the 2.4× speedup confirms they remain beneficial even when a larger fraction of neurons must be processed.

Memory capacity sweep (Figure 10): Performance scales from 2.13 tokens/s at 7GB to 11.68 tokens/s at 19GB on TurboSparse-Mixtral-47B, demonstrating linear scaling with cache size. This validates that the neuron cache's hit rate is the dominant performance factor under memory constraints — as cache size increases and miss rate decreases, performance improves proportionally. The paper does not report the exact miss rates at each memory point, which would have strengthened this analysis.

Cache miss rate distribution (Table 5, Section 7.2.4): The P99 cache miss rate (18.9%) being 5.4× higher than the average (3.5%) reveals a meaningful tail latency problem. The paper identifies this as arising from "differing activation patterns between consecutive tokens" — when a token activates rarely-used neurons, cache misses spike. This is a genuine limitation: even with the neuron-cluster pipeline hiding most I/O latency, P99 tokens take 40.9% longer than average. The paper does not propose a solution beyond acknowledging the phenomenon.

Downstream task consistency (Figure 11): Speed varies only 3.4% across four tasks (11.4–11.8 tokens/s), demonstrating that performance is not sensitive to task-specific activation pattern differences for this model. This is a practical robustness check: real users will run diverse tasks, and a system that performs well on benchmarks but degrades on certain query types would have limited practical value.

Prefill prompt length scaling (Figure 8): Two prompt lengths (128 and 512 tokens) are tested, showing consistent speedup ratios across the range. The paper does not test longer prompts (e.g., 2048 tokens, the typical context window for these models) or very short prompts (e.g., single-sentence queries), which would reveal whether the sequential I/O prefetching saturates at some prompt length or becomes inefficient for short prompts where attention computation time may not fully hide I/O.

Device heterogeneity (OnePlus 12 vs. OnePlus Ace 2, Figure 7): Speedup ratios remain substantial on the lower-tier device (2.93× average over LLMFlash vs. 3.84× on the flagship), validating that the system's benefits are not specific to the highest-end hardware. However, absolute speeds differ significantly: Bamboo-7B drops from 11.1 to 7.0 tokens/s, reflecting the weaker SoC and slower UFS 3.1 storage. The offline planner's device-specific profiling and plan generation is implicitly validated — the same system works on both devices without code changes, with performance scaling roughly with hardware capability.

Multi-model architecture coverage (Figure 7): Five models spanning dense SiLU, dense ReLU, sparse ReLU, and sparse MoE architectures are evaluated. The speedup consistency across these diverse architectures (2.93×–4.63× over LLMFlash on OnePlus 12) suggests the neuron-cluster abstraction is not architecture-specific. However, all models use FFN-dominant architectures typical of modern decoder-only transformers — the paper does not test encoder-decoder models (T5), mixture-of-experts models with different routing mechanisms, or models where attention blocks dominate parameter count (which would reduce the relative importance of FFN optimizations).

Two-phase I/O loading for quantized models (Section 4.4): The paper describes but does not separately ablate the two-phase loading strategy (loading Gate weights first, computing, then conditionally loading Up/Down). Its contribution is bundled into the overall "Bundle" optimization in Figure 14. A separate ablation would have isolated the ~20% I/O savings from skipping unnecessary Up/Down loads, quantifying whether the conditional logic's overhead outweighs its benefit.

Single vs. multi-core I/O (Table 1, Section 2.3.2): The paper reports that multiple I/O cores degrade performance by up to 40% but does not present a direct ablation comparing single-core I/O to multi-core I/O within PowerInfer-2. The single-core design is motivated by the UFS characterization data, but a direct system-level validation would strengthen the claim.

Critical Assessment

Claim 1: PowerInfer-2 achieves up to 27.8× speedup over state-of-the-art frameworks. The experiments directly support this claim with qualifications. Figure 7 shows 27.8× over llama.cpp on Bamboo-7B (OnePlus 12, 50% FFN offloading) and Figure 14's ablation shows the speedup accumulates from identifiable components. However, the comparison baseline deserves scrutiny: llama.cpp with mmap is a weak baseline because it has no sparsity awareness, no predictor, no caching, and no XPU utilization — it represents a lower bound on what is achievable. The more meaningful comparison is against LLMFlash (3.84× average speedup), which implements comparable optimizations (sparsity prediction, bundling, caching) but at matrix granularity. The 3.84× versus LLMFlash isolates the benefit of the neuron-cluster abstraction specifically, and this is the stronger evidence for the paper's core claim.

The "up to" framing is important — the 27.8× figure is the maximum observed (Bamboo-7B, which has the highest activation sparsity and benefits most from the sparse computation mechanisms), not the average. On SiLU-based models, the speedup is 2.4×, and on the largest model (TurboSparse-Mixtral-47B), it is 3.39× over LLMFlash. The paper is transparent about this variation, but the headline number oversells the typical case.

Claim 2: PowerInfer-2 is the first system to serve a 47B model on a smartphone. Strongly supported. Figure 7 shows TurboSparse-Mixtral-47B running at 9.5 tokens/s on OnePlus 12 with 50% FFN offloading, and Figure 10 shows it reaching 11.68 tokens/s with full available memory (19GB). No baseline can run this model: llama.cpp and LLMFlash achieve 0.4 and 2.8 tokens/s respectively (sub-interactive speeds), QNN and MLC-LLM cannot handle models exceeding memory (marked with "✗" in Figure 12), and none of the baselines were designed for a model of this scale on mobile hardware. The caveat is that TurboSparse-Mixtral-47B, despite having 47B total parameters, activates only ~3B per token due to its mixture-of-experts architecture and high sparsity. It is a 47B model in name but behaves more like a 7B model in computational cost per token. Whether a dense 47B model (which would activate all parameters) could run at interactive speeds is not tested because such models are not practically deployable in this regime.

Claim 3: PowerInfer-2 reduces memory usage by 40% while matching in-memory baselines. Supported with a specific configuration. Figure 12 shows that on Bamboo-7B with 50% FFN offloading, PowerInfer-2 achieves 10 tokens/s decoding and 404 tokens/s prefill, comparable to llama.cpp's 11 tokens/s and MLC-LLM's 6 tokens/s in-memory. The 40% figure refers to the 1.5GB reduction in memory usage for this specific model (Bamboo-7B's FFN weights are roughly 3GB in FP16, and offloading 50% saves ~1.5GB). However, the comparison is slightly asymmetric: the baselines are running entirely in memory with no I/O overhead, while PowerInfer-2 is running with active flash I/O and still matching their speed. This actually makes the result stronger than the "matching" claim suggests — PowerInfer-2 is matching in-memory speeds while doing additional work (flash I/O) that the baselines avoid entirely, thanks to efficient pipelining that hides the I/O cost. The paper does not explicitly frame it this way, but the implication is that for memory-constrained scenarios, PowerInfer-2 delivers the speed of an in-memory system while using substantially less memory.

Claim 4: Performance gains come from jointly addressing NPU heterogeneity and UFS bottlenecks. The ablation study (Figure 14) provides the cleanest evidence, but it reveals an asymmetry the paper underemphasizes. I/O-focused optimizations (Bundle, Cache, Pipeline) account for 24.0× of the 27.68× total speedup. The XPU (NPU) optimization adds only 1.15× on top of that. This suggests that UFS storage is the dominant bottleneck and NPU heterogeneity is a secondary contributor — the system would achieve the vast majority of its gains even without NPU involvement, as long as the I/O pipeline and caching are present. The paper's framing presents these as roughly co-equal design principles ("Sparsity-Aware Adaptation" and "I/O-Aware Orchestration"), but the experimental evidence indicates I/O is the primary constraint. The NPU contribution matters more for prefill (where it enables 700+ tokens/s vs. CPU-only at <50 tokens/s) and for Best-of-N sampling with larger batch sizes (Figure 13), but for single-batch decoding — the most common interactive use case — the I/O mechanisms dominate.

Potential weaknesses in the experimental design:

No end-to-end latency measurement for interactive use. All results report tokens/s (a throughput metric), but interactive AI assistants care about time-to-first-token (TTFT) and per-token latency jitter. Table 5 provides token-level timing but only for the decoding phase — prefill latency (which can be several seconds for long prompts at 400 tokens/s) is not reported as an end-to-end metric. A 512-token prompt at 400 tokens/s takes 1.28 seconds before any output appears, which is noticeable to users. The paper does not analyze whether this is acceptable or how it compares to baselines in end-to-end terms.
Single hardware platform family. Both test devices use Qualcomm Snapdragon SoCs with the same XPU architecture (Hexagon NPU, Kryo CPU, Adreno GPU). The paper does not test on Apple Silicon (A-series/M-series with Apple Neural Engine), MediaTek Dimensity, or Samsung Exynos platforms, each of which has different NPU architectures with different sparse compute characteristics and different memory subsystem designs. The claim that the offline planner enables portability (Section 5) is not empirically validated across manufacturers.
No evaluation with concurrent applications. The paper motivates mobile deployment with privacy and offline operation but evaluates with the smartphone as a dedicated inference device. Real smartphones run background services, notifications, and user applications that compete for memory, CPU cores, and memory bandwidth. PowerInfer-2's memory bandwidth improvements (59.6 GB/s combined CPU+NPU) might be degraded if other processes are simultaneously accessing DRAM, and the system's sensitivity to this contention is not tested.
Energy measurement methodology is sparse. Table 8 reports a single number (0.257 J/token) with minimal detail: the measurement tool/method is not specified (battery fuel gauge? external power monitor? onboard sensors?), the measurement duration and thermal conditions are not described, and the variance across the 100 prompts is not reported. Energy consumption on mobile devices is highly sensitive to screen state, radio activity, and thermal throttling — the paper does not control for these confounds.
The 10M-token calibration dataset may not generalize. The offline planner uses Wikipedia and RefinedWeb for activation profiling, which skews toward formal, factual text. Real-world LLM usage includes casual conversation, code, creative writing, and domain-specific queries that may exhibit different activation patterns. The paper's downstream task evaluation (Figure 11) shows only 3.4% speed variation, suggesting robustness, but the four tested tasks may not cover the full distribution of real-world prompts.
Accuracy evaluation limited to four benchmarks on two models. Table 7 covers Qwen2-7B and Bamboo-7B on four standard benchmarks, but the paper does not evaluate accuracy on the 13B or 47B models, nor does it test the impact of offloading on accuracy (does dynamic weight loading from flash ever cause weight corruption or version mismatches?). The accuracy claim applies only to the quantization strategy, not to the full system under offloaded conditions.
No comparison against speculative decoding. Section 8 recognizes speculative decoding as an orthogonal optimization, but does not evaluate whether PowerInfer-2's mechanisms compose with it or whether a baseline with speculative decoding (e.g., llama.cpp with Medusa or EAGLE) would narrow the speedup gap. This is an acknowledged open question, but it means the speedup numbers are relative to baselines that lack this increasingly common optimization.

Experiments that would have strengthened the paper:

Ablation isolating two-phase I/O loading: The conditional Gate-then-Up/Down loading strategy for quantized models is described in Section 4.4 but its marginal contribution is bundled into the "Bundle" bar in Figure 14. A separate bar isolating this optimization would quantify whether the 20% expected I/O savings materialize in practice or are offset by the additional synchronization overhead.
Sensitivity to hot/cold classification threshold: The offline planner uses a joint optimization of activation frequency and I/O budget, but the paper never evaluates how sensitive performance is to this threshold. If the threshold were shifted by ±10%, how much would decoding speed change? This would reveal whether careful per-device profiling (as the planner does) is genuinely necessary or whether a simple heuristic suffices.
Cache hit rate curves for different memory budgets: Figure 10 shows speed scaling with memory, but does not report the underlying cache hit rates. A miss rate curve (misses vs. cache size) would characterize the workload's memory access pattern in a device-independent way and allow prediction of performance on devices with different memory capacities.
Direct measurement of compute unit utilization during offloaded decoding: Figure 14 implies that the baseline is I/O-bound (0.40 tokens/s) but does not report NPU/CPU utilization percentages under each configuration. Utilization numbers would directly validate the claim that prior systems leave compute units idle and that the pipeline fills those idle periods.
Thermal throttling behavior: Smartphones aggressively throttle performance under sustained load due to thermal constraints. A 10-minute continuous generation test would reveal whether PowerInfer-2's higher throughput triggers earlier or more severe throttling than the lower-throughput baselines, potentially narrowing the effective speedup in sustained-use scenarios.

6. Limitations and Trade-offs

Assumption: Difficulty Estimation Cost Is Not Accounted For

The online inference engine's compute-optimal strategy selection depends on knowing each prompt's difficulty before allocating the budget. The paper estimates difficulty by generating and scoring 2048 samples per prompt, which is more expensive than the largest inference budgets studied (256–512 generations). The authors explicitly flag this:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2)

Consequence. The reported 4× efficiency gains are computed after difficulty is known, without amortizing the estimation cost. In deployment, total cost would include both difficulty estimation and strategy execution. Since the estimation step (2048 samples) dwarfs the studied inference budgets (up to 512 generations), the realized efficiency gain in a production system would be substantially lower — potentially negative if difficulty estimation is required for every query. This is not a minor accounting oversight; it is a first-order omission that affects the central quantitative claim of the paper.

What evidence exists. The paper provides none — this is an acknowledged gap. The predicted (non-oracle) difficulty bins perform nearly as well as oracle bins (Figures 4, 8), demonstrating that the PRM's own scores can substitute for ground-truth labels, but the 2048-sample cost remains regardless of whether correctness is checked against labels or against PRM score distributions. The paper does not report the wall-clock time or FLOPs for difficulty estimation on any configuration.

Mitigation status. The paper explicitly calls this out as a limitation and suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" without the 2048-sample cost (Section 8). It also suggests adaptive estimation where a small number of initial samples inform difficulty assessment, with the remaining budget allocated dynamically. However, no such lightweight estimator is developed or evaluated, and the current results provide no evidence that difficulty can be predicted accurately from the prompt text alone with a small model. Until this gap is closed, the 4× figure is best understood as an upper bound on achievable efficiency in a deployment where difficulty is known cost-free, rather than a realized system improvement.

Hard Problems Remain Unsolved Regardless of Budget

Across every mechanism studied — PRM-guided search, iterative revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5, the bottom quintile) show near-zero improvement from any amount of test-time compute. The paper's results on this are unambiguous:

In Figure 3 (right, Section 5.3), bin 5 accuracy hovers at 1–3% for all search methods across all budgets from 4 to 256 generations. In Figure 7 (right, Section 6.2), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9, Section 7), the bin 5 scaling line is essentially flat near 0–5% across all values of R (the inference-to-pretraining token ratio). The paper states:

"On the hardest questions (bin 5), no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated." (Section 5.3)

Consequence. Test-time compute can amplify existing capability but cannot create it from nothing. If the base model's pass@1 is near zero on a problem class, no amount of search, revision, or adaptive allocation will help — there are no correct solutions in the proposal distribution to find or refine. This establishes a hard boundary on the method's applicability: it cannot extend a model's capability frontier beyond what is already accessible through random sampling at non-trivial rates. For problems where the base model lacks the necessary knowledge or reasoning depth, pretraining remains the only viable path.

This limitation is fundamental to the approach, not a shortcoming of the specific implementation. It applies equally to search (which selects among existing candidates) and revisions (which refine existing candidates — but cannot create correctness from an incorrect candidate that shares no structural similarity with any correct solution).

What evidence exists. The difficulty-bin analyses in Figures 3, 7, and 9 provide consistent, replicated evidence across multiple mechanisms. The FLOPs-matched comparison (Figure 9) is particularly revealing: on bin 5, even with R ≪ 1 (which gives the smaller model a large inference budget), test-time compute does not close the gap with the ~14× larger pretrained model. The paper acknowledges this explicitly in the Section 7 takeaway box: for hard problems, "pretraining is almost always more effective."

Mitigation status. None attempted. The paper identifies this as a limitation (Section 8 notes that test-time compute is bounded by the base model's capability) but does not propose solutions. This is reasonable — the limitation is inherent to the approach — but it means practitioners must carefully characterize their problem distribution's difficulty relative to their base model before deciding whether to invest in test-time compute infrastructure. If a substantial fraction of expected queries fall into bin 5 (base model pass@1 near zero), test-time compute will not help, and the only recourse is a better base model.

The 14× Larger Model Baseline Is Artificially Weak

The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters, but the larger model uses only greedy decoding with no test-time compute of its own. The comparison also fixes training data and scales only model parameters, following the LLaMA paradigm rather than compute-optimal pretraining (Hoffmann et al., 2022), where both data and parameters are scaled equally. The authors acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

Consequence. Both choices weaken the pretraining baseline, potentially overstating the advantage of test-time compute. A compute-optimally trained larger model (scaling both parameters and data to maximize performance per FLOP) would likely outperform a parameter-only-scaled model at the same total FLOPs budget. Additionally, giving the larger model even a modest test-time compute budget — best-of-8 sampling, for instance — would create a much stronger baseline that is never tested. The paper's headline finding that "a smaller model with additional test-time compute can outperform a ~14× larger model" (Section 7) should therefore be qualified: it outperforms a specifically configured larger model under specific assumptions about how pretraining compute is allocated.

The practical consequence is that practitioners comparing pretraining investment against inference-time infrastructure investment may draw overly optimistic conclusions about the substitutability of test-time compute. In a realistic scenario where the larger model also receives some inference-time compute (e.g., best-of-4 or best-of-8), the performance gap would narrow or potentially reverse.

What evidence exists. The paper does not provide ablations with stronger large-model baselines (e.g., larger model with best-of-N, or compute-optimally trained larger model). The only comparison point is greedy decoding from the 14× larger model. Figure 9 shows stars at three x-axis positions corresponding to three values of R — these stars represent the larger model's performance, and they receive no test-time compute augmentation. The paper does acknowledge the parameter-only-scaling caveat in the Section 7 text, but not the greedy-decoding-only caveat (the larger model is never given any of the test-time strategies that PowerInfer-2's own compute-optimal policy would prescribe for it).

Mitigation status. The paper frames the parameter-only scaling choice as deliberate and representative (matching the LLaMA training paradigm) and defers the compute-optimal pretraining comparison to future work. This is a reasonable scoping decision for a first paper on the topic, but the lack of any test-time compute for the larger model is harder to justify — the larger model would benefit from the same techniques PowerInfer-2 uses, and denying it those techniques makes the comparison asymmetric in a way that systematically favors test-time compute over pretraining. The paper does not propose future work to address this asymmetry.

Revisions and Search Are Studied Independently, Not Combined

The paper treats PRM-guided search (Section 5) and iterative revisions (Section 6) as separate, independently evaluated mechanisms, never combining them into a single system. This is a significant gap because the two mechanisms have complementary, difficulty-dependent strengths documented within the paper itself: search helps most on medium problems where the model needs to explore different solution strategies; revisions help most on easy problems where the model needs local refinement of roughly-correct initial attempts. The authors explicitly acknowledge this:

"we did not experiment with PRM tree-search techniques in combination with revisions" (Section 8)

Consequence. The paper reports performance ceilings for each mechanism individually, but a combined system — using the revision model as the proposal distribution within beam search, or using the PRM to guide which revision branches to pursue — could exceed either mechanism's individual performance. The current results therefore represent a lower bound on what the compute-optimal framework can achieve when both proposal-distribution modification and verifier-guided selection are deployed simultaneously.

This gap matters for practical deployment decisions. A practitioner reading the paper might conclude that search and revisions are alternatives and choose one based on their problem difficulty distribution. But the paper's own difficulty analysis shows they help in different regimes, suggesting the optimal system would deploy both and switch between them adaptively. The paper provides no guidance on how to combine them, what the combined compute-optimal policy would look like, or what performance gains the combination would enable.

What evidence exists. None — this is a pure gap. The paper presents individual results for search (Figure 4) and revisions (Figure 8) but no joint results. Section 8 frames this as future work.

Mitigation status. Acknowledged but not addressed. The paper suggests the combination as natural future work but provides no preliminary experiments or design sketches. This is a reasonable scoping choice given the complexity of each mechanism individually, but it means the paper's results are not directly comparable to a future system that combines both axes, and the reported performance numbers should be understood as achievable without full integration of the proposal-verifier framework laid out in Section 2.

Single Benchmark and Single Model Family

All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model, plus a related ~14× larger model from the same family for the FLOPs-matched comparison. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is not empirically validated across model families or benchmarks.

Consequence. Several aspects of the findings could be specific to this model-benchmark combination and may not transfer to other settings:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution — its calibration, its error patterns, and the separability of correct versus incorrect solutions in its representation space. A model with different calibration properties could exhibit different over-optimization thresholds, shifting the difficulty boundaries where beam search helps versus hurts.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families (some models are much better at in-context correction than others).
The MATH benchmark consists of competition-level math problems requiring symbolic reasoning and multi-step deduction. The difficulty-dependent patterns (beam search over-optimizing on easy problems, revisions helping on easy problems, no method helping on the hardest problems) may not generalize to other reasoning domains — code generation (where verifier signals from unit tests are cleaner), logical reasoning (where error patterns differ), or factual QA (where "difficulty" has different meaning).
The test set of 500 questions, split into five difficulty quintiles (~100 each) and further cross-validated within each bin (~50 per fold), means the compute-optimal policy is selected based on small samples. The paper does not report confidence intervals on the compute-optimal scaling curves, making it impossible to assess whether observed differences between strategies are statistically reliable at this sample size.

What evidence exists. The paper provides no multi-model or multi-benchmark replication. The entire experimental corpus uses one base model family and one benchmark. The authors' belief that PaLM 2-S* is "representative" is stated but not defended with evidence.

Mitigation status. The paper does not address this limitation or propose cross-model/cross-benchmark validation as future work. This is understandable given the computational expense of the experiments (2048 samples per question for difficulty estimation, multiple search configurations, revision model training), but it means the findings should be treated as existence proofs — demonstrations that compute-optimal test-time scaling can achieve 4× efficiency gains under specific conditions — rather than as universal scaling relationships that hold across models and tasks.

The Revision Model Exhibits a 38% Correct-to-Incorrect Reversion Rate

A direct consequence of the revision model's training data construction — sequences of 0–4 incorrect answers followed by a correct answer — is that the model has never seen a correct answer in its context during training. At test time, when the revision chain produces a correct answer (at some step k), the next revision step (k+1) conditions on that correct answer and may "revise" it into an incorrect one. The paper reports:

"approximately 38% of correct answers get converted back to incorrect ones" using a naive approach where the last revision is taken as final (Section 6.1)

Consequence. The revision model is not a reliable self-improvement mechanism — it is a local search operator that sometimes improves answers and sometimes degrades them. The system cannot simply run a long revision chain and take the final output; it must use a selection mechanism (majority voting or verifier-based selection) across the entire chain to pick the best answer from any point. This adds complexity and means the effective compute budget includes the cost of generating revisions that end up being discarded.

The 38% reversion rate also reveals a fundamental mismatch between training and inference distributions. The model was trained to produce correct answers when conditioned on incorrect ones, but at inference time it encounters both correct and incorrect answers in its context — and it has no training signal for what to do when the current answer is already correct. The system does not know when to stop revising.

What evidence exists. The paper states the 38% figure explicitly in Section 6.1 and uses it to motivate the chain-wide selection mechanism. The ReST $^{EM}$ experiment (Appendix K, Figure 16) provides additional evidence that revision training is fragile: attempting to optimize the revision model with RL-style training caused performance to "substantially hurt" with sequential revisions (fully sequential dropping to ~33.5% versus ~38.5% at the optimal ratio at 256 generations). The authors hypothesize that on-policy data collection amplified spurious correlations.

Mitigation status. Partially addressed through chain-wide selection (majority voting or verifier-based scoring across all steps of the revision chain), which prevents a single incorrect reversion from being the final output. However, this is a patch, not a solution — it means that some fraction of the compute budget is wasted on generating and then discarding incorrect reversions. A more principled fix (training the model to recognize when no revision is needed, or including sequences with correct in-context answers in the training data) is not explored. The paper does not suggest specific future work to address the reversion problem beyond the general call for better revision models.

7. Implications and Future Directions

How This Work Changes the Landscape

PowerInfer-2 establishes that neuron-cluster granularity is the correct abstraction level for mobile LLM inference under combined compute-storage constraints, and in doing so, it shifts the conversation around on-device AI from "what models can we fit?" to "how should we decompose the ones we want to run?" This is a systems-level reframing with diagnostic depth, not merely an optimization of existing techniques.

The decomposition granularity insight. Before PowerInfer-2, the field operated under a tacit assumption carried over from PC environments: LLM inference is optimized at matrix granularity. PowerInfer split matrices into hot and cold sections; LLMFlash bundled co-activated neurons within matrices; both remained bounded by matrix-level synchronization. PowerInfer-2 demonstrates that breaking the matrix barrier — allowing computation from one FFN component to proceed while I/O for another component is still in flight — changes the performance ceiling structurally, not incrementally. The evidence is in the critical path breakdown (Table 4): the compute-to-I/O ratio inverts from 23:77 (LLMFlash) to 86:14 (PowerInfer-2). This is not a tuning improvement; it is the difference between a system that is storage-bound and one that is compute-bound. The implication for future systems is clear: design your decomposition granularity so that I/O and computation can be interleaved across rather than within operators. Matrix-level designs cannot achieve this regardless of how aggressively they cache or prefetch.

The diagnostic contribution: UFS as a first-class architectural constraint. The paper's characterization of UFS 4.0 performance — four specific properties (block size sensitivity, data range sensitivity, CPU core dependency, single-queue concurrency limits) — provides a reusable diagnostic framework for any mobile system that stresses flash I/O. Previous mobile LLM work treated storage as a simpler slower-than-RAM pipe. PowerInfer-2 shows that UFS behavior is idiosyncratic enough to drive architectural decisions: the two-phase I/O loading for quantized models (Section 4.4) exists because empirical measurements showed 2×4KB outperforms 1×8KB random reads on UFS 4.0 — a fact that no abstract storage model would predict. The dedicated I/O thread pinned to a big core (Section 4.3) exists because Table 1 shows a 29% throughput gap between big and little cores for random reads — a CPU-storage coupling that is specific to UFS driver architecture. Future mobile systems papers can adopt this characterization methodology directly: measure block size × range × core dependency × concurrency, and let those measurements shape the architecture from the data layout upward.

Reconciling the sparse-compute contradiction. The paper resolves a tension that made prior PC-derived approaches fail on smartphones. On PCs, the GPU handles both dense and sparse compute efficiently relative to the CPU, so the optimal strategy is "route everything possible to the GPU, use CPU for overflow." PowerInfer followed this strategy and achieved strong results on consumer GPUs. On smartphones, the NPU is the throughput-dominant processor but has no sparse compute support — its sparse primitives run slower than CPU equivalents (Section 2.3.1, Figure 3a). The PC playbook of "maximize GPU utilization" becomes counterproductive when ported naively to NPU. PowerInfer-2 resolves this by making the NPU-CPU split a dynamic variable indexed by activation density, not a static hardware preference. The NPU gets dense clusters where its throughput advantage is real; the CPU gets sparse clusters where its flexibility advantage dominates. This reframes heterogeneous scheduling from a hardware-capability question (which processor is faster?) to a workload-characteristic question (how dense is this cluster right now, given the current batch size and activation pattern?). The implication is that mobile heterogeneous systems should schedule based on runtime workload properties, not static hardware rankings — and that offline profiling must capture the interaction between workload characteristics and hardware performance, not just hardware performance in isolation.

Dynamic sparsity as a new first-class concern. The paper identifies a phenomenon that no prior LLM inference system had meaningfully addressed: activation sparsity changes with batch size under advanced decoding strategies, and these changes occur within a single inference session as Best-of-N candidates terminate at different rates (Section 2.2, Figure 2). This is more than an optimization target — it is a new system dynamics problem. Prior sparsity-aware systems assumed static activation patterns profiled once and applied uniformly. PowerInfer-2 demonstrates both that this assumption is false under realistic decoding workloads (Best-of-N, MCTS, map-reduce) and that the performance penalty for ignoring it is substantial: Figure 13 shows that a static NPU-centric configuration loses 1.77× versus dynamic adaptation at batch size 1, while a static CPU-centric configuration loses 1.28× at batch size 4. As advanced decoding strategies proliferate (speculative decoding, tree search, multi-agent orchestration), the ability to dynamically rebalance processor assignments in response to shifting sparsity will become a requirement, not an optimization. PowerInfer-2 provides the first worked example of how to build this capability — offline planning across batch sizes, runtime batch size monitoring, asynchronous graph swapping to hide reconfiguration latency — and future systems will need to address the same dynamics.

What becomes more attractive as a research direction. I/O-computation codesign for mobile accelerators becomes central, not peripheral. The paper's ablation (Figure 14) shows that I/O-focused optimizations (Bundle, Cache, Pipeline) account for ~87% of the total speedup; adding NPU computation provides the remaining ~13%. This empirically demonstrates that flash I/O, not raw compute throughput, is the binding constraint on offloaded mobile LLM inference. Research investment should flow toward better storage-aware system design (smarter prefetching, flash-conscious data layouts, compressed weight formats that reduce I/O volume) rather than toward faster matrix multiplication kernels. The NPU is already fast enough; the storage pipe is what starves it.

What becomes less attractive. Brute-force approaches to mobile LLM deployment — "just use a bigger phone with more RAM" or "quantize more aggressively until the model fits" — are revealed as missing the point. PowerInfer-2 runs a 47B model on a device with 19GB available memory by accepting that weights will live on flash and building a system that makes flash access fast enough. The alternative path — waiting for smartphones with 48GB of RAM — is a hardware roadmap, not a systems contribution. Similarly, quantization-only approaches that sacrifice accuracy to fit models in memory (as QNN's per-channel INT4 does, producing a 48.6% GSM8K score on Qwen2-7B versus PowerInfer-2's 75.2% in Table 7) are less attractive than hybrid approaches that preserve accuracy and handle the memory constraint through intelligent offloading. The paper's hybrid quantization strategy (INT8 for outliers, INT4 per-channel for remaining weights) demonstrates that accuracy and memory efficiency can be jointly optimized rather than traded off.

Follow-Up Research This Work Enables

Density-adaptive speculative decoding with PowerInfer-2's dynamic XPU ratio mechanism. Speculative decoding uses a draft model to propose multiple candidate tokens, which a target model verifies in parallel — effectively creating a variable batch size that changes based on how many candidates are accepted. PowerInfer-2 already supports dynamic batch size adjustment with per-batch-size NPU computation graphs (Section 4.1.3). A direct integration would use the draft model's acceptance rate as a signal for the CPU-NPU split: high acceptance (large effective batch, low sparsity) shifts work toward the NPU; low acceptance (small effective batch, high sparsity) shifts work toward the CPU. The key experiment is measuring whether the combined system achieves super-additive speedups — i.e., whether the speedup from speculative decoding + PowerInfer-2 exceeds the product of their individual speedups, since PowerInfer-2's dynamic adjustment should be particularly well-suited to speculative decoding's variable-batch dynamics. A negative result (the overhead of graph swapping outweighs the benefit of adaptive splitting at the fine timescale of token verification) would establish an important boundary on how quickly dynamic reconfiguration can be amortized.

Cross-manufacturer NPU characterization and planner portability. The paper evaluates exclusively on Qualcomm Snapdragon SoCs (8 Gen 3, 8+ Gen 1), but the offline planner is described as device-adaptive (Section 5: "hardware-aware optimization... enables PowerInfer-2 to automatically adapt its execution strategy to different mobile platforms"). A direct follow-up would port PowerInfer-2 to at least two additional platforms: an Apple device (A17 Pro or M-series with Apple Neural Engine) and a MediaTek Dimensity device. The critical measurement is whether the planner's joint optimization of activation frequency and I/O budget produces qualitatively different hot/cold splits on different NPU architectures — e.g., does the Apple Neural Engine's different tile size or sparsity support change which neurons are classified as hot? A finding that the planner produces substantially different splits on different hardware would validate the device-specific planning approach. A finding that the splits are similar across hardware would suggest a simpler, device-independent heuristic may suffice, reducing the barrier to deployment.

Neuron-cluster pipeline with NVMe-class mobile storage. UFS 4.0's random read performance (100K IOPS, ~1 GB/s for 4KB reads within a 128MB range) is the primary constraint PowerInfer-2 is designed around. Emerging mobile storage technologies (UFS 5.0 roadmap targets, or potential NVMe adoption in high-end mobile devices) would change the I/O landscape substantially. A forward-looking experiment would profile PowerInfer-2 on a device with faster random-read storage — perhaps a tablet or handheld gaming device with NVMe — to determine whether the neuron-cluster pipeline's fine-grained I/O interleaving remains beneficial when storage latency drops. The hypothesis: as storage gets faster, the relative contribution of the pipeline versus the cache shifts (the cache matters less because misses are less painful), and the XPU contribution becomes relatively more important. Quantifying this tradeoff would produce a storage-speed-dependent design space where the optimal configuration (cache size, pipeline depth, XPU split ratio) is parameterized by storage IOPS, enabling system designers to target specific hardware tiers without redoing the full planner profiling.

SiLU-specific cold neuron prediction for non-ReLU architectures. PowerInfer-2 achieves 2.4× speedup on SiLU-based Mistral-7B versus 4.6× on ReLU-based Bamboo-7B (Table 6). The gap is attributed to SiLU's ~50% activation sparsity versus ReLU's much higher sparsity. A targeted follow-up would investigate whether SiLU-based models exhibit predictable sparsity patterns that a predictor could exploit, even if the absolute sparsity is lower. Recent work (CATS, CHESS, cited by the paper) suggests SiLU activations can be thresholded to produce sparsity, but the thresholds may need to be dynamic. The concrete experiment: train a predictor specifically on SiLU activation patterns (possibly using a lower activation threshold than the standard zero-cutoff for ReLU), integrate it into PowerInfer-2's cold neuron pipeline, and measure whether the speedup gap closes. A finding that SiLU predictability is fundamentally lower than ReLU predictability would establish that activation function choice is a first-order deployment consideration for mobile LLMs — a result that would influence model architecture decisions at training time, not just inference-time optimization.

End-to-end latency and thermal characterization under sustained interactive workloads. The paper reports tokens/s (throughput) but not time-to-first-token or per-token latency distributions under realistic interactive usage patterns (Section 5, critical assessment). A measurement-focused follow-up would instrument PowerInfer-2 during multi-turn conversations with realistic think time between user inputs, measuring: (1) prefill latency for prompts of varying length (64–2048 tokens), (2) P50/P95/P99 per-token latency during decoding (extending Table 5 to cover the prefill phase and full sessions), (3) thermal throttling onset time and performance degradation curve under 10-minute continuous generation, and (4) memory bandwidth contention when background applications (messaging, web browsing) actively use DRAM. The key finding would be whether PowerInfer-2's throughput advantage translates to perceptibly better user experience — does the 11.68 tokens/s on a 47B model feel responsive, or do tail latencies (P99 at 140ms, 40.9% above mean per Table 5) and prefill delays (1.3 seconds for a 512-token prompt at 400 tokens/s prefill) dominate the user-perceived latency budget? A negative result — that interactive users cannot distinguish PowerInfer-2's throughput from a slower but more latency-consistent baseline — would redirect optimization effort toward tail-latency reduction rather than average-throughput improvement.

Practical Applications and Downstream Use Cases

High-quality offline AI assistants on flagship smartphones. The clearest immediate application is deploying capable LLMs (7B–47B parameters) for offline use cases where cloud connectivity is unavailable, undesirable, or expensive. PowerInfer-2 achieves 11.68 tokens/s on TurboSparse-Mixtral-47B with 19GB available memory (Figure 10) and at least 11.4 tokens/s across diverse tasks including dialogue, code generation, math, and role-play (Figure 11). At these speeds, a 47B model generates responses at roughly reading-comfortable rates (typical reading speed is 200–300 words per minute, or roughly 5–8 tokens/s at ~0.75 words per token) with minimal quality compromise (Table 7 shows 68.4% average accuracy on Bamboo-7B, competitive with llama.cpp's 70.1%). Use cases include: international travel where data roaming is expensive or blocked, secure facilities where network-connected devices are prohibited, field research in remote areas, and privacy-sensitive applications (medical consultation, legal document review) where users will not accept cloud processing of their data. The 40% memory reduction for in-memory models (Section 7.3: 1.5GB saved on Bamboo-7B while matching in-memory baselines) means the same device can run a capable LLM alongside other applications without aggressive memory pressure.

Cost-efficient batch inference for mobile-edge applications. Beyond interactive use, PowerInfer-2's offloading capability enables batch processing workloads that were previously infeasible on mobile devices. Examples include: on-device email summarization (processing dozens of emails overnight), local document indexing and question-answering over a personal file store, and privacy-preserving data extraction (processing sensitive documents without uploading them). The 404 tokens/s prefill throughput on Bamboo-7B with 50% FFN offloading (Figure 8, 512-token prompts) means that processing 100 emails averaging 500 tokens each would take approximately 124 seconds — feasible as a background task. The energy efficiency (0.257 J/token, Table 8) means this batch would consume roughly 12,850 Joules total, or about 7% of a typical 5,000 mAh smartphone battery — acceptable for an overnight operation while charging. Current alternatives (cloud-based batch processing that uploads private data, or PC-based processing that requires manual file transfer) are less convenient or less private.

Enabling on-device self-improvement and personalization loops. A more speculative but potentially transformative application is using PowerInfer-2's ability to run larger models for on-device fine-tuning and personalization. Current mobile deployments are limited to SLMs (~3B parameters) whose outputs may not be high-quality enough to serve as self-improvement training data. PowerInfer-2's 47B-capable inference means a smartphone can (1) run a large, capable model to generate high-quality responses to user queries, (2) use those responses as training data for a smaller, personalized model that fits entirely in memory for low-latency interaction, and (3) repeat periodically as user preferences evolve. The 11.68 tokens/s generation speed on the 47B model makes this pipeline practical: generating 1,000 high-quality training examples (each ~200 output tokens) would take roughly 4.7 hours of background processing, feasible as a weekly overnight task. The key enabler is PowerInfer-2's demonstration that large models are not categorically impossible on smartphones — they are merely slower, and for batch personalization tasks, throughput matters more than latency.