Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIs

🎯 Pitch

OpenAI’s davinci appears fast only because of API-level optimizations—once you strip away hardware and contention differences, its idealized runtime is worse than smaller open models. This paper shows how to cheaply estimate an apples-to-apples inference cost for any black-box LLM API using just a handful of measurements, revealing a completely different efficiency-capability frontier.

1. Executive Summary

This paper proposes a new inference efficiency metric called the idealized runtime that enables apples-to-apples comparison of autoregressive Transformer models served through black-box APIs by measuring query runtime on uniform hardware and software without performance contention, and develops an efficient estimation procedure based on a closed-form parameterization showing that end-to-end runtime is the sum of a piecewise linear function of prompt size and a linear function of the number of output tokens. Using this methodology on ten state-of-the-art LLMs ranging from 6 to 530 billion parameters across four NLP benchmarks (sentiment analysis, question answering, classification), the authors demonstrate that models appearing on the Pareto frontier under raw runtime metrics—notably OpenAI/davinci—fall off the frontier when compared using idealized runtime, establishing that superior raw latency often reflects API-level optimizations rather than inherent model efficiency, and revealing fundamentally different efficiency-capability tradeoff landscapes only when comparison is conducted on equal footing.

2. Context and Motivation

The Core Problem: You Can't Compare LLM Inference Efficiency Fairly

The fundamental problem this paper tackles is deceptively simple: given two large language models accessible only through black-box APIs, how do you determine which one is more inference-efficient? This question matters enormously for practical decision-making — model creators need to understand whether architectural choices (deeper, wider, or sparse models) will yield runtime-competitive models once deployed; end users and companies building LLM-powered products need to select models that meet latency service-level objectives (SLOs) and fit within operational budgets; and researchers studying scaling laws need to understand how capability gains trade off against the computational cost of serving those capabilities.

The difficulty arises because the statistics you can actually observe — what the paper calls raw runtimes measured by timing API calls — are contaminated by two categories of confounds that have nothing to do with a model's inherent efficiency:

Hardware and software heterogeneity across providers. One provider might serve their model on cutting-edge NVIDIA H100 GPUs with custom CUDA kernels and aggressive kernel fusion, while another serves theirs on older V100 GPUs with a standard PyTorch implementation. The first model might return results faster, but that speed advantage is a property of the serving infrastructure, not the model architecture itself. If both models were ported to the same serving stack, the ranking could reverse entirely.
Performance contention from shared infrastructure. Black-box APIs are typically served from multi-tenant clusters where multiple users' queries compete for compute, memory bandwidth, and network resources. This introduces a noise term η ≥ 0 (captured as overhead in the paper's modified Equation 4) that can vary substantially depending on time of day, query load, and the provider's load-balancing policies. The paper quantifies this effect directly: Figure 6 shows that as the number of parallel queries to the Anthropic/v4-s3 API increases, runtime can degrade by up to 2×. Figure 5 shows per-trial variability where outlier queries experience up to 3× higher latency than the median, and the AI21/J1-Grande v1 model exhibits a coefficient of variation of roughly 0.55 compared to ~0.2 for other models from the same provider. This means that two queries to the same model can produce wildly different runtime measurements, making any single timing unreliable as a model-level efficiency signal.

Why This Problem Has Real-World Consequences

This isn't a purely academic concern — the inability to compare models on equal footing has direct financial, environmental, and research implications the paper raises throughout Section 1 and Section 5.4.

For model creators and researchers, inference costs can dominate the total cost of ownership for deployed LLMs. As the paper notes in Section 1, "model inference costs might outweigh training costs for certain applications (e.g., ChatGPT)." When an architecture team decisions whether to scale model depth versus width, or whether to adopt mixture-of-experts routing, they need to understand the inherent runtime implications of each choice — not the runtime as measured on whatever hardware happens to be available in their lab versus what a competitor uses. Without a standardized efficiency metric, researchers cannot reliably answer questions like "Is a 52B dense model more efficient than a 100B sparse model with the same FLOPs?" or "Did Hyperparameter Configuration A genuinely produce a faster model than Configuration B, or is the difference just measurement noise?"

For practitioners deploying LLM-powered applications, latency SLOs are hard requirements. If a chatbot must respond within 200 milliseconds and model X's raw API latency is 150ms while model Y's is 180ms, a naive comparison suggests model X is safe and model Y is risky. But if model X's 150ms measurement was taken during a low-load period and it degrades to 300ms under peak load (a 2× degradation consistent with the Anthropic observations in Figure 6), while model Y's runtime is stable regardless of load, the risk assessment is completely wrong. The paper's denoised runtime metric is partially designed to address this by factoring out load-induced variance from the efficiency signal.

For environmental and cost accounting, estimating the energy or dollar cost of serving a model requires knowing its true computational requirements. If you measure API latency during a period of light load, you'll underestimate the energy consumption (since the model is running on underutilized hardware that draws power regardless) and potentially underestimate the cloud compute bills. The paper's idealized energy cost and idealized dollar cost metrics (Section 4.2) are explicit attempts to enable such accounting by grounding cost estimates in a known hardware configuration rather than observable but noisy API behavior.

For the scaling laws research community, the default practice has been to use model parameter count as a proxy for computational cost (Kaplan et al., 2020; Hoffmann et al., 2022; Wei et al., 2022). The paper argues forcefully that this proxy is inadequate: "two models with the same size can have vastly different inference runtimes" (Section 1), citing work on Switch Transformers (Fedus et al., 2021) and convolutional architectures (Jeon & Kim, 2018) where architectural choices produce large runtime gaps at fixed parameter count. This means that scaling law analyses that use parameter count as the x-axis are implicitly assuming that all models with N parameters cost the same to serve — an assumption the paper's data contradicts (Figure 9 shows that models with similar parameter counts can have dramatically different idealized runtimes).

Where Existing Approaches Fall Short

The paper identifies a fragmented landscape of existing efficiency metrics, each with critical limitations for the cross-provider comparison use case.

Raw API runtime is what practitioners actually observe and what matters for end-user experience, but it is inherently non-comparable across providers for the reasons detailed above. The paper doesn't dismiss raw runtime as useless — it acknowledges that "Raw latency is still a good metric for end users who are directly impacted by slow (or fast) predictions" (Section 1) — but it argues that raw runtime cannot serve the comparative function needed by model creators and researchers. The paper's quantitative evidence in Figure 7a supports this: 96.6% of raw runtime measurements are greater than the corresponding denoised runtime (the noise-free lower bound), confirming that raw measurements are systematically inflated by variable overhead.

Model parameter count is the most commonly used proxy for scale and efficiency in the LLM literature. For example, the BIG-Bench paper (Srivastava et al., 2022) and the Gopher paper (Rae et al., 2021) plot capabilities against model size (in parameters), implicitly treating parameter count as a stand-in for computational cost. PaLM (Chowdhery et al., 2022) reports parameter counts as a primary axis of comparison. The paper identifies two problems with this approach:

Interpretability: "It is hard to answer questions like 'Can model X meet a latency SLO of 100 milliseconds?' or 'How much will it cost to use model X in this concrete application?'" using parameter count alone (Section 6). A model with 175B parameters could run in 80 milliseconds or 800 milliseconds depending on hidden size, number of layers, attention head configuration, and the degree of model parallelism required — all factors that parameter count collapses into a single scalar.
Cross-architecture incomparability: Different architectural choices (dense vs. mixture-of-experts, deep vs. wide, standard attention vs. efficient attention variants) can produce identical parameter counts with vastly different runtime characteristics. The paper cites Scao et al. (2022) as evidence that "two models with the same size can have vastly different inference runtimes."

Number of floating-point operations (FLOPs) improves on parameter count by accounting for the actual computation performed, but the paper identifies two remaining limitations (Section 6):

Runtime does not correlate exactly with FLOPs because different operators with identical FLOP counts can execute at different efficiencies. This is a well-known phenomenon in high-performance computing formalized by the roofline model (Williams et al., 2009), which the paper cites: memory-bound operators (like element-wise additions or layer normalization) run far slower than their FLOP count would suggest relative to compute-bound operators (like large matrix multiplications). Two models with identical total forward-pass FLOPs but different ratios of attention to FFN computation — or different hidden-size-to-attention-head ratios — will have different actual runtimes on real hardware.
Interpretability again: FLOPs is an abstract quantity that is "hard to translate to something actionable" (Section 6) for practitioners making latency- or cost-constrained decisions.

The paper's Figure 9 demonstrates the practical consequence of relying on FLOPs. In the bottom row comparing idealized cost to accuracy, BigScience/BLOOM achieves cheaper ideal cost than Yandex/YaLM despite using more FLOPs (visible in the FLOPs row where YaLM is to the left of BLOOM on the x-axis). This reversal happens because BLOOM's architecture has been optimized for efficient execution on GPU hardware (Scao et al., 2022), while YaLM's architecture performs operations that, despite being fewer in total FLOPs, run less efficiently. A FLOPs-only analysis would misleadingly conclude YaLM is more efficient.

Carbon and energy accounting tools exist for training (Cao et al., 2020; Henderson et al., 2020; Strubell et al., 2019; Patterson et al., 2021) but the paper notes that "information on inference-time costs of LLMs is more scant" (Section 6). This gap exists partly because model providers treat their inference infrastructure as proprietary (since serving efficiency is a competitive advantage) and partly because the measurement methodology doesn't transfer automatically: training involves a known, fixed amount of computation performed on known hardware, while inference involves variable-length queries and output generation on unknown hardware with unknown parallelism strategies. The paper's idealized energy cost metric (Section 4.2) is explicitly positioned as extending these training-focused carbon accounting approaches to the inference domain.

HELM-style benchmarking (Liang et al., 2022), which the paper explicitly builds on and cites as infrastructure for its efficiency-capability analyses in Section 5.4, provides standardized capability evaluation across models but does not include standardized efficiency metrics. The paper's metrics are designed to complement such benchmarks by adding the efficiency dimension currently missing.

How This Paper Positions Itself

The paper positions its contribution as measurement infrastructure, not a new efficiency optimization. It does not propose faster attention mechanisms, better model parallelism strategies, or more efficient hardware architectures. Instead, it proposes a measurement methodology that enables comparisons that were previously impossible or misleading when using existing metrics.

The core insight is that you can estimate how a model would perform on standardized hardware and software without actually having access to the model's trained parameters or the provider's infrastructure — you only need knowledge of the model's architecture (hidden size h, number of layers l, number of attention heads n). This is possible because for Transformer models, the amount of computation — and therefore the runtime on a given hardware/software stack — depends only on the model architecture and the prompt/output dimensions, not on the model's specific learned parameter values. The paper exploits this by instantiating "skeleton" versions of each model (architecture only, no trained weights) on a reference system (NVIDIA A100 GPUs with NVIDIA's Megatron inference implementation) and profiling them to extract the parameters of the closed-form runtime expression.

The parametric insight in Section 3 is what makes this efficient rather than prohibitively expensive. If you had to profile every query in a benchmark (e.g., thousands of prompt-output pairs) by running them through the skeleton model on the reference hardware, the measurement process would cost as much as actually serving the benchmark — defeating the purpose. Instead, the paper derives and empirically validates that for autoregressive Transformer inference, the end-to-end runtime follows:

$t(\text{prompt size } p, \text{ number of output tokens } o) = \text{prompt\_encoding\_time}(p) + (o - 1) \cdot g$

where prompt_encoding_time is a piecewise linear function of p and g is a constant cost-per-output-token (Equation 3). This means you only need to profile a small grid of (p, o) configuration points (the paper suggests ranges like P = {1, 256, 512, 1024, 1536} for models with 2048-token context windows) and then use linear regression to predict the runtime of any arbitrary query in the benchmark — reducing profiling from "hours per benchmark" to "<2 hours once" (Section 5.3).

The paper explicitly positions itself relative to scaling laws research by arguing that scaling laws are necessary but insufficient. The paper acknowledges that scaling laws (Kaplan et al., 2020) show how model size affects training/validation loss, and that emergent abilities (Wei et al., 2022) demonstrate that some capabilities appear discontinuously with scale. But the paper argues that empirical efficiency-capability evaluation — running actual models on actual tasks with standardized efficiency metrics — provides information that scaling laws cannot: which concrete models sit on the Pareto frontier for specific use cases, how different providers' optimization efforts affect the efficiency landscape, and whether architecture-level efficiency improvements (like BLOOM's careful operator optimization) translate into better capability-per-dollar tradeoffs.

The paper also positions itself in the lineage of infrastructure-focused ML systems work. The parametric runtime analysis in Section 3 draws directly on the FLOP-counting methodology from Narayanan et al. (2021)'s Megatron-LM training systems paper, adapting it from the training regime (where sequence length s is fixed and known in advance) to the autoregressive inference regime (where prompt size p varies per query and output generation is iterative). The distinction between prompt_encoding_time (where computation scales with p and benefits from high arithmetic intensity at large p) and output_generation_time (where each new token requires a fundamentally different, memory-bound computation pattern) is the key analytical contribution that makes the efficient estimation procedure tractable.

Finally, the paper positions its contribution as enabling better science in the LLM space. The efficiency-capability analysis in Section 5.4 and Figure 9 is not presented as a definitive ranking of models but as a demonstration that different efficiency metrics lead to different conclusions about which models are Pareto-optimal. The finding that OpenAI/davinci appears on the Pareto frontier under raw runtime but consistently falls off under idealized runtime is the paper's most concrete example of how measurement methodology can reverse qualitative conclusions. The paper's goal is to provide tools that prevent the research community from drawing incorrect architectural or scaling conclusions from contaminated efficiency data.

3. Technical Approach

3.1 Reader Orientation

This paper develops a measurement methodology that estimates how fast a large language model would run if it were served on standardized hardware and software, without needing access to the model's trained parameters or the provider's infrastructure. The system solves the problem that raw API runtimes are contaminated by provider-specific optimizations and performance noise, making cross-model efficiency comparisons misleading, by exploiting the fact that Transformer inference runtime depends only on model architecture (not trained weights) and follows a simple parametric form — a sum of a piecewise linear function of prompt size and a linear function of output token count — that can be estimated from profiling a small grid of synthetic queries on skeleton models instantiated with known architecture hyperparameters.

3.2 Big-Picture Architecture

The system has four major components:

Skeleton model instantiation (Section 2, Table 1): For each model to be evaluated, the system constructs a "skeleton" — a Transformer architecture with the same hidden size h, number of layers l, and number of attention heads n as the target model, but with random or zero-valued parameters. This skeleton is loaded onto a reference hardware/software stack (the paper uses NVIDIA A100 GPUs with Megatron's inference implementation as the default uniform platform).
Configuration query profiler (Section 3.4): Rather than running every query in a benchmark through the skeleton model, the system profiles a small grid of synthetic queries with carefully chosen prompt sizes p and output token counts o. For each (p, o) pair, it measures the end-to-end wall-clock runtime on the reference system. This produces the raw data from which the closed-form runtime expression's parameters are fit.
Linear regression estimator (Section 3.4): The profiling data is fed into a two-step regression procedure. First, for each prompt size p, runtimes are regressed against o to extract the prompt-encoding time (the y-intercept) and the per-output-token cost g (the slope). Second, the prompt-encoding times across different p values are interpolated to handle arbitrary prompt sizes not in the profiling grid. The result is a complete parameterization of t(p, o) for that model on that reference system.
Idealized metric computation (Section 4): Given a benchmark of real queries (each with a specific prompt size and number of output tokens), the system plugs each query's (p, o) into the parameterized runtime function to compute t_idealized(p, o). This can be extended with hardware cost factors to produce idealized dollar cost and idealized energy cost (Section 4.2). For black-box API comparisons, a parallel denoised runtime metric is estimated by applying the same regression procedure to min-trial runtimes measured through the actual API, factoring out performance variance.

Information flows as follows: model architecture details → skeleton model on reference hardware → profiling of synthetic (p, o) grid → regression fits produce (prompt_encoding_time(p), g) parameters → benchmark queries → per-query idealized runtimes → efficiency-capability tradeoff analysis (e.g., Pareto frontier plots in Figure 9).

3.3 Roadmap for the Deep Dive

First, the closed-form runtime parameterization (Section 3.1): Why does inference runtime decompose into a prompt-encoding term plus a per-output-token term, and what makes prompt encoding piecewise linear? This is the theoretical foundation that enables efficient estimation.
Second, the empirical validation of the parametric form (Section 3.2): How do we know the decomposition holds in practice? Walk through Figures 3 and 4 showing linearity of runtime with o and piecewise linearity with p.
Third, the profiling and regression estimation procedure (Section 3.4): Given that the form holds, how do you actually estimate its parameters from a small number of profiling runs? The two-step regression algorithm and interpolation strategy.
Fourth, the two derived metrics — idealized and denoised runtime (Section 4.1): How these metrics are defined from the same parametric form but differ in what profiling data they use and what they represent.
Fifth, the scale-aware cost metrics (Section 4.2): How idealized dollar cost and idealized energy cost extend the runtime metric by incorporating the number and type of accelerators needed to serve the model.
Sixth, the black-box API variance characterization (Section 3.5): Why denoised runtime is necessary — quantitative evidence of performance contention and run-to-run variation that motivates factoring noise out of cross-model comparisons.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a measurement methodology paper whose core idea is that inference runtime for autoregressive Transformer models follows a simple parametric form that can be estimated from a small number of profiling runs on skeleton models, enabling standardized efficiency comparisons across models that are otherwise only accessible through non-comparable black-box APIs.

Deriving the Closed-Form Parametric Expression

The paper's key analytical contribution is the derivation of a closed-form expression for autoregressive inference runtime in terms of two query-level quantities (prompt size p and number of output tokens o) and a set of model-and-system-specific parameters. The derivation proceeds in two stages: first counting floating-point operations for the two distinct computational phases of autoregressive generation, and then converting FLOP counts to runtime using the system's achievable throughput.

The two-phase structure of autoregressive inference. Unlike standard ML inference, where a single forward pass produces the output, autoregressive text generation proceeds in two qualitatively different computational stages:

Prompt encoding phase: The model processes all p tokens of the input prompt simultaneously through all l Transformer layers. Because all prompt tokens are available at once, the self-attention operations in each layer can be computed in parallel across the prompt positions. This phase produces the hidden state representation of the prompt and the logits for the first generated token (which is sampled from the conditional distribution Pr(x_{p+1} | x_{1:p})).
Token-at-a-time generation phase: For each subsequent output token (token index i > p), the model must run a full forward pass through all l layers sequentially. Critically, the computation for generating token i is different from training-time computation because only the new token's query representation needs to be computed; the key and value representations for all preceding tokens (including the prompt tokens and all previously generated tokens) are cached and reused. This means the generation of each new output token requires a fundamentally different, memory-bound computation pattern that depends on the number of tokens generated so far i.

The paper formalizes this as a simple additive decomposition in Equation 3 (presented in Section 3.3, but derived in Section 3.1):

$t(\text{prompt size } p, \text{ number of output tokens } o) = \text{prompt\_encoding\_time}(p) + \text{output\_generation\_time}(o)$

where prompt_encoding_time(p) is the wall-clock time to process a prompt of size p through all layers, and output_generation_time(o) is the total wall-clock time to generate o output tokens (including the first token generated during prompt encoding, so o - 1 additional forward passes are needed).

FLOP counting for prompt encoding. Drawing on the FLOP-counting methodology from Narayanan et al. (2021) for Transformer training, the paper computes the number of floating-point operations required for one forward pass through l Transformer layers with hidden size h on a prompt of p tokens (with micro-batch size b = 1 since we are serving one query at a time):

$\text{FLOPs}_{\text{pe}} = 24 \cdot b \cdot p \cdot h^2 \cdot l \cdot \left(1 + \frac{p}{6h}\right)$

where b = 1 is the batch size (one prompt), p is the prompt size in tokens, h is the hidden dimension, and l is the number of Transformer layers.

Why this form matters — and when it simplifies: The term (1 + p/(6h)) captures the contribution of the attention score computation, which scales quadratically with sequence length. For most LLMs operating within typical context windows, p ≪ 6h. For OpenAI/davinci, the paper notes h = 12288 and the maximum context length is 2048 tokens, so p/(6h) ≤ 2048/(6 · 12288) ≈ 0.0278, meaning the quadratic attention term contributes less than 3% of total FLOPs. Under this condition, the expression simplifies to:

$\text{FLOPs}_{\text{pe}} ≈ 24 \cdot p \cdot h^2 \cdot l = θ_{pe} \cdot p$

where θ_{pe} = 24 · h^2 · l is a constant for a given model architecture. This is a linear function of prompt size p — doubling the prompt length approximately doubles the prompt-encoding computation. This linearity is the first key property that enables the efficient estimation procedure.

FLOP counting for output token generation. The computation for generating the (i + 1)-th token (where i tokens have been generated so far, including the prompt) is fundamentally different because only one new token's query representation needs to be computed, but it must attend to all i previous tokens. The paper derives (in Appendix A.2):

$\text{FLOPs}_{\text{og}} = 24 \cdot b \cdot h^2 \cdot l + 4 \cdot b \cdot i \cdot h \cdot l = 24 \cdot b \cdot h^2 \cdot l \cdot \left(1 + \frac{i}{6h}\right)$

where the first term captures the matrix multiplications for the new token's key/query/value projections and the FFN layers, and the second term captures the attention score computation between the new token and all i previous tokens.

Why this simplifies too: The same condition i ≪ 6h holds during generation—even for a model generating its 2048th token, i/(6h) ≈ 0.028 for OpenAI/davinci. Under this approximation, the FLOPs per generated token becomes:

$\text{FLOPs}_{\text{og}} ≈ 24 \cdot h^2 \cdot l = θ_{og}$

which is independent of i — every output token costs roughly the same number of FLOPs to produce, regardless of position. This is the second key property: it means the total output generation FLOPs is simply θ_{og} · o, a linear function of the number of output tokens.

Converting FLOPs to runtime. The actual runtime is the ratio of FLOPs to the achievable throughput (FLOPs per second) of the hardware/software system:

$\text{prompt\_encoding\_time}(p) = \frac{\theta_{pe} \cdot p}{\text{throughput}_{pe}(p)} \quad \quad \text{(Equation 1)}$

$\text{output\_generation\_time}(o) = \sum_{\text{token } i} \frac{\theta_{og}}{\text{throughput}_{og}(i)} \quad \quad \text{(Equation 2)}$

The critical nuance — why prompt encoding is piecewise linear, not purely linear: If throughput_{pe} were constant across all prompt sizes, prompt_encoding_time(p) would be purely linear in p. But it is not constant for small p. The reason lies in arithmetic intensity — the ratio of FLOPs performed to bytes transferred from memory. For small prompts, the computation has low arithmetic intensity because the h × h weight matrices must be read from GPU memory but are reused across only a small number of token positions. As p increases, the same weight reads are amortized over more tokens, increasing arithmetic intensity and pushing the computation from the memory-bound regime toward the compute-bound regime. This means throughput_{pe}(p) increases with p for small p and eventually saturates at the hardware's peak compute throughput for large p. Consequently, prompt_encoding_time(p) as a function of p has a steeper slope at small p (where throughput is low) and a shallower slope at large p (where throughput has plateaued) — hence piecewise linear, not purely linear.

For output generation, throughput is constant: The computation for each new output token is heavily memory-bound (each forward pass requires reading all l layers' weight matrices from GPU memory but only processes b = 1 token), meaning throughput_{og}(i) does not improve with i. The paper states: "Usually, throughput_og is a constant independent of the token being generated meaning output_generation_time is a linear function of o" (Section 3.1.2). This is the third key property and the one that makes the estimation procedure especially simple — the per-token generation cost g is a single scalar for a given model-system pair.

The final parametric form (Equation 3):

$t(\text{prompt size } p, \text{ number of output tokens } o) = \text{prompt\_encoding\_time}(p) + (o - 1) \cdot g$

where prompt_encoding_time(p) is a piecewise linear function of the prompt size p (in tokens), o is the number of output tokens (including the first token generated from the prompt), o - 1 is the number of additional forward passes needed, and g is the constant runtime cost of generating one additional output token (in seconds per token) for a given model on a given hardware/software stack.

What it computes: Given a query with a specific prompt length and a target number of output tokens, this expression predicts the total wall-clock time to serve that query on a known hardware/software configuration. The first term accounts for the one-time cost of processing the prompt through all Transformer layers (which produces the first output token's logits), and the second term accounts for the iterative cost of generating each subsequent token through repeated forward passes with key/value caching.

Why this form: The decomposition into prompt-encoding and output-generation components follows directly from the two-phase structure of autoregressive inference. The piecewise linearity of prompt_encoding_time follows from the arithmetic intensity curve of GPU computation — small prompts are memory-bound (low throughput), large prompts are compute-bound (high, saturated throughput). The linearity of output generation with o follows from the constant-per-token FLOP count (since i ≪ 6h in the attention term) combined with constant throughput (since each token generation is memory-bound and gets no benefit from amortization). This specific parametric structure is what enables efficient estimation: rather than profiling every (p, o) combination in a benchmark, you only need to profile enough points to fit the piecewise linear prompt-encoding curve and a single slope for output generation.

Empirical Validation of the Parametric Form

The paper validates the derived form empirically by profiling skeleton models — model architectures instantiated on the reference system without trained parameters — since "runtime is independent of the model's parameters given a prompt size and number of output tokens" (Section 3.2). The computation performed, the data movement patterns, and therefore the runtime are determined entirely by the architecture hyperparameters (h, l, n) and the query dimensions (p, o).

Validation of output-generation linearity (Figure 3). Figure 3 shows end-to-end runtime versus number of generated output tokens for four models (OpenAI/davinci, AI21/J1-Jumbo v1, AI21/J1-Grande v1, Anthropic/v4-s3) at different prompt sizes (shown as separate lines in the legend). For each prompt size, the data points fall tightly along a straight line, and the paper reports: "the coefficients of determination (R^2) for the resulting time estimates are very close to 1.0 (> 0.999) for all models." This confirms two things simultaneously:

The per-output-token generation time g is indeed constant (independent of token position i), since a constant-per-token cost would produce the observed exact linearity with o.
The throughput_og is constant across tokens, validating the assumption that the memory-bound nature of token generation doesn't change as more KV-cache entries accumulate.

Visually, the different prompt sizes produce lines with different y-intercepts (the prompt encoding time) but similar slopes (the per-token generation time g), confirming the decomposition.

Validation of prompt-encoding shape (Figure 4). Figure 4 plots prompt encoding time versus prompt size for the same four models, with dotted best-fit lines. The curves show the predicted piecewise linear behavior: at small p (roughly p < 256), the curve is steeper because throughput is low (the GPU is memory-bound and underutilized). At larger p (roughly p > 256), the curve becomes shallower and more linear as throughput approaches the hardware's peak. The paper notes: "We see that runtime and the prompt size have a roughly linear relationship, especially at large prompt sizes. However, this linear relationship breaks down at smaller prompt sizes." This matches the arithmetic intensity explanation.

Implication for estimation: The empirical validation means you can estimate the full runtime function t(p, o) for any (p, o) by profiling a sparse grid: pick several prompt sizes spanning the range of interest (e.g., P = {1, 256, 512, 1024, 1536} for a model with 2048-token context), run each with several output token counts to fit the g slope and extract the y-intercept as prompt_encoding_time(p), then interpolate between fitted prompt-encoding times for prompt sizes not in P.

The Profiling and Regression Estimation Procedure

The paper's efficient estimation procedure (Section 3.4) transforms the theoretical parametric form into a practical two-step regression algorithm. The key efficiency claim is that this requires profiling "on the order of hours (< 2 hours for most models) once, compared to hours per benchmark (depending on number of queries in the benchmark) for exhaustive profiling" (Section 5.3).

Step 1: Extract prompt-encoding times and per-token cost g. For each prompt size p in the chosen profiling set P:

Run the skeleton model on the reference hardware/software stack with queries that have prompt size p and varying numbers of output tokens o (the paper does not specify the exact o values profiled, but the methodology works as long as you have at least 2–3 different o values per p to fit a line).
Measure the end-to-end wall-clock runtime for each (p, o) pair.
Fit a linear regression model of the form runtime = intercept + slope × (o - 1). The fitted intercept is an estimate of prompt_encoding_time(p) — this works because when o = 1, the expression evaluates to just the prompt-encoding time. The fitted slope is an estimate of g for that model-system pair.

Step 2: Interpolation for arbitrary prompt sizes. Real benchmark queries will have prompt sizes that are not necessarily in P. Since prompt_encoding_time(p) is piecewise linear in p, you can interpolate between the fitted prompt-encoding times for adjacent profiled prompt sizes. The paper uses a simple procedural approach: "if max_context_length = 2048, then one possible range of prompt sizes to explore is P = {1, 256, 512, 1024, 1536}" and for any prompt size p between these values, "we can interpolate between known data points, since prompt_encoding_time is piecewise linear." The interpolation is linear between adjacent points in P.

Alternative single-regression approach. The paper describes a slightly different formulation that can also be used: "Equipped with these prompt encoding runtimes, we can leverage the fact that total_runtime(p, o) - prompt_encoding_time(p) is a linear function in o: we can fit a single linear regression model with y = runtime difference and x = o to obtain an estimate for the slope g." In this variant, you pool the runtime-difference data across all prompt sizes and fit a single g, rather than fitting a separate slope per prompt size. The choice depends on how much you trust that g is truly constant across prompt sizes (the empirical results in Figure 3, where slopes are similar across prompt sizes, support this assumption).

Output of the procedure. For a given model on a given hardware/software system, the procedure produces:

A piecewise linear function prompt_encoding_time(p) defined by its values at the profiled prompt sizes, with linear interpolation between them.
A scalar g (seconds per output token).

Together, these allow computing t(p, o) for any query with known p and o without running that specific query through the skeleton model.

Why this order: The two-step structure mirrors the decomposition in the runtime expression. Step 1 separates prompt encoding (intercept) from output generation (slope) for each profiled prompt size, exploiting the empirical linearity of runtime with o. Step 2 connects the prompt-encoding estimates across prompt sizes via piecewise linear interpolation, exploiting the arithmetic intensity curve. If you tried to fit a single global model, you would lose this structure and get worse estimates at small prompt sizes where the piecewise linearity matters most.

The Two Derived Runtime Metrics: Idealized and Denoised

The paper uses the same parametric form to define two distinct runtime metrics (Section 4.1), differing in what profiling data they use and what they aim to represent.

Idealized runtime. This is the metric for apples-to-apples comparison across models:

$t_{(m,s,h)}^{\text{idealized}}(\text{prompt size } p, \text{ number of output tokens } o) = \text{prompt\_encoding\_time}_{(m,s,h)}^{\text{idealized}}(p) + (o - 1) \cdot g_{(m,s,h)}^{\text{idealized}}$

where the subscript (m, s, h) indicates that the runtime is estimated for model m on software stack s (e.g., Megatron) and hardware h (e.g., NVIDIA A100 GPUs) that are explicitly chosen to be uniform across all compared models.

What it computes: The wall-clock time that model m would take to serve query (p, o) if it were implemented on the specified uniform software and hardware platform. By fixing (s, h) across all models, any remaining runtime differences are attributable to the model architecture m itself — its hidden size, depth, attention head configuration, and the parallelism strategies required to fit its parameters in GPU memory.

How it is estimated: Profile the skeleton model on the uniform reference system (A100 GPUs with Megatron) using the procedure in Section 3.4, and extract the model-specific (prompt_encoding_time_{idealized}, g_{idealized}) parameters.

Denoised runtime. This is a companion metric designed to test whether the idealized runtime is plausible and to provide a noise-free lower bound on what the API could achieve on its own hardware:

$t_{m \text{ on API } a}^{\text{denoised}}(\text{prompt size } p, \text{ number of output tokens } o) = \text{prompt\_encoding\_time}_{m \text{ on API } a}^{\text{denoised}}(p) + (o - 1) \cdot g_{m \text{ on API } a}^{\text{denoised}}$

What it computes: The runtime for model m on API a with performance contention factored out — essentially, the best-case runtime that the API provider's infrastructure can deliver for that model query, absent queuing delays, interference from other queries, and transient system load.

How it is estimated: The same regression procedure is applied, but the profiling data comes from running synthetic queries through the actual black-box API (not the skeleton model on reference hardware). To factor out performance noise, which the paper models as "a random variable η ≥ 0" (Section 4.1), the regression is performed using the minimum runtime across multiple trials for each (p, o) pair. The intuition is that the minimum runtime across trials represents the query with minimum variable overhead — closest to the true computational cost of serving the query on the provider's infrastructure. The paper confirms this empirically: Figure 5 shows that different models exhibit different per-trial variance, and the minimum across trials provides a stable lower bound.

The inequality relationship. The paper establishes that, by construction, the following inequality should hold for any model m on API a, provided that the idealized software s* and hardware h* are at least as fast as what the API provider uses:

$t_{m \text{ on API } a}^{\text{raw}}(p, o) \geq t_{m \text{ on API } a}^{\text{denoised}}(p, o) \geq t_{(m, s^*, h^*)}^{\text{idealized}}(p, o)$

What this inequality means operationally: The raw runtime (what you actually measure) is always at least as large as the denoised runtime (because η ≥ 0 adds overhead). The denoised runtime is always at least as large as the idealized runtime, assuming (s*, h*) is a floor on efficiency (i.e., if the API provider uses slower hardware, the idealized metric on faster hardware will predict a lower runtime than even the provider's noise-free runtime). If the inequality is violated — i.e., denoised runtime is lower than idealized runtime — it indicates that the API provider is using software or hardware optimizations beyond what the reference system provides.

Empirical validation of the inequality (Figure 7): Figure 7a shows that 96.6% of raw runtime points are above the denoised runtime (below the y = x line), confirming that η is indeed non-negative and that denoised runtime serves as a valid lower bound. Figure 7b tests the second inequality: for AI21 Labs models, idealized runtimes are greater than denoised runtimes 15.7% of the time, and for OpenAI models, idealized runtimes are greater than denoised runtimes 64.2% of the time. These violations indicate that both providers have implemented optimizations beyond what Megatron on A100 GPUs provides. For all other model providers (Cohere, Anthropic, Together, Microsoft), idealized runtimes are always lower than denoised runtimes, indicating that the A100+Megatron reference stack is at least as fast as their serving infrastructure.

Table 3 concretizes this. For OpenAI/davinci, the idealized per-output-token cost g_idealized = 0.081 seconds on Megatron+A100, while the denoised per-token cost g_denoised = 0.030 seconds through OpenAI's API — a 2.7× speedup attributable to API-level optimizations. Similarly, for AI21/J1-Grande v1, g_idealized = 0.038 vs. g_denoised = 0.021 (1.8× API speedup). For Anthropic/v4-s3, the values are much closer: g_idealized = 0.054 vs. g_denoised = 0.057 (idealized is actually slightly faster), consistent with Figure 7b where Anthropic's idealized runtimes are below denoised runtimes, indicating their API is not using optimizations beyond the reference stack.

Scale-Aware Cost Metrics: Dollar Cost and Energy Cost

Runtime alone does not capture the full cost of serving a model if different models require different numbers of accelerators. The paper extends the idealized runtime into two cost metrics that account for scale (Section 4.2).

Why runtime is insufficient for cost comparisons: A model requiring 8 GPUs with a 100-millisecond inference time incurs 8× the hardware cost and energy consumption of a model requiring 1 GPU with the same 100-millisecond runtime. Simply comparing runtimes would misleadingly suggest they are equally efficient.

Idealized dollar cost:

$t_{(m, s, h)}^{\text{idealized}}(\text{secs}) \times n_{\text{accelerator } h} \times c_{\text{accelerator } h} (\$ / \text{sec})$

where n_accelerator_h is the number of accelerators of type h needed to serve a single inference request for model m (this is 1 if the model fits on a single GPU without model parallelism, and >1 if tensor or pipeline model parallelism is required), and c_accelerator_h is the per-unit-time cost of renting one accelerator of type h (e.g., the AWS on-demand hourly price for an p4d.24xlarge instance with 8 A100 GPUs, divided by 8 and 3600 to get dollars per GPU-second).

What it computes: The dollar cost of serving a single query (p, o) on the reference hardware, accounting for the number of GPUs that must be allocated simultaneously to fit the model in memory and execute the computation. This is directly comparable to the per-query pricing charged by black-box API providers.

How n_accelerator_h is determined: Table 1 shows the exact GPU configurations used. For AI21/J1-Large v1 (6.7B parameters, h = 4096), only 1 × 80GB-A100 is needed — the model fits in a single GPU's memory. For OpenAI/davinci (175B parameters, h = 12288), 8 × 80GB-A100 with tensor model parallelism within a single server is required. For Microsoft+NV/TNLG v2 (530B parameters), 24 × 80GB-A100 with both tensor and pipeline parallelism is needed.

The paper's note on parallelism choices: "We also use the minimum number of GPUs necessary" and "Tensor model parallelism is optimal within a multi-GPU server (Narayanan et al., 2021) since expensive all-to-all communication is limited to fast high-bandwidth NVLink." These choices are deliberate: the idealized metric should represent the best-case cost of serving the model, not a suboptimal deployment.

Idealized energy cost:

$t_{(m, s, h)}^{\text{idealized}}(\text{secs}) \times n_{\text{accelerator } h} \times p_{\text{accelerator } h} (\text{W})$

where p_accelerator_h is the power draw of accelerator h in watts (e.g., the thermal design power of an A100 GPU).

What it computes: The energy consumed (in watt-seconds or joules) to serve a single query on the reference hardware, accounting for all accelerators used simultaneously. This extends the training energy accounting frameworks (Cao et al., 2020; Henderson et al., 2020; Strubell et al., 2019; Patterson et al., 2021) to the inference domain. The paper notes this enables comparisons like: "the number of inference queries needed to amortize the significant overhead of training models."

Why denoised cost variants are not defined: The paper explicitly states that for denoised runtime, "we cannot similarly modify [it] since we do not know the type of hardware and the number of chips used by the model provider." The denoised metric is limited to runtime because extending it to cost would require information (GPU count, GPU type, power draw) that black-box API providers do not disclose. This is a fundamental asymmetry: idealized metrics can be fully cost-aware because the reference hardware is chosen and known, while denoised metrics can only be runtime-based because the provider's infrastructure is opaque.

The paper reports a striking cost comparison: "We also compare these estimated inference costs to the costs charged by the black-box API provider. We observe that they are up to an order of magnitude lower than the charged actual costs." This gap represents the provider's margin, training cost amortization, infrastructure overhead beyond bare GPU costs, and potentially the cost of parallelism and redundancy for handling variable load. The paper does not break down this gap further but uses it to illustrate that idealized cost is a lower bound — the actual price paid by users reflects additional business costs that the metric intentionally does not capture.

Black-Box API Variance Characterization

Before proposing denoised runtime as a solution, the paper quantifies the performance variance that makes raw API runtimes unreliable as an efficiency metric. This empirical characterization (Section 3.5) provides the motivation for the denoising procedure.

The modified parametric form for black-box APIs (Equation 4):

$t(\text{prompt size } p, \text{ number of output tokens } o) = \text{prompt\_encoding\_time}(p) + (o - 1) \cdot g + \text{overhead}$

where overhead captures two distinct sources of extra time: "fixed costs of using an API to serve model predictions instead of using accelerators locally (e.g., round-trip latency of communicating with a remote API server) and performance variability (e.g., queuing delay or performance interference across requests)."

What this equation computes: The actual, observed runtime when calling a black-box API. It decomposes into the same deterministic computation (prompt_encoding_time(p) + (o - 1) · g) as the idealized runtime, plus a variable overhead term that is specific to the API infrastructure and the current system state (load, contention).

Why overhead is not a clean constant: Unlike g, which is a deterministic property of the model and hardware, overhead is a random variable that can vary across queries, across time, and across models from the same provider. This variability is what makes raw runtimes non-comparable.

Per-trial variance evidence (Figure 5). The paper runs multiple trials of synthetic queries with controlled (p, o) through various black-box APIs and plots per-trial runtimes. Figure 5 shows results for the three AI21 models. The key observations:

Model-specific variability: "AI21/J1-Grande v1 has much higher performance variance than AI21/J1-Large v1 or AI21/J1-Jumbo v1 (larger spread among points for a query of given size)." The coefficient of variation for AI21/J1-Grande v1 averages approximately 0.55, compared to roughly 0.2 for the other AI21 models. This means that Grande's runtime can vary by ±55% from its mean, while Large and Jumbo vary by only ±20%.
Outlier queries experience dramatically worse performance: "Even for models with lower spreads (e.g., AI21/J1-Large v1), we see that outlier points can have as much as 3× higher latency." A single slow query could violate an SLO even if the median performance is acceptable, making raw runtime unreliable for latency-critical decision-making.
The pattern is not consistent across providers or models from the same provider: The checkmarks and crosses in Figure 5 annotate trials with details about prompt format; the paper notes "we see discernible performance variance across multiple trials for different models, across prompt sizes and number of generated output tokens," emphasizing that the variance is not a simple function of query dimensions that could be factored out analytically.

Load-dependent variance evidence (Figure 6). To isolate the effect of contention specifically, the paper varies the number of parallel queries sent simultaneously to the Anthropic/v4-s3 API (with fixed prompt size p = 512 and varying o) and measures the minimum runtime across 10 trials for each parallel query count. The result: "We observe as much as a 2× increase in runtime, indicating that load can lead to increased contention on API servers and consequently increased observed runtime." As the number of parallel queries increases, the minimum runtime (which already factors out within-trial noise by taking the minimum across 10 repeats) still degrades, meaning that the shared infrastructure's capacity is being saturated and even the best-case query experiences queuing or bandwidth contention.

Why this motivates the denoising procedure: The overhead term in Equation 4 is composed of two sub-components: a fixed API overhead (round-trip latency, tokenization, request parsing) and a variable noise term η ≥ 0 that captures load-dependent and random variance. By taking the minimum runtime across multiple trials for each (p, o) profiling point before fitting the regression, the estimation procedure for denoised runtime effectively removes the η term while retaining the fixed overhead. The resulting t_denoised represents the best-case runtime the API can deliver — "the runtime with minimum variable overhead" — which is then fit to the same parametric form to extract (prompt_encoding_time_denoised, g_denoised).

Accuracy of the regression despite noise (Table 2): Even though raw runtimes are noisy, the linear regression using the parametric form is accurate. Table 2 reports R^2 values for denoised runtime estimates: 0.985 for OpenAI/davinci, 0.990 for AI21/J1-Large, 0.917 for AI21/J1-Grande (lower due to its higher variance), 0.995 for AI21/J1-Jumbo, 0.997 for Cohere/XL, and 0.924 for Anthropic/v4-s3. These high R^2 values confirm that the underlying parametric form holds even when estimated from noisy API measurements — the linear structure emerges reliably despite the η noise, especially when the minimum across trials is used.

Design Choices and Their Justifications

Why skeleton models without trained weights? Because the computation performed by a Transformer forward pass depends on the tensor dimensions (batch, sequence, hidden) and the learned weight matrix shapes, not on the specific floating-point values stored in those matrices. A matrix multiplication (m, n) × (n, k) performs 2mnk FLOPs regardless of what numbers are in the matrices. This means profiling a skeleton architecture produces the same runtime as profiling the fully trained model, provided the skeleton uses the same data types and tensor layouts. This is the key insight that makes idealized runtime estimation possible without access to proprietary model weights.

Why Megatron and A100 GPUs as the reference stack? The paper states: "We use A100 GPUs because they are the fastest widely available GPU right now. Other accelerators like the TPU or the NVIDIA H100 GPU could also be used." The choice is pragmatic — A100s are available in cloud instances with known pricing, making the dollar cost metric actionable, and Megatron is a high-performance, open-source inference implementation that represents a strong but not unreachable baseline. The paper emphasizes that the methodology works with any (s, h) pair, and Section 5.3 demonstrates this by comparing A100 to V100 GPUs as an alternative hardware choice.

Why tensor model parallelism within a single server? For models too large to fit on a single GPU, the paper uses tensor model parallelism (sharding each layer's weight matrices across GPUs and performing all-reduce on activations between layers) because it "is optimal within a multi-GPU server (Narayanan et al., 2021) since expensive all-to-all communication is limited to fast high-bandwidth NVLink." Pipeline parallelism (sharding different layers across different GPUs) is used only when tensor parallelism is insufficient (e.g., for Microsoft+NV/TNLG v2 with 530B parameters), because pipeline parallelism incurs pipeline bubbles that reduce throughput. The "minimum number of GPUs necessary" principle ensures the idealized cost reflects an efficient deployment, not an artificially inflated one.

Why profiling on the order of hours rather than per-benchmark hours? This is the central efficiency claim. Without the parametric form, estimating idealized runtime for a benchmark of thousands of queries would require running each query's specific (p, o) through the skeleton model — taking hours per benchmark. With the parametric form, you profile a small grid of (p, o) once (taking <2 hours for most models) and then compute any query's runtime via the fitted function in constant time. The profiling cost is amortized over all benchmarks and analyses, making the methodology practical for systematic efficiency-capability studies.

Why take the minimum across trials for denoised runtime? The paper models performance noise as η ≥ 0, meaning observed runtime = true computational runtime + non-negative noise. Under this model, the minimum across multiple trials is the best estimate of the true noise-free runtime — it is the trial where η was closest to zero. Using the mean or median would produce a biased estimate that includes a non-zero expected η. This is a standard technique in systems benchmarking where transient interference (network contention, OS scheduling, thermal throttling) adds non-negative latency.

4. Key Insights and Innovations

Innovation 1: Inference Efficiency as a Measurement Infrastructure Problem, Not an Optimization Problem

The paper's most fundamental conceptual move is reframing LLM inference efficiency from "how do we make models run faster?" to "how do we measure model efficiency in a way that is comparable across providers?" This is a shift from optimization to measurement infrastructure, and it is more profound than it sounds. The dominant paradigm in ML systems research has been to build faster kernels, better parallelism strategies, or more efficient attention mechanisms—each paper claiming speedups over a baseline measured on the authors' particular hardware. The unstated assumption has been that if you care about efficiency, you should build a faster system. This paper argues that before optimization, the field needs standardized measurement: a way to answer whether Model A is inherently more efficient than Model B, independent of whose engineers wrote better CUDA kernels.

This shift mirrors the role that standardized benchmarks like ImageNet played in computer vision—not a contribution to model architecture per se, but infrastructure that enabled the field to compare architectures fairly and thereby drive progress. Before ImageNet, researchers reported accuracy on their own test sets with their own preprocessing; comparisons across papers were meaningless. Before this paper, efficiency comparisons across LLM providers were similarly meaningless—raw API runtimes conflate model architecture with serving infrastructure. The paper's idealized runtime metric is an attempt to provide the efficiency analogue of a standardized test set: a measurement protocol that factors out implementation-specific confounds so that the model architecture itself can be evaluated.

This is not an incremental refinement of existing metrics. Using FLOPs or parameter count as efficiency proxies was the previous state of the art (Section 6), and those proxies have fundamental flaws—FLOPs ignores hardware efficiency, parameter count ignores architecture. The paper doesn't just propose a better proxy; it proposes an entirely different category of metric: a directly measurable quantity (runtime) that has been normalized by controlling the measurement environment rather than by approximating the quantity from model properties. This is a methodological innovation that is more about how we do science in the LLM era than about any specific technical technique.

Innovation 2: The Inference-Time Hardware Portability of Transformer Architectures

A central enabling insight of the paper is that the inference runtime of a Transformer model depends entirely on its architecture hyperparameters—hidden size h, number of layers l, number of attention heads n—and not on its trained parameter values. This property, which the paper exploits by profiling "skeleton" models without trained weights (Section 3.2), is what makes the idealized runtime metric practically feasible: you can estimate how a proprietary model would run on your reference hardware without needing access to the model's weights.

This insight is obvious in retrospect—matrix multiplication FLOPs depend on tensor dimensions, not tensor values—but its implications for measurement methodology had not been exploited before this paper. Prior work on inference benchmarking (e.g., MLPerf) profiled actual trained models on actual hardware, which limited comparisons to models you could download and run yourself. Black-box API models were simply inaccessible for such profiling. The skeleton-model trick circumvents the access problem entirely, converting opaque commercial offerings into measurable quantities as long as the provider discloses architecture details.

This is a fundamentally new capability in the LLM evaluation landscape. It is distinct from "you can estimate model size from API behavior" (a form of reverse engineering); here, the manufacturer tells you the architecture but doesn't give you the weights, and that's sufficient information to characterize inference efficiency on any hardware platform. The paper's Table 1 makes this explicit: all 10 models studied had publicly available architecture details (h, l, n) even when the model weights were proprietary (OpenAI, AI21, Anthropic) or the model was served through a third-party API (Together). The only models excluded from the study were those with "publicly unavailable model architecture details (including OpenAI's ChatGPT and GPT-4)" (Section 5.1), which is a telling limitation—the methodology works precisely when architecture transparency exists, and creates an incentive for providers to disclose architecture details if they want their models included in efficiency benchmarks.

Innovation 3: A Parametric Form That Converts Profiling from O(N) to O(1) in Benchmark Size

The decomposition of autoregressive inference runtime into a sum of a piecewise linear function of prompt size and a linear function of output token count (Equation 3) is not itself the innovation—it follows directly from the Transformer computation structure described in Narayanan et al. (2021). The innovation is recognizing that this parametric form enables an estimation procedure whose cost is independent of benchmark size: profile a small grid of (p, o) points once (<2 hours per model), then compute any query's runtime in closed form thereafter.

To appreciate why this matters, consider the alternative. A benchmark like HELM might contain thousands of queries with diverse prompt sizes and output lengths. Exhaustively profiling a skeleton model on every query—running each benchmark instance through the reference hardware and timing it—would take hours per benchmark, making systematic efficiency-capability studies across multiple benchmarks prohibitively expensive. The parametric approach reduces this to a one-time profiling cost per model, amortized across all benchmarks and all future analyses. The paper reports that profiling takes under 2 hours for most models (Section 5.3), after which computing the idealized runtime for an entire benchmark of thousands of queries is essentially free.

This is analogous to the role that closed-form loss functions played in making scaling laws practical—Kaplan et al. (2020) didn't need to train models at every possible compute budget; they fit a parametric curve to a sparse set of training runs and interpolated. The paper's parametric runtime model does the same for inference efficiency. It is a conceptual contribution to measurement methodology, not to Transformer theory: the value is in recognizing that the runtime form is simple enough to be captured by a handful of regression coefficients, and that this simplicity makes systematic efficiency evaluation tractable at scale.

Innovation 4: The Provider Optimization Gap as a Discoverable Empirical Fact, Not an Assumption

The paper makes a diagnostic move that elevates it beyond a new metric proposal: it uses the relationship between idealized and denoised runtime to measure how much optimization each API provider has implemented beyond the reference stack. The inequality t_raw ≥ t_denoised ≥ t_idealized (Section 4.1) is an accounting identity by construction, but the violations of the second inequality are where the interesting science lives.

Figure 7b and Table 3 reveal something that was previously only speculation in the LLM community: some providers have invested heavily in inference optimization, while others have not. OpenAI's denoised per-output-token generation time is 0.030 seconds versus 0.081 seconds idealized on Megatron+A100—a 2.7× gap attributable to API-level optimizations (custom kernels, kernel fusion, better batching, possibly quantization). AI21 shows a 1.8× gap for its Grande model. Anthropic shows essentially no gap (0.057 vs. 0.054), and other providers (Cohere, Together, Microsoft) fall on the "idealized is faster" side, meaning their APIs run on hardware/software that is no more optimized than the reference stack.

This is a new kind of empirical finding in the LLM efficiency literature. Prior work either assumed all providers were similarly optimized (making raw runtimes comparable) or assumed they were incomparable (making efficiency analysis hopeless). The paper instead quantifies the optimization gap per provider, turning it from an unknown confound into a measured variable. This enables statements like "the OpenAI/davinci model appears on the Pareto frontier under raw runtime but consistently falls off under idealized runtime" (Section 5.4, Figure 9)—a specific, falsifiable claim about the source of OpenAI's latency advantage that could not have been made before these metrics existed.

The significance extends beyond the specific numbers. The paper provides a template for monitoring how provider optimization evolves over time: if OpenAI's optimization gap widens in future measurements, it suggests ongoing systems investment; if it narrows, the reference stack may be catching up. This turns efficiency measurement from a one-time comparison into a longitudinal capability.

Innovation 5: Efficiency-Capability Pareto Analysis at the Architecture Level

The paper's efficiency-capability tradeoff plots (Figure 9) are not the first to plot capability against a cost metric—Kaplan et al. (2020), Hoffmann et al. (2022), and the HELM benchmark (Liang et al., 2022) all produce capability-vs-scale curves. The innovation is in conducting this analysis at the level of specific model architectures with a metric that reflects deployment cost rather than training cost or an abstract proxy like parameter count.

The distinction matters concretely. Figure 9 shows that different efficiency metrics produce different Pareto frontiers for the same models on the same tasks. Under raw runtime, OpenAI/davinci sits on the frontier across multiple benchmarks. Under idealized runtime, it falls off. Under FLOPs or model size, the rankings change again—Yandex/YaLM uses fewer FLOPs than BigScience/BLOOM, but BLOOM's architecture is more runtime-efficient on GPU hardware (a finding attributed to Scao et al., 2022's careful operator optimization). This means that the answer to "which models are on the efficiency frontier?" depends on which efficiency metric you choose, and the paper makes a normative argument that idealized runtime (and its cost-aware variants) is the right metric for comparing inherent model efficiency because it controls for confounding implementation factors.

A further finding from this analysis is that OpenAI/davinci is the only model whose Pareto-optimal status reverses when switching from raw to idealized runtime, while other models maintain consistent relative positioning across metrics. This is a specific, falsifiable diagnostic: OpenAI's apparent efficiency advantage in raw API measurements is a property of their serving infrastructure, not their model architecture. This finding has direct implications for model creators evaluating architectural tradeoffs—if they benchmark their prototype on their own optimized serving stack while comparing against a competitor's raw API latency, they will draw incorrect conclusions about relative architectural efficiency. The idealized metric provides a remedy.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates efficiency-capability tradeoffs on four diverse tasks drawn from the HELM benchmark (Liang et al., 2022): a sentiment analysis task (IMDB; Maas et al., 2011), two question answering tasks (MMLU college chemistry; Hendrycks et al., 2020, and BoolQ; Clark et al., 2019), and a classification task (RAFT terms of service; Alex et al., 2021). The exact number of instances per task is not specified in the main text, but these are standard HELM evaluation scenarios. The paper does not report the size of the profiling "synthetic queries" dataset used to fit the runtime parameters—these are synthetically constructed prompts with controlled (p, o) dimensions, not drawn from any existing NLP benchmark.
Base model(s). The paper evaluates 10 state-of-the-art autoregressive Transformer LLMs ranging from 6 to 530 billion parameters, drawn from six different providers: OpenAI (davinci, 175B), AI21 Labs (J1-Large v1 at 6.7B, J1-Grande v1 at 17B, J1-Jumbo v1 at 178B), Cohere (XL v20220609, 52B), Anthropic (v4-s3, 52B), Microsoft+NVIDIA (TNLG v2, 530B), and three models served through the Together Open Models API: EleutherAI/GPT-J (6B), Yandex/YaLM (100B), and BigScience/BLOOM (176B). All 10 models have publicly known architecture hyperparameters (h, l, n) listed in Table 1, which is the sole requirement for the idealized runtime estimation procedure. Models with undisclosed architectures—including OpenAI's ChatGPT and GPT-4—are explicitly excluded from evaluation (Section 5.1).
Metrics. The paper defines and compares six efficiency metrics, each averaged over all instances in a scenario: (1) model size in billions of parameters; (2) per-query floating-point operations (FLOPs) computed from the forward-pass FLOP formula in Appendix A; (3) raw runtime in seconds, measured by timing black-box API calls; (4) denoised runtime in seconds, estimated by fitting the parametric form (Equation 3) to minimum-trial runtimes measured through the black-box API; (5) idealized runtime in seconds, estimated by profiling skeleton models on the uniform reference stack (NVIDIA A100 GPUs with Megatron); and (6) idealized cost in cents, computed as t_idealized × n_accelerator × c_accelerator (Section 4.2) using per-hour AWS A100 GPU pricing. The capability metric is always accuracy on the target HELM task. For the ranking analysis in Figure 10, two composite objective functions are used: f₁ = accuracy / idealized_runtime and f₂ = accuracy / log(idealized_runtime).
Baselines. Because the paper proposes a measurement methodology rather than a new model or optimization technique, there are no "baseline methods" in the traditional sense. The comparison is between different efficiency metrics applied to the same set of models. The key analytical baseline is raw runtime (what practitioners currently observe and what prior benchmarking efforts implicitly rely on) compared against the paper's proposed idealized runtime and denoised runtime. For the FLOPs comparison, the paper uses the standard forward-pass FLOP formula from Narayanan et al. (2021). For capability measurement, the paper relies on HELM's existing accuracy evaluation pipeline rather than proposing new capability baselines.
Generation budget / compute accounting. There is no "generation budget" in the sense of the prior paper analyzed (which studied how to allocate a fixed number of LLM samples). Instead, computational cost is measured in three distinct ways: (1) for FLOPs, the paper computes per-query forward-pass floating-point operations from the architecture hyperparameters using equations in Appendix A; (2) for runtime metrics, wall-clock time is measured on the reference system (A100 GPUs) or through the black-box API; (3) for idealized cost, runtime is multiplied by the number of GPUs required (Table 1, column 7) and the per-second rental cost of an A100 GPU on AWS. The accounting for GPU count follows a "minimum number of GPUs necessary" principle with tensor model parallelism within a single server when possible (Section 3.2). All runtime measurements for idealized metrics use batch size 1 (single query at a time), representing the latency of serving one request in isolation.
Cross-validation / statistical protocol. The paper does not use cross-validation or train-test splits because it is not training models or making predictions that could overfit. Instead, statistical reliability is addressed through: (1) multiple trials for denoised runtime estimation, where the minimum across trials is used to factor out performance variance η ≥ 0 (Section 4.1); (2) coefficient of determination (R^2) reported for the linear regression fits (Table 2 for denoised runtime, Section 3.2 for idealized runtime where all R² > 0.999); and (3) qualitative assessment of the inequality t_raw ≥ t_denoised ≥ t_idealized as an empirical validation that the metrics behave as theoretically expected (Figure 7, where 96.6% of raw runtime points lie above denoised runtime). For the Pareto frontier analysis in Figure 9, the paper does not report error bars, confidence intervals, or statistical tests for whether a model's position on the frontier is significant. All experiments were run in September or October 2022 with the latest API versions available at the time, and the paper acknowledges that results may vary by time period.

Main Quantitative Results

The paper's experimental results are organized around four clusters of evidence: (1) validation that the parametric runtime form holds empirically, (2) characterization of performance variance in black-box APIs, (3) efficiency-capability tradeoff analysis comparing the six metrics, and (4) a case study on using the methodology to evaluate alternative hardware/software stacks.

Validation of the Parametric Runtime Form

The paper establishes the empirical validity of Equation 3's decomposition into prompt-encoding time and per-output-token generation time through controlled profiling of skeleton models on the reference A100+Megatron system.

Output-generation linearity (Figure 3). For each of four models (OpenAI/davinci, AI21/J1-Jumbo v1, AI21/J1-Grande v1, Anthropic/v4-s3), the paper plots end-to-end runtime against number of generated output tokens at different fixed prompt sizes (shown as separate lines in the legend). The key finding is that "the coefficients of determination (R^2) for the resulting time estimates are very close to 1.0 (> 0.999) for all models." This near-perfect linearity confirms that the per-output-token generation cost g is indeed constant—independent of token position i—and that throughput_og does not vary with the number of tokens generated so far. The different lines (different prompt sizes) have similar slopes but different y-intercepts, visually confirming the additive decomposition t = prompt_encoding_time(p) + (o - 1) · g.

For example, for OpenAI/davinci at prompt size p = 512 (extrapolating from the fitted parameters in Table 3), the prompt-encoding time is 0.178 seconds and g = 0.081 seconds per output token. A query generating 100 tokens would take approximately 0.178 + 99 × 0.081 ≈ 8.20 seconds.

Prompt-encoding piecewise linearity (Figure 4). The paper plots prompt-encoding time against prompt size for the same four models, with dotted best-fit lines. The curves reveal the theoretical piecewise linear behavior: "We see that runtime and the prompt size have a roughly linear relationship, especially at large prompt sizes. However, this linear relationship breaks down at smaller prompt sizes." For OpenAI/davinci, the prompt-encoding time grows from 0.178 seconds at p = 512 to 0.323 seconds at p = 1024 to 0.476 seconds at p = 1536 (Table 3), showing the expected sub-linear growth in the per-token cost as arithmetic intensity increases with p.

The paper explicitly attributes this to GPU arithmetic intensity: at small p, the computation is memory-bound because the model's weight matrices must be read from GPU memory but are amortized over few token positions. As p grows, the same weight reads are reused across more tokens, pushing throughput higher until it saturates at the hardware's peak compute throughput. This transition from memory-bound to compute-bound produces the piecewise linear shape.

Accuracy of the estimation when applied to black-box APIs (Table 2). Even when the regression is performed on noisy API runtimes (for denoised runtime), the parametric form holds with high fidelity. The R^2 values are: 0.985 for OpenAI/davinci, 0.990 for AI21/J1-Large v1, 0.917 for AI21/J1-Grande v1, 0.995 for AI21/J1-Jumbo v1, 0.997 for Cohere/XL v20220609, and 0.924 for Anthropic/v4-s3. The lower R^2 for AI21/J1-Grande v1 (0.917) and Anthropic/v4-s3 (0.924) correlates with their higher observed performance variance (discussed below), confirming that the parametric form itself is sound—the residual variance is in the API overhead, not in the underlying computation structure.

Characterization of Black-Box API Performance Variance

This cluster of results quantifies the performance noise that motivates the denoised runtime metric and provides the first systematic evidence of how much black-box API latency can vary depending on provider, model, and load.

Per-trial variance across models from the same provider (Figure 5). The paper runs multiple trials of synthetic queries with controlled (p, o) through the AI21 Labs API for three models (AI21/J1-Large v1, AI21/J1-Grande v1, AI21/J1-Jumbo v1) at fixed prompt size p = 512 and varying output token counts. The key finding is that variance is model-specific, not uniform across a provider's offerings: "AI21/J1-Grande v1 has much higher performance variance than AI21/J1-Large v1 or AI21/J1-Jumbo v1 (larger spread among points for a query of given size)." The coefficient of variation for AI21/J1-Grande v1 averages approximately 0.55, compared to approximately 0.2 for the other two models. This means that Grande's runtime can swing by ±55% from its mean across trials, while Large and Jumbo vary by only ±20%.

Additionally, "even for models with lower spreads (e.g., AI21/J1-Large v1), we see that outlier points can have as much as 3× higher latency." This is critical for latency-SLO-constrained applications: even a model with generally stable performance can produce occasional queries that violate tight latency bounds, making median runtime an insufficient characterization.

Load-dependent variance (Figure 6). For the Anthropic/v4-s3 model, the paper varies the number of parallel queries sent simultaneously (with fixed p = 512) and measures the minimum runtime across 10 trials for each parallel query count. The finding: "We observe as much as a 2× increase in runtime, indicating that load can lead to increased contention on API servers and consequently increased observed runtime." This 2× degradation is measured even on the minimum runtime across trials, meaning it reflects a genuine saturation of the serving infrastructure's capacity rather than random within-trial noise. As the number of concurrent queries increases, even the best-case query experiences queuing delays or bandwidth contention.

Empirical validation of the inequality (Figure 7). Figure 7a confirms that denoised runtime (computed from minimum-trial API profiling) is a valid lower bound on raw runtime: "96.6% of points" lie above the y = x line, meaning raw runtime ≥ denoised runtime for the overwhelming majority of queries. The 3.4% of points that fall below may represent trials where randomness worked in the opposite direction (a trial used for raw runtime happened to be faster than the minimum from the profiling trials) or where the API's performance characteristics shifted between profiling and evaluation.

Figure 7b tests the second inequality t_denoised ≥ t_idealized. For AI21 Labs models, idealized runtimes are greater than denoised runtimes 15.7% of the time. For OpenAI models, this violation rate is 64.2%. For all other providers (Cohere, Anthropic, Together, Microsoft), idealized runtimes are always lower than denoised runtimes. The interpretation: AI21 and especially OpenAI have implemented software/hardware optimizations beyond what Megatron on A100 GPUs provides, making their API faster than the reference stack for the same model architecture. Other providers are using infrastructure that is no faster than the reference—and often slower.

Quantifying the provider optimization gap (Table 3). The fitted parameters from the regression make the optimization gap concrete for four models. For OpenAI/davinci, the idealized per-output-token generation time g_idealized = 0.081 seconds (on Megatron+A100) versus g_denoised = 0.030 seconds (through OpenAI's API)—a 2.7× speedup from provider optimizations. For AI21/J1-Grande v1: g_idealized = 0.038 vs. g_denoised = 0.021 (1.8× speedup). For AI21/J1-Jumbo v1: g_idealized = 0.064 vs. g_denoised = 0.042 (1.5× speedup). For Anthropic/v4-s3: g_idealized = 0.054 vs. g_denoised = 0.057 (idealized is slightly faster, no provider speedup). The prompt-encoding times tell a similar story: OpenAI's denoised prompt-encoding times are substantially lower than idealized times across all prompt sizes (e.g., at p = 1024: 0.033 denoised vs. 0.323 idealized), while Anthropic's denoised and idealized times are close (0.191 vs. 0.189).

Efficiency-Capability Tradeoff Analysis

This is the paper's central empirical contribution: comparing how six different efficiency metrics produce different conclusions about which models are Pareto-optimal on four NLP tasks.

Pareto frontier analysis across six metrics (Figure 9). Figure 9 presents six rows of graphs (one per metric: model size, FLOPs, raw runtime, denoised runtime, idealized runtime, idealized cost) × four columns of tasks (IMDB, MMLU college chemistry, RAFT terms of service, BoolQ). Each graph plots average accuracy against the metric for all 10 evaluated models, with Pareto-optimal models shown as squares connected by a dotted black line and sub-optimal models as circles.

Finding 1: Model size is a poor predictor of capabilities. "Only a subset of the evaluated models fall on a Pareto frontier, with different models on the Pareto frontier for different tasks." The paper notes that this "suggests that scale alone does not predict model capabilities," connecting to the emergent abilities literature (Wei et al., 2022). For example, in the BoolQ panel (Figure 9d), the model size row shows that some smaller models achieve higher accuracy than larger ones, placing them on the efficiency frontier despite having fewer parameters.

Finding 2: Raw runtime misleads about inherent model efficiency. This is the headline result: "The OpenAI/davinci model appears in the Pareto frontier for each benchmark when using raw runtimes but not the idealized metrics." This pattern is visible by comparing the raw runtime row to the idealized runtime row across all four tasks. In the raw runtime row, the green OpenAI/davinci point consistently sits on or near the frontier (connected by the black dotted line). In the idealized runtime row, OpenAI/davinci drops off the frontier, replaced by other models. The paper interprets this as direct evidence that "the superior inference runtime performance of certain APIs is often a byproduct of optimizations within the API rather than the underlying model" (Abstract).

Finding 3: The Pareto frontier depends on which metric you trust. No single model dominates across all metrics. The paper highlights the comparison between BigScience/BLOOM and Yandex/YaLM: "while BigScience/BLOOM achieves cheaper idealized cost (which takes into account the lower number of GPUs that Yandex/YaLM requires), Yandex/YaLM uses fewer floating-point operations." In the FLOPs row, YaLM generally sits to the left of BLOOM (fewer FLOPs at comparable accuracy). In the idealized cost row (bottom row), this reverses—BLOOM achieves lower cost. The paper attributes this to BLOOM's architecture being "more [thoroughly searched] through model architectures for minimum runtime with a given number of floating-point operations in the forward pass (Scao et al., 2022)." This is a concrete demonstration of why FLOPs alone is an inadequate efficiency metric: hardware-aware architecture optimization can make a model with more FLOPs achieve lower runtime and cost.

Finding 4: Run-to-run variance distorts raw runtime comparisons. The paper notes: "AI21/J1-Grande v1 often achieves worse raw runtime than AI21/J1-Jumbo v1 despite having 10× fewer parameters, since the Grande model experiences higher performance variance (Figure 5)." This is visible in Figure 5 and manifests in the efficiency-capability plots where AI21/J1-Grande v1's average raw runtime can be higher than AI21/J1-Jumbo v1's despite the ~10× parameter difference, purely because Grande's high variance drives up the mean. The idealized metrics "factor out run-to-run variance, making it easier to see the true efficiency-capability tradeoffs."

Finding 5: Estimated inference costs are an order of magnitude below charged API prices. The paper reports: "We observe that they are up to an order of magnitude lower than the charged actual costs." The exact per-query idealized costs are not tabulated in the paper, so the magnitude of the gap cannot be quantified precisely from the provided figures, but the paper explicitly notes this does not incorporate training cost amortization, which "presumably gets amortized into the cost users pay with black-box APIs."

Composite objective function ranking (Figure 10). Figure 10 plots the rank of each model (y-axis) using f₁ = accuracy / idealized_runtime across the four benchmarks (x-axis). The finding is that "each model achieves similar ranks across benchmarks" under this objective—the rank ordering is largely consistent regardless of which task is being evaluated. However, switching to f₂ = accuracy / log(idealized_runtime) (which de-emphasizes inference efficiency relative to accuracy by using a log transformation) "increases variation across benchmarks and impacts relative model ordering," with the paper specifically noting that "Microsoft+NV/TNLG v2's average rank significantly improves" under f₂. This is expected because TNLG v2 is the largest model (530B parameters) and achieves high accuracy but has high idealized runtime; the log transformation reduces the penalty for high runtime, allowing its accuracy advantage to dominate.

Efficient Evaluation of Alternative Hardware Stacks

Section 5.3 demonstrates that the methodology is not just a one-time metric but a tool for ongoing systems analysis by comparing the reference A100 GPU configuration against older V100 GPUs.

A100 vs. V100 comparison (Figure 8). Figure 8 shows idealized runtime and idealized cost (in cents) estimated on A100 GPUs (x-axis) versus V100 GPUs (y-axis) for the 10 evaluated models. Each model is a single point in these scatter plots. If V100 were cheaper or comparably priced, points would lie on or below the y = x line. The paper reports the opposite: "we find that this is not the case, suggesting V100 GPUs are both slower and more expensive." The V100 GPUs produce higher runtime (slower) and higher cost (more expensive) than A100 GPUs.

The paper explains: "This is partially because we often have to use double the GPUs to fit the model parameters in GPU memory, since V100 GPUs only have 32GB of device memory compared to 80GB on the A100 GPUs." The reduced memory capacity forces more aggressive model parallelism, increasing the total GPU count (n_accelerator in the cost formula) and thus the total cost despite the V100's potentially lower per-GPU-hour price. This demonstrates how the idealized cost metric captures deployment-level tradeoffs (memory capacity, parallelism overhead) that pure runtime or FLOPs metrics miss.

Efficiency of the estimation procedure. The paper claims that this differential analysis "requires profiling on the order of hours (< 2 hours for most models, depending on the number of (p, o) values profiled) once, compared to hours per benchmark (depending on number of queries in the benchmark) for exhaustive profiling." The profiling cost is dominated by the grid of (p, o) configurations profiled; for a model with 2048-token context window and profiling set P = {1, 256, 512, 1024, 1536} with, say, 3–5 output token counts per prompt size, this amounts to 15–25 profiling runs. Each run involves executing the skeleton model on the reference hardware and timing it, which for large models requiring multi-GPU tensor parallelism might take seconds to tens of seconds per run. The paper does not report the exact number of profiled configurations or wall-clock profiling times per model.

Ablation Studies and Robustness Checks

Because the paper proposes a measurement methodology rather than a trained system, the "ablation studies" are analyses that test the sensitivity of conclusions to different metric choices and validate that the metrics behave as theoretically predicted.

Raw runtime vs. denoised runtime inequality (Figure 7a): The paper verifies that denoised runtime (minimum-trial regression) serves as a valid lower bound on raw runtime. 96.6% of raw runtime data points lie above the corresponding denoised runtime, confirming the noise model η ≥ 0. The 3.4% of violating points are not analyzed further, but likely represent cases where the profiling trials captured a lower minimum than the specific trial used for the raw runtime comparison, or time-dependent changes in API performance.

Idealized vs. denoised runtime inequality (Figure 7b): This comparison reveals which providers have implemented optimizations beyond the reference A100+Megatron stack. The violation rates—15.7% for AI21, 64.2% for OpenAI, 0% for all others—provide a quantitative signature of each provider's infrastructure investment. This is presented as a robustness check that the idealized metric is genuinely measuring model-level efficiency: if idealized runtime were always lower than denoised runtime across all providers, it would suggest the reference stack is unrealistically fast and the idealized metric is too optimistic. The fact that some providers achieve faster denoised runtime confirms that the reference stack represents a strong but achievable baseline.

Model size vs. FLOPs vs. runtime as efficiency proxies: The efficiency-capability plots in Figure 9 collectively serve as an ablation of which metric most faithfully captures deployment-relevant efficiency. The paper does not run a formal ablation varying one factor while holding others constant, but the qualitative finding—that different metrics produce different Pareto frontiers, and that raw runtime misattributes provider optimizations to model architecture—constitutes evidence that idealized runtime is preferable for comparing inherent model efficiency.

Objective function sensitivity (Figure 10): The comparison between f₁ = accuracy / idealized_runtime and f₂ = accuracy / log(idealized_runtime) demonstrates that model rankings are sensitive to how strongly the objective penalizes inference cost. Under f₁, models with high runtime are heavily penalized, and the ranking is relatively consistent across tasks. Under f₂, the log transformation reduces the penalty, allowing high-accuracy, high-runtime models like Microsoft+NV/TNLG v2 to rise in the rankings. This is presented as evidence that no single model dominates across all reasonable objective functions—practitioners must choose an objective that reflects their specific latency-vs-accuracy tradeoff preferences.

Hardware portability (Figure 8): The A100 vs. V100 comparison validates that the methodology cleanly separates model architecture from deployment hardware: the same models, when evaluated on different hardware using the same parametric estimation procedure, produce systematically different efficiency landscapes. This demonstrates that the methodology can be used to answer "what if" questions about hardware upgrades or cost optimization without physically migrating models.

Multiple-provider coverage: The inclusion of 10 models from 6 different providers (commercial APIs, research-access APIs, and open-source models served through Together) demonstrates that the methodology generalizes across the LLM serving ecosystem. The only excluded models are those with undisclosed architectures (ChatGPT, GPT-4), which is a transparency requirement rather than a methodological limitation.

Negative results and unexpected findings:

AI21/J1-Grande v1's anomalous variance (Figure 5): The fact that a 17B-parameter model can exhibit worse raw runtime than a 178B-parameter model from the same provider (AI21/J1-Jumbo v1) due solely to performance variance is an unanticipated finding. It demonstrates that raw runtime is not just noisy but can produce qualitatively wrong rankings when variance is model-specific.
V100 GPUs are both slower and more expensive (Section 5.3): This finding contradicts the natural assumption that older hardware trades speed for cost savings. Because V100's 32GB memory limit forces double the GPU count for large models, the total cost exceeds that of fewer, faster A100 GPUs. This is a concrete example of how the idealized cost metric captures deployment-level constraints that FLOPs or runtime metrics miss.
The order-of-magnitude gap between idealized and charged costs (Section 5.4): The paper reports that idealized inference costs are "up to an order of magnitude lower than the charged actual costs" but does not break down the components of this gap (provider margin, training amortization, infrastructure redundancy, dynamic provisioning overhead). This is a finding that raises questions the paper does not fully answer.

Critical Assessment

Claim 1: "Idealized runtime enables apples-to-apples comparisons across models"

This is the paper's central methodological claim. The experiments support it with meaningful scope limitations. The idealized runtime metric demonstrably produces different efficiency rankings than raw runtime (Figure 9: OpenAI/davinci falls off the Pareto frontier when switching from raw to idealized runtime), which validates that the metric is capturing something distinct from observable API behavior. The inequality validation in Figure 7 confirms that idealized runtime behaves as theoretically expected: it is a lower bound on denoised runtime for providers without significant API-level optimizations, and it is exceeded (denoised < idealized) specifically for providers known to have invested in custom inference infrastructure (OpenAI, AI21).

However, the claim of "apples-to-apples" comparison rests on an assumption that is not empirically tested: that Megatron on A100 GPUs is a "fair" uniform reference for all model architectures. Different model architectures may benefit differently from Megatron's specific implementation choices (e.g., its kernel fusion strategy, its memory management for KV-caching, its parallelism heuristics). A model with a hidden size that aligns poorly with Megatron's tensor-parallelism granularity or an attention-head configuration that triggers a performance cliff in NVIDIA's CUDA kernels would appear artificially disadvantaged on the reference stack, not because the architecture is inherently inefficient, but because the specific reference software does not optimize for it as well as a provider's custom implementation might. The paper does not test sensitivity to the choice of reference software (e.g., comparing Megatron to FasterTransformer, or to a naive PyTorch implementation). The A100 vs. V100 comparison (Figure 8) tests hardware sensitivity but not software sensitivity. This matters because the paper's key finding—that OpenAI/davinci falls off the Pareto frontier under idealized runtime—could be partially attributable to Megatron not being as well-tuned for OpenAI's specific architecture as OpenAI's own serving infrastructure is, rather than to the architecture being inherently inefficient.

The paper would have been strengthened by profiling on at least one alternative software stack (e.g., NVIDIA's FasterTransformer or vLLM) to test whether the OpenAI/davinci Pareto-frontier reversal is robust to reference software choice.

Claim 2: "Superior inference runtime performance of certain APIs is often a byproduct of optimizations within the API rather than the underlying model"

This claim is well-supported for OpenAI/davinci specifically and to a lesser extent for AI21 models. The evidence chain is clean:

Raw runtime places OpenAI/davinci on the Pareto frontier (Figure 9, raw runtime row).
Idealized runtime (Megatron+A100) removes OpenAI/davinci from the frontier (Figure 9, idealized runtime row).
The provider optimization gap is quantified: OpenAI's denoised per-token generation time is 2.7× faster than idealized (0.030 vs. 0.081 seconds, Table 3), confirming that OpenAI's API is substantially more optimized than the reference stack.
For Anthropic (no provider gap: 0.057 denoised vs. 0.054 idealized) and Cohere/Together/Microsoft (idealized faster than denoised), the reference stack is at least as fast as their APIs, so there is no spurious API-optimization advantage to confound their efficiency positioning.

However, the claim's generalizability is limited by sample size. "Often" in the claim is based on one provider (OpenAI) out of six evaluated showing a clear Pareto-frontier reversal due to API optimizations. AI21 models show a smaller optimization gap (1.5–1.8× for token generation) but the paper does not explicitly state whether any AI21 models also reverse their frontier status. Since only OpenAI/davinci is called out as exhibiting the reversal, the strength of "often" rests on a single example. This is partially a function of the models available for study—many frontier models had undisclosed architectures—but the quantitative strength of the claim should be calibrated to the evidence: demonstrated for OpenAI, likely for AI21 to a lesser extent, not tested for providers whose architecture details are unknown.

Claim 3: "The parametric form enables efficient estimation requiring hours of profiling once rather than hours per benchmark"

This claim is supported in principle but the paper reports insufficient timing data to verify the "<2 hours" figure independently. The paper states that profiling takes "<2 hours for most models, depending on the number of (p, o) values profiled" but does not report:

The exact number of (p, o) configurations profiled per model.
The wall-clock time for each profiling run.
The total profiling time broken down by model.
Whether the "<2 hours" includes the time to set up the skeleton model, configure tensor/pipeline parallelism, compile CUDA kernels, etc., or only the runtime measurement portion.

This makes the efficiency claim difficult to evaluate. For a model like OpenAI/davinci requiring 8 A100 GPUs with tensor parallelism, a single forward pass generating 100 tokens might take ~8 seconds (using the idealized parameters from Table 3: 0.178 + 99 × 0.081 ≈ 8.2 seconds at p = 512). Profiling a grid of, say, 5 prompt sizes × 4 output token counts = 20 configurations would take approximately 20 × 8 = 160 seconds of pure inference time, which is well under 2 hours. But this doesn't account for the time to warm up the GPU, run each configuration multiple times to get a stable measurement, or handle the larger prompt sizes that take longer per run. The claim is plausible but unverified by the data presented.

More critically, the paper does not report the cost of the difficult-estimation step that is implicit in the methodology. For the idealized metrics, you need to know each model's architecture (h, l, n)—information that was publicly available for the 10 studied models but is not guaranteed for future models. The paper excludes "models with publicly unavailable model architecture details" (Section 5.1), which means the methodology's applicability depends on a transparency precondition that the paper does not quantify (what fraction of current LLM APIs disclose their architectures?).

Claim 4: "Model size and FLOPs are inadequate proxies for inference efficiency"

This claim is strongly supported by the empirical evidence in Figure 9. The model size row shows that similarly-sized models can have vastly different accuracies (rendering size alone a weak predictor of where a model sits in the capability distribution). The FLOPs row shows Yandex/YaLM using fewer FLOPs than BigScience/BLOOM, yet BLOOM achieves cheaper idealized cost because its architecture is more runtime-efficient on GPU hardware. This is a direct counterexample to the claim that FLOPs is a sufficient efficiency proxy: two models with different FLOPs can have the opposite ordering on deployment-relevant cost.

The paper would have been strengthened by a more explicit analysis of why FLOPs fails in specific cases. The paper attributes BLOOM's advantage to architecture search (citing Scao et al., 2022) but does not analyze which architectural features (attention-head count, FFN ratio, layer count vs. hidden size ratio) drive the FLOPs-vs-runtime discrepancy. This limits the actionable insight for model creators: knowing that architecture search helps is less useful than knowing what to search for.

Broad Structural Weaknesses

Single reference hardware generation. All idealized measurements use A100 GPUs. The V100 comparison (Figure 8) is a point comparison showing the methodology can handle other hardware, but the main efficiency-capability analysis does not report idealized runtimes on, for example, H100 GPUs or TPUs. Since the idealized metric is defined relative to a chosen hardware target, and the paper's key finding (OpenAI/davinci falls off the Pareto frontier) is measured on A100 GPUs, this finding could in principle differ on other hardware. A model that is memory-bandwidth-bound on A100 might be compute-bound on H100 (with its higher memory bandwidth), changing its relative efficiency ranking versus other models. The paper acknowledges that "Other accelerators like the TPU or the NVIDIA H100 GPU could also be used" but does not test this sensitivity.

No capability confidence intervals. The efficiency-capability plots in Figure 9 present average accuracy as a point estimate without error bars. Given that the HELM scenarios used (IMDB, MMLU college chemistry, BoolQ, RAFT) have finite test sets, the accuracy estimates have sampling variance that could affect which models fall on the Pareto frontier. A model sitting just below the frontier in the point-estimate plot could be statistically indistinguishable from the frontier models if confidence intervals were shown. The paper does not report test set sizes, standard errors, or any statistical comparison between frontier and non-frontier models.

Temporal validity. All experiments were run in "September or October 2022." Black-box API performance can change rapidly as providers update their serving infrastructure, switch hardware, or modify rate limits. A model that showed a 2.7× provider optimization gap in October 2022 may show a different gap today. The paper does not discuss the temporal stability of the metrics or propose a versioning system for efficiency measurements.

Model architecture details as gating dependency. The methodology's applicability is entirely gated on architecture transparency from model providers. Table 1 lists h, l, and n for all 10 evaluated models, and the paper notes that models without these details (ChatGPT, GPT-4) are excluded. This is not a weakness of the methodology per se—it is a fundamental requirement for any measurement approach that needs model architecture to estimate computation—but it limits the scope of models that can be evaluated. As of the paper's writing, the two most commercially significant models (ChatGPT and GPT-4) could not be included, which means the efficiency-capability analysis in Figure 9 omits precisely the models that practitioners are most likely to be choosing between for deployment.

The FLOPs counting assumes p ≪ 6h. The simplified FLOP formulas θ_pe · p and θ_og rely on the approximation that the quadratic attention term p²/(6h) is negligible. For models with small hidden sizes relative to their context windows (e.g., a hypothetical model with h = 2048 and p = 8192), this approximation would break down, and the per-token FLOPs would not be constant. The paper validates the approximation for the studied models (for davinci, p/(6h) ≤ 0.028), but the methodology's claimed generality to "autoregressive Transformer models" (Section 3.3) should be qualified with this assumption.

6. Limitations and Trade-offs

1. Architecture Specification Availability as a Gating Requirement

The assumption or constraint. The idealized runtime estimation procedure requires knowledge of three model architecture hyperparameters: hidden size h, number of Transformer layers l, and number of attention heads n (Table 1). Without these, the skeleton model cannot be instantiated on the reference hardware, and the profiling-and-regression pipeline cannot be applied. The paper is explicit about this scope restriction: "We do not evaluate models with publicly unavailable model architecture details (including OpenAI's ChatGPT and GPT-4)" (Section 5.1).

The consequence. The methodology is entirely inapplicable to the most commercially significant and widely deployed LLMs at the time of writing — ChatGPT and GPT-4 — and, more broadly, to any future model whose provider chooses not to disclose architecture details. This means the efficiency-capability analysis in Figure 9 omits precisely the models that practitioners are most likely to be choosing between for real-world deployment decisions. The paper's central finding — that OpenAI/davinci falls off the Pareto frontier under idealized runtime while appearing on it under raw runtime — is demonstrated on an older model (davinci, from the GPT-3 family) whose architecture happens to be publicly known, while the successor models that replaced it in production (GPT-3.5-turbo, GPT-4) cannot be evaluated. A practitioner deciding whether to deploy GPT-4 versus Anthropic's Claude or Google's Gemini in 2024 would find that the paper's methodology cannot compare any of these models unless all providers choose to disclose their architectures — a choice they may actively resist since architecture details can be considered proprietary competitive information. The paper's stated goal of enabling "model creators [to] make better informed decisions about long-term model investments" (Section 7) is thus limited to the subset of the model landscape where architecture transparency exists.

What evidence exists in the paper. Table 1 lists h, l, and n for all 10 evaluated models, confirming that architecture disclosure was the inclusion criterion. Section 5.1 explicitly notes the exclusion of ChatGPT and GPT-4. The paper provides no quantitative assessment of what fraction of current LLM APIs disclose their architectures — this is an unmeasured scope limitation. The sample of 10 models spans six providers and demonstrates the methodology's applicability where architecture is known, but says nothing about its coverage of the overall LLM ecosystem.

Mitigation status. The paper does not attempt to address this limitation. It acknowledges the exclusion (Section 5.1) but does not propose workarounds — such as inferring architecture from observable API behavior, developing approximation techniques that require less architectural detail, or advocating for architecture transparency standards. The limitation is a fundamental constraint of the skeleton-model approach: if you cannot build the skeleton, you cannot profile it. No amount of methodological refinement can circumvent the information requirement.

2. Single Reference Software Stack Without Sensitivity Analysis

The assumption or constraint. The idealized runtime metric fixes both hardware (h* = NVIDIA A100 GPUs) and software (s* = NVIDIA's Megatron inference implementation) as the uniform reference platform. The paper states: "We use Megatron's (a high-performance GPU implementation) Transformer and autoregressive inference functionality" (Section 3.2) and justifies A100 GPUs as "the fastest widely available GPU right now." The paper explicitly acknowledges that "Other accelerators like the TPU or the NVIDIA H100 GPU could also be used" (Section 3.2), but treats the software choice as fixed without testing alternatives.

The consequence. Different model architectures may interact differently with Megatron's specific implementation choices — its kernel fusion strategy, its memory management for KV-caching during autoregressive generation, its tensor parallelism granularity, and its heuristics for partitioning attention heads across GPUs. A model whose hidden size h or attention-head count n aligns poorly with Megatron's internal assumptions could appear artificially disadvantaged on the reference stack — not because the architecture is inherently inefficient, but because the specific reference software does not optimize for it as effectively as another implementation (e.g., NVIDIA's FasterTransformer, vLLM, or a provider's custom serving stack) might. The paper's headline finding — that OpenAI/davinci falls off the Pareto frontier under idealized runtime while appearing on it under raw runtime (Figure 9, Section 5.4) — could be partially attributable to Megatron not being as well-tuned for OpenAI's specific architecture (hidden size 12288, 96 layers, 96 attention heads) as OpenAI's own production serving infrastructure is, rather than reflecting an inherent inefficiency in the davinci architecture itself. If a different reference software stack (e.g., FasterTransformer with its own operator fusion and memory management policies) produced a different idealized runtime for davinci relative to other models, the Pareto frontier might shift, potentially altering the paper's key qualitative conclusion about where davinci's efficiency advantage comes from.

What evidence exists in the paper. The paper provides evidence of hardware sensitivity: Figure 8 compares idealized runtimes on A100 vs. V100 GPUs using the same software (Megatron), demonstrating that hardware choice affects absolute runtime and cost but not the relative ranking methodology. However, the paper provides no equivalent software sensitivity analysis. There is no comparison of Megatron to FasterTransformer, to a naive PyTorch implementation, to TensorRT-LLM, or to any other inference framework. The paper's A100 vs. V100 comparison validates that the methodology can handle hardware variation, but it does not test whether the methodology's conclusions are robust to software variation. The paper attributes OpenAI's 2.7× provider optimization gap (Table 3: g_idealized = 0.081 vs. g_denoised = 0.030) to "optimizations within the API rather than the underlying model," but this gap conflates hardware advantages (OpenAI may use newer GPUs than A100s), software advantages (custom CUDA kernels), and system-level advantages (query caching, continuous batching). The software component of this gap — which could be closed by using a better reference software stack — is not isolatable from the data presented.

Mitigation status. The paper does not attempt to address software sensitivity. The choice of Megatron is justified pragmatically (it is a high-performance, open-source implementation), but the paper does not discuss why alternative software stacks were not tested, whether Megatron's implementation choices might differentially advantage certain architectures, or whether the key findings would survive a switch to a different reference implementation. Section 8 (Future Work) does not mention software sensitivity analysis as an avenue for further investigation.

3. Cost Estimation Omits Training Amortization, Provider Margin, and Infrastructure Redundancy

The assumption or constraint. The idealized dollar cost metric (Section 4.2) computes inference cost as t_idealized × n_accelerator × c_accelerator — the product of idealized runtime, number of GPUs required to fit the model, and the per-second cloud rental price of the reference GPU. This is a bare-metal compute cost that assumes: (1) the model is served with exactly the minimum number of GPUs needed to fit it in memory, with no redundancy for fault tolerance or load balancing; (2) the GPUs are utilized at 100% (no idle time between queries); (3) there is no additional infrastructure cost (networking, storage, API gateway servers, monitoring); and (4) the cost of training the model is not amortized into the per-query price.

The consequence. The paper reports that idealized inference costs are "up to an order of magnitude lower than the charged actual costs" (Section 5.4). This gap — which the paper acknowledges but does not decompose — means that idealized dollar cost cannot be directly compared to API pricing for budget planning purposes. A practitioner comparing two models based on idealized cost per query might conclude that both are affordable for their application, only to discover that the actual API prices are 10× higher. The gap also means that idealized cost understates the true cost of self-hosting a model: if a company decides to deploy an open-source model on their own A100 cluster based on idealized cost estimates, they will encounter additional costs (cluster management, load balancers, redundant capacity for handling traffic spikes, engineering time for deployment and maintenance) that the idealized metric does not capture.

For the carbon accounting use case the paper mentions (Section 4.2: "We can compare the idealized energy cost of running a specific inference query to the energy cost of training a full model end-to-end to better understand the number of inference queries needed to amortize the significant overhead of training models"), the idealized energy metric similarly omits the energy overhead of data center cooling, networking equipment, and storage systems, and assumes 100% GPU utilization. These overheads can be substantial — data center power usage effectiveness (PUE) ratios typically range from 1.1 to 1.6, meaning 10–60% additional energy beyond what the GPUs draw.

What evidence exists in the paper. The "order of magnitude" gap between idealized and charged costs is mentioned qualitatively in Section 5.4 but never quantified with specific numbers. The paper does not report what the actual API per-query prices were for the evaluated models, what the idealized costs were, or what the ratio between them was for each model. Figure 9 (bottom row) shows idealized cost in cents on the x-axis for the efficiency-capability plots, but does not overlay actual API prices for comparison. The gap between idealized and charged cost is therefore an observed-but-unmeasured phenomenon in this paper.

Mitigation status. The paper partially acknowledges the gap: "we note that these reported costs do not incorporate the significant cost of training models, which presumably gets amortized into the cost users pay with black-box APIs" (Section 5.4). However, the paper does not attempt to model or estimate any of the missing cost components — training amortization, provider margin, infrastructure redundancy, or data center overhead. The idealized dollar cost metric is presented as a lower bound on true cost, but the paper does not characterize how tight or loose that bound is, making it difficult for practitioners to use the metric for actual budgeting rather than relative comparison.

4. Temporal Instability of Black-Box API Measurements

The assumption or constraint. All black-box API measurements used for denoised runtime estimation and raw runtime comparison were collected during a specific two-month window: "Unless otherwise noted, all experiments in this paper were run in September or October 2022 with the latest API versions available at the time" (Section 3.5). The denoised runtime parameters in Table 3 (e.g., OpenAI/davinci's g_denoised = 0.030, Anthropic/v4-s3's g_denoised = 0.057) and the provider optimization gap estimates (2.7× for OpenAI, 1.8× for AI21/Grande) are snapshots from this period.

The consequence. Black-box API performance is not static. Providers continuously update their serving infrastructure — deploying new GPU generations, optimizing CUDA kernels, improving batch scheduling, upgrading network interconnects, or changing rate-limiting policies. A model that showed a 2.7× provider optimization gap (idealized vs. denoised runtime) in October 2022 might show a 1.5× gap or a 4× gap in 2024, depending on whether the provider invested further in inference optimization or whether the reference stack (Megatron on A100s) became outdated relative to newer software frameworks and hardware. The empirical efficiency-capability tradeoff landscapes in Figure 9 — and specifically the finding that OpenAI/davinci falls off the Pareto frontier under idealized runtime — are version-locked to the API implementations and model versions available in fall 2022. The paper provides no mechanism for tracking these measurements over time, no versioning scheme for efficiency metrics, and no discussion of how frequently measurements should be updated to remain relevant. A practitioner reading the paper in 2024 cannot know whether OpenAI's API is still 2.7× faster than Megatron+A100 for davinci, or whether Anthropic's API has since been optimized to close its gap, without re-running the entire measurement pipeline.

Additionally, API providers may change the underlying model (e.g., a "davinci" endpoint might be quietly upgraded from one checkpoint to another) or the model's architecture (e.g., switching to a mixture-of-experts variant) without updating the public API name or documentation. Since the idealized runtime estimation depends on the architecture hyperparameters (h, l, n) disclosed in model documentation, an undisclosed architectural change would render the idealized estimate invalid while denoised measurements through the API would shift, creating a discrepancy that the user might not detect.

What evidence exists in the paper. The paper provides strong evidence of within-window temporal variability — Figure 5 shows substantial run-to-run variance in API latency, and Figure 6 shows load-dependent degradation of up to 2× — but provides no evidence about across-window temporal drift. There is no longitudinal measurement comparing API performance in September 2022 to API performance in, say, August 2022 or November 2022. The paper does not report whether any of the profiled APIs changed their behavior during the two-month collection window. The single time-window constraint is stated but not analyzed as a limitation.

Mitigation status. The paper does not address temporal stability. There is no discussion of how often idealized and denoised measurements should be refreshed, no proposal for a versioning or provenance-tracking system for efficiency metrics, and no call for longitudinal benchmarking infrastructure in the future work discussion. The limitation is inherent in any empirical measurement of a moving target, but the paper does not provide guidance on how practitioners should interpret or update the reported numbers as the underlying systems evolve.

5. No Capability Confidence Intervals on the Efficiency-Capability Tradeoff Analysis

The assumption or constraint. The Pareto frontier analysis in Figure 9 treats model accuracy as a point estimate and model efficiency metrics (raw runtime, denoised runtime, idealized runtime, idealized cost) as fixed values. The paper reports average accuracy per model per task but does not report any measure of statistical uncertainty — no standard errors, no confidence intervals, no test set sizes for the four HELM scenarios, and no statistical test for whether a model's position on or off the Pareto frontier is significant.

The consequence. The paper's headline findings about which models are Pareto-optimal depend on the point estimates of accuracy and runtime. If the accuracy estimates have non-trivial sampling variance (because the HELM test sets are finite), a model placed just below the Pareto frontier in Figure 9 could be statistically indistinguishable from a model on the frontier. For example, in the IMDB panel (Figure 9a), if two models differ in accuracy by 1 percentage point but the standard error of that difference is 2 percentage points, their apparent Pareto ordering is not statistically reliable. The paper's claim that "the OpenAI/davinci model appears in the Pareto frontier for each benchmark when using raw runtimes but not the idealized metrics" (Section 5.4) is a qualitative observation about point estimates but is not accompanied by evidence that the frontier membership changes are statistically significant rather than artifacts of estimation noise.

Similarly, the denoised runtime estimates are themselves regression estimates with associated uncertainty — the R^2 values in Table 2 (e.g., 0.917 for AI21/J1-Grande v1, 0.924 for Anthropic/v4-s3) indicate that some models' runtime parameters are estimated with more residual variance than others. This uncertainty propagates into the x-axis positions of models in the efficiency-capability plots, but is not reflected in the visualization (models are shown as points, not error ellipses). A model whose denoised runtime estimate has wide uncertainty could overlap on the x-axis with another model whose estimate is more precise, complicating claims about which model is more efficient.

What evidence exists in the paper. The paper provides test set sizes and accuracy variance nowhere in the main text. The four HELM scenarios (IMDB, MMLU college chemistry, BoolQ, RAFT terms of service) are standard benchmarks with known test set sizes in the literature, but the paper does not report them or discuss their implications for statistical power. The runtime estimation uncertainty is partially characterized: R^2 values for denoised runtime regressions are reported in Table 2 (ranging from 0.917 to 0.997), and idealized runtime regressions are described as having R^2 > 0.999 (Section 3.2). However, these R^2 values describe the goodness-of-fit of the linear model to the profiling data, not the uncertainty in the estimated parameters (prompt_encoding_time, g) that would propagate to per-query runtime estimates. The paper does not report standard errors for the fitted regression coefficients, prediction intervals for query-level runtimes, or how these uncertainties would affect Pareto frontier membership.

Mitigation status. The paper does not address statistical uncertainty in any form. The efficiency-capability analysis is presented as a deterministic point-estimate comparison, with no discussion of confidence intervals, significance testing, or sensitivity of conclusions to estimation noise. Given that the paper's contribution is a measurement methodology, the absence of measurement uncertainty quantification is a notable omission — any measurement comes with error bars, and a measurement methodology paper should characterize the size of those error bars so that users know how much to trust the resulting comparisons.

6. Batch-Size-1 Assumption Does Not Reflect Throughput-Oriented Deployment

The assumption or constraint. All idealized runtime measurements are performed at batch size 1 — a single query processed in isolation. The paper states: "we assume that the number of sequences b (the microbatch size) is 1" (Appendix A.2). This is the appropriate setting for measuring latency — how long a single user waits for a response — but it does not capture throughput — how many queries can be processed per unit time when queries are batched together. The memory-bound nature of autoregressive token generation (discussed in Section 3.1.1) means that batching multiple queries can substantially improve hardware utilization: when b > 1, the weight matrices read from GPU memory are reused across b independent sequences, increasing arithmetic intensity and achievable throughput per query.

The consequence. The idealized runtime metric systematically favors model architectures that perform well at batch size 1 (low single-query latency) but may misrepresent relative efficiency in throughput-oriented deployments where batching is used — which is the dominant operational mode for high-traffic LLM services. A model with a deep-but-narrow architecture (many layers, small hidden size) might have lower single-query latency than a shallow-but-wide model (fewer layers, larger hidden size) at batch size 1 because the deep model's per-layer latency is low. But at larger batch sizes, the wide model's larger matrix multiplications achieve higher GPU utilization (closer to peak throughput), while the deep model's sequential layer dependencies create a latency floor that batching cannot improve. The batch-size-1 idealized metric would rank the deep model as more efficient, while a throughput-oriented metric (queries per second at batch size 16 or 32) might rank the wide model higher. The paper's efficiency-capability tradeoff analysis in Figure 9 thus represents the latency-optimal frontier, not the throughput-optimal frontier, and the paper does not distinguish between these two deployment objectives.

This is particularly relevant for the paper's cost metrics. The idealized dollar cost formula t_idealized × n_accelerator × c_accelerator (Section 4.2) computes the cost of serving one query in isolation, which is not how production systems operate — they batch queries to amortize GPU costs. The true per-query cost in a batched deployment would be lower (sometimes substantially) than the idealized cost, and the reduction factor would depend on the model's batching efficiency. A practitioner using idealized cost for budgeting would overestimate inference costs, potentially by a factor proportional to the achievable batch size for that model on that hardware.

What evidence exists in the paper. The paper provides no batch-size sensitivity analysis. There is no comparison of idealized runtime at b = 1 versus b = 4, 8, 16, no measurement of how per-query runtime changes with batching for different models, and no discussion of how batch-size-dependent efficiency would affect the Pareto frontier. The paper's batch-size-1 assumption is stated implicitly (through the b = 1 substitution in Appendix A.2 and the description of profiling "queries" rather than "batches") but is never explicitly analyzed as a limitation or design choice.

Mitigation status. The paper does not address this limitation. Section 7 (Conclusion) does not discuss throughput-oriented extensions. The paper's stated use case includes helping "model creators and researchers to understand the true inference costs that result from a particular training process and model architecture" (Section 5.4), but "true inference costs" in a production setting depend on batching efficiency, which the batch-size-1 metric does not capture. The limitation is partially inherent in the paper's focus on the black-box API setting — where users submit one query at a time and cannot control batching — but the idealized metric is derived from a controlled profiling setup where batch size can be varied, and the paper does not exploit this capability to provide throughput-aware cost estimates alongside latency-focused ones.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a methodological intervention, not an optimization contribution—it changes how we measure inference efficiency, not how we achieve it. The shift is more diagnostic than paradigmatic: it identifies a measurement confound (provider-specific optimizations and performance noise) that the field has been implicitly ignoring, and provides a standardized measurement protocol that factors those confounds out. This is analogous to the role that standardized benchmarks played in computer vision: not a contribution to model architecture per se, but infrastructure that enables fair comparisons and thereby accelerates architectural progress.

The reframing is deceptively simple but foundational: inference efficiency should be evaluated on a level playing field, not on whatever infrastructure the provider happens to deploy. Before this paper, the LLM community had largely accepted raw API latency or model parameter count as the de facto efficiency metrics—the former because it was what practitioners observe, the latter because it was easy to compute. This paper systematically demonstrates that both are misleading for architectural comparison. The evidence that OpenAI/davinci appears on the Pareto frontier under raw runtime but consistently falls off under idealized runtime (Figure 9, Section 5.4) is not just a curiosity—it is a proof-by-example that raw runtime can produce qualitatively wrong conclusions about which architectures are efficient.

Reconciling prior contradictions. The field contained an unstated tension: practitioners knew that different API providers delivered different latency experiences, but the scaling laws literature (Kaplan et al., 2020; Hoffmann et al., 2022) implicitly treated all models of a given parameter count as equally expensive to serve. The paper demonstrates that these two positions are compatible only if you distinguish between observed latency (which includes provider optimizations and noise) and inherent model efficiency (which does not). The paper shows that provider optimizations can produce a 2.7× speedup (OpenAI/davinci's g_denoised = 0.030 vs. g_idealized = 0.081 in Table 3), which is large enough to flip efficiency rankings entirely. This reconciles: scaling laws are not wrong to use parameter count as a proxy for computational cost at the level of abstraction they need (predicting training loss trends), but they are insufficient for answering deployment-level questions about which specific model to serve. The paper thus carves out a distinct niche for empirical efficiency evaluation alongside theoretical scaling law analysis.

Which research directions become more attractive. The paper makes architecture-aware efficiency benchmarking into a credible research activity. Before this work, benchmarking inference efficiency across providers was something you could do only by comparing raw API latencies (which the paper shows is unreliable) or by counting FLOPs (which the paper shows correlates imperfectly with runtime). The skeleton-model trick—that you can estimate how a model would run on uniform hardware without access to its trained weights—is the key enabling insight. It means that any researcher with access to a GPU cluster can now produce standardized efficiency measurements for any model whose architecture is publicly documented. This lowers the barrier to entry for efficiency-capability studies and opens the possibility of community-maintained efficiency leaderboards analogous to accuracy leaderboards.

Which research directions become less attractive. The paper implicitly argues that further optimization of raw API latency as a research metric is misguided for cross-model comparison. If the goal is to understand whether Architecture A is more efficient than Architecture B, optimizing raw API latency is optimizing a contaminated signal—you might be measuring your CUDA kernel engineering skills rather than the model's inherent efficiency. The paper does not argue against deploying optimizations (clearly, providers should optimize their serving stacks), but it argues against using optimized, provider-specific measurements as the basis for architectural conclusions. This narrows the scope of what raw latency measurements are valid for: they remain appropriate for end-user experience evaluation (the paper explicitly says "Raw latency is still a good metric for end users" in Section 1), but they should not be used for the kind of comparative architecture analysis that scaling laws and model design require.

A new diagnostic for provider infrastructure investment. The paper's inequality relationship t_raw ≥ t_denoised ≥ t_idealized (Section 4.1) provides a quantitative framework for measuring how much optimization each provider has implemented. The 2.7× gap for OpenAI and 1.8× gap for AI21 (Table 3) are not just comparisons—they are measurements of infrastructure investment. This creates a new category of empirical finding in the LLM efficiency literature: you can track how a provider's optimization gap evolves over time, compare optimization investment across providers, and potentially correlate optimization gaps with business outcomes (pricing, market share, user retention). This reframes inference efficiency from a purely technical question into one with economic and strategic dimensions.

The training-inference cost tradeoff gains a measurement tool. The paper notes that "model inference costs might outweigh training costs for certain applications (e.g., ChatGPT)" (Section 1). Before this work, estimating inference costs for black-box models required guesswork about the provider's hardware and software. The idealized cost metric provides a standardized lower bound on inference cost that is independent of provider choices. Combined with training cost estimates (which are increasingly available, as the paper notes in Section 6), this enables the first systematic comparisons of total lifetime cost for different models. The paper does not conduct this analysis itself, but it builds the inference-cost-measurement infrastructure that makes it possible.

Follow-Up Research This Work Enables

1. Sensitivity analysis of Pareto frontier membership to reference software choice. The paper's headline finding—that OpenAI/davinci falls off the Pareto frontier under idealized runtime—is measured on a single reference stack (Megatron on A100 GPUs). Different inference frameworks (FasterTransformer, vLLM, TensorRT-LLM, DeepSpeed-Inference) implement different operator fusion strategies, memory management policies, and parallelism heuristics that may interact differently with different model architectures. A direct follow-up would replicate the Figure 9 efficiency-capability analysis using 2–3 alternative software stacks (all on the same hardware, e.g., A100 GPUs) and test whether OpenAI/davinci's frontier status reversal is robust to software choice, or whether it is partially an artifact of Megatron being suboptimal for OpenAI's specific attention-head configuration (h=12288, n=96) or layer count. The experiment would require profiling all 10 skeleton models on each software stack (estimated 3× additional profiling time) and recomputing the Pareto frontiers per stack. A null result (davinci's frontier status is independent of software choice) would strengthen the paper's claim; a positive result (davinci returns to the frontier under a different framework) would qualify the claim and reveal software-architecture interactions that model creators need to understand.

2. Longitudinal tracking of provider optimization gaps as a time series. The paper's provider optimization gap estimates (Table 3: 2.7× for OpenAI, 1.8× for AI21/Grande, 0× for Anthropic) are snapshots from September–October 2022. Re-running the denoised runtime profiling pipeline on the same models through the same APIs at quarterly or semi-annual intervals would produce a time series of optimization gaps per provider. This would reveal: (a) whether providers are continuously investing in inference optimization (widening gaps over time) or whether they stabilize after initial deployment, (b) whether the reference stack (Megatron+A100) is catching up as the open-source ecosystem improves, narrowing gaps organically, and (c) whether changes in optimization gap correlate with observable API changes (pricing adjustments, model version updates, latency SLO announcements). The experiment requires only API access (no skeleton models or GPUs for the denoised component) and could be partially automated. A practitioner-oriented extension would publish a continuously updated dashboard of provider optimization gaps.

3. Extending the parametric form to throughput-oriented (batched) inference metrics. The paper's idealized runtime metric measures single-query latency at batch size 1, which captures the user experience for interactive applications but not the throughput achievable in batch-processing or high-traffic serving scenarios. A follow-up would extend the parametric form to include batch size b as an additional independent variable: t(p, o, b) = prompt_encoding_time(p, b) + (o - 1) · g(b). The key question is whether the linearity-in-o and piecewise-linearity-in-p properties hold at b > 1, and whether g(b) (per-token generation time) decreases with b at different rates for different model architectures (shallow-wide vs. deep-narrow). The experiment would profile the same 10 models at batch sizes b ∈ {1, 2, 4, 8, 16} and produce both latency-optimal and throughput-optimal Pareto frontiers. This would test whether the paper's conclusion that certain models (like BLOOM) are architecture-level efficiency winners holds under both latency and throughput objectives, or whether different models dominate under different serving regimes.

4. Architecture inference for models with undisclosed hyperparameters. The paper's Achilles' heel is its dependence on disclosed architecture details (h, l, n from Table 1). A follow-up could develop methods to infer these hyperparameters from observable API behavior, extending the methodology's coverage to models like GPT-4 and ChatGPT that are currently excluded. Potential approaches: (a) measuring how runtime scales with prompt size and output token count to back out h and l from the FLOP formulas (since the slope of prompt_encoding_time(p) depends on 24 · h² · l), (b) using the observation that different attention-head configurations produce subtly different KV-cache memory access patterns that might be detectable in timing side channels, or (c) exploiting the fact that the per-output-token generation time g is approximately 24 · h² · l / throughput_og, and throughput_og is largely determined by memory bandwidth and weight matrix dimensions. A validation experiment would apply the inference method to models with known architectures (treating them as black boxes) and compare inferred hyperparameters to ground truth from Table 1, then apply the method to models with undisclosed architectures and report the resulting idealized runtime estimates with appropriate uncertainty bounds.

5. Decomposing the provider optimization gap into hardware, software, and system-level components. The paper reports a 2.7× gap between OpenAI's denoised and idealized runtime (Table 3) but does not decompose it into: (a) hardware advantages (e.g., OpenAI using H100 GPUs rather than A100s), (b) software advantages (custom CUDA kernels, kernel fusion, optimized attention implementations like FlashAttention), and (c) system-level advantages (query caching, continuous batching, speculative decoding). A follow-up would attempt to isolate these components by: for (a), profiling the idealized metric on multiple hardware generations (A100, H100, and if available, whatever inference-specialized hardware exists) and testing which hardware choice brings idealized runtime closest to denoised; for (b), profiling with different software stacks that incorporate specific known optimizations (e.g., Megatron with and without FlashAttention enabled) and measuring the incremental speedup; for (c), constructing synthetic workloads that exercise specific system-level optimizations (e.g., repeated identical prompts to detect caching) through the API. This decomposition would tell model creators where to invest: in better hardware procurement, in custom kernel development, or in system-level batching and caching infrastructure.

6. Cross-domain validation: code generation and multimodal models. The paper's parametric form is derived specifically for autoregressive text generation with Transformer models. A natural extension tests whether the same form holds for code generation (e.g., CodeLlama, StarCoder models served through APIs), where output tokens follow a different distribution (more structured, more repetitive patterns that might enable different caching strategies) and where prompt sizes can be much larger (entire codebases as context). A second extension tests the methodology on autoregressive multimodal models (e.g., GPT-4V, Gemini) where the prompt includes both text tokens and image tokens, and the image encoding phase introduces a new computational component that may not follow the same piecewise linear form. The experimental design: profile a set of code-generation models on the reference stack using the same (p, o) grid methodology, fit the parametric form, and test whether R² remains high (as it did for text models in Table 2) or whether code-specific phenomena (e.g., variable-length token representations, structural repetition) introduce non-linearities that break the simple form. For multimodal models, the experiment would need to control for image token count as an additional dimension in the profiling grid and test whether the prompt-encoding piecewise linearity survives when the "prompt" includes both discrete text tokens and continuous image features.

Practical Applications and Downstream Use Cases

1. Cloud cost optimization for self-hosted open-source models. Organizations deploying open-source LLMs (e.g., LLaMA, Mistral, Falcon) on their own GPU clusters face a hardware procurement and configuration problem: which GPU generation (A100 vs. H100 vs. V100), how many GPUs per model instance, and which software stack minimizes cost per query? The paper's Figure 8 demonstrates the methodology directly: comparing A100 to V100 revealed that V100 GPUs are "both slower and more expensive" because their 32GB memory limit forces double the GPU count for large models, causing the per-query cost to increase despite the lower per-GPU-hour price. A practitioner evaluating, say, a 70B-parameter LLaMA variant could replicate this analysis: profile the skeleton model on A100-80GB vs. H100-80GB (where H100 has higher per-hour cost but 3× the memory bandwidth and potentially lower latency), compute idealized cost per query on both, and determine the cost-optimal hardware configuration. The paper's methodology provides the recipe—the practitioner needs only the model architecture (h, l, n) and access to the candidate GPU types for profiling.

2. Vendor-agnostic model selection for API consumers. Companies building LLM-powered products that compare multiple API providers (OpenAI, Anthropic, Cohere, Google, Together) need to select models based on capabilities and inherent efficiency—not just pricing, which can change. The paper's idealized runtime metric enables a separation of concerns: capability evaluation (does the model perform well on my task?) from efficiency evaluation (is the model architecture inherently efficient on standardized hardware?). A practitioner could: (1) evaluate all candidate models on their task-specific accuracy using HELM or custom evaluation, (2) estimate idealized runtime and idealized cost for each model using the paper's methodology (requiring only disclosed architectures and a one-time profiling investment), and (3) select the model on the capability-efficiency Pareto frontier for their specific accuracy-vs-cost tradeoff. This decouples model selection from provider pricing strategy—if a provider later raises prices, the efficiency assessment remains valid, and the practitioner knows whether an alternative model architecture can serve as a cost-comparable substitute.

3. Training budget allocation: pretraining vs. inference cost tradeoff. The paper notes that "model inference costs might outweigh training costs for certain applications" but does not perform the training-inference breakeven analysis itself. A downstream application would use the idealized cost metric to compute the lifetime cost per query for a model family, incorporating: (a) amortized training cost (increasingly available from model cards; e.g., LLaMA 2's training FLOPs are published), (b) per-query inference cost estimated via the idealized dollar cost formula (Section 4.2), and (c) projected query volume over the model's deployment lifetime. For a given application with known query volume and latency requirements, the practitioner could determine whether investing a larger training budget in a larger model recoups its cost through higher accuracy (and thus fewer failed queries requiring retry or human review) or whether the inference cost of the larger model dominates any accuracy benefit. The paper's methodology provides the inference-cost half of this equation in a standardized, provider-independent form, making it possible to compare across model families rather than being locked into one provider's pricing.

4. Regulatory reporting and carbon accounting for LLM deployments. The paper extends the idealized metric to energy cost (Section 4.2: t_idealized × n_accelerator × p_accelerator), directly connecting to the carbon accounting literature the paper cites (Henderson et al., 2020; Patterson et al., 2021). As jurisdictions begin requiring AI systems to report energy consumption (the EU AI Act's high-risk system provisions, for instance), the idealized energy metric provides a standardized, auditable method for estimating inference energy that does not depend on the provider's proprietary infrastructure details or transient load conditions. A company deploying an LLM-powered service could report: "Our model, if served on A100 GPUs with Megatron, consumes approximately X watt-hours per 1000 queries; actual consumption on our optimized serving stack is Y watt-hours (denoised measurement), representing a Z% improvement over the reference." This separates the model's inherent energy efficiency (idealized) from the deployment's realized efficiency (denoised), enabling regulators and the public to distinguish between architectural choices and infrastructure investments when evaluating environmental impact.

When to Prefer This Method

The paper explicitly positions idealized runtime against raw API latency and FLOPs/proxy metrics (model size, parameter count) as alternatives for efficiency evaluation. The decision rule it implies is:

Prefer idealized runtime (or idealized cost) when:

You are comparing models from different providers and need to understand inherent architectural efficiency (e.g., "Is a 52B dense model architecturally more efficient than a 100B sparse model?")—Figure 9 demonstrates that raw runtime and FLOPs produce different answers to this question.
You are making long-term architectural decisions (e.g., choosing a model architecture for a new training run) where provider-specific optimizations are not transferable and you need to know which architecture will be efficient on your own serving infrastructure, not on the provider's.
You have access to model architecture details (h, l, n) and can afford the one-time profiling cost (<2 hours per model per hardware/software configuration, per Section 5.3).
You need deployment cost estimates that account for the number and type of accelerators required (idealized dollar cost in Section 4.2)—this is unavailable from FLOPs or raw API latency.
You need to compare hardware options for serving a model (Figure 8 demonstrates comparing A100 vs. V100 GPUs)—this is a capability unique to the idealized metric.

Prefer raw API runtime (or denoised runtime) when:

You are an end user making a deployment decision based on what latency you will actually experience (the paper explicitly says raw latency "is still a good metric for end users" in Section 1).
The models you are comparing come from a single provider with consistent infrastructure (e.g., choosing between GPT-3.5-turbo and GPT-4 from OpenAI), where provider-specific optimizations are shared and the comparison already controls for infrastructure.
You need to monitor API performance in production for SLO compliance, where the observed latency (including noise and contention) is exactly what matters.
You cannot obtain the model's architecture details—the idealized methodology is inapplicable (Section 5.1), and raw/denoised latency is your only option.

Prefer FLOPs when:

You are doing theoretical scaling law analysis where runtime is not the target variable and you need a hardware-independent unit of computation (FLOPs serves this purpose, albeit imperfectly, as the paper acknowledges in Section 6).