Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs

ArXiv: 2510.18245

🎯 Pitch

For a fixed parameter and training token budget, simply tweaking hidden size and MLP-to-attention ratio yields models that are both 2.1% more accurate and 42% faster at inference than LLaMA-3.2. The key is a conditional scaling law that predicts how these architectural choices shift the loss curve, enabling a direct search for Pareto-optimal designs without guesswork.

---

1. Executive Summary

This paper studies how key architectural factors—hidden size, the mlp-to-attention ratio, and grouped-query attention (GQA)—jointly influence both the accuracy and inference efficiency of decoder-only transformers, using over 200 trained models spanning 80M to 3B parameters on 8B to 100B training tokens. The core contribution is a conditional scaling law that extends the Chinchilla framework by calibrating the training loss of architectural variants relative to an optimal baseline via separable multiplicative or additive terms in hidden size and the mlp-to-attention ratio, paired with a search framework (Algorithm 1) that maximizes inference throughput subject to a loss constraint, followed by a local GQA enumeration. Under identical training budgets, the resulting architectures—named Panda and Surefire—achieve up to 2.1% higher accuracy and 42% greater inference throughput compared to LLaMA-3.2 at the 1B and 3B scales, establishing that architectural optimization under a fixed parameter and token budget can yield simultaneous gains in both accuracy and efficiency, provided the number of layers is held fixed and the target model size is within approximately 3× of the fitting-data scale.

2. Context and Motivation

The Core Problem: We Optimize Training, Not Inference

The fundamental question driving this paper is deceptively direct: given a fixed budget of parameters and training tokens, what architecture minimizes pretraining loss while maximizing inference throughput? This matters because the conventional wisdom in language model development says you choose a parameter count and a training data volume, then design the architecture around those constraints. But "design the architecture" hides a multitude of interdependent choices—hidden size, number of attention heads, MLP intermediate size, the grouping pattern for key-value heads—each of which pulls accuracy and inference efficiency in potentially opposite directions. The paper argues that the prevailing approach to these choices is ad hoc and that the field lacks a principled framework for making them.

This gap has significant practical consequences that the authors highlight in Section 1:

• Inference is the dominant cost in deployment. A model trained once may be served millions or billions of times. Inference FLOPs, memory bandwidth for the KV cache, and the consequent throughput (tokens generated per second) determine the real-world cost and latency of serving LLMs. Yet the scaling laws that guide model development—most notably Hoffmann et al. (2022)'s Chinchilla framework—focus exclusively on the training budget. They tell you how many parameters and training tokens to use given a FLOPs constraint, but say nothing about what happens when those parameters are deployed.
• Architectural heterogeneity is widespread and unexplained. As the paper observes, open-weight models at similar parameter counts adopt markedly different architectures. LLaMA-3.2-1B uses hidden size 2048 with an mlp-to-attention ratio of 4.8 and GQA=4; Qwen3-0.6B uses hidden size 1024 with a ratio of 1.5 and GQA=2. These are not minor tweaks—they represent fundamentally different allocations of the parameter budget between attention and MLP, and different tradeoffs between compute and memory during inference. The paper asks: which of these choices is right, and why?
• The rise of reasoning systems amplifies inference costs. As the authors note, the growing use of LLMs for chain-of-thought reasoning, code generation, and multi-step problem solving (Snell et al., 2024; Brown et al., 2024; Luo et al., 2024) means models generate far more tokens per query during deployment than during a typical single-pass evaluation. In such regimes, inference efficiency becomes not merely a cost concern but a capability enabler—faster inference means more reasoning steps within a given latency budget.

The Specific Gap: Architecture Is Absent from Scaling Laws

The Chinchilla scaling law (Hoffmann et al., 2022) takes the form:

$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$

where $L$ is the loss, $N$ is the number of non-embedding parameters, and $D$ is the number of training tokens. Critically, $N$ is treated as a scalar—a single number that says nothing about how those parameters are organized. Two models with $N = 1\text{B}$ could have completely different hidden sizes, different MLP-to-attention ratios, different numbers of layers and heads, and yet are, in the Chinchilla framework, indistinguishable. The Chinchilla law predicts the same loss for both, which the paper demonstrates is empirically false.

This omission is not an oversight in the original scaling law literature—it is a deliberate simplification that makes the laws tractable. The Chinchilla optimization problem is:

$\arg\min_{N, D} L(N, D) \quad \text{s.t.} \quad \text{FLOPs}(N, D) = C$

where the constraint involves only $N$ and $D$. Introducing architectural variables would turn this into a higher-dimensional optimization that is harder to fit and harder to solve. But as the paper argues, this simplification has a cost: it leaves practitioners without guidance on what architecture to use for the $N$ and $D$ that the scaling law prescribes.

Prior Work and Where It Falls Short

The paper positions itself against three lines of prior work, each of which addresses part of the problem but leaves critical gaps.

1. Inference-aware scaling laws (Sardana et al., 2023). The most directly relevant prior work is Sardana et al. (2023), which proposed incorporating inference FLOPs into the scaling law framework by modeling total FLOPs as the sum of training FLOPs ($6ND$) and inference FLOPs (proportional to the number of tokens the model generates over its entire deployment lifetime). The paper identifies a fundamental limitation of this approach:

"their formulation requires estimating the total number of tokens generated over a model's entire lifespan. Because inference is performed repeatedly during deployment, this assumption renders the proposed scaling law impractical for real-world use."

In other words, Sardana et al. need you to know in advance how many tokens the model will ever generate—a number that depends on usage patterns, product decisions, and the model's own success or failure in the market. The paper's approach sidesteps this by optimizing architecture for a fixed $N$ and $D$ without requiring a lifetime inference token estimate. It treats inference efficiency as a separate objective (maximize throughput) subject to an accuracy constraint, rather than collapsing everything into a single FLOPs budget.

2. Aspect-ratio-focused architecture scaling (Bian et al., 2025). This prior work by two of the present paper's authors extended Chinchilla scaling laws to incorporate architecture, but only through a single variable: the aspect ratio, defined as hidden size divided by the number of layers. The present paper identifies three critical limitations of this approach:

• Aspect ratio alone is insufficient. As shown in the paper's Figure 2, models with the same number of layers can have dramatically different inference throughput depending on their hidden size, mlp-to-attention ratio, and GQA. For instance, Qwen2.5-1.5B achieves higher throughput than Qwen3-0.6B despite being larger, because its architecture makes more efficient use of hardware. Aspect ratio captures none of this variation.
• Layer count is a sensitive architectural degree of freedom. Petty et al. (2023) showed that reducing the number of layers impairs compositional generalization after fine-tuning, making depth a risky variable to optimize freely. The present paper therefore fixes the number of layers and focuses on other architectural factors, which the prior work did not do.
• No framework for GQA or the mlp-to-attention ratio. The prior work's single-variable formulation cannot accommodate the discrete choice of GQA grouping or the allocation of parameters between attention and MLP—both of which the present paper shows have substantial effects on inference efficiency.

3. Practice-driven architectural choices in open-weight models. The paper's implicit foil is the current state of practice, where model architectures are designed through a combination of intuition, precedent, and limited ablation. Table 7 and Table 8 in Appendix H illustrate this: LLaMA-3.2-1B uses an mlp-to-attention ratio of 4.8 and GQA=4; OLMo-2-1B uses a ratio of 3 and GQA=1; Qwen3-4B uses GQA=4 while Qwen3-1.7B uses GQA=2. This variation is not noise—it reflects genuine uncertainty about what works. As the paper demonstrates through its controlled ablations in Figures 4 and 5, both the hidden size and the mlp-to-attention ratio exhibit U-shaped relationships with training loss, meaning there exists a genuine interior optimum. Models that deviate from this optimum—in either direction—pay a penalty in accuracy that could be avoided with a principled allocation.

How This Paper Positions Itself

The paper frames its contribution as filling the gap between scaling laws (which ignore architecture) and practice (which varies architecture without scaling-law guidance). The conceptual move is to condition the scaling law on architecture rather than trying to absorb architecture into a single unified law.

Specifically, the paper proposes a two-step approach (Section 3.3):

1. Obtain a reference loss $L_{\text{opt}}(N, D)$ from the standard Chinchilla framework for given $N$ and $D$. This represents the best achievable loss if architecture were not a constraint—essentially, the loss floor at that scale.
2. Calibrate architectural variants around this reference using multiplicative or additive correction terms that are functions of $d_{\text{model}}/\sqrt{N}$ and $r_{\text{mlp/attn}}$, with learnable parameters shared across all $N$ and $D$.

This conditional approach has several advantages. First, it preserves the Chinchilla framework as a reference point, making the extension incremental rather than revolutionary. Second, it decouples the difficulty of fitting the base scaling law (which requires many models at different $N$ and $D$) from the difficulty of fitting the architectural correction (which can be done with a focused sweep of architectural variants at a few $N$ levels). Third, it naturally accommodates the separable effects of $d_{\text{model}}$ and $r_{\text{mlp/attn}}$—the paper finds empirically that these effects are largely independent, which the multiplicative and additive forms capture.

The search framework (Algorithm 1) then operationalizes this conditional law by solving:

$\arg\max_{P} \text{IN}(P) \quad \text{s.t.} \quad L(P \mid N, D) \leq L_t$

where $\text{IN}(P)$ is the inference efficiency of architecture $P$ (measured empirically via throughput benchmarks on specific hardware) and $L_t \geq L_{\text{opt}}$ is a maximum allowable loss. This is, in effect, a constrained optimization where accuracy is the constraint and inference efficiency is the objective—a deliberate inversion of the typical priority where accuracy is maximized subject to a compute budget. The paper argues this framing is more practical for deployment, where a minimum accuracy bar must be met and additional efficiency gains are always valuable.

The paper also explicitly limits its scope in ways that clarify its position: (1) it fixes the number of layers, motivated by the sensitivity of depth to fine-tuning performance (Petty et al., 2023); (2) it restricts attention to dense models, not mixture-of-experts, though it provides preliminary MoE inference efficiency results in Appendix M; (3) it focuses on pretraining only, leaving post-training (instruction tuning, RLHF) as future work. These choices reflect a deliberate decision to make the problem tractable and the results actionable at the scales studied (up to 3B parameters and 100B training tokens).

3. Technical Approach

3.1 Reader Orientation

This paper builds a predictive framework—a conditional scaling law paired with a search procedure—that, given a fixed budget of non-embedding parameters and training tokens, tells you which combination of hidden size, MLP-to-attention ratio, and GQA grouping will simultaneously minimize pretraining loss (accuracy) and maximize inference throughput (efficiency). The core problem is architectural choice under constraints, and the solution takes the shape of a two-step calibration: first, fit a standard Chinchilla scaling law to establish a loss floor for the parameter and token budget; second, learn correction factors that adjust this floor up or down depending on the architectural deviation from an empirically-observed optimum, where the correction factors are simple, separable functions in the normalized hidden size and MLP-to-attention ratio that exhibit consistent U-shaped behavior across model scales.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, organized into a fitting pipeline and a deployment pipeline:

1. Architecture Sampling Engine: For each target non-embedding parameter count $N_{\text{non-embed}}$ (80M, 145M, 297M, 1B, 3B), this component generates candidate architectures by systematically varying the hidden size $d_{\text{model}}$ and the intermediate MLP size $f_{\text{size}}$ (which jointly determine the mlp-to-attention ratio $r_{\text{mlp/attn}}$) while holding the number of layers $n_{\text{layers}}$ and GQA fixed at a default value (GQA=4). Changing $d_{\text{model}}$ and $f_{\text{size}}$ in opposite directions keeps the total parameter count approximately constant—for a larger hidden size, the number of attention heads and/or the intermediate size is reduced to compensate. The output is a set of over 200 concrete model configurations (detailed in Appendix D, Table 4) that span the architectural design space.

2. Training and Loss Measurement: Each candidate architecture is pretrained on 100 times its non-embedding parameters in tokens ($100 \times N_{\text{non-emb}}$), sampled from the Dolma-v1.7 dataset with probabilities proportional to each source's contribution. Training uses a LLaMA-3.2-style decoder-only transformer with standard hyperparameters (AdamW, cosine schedule, sequence length 2048). The result is a single training loss value per architecture, along with zero-shot downstream task accuracies on nine benchmarks for the larger-scale validation models (1B and 3B).

3. Reference Loss (Chinchilla) Fitting: For each $(N, D)$ pair, the component identifies the minimum training loss $L_{\text{opt}}(N, D)$ achievable among the architectural variants at that scale. This is not a fitted Chinchilla law in the traditional sense—the paper "empirically searched over architecture variants to find the optimal loss" for $N_{\text{non-embed}} < 1\text{B}$ (Section 4, Fitting Scaling Laws). For larger scales, the Chinchilla functional form is fitted to these empirical minima. The output is a scalar reference loss that represents the best the model family can do at that scale, independent of architecture.

4. Conditional Scaling Law (Calibration): This is the intellectual core. The component learns a multiplicative or additive correction function that takes the architectural descriptors $d_{\text{model}}/\sqrt{N}$ and $r_{\text{mlp/attn}}$ as input and outputs a predicted loss $L(d_{\text{model}}/\sqrt{N}, r_{\text{mlp/attn}} \mid N, D)$ relative to the reference loss. The correction function has learnable parameters shared across all $N$ and $D$, enabling extrapolation: fit on 80M, 145M, 297M data, predict for 1B and 3B. The fitting uses the Levenberg-Marquardt least-squares algorithm, and the paper ablates progressive fitting strategies (Task 1: 80M → 145M; Task 2: 80M+145M → 297M; Task 3: 80M+145M+297M → 1B) to validate predictive accuracy.

5. Constrained Search for Inference-Efficient Architectures: Given a target parameter count $N$, token budget $D$, and maximum allowable loss $L_t$ (e.g., the loss of the LLaMA-3.2 baseline at that scale), this component solves:

   $\arg\max_{P} \text{IN}(P) \quad \text{s.t.} \quad L(P \mid N, D) \leq L_t$

   Since $\text{IN}(P)$ (inference throughput) depends on hardware and serving framework details that are difficult to express analytically, the component enumerates over feasible configurations $P_i$ that satisfy the loss constraint (as predicted by the conditional scaling law) and selects Pareto-optimal points. It then performs a local search over GQA values—since GQA must be a prime factor of the number of attention heads, the search space is small—to maximize inference efficiency. The output is a concrete architecture specification: $d_{\text{model}}$, $f_{\text{size}}$, $n_{\text{heads}}$, and GQA.

Information flows sequentially: sampling → training → reference fitting → calibration fitting → constrained optimization. The calibration step is the inductive leap that allows small-scale measurements to guide large-scale architectural decisions without retraining the full design space at the target scale.

3.3 Roadmap for the Deep Dive

• First, the architectural degrees of freedom and how they are parameterized: what we vary ($d_{\text{model}}$, $r_{\text{mlp/attn}}$, GQA), what we hold fixed ($n_{\text{layers}}$, per-head dimension, total $N_{\text{non-embed}}$), and why these choices isolate the effects that matter for inference efficiency while keeping the problem tractable.
• Second, the empirical U-shaped relationships: why loss curves as functions of $d_{\text{model}}/\sqrt{N}$ and $r_{\text{mlp/attn}}$ consistently exhibit minima at intermediate values, what structural properties of transformers produce this shape, and why this motivates the specific functional forms ($c_0 + c_1 \log x + c_2/x$) used in the calibration terms.
• Third, the conditional scaling law formalism: the two-step reference-and-calibration framework, the multiplicative and additive variants, the learnable parameters, and why separability of $d_{\text{model}}$ and $r_{\text{mlp/attn}}$ effects is assumed and empirically validated.
• Fourth, the fitting and validation protocol: the progressive fitting strategy (Tasks 1-3), the Levenberg-Marquardt least-squares algorithm, the MSE and Spearman correlation metrics, and the ablation studies on outliers and calibration forms that justify the design choices.
• Fifth, the constrained optimization and search procedure: how inference efficiency is measured (vLLM on A100, SGLang on H200), how the loss constraint $L_t$ is set (matching LLaMA-3.2 baselines), what "Pareto-optimal" means in this context, and how the local GQA search operates given the discrete constraint that GQA must divide the number of attention heads.
• Sixth, the inference FLOPs analysis: why larger $d_{\text{model}}$ and larger $r_{\text{mlp/attn}}$ reduce total inference FLOPs under a fixed non-embedding parameter budget, connecting the empirical throughput gains to a first-principles compute model (Appendix K).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical methodology paper whose core idea is that architectural factors not captured by parameter count alone have predictable, separable effects on training loss, and that a conditional extension of the Chinchilla scaling law can learn these effects from small-scale sweeps and extrapolate them to guide large-scale architectural decisions.

---

Architectural Degrees of Freedom and Fixed Constraints

The paper studies decoder-only transformers in the LLaMA family, where the architecture is defined by a stack of $n_{\text{layers}}$ identical decoder blocks. Each block contains a multi-head self-attention sublayer and a feed-forward (MLP) sublayer. The total non-embedding parameters for such a model are approximately:

$N_{\text{non-embed}} \approx n_{\text{layers}} \times \left(4 d_{\text{model}}^2 + 2 d_{\text{model}} d_{\text{kv}} + 3 d_{\text{model}} f_{\text{size}}\right)$

where $d_{\text{model}}$ is the hidden size (the dimensionality of the residual stream), $f_{\text{size}}$ is the intermediate size of the MLP (typically expanded relative to $d_{\text{model}}$ by a factor of 2–4), $d_{\text{kv}}$ is the total dimensionality of the key and value projections, and the constants reflect: 4 attention projection matrices (query, key, value, output) each of size approximately $d_{\text{model}} \times d_{\text{model}}$, plus 3 MLP matrices (up-projection, gate-projection, down-projection) each of size $d_{\text{model}} \times f_{\text{size}}$ (for SwiGLU-style activations).

What this equation means for architecture design: Given a fixed $N_{\text{non-embed}}$ and fixed $n_{\text{layers}}$, increasing $d_{\text{model}}$ consumes more of the parameter budget in the attention projections (since attention parameters scale as $d_{\text{model}}^2$), forcing a reduction in either $f_{\text{size}}$ (the MLP expansion) or $d_{\text{kv}}$ (the KV projection dimension) to stay within the parameter budget. The paper manipulates these tradeoffs systematically.

What the paper fixes and why:

• Number of layers $n_{\text{layers}}$: Held fixed at 12 for models up to 297M, 16 for 1B models, and 28 for 3B models. This is motivated by the finding that "cutting layers tends to impair the model's generalization after fine-tuning" (Section 1, citing Petty et al., 2023). Changing $n_{\text{layers}}$ under a fixed parameter budget also changes the aspect ratio $d_{\text{model}}/n_{\text{layers}}$, which the prior work Bian et al. (2025) showed is predictive of loss but insufficient for capturing inference efficiency. By fixing $n_{\text{layers}}$, the paper isolates the effects of the internal structure of each block (how attention and MLP are balanced) from the effects of depth.

• Per-head dimension $d_{\text{head}}$: Fixed to 64 for models with $N_{\text{non-embed}} \leq 1\text{B}$ and to 128 for models with $N_{\text{non-embed}} \geq 3\text{B}$ (Section 3.1). This is a standard practice in open-weight models (e.g., GPT-3, LLaMA) that keeps the per-head computation roughly constant across scales. Fixing $d_{\text{head}}$ means that changing the hidden size $d_{\text{model}}$ changes the number of attention heads $n_{\text{heads}} = d_{\text{model}} / d_{\text{head}}$, which in turn affects the degree of parallelism in attention and the total dimension of the KV cache.

• Default GQA of 4 during the scaling-law fitting phase: GQA is held at 4 while sweeping $d_{\text{model}}$ and $r_{\text{mlp/attn}}$, and then optimized separately via local search (Algorithm 1, line 3). This design choice reflects the paper's finding that "unlike hidden size and the mlp-to-attention ratio, GQA does not exhibit a consistent continuous relationship with loss" (Section 3.4)—GQA affects loss in a highly fluctuating, non-monotonic way (Figure 24, Appendix I), making it unsuitable for inclusion in a continuous parametric scaling law. The local search over GQA is tractable because once $d_{\text{model}}$, $n_{\text{heads}}$, and $r_{\text{mlp/attn}}$ are fixed, the feasible GQA values are constrained: "GQA must be a prime factor of the number of attention heads" (Section 3.4). So the search enumerates these factors and picks the one that maximizes throughput without violating the loss constraint.

What the paper varies:

• Hidden size $d_{\text{model}}$: The paper varies $d_{\text{model}}$ while holding $N_{\text{non-embed}}$ and $n_{\text{layers}}$ constant. To keep the parameter count fixed, the number of attention heads $n_{\text{heads}}$ and the intermediate size $f_{\text{size}}$ are adjusted inversely. For example, in the 80M model grid (Table 4, Appendix D), $d_{\text{model}}$ ranges from 384 to 2048. A larger $d_{\text{model}}$ means more parameters allocated to attention projections ($d_{\text{model}}^2$), so the MLP must shrink. The paper normalizes $d_{\text{model}}$ by $\sqrt{N_{\text{non-embed}}}$ to make the quantity scale-invariant—since attention parameters scale as $d_{\text{model}}^2$ (assuming square projection matrices), $d_{\text{model}}/\sqrt{N}$ is approximately constant at the architectural optimum across model scales.

• MLP-to-attention ratio $r_{\text{mlp/attn}}$: This ratio quantifies how the parameter budget is split between the MLP and attention sublayers. The paper defines it implicitly via the ratio of MLP parameters to attention parameters. In practice, given fixed $d_{\text{model}}$ and $n_{\text{heads}}$, changing $f_{\text{size}}$ changes the MLP parameter count ($3 d_{\text{model}} f_{\text{size}}$) and thus the ratio. The attention parameter count (with GQA) is approximately $2 d_{\text{model}} d_q + 2 d_{\text{model}} d_{\text{kv}}$ where $d_q = n_{\text{heads}} \times d_{\text{head}}$ and $d_{\text{kv}} = n_{\text{kv-heads}} \times d_{\text{head}}$. Running through the 80M variants in Table 4: v1 has $d_{\text{model}}=768$, $f_{\text{size}}=2048$, giving $r_{\text{mlp/attn}}=2.40$; v6 has $d_{\text{model}}=768$, $f_{\text{size}}=768$, giving $r_{\text{mlp/attn}}=0.36$. The range spans from 0.10 (almost no MLP) to 12.6 (MLP-dominant).

• GQA (during the final search phase): Grouped-query attention reduces the number of key-value heads relative to query heads, shrinking the KV cache and reducing memory I/O during inference. For instance, GQA=4 means there are 4 times fewer KV heads than query heads. Since $n_{\text{heads}}$ is determined by $d_{\text{model}}/d_{\text{head}}$, feasible GQA values must be divisors of $n_{\text{heads}}$. The paper explores GQA values up to the point where training loss degrades below the baseline (early stopping in the local search).

Why this parameterization isolates inference-relevant effects: The inference cost of a transformer is dominated by two factors: (1) compute—the FLOPs required for the attention projections and MLP layers, and (2) memory I/O—the size of the KV cache, which scales with $n_{\text{layers}} \times d_{\text{kv}} \times \text{sequence length}$. Increasing $d_{\text{model}}$ (at fixed $N_{\text{non-embed}}$) reduces $n_{\text{heads}}$ (since $d_{\text{head}}$ is fixed), which in turn reduces $d_{\text{kv}}$ if GQA is held constant, shrinking the KV cache and reducing both FLOPs and I/O. Increasing $r_{\text{mlp/attn}}$ shifts parameters from attention (which has a quadratic-in-$d_{\text{model}}$ cost and a KV cache footprint) to the MLP (which is purely compute-bound with no persistent state), further improving throughput. The paper's inference FLOPs analysis (Appendix K) formalizes this: the total non-embedding inference FLOPs are:

$\text{Total-FLOPs} = 2P_{\text{non-emb}} + 2 n_{\text{layers}} T d_q$

where $T$ is the sequence length prior to token generation (the prompt length), $d_q = A d_h$ is the query projection dimension ($A$ = number of query heads, $d_h$ = per-head dimension), and $P_{\text{non-emb}}$ is the total non-embedding parameter count. The term $2 n_{\text{layers}} T d_q$ represents the cost of computing query-key dot products during prefill. Increasing $r_{\text{mlp/attn}}$ reduces $d_q$ at fixed total parameters, directly shrinking this term. Increasing $d_{\text{model}}$ has a more complex effect: it increases the quadratic attention projection cost ($2P_{\text{non-emb}}$ grows, but the split between attention and MLP parameters matters) while potentially reducing $d_q$ if $n_{\text{heads}}$ decreases.

---

The Empirical U-Shaped Relationships and Their Functional Form

The central empirical observation that enables the conditional scaling law is that both $d_{\text{model}}/\sqrt{N}$ and $r_{\text{mlp/attn}}$ exhibit U-shaped relationships with training loss when other architectural factors are held fixed. This is not an assumption—it is documented in Figures 4 and 5 across three model scales (80M, 145M, 297M).

The hidden-size U-curve (Figure 4). For each panel in Figure 4, the x-axis is $d_{\text{model}}/\sqrt{N_{\text{non-embed}}}$, the y-axis is training loss, and different curves correspond to different fixed values of $r_{\text{mlp/attn}}$. The curves consistently dip to a minimum at intermediate values of $d_{\text{model}}/\sqrt{N}$, roughly around 0.06-0.10 depending on the ratio, and rise on both sides. The paper explains the two regimes:

• Left side (too small $d_{\text{model}}$): When the hidden size is too small relative to the parameter budget, the model allocates a large number of attention heads (since $n_{\text{heads}} = d_{\text{model}}/d_{\text{head}}$, and $d_{\text{head}}$ is fixed at 64). Each head processes a smaller slice of the hidden dimension, reducing the representational capacity per head. The paper notes this "can degrade accuracy—a phenomenon consistently observed in prior analyses of transformer capacity and head allocation" (Section 3.3, citing Kaplan et al., 2020; Hoffmann et al., 2022). Essentially, the attention mechanism becomes fragmented: many narrow heads may not individually capture meaningful patterns, and the information bottleneck at each head limits what the attention sublayer can learn.

• Right side (too large $d_{\text{model}}$): When the hidden size is too large, the number of attention heads $n_{\text{heads}}$ becomes very small (e.g., at $d_{\text{model}}=2048$ in the 80M models, $n_{\text{heads}}=2048/64=32$, but the model must still allocate enough parameters to reach the target $N_{\text{non-embed}}$, forcing a drastic reduction in $f_{\text{size}}$). The MLP, which provides non-linear transformation and stores factual knowledge, becomes too narrow. Additionally, with very few attention heads, the model loses the ability to attend to different representational subspaces in parallel—a key advantage of multi-head attention. The loss rises as the model becomes MLP-starved or attention-impoverished.

The MLP-to-attention ratio U-curve (Figure 5). Here, the x-axis is $r_{\text{mlp/attn}}$ and different curves correspond to different fixed values of $d_{\text{model}}$. Again, a consistent U-shape emerges, though the optimum shifts slightly with $d_{\text{model}}$:

• Left side (too small $r_{\text{mlp/attn}}$ — attention-dominant): When nearly all parameters are allocated to attention, the MLP is squeezed to a small intermediate size. Since the MLP is responsible for the feed-forward computation that processes each token independently after attention mixing, a tiny MLP limits the model's ability to perform the non-linear transformations needed for complex reasoning. The paper observes that "the attention mechanism is central to the success of transformers" (citing Vaswani, 2017) but also that "recent open-weight models have allocated a progressively smaller fraction of parameters to attention as overall model size increases" (Section 3.3). The U-curve shows that this trend has limits: below some ratio, reducing MLP capacity hurts more than the extra attention helps.

• Right side (too large $r_{\text{mlp/attn}}$ — MLP-dominant): When parameters are overwhelmingly allocated to the MLP, attention is starved. Attention projections become narrow (small $d_q$ or $d_{\text{kv}}$), which limits the model's ability to learn which tokens to attend to. Since attention is the only mechanism for routing information between token positions, constraining it too aggressively prevents the model from forming the long-range dependencies that transformers excel at. The loss rises because the model cannot effectively model context.

Why the U-shape matters for modeling: A U-shaped relationship implies there exists a unique interior optimum that balances the two sublayers. This is exactly the kind of relationship that a scaling law should capture—it cannot be modeled by a simple monotonic function (like a power law, which the Chinchilla law uses for $N$ and $D$). The paper chooses the functional form:

$f(x) = c_0 + c_1 \log x + \frac{c_2}{x}$

where $x$ is either $d_{\text{model}}/\sqrt{N}$ or $r_{\text{mlp/attn}}$, and $c_0$, $c_1$, $c_2$ are learnable parameters.

Why this specific functional form? The paper states it "effectively models the U-shaped behavior while ensuring sublinear growth as $x$ increases" (Section 3.3). Let's unpack this:

• The $\log x$ term grows slowly and without bound as $x \to \infty$, which captures the gradual degradation on the right side of the U (e.g., when the hidden size gets too large). The logarithmic growth is sublinear—it doesn't explode—which matches the empirical observation that loss degrades gradually rather than catastrophically on the right side.

• The $c_2/x$ term goes to zero as $x \to \infty$ (capturing that the right-side penalty is dominated by the log term at large $x$) but blows up as $x \to 0^+$, which captures the steep degradation on the left side of the U (e.g., when the MLP-to-attention ratio or hidden size gets too small). This asymmetric behavior—sharp rise on the left, gradual rise on the right—is exactly what the U-curves in Figures 4 and 5 show.

• The constant $c_0$ sets the vertical position of the curve.

• The parameters $c_1$ and $c_2$ control the asymmetry: if $c_2 \gg c_1$, the left side dominates (steep drop-off for small $x$); if $c_1 \gg c_2$, the right side dominates (gradual climb for large $x$). The fitted values in Section 5.1 provide concrete numbers: for the multiplicative calibration on Task 3, $a_2 = 0.0078$ (coefficient on the $1/x$ term for $d_{\text{model}}/\sqrt{N}$) and $b_2 = 0.0065$ (coefficient on the $1/r$ term for $r_{\text{mlp/attn}}$), both relatively small compared to their log-term counterparts, indicating that the left-side penalties are modest at the scales studied—the optima are fairly broad.

An alternative that would have been wrong: A simple power law $x^\gamma$ cannot produce a U-shape because it is monotonic. A quadratic $c_2 (x - x_{\text{opt}})^2$ would impose symmetry around the optimum, which the empirical data does not support—the left side is typically steeper than the right side. The chosen form $c_0 + c_1 \log x + c_2/x$ has exactly two degrees of freedom beyond the offset and can produce asymmetric U-shapes, making it the simplest functional form that captures the observed phenomenology.

---

The Conditional Scaling Law Formalism

The core intellectual contribution is the conditional scaling law that extends the Chinchilla framework. The paper begins by acknowledging the difficulty of a fully joint model:

"fitting a single all-purpose scaling law $L(d/\sqrt{N}, r, N, D)$ is unrealistic across all possible configurations"

The reason is practical: the number of possible combinations of $d_{\text{model}}$, $r_{\text{mlp/attn}}$, $N$, and $D$ is combinatorially large, and training a model for every combination is prohibitively expensive. Instead, the paper proposes a two-step approach that leverages the Chinchilla law as a reference point.

Step 1: Obtain the optimal reference loss. For given $N$ and $D$, the Chinchilla-optimal loss is:

$L_{\text{opt}}(N, D) = \min L(N, D) = \min \left(E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}\right)$

where $E$, $A$, $B$, $\alpha$, $\beta$ are the standard Chinchilla parameters. In practice, for $N_{\text{non-embed}} < 1\text{B}$, the paper "empirically searched over architecture variants to find the optimal loss" rather than fitting the Chinchilla curve to all data (Section 4). This means: for the 80M models, they train all architectural variants from Table 4 and record the minimum loss achieved. That minimum becomes $L_{\text{opt}}(80\text{M}, 100 \times 80\text{M})$. This empirical approach avoids the error that would come from fitting a Chinchilla curve to suboptimal architectures—the reference point is genuinely the best the architecture family can achieve at that scale.

Step 2: Calibrate architectural deviations relative to this reference. The paper proposes two calibration schemes, both assuming that the effects of $d_{\text{model}}$ and $r_{\text{mlp/attn}}$ on loss are separable (i.e., the penalty for a suboptimal hidden size does not depend on the mlp-to-attention ratio, and vice versa).

Multiplicative calibration:

$L\left(\frac{d}{\sqrt{N}}, r \mid N, D\right) = \left(a_0 + a_1 \log\frac{d}{\sqrt{N}} + a_2 \frac{\sqrt{N}}{d}\right) \cdot \left(b_0 + b_1 \log r + b_2 \frac{1}{r}\right) \cdot L_{\text{opt}}$

where $d = d_{\text{model}}$, $r = r_{\text{mlp/attn}}$, and $a_0$, $a_1$, $a_2$, $b_0$, $b_1$, $b_2$ are six learnable parameters shared across all $N$ and $D$.

What this equation computes: Given a specific architecture $(d_{\text{model}}, r_{\text{mlp/attn}})$ at a given scale $(N, D)$, the predicted loss is the product of three terms: (1) a hidden-size correction factor that is a U-shaped function of $d/\sqrt{N}$, (2) an MLP-to-attention ratio correction factor that is a U-shaped function of $r$, and (3) the Chinchilla-optimal reference loss. If the architecture is optimal in both dimensions, the correction factors equal 1 (or some constant offset depending on parameterization), and the predicted loss equals $L_{\text{opt}}$ (up to a constant). If the architecture deviates from optimality in either dimension, the corresponding correction factor increases the predicted loss multiplicatively.

Why multiplicative? The multiplicative form has a natural interpretation: the architecture imposes a relative penalty on top of the scale-determined loss floor. If the optimal architecture at scale $N_1$ reduces loss by 10% relative to a suboptimal architecture, the multiplicative form predicts roughly the same 10% relative reduction at scale $N_2$. This is a reasonable inductive bias if architectural effects are proportional rather than absolute—a bad architecture hurts a large model proportionally as much as a small model. The paper validates this assumption empirically through the cross-scale prediction tasks.

Additive calibration (the alternative):

$L\left(\frac{d}{\sqrt{N}}, r \mid N, D\right) = \left(a_0 + a_1 \log\frac{d}{\sqrt{N}} + a_2 \frac{\sqrt{N}}{d}\right) + \left(b_1 \log r + b_2 \frac{1}{r}\right) + L_{\text{opt}}$

Here, the architectural effects are additive: a suboptimal architecture adds a fixed number of nats (or bits) to the loss, regardless of the base loss level. The paper finds that "multiplicative and additive calibrations achieve similar MSE and Spearman correlations" (Section 5, Ablation of Calibration, and Figure 25 right), suggesting that at the scales studied (80M–1B), the distinction between proportional and absolute penalties is not yet critical. However, the multiplicative form is used for the main results because it is more natural for extrapolation: as $L_{\text{opt}}$ decreases with scale, an additive penalty would become an increasingly large fraction of the total loss, while a multiplicative penalty maintains a constant relative impact.

The separability assumption: Both forms assume that the hidden-size correction and the MLP-to-attention ratio correction are independent—they multiply or add without interaction terms. This is a modeling choice that simplifies fitting and extrapolation. The paper explicitly tests a joint, non-separable alternative:

$\left(a_0 + a_1 \log\frac{dr}{\sqrt{N}} + a_2 \frac{\sqrt{N}}{dr}\right) \cdot L_{\text{opt}}$

This form treats $d$ and $r$ as a single combined variable $d \cdot r / \sqrt{N}$, which assumes they trade off against each other in a specific, rigid way. Figure 26 in Appendix J shows that this joint formulation performs "significantly worse" than the separable multiplicative calibration, with MSE increasing from 0.0001 to 0.0067 and Spearman correlation dropping from 0.7451 to 0.2108. The failure of the joint form provides empirical support for the separability assumption: the optimal hidden size and the optimal MLP-to-attention ratio are largely independent choices—you cannot compensate for a bad hidden size by adjusting the MLP-to-attention ratio, and vice versa.

The learnable parameters and what they mean:

• $a_1$ and $b_1$ (coefficients on the $\log$ terms): These control the penalty for being too large. If $a_1$ is positive, increasing $d_{\text{model}}/\sqrt{N}$ beyond the optimum increases loss. The fitted values in Section 5.1 ($a_1 = 0.0974$, $b_1 = 0.0063$) indicate that the hidden-size penalty grows faster with log-scale than the ratio penalty.

• $a_2$ and $b_2$ (coefficients on the $1/x$ terms): These control the penalty for being too small. Larger values mean the left side of the U-curve is steeper. The fitted values ($a_2 = 0.0078$, $b_2 = 0.0065$) are relatively small, consistent with the broad U-shapes observed in Figures 4 and 5.

• $a_0$ and $b_0$ (offsets): These scale the multiplicative factors so that they equal approximately 1 at the optimum. In the multiplicative form, the product of the offset-containing terms determines the minimum achievable correction factor.

Solving for the optimal architecture: Given fitted parameters, the optimal $d_{\text{model}}/\sqrt{N}$ can be found by solving $\partial L / \partial d = 0$, which yields an analytical expression involving $a_1$ and $a_2$. For the 1B model (Section 5.1), this gives $d_{\text{model}}/\sqrt{N} = 0.08$, corresponding to $d_{\text{model}} = 0.08 \times \sqrt{1\text{B}} \approx 2560$. The optimal $r_{\text{mlp/attn}}$ is solved similarly, yielding $r = 1.032$ for the 1B model. These are the architectural choices that minimize predicted loss—they are not guessed or heuristically chosen; they are the closed-form solutions to the optimization problem defined by the fitted scaling law.

---

The Fitting Protocol and Progressive Validation Strategy

The paper's validation strategy is designed to test whether the conditional scaling law can extrapolate from small models to larger ones—the entire premise is that you can fit on cheap-to-train small models and use the fitted law to design expensive large models without training a full architectural grid at the large scale.

Training data and hyperparameters. All models are trained on data sampled from Dolma-v1.7 (Soldaini et al., 2024), which contains data from 15 sources. Tokens are sampled "with probability proportional to each source's contribution, ensuring the sampled dataset preserves a similar distribution to Dolma-v1.7" (Section 4). The total training tokens are set to $100 \times N_{\text{non-embed}}$, which is 5 times the Chinchilla-optimal ratio (where $D \approx 20 \times N_{\text{non-embed}}$)—this ensures that all models are trained to convergence and that the loss primarily reflects architectural capacity rather than under-training. The detailed hyperparameters (Table 5, Appendix E) are:

The learning rate decreases with model size (following standard practice), and the batch size increases for larger models to maintain training stability.

The progressive fitting tasks. To validate extrapolation, the paper defines three tasks of increasing difficulty (Section 4):

• Task 1: Fit the conditional scaling law on the 80M architectural variants, evaluate predictive performance on the 145M variants. This tests whether architectural optima are stable across a roughly 2× increase in parameters.

• Task 2: Fit on 80M and 145M variants jointly, evaluate on 297M variants. This tests whether aggregating data from multiple scales improves or degrades prediction.

• Task 3: Fit on 80M, 145M, and 297M variants jointly, evaluate on 1B variants. This is the hardest test—extrapolating from at most 297M to 1B, a 3.4× increase.

For each task, the paper reports two metrics:

1. Mean Squared Error (MSE): Defined as $\frac{1}{n}\sum_{i=1}^{n} (l_i - \hat{l}_i)^2$, where $l_i$ is the actual training loss of the $i$-th architectural variant and $\hat{l}_i$ is the scaling law's prediction. Lower is better.

2. Spearman's rank correlation coefficient: Measures whether the ordering of architectures by predicted loss matches the ordering by actual loss. This is critical for the search framework: even if the absolute loss predictions are off by a constant shift, the scaling law can still correctly identify which architecture is best.

Results for Task 1 (Figure 6, left): MSE = 0.0002, Spearman = 0.8909. The predicted vs. actual loss plot shows a tight cluster around the diagonal, with both fitting (orange dots, 80M) and evaluation (purple crosses, 145M) points well-aligned. The high Spearman indicates that the relative ranking of architectures is preserved across the 80M → 145M scale jump.

Results for Task 2 (Figure 6, center): MSE = 0.0001, Spearman = 0.7920. The MSE improves (as expected with more fitting data), but Spearman drops slightly from 0.89 to 0.79. This could indicate that the 80M and 145M data have slightly different optimal architectures, and fitting them jointly forces a compromise that is less accurate for either individually, though still strongly predictive.

Results for Task 3 (Figure 6, right): MSE = 0.0001, Spearman = 0.7451. The Spearman remains high at 0.745, showing that even extrapolating from ≤297M to 1B yields meaningful architectural guidance. The MSE is identical to Task 2 at 0.0001, indicating that the absolute accuracy of predictions does not degrade with the larger extrapolation—the variance in loss across 1B architectures is well-captured.

The Levenberg-Marquardt fitting algorithm. The paper uses the Levenberg-Marquardt algorithm, a standard nonlinear least-squares solver that interpolates between gradient descent and the Gauss-Newton method. Formally, it solves:

$\arg\min_{\beta} \sum_{i=1}^{m} \left[y_i - f(x_i, \beta)\right]^2$

where $(x_i, y_i)$ are the observed (architecture, loss) pairs, $f$ is the conditional scaling law function (Eq. 3), and $\beta$ is the vector of learnable parameters $(a_0, a_1, a_2, b_0, b_1, b_2)$. The algorithm is chosen because it handles the nonlinearity of the calibration function (which includes both log and reciprocal terms) better than linear least squares while being more efficient than gradient-free methods for this low-dimensional parameter space (6 parameters).

Ablation: effect of outlier architectures. The paper trains some extreme architectures with $r_{\text{mlp/attn}}$ values far outside the typical range of 0.5–5 (e.g., 0.10 for variant v7 in 80M and 12.6 for variant v2 in 80M, Table 4). These outliers are included in the full training grid to map out the shape of the U-curve, but the paper finds that including them in the scaling law fit degrades predictive performance. Figure 25 (left vs. center) shows the comparison on Task 3: with outliers excluded ($r_{\text{mlp/attn}} \in [0.5, 5]$), Spearman = 0.7451; with outliers included, Spearman drops to 0.3309. The likely reason is that the simple functional form $c_0 + c_1 \log x + c_2/x$ cannot perfectly capture the behavior at extreme ratios (the loss probably increases faster than logarithmic on the far right, or faster than $1/x$ on the far left), and including these points biases the fit for the regime of practical interest (the central region around the optimum). The paper therefore restricts the fitting data to architectures with $r_{\text{mlp/attn}} \in [0.5, 5]$, which covers all practical open-weight model configurations (LLaMA-3.2-1B uses 4.8, Qwen3-0.6B uses 1.5).

Ablation: fitting data strategy at larger scales. An important finding, shown in Figure 8 and Section 5.1, is that fitting on models closer in size to the target yields better extrapolation. When fitting on 80M, 145M, 297M, and 1B data to predict 3B architectures, the Spearman correlation is 0.500—significant but much worse than the 0.745 achieved when predicting 1B from ≤297M data. However, when fitting only on 1B data to predict 3B architectures, the Spearman jumps to 1.00 (perfect rank correlation on the test set). The paper attributes this to "the law's coefficients shift with model size" (Section 5.1). In practice, this means that when scaling up, "it is often sufficient, and sometimes preferable, to fit the law using models within a closer size range to the target, such as about one third of its scale." The coefficients for the 1B-fit-to-3B law are $a_0 = 2.319$, $a_1 = 0.238$, $a_2 = 0.0176$, $b_0 = 0.5104$, $b_1 = 0.0051$, $b_2 = 0.0062$, producing $d_{\text{model}}/\sqrt{N} = 0.074$ and $r = 1.229$ for the 3B optimum.

---

The Constrained Optimization and Search Procedure (Algorithm 1)

With a validated conditional scaling law in hand, the final step is operational: given a target parameter count $N$, training tokens $D$, and a maximum allowable loss $L_t$, find the architecture that maximizes inference throughput.

Step 1 (implicit in Algorithm 1, line 1): If $L_{\text{opt}}(N, D)$ is not already known from previous Chinchilla fits, train small models to fit it. In practice, for the 1B and 3B targets, the paper has already trained models at smaller scales and can extrapolate $L_{\text{opt}}$ from the Chinchilla parameters.

Step 2 (Algorithm 1, line 2): Solve the constrained optimization in Equation (4):

$\arg\max_{P} \text{IN}(P) \quad \text{s.t.} \quad L(P \mid N, D) \leq L_t$

How this is done in practice: The paper does not solve this analytically because $\text{IN}(P)$—inference throughput—depends on hardware and serving framework details that are not captured by a simple formula. Instead, the paper takes the following approach:

1. Use the conditional scaling law to predict the loss for a range of feasible $(d_{\text{model}}, r_{\text{mlp/attn}})$ combinations at the target $N$ and $D$.
2. Filter to those architectures whose predicted loss satisfies $L(P \mid N, D) \leq L_t$.
3. Among those, narrow down to a manageable set of candidate architectures by selecting Pareto-optimal points—architectures where neither throughput nor predicted loss can be improved without worsening the other.
4. Train these candidate architectures (or at least the most promising few, given budget constraints) and measure their actual inference throughput using vLLM on A100 GPUs.
5. Select the architecture with the highest measured throughput.

What $L_t$ is set to: For the 1B models, $L_t$ is set to match the training loss of the LLaMA-3.2-1B architecture trained under the same setup. For the 3B models, $L_t$ is set to match LLaMA-3.2-3B's training loss. This means the constraint is "do at least as well as the default architecture," and the objective is to exceed it in efficiency.

The Pareto-optimal selection: The paper names the architectures that satisfy the loss constraint and maximize throughput Surefire-1B and Surefire-3B. These are distinct from Panda-1B and Panda-3B, which are the architectures that minimize predicted loss without regard to inference efficiency (they solve $\partial L/\partial d = 0$ and $\partial L/\partial r = 0$). As Table 1 shows:

• Panda-1B: $d_{\text{model}}=2560$, $f_{\text{size}}=4096$, $r_{\text{mlp/attn}}=1.07$, GQA=4. This architecture achieves the lowest loss (2.782) among the 1B variants but has lower throughput than Surefire-1B.
• Surefire-1B: $d_{\text{model}}=2560$, $f_{\text{size}}=6144$, $r_{\text{mlp/attn}}=3.60$, GQA=9. This architecture achieves higher throughput (Figure 7, center) while matching LLaMA-3.2-1B's loss constraint (2.804 vs. 2.803). The larger $r_{\text{mlp/attn}}$ (3.60 vs. 1.07) shifts parameters from attention to MLP, reducing KV cache size and inference FLOPs, while the higher GQA (9 vs. 4) further reduces the KV cache. The loss is slightly worse (-0.022 nats) but still meets the constraint.

This illustrates the core tradeoff: the loss-minimizing architecture (Panda) is not the throughput-maximizing architecture (Surefire); the latter accepts a small accuracy penalty within the allowed tolerance to gain substantial efficiency.

Step 3 (Algorithm 1, line 3): Perform a local search over GQA values. Once $d_{\text{model}}$ and $r_{\text{mlp/attn}}$ are fixed, the number of attention heads $n_{\text{heads}} = d_{\text{model}}/d_{\text{head}}$ is determined. Feasible GQA values are the prime factors of $n_{\text{heads}}$. For Surefire-1B with $d_{\text{model}}=2560$ and $d_{\text{head}}=64$, $n_{\text{heads}} = 2560/64 = 40$. The prime factorization of 40 is $2^3 \times 5$, so feasible GQA values include 1, 2, 4, 5, 8, 10, 20, 40. The local search enumerates these, measures throughput (or predicts it based on the monotonic trend in Figure 11), and selects the highest GQA that does not cause the training loss to degrade below the constraint. The paper uses "early stopping—once performance falls below that of the GQA=4 baseline" (Section 3.4) to avoid evaluating all candidates.

Validation of the search results: The paper demonstrates that the resulting architectures generalize across hardware and serving stacks. Table 6 and Appendix G show inference throughput for Surefire-1B and Surefire-3B on both A100 and H200 GPUs using vLLM and SGLang. Across all combinations, Surefire models consistently outperform LLaMA-3.2 baselines:

• Surefire-1B achieves up to 47% higher throughput than LLaMA-3.2-1B (SGLang on H200, batch size 128).
• Surefire-3B achieves up to 42% higher throughput than LLaMA-3.2-3B (vLLM on A100, batch size 64).

This cross-framework, cross-hardware consistency is important because it suggests the architectural advantages are fundamental (stemming from reduced FLOPs and KV cache) rather than artifacts of a particular serving implementation.

---

Inference Efficiency Measurement Methodology

The paper measures inference throughput using two serving frameworks and two GPU types:

• vLLM (Kwon et al., 2023): Uses PagedAttention for efficient KV cache management. Default configuration: 4096 input tokens, 1024 output tokens.
• SGLang (Zheng et al., 2023): Uses RadixAttention for prefix caching. Same token configuration.
• GPUs: NVIDIA Ampere A100 (40GB) and NVIDIA H200 (141GB).

Throughput is defined as tokens generated per second (including both prefill and decode phases) and is averaged over 5 repeated runs. The batch sizes are swept across [16, 32, 64, 128] for 1B models and [16, 32, 64, 128] for 3B models (Figures 7, 9-11, 15-19). The paper notes that "all experiments are conducted on a single GPU" to isolate architectural effects without confounding from model parallelism or inter-GPU communication.

Why this measurement approach works: The consistent trend across GPU types and frameworks—larger $d_{\text{model}}$ and larger $r_{\text{mlp/attn}}$ improve throughput—suggests the efficiency gains are not specific to a particular software optimization. The inference FLOPs analysis (Appendix K) provides the first-principles explanation: reducing the query-head dimension $d_q$ and the KV-head dimension $d_{\text{kv}}$ lowers both the arithmetic intensity and the memory footprint of the attention operation. The empirical measurement confirms that these theoretical savings translate to wall-clock speedups.

---

4. Key Insights and Innovations

Innovation 1: Architecture as a First-Class Variable in Scaling Laws via Conditional Calibration

The paper's deepest conceptual move is not inventing a new scaling law but changing what a scaling law is conditioned on. The Chinchilla framework (Hoffmann et al., 2022) treats model size $N$ as a scalar—all 1B-parameter models are, in the scaling law's eyes, identical. This is a deliberate simplification that enables clean power-law fits, but it erases architectural heterogeneity entirely. The paper asks: what if we treat $N$ as a constraint envelope within which architecture can vary, and learn how that variation shifts the loss?

The mechanism for doing this—the two-step reference-and-calibration approach—is detailed in Section 3. What makes it innovative as an idea is its humility. Rather than attempting to absorb $d_{\text{model}}$, $r_{\text{mlp/attn}}$, $N$, and $D$ into a single monolithic law (which the paper explicitly calls "unrealistic across all possible configurations"), the conditional approach decouples the hard problem from the tractable one. Fitting how loss scales with $N$ and $D$ is hard and requires many models at many scales. Fitting how loss varies with architecture at a given scale is easier and can be done with a focused sweep. The conditional law stitches these together by using the Chinchilla optimum as a reference point and learning multiplicative corrections around it.

This conceptual move is significant beyond the specific functional forms used. It establishes a template for incrementally extending scaling laws with new degrees of freedom without breaking the existing framework. If a future researcher wants to incorporate, say, activation function choice or normalization scheme, they can add another multiplicative correction term rather than refitting the entire law from scratch. The separability assumption that makes this work—that the effects of hidden size and MLP-to-attention ratio are independent—is validated empirically (Appendix J, Figure 26) and provides a principle for what architectural factors can be cleanly incorporated and which (like GQA) require separate search.

Contrast this with the prior work Bian et al. (2025), which folded architecture into scaling laws by adding a single continuous variable (aspect ratio) to an otherwise standard power-law form. That approach is more elegant in principle but brittle in practice: it assumes all architectural effects can be captured by one degree of freedom, and it cannot accommodate discrete choices like GQA. The conditional calibration framework is less aesthetically unified but more practically extensible—it acknowledges that some architectural choices are continuously tunable with predictable effects (hidden size, MLP-to-attention ratio) while others are discrete and idiosyncratic (GQA), and it handles them differently.

Innovation 2: Inference Efficiency as a Constrained Optimization Objective, Not an Afterthought

Prior work on scaling laws treats inference cost either as absent (Chinchilla, Kaplan et al.) or as folded into a single total-FLOPs budget (Sardana et al., 2023). Both approaches implicitly assume that inference efficiency is something you measure after the fact rather than something you optimize for during architecture design. The paper flips this: inference throughput is the objective to maximize, and accuracy is the constraint.

This framing (Equation 4, Algorithm 1) is not merely cosmetic. It reflects the reality of production deployments, where a model must meet a minimum accuracy bar to be useful, and any additional inference efficiency directly translates to lower serving costs or higher throughput. The traditional framing—maximize accuracy subject to a training budget—is appropriate for research benchmarks but misaligned with deployment economics, where training is a one-time cost and inference is perpetually recurring.

What makes this more than a reframing exercise is that the paper demonstrates this tradeoff is actionable: the architecture that minimizes training loss (Panda) is not the architecture that maximizes throughput under a loss constraint (Surefire). As shown in Table 1, Panda-1B achieves loss 2.782 with $r_{\text{mlp/attn}} = 1.07$, while Surefire-1B achieves loss 2.804 (matching the LLaMA-3.2-1B constraint) with $r_{\text{mlp/attn}} = 3.60$ and delivers 42% higher throughput. The loss-minimizing architecture and the throughput-maximizing architecture are genuinely different points in the design space—if you only optimize for loss, you leave inference efficiency on the table. This is the architectural analog of the "overspecialization" phenomenon: optimizing a single metric to its extreme often produces a configuration that is suboptimal on other dimensions that matter in practice.

The paper's pragmatic approach to measuring inference efficiency—empirically benchmarking on real hardware (vLLM on A100, SGLang on H200) rather than attempting a closed-form analytical model—is also a conceptual choice worth noting. It acknowledges that throughput is not a clean mathematical function of architectural parameters but depends on hardware-specific factors (memory bandwidth, tensor core utilization, KV cache management) that are better measured than modeled. The cross-framework, cross-hardware validation (Table 6) provides confidence that the architectural advantages are not artifacts of a particular software stack.

Innovation 3: The U-Shaped Optimality of Hidden Size and MLP-to-Attention Ratio as a Universal Phenomenon

The empirical observation that both $d_{\text{model}}/\sqrt{N}$ and $r_{\text{mlp/attn}}$ exhibit U-shaped relationships with training loss is, in isolation, not surprising—many architectural hyperparameters have interior optima. What elevates this from an observation to an innovation is the paper's demonstration that these U-shapes are consistent across model scales (80M, 145M, 297M in Figures 4 and 5), that their optima occur at approximately the same normalized values ($d_{\text{model}}/\sqrt{N} \approx 0.06$–$0.10$, $r_{\text{mlp/attn}} \approx 1$–$2$), and that this scale invariance enables extrapolation from small models to larger ones.

This is a diagnostic contribution as much as a predictive one. Prior to this work, the dominant assumption in architecture design—implicit in the wildly varying choices across open-weight models (Table 7–8, Appendix H)—was that optimal architecture depends strongly on scale, and that rules of thumb derived from small models do not transfer. The paper provides systematic evidence against this assumption: the optimal normalized hidden size and MLP-to-attention ratio are approximately constant across the 80M–3B range. The fact that fitting on 1B data alone produces a nearly perfect prediction of 3B architectural rankings (Spearman = 1.00, Figure 8 right) while fitting on 80M–297M data yields weaker but still useful prediction (Spearman = 0.50, Figure 8 left) suggests that the architectural optimum shifts very slowly, if at all, across this range—the degradation in cross-scale prediction is more about the law's coefficients shifting slightly than about the optimum itself moving.

This finding has a direct practical consequence: you can find the optimal architecture for a 3B model by training a grid of 1B models, which is dramatically cheaper. The paper's progressive fitting strategy (Tasks 1–3) operationalizes this and quantifies the prediction quality at each step, giving practitioners a concrete recipe for how close in scale the fitting models need to be.

Innovation 4: GQA as a Separately Optimizable, Discrete Knob with No Continuous Scaling Behavior

Perhaps the most understated but conceptually interesting finding concerns Grouped-Query Attention. The paper documents that GQA has a substantial impact on inference throughput (Figures 11, 14, 18, 22 in Appendices F and G) but that its relationship to training loss is non-smooth, highly fluctuating, and lacks any consistent functional form (Figure 24 in Appendix I). Unlike hidden size and the MLP-to-attention ratio, which exhibit clean U-shaped curves amenable to parametric modeling, GQA produces a jagged, non-monotonic loss landscape as the grouping factor changes—sometimes improving loss, sometimes degrading it, with no simple pattern.

The paper's negative finding—that GQA cannot be incorporated into a continuous scaling law—is itself an intellectual contribution. It identifies a boundary of what scaling laws can capture and motivates a separate, discrete optimization step (the local GQA search in Algorithm 1). The paper's explanation for why GQA behaves this way is instructive: when the total number of heads is fixed, changing GQA redistributes capacity between query heads and key-value heads in opposite directions—increasing query heads expands query projection dimensionality but shrinks KV-expressivity by increasing sharing, while decreasing query heads improves KV capacity but narrows query projections. These opposing effects create a fundamental tradeoff with no guaranteed monotonic solution.

This insight connects to why open-weight models make inconsistent GQA choices (Qwen3 uses GQA=2 for 0.6B but GQA=4 for 4B, LLaMA-3 uses GQA=4 for 1B but GQA=3 for 3B): there is no universal rule, and the optimum depends on model-specific details of the attention head budget. The paper's response—treat GQA as a discrete search problem with early stopping—is pragmatic but theoretically honest: some architectural choices resist elegant parameterizations, and acknowledging that is better than forcing them into a framework that cannot accommodate them.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All training uses data sampled from Dolma-v1.7 (Soldaini et al., 2024), which contains data from 15 different sources. Tokens are sampled with probability proportional to each source's contribution, preserving the original distribution. The training data volume is set to $100 \times N_{\text{non-embed}}$ tokens for all models, which is 5× the Chinchilla-optimal ratio and ensures convergence. For zero-shot downstream evaluation, the paper uses nine benchmarks from lm-evaluation-harness: ARC-Easy, ARC-Challenge, LAMBADA, HellaSwag, OpenBookQA, PIQA, SciQ, WinoGrande, and CoQA (Section 4).

• Base model(s). All models are LLaMA-3.2-style decoder-only transformers (Dubey et al., 2024) with architectural variants generated by modifying hidden size, intermediate size, attention heads, and GQA while holding the number of layers fixed. The paper trains over 200 models spanning non-embedding parameter counts of 80M, 145M, 297M, 1B, and 3B. For the 1B and 3B validation, the paper compares against the standard LLaMA-3.2-1B and LLaMA-3.2-3B architectures trained under identical data and hyperparameter setups (Section 5.1, Table 1). The LLaMA family is chosen because it is "representative of a canonical approach to scaling" and its architectural configurability allows systematic variation of the factors under study.

• Metrics. The primary metric for scaling-law fitting is pretraining loss (cross-entropy loss on the training data). For downstream validation, the paper reports zero-shot accuracy on each of the nine benchmarks, with the average across all nine used as the aggregate (Tables 1, 9, 10). The scaling-law predictive quality is measured by Mean Squared Error (MSE), defined as $\frac{1}{n}\sum_{i=1}^{n}(l_i - \hat{l}_i)^2$ where $l_i$ is the actual loss and $\hat{l}_i$ is the predicted loss, and Spearman's rank correlation coefficient, which assesses whether the predicted ordering of architectures matches the actual ordering (Section 4, Fitting Scaling Laws). For inference efficiency, the metric is throughput in tokens per second, averaged over 5 repeated runs using vLLM or SGLang with 4096 input tokens and 1024 output tokens on A100 or H200 GPUs (Section 4, Inference Setup).

• Baselines. The primary architectural baselines are the LLaMA-3.2-1B and LLaMA-3.2-3B standard configurations, trained under the identical data and hyperparameter setup as the paper's models (Table 1). The paper also compares against open-weight baselines: LLaMA-3.2-1B-HF, OLMo-2-1B-HF, LLaMA-3.2-3B-HF, and Qwen2.5-3B-HF (Tables 7 and 8, Appendix H). For the scaling-law fitting, the baseline reference loss $L_{\text{opt}}(N, D)$ is obtained either by empirically searching over architectural variants (for $N < 1\text{B}$) or by fitting the Chinchilla functional form (for larger scales). This is not a baseline in the competitive sense but a reference point from which architectural deviations are calibrated (Section 3.3).

• Generation budget / compute accounting. The paper does not compare methods under a shared FLOPs budget in the traditional scaling-law sense—all models are trained with the same token budget ($100 \times N_{\text{non-embed}}$) and differentiated by their architecture, not their training cost. For inference, compute is measured empirically as throughput rather than modeled analytically, and the search framework (Algorithm 1) treats inference efficiency as the objective to maximize subject to a loss constraint. The inference FLOPs analysis in Appendix K provides a first-principles breakdown of how architectural changes affect total non-embedding FLOPs per token, but this is not used as the primary accounting metric for experimental comparisons. The critical budget unit is non-embedding parameters $N_{\text{non-embed}}$—all architectural variants at a given scale have approximately the same parameter count, ensuring fair comparison under the constraint the paper studies.

• Cross-validation / statistical protocol. The paper uses a progressive fitting strategy to validate extrapolation: Task 1 fits on 80M and evaluates on 145M; Task 2 fits on 80M+145M and evaluates on 297M; Task 3 fits on 80M+145M+297M and evaluates on 1B (Section 4, Fitting Scaling Laws). This is not traditional train/val/test splitting but a scale-based holdout protocol designed to test whether the fitted coefficients generalize across model sizes. The paper does not report confidence intervals on the fitted coefficients or on throughput measurements, and the 500-question MATH-style test set from prior work is not used—downstream evaluation uses only the nine lm-evaluation-harness benchmarks. For the ablated fitting data strategy (Section 5.1), the paper refits the law using only 1B data and evaluates on 3B architectures, yielding MSE and Spearman metrics for comparison.

Main Quantitative Results

Predictive Accuracy of the Conditional Scaling Law

The headline result is that the multiplicative conditional scaling law achieves MSE = 0.0001–0.0002 and Spearman correlation = 0.75–0.89 across all three progressive fitting tasks (Figure 6). On Task 3—the most challenging, extrapolating from ≤297M to 1B—the law achieves MSE = 0.0001 and Spearman = 0.7451 (Figure 6, right). This demonstrates that the fitted calibration terms capture meaningful architectural variation and that the optimal architecture is sufficiently stable across scales for extrapolation to be useful.

The predicted vs. actual loss plots in Figure 6 show that both the fitting data points (orange dots) and evaluation data points (purple crosses) cluster tightly around the diagonal for all three tasks. There is no visible systematic bias—the predicted loss neither consistently overestimates nor underestimates the actual loss across the range of 2.6–3.6 nats. The slightly higher spread in evaluation points compared to fitting points is expected and does not indicate a breakdown of the law; the Spearman values (0.89, 0.79, 0.75) confirm that the relative ordering of architectures is well-preserved even when absolute predictions degrade.

The paper also reports an ablation of the fitting data strategy for the 3B scale (Section 5.1, Figure 8). When the law is fitted on the full 80M+145M+297M+1B data and evaluated on 3B architectures, MSE = 0.0001 but Spearman drops to 0.500 (Figure 8, left). This is notably worse than the Task 3 extrapolation (0.745 Spearman from ≤297M to 1B). When the law is refitted using only the 1B variants and evaluated on 3B, MSE = 0.0001 and Spearman = 1.000 (Figure 8, right)—perfect rank correlation on the held-out 3B architectures. The paper's interpretation is that "the law's coefficients shift with model size" and that fitting on models closer to the target (approximately one-third the scale) yields better predictions than including data from much smaller scales.

Validation at the 1B Scale: Panda-1B and Surefire-1B vs. LLaMA-3.2-1B

Using the conditional scaling law fitted on 80M+145M+297M data (Task 3), the optimal 1B architecture is predicted to have $d_{\text{model}}/\sqrt{N} = 0.08$ and $r_{\text{mlp/attn}} = 1.032$, yielding Panda-1B with $d_{\text{model}} = 2560$, $f_{\text{size}} = 4096$, and GQA=4 (Section 5.1, Table 1).

Accuracy results (Table 1, detailed in Table 9):

• LLaMA-3.2-1B: training loss = 2.803, average downstream accuracy = 54.9%
• Panda-1B: training loss = 2.782, average downstream accuracy = 57.0%

Panda-1B outperforms LLaMA-3.2-1B by 0.021 nats in training loss and 2.1% in average downstream accuracy. On individual benchmarks (Table 9), Panda-1B achieves notably higher accuracy on LAMBADA (55.1% vs. 52.8%, +2.3%), COQA (55.3% vs. 48.7%, +6.6%), and SciQ (87.2% vs. 84.8%, +2.4%), while performance on ARC-Challenge is essentially tied (28.9% vs. 29.8%). The gains are broad rather than concentrated on a single benchmark.

Figure 7 (left) provides further validation: among "exhaustively trained 1B variants under the same setup," Panda-1B sits at the minimum of the loss curve when plotted against $r_{\text{mlp/attn}}$, confirming that the scaling law correctly identified the loss-minimizing architecture. The curve is U-shaped with a clear optimum near $r_{\text{mlp/attn}} \approx 1.0$–$1.3$, and LLaMA-3.2-1B (at $r_{\text{mlp/attn}} = 4.8$) sits on the right-hand rising limb, well off the optimum.

Inference efficiency results for Surefire-1B (Table 1, Figure 7 center, Table 6):

• Surefire-1B: $d_{\text{model}} = 2560$, $f_{\text{size}} = 6144$, $r_{\text{mlp/attn}} = 3.60$, GQA=9
• Training loss: 2.804 (matching LLaMA-3.2-1B's 2.803)
• Average downstream accuracy: 55.4% (vs. 54.9% for LLaMA-3.2-1B)

Throughput (vLLM on A100, batch size 128): LLaMA-3.2-1B achieves 3825.91 tokens/s; Surefire-1B achieves 4800.05 tokens/s, a 25.5% improvement (Table 6). At batch size 64, the improvement is 18.2% (4028.69 vs. 3409.85). Throughput (SGLang on H200, batch size 128): LLaMA-3.2-1B achieves 8608.57 tokens/s; Surefire-1B achieves 12645.55 tokens/s, a 47% improvement (Table 6). The gains are consistent across serving frameworks (vLLM, SGLang), hardware (A100, H200), and batch sizes (16–128), as shown in Figure 7 (center) and Figures 15, 19, 23 in Appendices F and G.

The Surefire-1B architecture achieves higher throughput than Panda-1B at the cost of slightly higher training loss (2.804 vs. 2.782). The tradeoff is explicit: Panda-1B minimizes loss, Surefire-1B maximizes throughput subject to the loss constraint set by the LLaMA-3.2-1B baseline.

Validation at the 3B Scale: Panda-3B and Surefire-3B vs. LLaMA-3.2-3B

The predicted optimal architecture for 3B (fitted on 80M+145M+297M+1B) is $d_{\text{model}}/\sqrt{N} = 0.08$ and $r_{\text{mlp/attn}} = 1.055$, yielding Panda-3B with $d_{\text{model}} = 4096$, $f_{\text{size}} = 4096$, and GQA=3 (Section 5.1, Table 1).

Accuracy results (Table 1, detailed in Table 10):

• LLaMA-3.2-3B: training loss = 2.625, average downstream accuracy = 61.9%
• Panda-3B: training loss = 2.619, average downstream accuracy = 62.5%

The improvement is smaller in absolute terms than at the 1B scale: 0.006 nats in training loss and 0.6% in average accuracy. On individual benchmarks (Table 10), Panda-3B shows gains on ARC-Challenge (35.2% vs. 33.3%, +1.9%), SciQ (91.2% vs. 89.4%, +1.8%), and WinoGrande (63.2% vs. 62.5%, +0.7%), with smaller differences on most other tasks.

Inference efficiency results for Surefire-3B (Table 1, Figure 7 right, Table 6):

• Surefire-3B: $d_{\text{model}} = 4096$, $f_{\text{size}} = 4096$, $r_{\text{mlp/attn}} = 1.0$, GQA=7
• Training loss: 2.620 (vs. 2.625 for LLaMA-3.2-3B)
• Average downstream accuracy: 62.6% (vs. 61.9%)

Throughput (vLLM on A100, batch size 64): LLaMA-3.2-3B achieves 1136.61 tokens/s; Surefire-3B achieves 1613.32 tokens/s, a 42% improvement (Table 6). At batch size 128, the improvement is 20.8% (1476.22 vs. 1222.03). Throughput (SGLang on H200, batch size 128): LLaMA-3.2-3B achieves 4183.09 tokens/s; Surefire-3B achieves 4877.16 tokens/s, a 16.6% improvement (Table 6). The relative gains are smaller on H200 with SGLang than on A100 with vLLM, suggesting some dependence on hardware and software stack, but the direction is consistent—Surefire-3B always outperforms LLaMA-3.2-3B.

Table 2 ablation on fitting data strategy: When the law is refitted using only 1B data (producing Panda-3B° with $d_{\text{model}} = 4096$, $f_{\text{size}} = 4608$, $r_{\text{mlp/attn}} = 1.23$, GQA=3), training loss drops further to 2.606 (vs. 2.625 for LLaMA-3.2-3B and 2.619 for Panda-3B). Downstream accuracy remains at 62.5%—the same as Panda-3B. This suggests that the optimal architecture for loss minimization is sensitive to the fitting data, with the 1B-fitted law identifying a configuration that achieves lower loss than the law fitted on the full 80M–297M range. The paper does not train a "Surefire-3B°" equivalent with this alternative fitting strategy.

Inference Efficiency Ablations Across Architectural Factors

The paper provides controlled single-factor ablations that isolate the effect of each architectural dimension on inference throughput.

Hidden size (Figures 3 left, 9): Across 1B, 3B, and 8B model scales, larger hidden sizes consistently improve inference throughput at all batch sizes. For the 1B variants (Figure 9, left): at batch size 128, $d_{\text{model}} = 1024$ achieves roughly 3200 tokens/s, $d_{\text{model}} = 2048$ achieves roughly 5200 tokens/s, and $d_{\text{model}} = 4096$ achieves roughly 8000 tokens/s. For the 8B variants (Figure 3, left): at batch size 128, $d_{\text{model}} = 2048$ yields approximately 600 tokens/s, $d_{\text{model}} = 4096$ yields approximately 1000 tokens/s, and $d_{\text{model}} = 8192$ yields approximately 1800 tokens/s. The relationship is monotonic—larger hidden size strictly improves throughput—but the magnitude of improvement varies with batch size and model scale.

MLP-to-attention ratio (Figures 3 right, 10): Higher ratios consistently improve inference throughput. For the 8B variants (Figure 3, right): at batch size 128, $r_{\text{mlp/attn}} = 0.90$ achieves roughly 750 tokens/s, $r_{\text{mlp/attn}} = 1.30$ achieves roughly 850 tokens/s, $r_{\text{mlp/attn}} = 2.40$ achieves roughly 1050 tokens/s, and $r_{\text{mlp/attn}} = 4.20$ achieves roughly 1200 tokens/s. For the 1B variants (Figure 10, left): at batch size 128, $r_{\text{mlp/attn}} = 0.45$ yields approximately 3800 tokens/s, $r_{\text{mlp/attn}} = 4.80$ yields approximately 5200 tokens/s. The gains are substantial but diminish at the highest ratios.

Grouped-query attention (Figure 11, Appendices F and G): Larger GQA values consistently improve throughput. For the 1B variants with $d_{\text{model}} = 2048$ and $r_{\text{mlp/attn}} = 4.0$ (Figure 11, left): at batch size 128, GQA=1 achieves roughly 4200 tokens/s, GQA=3 achieves roughly 5200 tokens/s, GQA=4 achieves roughly 5800 tokens/s, and GQA=9 achieves roughly 7500 tokens/s. The gains from GQA are comparable in magnitude to those from hidden size and ratio adjustments, confirming that inference efficiency requires optimizing all three factors simultaneously.

Cross-framework and cross-hardware consistency (Table 6): The throughput rankings (LLaMA-3.2 < Surefire) are preserved across all four combinations of serving framework (vLLM, SGLang) and GPU (A100, H200). The absolute throughput values differ substantially—SGLang on H200 achieves roughly 2–3× the throughput of vLLM on A100 for the same model—but the relative gains are robust. Surefire-1B achieves 25.5% improvement over LLaMA-3.2-1B on vLLM/A100 and 47% on SGLang/H200 (batch size 128). Surefire-3B achieves 42% improvement on vLLM/A100 (batch size 64) and 16.6% on SGLang/H200 (batch size 128). The variability in relative gains suggests some hardware- and framework-specific sensitivity but no reversal of the ranking.

Open-Weight Model Comparisons (Tables 7 and 8)

The paper compares its pretrained architectures against released open-weight checkpoints at the 1B and 3B scales using WikiText byte-level perplexity and inference throughput (vLLM on H200 at batch size 128).

1B scale (Table 7): LLaMA-3.2-1B-HF achieves WikiText perplexity of 1.5807 and throughput of 9306 tokens/s (4k/1k setup). Panda-1B achieves perplexity of 1.7016 and throughput of 6218 tokens/s—worse on both metrics than the HF checkpoint, likely because the open-weight model has been post-trained or trained on different data. Surefire-1B achieves perplexity of 1.7142 and throughput of 11283 tokens/s—21% higher throughput than LLaMA-3.2-1B-HF but at worse perplexity. OLMo-2-1B-HF achieves perplexity of 1.5798 and throughput of 7486 tokens/s (2k/1k setup only, since it does not support 4k context). The paper notes that OLMo-2-1B-HF is "relatively close to our predicted optimal design" with $r_{\text{mlp/attn}} = 3$, "but remains inference-inefficient due to its hidden dimension and GQA choices."

3B scale (Table 8): LLaMA-3.2-3B-HF achieves WikiText perplexity of 1.5164 and throughput of 4311 tokens/s. Panda-3B achieves perplexity of 1.6454 and throughput of 3335 tokens/s—both worse than the HF checkpoint. Surefire-3B achieves perplexity of 1.6462 and throughput of 4842 tokens/s—12.3% higher throughput but worse perplexity. Qwen2.5-3B-HF achieves perplexity of 1.6185 and throughput of 6470 tokens/s—50% higher throughput than Surefire-3B but worse perplexity (1.6185 vs. 1.6462). The paper notes this illustrates the accuracy-efficiency tradeoff: "LLaMA-3.2-3B-HF achieves good accuracy but is not inference-efficient, while Qwen2.5-3B-HF is inference-efficient but less accurate."

These comparisons against released checkpoints should be interpreted cautiously. The paper's models are trained only on Dolma-v1.7 data with a fixed training recipe, while the open-weight models may use different data mixtures, post-training procedures, and tokenizers—all of which affect perplexity independently of architecture. The fair comparison is between the paper's baselines (LLaMA-3.2-1B and LLaMA-3.2-3B trained under the same setup) and the paper's architectures (Panda, Surefire), which isolates the architectural effect.

Ablation Studies and Robustness Checks

• Outlier architectures in scaling law fitting (Figure 25, left vs. center): Including models with extreme MLP-to-attention ratios ($r_{\text{mlp/attn}} < 0.5$ or $> 5$) in the scaling law fitting degrades Spearman correlation from 0.7451 to 0.3309 on Task 3. The paper restricts the fitting data to $r_{\text{mlp/attn}} \in [0.5, 5]$, which covers all practical open-weight configurations. The functional form $c_0 + c_1 \log x + c_2/x$ appears unable to capture the loss behavior at extreme ratios, and including these points biases the fit in the practically relevant central region.

• Multiplicative vs. additive calibration (Figure 25, left vs. right): On Task 3, both calibration forms achieve MSE = 0.0001 and Spearman = 0.7451—virtually identical. The paper uses multiplicative for the main results, arguing it is "more natural for extrapolation" because architectural effects are more likely to be proportional than absolute as $L_{\text{opt}}$ decreases with scale. At the scales studied (80M–1B), the distinction does not yet produce measurable differences.

• Joint non-separable calibration (Figure 26): The alternative formulation that combines $d_{\text{model}}$ and $r_{\text{mlp/attn}}$ into a single combined variable $(dr)/\sqrt{N}$ performs significantly worse, with MSE = 0.0067 and Spearman = 0.2108 (vs. 0.0001 and 0.7451 for the separable multiplicative form). This is a critical ablation—it empirically validates the separability assumption that the conditional scaling law relies on. The hidden-size optimum and the MLP-to-attention ratio optimum are genuinely independent; you cannot compensate for one with the other.

• Fitting data strategy at scale (Figure 8, Table 2): When predicting 3B architectures, fitting the law on all available smaller-scale data (80M+145M+297M+1B) yields Spearman = 0.500 (Figure 8, left). Fitting on only 1B data yields Spearman = 1.000 (Figure 8, right)—perfect rank correlation on the 3B test set. The resulting architecture (Panda-3B°) achieves lower training loss than Panda-3B (2.606 vs. 2.619) and identical downstream accuracy (62.5%). This is a non-obvious finding: including data from very small models can hurt extrapolation rather than help it, because the law's coefficients shift with scale. The paper recommends fitting on models within approximately one-third of the target scale.

• Inference efficiency across serving frameworks and hardware (Table 6): The efficiency gains transfer across vLLM and SGLang on both A100 and H200 GPUs. For Surefire-1B, the throughput improvement over LLaMA-3.2-1B ranges from 18–47% depending on the combination. For Surefire-3B, it ranges from 6–42%. The gains are largest on the lower-throughput combinations (vLLM/A100) and smaller on the higher-throughput combinations (SGLang/H200), suggesting that the architectural advantages are partially absorbed by faster hardware or more efficient software, but never reversed.

• Qwen3 architecture variants (Figures 12–14, 16–18, 20–22): The paper replicates the hidden size, MLP-to-attention ratio, and GQA throughput ablations on Qwen3-0.6B, 1.7B, and 4B model configurations, confirming that the trends—larger hidden size, higher MLP-to-attention ratio, and larger GQA all improve throughput—are not specific to the LLaMA architecture family. This is a robustness check against the possibility that the throughput benefits are an artifact of LLaMA's specific attention or MLP implementation.

• MoE inference efficiency (Figure 27, Appendix M): The paper extends the throughput analysis to mixture-of-experts architectures, showing that larger hidden sizes and higher Active-Experts-to-Attention ratios improve MoE inference throughput, "consistent with observations in dense models." This is presented as preliminary evidence that the architectural principles may generalize beyond dense transformers, though the paper explicitly does not establish scaling laws for MoE architectures.

Critical Assessment

Claim: The conditional scaling law reliably predicts optimal architectural choices. What the experiments actually demonstrate: On Tasks 1–3, the law predicts 1B architecture rankings with Spearman = 0.745 and MSE = 0.0001 (Figure 6, right). The loss-minimizing architecture identified by the law (Panda-1B) does achieve the lowest training loss among all trained 1B variants (Figure 7, left). On the 3B scale, fitting on 1B data yields perfect rank correlation (Spearman = 1.00, Figure 8, right), and the predicted architecture (Panda-3B°) achieves lower loss than the LLaMA-3.2-3B baseline (2.606 vs. 2.625, Table 2). However, this perfect correlation is based on only a handful of 3B test architectures—the paper does not state how many 3B variants were trained for the test set, and with a small test set, Spearman = 1.00 can arise from evaluating on 2–3 points. The predictive accuracy for the 3B scale from smaller-scale data (80M–297M) is substantially weaker (Spearman = 0.500, Figure 8 left). The claim "reliably predicts" should be qualified: the prediction is reliable when the fitting data includes models within roughly 3× of the target scale. Extrapolating from 80M to 3B (a 37.5× increase) without intermediate-scale data would likely fail.

Claim: Optimized architectures achieve up to 2.1% higher accuracy and 42% greater inference throughput than LLaMA-3.2. The 2.1% accuracy gain and 42% throughput gain are for different architectures at different scales. Panda-1B achieves the 2.1% accuracy gain (57.0% vs. 54.9%, Table 1) with throughput lower than Surefire-1B. Surefire-3B achieves the 42% throughput gain (1613.32 vs. 1136.61 tokens/s on vLLM/A100 at batch size 64, Table 6) with accuracy only 0.7% higher than LLaMA-3.2-3B (62.6% vs. 61.9%, Table 1). No single architecture achieves both headline numbers simultaneously. This is not a flaw—it reflects the genuine accuracy-efficiency tradeoff the paper studies—but the abstract and introduction language could be read as claiming a single architecture delivers both gains.

Claim: The resulting models outperform existing open-source baselines. This claim requires careful unpacking. Panda-1B and Surefire-1B outperform their in-house LLaMA-3.2-1B baseline (trained under the same data and hyperparameters), but Table 7 shows that LLaMA-3.2-1B-HF (the released checkpoint) achieves substantially better WikiText perplexity (1.5807 vs. 1.7016 for Panda-1B, 1.7142 for Surefire-1B). The paper's architectures are optimized against the in-house baseline, not against the released checkpoint, which benefits from Meta's data curation, training infrastructure, and possibly post-training. The fair claim is that architectural optimization provides gains over an identically-trained baseline of the default architecture, not that the resulting models are state-of-the-art at their scale.

Weaknesses and missing experiments:

• No 7B validation. The paper acknowledges this as a limitation (Section 7), stating it is due to "limitations in resources and time." The 3B results already show diminishing accuracy returns from architectural optimization (0.6% gain at 3B vs. 2.1% at 1B, Table 1). It would be valuable to know whether this trend continues, and whether the 1B-fitted law can usefully predict 7B optimal architectures (which would require fitting on models around 2–3B, within the paper's recommended one-third-scale guideline).

• Single training data distribution. All models are trained on Dolma-v1.7 sampled proportionally to source contribution. The paper does not investigate whether the optimal architecture depends on the data distribution—for instance, whether code-heavy training data favors a higher MLP-to-attention ratio or whether multilingual data shifts the optimum. This is a plausible dependence that could limit the generality of the found optima.

• Fixed training recipe. All models use the same hyperparameters (learning rate, batch size, optimizer settings within each scale tier). It is possible that the optimal hyperparameters interact with architecture—a larger hidden size might benefit from a different learning rate or batch size—and the reported loss differences could partially reflect hyperparameter-architecture interactions rather than pure architectural effects. The paper does not ablate this.

• Small number of 3B test architectures for Spearman = 1.00. The perfect correlation in Figure 8 (right) is impressive but potentially fragile. If only 3–5 test architectures were evaluated, the Spearman could be inflated. The paper does not report the number of test points for this evaluation, and the configuration table (Table 4, Appendix D) lists only variants up to 1B in detail—3B variants are not exhaustively enumerated.

• Downstream evaluation limited to pretrained models. The paper evaluates zero-shot accuracy without any fine-tuning or instruction tuning. Given that Petty et al. (2023) found that architectural depth matters for fine-tuning generalization, it is plausible that the optimal architecture for pretraining loss differs from the optimal architecture for downstream performance after fine-tuning. The paper acknowledges this limitation (Section 7) but provides no empirical investigation.

• Inference efficiency measured on single GPU. All throughput measurements use a single A100 or H200 GPU with 4096/1024 input/output tokens. In multi-GPU deployments, model parallelism or tensor parallelism could interact with architecture in ways that change the optimal choices—for instance, a wider model may parallelize differently than a deeper one. The paper's throughput results should be understood as reflecting single-GPU serving, not distributed deployment.

• No confidence intervals. The paper reports point estimates for MSE, Spearman, and throughput without confidence intervals or standard errors. With 5 repeated runs for throughput, the variance should be quantifiable. The scaling-law fit could report parameter uncertainty, which would be especially informative for the cross-scale extrapolation tasks.

Where the claims hold conditionally:

• The throughput gains from architectural optimization are largest on the A100 with vLLM and diminish on the H200 with SGLang (Table 6). The claim of "42% greater inference throughput" (abstract) refers specifically to Surefire-3B on vLLM/A100 at batch size 64—other configurations show smaller gains (16.6% on SGLang/H200 for the same model at batch size 128).
• The accuracy gains from architectural optimization decrease with scale: 2.1% at 1B, 0.6% at 3B (Table 1). Extrapolating this trend, the gains at 7B and beyond might be very small, making the practical value of architectural optimization questionable at larger scales. The paper does not discuss this trend or its implications.
• The scaling law's predictive quality depends on the fitting data being within approximately one-third of the target scale (Section 5.1). Extrapolating across larger scale gaps requires intermediate fitting data, which partially defeats the cost-saving purpose of the scaling law—you need to train models at a substantial fraction of the target scale anyway.

What would strengthen the paper (not criticisms, but identified gaps):

• Training a 7B model using the predicted optimal architecture from 1B- or 3B-fitted laws to test whether the U-shaped relationships and separable effects persist at larger scales.
• Ablation of whether the optimal architecture depends on training data distribution (e.g., code-heavy vs. web-text-heavy vs. multilingual).
• A multi-GPU inference efficiency study to determine whether the optimal architecture for throughput changes under model parallelism.
• Confidence intervals on throughput measurements and scaling-law parameter estimates.
• Explicit reporting of how many architectures were in each test set for the progressive fitting tasks, so readers can assess the reliability of the reported Spearman values.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Not Accounted for in the Search Framework

The assumption or constraint. The paper's search framework (Algorithm 1, Section 3.4) and the conditional scaling law (Equation 3, Section 3.3) both depend on knowing the reference loss $L_{\text{opt}}(N, D)$—the Chinchilla-optimal loss for the target parameter and token budget. For $N_{\text{non-embed}} < 1\text{B}$, the paper "empirically searched over architecture variants to find the optimal loss" (Section 4, Fitting Scaling Laws) by training a full grid of architectural variants and selecting the minimum. For extrapolation to larger scales, $L_{\text{opt}}$ is obtained by fitting the Chinchilla functional form to these empirical minima, which still requires training the full architectural grid at the fitting scales.

The consequence. The headline results—2.1% accuracy gain and 42% throughput improvement—are computed after the architectural design space has been exhaustively explored at smaller scales. The cost of this exploration is the training of over 200 models spanning 80M to 1B parameters on up to 100B tokens each. The paper's total training budget for fitting the scaling law is the sum over all trained variants in Table 4 (Appendix D): at least 80M × 100 tokens × (number of 80M variants) + 145M × 100 tokens × (number of 145M variants) + 297M × 100 tokens × (number of 297M variants) + 1B × 100 tokens × (number of 1B variants). With approximately 200 variants and an average size of roughly 200M parameters, this represents on the order of 200 × 200M × 100 × 100M ≈ $4 \times 10^{21}$ FLOPs for the fitting data alone—comparable to the cost of training a single 7B model. If the practitioner's goal is to find the optimal architecture for a 3B model, this fitting cost may exceed the savings from the improved architecture, depending on the inference volume.

What evidence exists in the paper. The paper does not report the total FLOPs or GPU-hours consumed by the fitting process. The progressive fitting strategy (Tasks 1–3, Section 4) quantifies how many model scales are needed for extrapolation but does not account for the number of variants required at each scale to fit the U-shaped curves. The ablation on fitting data strategy (Section 5.1, Figure 8) shows that fitting on 1B data alone suffices for predicting 3B architectures, which reduces the required fitting scale span but still requires training a full grid of 1B architectural variants (at least 17 variants from Table 4). The inference FLOPs analysis (Appendix K) addresses per-token inference cost but not the amortized cost of the architectural search.

Mitigation status. The paper acknowledges this implicitly through its progressive fitting strategy—the entire premise is to make the search cheaper by fitting on smaller models—but the absolute cost of even the "cheap" fitting sweep is never quantified or amortized against the savings. The paper does not investigate whether the number of architectural variants can be reduced, whether active learning or Bayesian optimization could find the optimum with fewer trained models, or whether the U-shaped curves can be fitted from partial data (e.g., training only three points per $N$ to estimate the optimum rather than 30–60). Section 7 (Limitations and Future Work) does not mention the fitting cost as an open challenge.

---

Extrapolation Across Scale Is Fragile Without Intermediate Fitting Data

The assumption or constraint. The conditional scaling law assumes that the learnable parameters ($a_0$, $a_1$, $a_2$, $b_0$, $b_1$, $b_2$ in Equation 3) are "shared across all $N$, $D$" (Section 3.3). This means that the U-shaped relationships between architecture and loss are assumed to be identical at all model scales up to a rescaling of the x-axis (normalizing $d_{\text{model}}$ by $\sqrt{N}$). The paper's validation directly tests this assumption by extrapolating across scale gaps.

The consequence. The extrapolation performance degrades sharply as the gap between fitting scale and target scale grows. On Task 3 (fit on 80M+145M+297M, evaluate on 1B), Spearman correlation is 0.745—useful but imperfect. When extrapolating from the full 80M–1B range to 3B, Spearman drops to 0.500 (Figure 8, left)—barely above chance for architecture ranking. The paper's own finding is that "it is often sufficient, and sometimes preferable, to fit the law using models within a closer size range to the target, such as about one third of its scale" (Section 5.1). This means that to design a 7B model, you would need to train a grid of architectural variants at approximately 2–3B parameters—which is itself a substantial training investment. The scaling law does not provide a cheap shortcut from toy-scale models to production-scale models; it requires fitting data at a meaningful fraction of the target scale.

What evidence exists in the paper. Figure 8 shows this fragility directly. Fitting on 80M+145M+297M+1B yields Spearman = 0.500 on 3B evaluation (left panel); fitting on 1B only yields Spearman = 1.000 on 3B (right panel). The paper acknowledges that "the law's coefficients shift with model size" (Section 5.1), which is an admission that the sharing assumption is only approximately true at small scale gaps and breaks down at larger gaps. The paper does not systematically characterize how the correlation decays with the fitting-to-target scale ratio—no curve of Spearman vs. scale gap is provided.

Mitigation status. The paper's recommended practice—fit within one-third of the target scale—is a pragmatic workaround but partially defeats the purpose of a scaling law. If you must train dozens of models at 2B to design a 7B model, the savings relative to training a few candidate 7B architectures directly are less dramatic than the abstract suggests. The paper does not propose methods to improve cross-scale extrapolation (e.g., making the calibration parameters themselves functions of $N$, or incorporating scale-dependent corrections).

---

Accuracy Gains Diminish with Scale, Threatening Practical Value at Larger Sizes

The assumption or constraint. The paper validates its approach at 1B and 3B scales and argues that the architectural principles identified (larger hidden size, lower MLP-to-attention ratio for accuracy; higher ratio for throughput) are universal. It does not train or evaluate models above 3B parameters, explicitly noting that "due to limitations in resources and time, our evaluation does not extend to 7B models" (Section 7).

The consequence. The accuracy gains from architectural optimization are already diminishing: from 2.1% average downstream improvement at 1B (Panda-1B vs. LLaMA-3.2-1B, Table 1) to 0.6% at 3B (Panda-3B vs. LLaMA-3.2-3B, Table 1). If this trend continues, the gains at 7B and beyond may be well under 0.5%, possibly within the noise of training stochasticity and downstream evaluation variance. The throughput gains from architectural optimization may also shift at larger scales—the inference FLOPs analysis (Appendix K) shows that the $2 n_{\text{layers}} T d_q$ term becomes proportionally smaller as the parameter count grows (since $d_q$ scales sub-linearly with $N$ for fixed $d_{\text{head}}$), so the efficiency benefits of reducing $d_q$ through larger $d_{\text{model}}$ or higher $r_{\text{mlp/attn}}$ may shrink. If both accuracy and efficiency gains diminish, the practical value of architectural optimization at production scales (7B, 13B, 70B) is uncertain. The paper's search framework could recommend training dozens of small-scale models to find an architecture that yields negligible improvement at the target scale—a poor return on the fitting investment.

What evidence exists in the paper. The diminishing trend is visible but not discussed. Table 1 shows the 1B → 3B accuracy gain contraction. The paper provides no explicit analysis of this trend and no extrapolation of what gains would be expected at 7B. The ablation on fitting data strategy (Table 2) shows that Panda-3B° achieves 2.606 training loss vs. 2.625 for LLaMA-3.2-3B (0.019 nats improvement) but the same downstream accuracy as Panda-3B (62.5%), suggesting that training loss improvements do not monotonically translate to downstream gains—a further complication for predicting practical impact at scale.

Mitigation status. The paper acknowledges the absence of 7B validation as a limitation (Section 7) but does not discuss the trend or its implications. It frames the lack of 7B results as purely a resource constraint, not as a potential signal that the approach may not scale. The future work section does not prioritize investigating whether the diminishing returns are fundamental or specific to the 1B–3B transition.

---

The Search Framework Treats Inference Efficiency as an Empirically Measured, Hardware-Dependent Objective

The assumption or constraint. The constrained optimization in Equation (4) and Algorithm 1 treats inference throughput $\text{IN}(P)$ as an objective to be maximized, but $\text{IN}(P)$ is not expressed as an analytical function of architectural parameters. Instead, it is measured empirically on specific hardware (A100, H200) with specific serving frameworks (vLLM, SGLang) at specific token configurations (4096 input, 1024 output) and batch sizes (Section 4, Inference Setup). The paper states that "inference efficiency... depends heavily on hardware and inference configurations" and therefore "rather than solving for $\text{IN}(P)$ directly, we search over feasible configurations $P_i$ that satisfy the loss constraint... and select Pareto-optimal points" (Section 5.1).

The consequence. The optimal architecture found by the search is implicitly tuned to the specific hardware and framework used for measurement. If a practitioner deploys on different hardware (e.g., NVIDIA L40S, AMD MI300X, Google TPUv5, Apple Neural Engine), uses a different serving framework, or serves with different sequence length distributions (e.g., long-context applications with 32K+ tokens), the Pareto-optimal architecture may be different. The paper demonstrates cross-framework and cross-hardware consistency for the specific architectures it selected (Table 6), showing that Surefire-1B outperforms LLaMA-3.2-1B on both vLLM/A100 and SGLang/H200, but this does not guarantee that the ranking of all candidate architectures is invariant. An architecture that is Pareto-optimal on A100 might be dominated by a different architecture on H200 if the relative costs of attention vs. MLP compute or KV cache memory I/O change. Similarly, for long-context applications where the KV cache dominates memory usage, GQA and hidden size may have even larger relative effects, potentially shifting the optimal $r_{\text{mlp/attn}}$ toward even higher values.

What evidence exists in the paper. Table 6 shows throughput comparisons for the specific architectures (LLaMA-3.2 and Surefire) across four hardware-framework combinations. The relative gain varies substantially: Surefire-1B achieves 25.5% improvement on vLLM/A100 at batch size 128 vs. 47% on SGLang/H200 at batch size 128. This variability in magnitude suggests that the ranking could potentially shift in other configurations not tested. However, the paper does not sweep the full architectural grid on multiple hardware platforms to verify that the Pareto frontier is stable. The paper also fixes input length to 4096 and output length to 1024 for all measurements—there is no ablation over sequence length to test whether the optimal architecture changes for short-prompt (e.g., 128 tokens) or long-context (e.g., 32K tokens) scenarios.

Mitigation status. The paper partially addresses this by evaluating on two frameworks (vLLM and SGLang) and two GPU types (A100 and H200), showing consistent direction of improvement (Appendix F, G, Table 6). This is more thorough than most architecture-search papers, which typically evaluate on a single hardware configuration. However, the paper does not provide an analytical model of inference efficiency that would allow practitioners to predict the optimal architecture for their specific deployment setup without re-running the search. The inference FLOPs analysis (Appendix K) provides a first-principles breakdown of compute costs but does not model memory bandwidth, KV cache I/O, or batch-size-dependent utilization effects—all of which influence throughput on real hardware. The paper's recommendation to "search over feasible configurations... and select Pareto-optimal points" implicitly assumes the practitioner has the resources to re-run the throughput benchmarking on their target hardware.

---

The Fixed Number of Layers Restricts Architectural Expressiveness and May Not Generalize to Optimal Depth Configurations

The assumption or constraint. The paper "fix[es] the number of layers and study[s] the effect of other architectural factors" (Section 1). The motivation is that "cutting layers tends to impair the model's generalization after fine-tuning" (Section 1, citing Petty et al., 2023). All 80M–297M models use $n_{\text{layers}} = 12$; 1B models use $n_{\text{layers}} = 16$; 3B models use $n_{\text{layers}} = 28$. The choice of $n_{\text{layers}}$ at each scale follows the LLaMA-3.2 convention and is not optimized by the search framework.

The consequence. The optimal architecture found by the search is conditional on the fixed layer count. It is possible—and arguably likely, given that open-weight models at similar parameter counts vary substantially in depth (Qwen2.5-3B uses 36 layers vs. LLaMA-3.2-3B's 28, Table 8)—that a jointly optimal configuration of depth, hidden size, and MLP-to-attention ratio would differ from the conditional optimum found by the paper. For instance, a deeper but narrower model (more layers, smaller hidden size) might achieve better accuracy-efficiency tradeoffs than the paper's architectures, but this is never explored. The inference cost tradeoff is complex: deeper models have smaller per-layer KV caches (since $d_{\text{model}}$ is smaller) but more layers, meaning more sequential attention operations. On hardware with high memory bandwidth, depth-constrained width-optimized architectures (the paper's approach) may be optimal; on hardware with lower memory bandwidth relative to compute, wider-but-shallower architectures may dominate. By fixing depth, the paper cannot capture this dimension of the tradeoff space.

What evidence exists in the paper. The paper provides no ablation or analysis of how varying $n_{\text{layers}}$ under a fixed $N_{\text{non-embed}}$ would affect accuracy or throughput. Table 7 and Table 8 (Appendix H) show that existing open-weight models at the same parameter scales use different layer counts—OLMo-2-1B uses 16 layers (same as LLaMA-3.2-1B) but Qwen2.5-3B uses 36 layers (vs. 28 for LLaMA-3.2-3B)—which is presented as evidence that architecture matters but not as a systematic exploration of depth. The paper's defense of fixed depth ("cutting layers tends to impair generalization after fine-tuning") cites one prior work (Petty et al., 2023) but does not empirically verify that this concern applies at the scales and with the training recipe used in the paper.

Mitigation status. The paper explicitly scopes its contribution to the architectural dimensions it studies (Section 1: "we fix the number of layers and study the effect of other architectural factors") but does not treat this as a limitation to be addressed in future work. The fixed-depth constraint is presented as a deliberate design choice, not as an acknowledged gap. A reader deploying the paper's methodology would need to independently determine whether the fixed depth (inherited from LLaMA-3.2 conventions) is appropriate for their use case, or whether a deeper or shallower model would be preferable. The conditional scaling law provides no guidance on this question.

---

Pretraining Loss Reductions Do Not Reliably Translate to Downstream Accuracy Gains

The assumption or constraint. The conditional scaling law is fitted to and predicts training loss (cross-entropy loss on the Dolma-v1.7 training data). The search framework (Equation 4) constrains architectures by their predicted training loss and selects Pareto-optimal points. The underlying assumption is that lower training loss implies better downstream task performance, which is the standard scaling-law assumption (Hoffmann et al., 2022) but is known to be noisy at small loss differences.

The consequence. The relationship between training loss improvements and downstream accuracy gains is inconsistent in the paper's own results. At the 1B scale, Panda-1B improves training loss by 0.021 nats over LLaMA-3.2-1B and downstream accuracy by 2.1%—a substantial and correlated gain (Table 1). At the 3B scale, Panda-3B improves training loss by 0.006 nats but downstream accuracy by only 0.6% (Table 1). Panda-3B° improves training loss by 0.019 nats over LLaMA-3.2-3B (a similar magnitude to Panda-1B's improvement) but achieves the same downstream accuracy as Panda-3B (62.5%), which only improved loss by 0.006 nats (Table 2). This is a clear case where a larger training loss improvement (0.019 vs. 0.006 nats) produces no additional downstream benefit. The lack of monotonic translation from training loss to downstream performance means that the constrained optimization (Equation 4), which uses predicted training loss as a proxy for accuracy, may select architectures that are Pareto-optimal in the (throughput, training loss) space but not in the (throughput, downstream accuracy) space that practitioners actually care about.

What evidence exists in the paper. Table 2 directly demonstrates the breakdown: Panda-3B° achieves training loss 2.606 vs. Panda-3B's 2.619, but both achieve 62.5% average downstream accuracy. This is not discussed as a limitation. The paper's downstream evaluation (Tables 9 and 10) shows that individual benchmark scores can move in opposite directions even when aggregate accuracy improves—for example, Panda-1B improves over LLaMA-3.2-1B on most benchmarks but slightly degrades on ARC-Challenge (28.9% vs. 29.8%), showing that the architectural changes do not uniformly benefit all task types. The paper does not investigate whether the optimal architecture for one downstream task differs from the optimal architecture for another, or for the average.

Mitigation status. The paper does not acknowledge this as a limitation. The training-loss-to-downstream-accuracy gap is visible in the data (Tables 1, 2, 9, 10) but is not discussed or analyzed. The search framework's loss constraint $L_t$ (Equation 4) is set based on training loss (matching the LLaMA-3.2 baseline training loss), but the paper validates the resulting architectures on downstream accuracy as a post-hoc check rather than incorporating downstream performance into the search objective. This is consistent with standard scaling-law practice but represents a gap between what the method optimizes and what the user values. Future work could incorporate downstream evaluation directly into the search loop, but this would dramatically increase the cost (requiring full evaluation on multiple benchmarks for each candidate architecture rather than just measuring training loss).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around scaling laws from a parameter-and-data-centric view toward one where architecture is a first-class variable in the optimization, not an afterthought to be chosen by precedent or intuition. The Chinchilla framework (Hoffmann et al., 2022) answers the question "how many parameters and how many training tokens should I use for a given FLOPs budget?" but is silent on how those parameters should be organized. This paper provides a principled answer to that second question—and shows that the answer matters concretely, delivering up to 2.1% higher accuracy and 42% greater throughput at the 1B and 3B scales (Table 1, Table 6) purely through architectural reallocation under fixed parameter and token budgets.

The magnitude of this shift is best characterized as a methodological reframing with practical tooling, not a paradigm shift in the Kuhnian sense. The paper does not overturn the Chinchilla framework—it extends it. The core insight that training loss follows power-law behavior in $N$ and $D$ remains intact. What changes is that $N$ is no longer treated as a sufficient statistic for model capability. Two models with the same $N$ and $D$ can have measurably different training losses and inference throughputs, and the paper provides a framework for predicting those differences from small-scale sweeps and exploiting them at deployment scale. This is an incremental but practically significant refinement—the difference between knowing how big a model to build and knowing how to build it.

The reconciliation of architectural heterogeneity across open-weight models is perhaps the paper's most immediately clarifying contribution. Prior to this work, a practitioner surveying the landscape would see LLaMA-3.2-1B with r_mlp/attn = 4.8 and GQA=4, Qwen3-0.6B with ratio 1.5 and GQA=2, OLMo-2-1B with ratio 3 and GQA=1, and have no framework for deciding which configuration was better or why they diverged. The paper's U-shaped loss curves (Figures 4 and 5) demonstrate that there exist genuine interior optima for both hidden size and MLP-to-attention ratio, and that many open-weight models sit substantially off these optima—LLaMA-3.2-1B's ratio of 4.8 places it on the right-hand rising limb of the U-curve, where accuracy is sacrificed for inference efficiency (or the efficiency gain was valued without awareness of the accuracy penalty). This doesn't mean LLaMA-3.2-1B is "wrong"—it may reflect a deliberate efficiency-over-accuracy tradeoff, which the Surefire architecture formalizes explicitly—but it means the field now has a language for discussing architectural choices as points on a continuous Pareto frontier rather than as discrete, incommensurable design decisions.

The paper also redirects research attention from exotic architectural innovations toward principled allocation within standard architectures. The finding that simple continuous adjustments—making the hidden size larger, shifting parameters from attention to MLP, increasing GQA—can deliver throughput gains comparable to or exceeding those claimed by more radical architectural modifications (sparse attention, state-space models, mixture-of-experts routing innovations) suggests that a substantial fraction of achievable efficiency is left on the table by suboptimal configurations of standard dense transformers. This does not make architectural innovation irrelevant, but it raises the bar for what counts as a meaningful improvement: a new attention mechanism should be compared not just against a default transformer but against a compute-optimal configuration of the default transformer, since the latter may already recover much of the claimed gain.

The conditional calibration framework—using the Chinchilla optimum as a reference point and learning separable multiplicative corrections for architectural factors—is a methodological template that extends beyond the specific factors studied here. The paper demonstrates that some architectural choices (hidden size, MLP-to-attention ratio) admit clean parametric modeling with consistent U-shaped behavior across scales, while others (GQA) do not and require separate discrete search. This distinction between continuously tunable and discrete, idiosyncratic architectural factors is a conceptual contribution in itself—it tells future researchers which dimensions of architecture are amenable to scaling-law treatment and which will require case-by-case optimization. The failure of the joint non-separable calibration (Appendix J, Figure 26: Spearman dropping from 0.745 to 0.211 when d_model and r_mlp/attn are combined into a single variable) provides a negative result that is methodologically instructive: not all architectural parameters interact, and forcing them into a unified law degrades rather than improves prediction.

Finally, the paper establishes that architectural optima are approximately scale-invariant within the 80M–3B range when variables are appropriately normalized (d_model / sqrt(N), r_mlp/attn). The fact that fitting on 1B data produces perfect rank correlation on 3B architectures (Spearman = 1.00, Figure 8 right) while extrapolating from ≤297M to 3B yields weak correlation (Spearman = 0.50, Figure 8 left) bounds the regime where this scale invariance holds: the fitting scale should be within roughly 3× of the target scale. This is not a universal scaling law in the Chinchilla sense—it doesn't predict optimal architecture at 70B from measurements at 80M—but it is a practical engineering guideline: to design a model at scale $S$, train a grid of architectural variants at scale approximately $S/3$ and fit the conditional law.

Follow-Up Research This Work Enables

Stress-testing the scale invariance boundary: where does the U-shaped optimum shift? The paper finds that the optimal d_model/sqrt(N) and r_mlp/attn are approximately constant from 80M to 3B, but the diminishing accuracy gains (2.1% at 1B → 0.6% at 3B, Table 1) and the degradation of cross-scale prediction when fitting data includes models more than ~3× smaller (Spearman 0.50 vs. 1.00, Figure 8) hint that the invariance may break at larger scales. A direct follow-up would train a grid of architectural variants at 7B parameters—varying d_model and r_mlp/attn in a range around the optimum predicted by a 1B-fitted and a 3B-fitted law—and measure whether the U-shaped curves persist with the same minimum, whether the optimum shifts systematically (e.g., toward higher r_mlp/attn as scale increases, consistent with the trend in LLaMA and Qwen families), and whether the separable calibration form continues to fit. This would establish the practical ceiling of the method: if the optimum is stable to 7B, architectural optimization is a one-time cost amortized across the entire model family; if it shifts, scaling laws need scale-dependent correction terms.

Incorporating depth (number of layers) as an architectural variable in the conditional framework. The paper's deliberate choice to fix n_layers (Section 3.1) leaves open the question of whether the optimal hidden size and MLP-to-attention ratio depend on depth, and whether depth itself can be optimized under a fixed parameter budget. A natural extension would train a three-dimensional grid varying n_layers, d_model, and r_mlp/attn at a fixed N_non-embed (e.g., 300M, to keep training costs manageable) and fit a conditional scaling law with an additional depth term. The key questions: does depth exhibit a U-shaped optimum analogous to hidden size? Do depth and hidden size interact—is the optimal d_model a function of n_layers? And does the Petty et al. (2023) finding that shallow models degrade after fine-tuning manifest as a predictable downstream accuracy penalty that should be incorporated into the loss constraint in Equation 4? The inference cost dimension is particularly interesting for depth: deeper models have smaller per-layer hidden sizes (reducing KV cache per layer) but more layers (increasing sequential computation and total KV cache), creating a complex throughput tradeoff that the paper's single-GPU measurements (Table 6) would need to be extended to characterize.

Architecture-conditioned data scaling: does optimal architecture depend on training data volume or composition? The paper fixes training tokens at 100 × N_non-embed for all models and uses a single data distribution (Dolma-v1.7 with proportional sampling). It is plausible—but unknown—that the optimal r_mlp/attn shifts with training data volume: at small D, a larger MLP might overfit or underfit differently than attention, and the U-shaped curve might change shape or location. Similarly, data composition could matter: code-heavy training data, which contains highly structured, long-range dependencies, might favor more attention capacity (lower r_mlp/attn), while knowledge-intensive data (Wikipedia, textbooks) might favor more MLP capacity (higher r_mlp/attn). A follow-up would replicate the paper's 80M or 145M architectural grid at multiple data volumes (e.g., 20×, 50×, 100×, 200× N_non-embed) and/or on domain-specific data mixtures (code-only, Wikipedia-only, multilingual) and test whether the fitted calibration parameters a_i, b_i change. If the optimum is data-stable, the paper's methodology applies broadly; if it shifts, architectural optimization becomes data-dependent, and practitioners would need to fit the law on data representative of their deployment domain.

Training loss vs. downstream accuracy: closing the gap for architecture search. The paper's search framework optimizes architectures for training loss and validates downstream accuracy post-hoc, but Table 2 reveals a concrete failure of this proxy: Panda-3B° achieves 0.013 nats lower training loss than Panda-3B (2.606 vs. 2.619) but identical downstream accuracy (62.5%). This is a single data point but a concerning one—it suggests that beyond some threshold, further training loss reductions do not translate to meaningful capability improvements, and architectures optimized purely for loss may waste parameters on easily-compressible but practically irrelevant patterns. A methodologically important follow-up would replace the training loss constraint in Equation 4 with a downstream accuracy constraint, either by fitting a scaling law for downstream accuracy directly (analogous to Gadre et al., 2024) or by learning a mapping from training loss to downstream performance that accounts for the observed saturation. This would require evaluating candidate architectures on the full nine-benchmark suite (Table 9, 10) or a representative subset, making the search more expensive but better aligned with practitioner goals. A less expensive alternative would be to use a single diagnostic downstream task (e.g., LAMBADA or HellaSwag, which the paper shows are sensitive to architectural variation) as a proxy for the full average during the search phase.

Principled handling of discrete architectural choices: a theory for when GQA helps or hurts accuracy. The paper's finding that GQA has a "highly fluctuating, non-monotonic" relationship with training loss (Figure 24, Appendix I) and is therefore relegated to a separate local search (Algorithm 1) is pragmatically sound but intellectually unsatisfying. The paper provides a mechanistic hypothesis—when total heads are fixed, increasing GQA trades off query expressivity (more query heads, higher d_q) against KV expressivity (fewer KV heads, more sharing, lower d_kv)—but does not test it systematically. A follow-up would isolate this tradeoff by fixing the total attention head budget and varying GQA across a wide range while measuring both loss and per-layer attention pattern statistics (e.g., head diversity, KV cache utilization). If the hypothesis is correct, there should exist a predictable relationship between the optimal GQA and the ratio of query-to-KV dimensionality, and architectures where query expressivity is the bottleneck (e.g., tasks requiring fine-grained token-level attention) should prefer lower GQA while architectures where KV expressivity is less critical (e.g., tasks where attention patterns are diffuse and redundant) should tolerate higher GQA. This could lead to a task- or layer-dependent GQA assignment rather than a uniform one, which several recent architectures already employ informally (e.g., different GQA in early vs. late layers).

Extension to mixture-of-experts: do the same principles govern MoE efficiency-accuracy tradeoffs? The paper provides preliminary MoE throughput measurements (Figure 27, Appendix M) showing that larger hidden sizes and higher Active-Experts-to-Attention ratios improve MoE inference throughput, consistent with dense models. But the accuracy side is unexplored: MoE architectures introduce additional degrees of freedom (number of experts, active experts per token, expert granularity, routing mechanism) that interact with the attention-MLP parameter allocation in complex ways. A natural extension would apply the conditional scaling law framework to MoE models by treating the number of experts and active experts as additional architectural variables (analogous to Krajewski et al., 2024; Abnar et al., 2025), with the key question being whether the optimal r_mlp/attn differs between dense and MoE architectures at the same total active parameter count. MoE models concentrate MLP capacity in specialized experts while sharing attention across all tokens, which might shift the optimal ratio toward relatively more attention (since MLP capacity is already expanded through expert multiplicity) or toward relatively less attention (since the MLP is the primary locus of specialization and should be rich). The answer has direct practical implications for architectures like DeepSeek-V3 and Qwen3-MoE.

Practical Applications and Downstream Use Cases

Cost-efficient pretraining of model families under fixed inference SLAs. An organization planning to deploy a family of models at multiple scales (e.g., 1B for on-device, 3B for latency-sensitive cloud, 7B for quality-sensitive cloud) with fixed inference throughput targets can use the conditional scaling law to design the architecture at each scale without training a full grid at the largest size. Rather than inheriting the default architecture of whatever base model family they adopt (e.g., LLaMA-3.2, Qwen3), they would: (1) train architectural variant grids at 1B and identify the loss-minimizing architecture (Panda equivalent) and the throughput-maximizing architecture under their specific latency constraint (Surefire equivalent), (2) extrapolate to 3B and 7B using the fitted law, and (3) train only the predicted optimal configurations at the larger scales for final validation. The paper's finding that fitting on models within ~3× of the target scale is sufficient (Figure 8) means the 1B grid (which the paper trained with ~55 variants, Table 4) can guide 3B design, and a 3B grid can guide 7B design. The concrete savings: training a 7B model on 100B tokens costs approximately 6 × 7e9 × 100e9 ≈ 4.2e21 FLOPs. Training the full architectural grid at 7B might require 20–30 such runs. Training a grid at 3B (costing ~6 × 3e9 × 100e9 ≈ 1.8e21 FLOPs per run, or 36–54e21 total for 20–30 runs) plus a single 7B validation run costs roughly 40–55e21 FLOPs vs. 84–126e21 for the full 7B grid—a ~2× reduction in total pretraining FLOPs for architecture design.

Hardware-specific model customization for edge and mobile deployment. The paper's throughput measurements show that the relative gain from architectural optimization varies with hardware: Surefire-1B achieves 47% improvement on SGLang/H200 but 25.5% on vLLM/A100 at the same batch size (Table 6). For edge deployments on mobile GPUs, laptop NPUs, or specialized inference accelerators with very different compute-to-memory-bandwidth ratios than datacenter GPUs, the optimal architecture is likely different still—a memory-bandwidth-constrained device benefits disproportionately from KV cache reduction (favoring larger d_model and higher GQA), while a compute-constrained device might favor smaller hidden sizes with larger batch parallelism. A practitioner can apply the paper's search framework (Algorithm 1) by measuring throughput on the target edge hardware for a small grid of architectural variants (e.g., at 300M parameters, cheap to train and profile), fitting the conditional scaling law, and solving for the Pareto-optimal architecture at the deployment parameter count. The key benefit is that the expensive step—training many model variants to convergence to measure loss—can be done once on GPU, while only the cheap step—inference profiling—needs to be repeated per target hardware. The scaling law decouples the accuracy measurement (which is hardware-independent) from the throughput optimization (which is hardware-dependent).

Architecture-aware model merging and distillation. When distilling a large teacher model into a smaller student, or merging multiple fine-tuned models into a single multi-task model, the student or merged architecture is typically chosen by convention (same architecture as the teacher but smaller, or the same as some base model). The paper's framework suggests an alternative: given a target parameter budget for the student, use the conditional scaling law to design an architecture that maximizes inference throughput while matching or exceeding the accuracy of the default student architecture. This is particularly relevant for distillation because the teacher provides a stronger training signal than pretraining from scratch—the loss landscape of the student may be different, and the optimal architecture for learning from a teacher's output distribution may differ from the optimal architecture for learning from next-token prediction on raw text. A practitioner could train a small grid of student architectures with distillation at a manageable scale (e.g., distilling a 7B teacher into 300M student variants), fit the conditional law, and extrapolate to the target student size. The 42% throughput improvement demonstrated for Surefire-3B (Table 6) would directly reduce the serving cost of the distilled model without requiring any change to the distillation procedure itself.

When to Prefer This Method

The paper explicitly positions its search framework against the default practice of using off-the-shelf architectures (LLaMA-3.2, Qwen3) at a given parameter count. The decision rule is sharp and grounded in the paper's own empirical boundaries:

• Prefer the conditional scaling law plus constrained search when: (1) you have a fixed non-embedding parameter budget and training token budget, and your primary goal is to maximize inference throughput subject to a minimum accuracy constraint (Equation 4); (2) you are willing to fix the number of layers at a conventional value for your scale (e.g., following LLaMA-3.2 conventions, Section 3.1) and optimize only hidden size, MLP-to-attention ratio, and GQA; (3) you can afford to train a grid of architectural variants at a scale within roughly 3× of your target (the paper recommends fitting data at ~one-third of target scale for best extrapolation, Section 5.1); (4) your accuracy constraint can be expressed via pretraining loss rather than requiring direct downstream evaluation during search; and (5) you are operating in the 80M–3B range where the method is validated, or are willing to accept the risk of extrapolating beyond this range with diminished predictive quality (Spearman dropping from ~0.75 to ~0.50 when extrapolating across larger gaps, Figure 8).

• Stick with a default architecture (LLaMA-3.2, Qwen3, or similar) when: (1) you are working at scales well above 3B where the paper has no validation and where the diminishing accuracy gains (2.1% at 1B → 0.6% at 3B, Table 1) suggest the benefit of optimization may approach zero; (2) you need to vary the number of layers, use non-standard attention mechanisms, or explore architectural innovations beyond the factors studied—the conditional scaling law provides no guidance; (3) your deployment scenario has inference characteristics very different from the paper's measurement setup (e.g., long-context generation with 32K+ token sequences, multi-GPU tensor-parallel serving, or very different hardware like TPUs), where the Pareto-optimal architecture found on A100/H200 with 4096/1024 tokens may not transfer; or (4) the cost of the architectural search—training dozens of model variants at one-third target scale—exceeds the expected savings from the improved architecture given your inference volume. The paper provides no amortization analysis to guide this calculation, but a rough threshold can be derived: if your total inference tokens over the model's lifetime exceed ~100× the tokens used for the fitting grid, the efficiency gains from a 25–42% throughput improvement (Table 6) likely justify the search cost.

• Use the multiplicative calibration (Eq. 3, top) over the additive form when: (1) you are extrapolating to scales where the base loss $L_{\text{opt}}$ differs substantially from the fitting scales, since a multiplicative penalty maintains a constant relative impact while an additive penalty becomes an increasingly large fraction of the loss; (2) you believe architectural effects are proportional rather than absolute—e.g., a suboptimal hidden size hurts a 3B model proportionally as much as a 300M model. At the scales studied in the paper (80M–1B), both forms perform equivalently (Figure 25 right), so this choice matters only for larger-scale extrapolation.

• Exclude extreme architectural outliers from fitting data when: your target architecture falls within the conventional range of r_mlp/attn ∈ [0.5, 5] (which covers all current open-weight models, Section 5 Ablation of Outliers). Including architectures with ratios below 0.5 or above 5 in the fitting data degrades Spearman correlation from 0.745 to 0.331 (Figure 25 left vs. center) because the functional form c_0 + c_1 log x + c_2/x cannot capture the loss behavior in these extreme regimes. If you specifically need an architecture outside this range (unlikely for standard dense transformers), the conditional scaling law is not reliable and you should train the target architecture directly.