Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

🎯 Pitch

Most LLMs dramatically underuse vocabulary—your Llama2-70B likely needed a 7× larger vocabulary to reach its full potential. By uncovering a power-law relationship that scales vocabulary with compute budget, this work shows that simply increasing vocabulary size from 32K to 43K can boost downstream accuracy by nearly 3 points on ARC-Challenge at identical FLOPs, without any architecture changes.

1. Executive Summary

This paper investigates how vocabulary size impacts LLM scaling laws by training models ranging from 33M to 3B parameters on up to 500B characters with various vocabulary configurations, using the SlimPajama dataset. The authors propose three complementary approaches for predicting the compute-optimal vocabulary size—IsoFLOPs analysis (fitting power laws relating FLOPs to non-vocabulary parameters, vocabulary parameters, and training data), derivative-based estimation (finding the vocabulary size that minimizes FLOPs for a target loss), and parametric fit of the loss function (modifying Chinchilla scaling laws to incorporate vocabulary as an explicit term)—all converging on the finding that optimal vocabulary parameters follow a power-law relationship with compute budget ( $N_v^{\text{opt}} \propto N_{nv}^\gamma$ with $\gamma \approx 0.83$ ), meaning vocabulary should scale slower than non-vocabulary parameters but still grow substantially. For example, the paper predicts that Llama2-70B's optimal vocabulary size should have been at least 216K—roughly 7× larger than its actual 32K—and validates this empirically on 3B-parameter models, where adopting the predicted optimal vocabulary size of 43K instead of the conventional 32K improved ARC-Challenge accuracy from 29.1 to 32.0 under the same 2.3×10²¹ FLOPs budget, establishing that most existing LLMs systematically underallocate parameters to vocabulary and that this allocation should depend jointly on model size and training data volume.

2. Context and Motivation

The Core Problem: Vocabulary Size Has Been Ignored in Scaling Laws

The fundamental question this paper tackles is straightforward but has been systematically overlooked: given a fixed computational budget for pretraining, what vocabulary size should a language model use? This matters because every language model must commit to a vocabulary before training begins—it determines how text is split into tokens, how large the embedding and output layers are, and how efficiently the model can process language. Yet, despite extensive research on how to optimally allocate compute between model parameters and training data (the "scaling laws" line of work), vocabulary size has been treated as an arbitrary constant rather than a variable to be optimized.

The paper quantifies the consequences of this neglect concretely. As shown in their abstract and elaborated throughout Section 5, the authors find that Llama2-70B—one of the most widely deployed open-source LLMs—should have used a vocabulary of at least 216K tokens rather than its actual 32K, a sevenfold underestimation. In their validation experiments on 3B-parameter models, simply adjusting vocabulary size from the conventional 32K to their predicted optimal 43K (while keeping total FLOPs constant) improved ARC-Challenge accuracy from 29.1 to 32.0—a nearly 3 percentage point gain from a change that affects no other hyperparameters. These are not marginal improvements; they represent a significant allocation error that has been baked into the design of most major LLMs.

Why This Gap Exists: The Historical Treatment of Vocabulary in Scaling Laws

To understand why vocabulary has been neglected, one must examine how the two foundational scaling law papers treated it.

Kaplan et al. (2020) explicitly excluded vocabulary parameters from their analysis. In their framework, model size $N$ refers only to non-vocabulary parameters—the Transformer blocks, attention layers, and feed-forward networks—while the embedding and output layers (which together constitute approximately $2Vd$ parameters for vocabulary size $V$ and embedding dimension $d$ ) were treated as fixed overhead. Their predictive formulas therefore answer the question "given a compute budget, how many non-vocabulary parameters should I train for how many tokens?" while assuming the vocabulary size is already decided. This was a reasonable choice for their study, which used a fixed tokenizer, but it left open the question of whether that fixed vocabulary was itself optimal.

Hoffmann et al. (2022) in the Chinchilla paper continued this tradition—their loss predictor $L(N, D)$ depends on model size $N$ and training tokens $D$ , with vocabulary size again held fixed. Their famous finding that model parameters and training tokens should be scaled equally (both doubling when compute doubles) implicitly assumes a fixed vocabulary, meaning the optimality of that vocabulary was never tested.

The practical consequence of this omission is visible in the striking heterogeneity of vocabulary sizes across contemporary LLMs of similar scale. The paper highlights this concretely: Llama2-7B uses a vocabulary of 32K tokens, while Gemma-7B uses 256K—both have roughly the same number of total parameters (excluding vocabulary differences), yet their vocabularies differ by 8×. This is not a minor implementation detail; for a model with embedding dimension 4096, the difference between 32K and 256K represents approximately $2 \times 4096 \times (256K - 32K) \approx 1.8$ billion additional parameters dedicated solely to vocabulary, which could otherwise be allocated to additional Transformer layers. Some model series have made dramatic shifts between generations: Llama3-8B uses 128K vocabulary (4× Llama2's 32K), suggesting an industry intuition that larger vocabularies help, but without a principled framework for deciding how large.

The Two-Sided Nature of Vocabulary Size: Why "Bigger" Isn't Obviously Better

The paper's motivation rests on articulating a genuine tension that makes the optimal vocabulary size non-obvious. This tension operates on two fronts:

The case for larger vocabularies rests on tokenization fertility. A tokenizer with more distinct tokens can represent common words, subwords, and even multi-word phrases as single tokens. This increases what the paper calls tokenization fertility: for a fixed amount of raw text (measured in characters), a larger vocabulary produces fewer tokens. Since Transformer FLOPs scale with sequence length (tokens), and since what a model "learns from" is ultimately measured in characters of raw text, a larger vocabulary effectively lets the model process more semantic content per FLOP. Mathematically, the paper captures this via the compression ratio $f(V)$ —a function mapping vocabulary size $V$ to tokens-per-character—which decreases as $V$ increases (fewer tokens needed for the same text). This means that for a fixed training budget in characters, a larger vocabulary reduces the number of tokens the model processes, potentially freeing compute for other uses or effectively increasing data throughput.

Additionally, larger vocabularies enable models to capture a wider range of concepts directly. Rather than decomposing "photosynthesis" into three subword tokens that the model must learn to compose, a tokenizer with sufficient capacity can represent it as a single token with its own dedicated embedding, potentially making learning more efficient for frequent domain-specific terminology.

The case against larger vocabularies rests on undertraining of embedding parameters. Each token in the vocabulary has an associated embedding vector of dimension $d$ (the model's hidden size). For a 7B-parameter model with $d = 4096$ , adding 100K vocabulary entries adds $100K \times 4096 \approx 410$ million parameters—to the embedding layer alone (and another 410M to the output projection layer, for roughly 820M total new parameters). These parameters need to be trained, which means they need to appear in training data. The paper articulates the problem explicitly in Section 3, noting that "the risk of under-fitting for rare tokens increases with larger vocabulary sizes, especially in the data-constrained regime."

This is a genuine statistical problem: tokens that appear infrequently in the training corpus get few gradient updates, and their embeddings remain poorly estimated. The paper provides empirical evidence for this via SVD visualizations (Figure 9, Appendix A.3), showing that for a model with $N_{nv} = 85M$ , embeddings from a 64K vocabulary exhibit significantly more clustering (average Euclidean distance $D_{avg} = 0.952$ ) than those from a 16K vocabulary ( $D_{avg} = 1.011$ ), indicating that rare tokens' embeddings collapse toward each other due to insufficient training signals. This is the classic bias-variance tradeoff applied to tokenization: more vocabulary capacity gives more representational flexibility but reduces the effective data per parameter, risking overfitting.

The optimal vocabulary size emerges from balancing these two forces: making $V$ large enough to benefit from tokenization efficiency and conceptual coverage, but not so large that rare-token embeddings become degenerate noise. The key insight the paper develops—and which prior work missed entirely—is that this balance point depends on the total compute budget. As models scale up (more non-vocabulary parameters, more training data, more FLOPs), the "undertraining penalty" for rare tokens diminishes because the total number of tokens seen increases, shifting the optimal vocabulary size upward. This is the central empirical finding the three approaches converge on.

Prior Work on Vocabulary: Fragmented Insights, No Scaling Framework

The paper does not claim that vocabulary size has never been studied—only that it has never been integrated into a scaling law framework that jointly optimizes it with model size and training data. The existing literature provides scattered empirical observations but lacks the predictive formalism this paper contributes.

In pre-LLM NLP, vocabulary size was a standard hyperparameter in neural machine translation and language modeling. Approaches like BPE (Sennrich et al., 2016)—which the paper uses for its tokenizers—were developed specifically to address the out-of-vocabulary problem by constructing subword vocabularies of moderate size (typically 8K–32K for bilingual tasks, up to 50K for monolingual language modeling). These choices were driven by practical considerations (training speed, memory constraints) rather than optimality analysis.

In the multilingual literature, vocabulary size has received more explicit attention. Works like Zheng et al. (2021), Liang et al. (2023), and the broader literature on cross-lingual vocabulary adaptation (Wang et al., 2019; Chung et al., 2020) recognize that multilingual models need larger vocabularies to cover diverse scripts and morphological systems. The XLM-V approach (Liang et al., 2023) explicitly frames vocabulary as a bottleneck in multilingual models, showing that increasing vocabulary capacity improves multilingual transfer. However, these works treat vocabulary size as a design choice determined by language coverage requirements, not as a variable to be optimized jointly with compute allocation.

Recent work on domain-specific vocabulary (Dagan et al., 2024) has explored vocabulary size tradeoffs specifically for code generation, identifying optimal vocabulary sizes for memory efficiency and inference speed. Similarly, Dou et al. (2024) studied vocabulary expansion during continual pretraining for South-East Asian languages. These are valuable empirical contributions but remain domain- or task-specific and do not provide a general scaling theory.

Byte-level language models (Yu et al., 2024; Wang et al., 2024) represent an extreme point on the vocabulary spectrum: by using raw bytes (vocabulary size = 256), they eliminate tokenization entirely, trading off sequence length for vocabulary simplicity. The paper notes (Appendix A.12) that such models have generally not been successfully scaled beyond ~1B parameters, and suggests their limited vocabulary—fixed regardless of model scale—may explain this scaling difficulty. This corroborates the paper's central claim that vocabulary must scale with model size.

Where the Paper Positions Itself

The paper positions itself as filling the gap between two established bodies of work: scaling laws (which ignore vocabulary) and vocabulary design (which has been studied without scaling theory). This is captured in their statement from Section 1:

"This negligence has resulted in substantial variability in the vocabulary size of current LLMs... This variability in vocabulary sizes across LLMs raises the research question: What is the compute-optimal vocabulary size for a LLM?"

The paper's contribution is not a new tokenization algorithm or a new model architecture, but rather a principled framework for deciding vocabulary size as a function of compute budget, model size, and training data volume. This framework operates at the intersection of three components:

A fair evaluation metric. Because standard language modeling loss depends on vocabulary size (larger vocabularies create more prediction options, raising loss independent of model quality), the paper introduces a unigram-normalized loss $L_u$ that subtracts the unigram log-probability of each token, making losses comparable across vocabulary sizes. This is essential for any systematic study of vocabulary scaling—without it, models with larger vocabularies would appear worse by construction.
A cost model that accounts for vocabulary. The paper decomposes total parameters as $N = N_{nv} + Vd$ and models training tokens via the compression function $D = Hf(V)$ , where $H$ is training characters. This connects vocabulary choice to FLOPs via the standard approximation $C \approx 6ND = 6(N_{nv} + Vd)Hf(V)$ , making the vocabulary-FLOPs relationship explicit and differentiable.
Three complementary prediction frameworks (IsoFLOPs fitting, derivative estimation, parametric loss function) that all yield the same qualitative conclusion: $N_v^{\text{opt}} \propto N_{nv}^\gamma$ with $\gamma \approx 0.83$ . The convergence of these distinct methods—each making different assumptions and using different mathematical machinery—is presented as evidence for the robustness of the finding.

The paper explicitly distinguishes its approach from prior work that studied vocabulary in isolation: rather than asking "what vocabulary size works best for this specific task?" it asks "how should vocabulary size scale as we scale everything else?" This shifts vocabulary from a fixed hyperparameter to a scaling dimension alongside model parameters and training data, making it subject to the same kind of optimal allocation analysis that Hoffmann et al. applied to parameters-vs-data.

The Practical Significance: Wasted Compute at Scale

The motivation for this work is not purely theoretical. The paper quantifies the practical stakes by examining popular open-source LLMs (Figure 2). Under the assumption that these models were trained with compute-optimal data allocations (following Chinchilla), the paper finds that essentially all major models underallocate vocabulary parameters relative to their predicted optimal values. The specific predictions (Table 1) are striking:

Llama2-70B ( $N_{nv} \approx 70$ B): predicted optimal vocabulary 212K–231K (depending on the approach), actual vocabulary 32K—roughly 7× too small.
Falcon-180B ( $N_{nv} \approx 180$ B): predicted optimal vocabulary parameters ~5B (vocabulary ~380K), actual vocabulary parameters ~1B—roughly 5× too small.
DeepSeek-67B: similarly underallocated.

The exceptions—models that come closer to the predicted optimal allocation—include StarCoder2-3B, OLMo-7B, InternLM2-20B, and Gemma2-27B. The paper notes that the community is beginning to shift: Llama3 increased to 128K vocabulary from Llama2's 32K, and Gemma2 uses 256K. This suggests an industry intuition aligning with the paper's formal analysis, but without the predictive framework needed to determine the exact optimal size for any given model scale and training budget.

The economic significance is substantial. Vocabulary parameters affect both training cost (via FLOPs) and inference cost (via the output softmax over $V$ tokens, which can dominate inference time for large vocabularies). Getting vocabulary size wrong by a factor of 7× means either:

If too small: sacrificing model quality that could have been achieved with the same compute budget, or equivalently, requiring more total FLOPs to reach a target performance.
If too large: wasting parameters on embedding vectors that receive insufficient training, and increasing inference latency due to the larger output projection.

The paper's framework provides a way to avoid both failure modes by making vocabulary size a predicted rather than arbitrary quantity. This is particularly relevant as the field moves toward training runs costing tens or hundreds of millions of dollars, where systematic misallocation of even 5–10% of parameters to suboptimal vocabulary sizing represents enormous waste.

The Unique Challenge Vocabulary Poses for Scaling Analysis

The paper identifies several technical challenges that make vocabulary scaling harder to study than parameter or data scaling, which helps explain why the gap persisted until this work:

Vocabulary changes the measurement of data. When you change vocabulary size, you change how many tokens are needed to represent a given corpus. This means that the standard scaling law variable $D$ (training tokens) is not comparable across vocabulary choices—1 billion tokens from a 4K vocabulary and 1 billion tokens from a 64K vocabulary represent very different amounts of raw text. The paper solves this by measuring data in training characters ( $H$ ) and using the compression function $f(V)$ to map between tokens and characters, enabling fair comparisons.

Vocabulary changes the loss metric. A model with a 64K vocabulary must predict among 64K possible next tokens; one with a 4K vocabulary only among 4K. The standard cross-entropy loss will be higher for the 64K vocabulary model even if both have identical language understanding, because the prediction task is harder by construction. The paper's introduction of a unigram-normalized loss $L_u$ (Section 2.2, Equation 4) addresses this, normalizing by token frequency so that the loss reflects the model's improvement over a context-free baseline.

Vocabulary parameters are structurally different from other parameters. As the paper explains (Appendix A.6), non-vocabulary parameters can benefit from increases in both depth (more layers) and width (larger hidden dimensions), while vocabulary parameters are confined to the embedding and output layers—they can only grow in width, not depth. This means that vocabulary parameters and non-vocabulary parameters have different growth dynamics with scale, which the paper's finding that $\gamma < 1$ (vocabulary scales slower) captures quantitatively.

The optimal vocabulary depends on the training regime. Prior work assumed a single "optimal" vocabulary size per model architecture, but the paper shows (Section 5, Table 3) that the optimal vocabulary shifts with the amount of training data: for $N_{nv} = 2.87$ B with insufficient data ("undertraining" at $2.8 \times 10^{20}$ FLOPs), the optimal vocabulary is 24K (smaller than the conventional 32K), while with overly sufficient data ("overtraining" at $2.3 \times 10^{21}$ FLOPs), the optimal jumps to 43K (larger than the conventional 32K). This means vocabulary choice is not just model-dependent but training-regime-dependent, a nuance that the paper's Approach 3 (parametric loss fit) is specifically designed to handle.

3. Technical Approach

3.1 Reader Orientation

This paper develops a predictive framework for choosing vocabulary size in large language models, not a new tokenization algorithm or architecture. The system takes as input a compute budget (FLOPs, model size, training data volume) and outputs the vocabulary size that minimizes loss—solving the problem that vocabulary size has been treated as an arbitrary hyperparameter rather than something that should scale with compute. The solution is three complementary mathematical approaches (IsoFLOPs fitting, derivative-based estimation, parametric loss function) that all converge on the same power-law relationship: optimal vocabulary parameters scale as $N_v^{\text{opt}} \propto N_{nv}^{0.83}$ , meaning vocabulary should grow with model size, but more slowly than non-vocabulary parameters.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components:

Tokenizer with configurable vocabulary size ( $V$ ) — a BPE tokenizer trained on SlimPajama data, where vocabulary sizes range from 4K to 96K (and up to 1024K for the compression function fitting). This determines how raw text is split into tokens and directly affects both the number of vocabulary parameters ( $N_v = Vd$ ) and the compression ratio (tokens per character).
Compression function $f(V)$ — a fitted quadratic function on $\log(V)$ that predicts the tokens-per-character ratio for any vocabulary size. This enables mapping between training characters ( $H$ , the vocabulary-independent data measure) and training tokens ( $D$ , the vocabulary-dependent measure used in FLOPs calculations).
Vocabulary-insensitive loss metric ( $L_u$ ) — a unigram-normalized language modeling loss that subtracts the unigram log-probability of each token, making losses comparable across different vocabulary sizes. This replaces standard cross-entropy loss, which is vocabulary-dependent and would falsely penalize larger vocabularies.
Model training pipeline with decomposed parameter accounting — a set of LLMs (33M to 3B non-vocabulary parameters) trained on fixed character budgets, where total parameters are split into $N_{nv}$ (non-vocabulary: Transformer blocks) and $N_v = Vd$ (vocabulary: embedding + output layers). FLOPs are computed as $C \approx 6(N_{nv} + Vd)Hf(V)$ .
Three prediction approaches — mathematical frameworks that take training data (losses, FLOPs, configurations) as input and output the compute-optimal vocabulary size for any target model scale and training regime.

Information flows as follows: select a set of ( $N_{nv}$ , $V$ , $H$ ) configurations → train models for each configuration → evaluate using $L_u$ → fit power laws (Approach 1), solve derivative equations (Approach 2), or fit a parametric loss function (Approach 3) → predict optimal $V$ for target $N_{nv}$ and training data regime.

3.3 Roadmap for the Deep Dive

First, the unified cost model (FLOPs equation incorporating vocabulary, the compression function $f(V)$ , and the parameter decomposition $N = N_{nv} + Vd$ ), since every prediction approach depends on mapping between vocabulary choice and compute cost.
Second, the vocabulary-insensitive loss metric $L_u$ , since fair comparison across vocabulary sizes is the prerequisite for all empirical work.
Third, Approach 1 (IsoFLOPs power-law fitting), the most direct empirical method: train models with varying vocabularies at fixed FLOPs budgets, find the best configuration per budget, and fit power laws.
Fourth, Approach 2 (derivative-based estimation), which flips the optimization to find the vocabulary that minimizes FLOPs for a target loss, requiring only the cost model and a scaling exponent from a small reference model.
Fifth, Approach 3 (parametric loss fit), which models loss as a function of $N_{nv}$ , $N_v$ , and $H$ (training characters) jointly, enabling predictions even when the training regime is non-optimal (undertraining or overtraining).
Sixth, the fitting procedures and hyperparameters—how each approach's free parameters are estimated from data, including optimization algorithms, loss functions (Huber loss), and constraints.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical scaling law paper whose core idea is that vocabulary size should be treated as a scaling dimension alongside model parameters and training data, and that the optimal vocabulary follows a power-law relationship with compute budget that can be predicted through three complementary mathematical frameworks. The key innovation is not any single prediction method but rather the unified framework for incorporating vocabulary into scaling analysis through careful metric design, cost modeling, and parameter decomposition.

The Unified Cost Model: How Vocabulary Affects FLOPs

The foundation for all three prediction approaches is a cost model that explicitly accounts for vocabulary size in the FLOPs calculation. Without this, vocabulary cannot be treated as an optimizable variable—there would be no mathematical connection between choosing $V$ and the resulting computational cost.

The paper decomposes the standard Transformer FLOPs approximation from Kaplan et al. into vocabulary-dependent and vocabulary-independent components. The starting point is the established estimate that Transformer training FLOPs are approximately:

$C \approx 6ND$

where $C$ is total training FLOPs, $N$ is total model parameters, and $D$ is the number of training tokens seen. The factor 6 comes from the forward pass (2 FLOPs per parameter per token for matrix multiplies) and backward pass (roughly 2× the forward pass), totaling approximately 6 FLOPs per parameter per token.

The paper then decomposes both $N$ and $D$ into vocabulary-dependent components.

Parameter decomposition. Total parameters $N$ are split into two categories that scale differently with vocabulary size:

$N = N_{nv} + N_v$

where $N_{nv}$ represents non-vocabulary parameters—all parameters in the Transformer blocks (attention projections, feed-forward networks, layer norms) that do not depend on vocabulary size—and $N_v$ represents vocabulary parameters. The paper models vocabulary parameters as:

$N_v = Vd$

where $V$ is the vocabulary size (number of distinct tokens) and $d$ is the embedding dimension (model hidden size). This approximates the parameters in both the input embedding layer and the output projection layer, though the paper acknowledges it uses $Vd$ rather than $2Vd$ "for clarity and analytical simplicity" (Section 2.2), noting that the main computational burden in FLOPs comes from the output layer but not the embedding layer. This is a practical simplification: the embedding lookup is a sparse operation that contributes sub-linearly to FLOPs, while the output projection is a dense matrix multiply over the entire vocabulary that scales linearly with $Vd$ .

What this decomposition captures conceptually: When you increase vocabulary size from 32K to 64K while keeping embedding dimension fixed at 4096, you add $V \cdot d = 32K \cdot 4096 \approx 131$ million new parameters. These parameters exist in both the embedding layer (where they convert token IDs to vectors) and the output projection (where they convert hidden states to logits over the vocabulary). The non-vocabulary parameters—the actual Transformer stack—remain unchanged.

Token-count decomposition. Training tokens $D$ are not directly comparable across vocabulary sizes because the same raw text produces different numbers of tokens depending on the tokenizer's vocabulary. The paper measures training data in training characters ( $H$ ), which is vocabulary-independent: a corpus of 500 billion characters is 500 billion characters regardless of how it's tokenized. To convert between characters and tokens, the paper models the compression ratio:

$D = H \cdot f(V)$

where $f(V)$ is a function that maps vocabulary size $V$ to the tokens-per-character ratio (equivalently, $1/f(V)$ is characters-per-token). As $V$ increases, tokenization becomes more efficient, so $f(V)$ decreases—fewer tokens are needed to represent the same characters. A tokenizer with $V = 4K$ might produce 0.35 tokens per character, while one with $V = 64K$ might produce 0.25 tokens per character, meaning the same corpus requires 40% fewer tokens with the larger vocabulary.

The compression function $f(V)$ . The paper develops a parametric form for $f(V)$ by training 25 BPE tokenizers with vocabulary sizes ranging from 1K to 1024K on a uniformly sampled subset of SlimPajama, then measuring the tokens-per-character ratio on a validation set. The fitted function is:

$f(V) = a \log^2(V) + b \log(V) + c$

where $a = 0.0064$ , $b = -0.1581$ , and $c = 1.2047$ are fitted constants.

What this form captures: The quadratic in $\log(V)$ models the diminishing returns of larger vocabularies. At very small $V$ , increasing vocabulary size dramatically improves compression (e.g., going from 1K to 4K tokens sharply reduces tokens-per-character because common words can be represented as single tokens rather than character sequences). At large $V$ , the gains saturate—once all common words and subwords have dedicated tokens, additional vocabulary entries are used only for rare or domain-specific terms, providing marginal compression improvement. The logarithmic form captures this because $\log(V)$ grows slowly: doubling $V$ from 32K to 64K increases $\log(V)$ by only $\log(2) \approx 0.69$ , corresponding to a small change in $f(V)$ .

The paper reports the fit quality as "a low relative mean square error (RMSE) and a high coefficient of determination ( $R^2$ )" and provides visualizations in Appendix A.9 showing the function works well across BPE, unigram, and word-based tokenizers. At very large $V$ , the function would eventually increase (since quadratics are single-peaked), so the paper clamps $V$ to a maximum of 200K in the function—a pragmatic choice based on the observation that compression gains become negligible beyond this point.

The full FLOPs equation. Substituting the decompositions into $C \approx 6ND$ gives:

$C \approx 6(N_{nv} + Vd) \cdot H f(V)$

What this equation computes operationally: Given a choice of non-vocabulary parameters $N_{nv}$ , vocabulary size $V$ , embedding dimension $d$ (determined by $N_{nv}$ via Table 5 in the appendix), and training characters $H$ , this equation estimates the total training FLOPs. The term $(N_{nv} + Vd)$ is total parameters; multiplying by $H f(V)$ gives total tokens; multiplying by 6 gives total FLOPs.

Why this decomposition matters for optimization: The equation reveals the two competing effects of increasing $V$ . First, $V$ appears positively in $(N_{nv} + Vd)$ —larger vocabulary means more parameters and thus more FLOPs per token. Second, $V$ appears negatively through $f(V)$ (since $f(V)$ decreases with $V$ )—larger vocabulary means fewer tokens for the same characters, reducing total FLOPs. These opposing forces create the U-shaped curve that motivates the existence of an optimal vocabulary size: at small $V$ , the tokenization efficiency gains dominate (total FLOPs decrease as $V$ increases); at large $V$ , the parameter burden dominates (total FLOPs increase as $V$ increases); there exists a minimum in between.

The paper illustrates this in Figure 3 (left), which shows FLOPs as a function of vocabulary size for a fixed loss target. The curve has a clear minimum—this minimum defines the derivative-based Approach 2.

The Vocabulary-Insensitive Loss Metric: Making Losses Comparable

Standard language modeling loss is vocabulary-dependent by construction. The cross-entropy loss for next-token prediction is:

$L = -\frac{1}{T} \sum_{i=1}^{T} \log p(w_i | w_{1:i-1}, V)$

where $T$ is the number of tokens, $p(w_i | w_{1:i-1}, V)$ is the model's predicted probability for the correct next token $w_i$ given context $w_{1:i-1}$ and a tokenizer with vocabulary size $V$ .

Why this metric is broken for vocabulary scaling studies. Consider two models with identical Transformer architectures and identical training, but one using $V = 4K$ and the other using $V = 64K$ . The 64K-vocabulary model has 16× more possible next tokens to choose from at each prediction step. Even if both models have identical language understanding, the 64K model will assign lower probability to each individual correct token simply because the probability mass must be distributed across more options. The cross-entropy loss will therefore be higher for the 64K model, creating the false impression that it is a worse language model. The paper demonstrates this empirically in Appendix A.10, Figure 13(a): models with larger vocabularies have higher standard cross-entropy loss yet better downstream task performance—a perverse positive correlation that would mislead any optimization procedure.

The paper adopts the unigram-normalized loss from Roh et al. (2020) to address this:

$L_u = -\frac{1}{T} \sum_{i=1}^{T} \log \frac{p(w_i | w_{1:i-1}, V)}{p(w_i | V)}$

which can be rewritten as:

$L_u = L + \frac{1}{T} \sum_{i=1}^{T} \log p(w_i | V)$

where $p(w_i | V)$ is the unigram frequency of token $w_i$ in the tokenized training corpus—the probability of encountering that token if you sampled randomly from the corpus without context.

What this computes operationally: For each token in the sequence, the model's predicted probability $p(w_i | w_{1:i-1}, V)$ is divided by the token's baseline frequency $p(w_i | V)$ . If the model assigns the same probability as the unigram baseline ( $p(w_i | w_{1:i-1}, V) = p(w_i | V)$ ), the ratio is 1, $\log(1) = 0$ , and that token contributes zero to the loss—the model has learned nothing beyond the token frequency. If the model assigns higher probability than the baseline ( $p(w_i | w_{1:i-1}, V) > p(w_i | V)$ ), the ratio exceeds 1, $\log$ is negative, and the contribution is negative—the model has learned to predict this token better than chance. $L_u$ therefore measures the improvement over a context-free unigram model.

Why this normalization works for vocabulary comparison. The unigram term $p(w_i | V)$ scales with vocabulary size in the same direction as the model's prediction difficulty: larger vocabularies make each individual token less frequent (since the total probability mass of 1.0 is distributed across more tokens), so $p(w_i | V)$ is smaller, and $\log p(w_i | V)$ is more negative, adding a larger negative offset to the loss. This exactly compensates for the larger-vocabulary model's lower raw probabilities, making $L_u$ comparable across $V$ .

Empirical validation. Appendix A.10, Figure 13(b) shows that $L_u$ exhibits the expected negative correlation with downstream performance: models with lower $L_u$ perform better on seven benchmark tasks (WinoGrande, PIQA, OpenBookQA, HellaSwag, BoolQ, ARC-Easy, ARC-Challenge). The correlation reverses from positive (broken) to negative (correct) when switching from $L$ to $L_u$ , confirming that $L_u$ is the appropriate metric for vocabulary scaling analysis.

Relation to BPC (Bits Per Character). The paper notes that BPC—another vocabulary-insensitive metric measuring bits needed to encode each character—is highly correlated with $L_u$ (Pearson $\rho = 0.9888$ in Appendix A.5, Figure 11). The difference is that BPC normalizes by characters (dividing total loss by number of characters in the text), while $L_u$ normalizes by token frequency. The high correlation validates that $L_u$ captures the same underlying model quality as BPC, but $L_u$ has the advantage of directly connecting to the token-level predictions that scaling laws model.

Practical note. $L_u$ can be negative, which is intentional: when the model consistently assigns higher probability than the unigram baseline, the ratio exceeds 1 on average, and the log average is negative. A negative $L_u$ means the model is outperforming a simple frequency-based predictor—which is the expected case for any trained language model.

Parameter Decomposition and Architecture Choices

The paper makes several architectural choices that enable clean decomposition of vocabulary effects from other model properties.

Non-vocabulary parameters ( $N_{nv}$ ) vs. vocabulary parameters ( $N_v$ ). The paper defines:

$N_{nv}$ : all parameters in Transformer blocks (attention query/key/value/output projections, feed-forward network weights, layer normalization parameters). These are independent of vocabulary size.
$N_v = Vd$ : parameters in the embedding layer and output projection that map between token IDs and hidden states. These scale linearly with vocabulary size $V$ .

Why separate them. The paper explains in Appendix A.6 that non-vocabulary parameters can grow through both depth (more layers) and width (larger hidden dimensions), while vocabulary parameters can only grow through width (since embeddings and output projections are single layers). This structural difference means that if you scale total parameters uniformly—without distinguishing $N_{nv}$ from $N_v$ —you might inadvertently constrain vocabulary growth relative to what would be optimal. The separate treatment enables independent power-law exponents for $N_{nv}$ and $N_v$ , which the paper finds are indeed different ( $\alpha_1 = 0.50$ for $N_{nv}$ vs. $\alpha_2 = 0.42$ for $N_v$ in Approach 1).

Embedding dimension scheduling. The paper uses empirically determined embedding dimensions for different $N_{nv}$ ranges, listed in Appendix A.7.2, Table 5. For example, models with $N_{nv} \leq 50$ M use $d = 512$ ; models with $2B < N_{nv} \leq 5B$ use $d = 3200$ ; models with $50B < N_{nv} \leq 100B$ would use $d = 8192$ . This follows the observation by Kaplan et al. that "the performance of models with varying depth-to-width ratios converges to a single trend" (Section 2.2)—meaning the exact width chosen matters less than the total non-vocabulary parameter count, so fixing the width based on $N_{nv}$ is reasonable. This scheduling is used for prediction in Table 1, where each model scale has its embedding dimension taken as given from this table.

Model architecture details. All models use the LLaMA architecture (pre-norm Transformer with rotary position embeddings, SwiGLU activations in feed-forward networks) with the specific depth and width configurations per $N_{nv}$ group listed in Appendix A.7.1, Table 4. For example, the $N_{nv} = 33$ M models use 8 layers, 8 attention heads, embedding dimension 512, intermediate size 2048, and sequence length 2048. The $N_{nv} = 2.87$ B models use 24 layers, 32 heads, embedding dimension 3200, intermediate size 8192. Vocabulary sizes are chosen to be multiples of 128 for compatibility with NVIDIA tensor core acceleration. The specific vocabulary sizes tested are: 4096, 6144, 8192, 10240, 16384, 24576, 32768, 48128, 64512, and 96256.

Training data and FLOPs accounting. For each model group ( $N_{nv}$ fixed), the paper determines the number of training characters based on a reference vocabulary size (typically $V = 16384$ for smaller models, $V = 32K$ for the 2.87B model) to hit a target FLOPs budget. This means that when vocabulary size varies within a group, the number of training tokens $D = Hf(V)$ varies slightly—models with larger $V$ train on fewer tokens (since $f(V)$ is smaller) but the same number of characters $H$ , keeping the comparison FLOPs-matched. The training data is uniformly sampled from SlimPajama, a 627B-token cleaned and deduplicated version of RedPajama.

Training hyperparameters. All models use: AdamW optimizer, maximum learning rate $4 \times 10^{-4}$ decaying to 10% ( $4 \times 10^{-5}$ ), global batch size 512, bfloat16 mixed precision, sequence length 2048. Models with $N_{nv} < 1130$ M are trained on a single 8-GPU node; larger models use Megatron-LM for multi-node training with 8 GPUs per node. The largest models ( $N_{nv} = 2870$ M) require approximately 120 hours on 64 A100-40GB GPUs to train on over 500B training characters.

Training curves. Figure 4 visualizes all training runs used for Approach 1 and Approach 3. Each line represents one ( $N_{nv}$ , $V$ ) configuration, plotting $L_u$ against training characters. Within each $N_{nv}$ group (same color), different line styles correspond to different vocabulary sizes. The curves show the expected pattern: within each group, moderate vocabulary sizes tend to achieve lower $L_u$ than either very small or very large vocabularies, and the optimal vocabulary size shifts upward as $N_{nv}$ increases (comparing across colors).

Approach 1: IsoFLOPs Power-Law Fitting

This approach directly answers the question: Given a fixed FLOPs budget, what combination of $N_{nv}$ , $N_v$ , and $H$ achieves the lowest loss? It is the most empirical of the three methods, relying on actual training runs rather than mathematical approximations.

Setup. The paper defines six groups of models with fixed $N_{nv}$ (33M, 85M, 151M, 302M, 631M, 1130M). Within each group, vocabulary size $V$ is varied across the standard set (4K to 96K). Each configuration is trained for a fixed number of training characters $H$ . The training curves in Figure 4 show $L_u$ as a function of training progress for all configurations.

Selecting compute-optimal points. For each FLOPs budget (from small to large), the paper identifies the ( $N_{nv}$ , $V$ , $H$ ) configuration that achieves the minimum $L_u$ among all configurations that fall at or near that budget. This is the standard IsoFLOP methodology from Hoffmann et al.: rather than training a single configuration at many budgets, you train many configurations and find the best performer per budget.

Interpolation for denser coverage. Because training every possible ( $N_{nv}$ , $V$ , $H$ ) combination is combinatorially prohibitive, the paper uses interpolation in logarithmic space to generate additional data points cheaply. Specifically, for configurations near existing training runs, they interpolate the ( $N_{nv}$ , $N_v$ , $H$ ) triple in log-space and predict the corresponding validation loss from neighboring real data points. The FLOPs for each interpolated point are computed using the cost model in Equation 5. This provides a denser sampling of the configuration space without additional training, enabling more reliable fitting of the power laws.

Power-law fitting. The paper hypothesizes that the compute-optimal values of $N_{nv}$ , $N_v$ , and $H$ each follow power-law relationships with FLOPs $C$ :

$N_{nv} = k_1 C^{\alpha_1}$

$N_v = k_2 C^{\alpha_2}$

$H = k_3 C^{\alpha_3}$

where $k_1$ , $k_2$ , $k_3$ are scaling coefficients and $\alpha_1$ , $\alpha_2$ , $\alpha_3$ are the power-law exponents.

Fitting procedure. The paper fits these relationships by minimizing Huber loss in log-space. Taking $N_{nv}$ as an example, they optimize:

$\min_{K_1, \alpha_1} \text{Huber}_\delta(K_1 + \alpha_1 \log(C), \log(N_{nv}))$

where $K_1 = \log(k_1)$ and $\text{Huber}_\delta$ is the Huber loss with $\delta = 0.001$ . The Huber loss is chosen because it is more robust to outliers than mean squared error: it behaves like MSE for small residuals (within $\delta$ ) and like absolute error for large residuals, preventing individual poorly-fit points from dominating the optimization. The delta of 0.001 in log-space corresponds to approximately a 0.1% relative error, meaning errors larger than this get linear rather than quadratic penalty.

The optimization uses the LBFGS algorithm (a quasi-Newton method well-suited to smooth optimization problems with moderate numbers of variables) with 20 random initializations drawn from uniform grids: $K \in [-20, 15]$ , $\alpha \in [0, 1]$ . The best solution across initializations is selected.

Constraint from Chinchilla. Following established scaling law results, the paper constrains $\alpha_1 = \alpha_3$ , meaning non-vocabulary parameters and training characters should scale at the same rate with FLOPs. This is the key finding from Hoffmann et al.: for compute-optimal training, model size and data size should be scaled equally (both growing as $C^{0.5}$ ). The paper adopts this constraint to maintain consistency with prior scaling law results while adding vocabulary as a new dimension.

Fitted results. The resulting power laws are (Figure 5):

$N_{nv} = 0.08 \cdot C^{0.50}$

$N_v = 0.20 \cdot C^{0.42}$

$H = 6.42 \cdot C^{0.50}$

What these equations compute operationally. Given a FLOPs budget $C$ , you can compute the optimal non-vocabulary parameters as $0.08 \cdot C^{0.50}$ , the optimal vocabulary parameters as $0.20 \cdot C^{0.42}$ , and the optimal training characters as $6.42 \cdot C^{0.50}$ . To get vocabulary size $V$ , divide $N_v$ by the embedding dimension $d$ appropriate for that $N_{nv}$ (from Table 5). For example, with $C = 7.1 \times 10^{23}$ (the estimated budget for a 70B-parameter model under compute-optimal training), $N_v = 0.20 \cdot (7.1 \times 10^{23})^{0.42} \approx 1.7 \times 10^9$ vocabulary parameters, and with $d = 8192$ , $V = 1.7 \times 10^9 / 8192 \approx 212$ K.

Key findings from the fitted exponents. The exponent for vocabulary parameters $\alpha_2 = 0.42$ is smaller than the exponent for non-vocabulary parameters $\alpha_1 = 0.50$ . This means vocabulary parameters should be scaled slower than non-vocabulary parameters as compute increases. The ratio $\gamma = \alpha_2 / \alpha_1 = 0.42 / 0.50 = 0.84$ (which rounds to the commonly cited 0.83) gives the scaling relationship:

$N_v^{\text{opt}} \propto N_{nv}^\gamma$

or equivalently:

$\frac{N_v^{\text{opt}}}{N_v^{\text{ref}}} = \left(\frac{N_{nv}}{N_{nv}^{\text{ref}}}\right)^\gamma$

where $N_{nv}^{\text{ref}}$ and $N_v^{\text{ref}}$ are the optimal values for a reference (small) model.

Why $\gamma < 1$ . The paper hypothesizes that non-vocabulary parameters benefit more from scaling because they can leverage increased depth (more layers to build hierarchical representations) and width (more capacity per layer), while vocabulary parameters—confined to the embedding and output layers—only benefit from increased width. Once a sufficiently rich embedding space exists (via a moderately large vocabulary), the marginal benefit of adding more vocabulary entries diminishes relative to adding more Transformer capacity to process those embeddings.

Interpreting Figure 1. Figure 1 in the paper visualizes this relationship: the x-axis is non-vocabulary parameters $N_{nv}$ , the y-axis is optimal vocabulary parameters $N_v^{\text{opt}}$ , and the solid line shows the power-law fit $N_v^{\text{opt}} \propto N_{nv}^{0.83}$ . The circles represent empirical optimal points from Approach 1, with larger circles indicating higher loss values. The alignment between the empirical points and the fitted power law validates the functional form.

Usage for prediction. Given a target model with non-vocabulary parameters $N_{nv}$ , you compute the required FLOPs budget for compute-optimal training from $N_{nv} = 0.08 \cdot C^{0.50}$ (inverting: $C = (N_{nv} / 0.08)^{1/0.50}$ ), then compute $N_v^{\text{opt}} = 0.20 \cdot C^{0.42}$ , then $V^{\text{opt}} = N_v^{\text{opt}} / d$ . This yields predictions like those in Table 1: for $N_{nv} = 3$ B with $d = 3200$ , $V^{\text{opt}} \approx 39K$ ; for $N_{nv} = 70$ B with $d = 8192$ , $V^{\text{opt}} \approx 212$ K.

Limitations of Approach 1. The paper acknowledges (Appendix B.1) that this approach is "constrained by the granularity and range of the experimental data points available, which can introduce errors in the fitting process." The models used for fitting only go up to $N_{nv} = 1.13$ B, so predictions for 70B+ models require extrapolation over nearly two orders of magnitude. The power-law form is an empirical regularity—there is no theoretical guarantee it holds at larger scales. However, the convergence of all three approaches on similar predictions provides some triangulation.

Approach 2: Derivative-Based Fast Estimation

This approach addresses the practical limitation of Approach 1: training many models at different configurations is expensive. Instead, it derives the optimal vocabulary analytically from the FLOPs equation, requiring only a single scaling exponent that can be estimated from a small reference model.

The optimization perspective. Approach 2 flips the optimization problem. Rather than minimizing loss for a fixed FLOPs budget, it asks: For a target loss value $\ell$ , what vocabulary size minimizes the FLOPs needed to achieve that loss? Formally:

$V = \underset{V | L_u(N_{nv}, V, H) = \ell}{\arg\min} \, C(N_{nv}, N_v, H)$

The assumption is that, under compute-optimal allocation, the loss $\ell$ is achievable through some combination of $N_{nv}$ and $H$ , and we want to find the $V$ that minimizes the computational cost of reaching that loss.

Taking the derivative. The FLOPs equation $C \approx 6(N_{nv} + Vd)H f(V)$ depends on $V$ through both the parameter term $(N_{nv} + Vd)$ and the compression function $f(V) = a \log^2(V) + b \log(V) + c$ . Taking the derivative with respect to $V$ :

$\frac{\partial C}{\partial V} = 6H \frac{\partial}{\partial V} \left[(N_{nv} + Vd)(a \log^2(V) + b \log(V) + c)\right]$

Applying the product rule:

$\frac{\partial C}{\partial V} = 6H \left[(N_{nv} + Vd) \frac{d}{dV}(a \log^2(V) + b \log(V) + c) + (a \log^2(V) + b \log(V) + c) \frac{d}{dV}(N_{nv} + Vd)\right]$

The derivative of the log terms: $\frac{d}{dV} \log^2(V) = \frac{2 \log(V)}{V}$ and $\frac{d}{dV} \log(V) = \frac{1}{V}$ . The derivative of $N_{nv} + Vd$ with respect to $V$ is simply $d$ . Substituting:

$\frac{\partial C}{\partial V} = 6H \left[(N_{nv} + Vd) \cdot \frac{2a \log(V) + b}{V} + (a \log^2(V) + b \log(V) + c) \cdot d\right]$

What this derivative represents. $\frac{\partial C}{\partial V}$ is the rate of change of FLOPs with respect to vocabulary size. When it is positive, increasing $V$ increases total FLOPs (vocabulary parameter cost dominates tokenization efficiency gain). When it is negative, increasing $V$ decreases total FLOPs (tokenization efficiency gain dominates). At the optimal vocabulary size, the two effects exactly balance, and $\frac{\partial C}{\partial V} = 0$ .

Finding the optimal $V$ . The equation $\frac{\partial C}{\partial V} = 0$ cannot be solved analytically in closed form because $V$ appears both inside and outside logarithms. Instead, the paper uses numerical root-finding (specifically scipy.optimize.fsolve) to find the $V$ that makes the derivative zero. This requires only the parameters $a$ , $b$ , $c$ from the compression function fit (which are obtained once and reused) and the values $N_{nv}$ , $d$ , and $H$ .

From a single reference model to a scaling law. To use this approach for scaling predictions, the paper needs to establish how the optimal $V$ changes as $N_{nv}$ grows. Rather than solving the derivative equation for every possible $N_{nv}$ (which would require knowing $H$ for each case), the paper leverages the fact that under compute-optimal training, the ratio between the derivative-optimal vocabulary at different model scales follows a power law.

The procedure is:

For a small reference model (e.g., $N_{nv}^0 = 33$ M), find the optimal vocabulary $N_v^0$ empirically (by training models with different $V$ at fixed $N_{nv}^0$ and selecting the one with minimum $L_u$ ).
For a range of $N_{nv}$ values, solve $\frac{\partial C}{\partial V} = 0$ numerically to get the derivative-optimal $N_v$ for each $N_{nv}$ .
Fit the power law $N_v \propto N_{nv}^\gamma$ to the pairs $(N_{nv}^i, N_v^i)$ using Huber loss minimization (Equation 12 in Appendix A.7.4).
The fitted $\gamma$ can then be combined with the empirical reference point to predict the optimal vocabulary for any target scale:

$N_v^{\text{opt}} = N_v^0 \cdot \left(\frac{N_{nv}}{N_{nv}^0}\right)^\gamma$

Fitted result. The paper obtains $\gamma = 0.83$ from this procedure, consistent with the 0.84 derived from Approach 1. The predicted optimal vocabulary parameters for various $N_{nv}$ are shown in Table 1: for example, $N_{nv} = 3$ B with $d = 3200$ yields $V^{\text{opt}} \approx 43K$ (Approach 2) vs. 39K (Approach 1).

Interpretation through the FLOPs-minimization lens. Appendix A.1, Figure 8 provides intuition for why this derivative method works. The left panel shows $\frac{\partial C}{\partial V}$ as a function of $V$ : it is negative at small $V$ (increasing vocabulary reduces FLOPs) and crosses zero at the optimal $V$ , then becomes positive (further increases cost more than they save). The right panel shows that at the optimal vocabulary, the model can process the maximum number of training characters for a given FLOPs budget—the vocabulary is large enough for efficient tokenization but not so large that parameter costs dominate.

Advantages over Approach 1. The derivative method is computationally cheap: once the compression function $f(V)$ is fitted (requiring only tokenizer training, not LLM training), and the scaling exponent $\gamma$ is estimated from a small reference model, predictions for any scale are obtained via numerical root-finding. This makes it practical for quick estimation without extensive training runs. The paper notes (Appendix B.1) that "its reliance on numerical solutions rather than exhaustive deep learning experiments makes it rapid and broadly applicable across various tokenizers."

Limitations. Approach 2 relies on two assumptions: (1) the FLOPs approximation $C \approx 6ND$ accurately captures the relationship between vocabulary size and compute cost, and (2) the compression function $f(V)$ is accurately fitted. It also cannot independently determine the optimal allocation of $N_{nv}$ and $H$ —it requires scaling law results from Approach 1 (or prior work) to set those values. This makes it less useful as a standalone method but valuable as a complementary verification of Approach 1's predictions.

Approach 3: Parametric Fit of Loss Function

Approaches 1 and 2 assume compute-optimal training: non-vocabulary parameters and training data are scaled equally (Chinchilla regime). However, many real-world LLMs are deliberately trained on more data than would be compute-optimal for their size, because this produces smaller models that are cheaper to serve at inference time. Approach 3 extends the framework to handle these non-optimal regimes.

The parametric loss function. The paper proposes that the unigram-normalized loss $L_u$ can be decomposed into additive contributions from non-vocabulary parameters, vocabulary parameters, and training data:

$L_u = -E + \frac{A_1}{N_{nv}^{\alpha_1}} + \frac{A_2}{N_v^{\alpha_2}} + \frac{B}{D^\beta}$

where $D = H f(V)$ is the number of training tokens, and $E$ , $A_1$ , $A_2$ , $B$ , $\alpha_1$ , $\alpha_2$ , $\beta$ are learned parameters.

What each term represents. The first term $-E$ is the irreducible loss—the minimum achievable $L_u$ even with infinite parameters and data, representing the entropy of natural language after accounting for the unigram baseline. The term $\frac{A_1}{N_{nv}^{\alpha_1}}$ captures how loss decreases as non-vocabulary parameters increase: more Transformer capacity reduces loss, but with diminishing returns ( $\alpha_1 > 0$ means the denominator grows, making the term smaller). Similarly, $\frac{A_2}{N_v^{\alpha_2}}$ captures how vocabulary capacity reduces loss: larger vocabularies provide richer representations, with diminishing returns. The term $\frac{B}{D^\beta}$ captures how more training data reduces loss, again with diminishing returns.

Why this additive form. This decomposition follows the classical risk decomposition used in Hoffmann et al. and earlier scaling literature: total loss equals irreducible loss plus separate terms for model capacity and data. The innovation is adding the vocabulary term $\frac{A_2}{N_v^{\alpha_2}}$ as a separate component, allowing the model to learn how vocabulary size independently affects loss beyond its effect on tokenization (which is captured through $D = H f(V)$ ).

Constraint from prior work. Following Muennighoff et al. (2024), the paper constrains $\alpha_1 = \beta$ , meaning non-vocabulary parameters and training data affect loss through the same functional form (the exponents governing diminishing returns are equal). This is consistent with the Chinchilla finding that parameters and data should be scaled equally for compute-optimal training.

Fitting procedure. The loss function has 7 free parameters: $E$ , $A_1$ , $A_2$ , $B$ , $\alpha_1$ , $\alpha_2$ , $\beta$ (with $\beta = \alpha_1$ as a constraint). Fitting uses all training runs from the experiments in Section 4.1—not just the compute-optimal points selected for Approach 1, but every ( $N_{nv}$ , $N_v$ , $H$ , $L_u$ ) tuple from every training run. This larger dataset enables learning how loss varies across the full configuration space, including suboptimal allocations.

Fitting is performed by minimizing Huber loss:

$\min_{a_1, a_2, b, e, \alpha_1, \alpha_2} \text{Huber}_\delta(-\exp(e) + \exp(a_1 - \alpha_1 \log(N_{nv})) + \exp(a_2 - \alpha_2 \log(N_v)) + \exp(b - \beta \log(D)), L_u)$

where $A_1 = \exp(a_1)$ , $A_2 = \exp(a_2)$ , $B = \exp(b)$ , $E = \exp(e)$ . The exponential parameterization ensures these coefficients are positive (since they represent magnitudes of positive contributions to loss). The Huber delta is 0.001 as in Approach 1.

Parameters are initialized from uniform grids: $a_1, a_2, b \in [0, 5]$ , $e \in [0, 2]$ , $\alpha_1 \in [0, 1]$ , $\alpha_2 \in [0, 1]$ , with 3 initial guesses each. A constraint $0.1 < \alpha_1, \alpha_2 < 1$ is added based on the prior that scaling exponents for neural networks typically fall in this range. Points with very small FLOPs are filtered out following Hoffmann et al., as these tend to be noisy and can distort the fit.

Fitted parameters. The resulting values are: $A_1 = 1.831$ , $A_2 = 0.196$ , $B = 2.124$ , $E = 5.533$ , $\alpha_1 = \beta = 0.447$ , $\alpha_2 = 0.671$ .

Interpreting the fitted values. $\alpha_2 = 0.671$ is larger than $\alpha_1 = 0.447$ , meaning the vocabulary term $\frac{A_2}{N_v^{\alpha_2}}$ decays faster with increasing $N_v$ than the non-vocabulary term $\frac{A_1}{N_{nv}^{\alpha_1}}$ decays with increasing $N_{nv}$ . This implies that increasing vocabulary parameters provides diminishing returns more quickly than increasing non-vocabulary parameters—consistent with the finding that vocabulary should scale slower ( $\gamma < 1$ ). The coefficient $A_2 = 0.196$ is much smaller than $A_1 = 1.831$ , meaning that at small parameter counts, non-vocabulary parameters dominate the loss, but vocabulary becomes relatively more important as both grow (since the faster exponent on $N_v$ means its term shrinks more quickly, so at large scales both terms are small but the vocabulary term shrinks proportionally faster).

Finding the optimal vocabulary. With the loss function fitted, the optimal $V$ for any $(N_{nv}, H)$ configuration can be found by substituting $D = H f(V)$ and minimizing $L_u$ with respect to $V$ . For compute-optimal training, $H$ is determined from the FLOPs budget constraint $C$ : since $C \approx 6(N_{nv} + Vd)H f(V)$ , we can express $H f(V) = C / (6(N_{nv} + Vd))$ , substitute into the loss, and solve $\frac{\partial L_u}{\partial V} = 0$ .

The derivative is derived in detail in Appendix A.2:

$\frac{\partial L_u}{\partial V} = -\alpha_2 \frac{A_2 d}{(V d)^{\alpha_2+1}} + \beta \frac{B}{C} \cdot \frac{d \cdot 6(N_{nv} + Vd)}{(C / (6(N_{nv} + Vd)))^{\beta+1}} \cdot \frac{1}{6(N_{nv} + Vd)^2}$

The first term (negative) captures how increasing $V$ reduces loss through improved vocabulary representation (more parameters → lower loss). The second term (positive because of the overall structure of the derivative) captures how increasing $V$ diverts FLOPs from training non-vocabulary parameters or processing more data, increasing loss. The optimal $V$ balances these forces, found numerically via scipy.optimize.fsolve.

The key advantage: handling non-optimal training regimes. Because the loss function is fitted on all training points (not just compute-optimal ones), it can predict the locally optimal vocabulary for any $(N_{nv}, H)$ pair, not just those satisfying the Chinchilla equal-scaling condition. This is crucial for modern LLM training where models are often intentionally overtrained (trained on more data than compute-optimal for their size to reduce inference costs).

Usage for non-optimal regimes. Given a specific model with known $N_{nv}$ and known training characters $H$ (derived from the actual training token count and the compression function), you can find the vocabulary that minimizes $L_u$ by solving $\frac{\partial L_u}{\partial V} = 0$ numerically. This yields predictions that adapt to the specific training regime: for the same $N_{nv} = 2.87$ B, the optimal vocabulary is 24K when undertraining (few characters), 35K when compute-optimally training, and 43K when overtraining (many characters), as shown in Table 3.

Predictions for popular models (Figure 6). The paper applies Approach 3 to existing LLMs using their reported training token counts and non-vocabulary parameters. The results show that most models significantly underallocate vocabulary parameters relative to the locally optimal prediction. For example, Llama2-7B ( $N_{nv} \approx 7$ B, trained on 2T tokens) has approximately 0.13B vocabulary parameters (32K vocabulary × 4096 embedding dimension), while Approach 3 predicts approximately 0.68B optimal vocabulary parameters (corresponding to a vocabulary of approximately 166K). Llama2-70B ( $N_{nv} \approx 70$ B) has approximately 0.26B vocabulary parameters, while the optimal is predicted at approximately 1.8B (corresponding to approximately 218K vocabulary). The exception is Gemma2-9B, which appears to have near-optimal vocabulary allocation.

Why the predicted optimal changes with training data volume (Figure 7). The paper provides a heatmap analysis showing how the best vocabulary size shifts with training data for a fixed $N_{nv} = 302$ M. When training data is scarce (left side of heatmap, low FLOPs budget), the optimal vocabulary is small (16K drops to 10K)—rare tokens would be undertrained, so a smaller vocabulary protects against overfitting. When training data is abundant (right side, high FLOPs budget), the optimal vocabulary grows (16K → 24K)—with more tokens, rare tokens get sufficient gradient updates, and the benefits of larger vocabulary coverage can be realized. This explains why models like Llama2, which are overtrained relative to Chinchilla-optimal, should use even larger vocabularies than the compute-optimal prediction.

Recommendation despite overtraining. The paper notes that "expanding the vocabulary size also increases the computational demands during inference" (Section 5). Therefore, even when overtraining suggests an even larger vocabulary would be optimal, the paper recommends using the vocabulary size corresponding to compute-optimal training for that $N_{nv}$ —trading off some training efficiency for better inference efficiency, since inference costs accumulate over the model's entire deployment lifetime.

Summary of Design Choices and Their Justifications

Unigram-normalized loss ( $L_u$ ) over standard cross-entropy: The standard loss is vocabulary-dependent by construction (more prediction options → higher loss regardless of model quality). $L_u$ normalizes by token frequency, making the fair comparison required for vocabulary scaling analysis possible. Validated empirically by the reversal from positive to negative correlation with downstream performance (Figure 13).
Training characters ( $H$ ) over training tokens ( $D$ ) as the data measure: Since vocabulary size changes how many tokens represent the same text, using $D$ would confound vocabulary effects with data quantity effects. $H$ isolates data volume from tokenization, and the compression function $f(V)$ provides the mapping.
Parameter decomposition ( $N = N_{nv} + Vd$ ) over lumped parameters: Non-vocabulary and vocabulary parameters have different scaling dynamics (the former benefits from depth and width; the latter only from width). Separate treatment enables learning their distinct power-law exponents and the $\gamma < 1$ relationship.
Quadratic-in-log compression function ( $f(V) = a\log^2(V) + b\log(V) + c$ ) over alternatives: Captures diminishing returns as vocabulary grows, with only three parameters fitted once from tokenizer statistics. Validated across BPE, unigram, and word-based tokenizers with high $R^2$ .
Three complementary approaches rather than one: Approach 1 provides the most direct empirical evidence but requires extensive training. Approach 2 is computationally cheap but relies on assumptions about the FLOPs equation. Approach 3 handles non-optimal training regimes but requires fitting a 7-parameter function. Their convergence on $\gamma \approx 0.83$ triangulates the finding.
Huber loss ( $\delta = 0.001$ ) over MSE for fitting: Prevents outlier training runs from dominating the fit, while still penalizing large errors linearly rather than ignoring them entirely.
LBFGS optimization with multiple random restarts: The parameter spaces are low-dimensional (2–7 parameters) and smooth; LBFGS is efficient and reliable. Multiple restarts avoid local minima.
Vocabulary sizes as multiples of 128: Ensures compatibility with NVIDIA tensor core requirements, making the experimental setup practically reproducible.

4. Key Insights and Innovations

Innovation 1: Vocabulary Size as a Scaling Dimension—Not a Fixed Hyperparameter

The paper's most fundamental contribution is a reconceptualization of vocabulary size from an arbitrary architectural constant to a compute-dependent scaling dimension. Prior to this work, the scaling laws literature—from Kaplan et al. (2020) through Hoffmann et al. (2022) and onward—treated vocabulary as a fixed design choice made before scaling analysis begins. Kaplan et al. explicitly excluded vocabulary parameters from their model size variable $N$ , and Hoffmann et al.'s Chinchilla laws optimize the allocation of parameters versus data while assuming vocabulary is already decided. This created a conceptual blind spot: the scaling laws community systematically optimized over two dimensions (model size, data volume) while holding a third dimension fixed, without ever testing whether that third dimension should itself scale.

The paper closes this gap by demonstrating that vocabulary size is not merely a design choice but a variable that should grow with compute budget, following its own power-law relationship. The convergence of three methodologically distinct approaches on $\gamma \approx 0.83$ —IsoFLOPs fitting (Section 4.1), derivative-based estimation (Section 4.2), and parametric loss modeling (Section 4.3)—elevates this from an empirical curiosity to a robust finding. The fact that these approaches make different assumptions (Approach 1 requires extensive training runs; Approach 2 relies on the FLOPs equation's accuracy; Approach 3 handles non-optimal regimes) and yet produce consistent predictions triangulates the result.

What makes this more than "vocabulary should be bigger." The paper does not simply argue that existing vocabularies are too small—that would be an empirical observation, not an intellectual contribution. The innovation is in establishing that vocabulary has a quantifiable, predictable scaling trajectory analogous to model parameters and training data. The relationship $N_v^{\text{opt}} \propto N_{nv}^{0.83}$ means that vocabulary scaling is not linear with model scale—it grows, but more slowly than non-vocabulary parameters. This is conceptually significant because it shows that vocabulary occupies a distinct scaling regime: it is not simply "another kind of parameter" that should be scaled identically to Transformer blocks. The structural difference—vocabulary parameters only benefit from width scaling, while non-vocabulary parameters benefit from both depth and width—manifests quantitatively in the $\gamma < 1$ exponent.

Comparison to prior assumptions. The dominant implicit assumption in the field has been that vocabulary size is determined by tokenizer quality and domain coverage requirements, not by compute optimality. This assumption is visible in the heterogeneous vocabulary choices across model families (32K for Llama2, 256K for Gemma, 128K for Llama3) made without systematic justification. The paper provides the first framework for determining whether these choices are near-optimal or systematically misallocated. The finding that most existing LLMs underallocate vocabulary parameters by 3–7× (Figure 2 and Figure 6) transforms vocabulary from a "set it once and forget it" hyperparameter to a dimension that demands the same careful optimization as model size and data volume.

Significance for how the field thinks about scaling. This innovation implies that scaling laws are fundamentally incomplete without vocabulary. A three-dimensional optimization (non-vocabulary parameters, vocabulary parameters, training data) replaces the two-dimensional optimization (parameters, data) that has dominated the field. This is a structural change to the scaling laws framework, not an incremental refinement. The practical consequence—that getting vocabulary wrong at the scale of a 70B-parameter model means misallocating billions of parameters—underscores that this is not merely a theoretical concern.

Innovation 2: The Vocabulary-Insensitive Loss Metric as a Diagnostic Tool

The paper develops and validates a metric that makes vocabulary scaling studies possible in the first place: the unigram-normalized loss $L_u$ . While the metric itself is adapted from Roh et al. (2020), the paper's contribution is in recognizing that this metric is necessary for vocabulary scaling analysis and in providing the empirical validation that demonstrates why standard cross-entropy loss is broken for this purpose.

The diagnostic problem the metric solves. Standard language modeling loss creates a perverse incentive when comparing across vocabulary sizes: models with larger vocabularies have higher loss because they must distribute probability mass across more tokens, even when they have better language understanding. The paper's empirical demonstration of this—showing a positive correlation between standard loss and downstream performance when vocabulary varies (Appendix A.10, Figure 13a)—is a stark diagnostic. It means that any naive attempt to optimize vocabulary by comparing standard cross-entropy losses would systematically select too-small vocabularies, because smaller vocabularies artificially lower the loss metric. The field has been using a broken yardstick for vocabulary comparison without recognizing it.

What $L_u$ does differently. By subtracting the unigram log-probability of each token, $L_u$ measures the model's improvement over a context-free baseline. This normalization exactly compensates for the vocabulary-size-dependent difficulty of the prediction task: larger vocabularies make each individual token less frequent, which makes the unigram baseline worse, which adds a larger negative offset to $L_u$ , exactly canceling the artificial loss inflation. The metric thus isolates the model's genuine language understanding from the mechanical effect of vocabulary size on prediction difficulty. The paper validates this by showing that $L_u$ exhibits the expected negative correlation with downstream performance (Figure 13b)—lower $L_u$ means better downstream accuracy—unlike standard loss which shows a perverse positive correlation.

Why this matters beyond this paper. The metric is not just a tool for this study; it is a prerequisite for any future work on vocabulary scaling. Without a vocabulary-insensitive loss, researchers cannot run IsoFLOPs experiments varying vocabulary size and fairly compare the results. The paper essentially provides the measurement apparatus that enables vocabulary to be treated as a scaling dimension at all. The strong linear correlation with BPC (Pearson $\rho = 0.9888$ , Appendix A.5) provides additional validation and connects $L_u$ to the information-theoretic compression interpretation of language modeling.

The subtle limitation that makes this an insight rather than a solved problem. The paper is transparent that $L_u$ is an empirical construct—it relies on estimating unigram token frequencies from the training corpus, and these estimates themselves depend on the tokenizer. This creates a mild circularity: the normalization term $p(w_i | V)$ changes with $V$ , so $L_u$ is not perfectly vocabulary-invariant in a theoretical sense. However, the paper demonstrates that in practice, it produces the correct monotonic relationship with downstream performance, which is the operational requirement. This pragmatic approach—validate the metric empirically rather than prove theoretical invariance—is appropriate for the empirical scaling law methodology.

Innovation 3: The Compute-Dependence of Optimal Vocabulary as a Resolution to Conflicting Industry Practices

The paper's finding that optimal vocabulary size depends on both model scale and training data volume provides a unified explanation for the field's inconsistent vocabulary choices, resolving what might otherwise appear as arbitrary variation across model families. This is not merely a prediction—it is a diagnostic framework that explains why different models made different (and differently suboptimal) choices.

The explanatory framework. The paper identifies three regimes that produce different optimal vocabulary sizes for the same non-vocabulary parameter count:

Undertraining (insufficient data): The optimal vocabulary is smaller than the compute-optimal size, because rare tokens would be undertrained. This explains why early models trained on limited data (pre-2020) tended toward smaller vocabularies (8K–32K).
Compute-optimal training (Chinchilla regime): The optimal vocabulary follows $N_v^{\text{opt}} \propto N_{nv}^{0.83}$ . This is the baseline recommendation for new training runs that can afford to scale data with model size.
Overtraining (excessive data, common in modern LLMs): The optimal vocabulary is larger than the compute-optimal size, because abundant data provides sufficient gradient updates for rare token embeddings. This explains why the trend toward larger vocabularies (Llama3's 128K vs. Llama2's 32K, Gemma's 256K) accompanies the trend toward training on far more tokens than Chinchilla-optimal.

This three-regime framework is empirically validated in Table 3: for $N_{nv} = 2.87$ B, the optimal vocabulary is 24K under undertraining ( $2.8 \times 10^{20}$ FLOPs), 35K under compute-optimal training ( $1.2 \times 10^{21}$ FLOPs), and 43K under overtraining ( $2.3 \times 10^{21}$ FLOPs). These are not marginal differences—the optimal vocabulary varies by nearly 2× depending on the training regime.

Why this resolves the apparent contradiction. Without this framework, one might look at Gemma-7B (256K vocabulary) and Llama2-7B (32K vocabulary)—both similar-sized models—and conclude that someone made a large error. The paper shows that both choices were likely suboptimal, but in different directions and for different reasons. Models trained in the overtrained regime (like Llama2-7B, trained on 2T tokens when Chinchilla-optimal would be ~150B) should have larger vocabularies than compute-optimal predictions, yet Llama2-7B used only 32K. This means its vocabulary error is even more severe than the compute-optimal comparison suggests. Conversely, models that came closer to compute-optimal training and used larger vocabularies (like Gemma2) may be nearer to their locally optimal allocation.

The practical recommendation that emerges. The paper recommends using the compute-optimal vocabulary size even when overtraining, because larger vocabularies increase inference costs (the output softmax over $V$ tokens can dominate latency). This is a nuanced tradeoff: training efficiency would favor an even larger vocabulary under overtraining, but the cumulative inference cost over the model's deployment lifetime argues for restraint. This recommendation emerges not from a simple "bigger is better" finding but from a careful analysis of how the two competing effects of vocabulary size—better tokenization fertility (training benefit) and larger output softmax (inference cost)—play out across different deployment scenarios.

The diagnostic Figure 7 heatmap. This figure encodes the regime-dependent behavior compactly: for a fixed $N_{nv} = 302$ M, the optimal vocabulary shifts from 10K at the lowest data volumes to 24K at the highest, tracing a diagonal path through the configuration space. This single visualization captures the paper's central insight that vocabulary optimality is not a point estimate but a function of the full training configuration.

Innovation 4: Diagnosing Vocabulary Undertraining via Embedding Degeneration

The paper provides direct empirical evidence for the mechanism that limits vocabulary scaling, connecting the statistical intuition (rare tokens get insufficient gradient updates) to observable geometric structure in the learned embedding space. This transforms the undertraining concern from a theoretical worry into a diagnosable phenomenon.

The diagnostic approach. Using SVD visualizations of learned word embeddings (Appendix A.3, Figure 9), the paper shows that embeddings from a 64K vocabulary trained with constrained FLOPs exhibit significantly more clustering than those from a 16K vocabulary. The quantitative metric—average Euclidean distance between embeddings—decreases from 1.011 (16K) to 0.952 (64K), indicating that rare-token embeddings collapse toward each other. This is presented alongside the qualitative observation that "low-frequency word embeddings cluster together due to limited parameter updating," citing the representation degeneration literature (Gao et al., 2019).

Why this matters for the broader argument. The undertraining mechanism is intuitive—parameters that rarely appear in training data cannot be reliably estimated—but intuitions in deep learning are cheap. The paper provides geometric evidence that this intuition manifests measurably in the embedding space, and that the degree of degeneration correlates with vocabulary size. This does two things: (1) it validates the theoretical basis for why optimal vocabulary is bounded, and (2) it provides a diagnostic tool that could be used to detect vocabulary oversizing without benchmarking downstream tasks. If future models show embedding degeneration patterns similar to the 64K panel in Figure 9, that is prima facie evidence that their vocabulary is too large for their training budget.

Connection to the cost model. The embedding degeneration finding provides mechanistic justification for the U-shaped FLOPs curve in Figure 3. At small $V$ , increasing vocabulary improves tokenization fertility without underfitting embeddings, so total FLOPs decrease. At large $V$ , the marginal improvement in tokenization fertility is negligible (since $f(V)$ saturates), while the embedding parameters suffer from increasingly severe degeneration, wasting parameters that could have been allocated to Transformer blocks. The optimal $V$ sits at the transition point where these effects balance.

What this means for future work on vocabulary. The paper does not explore interventions to mitigate embedding degeneration (e.g., frequency-based regularization, adaptive embedding dimensions, or sparse updates), but the diagnostic itself points toward such interventions. If one could train large-vocabulary embeddings without degeneration—perhaps through better initialization or specialized optimization—the optimal vocabulary size might be larger than what the paper predicts. The current predictions are thus specific to standard training procedures, and the embedding degeneration diagnostic provides a way to test whether alternative procedures shift the optimal vocabulary.

Limitation. The SVD analysis is performed on a single small model ( $N_{nv} = 85$ M) and is qualitative rather than statistically rigorous. This is a minor supporting finding rather than a central empirical result, but it illustrates the kind of mechanistic evidence that strengthens scaling law claims beyond pure curve-fitting, connecting the macro-level optimization (which $V$ minimizes loss?) to micro-level model behavior (what happens to individual embeddings when $V$ is too large?).

Innovation 5: The FLOPs-Minimization Perspective—Revealing the Dual Role of Vocabulary in Compute Cost

Approach 2's formulation—finding the vocabulary that minimizes FLOPs for a target loss rather than minimizing loss for a target FLOPs budget—introduces a complementary optimization perspective that reveals the dual role of vocabulary in compute cost. While this is framed as an estimation method, it provides conceptual clarity that the loss-minimization perspective (Approach 1) obscures.

The conceptual move. In Approach 1, vocabulary size affects loss through two pathways: larger $V$ increases parameters (which should reduce loss, all else equal) and changes tokenization (which changes effective data volume). The optimal $V$ emerges as the point where these effects balance in loss space. This is intuitive but conflates two distinct mechanisms.

Approach 2 reframes the problem: for a fixed model capability (target loss), what vocabulary minimizes the computational cost of reaching that capability? This perspective isolates vocabulary's effect on FLOPs through a single equation, $\frac{\partial C}{\partial V} = 0$ , where the two competing terms—parameter cost vs. tokenization efficiency—are directly visible:

$\frac{\partial C}{\partial V} = 6H\left[(N_{nv} + Vd)\frac{2a \log(V) + b}{V} + f(V) \cdot d\right]$

The first term (containing the log derivative) captures how increasing $V$ changes the number of tokens needed: at small $V$ , this term is large and negative (tokenization improves dramatically), while at large $V$ , it approaches zero (saturation). The second term ( $f(V) \cdot d$ ) captures the fixed cost of adding another vocabulary entry: each new token adds $d$ parameters that must be multiplied by every training token. The optimal $V$ is where the marginal benefit of better tokenization equals the marginal cost of additional parameters.

Why this perspective matters beyond estimation. The FLOPs-minimization framing makes explicit what the loss-minimization framing leaves implicit: vocabulary choice is fundamentally about trading off parameter efficiency against data efficiency. A larger vocabulary converts character-processing capacity into token-processing capacity—it lets you process the same raw text with fewer tokens, which is like getting more data throughput per FLOP. But this conversion has diminishing returns (captured by the logarithmic derivative) while the parameter cost is linear. This tradeoff is isomorphic to the classic "model size vs. data" tradeoff in scaling laws, but operating at the tokenization level. The paper essentially shows that tokenization is a form of learned compression whose compression ratio is itself a function of compute investment (via vocabulary size), creating a nested optimization within the outer parameters-vs-data optimization.

The enabling role of the compression function. This perspective is only possible because of $f(V)$ —the fitted quadratic-in-log function that maps vocabulary size to tokens-per-character. Without this function, vocabulary size and data volume are confounded, and the FLOPs-minimization derivative cannot be taken. The $f(V)$ fit is thus not merely a practical convenience but a conceptual prerequisite for treating vocabulary as an optimizable variable. Its robustness across tokenizer types (BPE, unigram, word-based, as shown in Appendix A.9) means the perspective generalizes beyond the specific tokenization algorithm used in the paper.

The limitation that defines the approach's role. Approach 2 cannot independently determine the optimal $N_{nv}$ and $H$ —it requires these values from Approach 1 or prior scaling laws. This means it is a partial equilibrium analysis: it finds the optimal vocabulary given the outer-loop allocation of parameters and data, rather than solving the full three-dimensional optimization simultaneously. The paper is transparent about this (Appendix B.1), but it means Approach 2's predictions are conditional on the outer-loop allocation being correct. This is not a flaw—it is a useful decomposition of the optimization problem into separable steps—but it prevents Approach 2 from being a standalone scaling law. Its value is as a fast, interpretable complement to the full empirical approach.

Figure 8 from Appendix A.1 as the conceptual summary. The middle panel shows that at the derivative-optimal $V$ , FLOPs achieve their minimum for the given loss target. The right panel shows that this same $V$ maximizes the number of training characters that can be processed—the vocabulary is large enough to compress text efficiently but not so large that parameter overhead dominates. This dual interpretation (minimum FLOPs, maximum characters) captures the vocabulary optimization's essence in a single visualization.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the SlimPajama dataset (Soboleva et al., 2023), a 627B-token cleaned and deduplicated version of RedPajama. Training data is uniformly sampled from different domains for each run, and validation loss is evaluated on a held-out portion. For downstream evaluation, the paper uses 7 standard benchmarks: ARC-Challenge, ARC-Easy, HellaSwag, OpenBookQA, WinoGrande, PIQA, and BoolQ—all evaluated in zero-shot using the lighteval framework.
Base model(s). The paper trains custom Llama-architecture models (Vaswani-style Transformers with rotary position embeddings and SwiGLU activations) from scratch at 7 non-vocabulary parameter scales: 33M, 85M, 151M, 302M, 631M, 1130M, and 2870M parameters. All models use configurable BPE tokenizers trained on SlimPajama with vocabulary sizes ranging from 4K to 96K. This range is chosen because "the optimal vocabulary sizes for all the training configurations we used fall in this range" (Appendix A.4). The 2.87B model is used for the main validation experiments (Tables 2 and 3), while smaller models provide the data for fitting the three prediction approaches.
Metrics. The primary training metric is the unigram-normalized language modeling loss $L_u$ (Equation 4): $L_u = -\frac{1}{T} \sum_{i=1}^{T} \log \frac{p(w_i | w_{1:i-1}, V)}{p(w_i | V)}$ , where $p(w_i|V)$ is the unigram frequency of token $w_i$ in the training corpus. This metric subtracts the context-free baseline to enable fair comparison across vocabulary sizes. For downstream evaluation, the paper reports accuracy (percentage of questions answered correctly). For multiple-choice tasks, predicted likelihoods are normalized by choice length "to eliminate the effect of text length on predictions" (Table 2 caption). Standard deviations are reported for downstream results based on 3 evaluation runs with different random seeds.
Baselines. The primary baseline is the conventional vocabulary size of $V = 32K$ , which the paper identifies as the most common choice among current LLMs (used by Llama2, among others). This is not a method from prior work but rather the de facto standard that the paper argues against. In the main validation experiments (Tables 2 and 3), models with predicted optimal vocabulary ( $V^{\text{opt}}$ ) are compared directly against models with $V = 32K$ , keeping all other hyperparameters identical.
Generation budget / compute accounting. Compute is measured in FLOPs using the standard Transformer approximation $C \approx 6ND$ , decomposed to account for vocabulary as $C \approx 6(N_{nv} + Vd) \cdot H f(V)$ (Equation 5). For fair comparison, total FLOPs are held constant when comparing vocabulary choices: models with larger $V$ have more parameters but train on fewer tokens (since $f(V)$ decreases), keeping $C$ fixed. Training characters $H$ are held constant within each $N_{nv}$ group—the model with the reference vocabulary size ( $V = 16384$ for most groups, $V = 32K$ for the 2.87B group) determines $H$ , and other vocabulary sizes use the same $H$ with their token counts adjusting via $f(V)$ .
Cross-validation / statistical protocol. For the IsoFLOPs fitting (Approach 1), the paper selects data points with minimum $L_u$ for each FLOPs budget from the set of all training runs, then fits power laws using Huber loss with $\delta = 0.001$ and LBFGS optimization with 20 random restarts. For the parametric loss function (Approach 3), fitting uses all training points (not just compute-optimal ones) with the same Huber loss and LBFGS procedure, filtering out points with "very small FLOPs" following Hoffmann et al. (2022). Prediction quality is assessed via RMSE and $R^2$ values (reported in figure captions for Approach 1's power-law fits and for the compression function $f(V)$ ). For downstream evaluation, standard deviations are computed across 3 evaluation seeds.

Main Quantitative Results

Power-Law Fitting of Compute-Optimal Allocations (Approach 1)

The central empirical result from Approach 1 is that compute-optimal non-vocabulary parameters, vocabulary parameters, and training characters each follow power-law relationships with FLOPs, but with different exponents. Using data from 6 model groups ( $N_{nv}$ ranging from 33M to 1.13B) with vocabulary sizes varied from 4K to 96K, the paper selects the ( $N_{nv}$ , $N_v$ , $H$ ) configuration achieving minimum $L_u$ at each FLOPs budget and fits:

$N_{nv} = 0.08 \cdot C^{0.50}$

$N_v = 0.20 \cdot C^{0.42}$

$H = 6.42 \cdot C^{0.50}$

This is the key quantitative finding that underlies all vocabulary size predictions. The exponent for vocabulary parameters ( $\alpha_2 = 0.42$ ) is smaller than the exponent for non-vocabulary parameters ( $\alpha_1 = 0.50$ ), yielding a scaling ratio $\gamma = \alpha_2 / \alpha_1 = 0.84$ (which rounds to the commonly cited 0.83). Figure 5 visualizes these fits with RMSE and $R^2$ values that the paper characterizes as indicating "the strength of our fit," though exact goodness-of-fit statistics are embedded in the figure captions rather than the main text.

Three implications are drawn directly from these exponents (Section 4.1, "Results and Usage"):

LLMs are data-hungry: the coefficient $k_3 = 6.42$ is much larger than $k_1 = 0.08$ and $k_2 = 0.20$ , meaning that for a given FLOPs budget, the optimal allocation assigns substantially more compute to training data than to parameters—consistent with Hoffmann et al. (2022).
Vocabulary parameters scale in a power-law relation with FLOPs ( $N_v \propto C^{0.42}$ ): this is the first empirical demonstration that vocabulary size should not be held constant as compute increases, but rather should grow with a predictable exponent.
Vocabulary parameters should be scaled slower than non-vocabulary parameters: $\gamma = 0.42/0.50 = 0.84 < 1$ , meaning a 10× increase in $N_{nv}$ corresponds to approximately a $10^{0.84} \approx 6.9\times$ increase in $N_v$ , not a 10× increase.

These fitted relationships are used to produce the predictions in Table 1. For example, for $N_{nv} = 70\text{B}$ , the FLOPs budget from inverting $N_{nv} = 0.08 \cdot C^{0.50}$ is $C = (70\text{B} / 0.08)^{1/0.50} \approx 7.1 \times 10^{23}$ ; then $N_v^{\text{opt}} = 0.20 \cdot (7.1 \times 10^{23})^{0.42} \approx 1.7 \times 10^9$ vocabulary parameters; with $d = 8192$ , $V^{\text{opt}} \approx 212\text{K}$ (Approach 1 column in Table 1).

Derivative-Based Estimation (Approach 2)

Approach 2 produces predictions that closely match Approach 1, as shown in Table 1. For $N_{nv} = 3\text{B}$ with $d = 3200$ , Approach 2 predicts $N_v^{\text{opt}} \approx 0.1\text{B}$ ( $V^{\text{opt}} \approx 43\text{K}$ ), compared to Approach 1's $N_v^{\text{opt}} \approx 0.1\text{B}$ ( $V^{\text{opt}} \approx 39\text{K}$ ). For $N_{nv} = 70\text{B}$ with $d = 8192$ , Approach 2 predicts $N_v^{\text{opt}} \approx 1.9\text{B}$ ( $V^{\text{opt}} \approx 231\text{K}$ ), compared to Approach 1's $V^{\text{opt}} \approx 212\text{K}$ .

The fitted scaling exponent from Approach 2 is $\gamma = 0.83$ , consistent with the 0.84 from Approach 1. The procedure for obtaining this exponent is: compute derivative-optimal $N_v$ for different $N_{nv}$ values by solving $\partial C / \partial V = 0$ numerically (Equation 7), then fit $N_v \propto N_{nv}^\gamma$ to the resulting pairs (Equation 12 in Appendix A.7.4). The specific numerical values for the derivative-optimal pairs are not tabulated individually, but the resulting $\gamma = 0.83$ is reported in Section 4.2 and used for all Approach 2 predictions.

The practical advantage of Approach 2 is its computational cost. Once the compression function $f(V)$ is fitted (requiring only tokenizer training, not LLM training) and the scaling exponent $\gamma$ is estimated from a small reference model, predictions for any scale are obtained via numerical root-finding without additional training runs. The paper notes that "its reliance on numerical solutions rather than exhaustive deep learning experiments makes it rapid and broadly applicable across various tokenizers" (Appendix B.1).

Parametric Loss Function Fit (Approach 3)

Approach 3 fits the loss function $L_u = -E + A_1/N_{nv}^{\alpha_1} + A_2/N_v^{\alpha_2} + B/(H f(V))^\beta$ to all training points (not just compute-optimal ones), yielding (Section 4.3):

$L_u = -5.533 + \frac{1.831}{N_{nv}^{0.447}} + \frac{0.196}{N_v^{0.671}} + \frac{2.124}{(H f(V))^{0.447}}$

The key quantitative result is the vocabulary exponent $\alpha_2 = 0.671$ , which is larger than $\alpha_1 = 0.447$ . This means that the vocabulary term $\frac{A_2}{N_v^{\alpha_2}}$ decays faster with increasing $N_v$ than the non-vocabulary term decays with increasing $N_{nv}$ —vocabulary parameters exhibit stronger diminishing returns. At small parameter counts, non-vocabulary parameters dominate the loss (since $A_1 = 1.831$ is much larger than $A_2 = 0.196$ ), but as both $N_{nv}$ and $N_v$ grow, the vocabulary term shrinks proportionally faster due to the larger exponent, meaning vocabulary becomes relatively less important at scale—consistent with the $\gamma < 1$ finding.

Predictions from Approach 3 closely align with Approaches 1 and 2 in the compute-optimal setting (Table 1): for $N_{nv} = 3\text{B}$ , $V^{\text{opt}} \approx 37\text{K}$ ; for $N_{nv} = 70\text{B}$ , $V^{\text{opt}} \approx 218\text{K}$ .

The unique value of Approach 3 is handling non-optimal training regimes. By fixing $N_{nv}$ and varying the FLOPs budget $C$ , the paper predicts locally optimal vocabulary sizes for different training data volumes. This is visualized in the heatmap of Figure 7 (left) for $N_{nv} = 302\text{M}$ : as available training data increases (moving right on the x-axis), the best vocabulary size among the tested options shifts from 10K at the lowest data volumes to 24K at the highest. The paper's numerical results confirm this pattern at scale (Table 3): for $N_{nv} = 2.87\text{B}$ , Approach 3 predicts $V^{\text{opt}} = 24\text{K}$ under undertraining ( $2.8 \times 10^{20}$ FLOPs, insufficient data), $V^{\text{opt}} = 35\text{K}$ under compute-optimal training ( $1.2 \times 10^{21}$ FLOPs), and $V^{\text{opt}} = 43\text{K}$ under overtraining ( $2.3 \times 10^{21}$ FLOPs, overly sufficient data).

Figure 6 provides the real-world diagnostic. Using Approach 3 with each model's actual reported training tokens (not assuming compute-optimal allocation), the paper predicts locally optimal vocabulary parameters for popular LLMs. All models except Gemma2-9B allocate fewer vocabulary parameters than predicted. Specific comparisons: Llama2-7B's actual $N_v \approx 0.13\text{B}$ vs. predicted optimal $\approx 0.68\text{B}$ ; Llama2-70B's actual $N_v \approx 0.26\text{B}$ vs. predicted optimal $\approx 1.8\text{B}$ ; DeepSeek-67B, Falcon-180B, and Qwen2-72B similarly underallocate. Gemma2-9B is the notable exception, with its vocabulary parameters closest to the prediction.

Validation via Downstream Evaluation

Table 2 reports the core validation experiment at the largest tested scale ( $N_{nv} = 2.87\text{B}$ ) under compute-optimal training ( $1.2 \times 10^{21}$ FLOPs). The model with $V = 32\text{K}$ (the conventional choice) is compared against the model with Approach 3's predicted $V^{\text{opt}} = 35\text{K}$ :

Metric	$V = 32\text{K}$	$V^{\text{opt}} = 35\text{K}$
ARC-Challenge	28.5 ± 1.3	29.1 ± 1.3
ARC-Easy	49.2 ± 1.0	50.6 ± 1.0
HellaSwag	47.5 ± 0.5	48.1 ± 0.5
OpenBookQA	31.6 ± 2.1	31.6 ± 2.1
WinoGrande	50.4 ± 1.4	51.9 ± 1.4
PIQA	71.4 ± 1.1	71.4 ± 1.1
BoolQ	56.4 ± 0.9	57.1 ± 0.9
Average	47.9	48.5

The optimal-vocabulary model outperforms on 5 of 7 tasks and matches on 2, with an average improvement of 0.6 percentage points. While individual task improvements are within standard deviations for most tasks, the consistent direction of improvement across tasks supports the prediction, as the paper notes. The $V^{\text{opt}}$ model uses $N_v = 0.11\text{B}$ vs. $N_v = 0.10\text{B}$ for the baseline—a 10% increase in vocabulary parameters that yields the observed gains with the same total FLOPs (since training characters $H$ are matched, and the slightly larger vocabulary processes slightly fewer tokens $D = 67.1\text{B}$ vs. 67.3B).

Table 3 extends the validation to non-optimal training regimes. For $N_{nv} = 2.87\text{B}$ :

Under the undertraining regime ( $2.8 \times 10^{20}$ FLOPs), Approach 3 predicts $V^{\text{opt}} = 24\text{K}$ (smaller than the conventional 32K). The results show: the $V^{\text{opt}} = 24\text{K}$ model achieves an average downstream score of 43.9 vs. 43.2 for $V = 32\text{K}$ , with the largest gains on HellaSwag (34.4 → 36.0) and BoolQ (59.8 → 61.5). This validates the prediction that with insufficient data, a smaller vocabulary is optimal because rare tokens would be undertrained.
Under the overtraining regime ( $2.3 \times 10^{21}$ FLOPs), Approach 3 predicts $V^{\text{opt}} = 43\text{K}$ (larger than the conventional 32K). The results show: the $V^{\text{opt}} = 43\text{K}$ model achieves an average downstream score of 51.6 vs. 50.3 for $V = 32\text{K}$ , with the largest gains on ARC-Challenge (29.1 → 32.0, a 2.9 percentage point improvement), BoolQ (59.5 → 61.9), and PIQA (72.0 → 72.6). This validates the prediction that with abundant data, a larger vocabulary can be effectively trained and yields substantive performance improvements.

The ARC-Challenge gain of +2.9 points (from 29.1 to 32.0) under overtraining is the paper's headline validation number, reported in the abstract. This represents a ~10% relative improvement on a challenging reasoning benchmark, achieved solely by adjusting vocabulary size while holding FLOPs constant. The paper frames this as evidence that vocabulary allocation errors have been costing models measurable performance.

Consistency of the results across regimes. The three validation scenarios (undertraining, compute-optimal, overtraining) demonstrate a consistent pattern: adjusting vocabulary to the predicted optimal improves downstream performance in all cases, but the direction of adjustment differs (smaller vocabulary for undertraining, larger for compute-optimal and overtraining). This validates the core claim that optimal vocabulary depends on the training regime, not just model scale.

Comparison of All Three Approaches

Table 1 provides the side-by-side comparison for models ranging from 3B to 300B non-vocabulary parameters, assuming compute-optimal training. Key observations:

All three approaches agree on the qualitative direction: larger models need larger vocabularies.
All three agree that existing models (as shown in Figure 2) systematically underallocate vocabulary parameters.
For a 70B-parameter model, the approaches predict vocabularies of 212K, 231K, and 218K respectively—all roughly 7× the 32K used by Llama2-70B.
For a 300B model, predictions converge on 356K–389K, suggesting vocabularies approaching 400K may be optimal at the largest scales.

The convergence of predictions across methods is taken as evidence of robustness. The paper states that "the predictions from all proposed approaches align closely" (Section 5), and the variations (e.g., 212K vs. 231K for 70B parameters) are small relative to the gap from current practice (32K).

Ablation Studies and Robustness Checks

Compression function $f(V)$ robustness across tokenizer types: The paper fits $f(V) = a\log^2(V) + b\log(V) + c$ not only for BPE tokenizers (the main experiments) but also for unigram and word-based tokenizers (Appendix A.9, Figure 12). All three tokenizer types show high $R^2$ values (0.99 for BPE, 0.98 for unigram, 0.99 for word-based) and low RMSE values. BPE and unigram tokenizers show similar tokens-per-character ratios, while word-based tokenizers require substantially more tokens per character, as expected. This robustness check validates that the quadratic-in-log functional form is not specific to BPE and supports the generality of the derivative-based approach (Approach 2) across tokenization algorithms.

Vocabulary-insensitive loss validation ( $L_u$ vs. downstream performance): The paper validates that $L_u$ is a fair metric for comparing models with different vocabulary sizes by showing it exhibits the expected negative correlation with downstream performance (Appendix A.10, Figure 13b). Models with varying $V$ but fixed $N_{nv}$ are evaluated: when using standard cross-entropy loss $L$ , the correlation with downstream performance is positive (higher loss → better downstream, Figure 13a), which would penalize larger vocabularies unfairly. When using $L_u$ , the correlation becomes negative (lower loss → better downstream, Figure 13b), confirming that $L_u$ correctly identifies better models regardless of vocabulary size. The fit includes a regression line with confidence intervals. This validates the foundational metric without which fair vocabulary comparison is impossible.

Correlation between $L_u$ and BPC: Appendix A.5, Figure 11 shows a strong linear relationship between unigram-normalized loss $L_u$ and bits-per-character (Pearson $\rho = 0.9888$ , linear-fit RMSE = 0.0683), with data from all models at final training steps across all vocabulary sizes. This validates that $L_u$ and BPC—two independently motivated vocabulary-insensitive metrics—capture the same underlying model quality.

Vocabulary size range exploration beyond the fitting range: The paper explores vocabulary sizes from 0.5K to 512K for $N_{nv} = 33\text{M}$ (Appendix A.4, Figure 10), well beyond the 4K–96K range used for scaling law fitting. The loss curves consistently show a U-shape with performance degrading as vocabulary moves beyond the optimal point, confirming that the existence of an optimal vocabulary is not an artifact of the tested range. At 512K vocabulary, the loss is substantially higher than at 16K–24K, validating that very large vocabularies are genuinely detrimental at this model scale.

SVD visualization of embedding degeneration: Appendix A.3, Figure 9 shows SVD projections of learned word embeddings for $N_{nv} = 85\text{M}$ at $V = 4\text{K}$ , $V = 16\text{K}$ , and $V = 64\text{K}$ . The average Euclidean distance between all embeddings decreases from 1.067 (4K) to 1.011 (16K) to 0.952 (64K), indicating increasing clustering. Different log-frequency bands are color-coded, revealing that low-frequency tokens exhibit the most severe clustering in large vocabularies. This provides mechanistic evidence for the undertraining mechanism underlying the U-shaped vocabulary-loss relationship.

Prediction for Llama3 vocabulary: Appendix A.11, Figure 14 replicates the main Figure 1 analysis for Llama3 models, which use a 128K vocabulary. The paper predicts that while this is a significant improvement over Llama2's 32K, it remains suboptimal for the larger Llama3 models: the predicted optimal vocabulary for Llama3-400B is as high as 487K. This suggests that even recent models with larger vocabularies may still be underallocating at the largest scales.

Critical Assessment

Claim: "Optimal vocabulary size depends on compute budget, with larger models requiring larger vocabularies"

This is the paper's most foundational claim and is supported by converging evidence from three distinct approaches, but the strength of evidence varies across the three methods.

Approach 1 provides the most direct empirical support, deriving the relationship $N_v^{\text{opt}} \propto C^{0.42}$ from actual training runs spanning $N_{nv}$ from 33M to 1.13B and vocabulary sizes from 4K to 96K. The power-law fits (Figure 5) with reported low RMSE and high $R^2$ values substantiate the claim within the tested range. However, the predictions for models at 3B, 7B, 70B, and 300B parameters (Table 1) represent extrapolation of up to 2.5 orders of magnitude beyond the largest fitted model (1.13B → 300B). The paper does not provide confidence intervals or prediction bands for these extrapolations, and power-law relationships established at small scales do not always hold at larger scales.

Approach 2 provides theoretical triangulation but its predictions depend on the FLOPs approximation $C \approx 6ND$ , the compression function $f(V)$ , and the assumption that the derivative-optimal $V$ corresponds to the loss-optimal $V$ . The close match with Approach 1 (e.g., 212K vs. 231K for 70B parameters) is reassuring but not independent validation—both approaches share the same cost model and are fitted to data from the same model family.

Approach 3 provides the most flexible validation by fitting a loss function that can predict optimal vocabulary in non-compute-optimal regimes. The fitted exponent $\alpha_2 = 0.671$ (vocabulary) vs. $\alpha_1 = 0.447$ (non-vocabulary) independently confirms the $\gamma < 1$ relationship without assuming power-law forms for the optimal allocations—it emerges from fitting loss to all data points, not just the compute-optimal subset.

A genuine weakness: the extrapolation is untested at true LLM scale. The paper's validation experiments at $N_{nv} = 2.87\text{B}$ confirm that the framework works at roughly 3× the largest fitting scale, but this is still 20× smaller than Llama2-70B and 100× smaller than the 300B predictions. The claim that Llama2-70B should have used 216K vocabulary (7× its actual 32K) is a prediction, not a verified fact. Validating at 70B scale would be prohibitively expensive, but this doesn't change the extrapolation uncertainty—it means the paper's most headline-grabbing numbers are extrapolations, not measurements.

Claim: "Most LLMs use insufficient vocabulary sizes"

This claim is supported under the assumption that training data allocation is near-optimal (Figure 2) or under a more realistic accounting of actual training tokens (Figure 6).

Figure 2 assumes Chinchilla-optimal training: each model's non-vocabulary parameters are used to compute the implied FLOPs budget, from which optimal vocabulary is predicted. This assumption is violated for models like Llama2, which was overtrained—trained on 2T tokens when Chinchilla-optimal for 7B parameters would be ~150B. The paper is transparent about this: Section 5 notes that "considering that several LLMs are trained on substantially more data than optimal ones (e.g., Llama2), the optimal vocabulary sizes would likely be larger than currently estimated." This means Figure 2 underestimates the optimal vocabulary for overtrained models, making the conclusion of "underallocation" even stronger.

Figure 6 corrects for this by using Approach 3 with each model's actual reported training tokens. It shows the same qualitative result: most models underallocate vocabulary parameters by large margins. However, the paper does not document the source of the training token counts used for Figure 6, and the axis values (vocabulary parameters in billions) are not tabulated with numerical precision—they must be read from the chart, which limits exact verification.

A limitation: partial vocabulary parameter accounting. The paper models vocabulary parameters as $N_v = Vd$ rather than $2Vd$ (which would account for both the embedding and output layers). This simplification is acknowledged in Section 2.2 with the justification that "the main computational burden, as measured in FLOPs, is associated with the output layer, but not the word embedding layer." For the purpose of the scaling analysis, this is reasonable—the FLOPs equation $C \approx 6ND$ counts the output projection's contribution linearly in $Vd$ while the embedding lookup contributes sub-linearly. However, the predicted optimal vocabulary sizes would shift if both layers were accounted for (the paper would predict somewhat smaller vocabularies), and the comparison against actual models' vocabulary parameters may not consistently account for tied vs. untied embeddings across model families.

Claim: "Adopting the predicted optimal vocabulary size consistently improves downstream performance over commonly used vocabulary sizes"

This claim is supported by the validation experiments at 2.87B scale (Tables 2 and 3), but with important caveats about the magnitude and statistical reliability of the gains.

The compute-optimal comparison (Table 2) shows an average improvement of 0.6 percentage points (47.9 → 48.5) when moving from $V = 32\text{K}$ to $V^{\text{opt}} = 35\text{K}$ . Individual task improvements are within standard deviations for most tasks, meaning the result is directionally consistent but not statistically significant on any single task. This is expected—vocabulary size is a relatively small architectural change compared to scaling model parameters—but it means the claim of "consistently improves" rests on the pattern across tasks rather than individual statistical significance.

The overtraining comparison (Table 3) shows the strongest result: $V^{\text{opt}} = 43\text{K}$ improves ARC-Challenge from 29.1 to 32.0 (+2.9 points), with average improvement of 1.3 points (50.3 → 51.6). The ARC-Challenge gain of 2.9 points is the largest single-task improvement and is used as the headline number in the abstract. However, the standard deviations are ±1.3–1.4 for ARC-Challenge, meaning the gap is approximately 2 standard deviations—not overwhelming statistical evidence but clearly suggestive.

A missing baseline: comparison against other vocabulary sizes besides the predicted optimal. The validation experiments compare exactly two vocabulary sizes: the conventional 32K and the predicted optimal (24K, 35K, or 43K depending on the regime). The paper does not show performance at other vocabulary sizes (e.g., 24K vs. 32K vs. 43K all in the overtraining regime) to demonstrate that the predicted optimal is indeed the peak of the performance curve. Without this, the reader cannot distinguish between "the predicted optimal is better than 32K" and "larger vocabularies are monotonically better in this regime"—both would produce the observed result. Figure 7 (left) does provide such a curve for $N_{nv} = 302\text{M}$ , showing a clear optimum, but the same sweep is not provided for $N_{nv} = 2.87\text{B}$ .

A missing baseline: comparison against heuristic vocabulary choices. The paper compares against $V = 32\text{K}$ , the most common choice. But what about other heuristics—e.g., the vocabulary size that makes vocabulary parameters equal to 1% or 2% of non-vocabulary parameters, or following the scaling of an existing model family? The paper doesn't provide these baselines, so the improvement over 32K cannot be contextualized against other reasonable but non-optimal choices.

Claim: The three approaches "converge on the conclusion that the optimal vocabulary size depends on the compute budget"

The convergence is convincing for the $\gamma \approx 0.83$ exponent, with Approach 1 giving $\gamma = 0.42/0.50 = 0.84$ , Approach 2 giving $\gamma = 0.83$ via direct fitting, and Approach 3 giving $\alpha_2 = 0.671$ and $\alpha_1 = 0.447$ (whose relationship implies vocabulary scales slower). The numerical predictions for specific model sizes (Table 1) agree within ~10% across approaches (e.g., 212K vs. 231K vs. 218K for 70B parameters), which is tight given that these are different methodologies.

However, the convergence is partially by construction. Approaches 1 and 3 are fitted to the same training run data—they are different functional forms applied to the same empirical measurements. Approach 2 is more independent (relying on the FLOPs equation and compression function), but its $\gamma$ is fitted to approach-1-style optimal allocations. The three approaches are better understood as complementary perspectives rather than truly independent validations.

What experiments would have strengthened the paper

Multi-seed training for the 2.87B models. The downstream evaluation uses 3 seeds for inference but does not appear to use multiple training seeds. Training variance at 3B scale could be significant, and showing that the vocabulary effect survives training noise would strengthen the claim.
Vocabulary sweeps at 2.87B scale. Showing the full U-curve (vocabulary vs. downstream performance) at the largest model scale—analogous to the $N_{nv} = 302\text{M}$ heatmap in Figure 7—would directly validate that the predicted optimum is indeed the peak, rather than just an improvement over 32K.
Validation with a different architecture. All experiments use the Llama architecture. The paper argues that vocabulary scaling should be architecture-independent (since it's about parameter allocation, not architectural inductive bias), but this remains untested. A single experiment with a non-Llama architecture (e.g., a vanilla GPT-2-style Transformer) would test this.
Validation with a different dataset. All training uses SlimPajama. Different corpora have different token frequency distributions, which could shift the optimal vocabulary size—e.g., code-heavy datasets might benefit from larger vocabularies to capture language-specific syntax tokens. The compression function $f(V)$ would change, and the optimal vocabulary might shift accordingly.
Inference cost analysis. The paper recommends using compute-optimal vocabulary even when overtraining to control inference costs, but never quantifies the inference cost tradeoff. An experiment measuring inference latency across vocabulary sizes for a fixed model would make this recommendation more actionable.
Confidence intervals for extrapolated predictions. The 70B, 130B, and 300B predictions in Table 1 are point estimates without uncertainty quantification. Bootstrap or Bayesian methods could propagate fitting uncertainty to prediction intervals, which would help practitioners assess whether the distinction between, say, 212K and 231K vocabulary is meaningful.

Summary of what holds and what doesn't

What the experiments convincingly demonstrate:

Within the tested range (33M–1.13B $N_{nv}$ , up to 96K vocabulary), there exists a compute-dependent optimal vocabulary size that can be fitted with power laws.
The optimal vocabulary scales slower than non-vocabulary parameters ( $\gamma \approx 0.83$ ).
At 2.87B scale, adjusting vocabulary to the predicted optimum (whether smaller for undertraining or larger for overtraining) improves downstream performance compared to the conventional 32K vocabulary, with the largest gains under overtraining (+2.9 ARC-Challenge points).
The unigram-normalized loss $L_u$ is a valid metric for comparing models with different vocabulary sizes, validated by correlation with downstream performance.

What is extrapolated rather than demonstrated:

The specific vocabulary predictions for 7B–300B models (Table 1) are extrapolations of 1–2.5 orders of magnitude beyond the fitted range.
The claim that Llama2-70B "should have been at least 216K" is a prediction based on fitted power laws, not a measurement at that scale.

What is assumed rather than tested:

That the BPE tokenization algorithm is representative—predictions might shift for other tokenization methods.
That the compression function $f(V)$ fitted on SlimPajama generalizes to other corpora.
That the architectural choice of embedding dimension (Table 5 schedule) doesn't interact with vocabulary optimality in ways not captured by the parameter decomposition.

What the paper explicitly acknowledges as limitations (Appendix B):

Approach 1's predictions are "constrained by the granularity and range of the experimental data points available, which can introduce errors in the fitting process."
Approach 2 "does not allow us to independently determine the optimal allocation of non-vocabulary parameters and training data size."
Approach 3 is "constrained by the granularity and range of the experimental data points available to some extent."
All approaches are limited to the dense Transformer architecture, with extensions to non-Transformer models, multilingual scenarios, and multimodal scenarios identified as future work.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in Headline Gains

The assumption or constraint. The three prediction approaches require estimating the relationship between vocabulary size and loss—either through extensive training runs (Approaches 1 and 3) or by fitting a reference model combined with the compression function (Approach 2). For Approaches 1 and 3, this means training multiple models per non-vocabulary parameter scale at every vocabulary size of interest (4K, 6K, 8K, 10K, 16K, 24K, 32K, 48K, 64K, 96K—up to 10 vocabulary variants each) across 6 model scales, plus the 2.87B validation models. The paper does not amortize this fitting cost into the reported FLOPs savings. A practitioner who wants to determine the optimal vocabulary for their specific model scale and dataset using Approach 1 or 3 would need to replicate a substantial fraction of this experimental grid before making the vocabulary decision.

The consequence. The framing of "same FLOPs budget" in the validation experiments (Tables 2 and 3)—where the $V = 32\text{K}$ baseline and $V^{\text{opt}}$ model are compared at equal training FLOPs—is a conditional comparison that assumes the optimal vocabulary has already been identified. The total cost of identifying that optimum (the fitting experiments) is externalized. For a team training a single 70B-parameter model, the cost of running a 10× vocabulary sweep even at 1B scale to fit Approach 1 could rival or exceed the cost of the final training run itself. Approach 2 partially mitigates this by requiring only a small reference model and the compression function fit, but it cannot independently determine the scaling exponent $\gamma$ without empirical validation—the paper fits $\gamma$ to Approach 1-style data. The paper acknowledges this implicitly in Appendix B.1 when noting Approach 1 is "constrained by the granularity and range of the experimental data points available" and requires "substantial computational resources," but it does not quantify the fitting cost or propose a procedure for minimizing it.

What evidence exists in the paper. Figure 4 shows all training runs used for Approaches 1 and 3—6 model groups × up to 10 vocabulary sizes × training to convergence on up to 500B characters, representing thousands of GPU-hours. The validation experiments (Tables 2 and 3) compare only the final selected models, not the cost of selection. The paper does not provide a budget analysis that combines fitting cost with training cost.

Mitigation status. The paper partially addresses this through Approach 2 (derivative-based), which requires only the compression function $f(V)$ (fitted once from tokenizer statistics, not LLM training) and a scaling exponent $\gamma$ that could potentially be estimated from a single small reference model. However, the paper itself estimates $\gamma = 0.83$ by fitting derivative-optimal $N_v$ values across multiple $N_{nv}$ scales—which still requires knowing the optimal $V$ at those scales. A true single-reference-model procedure (train one 33M model at a few vocabulary sizes to find its optimum, then extrapolate via $\gamma = 0.83$ ) is suggested but not validated. The paper flags "exploring cheaper difficulty estimation" as future work in spirit, though not in these exact terms.

Extrapolation Across Two Orders of Magnitude Without Uncertainty Quantification

The assumption or constraint. The scaling law fits used to predict optimal vocabulary for large models (7B–300B parameters, Table 1) are derived from models with $N_{nv}$ ranging from 33M to 1.13B. The predictions for Llama2-70B, Falcon-180B, and hypothetical 300B models represent extrapolation of 1.5–2.5 orders of magnitude beyond the largest fitted model. The paper assumes that the power-law relationships $N_v \propto C^{0.42}$ (Approach 1) and $\gamma = 0.83$ (Approach 2) hold at these scales without providing confidence intervals, prediction bands, or any formal uncertainty quantification for the extrapolated values.

The consequence. The headline numbers—that Llama2-70B "should have been at least 216K" vocabulary, 7× its actual 32K—are point estimates from an extrapolated fit. If the true exponent $\alpha_2$ (vocabulary parameters vs. FLOPs) were 0.38 instead of 0.42, the predicted optimal vocabulary for a 70B-parameter model would be substantially smaller. If the power-law relationship degrades at larger scales—perhaps because the embedding dimension schedule (Table 5, Appendix A.7.2) introduces discontinuities, or because very large vocabularies encounter qualitatively different data sparsity patterns—the predictions could be far off. The distinction between 212K (Approach 1) and 231K (Approach 2) for 70B parameters is reported as "aligned," but without uncertainty estimates, a practitioner cannot distinguish between these or assess whether a 150K vocabulary would be essentially equivalent given fitting noise.

The paper's validation at 2.87B scale (roughly 2.5× the largest fitting model) shows the framework works at that extrapolation distance, but 2.87B → 70B is a further 25× extrapolation in parameter count. The gap between validated scale and predicted scale is larger than the gap between fitting scale and validated scale.

What evidence exists in the paper. Table 1 provides point estimates only. The paper reports RMSE and $R^2$ for the power-law fits (Figure 5 caption) but does not propagate these to prediction intervals. The convergence of three approaches (212K, 231K, 218K for 70B) provides some informal triangulation but is not a substitute for statistical uncertainty quantification. The figure captions mention low RMSE and high $R^2$ , but these are in-sample fit quality metrics that do not capture extrapolation uncertainty.

Mitigation status. Partially addressed by the convergence of three distinct methodologies on similar predictions. If Approach 1 (empirical power-law fitting), Approach 2 (derivative-based, no loss data), and Approach 3 (parametric loss function, different functional form) all produce $\gamma \approx 0.83$ and similar absolute predictions, it reduces the likelihood that any single method's assumptions are driving the result. However, this triangulation does not eliminate the extrapolation risk—all three methods share the underlying data (same model family, same dataset, same FLOPs equation) and could fail jointly if the scaling behavior changes at larger scales. The paper explicitly lists "larger models" as a limitation for future work in Appendix B.2.

Single Model Architecture and Single Training Dataset

The assumption or constraint. All experiments use the LLaMA architecture (pre-norm Transformer with RoPE and SwiGLU) trained on the SlimPajama dataset, a cleaned subset of RedPajama derived from CommonCrawl. The compression function $f(V)$ is fitted on SlimPajama, the training runs use SlimPajama, and the downstream evaluation uses standard English benchmarks (ARC, HellaSwag, PIQA, etc.). The paper makes no claims about how optimal vocabulary size changes with model architecture (dense vs. mixture-of-experts, Transformer vs. state-space models like Mamba, encoder-decoder vs. decoder-only) or with training data distribution (code-heavy corpora, multilingual data, domain-specific scientific text).

The consequence. The compression function $f(V) = a\log^2(V) + b\log(V) + c$ encodes the relationship between vocabulary size and tokenization efficiency for SlimPajama's token distribution. A corpus with a different character- or word-level entropy would produce a different $f(V)$ , shifting the optimal vocabulary size. For example, a code-heavy corpus with many repeated syntactic tokens (parentheses, indentation, keywords) might compress more efficiently with smaller vocabularies, while a multilingual corpus covering dozens of scripts might require larger vocabularies to avoid excessive fragmentation of non-Latin scripts. The paper's specific coefficients ( $a = 0.0064$ , $b = -0.1581$ , $c = 1.2047$ ) and the resulting predictions (Table 1) are calibrated to SlimPajama and would not transfer directly.

Similarly, the fitted power-law exponents ( $\alpha_2 = 0.42$ , $\gamma = 0.83$ ) might be architecture-dependent. The paper argues in Appendix A.6 that vocabulary parameters should scale slower than non-vocabulary parameters because they "can only grow in width" while non-vocabulary parameters benefit from both depth and width increases. If an architecture uses a different embedding structure—for example, factorized embeddings that decouple vocabulary size from embedding dimension, or Mixture-of-Experts models where vocabulary parameters are a smaller fraction of total parameters—the optimal scaling exponent might shift.

What evidence exists in the paper. Appendix A.9 validates the compression function $f(V)$ across BPE, unigram, and word-based tokenizers, showing high $R^2$ for all three. This addresses the tokenizer algorithm dimension but not the data distribution dimension—all three tokenizer types were trained on the same SlimPajama data. The paper does not fit $f(V)$ on a different dataset (e.g., code, scientific text, multilingual) to test sensitivity. Appendix B.4 identifies multilingual and multimodal extensions as future work, acknowledging the gap.

Mitigation status. The paper provides the compression function fitting methodology so that practitioners can refit $f(V)$ on their own data. However, the scaling law parameters ( $\alpha_1$ , $\alpha_2$ , $\gamma$ , and the loss function coefficients in Approach 3) would require re-estimation through new training runs, which is the expensive part. The paper does not propose a transfer learning approach (e.g., fitting scaling laws on SlimPajama and applying a correction factor for a new domain) or provide evidence that the exponents are dataset-independent. The recommendation is effectively: replicate our experimental grid on your data and architecture.

Vocabulary Parameters Are Modeled as $Vd$ Rather Than $2Vd$ , and the Embedding Dimension Schedule Is Externally Imposed

The assumption or constraint. The paper defines vocabulary parameters as $N_v = Vd$ (Section 2.2), capturing only one of the two vocabulary-dependent parameter matrices (the output projection layer). In standard Transformer implementations, vocabulary parameters exist in two places: the input embedding matrix ( $V \times d$ ) and the output projection matrix ( $d \times V$ ). The paper justifies this simplification by noting that "the main computational burden, as measured in FLOPs, is associated with the output layer, but not the word embedding layer"—the embedding lookup is a sparse indexing operation that contributes negligibly to FLOPs, while the output projection is a dense matrix multiply over all $V$ tokens for every sequence position. However, for parameter counting and the scaling law fits, $N_v = Vd$ undercounts vocabulary parameters by approximately a factor of 2 (for models without weight tying).

Additionally, the embedding dimension $d$ is not treated as a free variable. It is determined by $N_{nv}$ through a fixed schedule (Appendix A.7.2, Table 5), following Kaplan et al.'s observation that depth-to-width ratio has minimal effect on performance for fixed total parameters. This schedule determines what vocabulary size $V$ corresponds to a given $N_v$ (since $V = N_v / d$ ), making the vocabulary size predictions sensitive to the chosen schedule. If a practitioner uses a different depth-to-width ratio for their target model, the mapping from predicted $N_v$ to recommended $V$ would shift.

The consequence. Using $N_v = Vd$ rather than $N_v = 2Vd$ means the paper's predicted optimal vocabulary parameter counts ( $N_v$ ) are roughly half of what they would be under full accounting, and the predicted optimal vocabulary sizes ( $V$ ) are calibrated to a convention that may not match how practitioners count vocabulary parameters. When the paper compares its predictions against actual models (Figures 2, 6), it computes those models' vocabulary parameters as $V \times d$ (presumably). But if a model uses untied embeddings (separate input and output matrices, each $V \times d$ ), the paper's $N_v$ undercounts the model's actual vocabulary parameter allocation by 2×. If a model uses tied embeddings (shared input and output matrix), the paper's accounting is approximately correct. The paper does not specify how it handled tied vs. untied embeddings when computing the "Current" vocabulary parameters for models in Figures 2 and 6, creating ambiguity in the comparison.

The embedding dimension schedule introduces a separate concern. For $N_{nv} = 70\text{B}$ , the paper assumes $d = 8192$ (Table 5), yielding $V = 1.7\text{B} / 8192 \approx 212\text{K}$ . If a practitioner built a 70B-parameter model with $d = 12288$ (wider, shallower), the same $N_v = 1.7\text{B}$ would correspond to $V \approx 138\text{K}$ . Neither is clearly correct—the optimal $V$ might depend on $d$ in ways not captured by the $N_v = Vd$ parameterization alone. The paper's parameterization assumes vocabulary optimality depends only on the product $Vd$ , not on $V$ and $d$ independently. This is an untested assumption.

What evidence exists in the paper. The paper acknowledges the $Vd$ vs. $2Vd$ simplification in Section 2.2 with a footnote: "Vocabulary parameters typically encompass both the word embedding layer and the output layer. In this paper, for clarity and analytical simplicity, we employ $Vd$ rather than $2Vd$ to represent the vocabulary parameters. This choice is predicated on empirical observations: the main computational burden, as measured in FLOPs, is associated with the output layer, but not the word embedding layer." The embedding dimension schedule is provided in Appendix Table 5, with the observation that it follows Kaplan et al.'s finding about depth-width insensitivity.

Mitigation status. Not addressed beyond the footnote disclosure. The paper does not provide an ablation testing whether $Vd$ vs. $2Vd$ parameterization affects the fitted exponents, nor does it test sensitivity to the embedding dimension schedule. For a practitioner using the predictions, the practical recommendation would be: use the predicted $N_v$ and divide by your target model's actual embedding dimension to get $V$ , then double-check whether your model uses tied or untied embeddings to interpret $N_v$ correctly for parameter budgeting.

The Optimal Vocabulary Depends on Training Data Volume, but the Recommendation Defaults to Compute-Optimal

The assumption or constraint. Approach 3 demonstrates that the locally optimal vocabulary size shifts substantially depending on the training data volume, even for fixed non-vocabulary parameters. For $N_{nv} = 2.87\text{B}$ , the predicted optimal $V$ is 24K under undertraining, 35K under compute-optimal training, and 43K under overtraining—a range spanning nearly 2×. Despite this, the paper recommends "using the optimal vocabulary size corresponding to a given $N_{nv}$ , assuming optimal allocation of training data, even in scenarios where overtraining may occur" (Section 5). The justification is that "expanding the vocabulary size also increases the computational demands during inference," since the output projection must compute logits over all $V$ tokens for every generated token.

The consequence. This recommendation creates a tension the paper does not resolve quantitatively. Under overtraining (the most common regime for modern LLMs, since it produces smaller models that are cheaper to serve), Approach 3 predicts that an even larger vocabulary than the compute-optimal size would improve training-time loss and downstream performance. The paper validates this empirically: Table 3 shows $V = 43\text{K}$ outperforms $V = 32\text{K}$ under overtraining, and $V = 35\text{K}$ (the compute-optimal prediction) is not tested in the overtraining regime. A practitioner training in the overtrained regime faces a genuine tradeoff: use 43K vocabulary for best training-time performance (at the cost of ~12% more vocabulary parameters and a correspondingly larger output softmax at inference) or use 35K as the paper recommends (accepting lower training-time performance in exchange for inference efficiency). The paper provides no quantitative analysis of this tradeoff—no measurements of inference latency vs. vocabulary size, no estimates of total cost of ownership that would amortize inference savings over the model's deployment lifetime, and no framework for deciding when the inference cost penalty outweighs the training-time accuracy gain.

What evidence exists in the paper. Table 3 provides the training-time performance comparison: $V^{\text{opt}} = 43\text{K}$ achieves 51.6 average downstream accuracy vs. 50.3 for $V = 32\text{K}$ under overtraining, a 1.3-point improvement. Figure 7 (left) shows the optimal vocabulary shifting from 10K to 24K as data increases for $N_{nv} = 302\text{M}$ . The paper acknowledges the inference cost concern in Section 5 with exactly one sentence. No inference latency experiments are reported.

Mitigation status. The paper does not address this tradeoff quantitatively. The recommendation to use compute-optimal vocabulary even under overtraining is stated as a pragmatic guideline without supporting cost-benefit analysis. This is a significant gap for practitioners, since the inference cost of the output projection is often the latency bottleneck in autoregressive generation—the final linear layer mapping from hidden dimension $d$ to vocabulary size $V$ requires $O(Vd)$ multiply-adds per generated token, and for large $V$ (200K+) this can dominate attention and feed-forward computation. A rigorous treatment would require measuring or modeling inference throughput at different vocabulary sizes and determining the break-even point where the inference penalty outweighs the training-time accuracy gain. The paper identifies this as outside its scope but makes a recommendation nonetheless.

No Validation at True LLM Scale; the Largest Tested Model Is 3B Parameters

The assumption or constraint. The paper's empirical validation of its vocabulary predictions uses models with $N_{nv} = 2.87\text{B}$ (Tables 2 and 3), while its most impactful predictions target models at 7B, 13B, 30B, 70B, 130B, and 300B parameters (Table 1). All three prediction approaches are fitted on models ranging from 33M to 1.13B parameters—roughly 3× smaller than the validation models and 25–250× smaller than the headline prediction targets. The paper does not train or evaluate any model larger than 3B parameters with its predicted optimal vocabulary.

The consequence. The paper's central claim—that Llama2-70B should have used at least 216K vocabulary, 7× its actual 32K—rests entirely on extrapolated power-law fits. The validation at 3B parameters shows that the framework works at 3× the fitting scale, but this is a relatively mild extrapolation compared to the 25× gap to 70B parameters. There are several reasons the scaling relationship might break or shift at larger scales:

Data sparsity for rare tokens: At vocabulary sizes of 200K+, the tail of the vocabulary consists of tokens that appear extremely rarely—possibly only a handful of times in the entire training corpus. The paper's own SVD analysis (Figure 9) shows embedding degeneration for rare tokens at 64K vocabulary at 85M scale. At 70B scale with 216K vocabulary, whether the massively increased training data (trillions of tokens rather than billions) provides sufficient gradient updates for these ultra-rare token embeddings is unknown.
Discrete jumps in embedding dimension: The embedding dimension schedule (Table 5) places $d = 8192$ at $N_{nv} = 70\text{B}$ and $d = 16384$ at $N_{nv} = 300\text{B}$ . These are large, discrete changes—an 8K to 16K doubling—that could interact with vocabulary optimality in ways not captured by the smooth power-law fits derived from smaller models (where $d$ ranges from 512 to 3200).
Computational bottlenecks shifting: At very large vocabulary sizes, the output projection ( $d \times V$ ) can become a dominant fraction of total parameters and FLOPs. The paper's FLOPs equation $C \approx 6ND$ treats all parameters as equally contributing to FLOPs, but at extreme vocabulary sizes, the constant factors and memory access patterns might change the effective cost in ways the simple model doesn't capture.

What evidence exists in the paper. Tables 2 and 3 show validation at $N_{nv} = 2.87\text{B}$ with vocabulary sizes of 24K, 32K, 35K, and 43K—all within or near the fitted range. The paper does not validate at 7B parameters (the next step up) with a vocabulary of, say, 60K vs. the conventional 32K. The 3B validation demonstrates internal consistency of the framework at a modest extrapolation distance but cannot confirm that the power-law relationships extrapolate accurately over the much larger gap to 70B+ parameters.

Mitigation status. The paper identifies "larger models" as future work (Appendix B.2): "Exploring to what extent our findings hold in even larger models and with different architectures is a promising direction for future work." This is an honest acknowledgment but does not reduce the uncertainty of the current predictions. The convergence of three methods on similar predictions provides informal triangulation but is not a substitute for validation at scale. A partial mitigation would be to validate at an intermediate scale (e.g., 7B or 13B parameters) to test whether the extrapolation holds at one more step before jumping to 70B+, but the paper does not do this.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper effects a conceptual reframing rather than a paradigm shift: it elevates vocabulary size from a fixed, arbitrary design constant to a third scaling dimension that should be jointly optimized alongside model parameters and training data. Prior to this work, the scaling laws community—from Kaplan et al. (2020) through Hoffmann et al. (2022)—operated under what we can now recognize as an implicit assumption that vocabulary size is an architectural prerequisite (like activation function or normalization scheme) rather than a resource allocation variable (like depth or width). The paper's demonstration that optimal vocabulary size follows a predictable power law ( $N_v^{\text{opt}} \propto N_{nv}^{0.83}$ ) and that getting it wrong by 3–7× (as most existing LLMs do) measurably degrades downstream performance means that future scaling law analyses cannot defensibly ignore vocabulary.

This reframing has several concrete consequences for how the field operates:

Scaling laws become three-dimensional. The classical two-way split—"how should I divide my compute budget between model size and data volume?"—becomes a three-way split that adds "and vocabulary size." This is not an incremental parameter adjustment but a structural change to the optimization problem. The paper shows that vocabulary parameters scale with their own exponent ( $\alpha_2 = 0.42$ ) distinct from the non-vocabulary parameter exponent ( $\alpha_1 = 0.50$ ), meaning that vocabulary cannot be absorbed into a lumped "total parameters" variable—it genuinely occupies a different scaling regime. Future work on compute-optimal training recipes must now consider triples ( $N_{nv}$ , $N_v$ , $D$ ) rather than pairs ( $N$ , $D$ ), increasing the dimensionality of the design space that scaling law experiments must cover.

Vocabulary design becomes predictive rather than empirical. The paper provides a framework where vocabulary size is not discovered through trial-and-error sweeps but predicted from the target model scale and training budget. This shifts the vocabulary decision from the "hyperparameter tuning" phase (which occurs during a training run) to the "resource allocation" phase (which occurs before training begins). For organizations planning million-dollar training runs, this is practically significant: the vocabulary size determines the embedding and output layer dimensions, which affect distributed training strategies, memory allocation, and inference serving architecture. Getting vocabulary right from the start—using Approach 3's predictions for the specific training regime—avoids the costly discovery mid-training that the vocabulary is suboptimal.

The field's implicit vocabulary choices are retrospectively diagnosed. The paper's framework provides a lens for understanding why the community's vocabulary choices have been so heterogeneous (32K for Llama2, 256K for Gemma, 128K for Llama3) and why they've been drifting upward. Under compute-optimal training, a 7B-parameter model should use ~60K vocabulary (Table 1)—roughly 2× the Llama2-7B choice. Under overtraining (the actual Llama2-7B regime, with 2T tokens), the predicted optimal rises to ~166K (Figure 6). Llama3-8B's shift to 128K is thus partially vindicated by the framework but still falls short of the predicted optimum. Gemma2-9B's 256K vocabulary, which the paper identifies as closest to optimal among current models, provides an existence proof that large vocabularies are trainable and beneficial. The framework converts scattered industry intuitions ("bigger vocabulary seems to help") into quantitative predictions with specific numbers attached, enabling the community to converge on principled choices rather than copying the most recent successful model's vocabulary by default.

The finding reconciles conflicting results about vocabulary size in prior literature. Earlier work on multilingual models (Zheng et al., 2021; Liang et al., 2023) demonstrated that larger vocabularies improve cross-lingual transfer—a finding that makes sense under this paper's framework since multilingual training data effectively increases the training budget (more languages → more tokens → the model can support a larger vocabulary without undertraining). Conversely, prior work showing that vocabulary expansion during continual pretraining degrades low-resource language performance (Dou et al., 2024) makes sense since continual pretraining typically uses modest data volumes, shifting the optimal vocabulary downward rather than upward—adding vocabulary capacity without sufficient data to train it causes the embedding degeneration documented in Figure 9. The paper's framework provides a unified explanation: there is no universally "good" vocabulary size; the optimal size depends on the effective training budget in the relevant regime.

Verifier-based search and over-optimization dynamics find a parallel in vocabulary scaling. The paper's finding that the optimal vocabulary is bounded by the compute budget—too small wastes tokenization efficiency, too large causes embedding undertraining—is structurally analogous to the over-optimization phenomena documented in the test-time compute literature. Just as aggressive beam search eventually finds solutions that score highly under the verifier but are actually incorrect (the verifier over-optimization problem), aggressive vocabulary expansion eventually produces embeddings that occupy the parameter budget but fail to learn useful representations (the undertraining problem). Both are manifestations of the same principle: optimization against an imperfect signal (verifier score, loss on limited data) eventually exploits that signal's weaknesses. The vocabulary framework provides a quantitative language for this phenomenon—the optimal $V$ is where $\partial C / \partial V = 0$ , the point where the marginal benefit of better tokenization equals the marginal cost of additional parameters—which could inspire analogous quantitative treatments of verifier over-optimization.

The compression function $f(V)$ enables a new class of tokenization analyses. The paper's fitted quadratic-in-log function mapping vocabulary size to tokens-per-character is both simple (three parameters, fitted once from tokenizer statistics) and remarkably robust across tokenizer types (BPE, unigram, word-based, all with $R^2 > 0.98$ ). This function is the conceptual bridge that connects vocabulary choice to FLOPs accounting, and its existence means that any future work on tokenization can now reason quantitatively about the compute implications of vocabulary decisions. Previously, the relationship between vocabulary size and training efficiency was understood qualitatively ("larger vocabulary means fewer tokens but more parameters"), but $f(V)$ makes it algebraic, enabling the derivative-based optimization of Approach 2 and the parametric loss modeling of Approach 3.

Research directions that become more attractive. The paper makes the case that investing in vocabulary optimization is high-leverage: for 3B-parameter models under overtraining, simply adjusting vocabulary from 32K to the predicted optimal 43K yields a 2.9-point ARC-Challenge improvement with zero additional FLOPs (Table 3). This is essentially free performance—it changes only the tokenizer, the embedding layer dimensions, and the output projection dimensions, all of which are architectural choices made before training begins. Compared to other "free lunch" improvements (better data filtering, learning rate schedules, architectural tweaks), vocabulary optimization appears to have an unusually high return on research investment, especially since most existing models are far from optimal. This suggests that vocabulary-aware scaling should become a standard component of the pretraining planning process, analogous to how the Chinchilla laws are now standard for determining the parameters-vs-data split.

Research directions that become less attractive. The paper's finding that byte-level language models (vocabulary size fixed at 256) have generally not scaled beyond ~1B parameters (Appendix A.12) now has a quantitative explanation: the vocabulary exponent $\gamma \approx 0.83$ means that larger models need larger vocabularies to achieve optimal performance, and a fixed 256-token vocabulary becomes increasingly suboptimal as model scale increases. This suggests that research on scaling byte-level models to LLM scale would need to overcome a fundamental vocabulary bottleneck—the model would be severely vocabulary-constrained relative to what the scaling laws predict is optimal—and would likely require architectural innovations (hierarchical tokenization, multi-scale processing) beyond simply scaling up the byte-level architecture. The paper does not prove this is impossible, but it provides a quantitative argument for why it has been difficult.

Follow-Up Research This Work Enables

Validation of the extrapolated predictions at 7B–13B parameter scale. The paper's most impactful predictions—that Llama2-70B should have used ~216K vocabulary, that Llama3-400B should use ~487K—are extrapolated 25–250× beyond the largest fitted model (1.13B parameters). The validation at 2.87B (Table 2 and 3) shows the framework works at ~3× extrapolation, but the gap to 70B+ remains enormous. A single experiment at 7B parameters with $N_{nv} = 7\text{B}$ and $d = 4096$ , comparing $V = 32\text{K}$ (conventional), $V = 60\text{K}$ (the Approach 3 compute-optimal prediction from Table 1), and $V = 128\text{K}$ (the Llama3 choice, which is between the compute-optimal and overtrained predictions) under Chinchilla-optimal training (~150B tokens) and under overtraining (~1T tokens) would provide a critical intermediate validation point. If the 60K vocabulary outperforms 32K at 7B scale, it strengthens confidence in the extrapolation to 70B. If 128K outperforms 60K under overtraining, it validates the Approach 3 prediction that overtraining shifts the optimum upward. This experiment would require approximately 5× the compute of the paper's largest run (7B vs. 2.87B, plus 2–3 vocabulary variants), making it feasible for an academic lab with access to a moderate GPU cluster.

Training a PRM or verifier model on top of vocabulary-optimized representations. The paper demonstrates that larger vocabularies improve downstream task performance (Table 3), but it does not investigate whether the quality of internal representations improves in ways that would matter for downstream fine-tuning, especially for tasks requiring nuanced semantic understanding. A concrete follow-up would be: train two 3B-parameter models—one with $V = 32\text{K}$ and one with $V = 43\text{K}$ —under the overtraining regime, then use each as the base model for training a process reward model (PRM) on mathematical reasoning (following the Monte Carlo rollout procedure from Lightman et al., 2023), and measure whether the vocabulary-optimized model produces better step-level correctness predictions. The hypothesis is that larger vocabularies enable more precise tokenization of mathematical notation (symbols, numbers, operators may get dedicated tokens rather than being fragmented), which could improve the PRM's ability to identify reasoning errors at fine granularity. This experiment would connect the vocabulary scaling work to the rapidly growing literature on verifier-guided reasoning, and would test whether vocabulary optimization provides benefits beyond the perplexity and benchmark accuracy improvements already demonstrated.

Measuring the inference cost of large vocabularies and determining the training-inference Pareto frontier. The paper's recommendation to use compute-optimal vocabulary size even when overtraining is motivated by inference cost concerns, but provides no quantitative analysis of this tradeoff. A concrete follow-up would: train models with fixed $N_{nv} = 3\text{B}$ and vocabulary sizes spanning 16K to 128K, measure both (a) downstream accuracy on the 7-task benchmark suite and (b) inference throughput (tokens/second) on standard hardware (e.g., A100-80GB, H100-80GB) for batch sizes of 1, 8, and 32, then construct a Pareto frontier plotting accuracy vs. inference latency. This would answer: at what vocabulary size does the inference cost of the output projection ( $O(Vd)$ per generated token) begin to dominate the total latency, and how much accuracy does a practitioner sacrifice by choosing the inference-efficient vocabulary (closer to the compute-optimal prediction) over the training-optimal vocabulary (closer to the overtraining prediction)? The experiment could also profile the fraction of total FLOPs spent in the output projection vs. attention and feed-forward layers as $V$ increases, providing hardware-specific guidance for vocabulary selection that the current paper cannot offer. A strong follow-up would include both dense models and models with techniques like grouped-query attention (where the output projection fraction of total compute is larger because attention is cheaper), since vocabulary cost interacts with the overall compute distribution.

Fitting the vocabulary scaling laws on a code-heavy or multilingual corpus to test domain dependence. All the paper's experiments use SlimPajama, an English-heavy web-text corpus. The compression function $f(V)$ and the scaling law exponents ( $\alpha_1$ , $\alpha_2$ , $\gamma$ ) are therefore calibrated to a specific token distribution. A concrete stress-test: replicate the Approach 1 IsoFLOPs experiments (6 model groups, vocabulary sweeps from 4K–96K) on (a) a code-only corpus (e.g., The Stack) and (b) a balanced multilingual corpus (e.g., a sample from mC4 covering 10+ languages with different scripts). The key measurements would be: (1) how the compression function $f(V)$ differs across domains (code might show steeper gains from larger vocabularies due to language-specific keywords; multilingual text might show a different saturation point due to script diversity); (2) whether the scaling exponent $\gamma \approx 0.83$ is domain-invariant or domain-dependent. If $\gamma$ shifts substantially—for example, if code requires $\gamma \approx 0.9$ (vocabulary scales nearly as fast as non-vocabulary parameters) because code tokens are more diverse and benefit more from dedicated vocabulary entries—then the paper's headline recommendation ("use our predicted vocabulary for your model scale") would need to be qualified by domain. This experiment would also test whether Approach 3's parametric loss function generalizes across domains with re-fitted coefficients, or whether the functional form itself needs to change (e.g., adding an interaction term between vocabulary size and domain).

Exploring whether vocabulary optimality interacts with the embedding dimension independently of the $Vd$ product. The paper models vocabulary parameters as $N_v = Vd$ , effectively assuming that the optimal vocabulary depends only on the product of vocabulary size and embedding dimension, not on $V$ and $d$ separately. This is an untested assumption with practical implications: for a given $N_v$ budget, should a practitioner prefer a larger vocabulary with a smaller embedding dimension (e.g., $V = 128\text{K}$ with $d = 1024$ ) or a smaller vocabulary with a larger embedding dimension (e.g., $V = 32\text{K}$ with $d = 4096$ )? Both configurations use $N_v = Vd \approx 131\text{M}$ vocabulary parameters, but they make very different tradeoffs in tokenization granularity vs. embedding expressiveness. A concrete experiment: for fixed $N_{nv}$ (e.g., 300M) and fixed $N_v$ (e.g., 50M), train models at several $(V, d)$ pairs that satisfy $Vd \approx 50\text{M}$ —such as $(32\text{K}, 1562)$ , $(64\text{K}, 781)$ , and $(128\text{K}, 390)$ —and measure downstream performance. If all pairs achieve similar performance, the paper's $N_v = Vd$ parameterization is validated. If one configuration (likely the intermediate one) outperforms, then vocabulary optimality depends on $V$ and $d$ independently, and the paper's predictions would need to be augmented with a recommended ratio of $V$ to $d$ . This connects to the broader question of whether very high-dimensional embeddings ( $d > 4096$ ) for moderate vocabulary sizes provide benefits that the current framework cannot capture.

Developing a difficulty-estimation-free method for determining the optimal vocabulary pre-training. The paper's three approaches all require some form of empirical measurement before predicting the optimal vocabulary: Approach 1 needs extensive training runs, Approach 2 needs a reference model and the compression function, and Approach 3 needs the full loss function fit. A practical deployment of this work would benefit from a method that determines the optimal vocabulary without any model training, using only corpus statistics and the target model specifications. A concrete direction: using the compression function $f(V)$ (fitted on the target corpus) and the FLOPs equation $C \approx 6(N_{nv} + Vd)H f(V)$ , one could estimate the total number of gradient updates that each vocabulary entry would receive under a given training budget, then use a statistical criterion (e.g., each token's embedding should receive at least $k$ gradient updates, where $k$ is a threshold derived from the embedding dimension and optimizer dynamics) to determine the maximum vocabulary size such that even the rarest token meets this threshold. This would replace the empirical scaling law fitting with a theoretically-motivated threshold derived from optimization theory, potentially making vocabulary optimization a zero-cost pre-training design decision. The paper's SVD analysis (Figure 9) showing embedding degeneration at 64K vocabulary for 85M-parameter models provides a starting point for calibrating such a threshold.

Practical Applications and Downstream Use Cases

LLM pretraining planning for organizations with fixed compute budgets. For any team planning to train a large language model from scratch, the paper provides a method to determine vocabulary size before training begins rather than discovering it through expensive ablation. Concretely: given a target non-vocabulary parameter count $N_{nv}$ , a training data budget in characters $H$ , and the embedding dimension $d$ (determined by the depth-width schedule), the team can use Approach 3 to compute the locally optimal $V$ for their specific training regime—whether they are compute-optimally training (following Chinchilla), overtrained (following Llama2-style recipes with abundant data), or data-constrained. For a concrete use case: a team planning to train a 7B-parameter model on 2T tokens (overtrained regime) with Llama architecture and SlimPajama-like data can use Approach 3 to predict that $V^{\text{opt}} \approx 166\text{K}$ (Figure 6), nearly 3× the Llama2-7B vocabulary and 1.3× the Llama3-8B vocabulary. They would then train their BPE tokenizer with this vocabulary size, design their embedding and output layers accordingly, and proceed with training. The paper's validation shows that models using the predicted vocabulary outperform those using the conventional 32K under matched FLOPs (Table 3), so this decision directly translates to improved benchmark performance—potentially worth 1–3 points on ARC-Challenge at 3B scale, and possibly more at 7B+ scale if the extrapolation holds. The Approach 2 derivative-based method provides a cheaper alternative if the team cannot afford the Approach 3 fitting experiments: train one small model (~33M parameters) at a few vocabulary sizes to find its optimum, then extrapolate to 7B scale using $\gamma = 0.83$ , requiring only the compression function $f(V)$ fitted on their target corpus.

Vocabulary adjustment for continual pretraining and domain adaptation. When a pretrained LLM is further trained on a new domain (e.g., adapting a general English model to code, medicine, or a new language), practitioners often expand the vocabulary to include domain-specific tokens. The paper's framework provides a principled method for determining how much to expand: the optimal vocabulary size for the continual pretraining phase depends on the amount of new data available. If only a small amount of domain-specific data is available (the undertraining regime), the vocabulary should be expanded conservatively—adding too many new tokens with insufficient training data will cause their embeddings to degenerate (as shown in Figure 9). If abundant domain data is available (the overtraining regime), a larger expansion is justified. Concretely, a team adapting Llama2-7B ( $V = 32\text{K}$ ) to a specialized scientific domain with 50B tokens of new data (modest volume) should use Figure 7's logic to determine that the optimal vocabulary is closer to the compute-optimal size (~60K for 7B parameters) rather than the overtrained optimal (~166K), meaning they should add only about 28K new tokens rather than attempting a full expansion to Llama3-sized vocabulary. This prevents the performance degradation documented by Dou et al. (2024) for vocabulary expansion under data-constrained continual pretraining.

Tokenization quality assessment for existing models via the compression function. The compression function $f(V)$ provides a quantitative tool for evaluating whether an existing model's vocabulary size is appropriate for its intended domain. A practitioner deploying an LLM on a specialized text corpus (e.g., legal documents, medical records) can train a BPE tokenizer at various vocabulary sizes on their target domain, fit $f(V)$ , and compare the tokens-per-character ratio at their model's actual vocabulary size against what a larger vocabulary would achieve. If the derivative $\partial f / \partial V$ at their current $V$ is still substantially negative (meaning significant compression gains remain from larger vocabularies), and if their deployment scenario can tolerate the increased inference cost of a larger vocabulary, they have a quantitative justification for vocabulary expansion. Conversely, if $\partial f / \partial V$ is near zero (compression has saturated), expanding vocabulary would provide negligible tokenization efficiency gains and would only increase inference latency. This analysis requires only tokenizer training on the target corpus—no LLM training—making it a cheap pre-deployment diagnostic. The paper's validation of $f(V)$ across BPE, unigram, and word-based tokenizers (Appendix A.9, Figure 12) ensures the method works regardless of the tokenization algorithm used by the deployed model.

Hardware-aware vocabulary sizing for edge deployment. When deploying LLMs on edge devices or consumer hardware (laptops, phones), inference latency and memory bandwidth are the binding constraints, and the output projection over $V$ tokens is often the latency bottleneck for autoregressive generation (since it requires $O(Vd)$ multiply-adds per generated token). The paper's framework, combined with an inference profiling experiment (as proposed in the follow-up section), would enable hardware-aware vocabulary selection: given a target model scale and a latency budget, use the Pareto frontier of accuracy vs. vocabulary size to select the largest vocabulary that stays within the latency constraint. For example, a 3B-parameter model deployed on a phone with a 50ms per-token latency budget might find that $V = 32\text{K}$ is the maximum vocabulary that meets the constraint, even though $V = 43\text{K}$ would improve accuracy—and the paper's framework quantifies the accuracy penalty (roughly 1.3 points of average downstream accuracy, from Table 3's overtraining comparison) for accepting the inference-motivated constraint. This makes the vocabulary tradeoff explicit and data-driven rather than based on ad hoc rules of thumb.