Evaluating Morphological Alignment of Tokenizers in 70 Languages

🎯 Pitch

Despite widespread belief that tokenizers should align with morpheme boundaries for better model performance, this study finds morphological alignment actually explains almost no variance in downstream task scores across 70 languages and five model families—and even shows a slightly negative correlation. Expanding an existing metric to a broader typological sample, the authors test multiple alignment variants, yet none meaningfully predict model quality, challenging the core assumption that linguistically informed tokenization is a reliable intrinsic evaluation tool.

1. Executive Summary

This paper expands and evaluates MorphScore—a metric that measures how often a tokenizer's segmentation boundaries align with linguistic morpheme boundaries—from the original 22 languages to 70 languages, adding flexibility via parameters for frequency weighting and single-token word inclusion. Using pre-trained tokenizers from Llama2, BLOOM, XGLM, Llama3, and Gemma3 and correlating their morphological alignment scores against downstream task performance across seven benchmarks, the authors test whether morphological alignment (the degree to which token boundaries correspond to morpheme boundaries, e.g., segmenting "books" as [book + s] rather than [boo + ks]) predicts model quality. Despite testing multiple scoring variants, morphological alignment—whether measured by boundary-level or subword-level precision and recall—explains only a small fraction of variance in model performance (recall $R^2 = 0.024$ , precision $R^2 = 0.005$ ), and the correlation is negative, establishing that morphological alignment alone does not capture dimensions of tokenizer quality relevant to downstream task performance, even after controlling for model size and training data proportions.

2. Context and Motivation

The Core Problem: We Lack Reliable Ways to Evaluate Tokenizers Without Training Models

The fundamental question this paper tackles is deceptively simple: how do you know if a tokenizer is good before you use it to train a language model? Tokenization—the process of segmenting raw text into discrete units (tokens) from a fixed vocabulary—is the first and arguably most consequential preprocessing step in modern language modeling. Every byte of text that a model ever sees passes through the tokenizer. Yet, as the authors point out in Section 1, the field lacks principled, intrinsic evaluation methods for tokenizers. Intrinsic evaluations measure properties of the tokenizer itself (e.g., compression ratio, vocabulary efficiency) without requiring the expensive step of training a model and measuring its downstream performance. Extrinsic evaluations, by contrast, measure how tokenizer choices affect model training speed, inference cost, or task accuracy—but these require actually training or running models, which is computationally prohibitive for systematic comparison.

This gap matters enormously for both research and practice:

Tokenizer selection before training. If reliable intrinsic metrics existed, practitioners could evaluate candidate tokenizers (different algorithms, vocabulary sizes, preprocessing rules) and select the best one before committing to a multi-million-dollar training run. The computational and financial savings would be substantial—the authors explicitly note that "finding reliable intrinsic tokenizer evaluation would be enormously valuable, as it would enable tokenizer selection before model training, leading to significant computational and financial savings" (Section 1).
Multilingual fairness. Tokenization has been shown to introduce systematic unfairness across languages (Petrov et al., 2023; Ahia et al., 2023). Languages with non-Latin scripts, complex morphology, or limited representation in training data often require more tokens to encode the same semantic content, increasing inference cost and latency while potentially degrading performance. A good intrinsic metric would help diagnose and correct these disparities.
Understanding what makes tokenization "good." Beyond practical utility, there is a scientific question: what properties of a tokenizer actually matter for downstream model behavior? Is it compression (how few tokens are needed)? Is it morphological transparency (do tokens correspond to meaningful linguistic units)? Is it frequency calibration (are common words represented efficiently)? The field has hypotheses but limited systematic evidence.

Conflicting Evidence on Morphological Alignment

One prominent hypothesis—with substantial supporting literature—is that morphologically aligned tokenization benefits model performance. The intuition is straightforward: if a tokenizer segments words at linguistically meaningful boundaries (e.g., "books" → [book, s] rather than [boo, ks]), the resulting tokens carry interpretable semantic and grammatical information. A model learning from these tokens can more easily generalize morphological patterns—for example, recognizing that the suffix "-s" marks plurality across thousands of English nouns—rather than having to learn these patterns from fragmented subword units that span morpheme boundaries inconsistently.

The paper cites an extensive body of work arguing for this position (Section 1): Park et al. (2020), Vasiu & Potolea (2020), Bostrom & Durrett (2020), Hofmann et al. (2021), Nzeyimana & Niyongabo Rubungo (2022), Erkaya (2022), Toraman et al. (2023), Držík & Forgac (2024), Libovický & Helcl (2024), Jabbar (2024), Uzan et al. (2024), Bauwens & Delobelle (2024), and Asgari et al. (2025). Batsuren et al. (2024) developed a classification scheme for tokenization—categorizing word segmentations as morphemic, non-morphemic, or whole-word—and found that morphemic tokenization correlated with better performance. This is a substantial body of evidence spanning multiple languages, model architectures, and task types.

However, there is an equally substantial body of contradictory evidence. The original MorphScore paper by Arnett & Bergen (2025), covering 22 languages, found that morphological alignment was not predictive of model performance. Arnett et al. (2024) found that morphemic tokenization had only a small effect on subject-verb agreement in Spanish. Macháček et al. (2018), Saleva & Lignos (2021), and Choo & Kim (2023) each found that morphologically aligned tokenization did not benefit model performance in their respective settings.

This is not a minor empirical disagreement—it is a fundamental contradiction about what makes tokenization effective. If morphological alignment helps, then tokenizer design should prioritize linguistic structure. If it does not, then other properties (compression, frequency calibration, vocabulary efficiency) may be more important, and the extensive literature advocating for morphology-aware tokenizers may be optimizing the wrong objective.

Why the Evidence Is Mixed: Methodological Heterogeneity

The paper identifies specific reasons why prior work reaches contradictory conclusions (Section 1):

Limited and inconsistent language coverage. The original MorphScore covered 22 languages—diverse by some standards but missing many high-resource languages commonly studied in language model research, such as French and German. More critically, different studies test on different language samples. Morphological alignment may matter more for morphologically rich languages (e.g., Turkish, Finnish, Hungarian) and less for analytic languages (e.g., English, Chinese). If one study predominantly tests on agglutinative languages and another on fusional languages, their conclusions may appear contradictory even if both are correct for their respective samples.

Varied model architectures and supervision regimes. Prior work spans encoder-only models (BERT), decoder-only models (GPT-style), and encoder-decoder models (T5-style), trained with objectives ranging from masked language modeling to autoregressive generation. The amount of supervision also varies dramatically—from zero-shot evaluation to full supervised fine-tuning. Morphological alignment might interact differently with different architectures and training paradigms, making cross-study comparisons unreliable.

Different evaluation metrics. Some studies measure perplexity (a token-level metric that is itself sensitive to tokenization), while others measure downstream task accuracy. A tokenizer that appears beneficial under perplexity may not improve (or may even degrade) task performance, or vice versa. The relationship between these metrics is not straightforward.

No unified analysis framework. Most prior work tests one hypothesis in one setting—does morphological alignment help for this specific model on these specific languages? There is no framework for systematically comparing across languages, models, tasks, and scoring methodologies to determine when (if ever) morphological alignment matters and why previous studies disagree.

Limitations of the Original MorphScore

The original MorphScore (Arnett & Bergen, 2025) attempted to provide a standardized intrinsic evaluation for morphological alignment, but the current paper identifies several critical limitations that motivated this expansion (Section 1):

Missing high-resource languages. The 22-language sample excluded many languages central to NLP research, making it difficult to integrate MorphScore into standard evaluation pipelines or to compare against results reported for models like Llama and BLOOM that are extensively tested on languages like French, German, and Italian.

No contextual information. The items in the original MorphScore were isolated words without surrounding sentence context. While this does not affect tokenizers that use whitespace pre-tokenization (where context is irrelevant to how a given word is segmented), it makes MorphScore impossible to use with superword tokenizers—a class of methods that consider broader context when segmenting text. Examples include SuperBPE (Liu et al., 2025) and BoundlessBPE (Schmidt et al., 2025), which do not rely on whitespace pre-tokenization and can split across word boundaries. As these methods gain traction, the inability to evaluate them is a growing limitation.

No part-of-speech or morphological feature information. The Universal Dependencies treebanks used to create MorphScore contain rich annotations—part-of-speech tags, morphological features (case, number, tense, etc.), and dependency relations—but the original metric discarded all of this. This means MorphScore could not answer differentiated questions like "does morphological alignment matter more for nouns than verbs?" or "does alignment of inflectional morphology matter more than derivational morphology?"

No frequency weighting. The original MorphScore treated all words equally, regardless of how often they appear in real text. As Zouhar et al. (2023) argued in their work on Rényi efficiency, optimal tokenization likely depends on frequency distributions. It may be more important for high-frequency words to be tokenized efficiently (stored as single tokens for compression), while low-frequency words benefit from morphological segmentation (since the model sees them rarely and needs to generalize from their components). Alternatively, the reverse might be true—low-frequency items might be the ones where morphological alignment matters most, since the model has few opportunities to learn their behavior. The original MorphScore could not test these hypotheses.

The oversegmentation problem when measuring accuracy. If morphological alignment is measured using accuracy—the fraction of gold morpheme boundaries correctly identified by the tokenizer—then a trivial character-level tokenizer achieves a perfect score by splitting every word into individual characters. For "books," the character-level segmentation [b, o, o, k, s] places a boundary at every morphologically relevant position (between "k" and "s"), but this is clearly not what advocates of morphological alignment intend. The original MorphScore used accuracy as its primary metric, making it vulnerable to this oversegmentation confound. The authors note (Section 2.3) that "oversegmentation leads to high accuracy" and that "tokenizing words into more tokens is strongly correlated with morphological alignment as measured with accuracy"—a statistical artifact, not a genuine quality signal.

Where Other Intrinsic Metrics Fall Short

The paper also contextualizes morphological alignment within the broader landscape of intrinsic tokenizer evaluation (Section 1), identifying shortcomings of alternative metrics:

Compression metrics (fertility and corpus token count). The simplest intrinsic evaluation is compression: how many tokens does it take to encode a given text? Fertility (Rust et al., 2021) measures tokens per word but is difficult to generalize crosslinguistically because "word" is operationalized as whitespace-separated orthographic units—a definition that fails for languages without whitespace (Mandarin Chinese, Thai, Khmer). Corpus token count (CTC; Schmidt et al., 2024) measures total tokens for a parallel text across languages, enabling crosslinguistic comparison. While some work has argued that better compression increases information density and improves performance (Deletang et al., 2024; Goldman et al., 2024; Gallé, 2019), more recent evidence from Schmidt et al. (2024) found no robust relationship between tokenizer compression and language model performance—directly contradicting the compression-as-quality hypothesis.

Rényi efficiency. Zouhar et al. (2023) proposed Rényi efficiency, which evaluates whether a tokenizer allocates more compression to higher-frequency items and less to lower-frequency items—matching the theoretical optimum from information theory. The tokenization-scorer package enables easy calculation. However, later work by Cognetta et al. (2024) argued that Rényi efficiency may not provide a holistic metric of tokenization quality, highlighting specific counterexamples where the metric gives counterintuitive rankings. Like morphological alignment, Rényi efficiency has promising theoretical motivation but uncertain practical validity.

How This Paper Positions Itself

The paper positions itself not as proposing a new hypothesis about tokenizer quality, but as systematically testing an existing hypothesis with substantially expanded scope and improved methodology. The contribution structure is:

Expand the evaluation framework. Take MorphScore from 22 to 70 languages, add contextual information, POS tags, and morphological features (for future analyses), and fix the oversegmentation confound by switching from accuracy to precision and recall.
Add configurable parameters. Introduce frequency weighting and options for handling single-token words, enabling the community to test which settings best capture tokenizer quality.
Test against a broader set of models and tasks. Correlate morphological alignment (under all parameter settings) with downstream performance for five model families (Llama2, BLOOM, XGLM, Llama3, Gemma3) across seven tasks, with at least one task in every language in the 70-language sample (achieved by adding MultiBLiMP, which is derived from the same UD treebanks as MorphScore and covers all languages).
Determine whether morphological alignment predicts performance, and under which settings. The central empirical question is not "is morphological alignment good?" but rather "does morphological alignment, as measured by any reasonable variant of MorphScore, explain variance in how well models actually perform on downstream tasks?"

This framing is important because it converts a theoretical debate ("should tokenizers be morphologically aligned?") into an empirical question that can be answered with data. If morphological alignment were strongly predictive of performance, that would validate a large body of prior work advocating for morphology-aware tokenization and provide a clear design principle. If it is not predictive—as the paper's results ultimately show—that suggests the field needs to look elsewhere (or combine morphological alignment with other metrics) to understand what makes tokenizers effective.

The paper also positions itself as enabling future work beyond its own analyses. By releasing the expanded datasets with contextual information, POS tags, and morphological features—even though these are not analyzed in the current paper—the authors provide infrastructure for investigating more fine-grained questions: Does morphological alignment matter more for certain parts of speech? For inflectional versus derivational morphology? For certain morphological feature combinations? These questions are explicitly flagged as future work in Section 5, and the released resources make them tractable.

The Specific Gap: Can We Reconcile the Conflicting Evidence?

The intellectual gap the paper addresses can be stated precisely: after controlling for known confounding factors (model size, training data proportion), expanding language coverage to 70 languages, testing multiple scoring variants, and using metrics that avoid the oversegmentation confound, does morphological alignment predict downstream model performance? If the answer is yes, the conflicting prior evidence likely reflects methodological limitations of earlier studies. If the answer is no—which is what the paper finds—then the positive results in prior work may be attributable to uncontrolled confounds, the oversegmentation artifact in accuracy-based metrics, or the narrow language samples that happened to favor morphology-sensitive models.

The paper's contribution is thus both methodological (an improved evaluation framework) and empirical (a large-scale, multi-model, multi-task test of a widely-held hypothesis with a negative result). In a field where positive results dominate publication, this negative finding—that a popular intrinsic metric does not predict what it is supposed to predict—is valuable for redirecting research effort toward more promising directions.

3. Technical Approach

3.1 Reader Orientation

This paper builds an intrinsic tokenizer evaluation framework — a set of datasets, scoring algorithms, and configurable metrics that measure how well a tokenizer's output aligns with linguistically meaningful morpheme boundaries, without requiring any model training. The core problem is that we cannot currently tell whether a tokenizer is "good" without training a model and measuring downstream performance, which is computationally prohibitive; the framework solves this by providing a standardized, crosslinguistically applicable measure of morphological alignment that can be computed purely from tokenizer outputs and linguistic annotations, then tests whether that alignment measure actually predicts model performance across 70 languages, five model families, and seven tasks.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components:

Dataset Creation Pipeline — takes Universal Dependencies treebanks as input, extracts words with multi-morpheme segmentations, constructs gold-standard morphological segmentations using lemma-wordform alignment, filters out non-concatenative and irregular morphology, and produces evaluation datasets with contextual and linguistic metadata.
Morphological Scoring Engine — takes a tokenizer and an evaluation dataset, segments each word, compares predicted token boundaries against gold morpheme boundaries, and computes boundary-level and subword-level precision, recall, and F1 scores, with configurable parameters for frequency weighting and single-token word handling.
Parameter Sweep Framework — evaluates tokenizers under four combinations of configuration settings (frequency scaling on/off, single-token word inclusion on/off), producing 16 scores per tokenizer-language pair (4 boundary scores + 12 subword scores).
Correlation Analysis Pipeline — takes morphological alignment scores and reported downstream task performance for five pre-trained language models on seven multilingual benchmarks, fits linear mixed effects models controlling for model size and training data proportion, and tests whether morphological alignment explains additional variance in performance.

Information flows as follows: Universal Dependencies treebanks → dataset creation (lemma-based segmentation) → evaluation datasets (with gold segmentations, context, POS, morphological features) → morphological scoring engine (tokenizer applied, boundaries compared) → alignment scores (per language, per tokenizer, per parameter setting) → correlation analysis (alignment scores vs. downstream task performance, controlling for confounds) → conclusion about whether morphological alignment predicts model quality.

3.3 Roadmap for the Deep Dive

First, the dataset creation process — how gold morphological segmentations are derived from Universal Dependencies annotations, the filtering criteria that determine which words are retained, and why 86 initial languages reduce to 70 after quality filtering. This is foundational because all subsequent scoring depends on the quality of these gold segmentations.
Second, the morphological scoring metrics — boundary precision/recall versus subword precision/recall/F1, what each measures, why the switch from accuracy to precision/recall matters for avoiding the oversegmentation confound, and the mathematical definitions of each metric.
Third, the configurable parameters — frequency scaling (how word frequency weights are computed from treebank counts and incorporated into the scoring function) and single-token word handling (whether words that are not segmented at all count as perfectly aligned or are excluded), including the empirical relationships between frequency and tokenization behavior that motivate these parameters.
Fourth, the experimental design for testing whether morphological alignment predicts performance — the linear mixed effects model specification, the control variables (model parameters, training data proportion), the random effect structure (task as random intercept), and the ANOVA-based testing procedure.

3.4 Detailed Technical Breakdown

This is primarily an methodology and evaluation paper whose core idea is that an expanded and improved morphological alignment metric, tested across a much broader range of languages, models, and parameter settings than prior work, can determine whether morphological alignment — as a candidate intrinsic tokenizer evaluation — actually explains downstream model performance after properly controlling for known confounds.

Dataset Creation: Deriving Gold Morphological Segmentations from Universal Dependencies

Source data and language selection. All evaluation datasets are constructed from annotations in Universal Dependencies (UD), a multilingual collection of dependency-parsed treebanks with consistent annotation standards across languages. The authors select, for each language, "the largest available treebank" and use "all available splits (train, dev, and test)" including the test split because "for many of the languages that is the only split available" (Section 2.1). The exact treebanks and their sizes are listed in Appendix A (Table 2), with the number of retained evaluation items per language ranging from just over 100 (the filtering threshold) to tens of thousands for well-resourced languages.

Gold segmentation via lemma-wordform alignment. For each annotated word in a UD treebank, the authors have access to two forms: the wordform (the inflected or derived surface form as it appears in text, e.g., "launched") and the lemma (the canonical dictionary form, e.g., "launch"). The proposed gold morphological segmentation is determined by identifying the longest shared character sequence between the wordform and lemma:

Find the longest common substring between the wordform and lemma (matching from the start of both strings where possible, though the paper does not specify the exact alignment algorithm — the described behavior suggests prefix-aligned longest common subsequence).
Identify this substring as the stem.
Any characters in the wordform that precede the shared sequence are classified as a prefix.
Any characters in the wordform that follow the shared sequence are classified as a suffix.

For the wordform "launched" with lemma "launch," the longest shared sequence is "launch," giving the stem "launch" and the suffix "-ed." The gold segmentation is therefore [launch + ed], with one morpheme boundary between positions 5 and 6 (0-indexed: after character 5).

This procedure produces gold segmentations with "at least two morphemes (the stem and an affix) and at most three morphemes (a prefix, stem, and suffix)" (Section 2.1). The limitation to two or three morphemes is inherent to the alignment method — it cannot detect multiple affixes (e.g., a word with both a prefix and multiple suffixes would only have the outermost prefix and suffix identified, with everything in between treated as stem) — but the paper argues this captures the "most clear-cut cases" (Limitations).

Filtering criterion: concatenative morphology only. A critical filtering step removes words where the wordform cannot be recomposed by concatenating the proposed stem with the identified affixes. This is explicitly a filter for concatenative morphology — morphological processes where morphemes are linearly attached without changing the stem's internal structure. The authors exclude:

Suppletion: irregular forms where the stem changes entirely (e.g., English "go" → "went," where no substring of "went" corresponds to "go").
Umlaut/ablaut: vowel changes in the stem (e.g., English "sing" → "sang," German "Mutter" → "Mütter," where the stem vowel changes).
Non-concatenative morphology: root-and-pattern systems where morphological distinctions are encoded by changing vowels between consonants rather than adding affixes.

The authors motivate this exclusion with a concrete example from Afrikaans (Section 2.1): the verb "to be" has the infinitival form "wees" and the present form "is." Under the alignment procedure, the longest shared substring would be "s," yielding a proposed segmentation [i + s] with stem "-s." But "is" is an irregular form — it should not be thought of as having the stem "-s" — and this segmentation "would not be informative about the quality of tokenization." The concatenation recomposition check catches this case because "i" + "s" ≠ "wees" (the lemma), so the item is filtered out.

This filtering has systematic consequences for language coverage. The authors explicitly note that after filtering, there were "not sufficient remaining items from any of the Semitic languages (Amharic, Arabic, and Hebrew) or most isolating languages (e.g. Chinese, Vietnamese, and Thai)" (Section 2.1). Semitic languages use root-template patterns extensively — a triconsonantal root carries semantic meaning, and vowels between the consonants encode grammatical distinctions (e.g., Arabic "kataba" = "he wrote" versus "kātib" = "writer"). These are non-concatenative by nature, so the concatenation filter removes nearly all examples. Isolating languages like Vietnamese and Chinese lack overt morphological markings for number, tense, etc. — "most words do not have overt morphological markings" — so "there are not sufficient affixation patterns to create the kind of examples that are selected for by our dataset creation process" (Section 2.1).

The consequence is that the final dataset "only covers fusional and agglutinative languages" — languages where morphology is primarily expressed through affixation (prefixes and suffixes). This is an intentional scope limitation, not an accidental omission, but the authors acknowledge that "future work could focus on how to determine gold segmentations for both irregular items... and non-concatenative morphology."

Dataset sizes and filtering. From an initial 86 languages for which datasets were created, languages with "fewer than 100 items" were filtered out. This "leaves a set of 70 languages" (Section 2.1). The threshold of 100 items is a practical minimum for stable scoring — below this, a few mis-segmentations could dramatically swing the alignment score, making the metric unreliable. The unfiltered datasets for all 86 languages are released on Hugging Face for transparency and potential use in other analyses.

Included metadata. Each item in the evaluation dataset includes (Section 2.1):

Wordform: the surface form from the UD annotation.
Gold segmentation: the stem and affixes identified by lemma alignment.
Sentential context: the full sentence from which the word was extracted (new in this expanded MorphScore, enabling evaluation of superword tokenizers).
Part-of-speech (POS) tag: from UD annotations (new, enabling future analysis of whether alignment matters more for certain POS).
Morphological features: case, number, tense, etc. from UD annotations (new, unused in this paper but enabled for future work).

The sentential context is particularly important because it makes the evaluation framework compatible with superword tokenizers like SuperBPE and BoundlessBPE, which "do not rely on whitespace pre-tokenization and can split across word boundaries" (Section 1). The original MorphScore with isolated words was useless for these tokenizers because they need surrounding context to determine segmentation. The expanded datasets include the full sentence, so a superword tokenizer can tokenize the complete sentence and the evaluator can then identify which tokens correspond to the target word and evaluate their boundaries.

Morphological Scoring Metrics: Boundary-Level and Subword-Level Evaluation

The scoring engine compares a tokenizer's predicted segmentation of a word against the gold morphological segmentation and produces two families of metrics: boundary-level and subword-level. The paper adds subword-level metrics as an expansion beyond the original MorphScore, which only evaluated boundary placement.

Tokenization application. For a given tokenizer and evaluation item, the tokenizer is applied to the word (or, if using sentential context, to the full sentence, and the tokens corresponding to the target word are extracted). The tokenizer's output is a sequence of subword tokens that, when concatenated with appropriate spacing markers (e.g., the "Ġ" prefix in GPT-style tokenizers or "##" prefix in BERT-style tokenizers), reconstruct the original wordform. For example, the English word "books" might be tokenized as:

Morphologically aligned: ["book", "s"] → segmentation boundaries at position 4 (between "k" and "s").
Non-aligned: ["boo", "ks"] → segmentation boundaries at position 3 (between "o" and "k").
Oversegmented: ["b", "o", "o", "k", "s"] → boundaries at positions 1, 2, 3, 4.
Single-token: ["books"] → no boundaries.

Boundary-level metrics. The boundary-level evaluation focuses solely on whether the predicted segmentation correctly places boundaries at the gold morpheme boundary positions. It does not require that the predicted tokens exactly match the gold morphemes — only that the boundaries align.

For a word with gold segmentation [stem + suffix], there is exactly one gold boundary — the position between the stem and suffix. For a word with [prefix + stem + suffix], there are two gold boundaries. The predicted segmentation produces a set of boundary positions. The metrics are:

Macro-average boundary precision:

$\text{Boundary Precision} = \frac{|\text{predicted boundaries} \cap \text{gold boundaries}|}{|\text{predicted boundaries}|}$

where $|\text{predicted boundaries} \cap \text{gold boundaries}|$ is the count of predicted boundary positions that match gold boundary positions, and $|\text{predicted boundaries}|$ is the total number of boundary positions in the predicted segmentation. Macro-averaging means this is computed per word and then averaged across all words.

What it computes: the fraction of tokenizer-placed boundaries that correspond to actual morpheme boundaries. High precision means that when the tokenizer does split a word, it tends to split at linguistically meaningful positions. A character-level tokenizer gets very low precision because it places many boundaries (between every character), only one or two of which happen to be correct.

Why this form: precision penalizes oversegmentation. A tokenizer that splits a word into more tokens than there are morphemes will have many predicted boundaries, most of which do not match gold boundaries, driving precision down. This directly addresses the oversegmentation confound that plagued the original MorphScore's accuracy metric — under precision, character-level tokenization scores poorly.

Macro-average boundary recall:

$\text{Boundary Recall} = \frac{|\text{predicted boundaries} \cap \text{gold boundaries}|}{|\text{gold boundaries}|}$

where $|\text{gold boundaries}|$ is the total number of morpheme boundaries in the gold segmentation (1 for two-morpheme words, 2 for three-morpheme words).

What it computes: the fraction of gold morpheme boundaries that the tokenizer correctly identifies. High recall means the tokenizer rarely misses a morpheme boundary. A tokenizer that never splits (single-token for everything) gets zero recall because it places no boundaries and therefore matches none of the gold boundaries.

Why this form: recall penalizes undersegmentation. A tokenizer that stores whole words as single tokens avoids the oversegmentation penalty but fails to capture any morpheme boundaries, so recall is zero.

The choice of precision-recall decomposition over accuracy is the key methodological improvement. The paper provides an explicit example (Section 2.2):

"if the gold segmentation is [book + s] and the predicted tokens are [boo + k + s], only the boundary between 'k' and 's' is correct. This yields a boundary precision of 1/2 and a boundary recall of 1/1."

The predicted tokens [boo, k, s] place two boundaries (between "o" and "k," and between "k" and "s"), of which only the second matches the gold boundary, so precision = 1/2. There is one gold boundary, and it is matched, so recall = 1/1. Accuracy would be 1/2 (one correct boundary out of two total boundaries considered), which obscures the distinction between the oversegmentation issue (a correct boundary among spurious boundaries) and a genuine miss.

Subword-level metrics. The subword-level evaluation assesses whether "the predicted subword spans exactly match gold morphemes" (Section 2.2). This is stricter than boundary evaluation — not only must the boundaries be correct, but the entire token sequence must align with the morpheme sequence.

Continuing the example: gold = [book, s], predicted = [boo, k, s]. For subword metrics:

The token "s" matches a gold morpheme exactly, so it is a true positive.
The tokens "boo" and "k" do not match any gold morpheme — "boo" is not equal to "book," and "k" is not a substring match for "book" because spans must be exact, not substrings.
The gold morpheme "book" is not matched by any predicted token.

The subword metrics are computed token-by-token and morpheme-by-morpheme:

Micro-average subword precision:

$\text{Subword Precision}_{\text{micro}} = \frac{\sum_{i} |\text{predicted tokens}_i \cap_{\text{exact}} \text{gold morphemes}_i|}{\sum_{i} |\text{predicted tokens}_i|}$

where $i$ indexes over words, $\cap_{\text{exact}}$ means exact string match (not substring), and the numerator sums true positive token matches across all words while the denominator sums total predicted tokens across all words. Micro-averaging pools all tokens together before computing the ratio.

What it computes: across the entire evaluation set, what fraction of predicted subword tokens exactly match a gold morpheme. A value of 0.4 means that 40% of all tokens produced by the tokenizer on these words are exact matches for linguistic morphemes.

Macro-average subword precision:

$\text{Subword Precision}_{\text{macro}} = \frac{1}{N} \sum_{i=1}^{N} \frac{|\text{predicted tokens}_i \cap_{\text{exact}} \text{gold morphemes}_i|}{|\text{predicted tokens}_i|}$

where $N$ is the number of words. This computes precision per word and averages, giving equal weight to each word regardless of how many tokens it contains.

The same pattern applies to subword recall (micro and macro):

$\text{Subword Recall}_{\text{micro}} = \frac{\sum_{i} |\text{predicted tokens}_i \cap_{\text{exact}} \text{gold morphemes}_i|}{\sum_{i} |\text{gold morphemes}_i|}$

$\text{Subword Recall}_{\text{macro}} = \frac{1}{N} \sum_{i=1}^{N} \frac{|\text{predicted tokens}_i \cap_{\text{exact}} \text{gold morphemes}_i|}{|\text{gold morphemes}_i|}$

Why provide both micro and macro: micro-averaging gives more weight to words that are segmented into many tokens (since they contribute more to the total token count). Macro-averaging weights each word equally. The choice matters when tokenization behavior varies systematically with word length or morphological complexity — longer, more morphologically complex words tend to be segmented into more tokens, and macro-averaging prevents them from dominating the score.

Subword F1:

For both micro and macro variants, F1 is the harmonic mean of precision and recall:

$\text{F1} = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}$

The paper computes and releases all scores — boundary precision, boundary recall, subword F1 (micro and macro), subword precision (micro and macro), subword recall (micro and macro) — but the main correlation analyses focus on the macro-average boundary precision and recall, as these are the cleanest test of whether morphological boundary alignment predicts performance.

Score aggregation across languages. For each tokenizer, morphological alignment scores are computed per language independently. These per-language scores are what enter the correlation analysis — each data point is a (language, tokenizer, model, task) tuple, with the language's morphological alignment score for that tokenizer as the predictor and the model's task performance in that language as the outcome. This enables testing whether, for example, a language where XGLM's tokenizer is highly morphologically aligned is also a language where XGLM performs well on downstream tasks.

Configurable Parameters: Frequency Scaling and Single-Token Word Inclusion

The scoring function accepts two binary configuration parameters that modify how individual word scores are aggregated into a final score for a language-tokenizer pair.

Parameter 1: Frequency scaling (boolean). When enabled, each word's contribution to the final morphological alignment score is weighted by that word's frequency in the UD treebank used to create the dataset. Higher-frequency words contribute more to the aggregate score; lower-frequency words contribute less.

Frequency computation. Word frequency is computed as the count of occurrences of that wordform in the treebank text. This is a raw frequency, not normalized or smoothed. The authors acknowledge that this is a limitation: "our frequency metrics came only from the treebanks we used to create our datasets, meaning for some languages the sample was very small" and "many treebanks are created with data from one source, e.g. news articles" (Section 5.1). For languages with small treebanks (some have only hundreds of sentences), frequency estimates are noisy and domain-specific.

Mathematical form. With frequency scaling enabled, the final score (whether precision, recall, or F1) is a weighted average:

$\text{Weighted Score} = \frac{\sum_{w \in \text{words}} f_w \cdot s_w}{\sum_{w \in \text{words}} f_w}$

where $f_w$ is the frequency (count) of word $w$ in the treebank, and $s_w$ is the per-word alignment score (e.g., boundary precision for word $w$ ). Without frequency scaling, this reduces to the unweighted average:

$\text{Unweighted Score} = \frac{1}{N} \sum_{w \in \text{words}} s_w$

Why frequency scaling might matter. The motivation comes from information-theoretic considerations, referencing Zouhar et al. (2023)'s Rényi efficiency argument that optimal tokenization depends on frequency distributions. There are two competing hypotheses:

High-frequency items matter more. If a tokenizer aligns boundaries correctly for common words (which the model sees thousands of times during training), that provides consistent morphological signal. Misalignment on rare words matters less because the model rarely encounters them. In this view, frequency-weighted scores should correlate more strongly with performance than unweighted scores because they focus on what the model actually processes frequently.
Low-frequency items matter more. Rare words are precisely the ones that benefit from morphological decomposition — the model cannot memorize them as atomic units because it sees them too infrequently, so it must generalize from their morphological components. Misalignment on rare words degrades this generalization. In this view, frequency-weighted scores that upweight common words (which the model can memorize whole) might actually obscure the signal.

The paper tests these indirectly by comparing the predictive power of frequency-scaled versus unscaled scores on downstream performance (Appendix D), but does not directly test which hypothesis is correct — it only tests which parameter setting yields scores that better predict model quality.

Empirical relationship between frequency and tokenization. The authors report two relevant correlations (Section 3.1):

"a weak but statistically significant correlation ( $\rho = 0.119$ , $p < 0.0001$ )" between an item's frequency and the likelihood it is morphemically segmented. The relationship is positive: "more frequent items are more likely to be morphemically segmented."
"a negative correlation between word frequency and the number of tokens a word is segmented into ( $\rho = -0.108$ , $p < 0.0001$ )." More frequent words are segmented into fewer tokens.

Both correlations are weak (explaining about 1% of variance each) but highly significant due to the large sample size. They confirm that frequency and tokenization behavior are related — frequent words tend to be stored as single tokens or segmented into fewer, more morphologically aligned tokens — which is consistent with how BPE tokenization works (frequent substrings are merged earlier and more often). This provides empirical grounding for why frequency weighting might affect alignment scores: high-frequency words are systematically different in their tokenization behavior from low-frequency words, so weighting them differently changes the aggregate score and potentially its relationship to model performance.

Parameter 2: Single-token word inclusion (boolean). When a word is tokenized into a single token (the tokenizer does not split it at all), there are no predicted boundaries. The question is how to score this case. The parameter controls whether such words are:

Included (True): the word receives the score associated with a perfectly morphologically aligned tokenization — boundary precision = 1.0, boundary recall = 1.0, subword metrics = 1.0 (since the single token exactly matches the entire word, which is also the concatenation of the gold morphemes).
Excluded (False): the word is simply left out of the score calculation entirely. Only words that are actually segmented (split into multiple tokens) contribute to the final score.

Why this parameter matters. The inclusion/exclusion of single-token words trades off two considerations (Section 3.2):

Argument for exclusion: "One argument for excluding these items is that these cases do not give any indication of how morphologically aligned a segmentation of a word is, given that there is a segmentation." In other words, if you want to know how good the tokenizer is at segmenting words morphologically, you should only look at cases where it actually segments. Including single-token words inflates the score for languages with small vocabularies or high compression (where many words fit in the vocabulary as whole tokens) without providing any evidence about the quality of segmentations when they do occur.
Argument for inclusion: "However, excluding these cases might also essentially penalize a tokenizer for segmenting less. Fewer segmentations leads to better compression, which is thought to be an ideal feature of a tokenizer." A tokenizer that stores many common words as single tokens is efficient (good compression) and arguably should get credit for not unnecessarily splitting words. Excluding single-token words treats this tokenizer the same as one that segments those words but does so poorly, which penalizes compression.

Empirical effect. The authors find that "morphological alignment scores are generally higher with the inclusion of one-token items, which is what we predicted" (Section 3.2). This makes sense: including single-token words adds perfect scores (1.0) for those items, pulling the average up. The magnitude of this inflation depends on how many words in a language are stored as single tokens, which in turn depends on the tokenizer's vocabulary size and the language's morphological complexity — agglutinative languages with long words tend to have fewer single-token words.

Interaction with frequency. There is an interaction between frequency and single-token word behavior: "more frequent items are more likely to be stored in the vocabulary, instead of having to be composed of multiple tokens." The negative correlation between frequency and number of tokens ( $\rho = -0.108$ ) means that the effect of including versus excluding single-token words is concentrated on high-frequency items. Frequency scaling and single-token word inclusion therefore interact: with frequency scaling, high-frequency single-token words (which get a perfect score) receive extra weight, potentially amplifying the inflation effect.

Four configuration conditions. Combining the two boolean parameters yields four scoring configurations, referenced in Appendix D and the results:

True_True: frequency-scaled scores, single-token words excluded. This is the condition that the paper ultimately recommends as the "appropriate set of default scoring parameters" (Section 4.2) because it has "slightly more explanatory power for model performance."
True_False: frequency-scaled scores, single-token words included.
False_True: unscaled scores, single-token words excluded.
False_False: unscaled scores, single-token words included.

Optimal default settings determination. The authors do not select the optimal default settings by maximizing alignment scores themselves, but rather by determining "which is most predictive of language model performance" (Section 3.3). They fit linear mixed effects models for each of the four conditions (Appendix D) and find that "the condition which frequency-scales scores and does not include one-token words has slightly more explanatory power for model performance, though we note this difference is numeric and the amount of variance is still quite small" (Section 4.2). The difference between conditions is minimal — all four show small negative correlations with performance — so the recommendation is based on a marginal empirical advantage rather than a strong theoretical argument.

Relative tokenizer rankings across conditions (Table 1). The paper reports how the five pre-trained tokenizers rank according to precision under different conditions:

"XGLM consistently has the highest morphological alignment as measured by precision. The other tokenizers' rankings change depending on the different conditions. Measured with recall, Llama2 has the best recall. This is likely due to pervasive oversegmentations."

The instability of rankings across conditions and metrics underscores why parameter choices matter — if different reasonable parameter settings give different answers about which tokenizer is "best," then the metric's utility as an evaluation tool depends on selecting the setting that actually correlates with outcomes we care about (model performance). The fact that Llama2 leads on recall due to oversegmentation is a direct example of the metric's sensitivity: recall rewards placing many boundaries (because some will match gold boundaries by chance), and Llama2's tendency to oversegment in non-Latin scripts inflates its recall artificially.

Experimental Design for Testing Morphological Alignment's Predictive Power

The correlation analysis in Section 4 tests whether morphological alignment scores explain variance in downstream model performance beyond what is explained by known confounding factors. The design involves several deliberate choices to isolate the effect of morphological alignment.

Downstream task data. The authors use "reported model task performance results" from Arnett & Bergen (2025) plus newly added results:

XCOPA (Ponti et al., 2020): causal commonsense reasoning, covering 11 languages.
XNLI (Conneau et al., 2018): cross-lingual natural language inference, covering 15 languages.
SIB-200 (Adelani et al., 2024): topic classification, covering 200+ languages (but only languages overlapping with the MorphScore sample are used).
MultiBLiMP (Jumelet et al., 2025): subject-verb agreement evaluation derived from UD, added specifically because it covers all languages in the MorphScore sample — "the inclusion of MultiBLiMP means we have performance results for all languages in our sample, since MultiBLiMP is also derived from UD" (Section 4.1).

Model coverage. Performance scores are collected for:

Llama2 8B (Touvron et al., 2023)
BLOOM at four sizes: 560M, 1.1B, 3B, 7.1B (Le Scao et al., 2023)
XGLM 7.5B (Lin et al., 2021)
Llama3 at two sizes: 8B, 70B (Grattafiori et al., 2024) — MultiBLiMP only
Gemma3 at three sizes: 4B, 12B, 27B (Team et al., 2025) — MultiBLiMP only

Note that Llama3 and Gemma3 scores are only available for MultiBLiMP, not the other six tasks, because those results come from the MultiBLiMP paper's reported evaluations.

Training data proportions. A critical confound in any crosslinguistic model performance analysis is that models see vastly different amounts of text in different languages during pretraining. A model may perform poorly on a language not because of tokenization quality but simply because it was trained on very little text in that language. The paper controls for this using "estimated training data proportions from Hayase et al. (2024)" (Section 4.1), a method that infers language-specific data proportions from the tokenizer's own vocabulary distribution. This is necessary because "the model developers do not release that information about the pre-training data" for most models. BLOOM and XGLM are the only models that publicly report their training data language composition; for Llama2, Llama3, and Gemma3, the Hayase et al. estimates are the best available proxy.

Linear mixed effects model specification. The statistical model is:

$\text{Performance}_{ij} = \beta_0 + \beta_1 \cdot \text{Parameters}_i + \beta_2 \cdot \text{TrainingDataProp}_{ij} + \beta_3 \cdot \text{MorphAlign}_{ij} + u_j + \epsilon_{ij}$

where:

$\text{Performance}_{ij}$ is the reported task performance for model $i$ on task $j$ in a given language.
$\text{Parameters}_i$ is the model parameter count, a continuous fixed effect capturing the well-established scaling law that larger models perform better (Kaplan et al., 2020).
$\text{TrainingDataProp}_{ij}$ is the estimated proportion of the model's pretraining data in that language, a continuous fixed effect capturing that models perform better on languages they see more during training (Bagheri Nezhad & Agrawal, 2024; Li et al., 2024).
$\text{MorphAlign}_{ij}$ is the morphological alignment score (precision or recall, depending on the model variant) for that model's tokenizer on that language, the predictor of interest.
$u_j \sim \mathcal{N}(0, \sigma^2_u)$ is a random intercept for benchmark task $j$ , accounting for the fact that different tasks have different baseline difficulty levels — XCOPA accuracy numbers are not directly comparable to XNLI accuracy numbers because the tasks differ in inherent difficulty, chance performance, and evaluation methodology.
$\epsilon_{ij} \sim \mathcal{N}(0, \sigma^2_\epsilon)$ is the residual error.

Why this form: the random intercept for task avoids the false precision that would come from pooling observations across tasks with different difficulty scales. Without it, a model that scores 80% on XCOPA and 40% on XNLI would have these treated as independent observations of "performance," when in fact the XCOPA observation is systematically higher not because the model is better at that language but because XCOPA is an easier task. The random intercept accounts for this task-level variability.

The fixed effects of parameters and training data proportion are included specifically because the authors want to test whether morphological alignment explains variance above and beyond these known predictors. The testing procedure uses a nested model comparison:

Fit a baseline model: $\text{Performance} \sim \text{Parameters} + \text{TrainingDataProp} + (1 | \text{Task})$
Fit the full model adding morphological alignment: $\text{Performance} \sim \text{Parameters} + \text{TrainingDataProp} + \text{MorphAlign} + (1 | \text{Task})$
Compare using ANOVA (likelihood ratio test): does the full model fit significantly better than the baseline?

The test statistic is $\chi^2$ with degrees of freedom equal to the difference in the number of parameters (1, since MorphAlign adds one predictor). A significant $\chi^2$ indicates that morphological alignment explains variance in performance beyond what is already explained by model size and training data proportion. The authors also report marginal $R^2$ — the proportion of variance explained by the fixed effects (including morphological alignment) — separately from the total $R^2$ that includes the random effect of task.

Simple linear regression complement. In addition to the mixed effects model, the authors use "a simple linear regression to test how much variance morphological alignment explains in the model performance scores" (Section 4.1) without controlling for other factors. This provides a baseline correlation — if morphological alignment explained a large fraction of raw performance variance, that would be interesting even if the effect is partially confounded with model size or data proportion. The fact that even this un-controlled correlation is very small strengthens the negative result.

Why both precision and recall are tested separately. The paper runs the full analysis twice — once with morphological alignment measured as precision, once as recall. These metrics capture fundamentally different aspects of tokenization behavior:

Precision tests whether the tokenizer's segmentation decisions, when it segments, are morphologically meaningful. A tokenizer could have high precision by rarely segmenting (so most segmentations are on clear morphological boundaries) but low recall (because it misses many boundaries by not segmenting enough).
Recall tests whether the tokenizer captures all morphological boundaries. A tokenizer could have high recall by segmenting aggressively (character-level tokenization captures every boundary) but low precision (because it segments at many non-morphological positions).

The fact that the authors find recall significantly explains additional variance ( $\chi^2(1) = 391.42$ , $p < 0.001$ ) while precision does not ( $\chi^2(1) = -6.99$ , $p = 1$ ) is a nuanced finding: it suggests that whatever small relationship exists between morphological alignment and performance is driven by capturing boundaries (recall) rather than by avoiding spurious boundaries (precision). However, since both explain extremely little overall variance ( $R^2 = 0.024$ for recall, $R^2 = 0.005$ for precision), the practical significance is negligible regardless of statistical significance.

Cross-validation is not used. Unlike the original MorphScore, the paper does not describe any cross-validation or held-out evaluation of the scoring parameters themselves — the parameter selection ("optimal default settings") is based on which condition best predicts model performance, and this determination is made on the same data used to evaluate the predictive relationship. The authors acknowledge that the difference between conditions is "numeric and the amount of variance is still quite small," so overfitting to condition choice is unlikely to materially affect conclusions, but the lack of cross-validation for parameter selection is a methodological limitation.

Summary of Design Choices and Their Justifications

Lemma-wordform alignment for gold segmentation over human annotation: scalable across 70 languages without requiring linguists to manually segment thousands of words per language; leverages existing UD annotations consistently applied across languages.
Concatenative-only filtering over attempting to handle non-concatenative morphology: avoids the unsolved problem of defining gold segmentations for root-template patterns and irregular forms where there is no principled answer about what the "correct" segmentation should be; limits scope to cases where the gold standard is unambiguous.
Precision and recall over accuracy for scoring: eliminates the oversegmentation confound where character-level tokenization achieves perfect accuracy; decomposes alignment into two interpretable components (boundary detection vs. boundary precision) that capture different aspects of tokenizer behavior.
Boundary-level and subword-level metrics separately instead of combined: boundary metrics answer "does the tokenizer split at the right places?" while subword metrics answer "do the resulting tokens correspond to actual morphemes?" These are different questions, and a tokenizer could be good at one but poor at the other.
Four parameter configurations (2×2 frequency × single-token) over a single default: acknowledges uncertainty about the correct formulation and enables empirical determination of which settings best predict model performance rather than relying on a priori assumptions.
Linear mixed effects models with task random intercept over simple correlation: controls for the known confounds of model size and training data proportion, and accounts for different tasks having different difficulty baselines, providing a more stringent test than raw correlation.
Hayase et al. (2024) estimated data proportions over ignoring data composition or using only models that report it: enables inclusion of Llama2, Llama3, and Gemma3 despite their closed training data, substantially expanding the model sample.
MultiBLiMP as the coverage-complete task over relying only on XCOPA/XNLI/SIB-200: ensures every language in the 70-language sample has at least one task evaluation, so no language is excluded from the correlation analysis due to missing performance data.

4. Key Insights and Innovations

Innovation 1: Proving a Negative — Morphological Alignment Is Not a Useful Predictor of Downstream Performance

The paper's most intellectually significant contribution is not a new method or a positive result, but rather a rigorous, large-scale negative finding that challenges a widely-held assumption in the tokenization literature. Across 70 languages, five model families, seven tasks, and four scoring configurations, morphological alignment explains at most ~2.4% of the variance in downstream model performance (recall $R^2 = 0.024$ , precision $R^2 = 0.005$ ; Section 4.2), and the correlation is negative. This is not a failure to detect a real effect due to insufficient statistical power — the analysis has substantial power given the number of language-model-task combinations — but rather evidence that the effect, if it exists at all, is too small to be practically useful for tokenizer selection.

The significance of this finding lies in what it contradicts. The paper cites over a dozen prior works arguing that morphologically aligned tokenization benefits model performance (Park et al., 2020; Vasiu & Potolea, 2020; Bostrom & Durrett, 2020; Hofmann et al., 2021; Nzeyimana & Niyongabo Rubungo, 2022; Erkaya, 2022; Toraman et al., 2023; Držík & Forgac, 2024; Libovický & Helcl, 2024; Jabbar, 2024; Uzan et al., 2024; Bauwens & Delobelle, 2024; Asgari et al., 2025; Batsuren et al., 2024). This is not a fringe position — it spans multiple research groups, languages, and publication venues over at least five years. The present paper's null result does not necessarily prove these prior findings were wrong (they may hold in specific settings or with specific operationalizations of morphological alignment), but it does establish that morphological alignment as a general, crosslinguistically robust intrinsic evaluation metric does not work. If you are choosing between tokenizers before training a model, measuring morphological alignment will not tell you which tokenizer will produce a better-performing model.

This is a genuine conceptual advance because negative results that overturn consensus are rare and valuable in NLP. The field has a well-documented bias toward publishing positive findings and new methods. A paper that simply says "this widely-studied property does not predict what people thought it predicted" provides a different kind of value: it redirects research effort away from a likely dead end and toward more promising directions. Sections 5.2 and 5.3 explicitly suggest that morphological alignment may need to be combined with other metrics (compression, Rényi efficiency) or analyzed at finer granularity (by POS, by morphological feature) to capture what matters about tokenizer quality — directions that are now motivated by the demonstrated inadequacy of alignment alone, rather than by prior assumptions.

Innovation 2: Decomposing "Alignment" into Precision and Recall to Expose the Oversegmentation Confound

A more subtle but methodologically important innovation is the paper's diagnosis and correction of a fundamental confound in how morphological alignment had previously been measured. The original MorphScore (Arnett & Bergen, 2025) and much prior work used accuracy — the fraction of gold morpheme boundaries correctly identified by the tokenizer — as the alignment metric. The problem, which the paper makes explicit in Section 2.3, is that accuracy creates a perverse incentive: "a tokenizer can achieve a perfect alignment score by segmenting a word into characters." For the word "books," the character-level segmentation [b, o, o, k, s] places a boundary between "k" and "s" — the one gold morpheme boundary — and thus achieves perfect accuracy, even though this is clearly not what morphological alignment advocates intend.

This is not merely a theoretical edge case. The authors observe empirically that "tokenizing words into more tokens is strongly correlated with morphological alignment as measured with accuracy" (Section 2.3) and that Llama2 tokenizers — which oversegment aggressively in non-Latin scripts, "separating characters and diacritics or decomposing into bytes" — achieve inflated accuracy scores. This means that any prior study using accuracy-based morphological alignment metrics may have been measuring oversegmentation rather than genuine linguistic alignment, potentially explaining some of the positive results in the literature (if models with more aggressive tokenizers happened to perform better for unrelated reasons).

The fix — switching to precision and recall — is simple conceptually but has deep implications for how alignment is interpreted:

Precision penalizes oversegmentation. A character-level tokenizer achieves perfect recall but near-zero precision because most of its boundaries are spurious. High precision means "when this tokenizer splits, it splits at linguistically meaningful places."
Recall penalizes undersegmentation. A tokenizer that never splits achieves zero recall because it identifies no boundaries. High recall means "this tokenizer captures most morphological boundaries."

By providing both metrics, the paper enables a more nuanced understanding of tokenizer behavior: XGLM achieves the highest precision (its splits are morphologically meaningful), while Llama2 achieves the highest recall due to "pervasive oversegmentations" (Section 3.3, Table 1). These are fundamentally different properties that were conflated in accuracy-based metrics. The finding that recall explains slightly more variance in model performance than precision ( $R^2 = 0.024$ vs. $R^2 = 0.005$ ; Section 4.2), while still practically negligible, suggests that the mechanism making Llama2's tokenization behavior correlate with performance is boundary coverage (recall), not boundary precision (precision) — a distinction that would be invisible under accuracy or F1 alone.

This innovation is fundamental rather than incremental for the specific problem of intrinsic tokenizer evaluation. It identifies a measurement error in prior work, provides a corrected measurement framework, and demonstrates that the corrected measurements yield different tokenizer rankings (Table 1). This is analogous to the shift from accuracy to precision-recall curves in imbalanced classification problems — the metric itself changes what conclusions you draw.

Innovation 3: Configurable Evaluation Parameters as a Framework for Community Consensus-Building

The paper explicitly makes the evaluation framework configurable — with parameters for frequency scaling and single-token word inclusion — and then empirically determines which settings best predict model performance rather than asserting a single "correct" formulation. This is a methodological innovation in how intrinsic evaluation metrics should be developed and validated.

The standard approach in the literature is to define a metric, assert its theoretical justification, and then use it. MorphScore v1 did this. Rényi efficiency (Zouhar et al., 2023) did this — it argued from information theory that frequency-weighted token allocation is optimal, and then provided a metric that measures deviation from this optimum. The limitation of this approach is that theoretical justification does not guarantee empirical validity. A metric can be mathematically elegant and linguistically motivated but still fail to predict the outcomes practitioners actually care about.

The paper's alternative approach — parameterize the metric, sweep the parameter space, and select settings based on predictive validity — treats the evaluation framework itself as a hypothesis to be tested rather than a definition to be asserted. The four conditions (True_True, True_False, False_True, False_False in Appendix D) represent different hypotheses about what aspects of morphological alignment should matter:

Frequency scaling (True/False): "Should we care more about frequent words because the model sees them often, or treat all words equally?"
Single-token inclusion (True/False): "Should we credit a tokenizer for not splitting a word, or only evaluate the quality of its split decisions?"

The finding that the True_True condition (frequency-scaled, single-token words excluded) has "slightly more explanatory power for model performance" (Section 4.2) provides an empirically-grounded default while leaving the framework open for others to test alternative settings. This matters because reasonable people could disagree about these parameter choices a priori. By making the framework configurable and releasing all scores for all conditions (Appendix D, Figures 3–4), the paper enables the community to replicate, challenge, or refine the parameter choices without rebuilding the evaluation pipeline.

This innovation is incremental to the evaluation methodology but fundamental to the meta-methodology — it establishes a pattern for how intrinsic NLP metrics should be validated: define a parameterized family of metrics, test which member best predicts extrinsic outcomes, and release all variants so the community can verify and extend. This is not yet standard practice in tokenizer evaluation (or in NLP evaluation more broadly), making it a transferable contribution beyond the specific finding about morphological alignment.

Innovation 4: Expanding Language Coverage by 3.2× with a Systematic Filtering Rationale

While expanding from 22 to 70 languages might appear to be a purely engineering contribution, the paper's language coverage expansion is intellectually innovative in how it handles the boundary between inclusive coverage and metric validity. The paper explicitly documents which languages are excluded and why, establishing a principled scope limitation that is more valuable than an indiscriminate expansion would be.

The key conceptual move is the concatenative-only filtering criterion: only words where the wordform can be recomposed by concatenating the identified stem and affixes are retained for evaluation. The paper argues that without this criterion, gold segmentations "would not be informative about the quality of tokenization" — the Afrikaans "is" example (Section 2.1) demonstrates a case where the alignment procedure would produce a segmentation [i + s] that "should not be thought of as having the stem -s."

The systematic consequences of this filtering are substantial and honest:

Semitic languages (Amharic, Arabic, Hebrew) are excluded because their non-concatenative root-template morphology (Figure 1: "kataba" vs. "kātib") cannot be evaluated with a concatenation-based gold standard. The paper explicitly acknowledges that "recent work has sought solutions for effective tokenization in languages with these morphological patterns" (Gazit et al., 2025) but does not attempt to incorporate these solutions.
Isolating languages (Chinese, Vietnamese, Thai) are excluded because "most words do not have overt morphological markings" and "there are not sufficient affixation patterns to create the kind of examples that are selected for by our dataset creation process."
The resulting coverage is restricted to "fusional and agglutinative languages" (Section 2.1) — still a substantial and diverse set, but not a universal claim.

This approach contrasts with a common pattern in multilingual NLP where language coverage is expanded by lowering quality standards — including languages with noisy annotations, inadequate sample sizes, or annotation schemes that don't fit the evaluation framework well. The paper instead expands coverage aggressively (3.2×) while maintaining a clearly-defined validity boundary, and releases the filtered-out data (the 16 languages with fewer than 100 items) separately for transparency, rather than including them and hoping the noise averages out.

The innovation here is in how the expansion is framed: not as "we now cover 70 languages" (which would obscure the systematic exclusions), but as "our approach covers fusional and agglutinative languages; non-concatenative and isolating languages require fundamentally different evaluation strategies that we do not yet have." This is a more honest and scientifically useful contribution than a superficially broader but methodologically compromised expansion would be, because it identifies specific classes of languages for which morphological alignment evaluation remains an open problem rather than pretending the metric applies universally.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation is performed on custom-constructed morphological datasets derived from Universal Dependencies (UD) treebanks. For each of 70 languages, the authors selected "the largest available treebank" and used all available splits (train, dev, and test), including the test split "for many of the languages that is the only split available" (Section 2.1). After filtering for concatenative morphology and removing languages with fewer than 100 items, the final dataset contains per-language item counts ranging from just over 100 to tens of thousands (Appendix A, Table 2). These datasets contain words with gold morphological segmentations, sentential context, POS tags, and morphological features. The key evaluation variable is morphological alignment score — boundary-level precision/recall and subword-level precision/recall/F1 — computed per language for each tokenizer.
Base model(s). The tokenizers evaluated come from five pre-trained language model families, selected to span different architectures, scales, and training data distributions: Llama2 8B (Touvron et al., 2023), BLOOM at four sizes (560M, 1.1B, 3B, 7.1B; Le Scao et al., 2023), XGLM 7.5B (Lin et al., 2021), Llama3 at two sizes (8B, 70B; Grattafiori et al., 2024), and Gemma3 at three sizes (4B, 12B, 27B; Team et al., 2025). The paper states these models are chosen for their "representative" character, though downstream task performance is only available for subsets of them on specific benchmarks (Section 4.1). BLOOM and XGLM are the only models that "report their training data proportions" (Section 4.1); for Llama2, Llama3, and Gemma3, the paper uses "estimated training data proportions from Hayase et al. (2024)," a method that infers data composition from tokenizer vocabulary distributions.
Metrics. The primary intrinsic metric is morphological alignment score, computed in two variants (Section 2.2): boundary-level macro-average precision and recall (measuring whether tokenizer-placed boundaries align with gold morpheme boundary positions, irrespective of whether the resulting tokens exactly match gold morphemes) and subword-level micro- and macro-average precision, recall, and F1 (measuring whether predicted subword tokens exactly match gold morpheme spans). Boundary precision penalizes oversegmentation, boundary recall penalizes undersegmentation. The extrinsic metric is downstream task performance, measured as reported accuracy on seven multilingual benchmarks: XCOPA (causal reasoning, 11 languages), XNLI (natural language inference, 15 languages), SIB-200 (topic classification, 200+ languages), and MultiBLiMP (subject-verb agreement, all MorphScore languages). MultiBLiMP is the novel addition to the suite, added specifically because "the inclusion of MultiBLiMP means we have performance results for all languages in our sample" (Section 4.1).
Baselines. The paper does not evaluate "baselines" in the conventional sense — there is no alternative intrinsic metric against which MorphScore is compared for predictive power. However, the correlation analyses include as control variables model size in parameters and proportion of training data in each language, both of which are known to predict performance (Kaplan et al., 2020; Bagheri Nezhad & Agrawal, 2024). These serve as the baseline predictors in the nested model comparison: the question is whether morphological alignment explains variance above and beyond these controls. The original MorphScore (Arnett & Bergen, 2025) covering 22 languages serves as the methodological baseline for the expanded framework, with the key improvements being the larger language sample, the switch from accuracy to precision/recall, and the addition of configurable parameters.
Generation budget / compute accounting. There is no generation budget or compute accounting in this paper. The evaluation is purely intrinsic — it computes alignment scores by applying existing pre-trained tokenizers to evaluation datasets and comparing outputs against gold segmentations. No models are trained or fine-tuned as part of the evaluation. The tokenizers themselves are the pre-trained tokenizers distributed with the listed models; their vocabulary sizes and training procedures are fixed and inherited from the model releases. This is a strength of intrinsic evaluation (it is computationally cheap) but also a limitation (it assumes that differences in tokenizer training — e.g., data composition, vocabulary size choice — are captured indirectly through the tokenizer's behavior on the evaluation dataset).
Cross-validation / statistical protocol. The paper does not use cross-validation for the morphological alignment scoring itself — scores are computed on the full evaluation datasets without held-out splits. For the correlation between alignment and performance, the paper uses linear mixed effects models (Section 4.1) with downstream task performance as the dependent variable, model parameters and training data proportion as fixed effects, benchmark task as a random intercept (to account for different tasks having different difficulty baselines), and morphological alignment (precision or recall) as the additional predictor of interest. The testing procedure uses nested model comparison via ANOVA (likelihood ratio test): the baseline model with only controls is compared to the full model adding morphological alignment, and the $\chi^2$ statistic tests whether the improvement in fit is significant. The paper also reports marginal $R^2$ for the fixed effects. For comparing scoring configurations (True_True vs. True_False vs. False_True vs. False_False in Appendix D), the paper uses "the amount of variance explained by morphological alignment" under each configuration to select default settings, noting that "the difference is numeric and the amount of variance is still quite small" (Section 4.2). No correction for multiple comparisons across configurations or metrics is reported.

Main Quantitative Results

Parameter Setting Effects on Morphological Alignment Scores

Frequency scaling relationships. The paper reports two empirical tests of the relationship between word frequency and tokenization behavior (Section 3.1). Across all items in the 70-language evaluation datasets:

There is "a weak but statistically significant correlation ( $\rho = 0.119$ , $p < 0.0001$ )" between an item's frequency and the likelihood that it is morphemically segmented. The correlation is positive: "more frequent items are more likely to be morphemically segmented."
There is "a negative correlation between word frequency and the number of tokens a word is segmented into (Spearman's $\rho = -0.108$ , $p < 0.0001$ )." More frequent words are segmented into fewer tokens.

Both correlations are extremely weak — explaining approximately 1% of variance — but are statistically significant due to the large sample size (tens or hundreds of thousands of word-tokenizer pairs across 70 languages and five tokenizers). These results establish empirically that frequency and tokenization behavior are related in the expected direction (frequent words get preferential treatment), which provides the empirical grounding for why frequency scaling might matter.

Single-token word inclusion effects. The paper finds that "morphological alignment scores are generally higher with the inclusion of one-token items, which is what we predicted" (Section 3.2). The magnitude of this inflation depends on the language and tokenizer — languages and tokenizers where more words are stored as single tokens (due to smaller average word length, larger vocabulary coverage, or both) show larger differences between the include and exclude conditions. No quantitative magnitude is reported.

There is also "an interaction between word frequency and the likelihood that a tokenizer represents a word as a single token" (Section 3.2): frequent items are more likely to be stored as single tokens, consistent with the $\rho = -0.108$ correlation between frequency and token count. This means that frequency scaling and single-token inclusion interact — when both are enabled, high-frequency single-token words receive both a perfect alignment score and a high weight, potentially amplifying the inflation effect concentrated on common words.

Overall effects of parameter choices on scores. The linear mixed effects model with morphological alignment precision as the dependent variable, frequency scaling, one-token word inclusion, and training split as fixed effects, and tokenizer as a random intercept, finds "significant differences across the different categories" (Section 3.3). Full statistical results are reported in Appendix C (Tables 8 and 9), though the specific coefficient magnitudes are not discussed in the main text.

Relative tokenizer rankings across conditions (Table 1). Under morphological alignment measured with precision:

"XGLM consistently has the highest morphological alignment as measured by precision. The other tokenizers' rankings change depending on the different conditions."

Under recall:

"Measured with recall, Llama2 has the best recall. This is likely due to pervasive oversegmentations."

The instability of rankings across conditions and metrics is a key finding: different reasonable ways of computing the same underlying concept (morphological alignment) produce different answers about which tokenizer is "best." This is not a property of the metric per se but of the tokenizers — XGLM is genuinely more precise (its splits tend to be at morphological boundaries), while Llama2 achieves higher recall (it places more boundaries, some of which happen to be correct). The fact that which property is rewarded depends on which metric is used underscores the importance of validating the metric against an external criterion (downstream performance) rather than assuming one metric is "correct."

Correlation Between Morphological Alignment and Downstream Task Performance

Control variables explain significant variance. The baseline linear mixed effects model containing only model parameters and training data proportion as fixed effects (with task as a random intercept) explains significantly more variance than an intercept-only model ( $\chi^2(2) = 25.67$ , $p < 0.001$ ; Section 4.2). This confirms that the known predictors (larger models perform better, models perform better on languages they see more of during pretraining) are operating as expected in this dataset.

Morphological alignment measured by recall explains additional variance; precision does not. When morphological alignment measured by recall is added to the model:

"Morphological alignment, as measured with recall, explains additional variance above and beyond these factors ( $\chi^2(1) = 391.42$ , $p < 0.001$ )" (Section 4.2).

However, when measured by precision:

"precision does not ( $\chi^2(1) = -6.99$ , $p = 1$ )" (Section 4.2).

The negative $\chi^2$ value for precision is anomalous — $\chi^2$ statistics cannot be negative — and likely indicates a reporting error or a convergence issue in the mixed effects model (the paper does not explain this). Regardless, the $p = 1$ value clearly indicates that precision adds no explanatory power beyond the control variables.

This asymmetry between recall and precision is notable. It suggests that whatever small signal exists in morphological alignment's relationship with performance is driven by boundary coverage (does the tokenizer identify morphological boundaries?) rather than boundary precision (are the tokenizer's identified boundaries at correct positions?). A possible mechanism: models may benefit from having morphological boundaries consistently identified (high recall), even if the tokenizer also makes spurious splits elsewhere (low precision), because the consistent morphological signal provides a regularized representation of morphological structure. However, as the next result shows, even this effect is practically negligible.

Morphological alignment explains extremely little variance. Despite the statistically significant improvement in model fit for recall, the practical magnitude of morphological alignment's explanatory power is tiny:

"the full linear mixed effects model only explains a small fraction of the variance (recall $R^2 = 0.024$ , precision $R^2 = 0.005$ )" (Section 4.2).

These are marginal $R^2$ values for the fixed effects — meaning that morphological alignment, model size, and training data proportion together explain 2.4% of the variance in downstream performance when alignment is measured by recall, and only 0.5% when measured by precision. Even the 2.4% includes the contributions of model size and training data, which were already significant in the baseline model. The unique contribution of morphological alignment is the difference between the full model $R^2$ and the baseline model $R^2$ , which is not reported but is necessarily smaller than 0.024 — likely well under 0.01.

The correlation is negative (Figure 2). Critically, the direction of the relationship is negative:

"In addition to being a very small effect, the correlation between morphological alignment and model performance is negative. This is consistent with the findings in Arnett & Bergen (2025), and challenges claims that morphologically aligned tokenization can contribute to better model performance."

Figure 2 visualizes this relationship: model performance on different tasks (color-coded) is plotted against morphological alignment recall (left panel) and precision (right panel). The scatter plots show a slight downward trend — higher morphological alignment is associated with slightly lower performance, not higher. The paper does not report the slope or confidence intervals for these trends, relying on the mixed effects model statistics instead.

Comparison across scoring configurations (Appendix D). The paper evaluates all four parameter configurations (frequency scaling × single-token word inclusion) for their ability to predict model performance. Figures 3 and 4 in Appendix D show correlation plots for precision and recall respectively across all four conditions (True_True, True_False, False_True, False_False), with model task indicated by color. The key finding is:

"the condition which frequency-scales scores and does not include one-token words has slightly more explanatory power for model performance, though we note this difference is numeric and the amount of variance is still quite small. All of the conditions still show small negative correlations with model performance."

No quantitative comparison of $R^2$ values across conditions is reported in the main text or appendix — the determination is based on "slightly more explanatory power" rather than a formal model comparison. The consistency of the negative correlation across all four conditions is the more important finding: the qualitative conclusion (morphological alignment does not positively predict performance) is robust to parameter choices.

Ablation Studies and Robustness Checks

The paper does not contain conventional ablation studies — there is no model component or preprocessing step to remove and retest. However, the parameter sweep across the four scoring configurations (Appendix D) and the use of both precision and recall metrics serve a similar function: testing whether the conclusion is sensitive to methodological choices.

Frequency scaling (on vs. off): All four configurations show "small negative correlations with model performance" (Section 4.2), with the frequency-scaled, single-token-excluded condition having "slightly more explanatory power." The insensitivity of the qualitative conclusion to frequency scaling suggests that word frequency — despite its empirical relationship with tokenization behavior ( $\rho = 0.119$ for morphological segmentation likelihood, $\rho = -0.108$ for token count) — does not substantially mediate the (lack of) relationship between morphological alignment and performance.
Single-token word inclusion (included vs. excluded): Similarly, the negative correlation holds whether single-token words are included (treating them as perfectly aligned, which inflates scores) or excluded (only evaluating words that are actually segmented). The robustness across this parameter suggests that the null finding is not an artifact of how undersegmented languages are scored, since excluding them (which penalizes undersegmentation) produces the same qualitative result as including them (which rewards it).
Precision vs. recall as the alignment metric: The finding that recall ( $R^2 = 0.024$ ) has more explanatory power than precision ( $R^2 = 0.005$ ) is itself a form of ablation — it decomposes morphological alignment into two components and shows that neither is practically useful as a performance predictor, though recall captures slightly more signal.
MultiBLiMP as full-coverage task: The addition of MultiBLiMP (which covers all 70 languages in the MorphScore sample, unlike XCOPA's 11 or XNLI's 15) is a robustness check for language coverage bias. Without MultiBLiMP, the correlation analysis would be restricted to the subset of languages covered by XCOPA/XNLI/SIB-200, which are predominantly high-resource European languages. The inclusion of MultiBLiMP ensures that the null result holds even when all 70 languages (including lower-resource fusional and agglutinative languages) are included in the analysis.

What was not ablated. The paper does not test:

Different gold segmentation methods. The lemma-wordform alignment procedure is the only method used. Alternative segmentation approaches — using existing morphological analyzers (e.g., Morfessor, UD's own morphological feature annotations directly), or human-annotated segmentations — might produce different gold standards and different alignment scores. The paper acknowledges that "the operationalization of morphological boundaries is coarse" and "undoubtedly misses many informative cases" (Limitations).
Vocabulary size as a mediator. Tokenizers with larger vocabularies can store more whole words and tend to segment less, which affects morphological alignment scores (fewer segmentations → lower recall, higher precision when segmentations do occur). The paper does not control for vocabulary size in the correlation analysis, meaning that some of the (already minimal) variance attributed to morphological alignment may actually be vocabulary size effects.
Separate evaluation by POS or morphological feature. The datasets include POS and morphological feature annotations specifically to enable these analyses (Section 2.1), but the paper does not conduct them, flagging them as future work (Section 5.3). It is possible that morphological alignment matters for some linguistic categories (e.g., verbs, where inflectional morphology encodes critical syntactic information) but not others (e.g., nouns with predictable plural marking), and that pooling all categories together washes out category-specific effects.
Interaction with language morphological type. The paper notes that the dataset "only covers fusional and agglutinative languages" (Section 2.1). It does not test whether the predictive power of morphological alignment differs between these two types, even though agglutinative languages (with long sequences of transparently concatenated morphemes) might benefit more from morphological alignment than fusional languages (where morphemes are more fused and boundaries less clear).

Critical Assessment

Does the paper demonstrate that morphological alignment does not predict model performance?

The answer is yes, with important scope qualifications.

The core empirical finding — that morphological alignment scores explain only ~2.4% of the variance in downstream task performance at most, with a negative direction — is clearly supported by the data, the analysis, and the figures. The linear mixed effects model is appropriate for the question (it controls for the known confounds of model size and training data proportion), the random intercept for task accounts for different difficulty baselines, and the negative result is robust across all four scoring configurations and both precision and recall metrics. The consistency with the original MorphScore finding (Arnett & Bergen, 2025) further strengthens the case — two independent evaluations using different language samples (22 vs. 70 languages) and different metrics (accuracy vs. precision/recall) converge on the same qualitative conclusion.

However, the paper demonstrates something more specific than the broad claim "morphological alignment does not predict model performance." It demonstrates that morphological alignment, as operationalized by lemma-wordform alignment on concatenative morphology in UD treebanks, under the specific tokenizers evaluated, using boundary precision/recall metrics, does not explain meaningful variance in reported performance on the specific downstream tasks tested. Several of these qualifications matter:

The negative finding is only demonstrated for the tokenizers and models tested (Llama2, BLOOM, XGLM, Llama3, Gemma3). These are all large, pre-trained, predominantly English-centric multilingual models. It is possible that morphological alignment matters more for monolingual models trained specifically on morphologically rich languages, or for smaller models where efficient use of parameters is more critical. The paper does not test this. Someone training a dedicated Finnish or Turkish model might reasonably wonder whether these results (obtained with massively multilingual models where the tokenizer must serve dozens of languages simultaneously) generalize to their setting. The authors implicitly acknowledge this in the Limitations, noting their "focus is on large, autoregressive LMs" and they "exclude encoder models or those trained with masked language modeling."
The downstream tasks are a specific, limited set. MultiBLiMP tests subject-verb agreement; XCOPA tests causal reasoning; XNLI tests natural language inference; SIB-200 tests topic classification. None of these are tasks where morphological segmentation quality would obviously be the bottleneck — a model can perform well on topic classification without any morphological awareness, and causal reasoning likely depends more on world knowledge and logical structure than on morphological decomposition. Tasks that more directly test morphological generalization — e.g., inflection generation, morphological paradigm completion, or syntactic tasks sensitive to agreement morphology — might show stronger correlations with morphological alignment. The paper does not include such tasks, relying instead on the general-purpose benchmarks available in many languages.
The gold segmentations capture only the simplest cases of concatenative morphology. The lemma-wordform alignment method identifies at most one prefix and one suffix, and the concatenation recomposition check filters out any irregular or non-concatenative morphology. This means the gold standard systematically excludes the very morphological phenomena that might benefit most from aligned tokenization — complex multi-affix words in agglutinative languages, stem-altering derivational morphology, and morphophonological alternations. The paper is transparent about this ("This undoubtedly misses many informative cases," Limitations), but the consequence is that the evaluation may be measuring alignment on the easy cases (where the model can likely learn the pattern regardless of tokenization) while missing the hard cases (where tokenization might matter).
Tokenizers are evaluated at a fixed snapshot, not trained for different alignment properties. The tokenizers tested are off-the-shelf pre-trained tokenizers from released models, with whatever vocabulary sizes, training data, and BPE merge counts their developers chose. The paper is correlational, not causal — it observes that in the set of existing tokenizers, those with higher morphological alignment do not have better downstream models. It does not manipulate tokenizer alignment experimentally (e.g., taking the same architecture and training tokenizers with different amounts of morphological supervision, then training models from scratch and comparing performance). A causal claim would require this experimental manipulation; the paper's correlational design cannot rule out the possibility that morphological alignment would help if properly optimized, but the tokenizers that happen to achieve high alignment also have other properties (e.g., smaller vocabularies, less training data) that offset the benefit.
Training data proportions are estimated, not known, for most models. The paper uses Hayase et al. (2024)'s estimated data proportions for Llama2, Llama3, and Gemma3. These are estimates derived from the tokenizer's vocabulary distribution — clever but not ground truth. If the estimates systematically overestimate or underestimate data proportions in ways correlated with morphological complexity (which is plausible — languages with complex morphology may be over- or under-represented in certain types of web text, and the estimation method may have language-specific biases), the control for training data proportion could be incomplete, leaving residual confounding that obscures a real alignment-performance relationship.

Does the paper's parameter sweep identify optimal scoring settings?

The answer is provisionally yes, but the evidence is minimal.

The finding that the True_True condition (frequency-scaled, single-token words excluded) has "slightly more explanatory power for model performance" is the basis for the recommended defaults. However:

The paper does not report numerical comparisons of $R^2$ or $\chi^2$ across conditions — the determination is based on "slightly more explanatory power" without quantification.
The difference is described as "numeric and the amount of variance is still quite small," meaning that even the "best" condition explains very little.
There is no cross-validation for condition selection — the condition that performs best on the full dataset is selected as the default, and this selection may capitalize on noise (though the overall weakness of all conditions makes overfitting unlikely to change the qualitative conclusion).

A more rigorous approach would have used cross-validation to select the best-performing condition on held-out language-task combinations, then evaluated that condition's predictive power on the held-out set. The paper's approach effectively uses the same data to select the condition and evaluate its performance, which could introduce optimism bias, though again the small absolute magnitude of all correlations makes this a minor concern.

What experiments would have strengthened the paper?

Several experiments are conspicuously absent given the paper's stated goals:

A baseline intrinsic metric for comparison. The paper shows that morphological alignment does not predict performance, but does not show whether it predicts performance better or worse than other intrinsic metrics. Does compression (fertility or CTC) explain more variance? Does Rényi efficiency? Without comparing MorphScore against alternatives, the conclusion that "morphological alignment alone does not measure dimensions of tokenizer quality relevant to model performance" (Section 5) is ambiguous — it could mean that morphological alignment is uniquely poor, or that all intrinsic metrics are similarly poor, and the data cannot distinguish these.
Analysis by morphological type (fusional vs. agglutinative), by POS, or by morphological feature. The paper explicitly includes POS and morphological feature annotations "to enable a broad range of future work" (Section 2.1) and notes that "aspects of our evaluation we do not discuss here" include "the ability to retrieve morphological alignment score broken down by POS" (Section 5.3). This is framed as future work, but it is precisely the kind of analysis that would transform the paper from a purely negative result into a nuanced finding about when morphological alignment matters (e.g., "morphological alignment of verbs predicts performance in agglutinative languages but not in fusional languages"). The paper's current conclusion — that morphological alignment does not predict performance in aggregate — leaves open the possibility that it does predict performance in specific linguistically motivated subsets, and this possibility is not tested.
Correlation at different vocabulary sizes. The five tokenizers have different vocabulary sizes (Llama2: 32K, BLOOM: 250K, XGLM: 250K, Llama3: 128K, Gemma3: 256K). Vocabulary size directly affects segmentation behavior (larger vocabularies → fewer segmentations → lower recall, potentially higher precision) and is a known predictor of compression efficiency. The paper does not control for vocabulary size in the correlation analysis or test whether the alignment-performance relationship differs across vocabulary size regimes.
Permutation or randomization test for the $R^2$ values. Given the very small $R^2$ values (0.005 for precision, 0.024 for recall), a natural question is whether these are statistically distinguishable from zero. The $\chi^2$ test for recall is significant ( $p < 0.001$ ), but with large samples, statistical significance and practical significance diverge. A permutation test randomizing the alignment scores across languages and recomputing $R^2$ would establish the null distribution and confirm that the observed 0.024 is genuinely above chance, even if practically negligible. This is not reported.
Causal manipulation of tokenization. The ultimate test of whether morphological alignment matters would be an experiment where the same base model architecture is trained with tokenizers that differ only in their morphological alignment properties (e.g., standard BPE vs. BPE with morphological constraints vs. character-level). If the morphologically aligned tokenizer produces better models, that would be causal evidence. If not, it would confirm the correlational finding. The paper does not attempt this, and doing so would be expensive (requiring pretraining multiple models from scratch), but the absence of any experimental manipulation means the paper can only draw correlational, not causal, conclusions about the relationship between tokenization and performance.

Does the paper fairly represent its own scope limitations?

The Limitations section is unusually thorough and honest for a conference paper. It acknowledges:

"European languages are over-represented in our sample" due to "systemic over-representations in the field and in resources like Universal Dependencies."
"the operationalization of morphological boundaries is coarse" and "mostly covers inflectional morphology and items that appear as single orthographic words. This undoubtedly misses many informative cases."
"We use only a small number of tasks to represent model performance" and "such evaluations are limited and generally do not represent most of the languages in our sample."
"Our sample of models was not very large, because many models do not provide critical information about their training data."
"We also chose to exclude instruction-tuned models."

These are honest, specific, and correctly identify the key threats to generalizability. The one notable omission from the Limitations is the reliance on UD treebank frequency counts for frequency scaling — these are acknowledged as limited in the Discussion ("for some languages the sample was very small," "many treebanks are created with data from one source") but not flagged as a limitation of the frequency-scaling analysis specifically. For languages with a few hundred sentences of news text, word frequency estimates are noisy and domain-biased, and the effect of this noise on the frequency-scaled alignment scores is not assessed.

6. Limitations and Trade-offs

Limitation 1: The Evaluation Framework Excludes Non-Concatenative and Isolating Morphology by Construction

The assumption or constraint. The gold morphological segmentations are derived by identifying the longest shared substring between a wordform and its lemma from Universal Dependencies, then checking whether concatenating the identified stem and affixes reconstructs the original wordform. The paper explicitly acknowledges that this filtering criterion removes essentially all examples of non-concatenative morphology:

"In the process of creating and filtering the datasets, despite having very large treebanks, there were not sufficient remaining items from any of the Semitic languages (Amharic, Arabic, and Hebrew) or most isolating languages (e.g. Chinese, Vietnamese, and Thai)" (Section 2.1).

The consequence is that the final 70-language evaluation set "only covers fusional and agglutinative languages" (Section 2.1). This is not an accidental omission — it is a fundamental constraint of the operationalization.

The consequence. The paper's central claim — that morphological alignment does not predict downstream model performance — is only demonstrated for languages where morphology is expressed through concatenative affixation (prefixes and suffixes on a stable stem). This excludes two major morphological types:

Semitic languages (Arabic, Hebrew, Amharic), where morphological distinctions are encoded through root-template patterns — a triconsonantal root carries semantic meaning, and vowels between the consonants encode grammatical distinctions (e.g., Arabic kataba "he wrote" vs. kātib "writer," illustrated in Figure 1). A tokenizer that splits at consonant-vowel boundaries in these languages would be making morphologically meaningful splits, but the current gold standard has no way to recognize this because there is no concatenative stem+affix boundary to compare against.
Isolating languages (Chinese, Vietnamese, Thai), where most words lack overt morphological markings entirely. These languages are excluded not because the metric fails on them, but because the dataset creation procedure finds "not sufficient affixation patterns" to produce evaluation items above the 100-item threshold.

This matters because the very language types where morphological tokenization is most debated — Semitic languages, where recent work has "sought solutions for effective tokenization" specifically for non-concatenative patterns (Gazit et al., 2025, cited in Section 2.1) — are systematically absent from the evaluation. Any practitioner working on Arabic or Hebrew tokenization cannot use this MorphScore variant to evaluate their tokenizer at all, and cannot extrapolate the paper's null finding to their setting because the morphological phenomena are qualitatively different.

What evidence exists in the paper. The paper is transparent about this exclusion (Section 2.1) and reports that 16 of the 86 initially-constructed datasets were filtered out for having fewer than 100 items (Section 2.1), though it does not report how many of those 16 were Semitic or isolating languages versus fusional/agglutinative languages with small treebanks. The Afrikaans example ("wees" → "is") illustrates the filtering logic in detail, showing that the concatenation recomposition check is intentionally conservative.

Mitigation status. Not addressed. The paper frames this as a scope limitation for future work: "Future work could focus on how to determine gold segmentations for both irregular items, such as the example from Afrikaans, and non-concatenative morphology" (Section 2.1). No alternative evaluation protocol or adaptation is proposed. The released datasets include the unfiltered items, so other researchers could attempt different gold segmentation strategies, but the paper's own conclusions are bounded to concatenative morphology in fusional and agglutinative languages.

Limitation 2: The Difficulty Estimation for Correlation Analysis Relies on Estimated, Not Observed, Training Data Proportions for Most Models

The assumption or constraint. A critical confound in any crosslinguistic performance analysis is that models see vastly different amounts of text in different languages during pretraining. The paper controls for this by including "estimated training data proportions from Hayase et al. (2024)" as a fixed effect in the linear mixed effects model (Section 4.1). However, the paper acknowledges that "the model developers do not release that information about the pre-training data" for most models — BLOOM and XGLM are the only ones that publicly report their training data language composition. For Llama2, Llama3, and Gemma3, the Hayase et al. estimates are the best available proxy, but they are inferred from the tokenizer's vocabulary distribution, not observed from the training data itself.

The inference method (Hayase et al., 2024) works by analyzing which Unicode characters and byte sequences appear frequently in the tokenizer's vocabulary and matching these patterns to languages — clever, but necessarily approximate. The paper treats these estimates as ground-truth covariates in the regression without accounting for estimation uncertainty.

The consequence. If the Hayase et al. estimates systematically overestimate or underestimate training data proportions in ways that correlate with morphological complexity, the control for training data proportion is incomplete. Consider a scenario where:

Morphologically complex languages (which tend to produce more unique wordforms and thus more token types) are systematically underestimated by the vocabulary-based inference method because the tokenizer's vocabulary includes fewer whole-word tokens for these languages (they get split into subwords instead).
The true proportion of training data in these languages is higher than estimated.
These languages also tend to have higher morphological alignment scores (because their morphology is more transparently concatenative).

In this scenario, the control variable would underestimate the training data effect and over-attribute performance variance to morphological alignment — or, conversely, if the bias goes the other direction, it could mask a real alignment-performance relationship. The paper provides no sensitivity analysis for this potential confounding.

This is not a hypothetical concern. The paper itself demonstrates that morphological complexity affects tokenization behavior (Section 3.1), and the Hayase et al. method's validity depends on exactly these tokenization patterns. If tokenization behavior and data proportions are entangled (which they are, by the nature of the inference method), then controlling for estimated data proportions while testing tokenization quality as a predictor creates a subtle circularity that is not addressed.

What evidence exists in the paper. The paper cites Hayase et al. (2024) as the source of estimates and notes that BLOOM and XGLM are the only models that report training data proportions (Section 4.1). No diagnostic is reported for the quality of these estimates — no comparison against the known proportions for BLOOM and XGLM to calibrate trust in the method, and no analysis of how sensitive the regression results are to using the estimates versus omitting the data proportion control entirely. The baseline model ( $\chi^2(2) = 25.67$ , $p < 0.001$ ) confirms that the control variables are significant predictors, but does not distinguish whether this significance comes from model parameters (which are known exactly) or from the estimated data proportions.

Mitigation status. Not addressed. The paper treats the estimated proportions as exact without caveats in the statistical modeling. A robustness check comparing results with and without the data proportion control, or testing whether the morphological alignment effect differs between BLOOM/XGLM (where proportions are known) and Llama2/3/Gemma3 (where they are estimated), would help bound this uncertainty. The paper does not conduct such checks.

Limitation 3: The "Downstream Performance" Signal Is Limited to a Narrow Set of Tasks, None of Which Directly Test Morphological Generalization

The assumption or constraint. The paper tests whether morphological alignment predicts model performance on seven benchmarks: XCOPA (causal reasoning, 11 languages), XNLI (natural language inference, 15 languages), SIB-200 (topic classification), and MultiBLiMP (subject-verb agreement, all 70 languages). The paper acknowledges the narrowness implicitly:

"We use only a small number of tasks to represent model performance. We used evaluations which were available for a wide variety of languages, but such evaluations are limited and generally do not represent most of the languages in our sample." (Limitations)

MultiBLiMP is the task that provides full language coverage (since it is derived from the same UD treebanks as MorphScore), but it tests only subject-verb agreement — one specific morphosyntactic phenomenon. XCOPA tests causal commonsense reasoning, XNLI tests textual entailment, and SIB-200 tests document topic classification. None of these tasks require the model to produce morphologically complex forms, to generalize morphological patterns to novel words, or to decompose morphologically complex words into their constituents for semantic interpretation.

The consequence. The tasks tested are precisely the wrong tasks for detecting a morphological tokenization effect. A model can perform well on topic classification (e.g., determining whether a news article is about sports or politics) without any morphological awareness whatsoever — distributional keyword features suffice. Causal reasoning (XCOPA) depends on world knowledge and logical structure, not on whether the model correctly decomposes "unhappiness" into [un + happy + ness] versus [unh + appi + ness]. Natural language inference (XNLI) can often be solved through lexical overlap heuristics and syntactic structure, with morphological details playing a secondary role.

This creates a detection problem: the paper concludes that morphological alignment does not predict performance, but it tests this on tasks where morphological alignment would not be expected to matter very much even if tokenization quality did affect morphological processing. Tasks that would provide a stronger test include:

Morphological inflection generation — given a lemma and morphological features, produce the correct inflected form (e.g., English: run + PAST → "ran"). Tokenization that cleanly separates stems from affixes should make this task easier because the model can learn to manipulate affixes independently of stems.
Morphological paradigm completion — given some forms of a word, predict other forms.
Syntactic tasks that require morphological cues — subject-verb agreement (MultiBLiMP partially covers this), case government, gender agreement across phrases, where incorrect tokenization might obscure the morphological signal needed for grammaticality judgments.
Low-resource machine translation into morphologically rich languages — where producing correct inflectional morphology is the primary difficulty.

The paper's null result might genuinely reflect that morphological alignment does not matter. But it might equally reflect that morphological alignment does matter specifically for morphological processing tasks, and the benchmarks tested simply do not measure morphological processing. The paper cannot distinguish these.

What evidence exists in the paper. MultiBLiMP is the closest task to morphological processing, and it is the one that provides full language coverage. However, the paper does not report the correlation between morphological alignment and MultiBLiMP performance separately — all results pool across tasks with task as a random intercept (Section 4.1). This means we cannot see whether morphological alignment predicts MultiBLiMP performance better than (or differently from) XCOPA, XNLI, or SIB-200 performance. If morphological alignment had a positive correlation with MultiBLiMP but negative correlations with the other tasks, the pooled analysis would wash out the signal. The color-coding in Figure 2 distinguishes tasks visually, but the reported statistics ( $R^2$ , $\chi^2$ ) are from the pooled model.

Mitigation status. The paper acknowledges the task limitation in the Limitations section but does not address it analytically. The suggestion that "combining morphological alignment with other intrinsic tokenizer evaluation metrics, like compression or Rényi efficiency, could potentially be more informative" (Section 5.2) implicitly recognizes that performance on general downstream benchmarks may have multiple determinants, but the paper does not test whether morphological alignment specifically predicts morphological task performance.

Limitation 4: The Analysis Is Correlational, Not Causal — Tokenizers Are Evaluated at Fixed Off-the-Shelf Snapshots

The assumption or constraint. All tested tokenizers are pre-trained tokenizers distributed with released models (Llama2, BLOOM, XGLM, Llama3, Gemma3). The paper measures their morphological alignment and correlates these scores with the downstream performance of the models that use them. This design answers: "among existing, independently-developed tokenizers, do those with higher morphological alignment tend to be paired with better-performing models?" It does not answer: "if you take a given model architecture and improve its tokenizer's morphological alignment, does the resulting model perform better?"

This distinction — correlation versus causation — is fundamental because the tokenizers differ on multiple dimensions simultaneously that are not controlled for:

Vocabulary size: Llama2 uses 32K tokens; BLOOM and XGLM use 250K; Llama3 uses 128K; Gemma3 uses 256K. Vocabulary size directly affects segmentation behavior (larger vocabularies → fewer splits → lower recall but potentially higher precision) and compression efficiency.
Training data composition: The tokenizers are trained on different text corpora with different language distributions, which affects which subword merges are learned and therefore which morphological boundaries are respected.
Pre-tokenization rules: Different models use different whitespace handling, Unicode normalization, and special token conventions that affect how words are segmented independent of morphological structure.
Model architecture and scale: The models range from 560M to 70B parameters, with different architectures (BLOOM uses a different architecture from Llama), training objectives, and training data mixtures.

A model with higher morphological alignment might also have a larger vocabulary, or more training data in morphologically rich languages, or better Unicode handling — and any observed performance differences could be due to these confounds rather than morphological alignment per se.

The consequence. The paper's negative finding could arise in at least two fundamentally different ways:

True null: Morphological alignment genuinely has no causal effect on model performance. Improving a tokenizer's alignment would not improve the model.
Confounded null: Morphological alignment does have a causal benefit, but the tokenizers that achieve higher alignment in this sample also have other properties (smaller vocabularies? less multilingual training data? different pre-tokenization rules?) that offset or reverse this benefit, producing a net zero or negative correlation.

The correlational design cannot distinguish these. The finding that the correlation is negative — higher morphological alignment is associated with slightly worse performance (Section 4.2) — is particularly suggestive of confounding: it seems implausible that genuinely better morphological segmentation would directly cause worse performance, but quite plausible that, in this sample of tokenizers, the tokenizer with the highest alignment (XGLM, per Table 1) has other properties that happen to produce worse downstream models.

A causal experiment — e.g., training multiple versions of the same model architecture from scratch with tokenizers that differ only in their morphological alignment properties (standard BPE vs. morphology-constrained BPE vs. character-level), then comparing performance — would resolve this. The paper does not conduct such an experiment, and acknowledges in the Limitations that "our focus is on large, autoregressive LMs" whose pretraining cost makes experimental tokenizer manipulation prohibitive.

What evidence exists in the paper. Table 1 reports which tokenizer ranks highest on precision (XGLM) and recall (Llama2) under different scoring conditions. The paper does not report vocabulary sizes, tokenizer training data, or pre-tokenization rules for the five tokenizers, nor does it control for these in the regression. The only tokenizer-level control is the model parameter count (Section 4.1), which captures model scale but not tokenizer-specific properties. The significant but small correlation between frequency and tokenization behavior ( $\rho = 0.119$ for morphological segmentation, $\rho = -0.108$ for token count; Section 3.1) demonstrates that tokenizer behavior and word frequency are entangled, confirming that tokenizer properties are not orthogonal to the linguistic variables of interest.

Mitigation status. The paper acknowledges the small model sample in the Limitations ("Our sample of models was not very large, because many models do not provide critical information about their training data") and notes that BLOOM and XGLM are the only models reporting training data proportions. However, it frames this as a data transparency issue rather than a causal identification issue, and does not discuss what experimental design would be needed to establish causality. The explicit scoping to large autoregressive LMs is a practical acknowledgment of feasibility constraints but does not mitigate the inferential limitation.

Limitation 5: Frequency Estimates for Frequency Scaling Are Noisy, Domain-Specific, and Derived from Small Samples for Many Languages

The assumption or constraint. When the frequency scaling parameter is enabled, each word's contribution to the morphological alignment score is weighted by its frequency in the UD treebank used to create the evaluation dataset (Section 3.1). The paper selects the frequency-scaled, single-token-excluded condition (True_True) as the recommended default because it has "slightly more explanatory power for model performance" (Section 4.2).

However, the treebank frequency estimates vary dramatically in quality across languages:

"our frequency metrics came only from the treebanks we used to create our datasets, meaning for some languages the sample was very small. Additionally, many treebanks are created with data from one source, e.g. news articles." (Section 5.1)

The treebank sizes in Appendix A (Table 2) range from just over 100 items to tens of thousands. For a language with 150 evaluation items, the underlying treebank has perhaps a few hundred sentences — a sample from which word frequency estimates are extremely noisy. A word that appears three times in a small news treebank may be genuinely common in the language or may be an artifact of the specific news stories covered; the frequency weighting cannot distinguish these.

The consequence. The frequency-weighted alignment scores for low-resource languages (which are precisely the languages where morphological tokenization is often most debated) are computed with unreliable frequency weights. A single high-frequency domain-specific word (e.g., a politician's name in a news treebank) could receive disproportionate weight in the aggregate alignment score, making the score reflect tokenization quality on that one word rather than on the language as a whole. Across the 70-language sample, the quality of frequency estimates covaries with language resource level — high-resource European languages (English, German, French) have large treebanks with diverse genres, while low-resource languages (Buryat, Veps, Erzya) have small, single-domain treebanks. This means frequency scaling systematically changes the nature of the measurement across languages in ways that correlate with resource level, potentially introducing a confound into the crosslinguistic correlation analysis.

The paper's recommendation of frequency-scaled scores as the default (Section 4.2) means that future users of MorphScore will reproduce this differential measurement quality when evaluating tokenizers across languages — but the paper provides no guidance on whether frequency scaling should be enabled or disabled for low-resource languages specifically, or what minimum treebank size is needed for reliable frequency estimates.

Additionally, the paper notes that wordform frequency may not be the right frequency unit:

"Another possible change would be to use lemma frequency instead of wordform frequency. Particularly for agglutinative languages, e.g. Turkish, individual wordforms tend to be lower frequency. Any given verb, for example, can have thousands of different forms." (Section 5.1)

The current frequency scaling uses wordform frequency — how often a specific inflected form appears. In agglutinative languages, most individual wordforms are rare (because there are so many possible inflectional combinations), even though the underlying lemma is common. Weighting by wordform frequency may therefore systematically underweight morphologically complex words in agglutinative languages (they are frequent as a morphological type but each specific form is uncommon), while overweighting simple wordforms in fusional languages (where each form appears more frequently because there are fewer alternatives). This interacts with the paper's language coverage — which "only covers fusional and agglutinative languages" (Section 2.1) — potentially introducing a systematic difference in what the frequency-scaled metric measures in these two morphological types.

What evidence exists in the paper. The paper reports the two correlations between frequency and tokenization behavior ( $\rho = 0.119$ for morphological segmentation likelihood, $\rho = -0.108$ for token count; Section 3.1) but does not break these down by treebank size, domain, or morphological type. The acknowledged limitations of the frequency estimates are in the Discussion (Section 5.1), not in the Limitations section, and are framed as motivation for future work ("in the future, word frequency could be calculated using larger corpora from a wider range of domains") rather than as a threat to the validity of the current findings.

Mitigation status. Partially addressed through recommendation, not analysis. The paper rejects the False_False condition (no frequency scaling, single-token words included) and the True_False condition (frequency scaling with single-token words included) in favor of True_True, but the selection is based on marginal predictive power on the full dataset without testing whether frequency scaling's benefit varies with treebank size. The suggestion to use larger corpora and lemma frequency (Section 5.1) is presented as future work. For a practitioner deploying MorphScore on a low-resource language with a small treebank, the paper provides no guidance on whether the recommended True_True settings remain reliable or whether the unscaled False_True condition would be more appropriate.

Limitation 6: Gold Segmentations Capture Only the Simplest Concatenative Cases — at Most One Prefix and One Suffix — Even Within the Included Languages

The assumption or constraint. The lemma-wordform alignment procedure identifies the longest shared substring as the stem, then classifies preceding characters as a prefix and following characters as a suffix. The gold segmentation therefore has "at least two morphemes (the stem and an affix) and at most three morphemes (a prefix, stem, and suffix)" (Section 2.1).

This means that within the included fusional and agglutinative languages, words with multiple affixes are systematically simplified. An agglutinative word like Turkish "evlerimizden" ("from our houses") has the morphological structure [ev + ler + imiz + den] (stem + plural + possessive + ablative case) — four morphemes with three boundaries. The alignment procedure would identify the lemma "ev" as the stem and "lerimizden" as a single suffix, producing a gold segmentation [ev + lerimizden] with exactly one boundary. A tokenizer that correctly segments this word into [ev + ler + imiz + den] would be credited with identifying only one of three gold boundaries (boundary recall = 1/3), even though its segmentation is morphologically perfect.

The paper acknowledges this implicitly: "the operationalization of morphological boundaries is coarse. We aim mainly to capture the most clear-cut cases. This means that we mostly cover inflectional morphology and items that appear as single orthographic words. This undoubtedly misses many informative cases" (Limitations).

The consequence. The morphological alignment metric systematically undervalues correct multi-morpheme tokenization in agglutinative languages — precisely the languages where morphological alignment is most often argued to matter. A tokenizer that correctly identifies all four Turkish morphemes is scored the same as one that finds only the stem-suffix boundary and misses the internal suffix boundaries. Worse, a tokenizer that segments at arbitrary positions within the suffix — e.g., [ev + lerim + izden] — could achieve the same or better boundary precision/recall as the morphologically correct segmentation if it happens to place a boundary closer to the single gold boundary position.

This interacts perversely with the paper's language coverage and its conclusions. Agglutinative languages (Turkish, Finnish, Hungarian, etc.) are included in the 70-language sample and are the languages where morphological tokenization is theoretically most beneficial — each word carries substantial grammatical information encoded in its affix sequence, and a tokenizer that isolates these affixes would give the model direct access to that grammatical signal. But the gold standard gives these languages essentially the same evaluation structure as fusional languages (single-boundary words), failing to distinguish tokenizers that capture the rich multi-affix structure from those that do not. The null finding — that morphological alignment does not predict performance — may partly reflect that the metric cannot detect genuine morphological alignment in the very languages where it should matter most.

What evidence exists in the paper. The method description in Section 2.1 explicitly says "the gold segmentation will have at least two morphemes (the stem and an affix) and at most three morphemes (a prefix, stem, and suffix)." The Limitations section acknowledges that the operationalization "mostly covers inflectional morphology and items that appear as single orthographic words." No analysis is reported on how many words in agglutinative languages actually have more than three morphemes, how often the gold segmentation collapses multiple real morpheme boundaries into a single boundary, or whether the alignment-performance correlation differs between languages where multi-morpheme words are common versus rare. The recommendation to use UD morphological feature annotations for future analyses (Section 5.3) does not address the fundamental mismatch between the gold segmentation resolution and morphological reality.

Mitigation status. Not addressed. The paper treats the at-most-three-morpheme limitation as inherent to the lemma-alignment method and does not propose an alternative (e.g., using UD's own morphological feature annotations to identify additional morpheme boundaries within the suffix string). The released datasets include the full morphological feature annotations from UD (Section 2.1), which could in principle be used to identify additional boundaries (e.g., plural "-ler" and possessive "-imiz" and ablative "-den" could be segmented using the UD feature tags), but the paper's scoring framework uses only the coarse lemma-derived boundaries and does not provide this finer-grained evaluation.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new method that practitioners will adopt; rather, it serves as a large-scale, rigorous null result that closes a door in the tokenizer evaluation literature. By demonstrating that morphological alignment—measured under multiple reasonable operationalizations, across 70 languages, five model families, and seven tasks—explains at most ~2.4% of variance in downstream model performance (with a negative direction), the paper establishes that morphological alignment alone is not a useful intrinsic evaluation metric for tokenizer quality in the context of large, pretrained multilingual autoregressive language models. This is not an incremental refinement of a known finding—it is a direct challenge to a substantial body of prior work (at least 13 papers cited in Section 1) that argued for or assumed a positive relationship between morphologically aligned tokenization and model performance.

The conceptual shift is best understood as a recalibration of the field's assumptions about what makes tokenization "good." Prior to this work, the debate was largely: does morphological alignment help or not?—with conflicting evidence from different studies on different languages with different metrics. This paper does not merely add another data point to that debate; it restructures the question by showing that the answer is neither "yes" nor "no" universally, but rather "not in the way we've been measuring it." The finding that recall (boundary coverage) has slightly more explanatory power than precision ( $R^2 = 0.024$ vs. $R^2 = 0.005$ ; Section 4.2) but both are practically negligible shifts the conversation from "is morphological alignment good?" to "the simple boundary-alignment definition of morphological alignment—under any metric, with any parameter setting—does not capture what matters about tokenizer quality for these models on these tasks."

This reframing has several concrete consequences for the research landscape:

Morphology-aware tokenizer design becomes a harder sell. The extensive literature advocating for tokenizers that respect morphological boundaries (Bostrom & Durrett, 2020; Hofmann et al., 2021; Toraman et al., 2023; Asgari et al., 2025; and others) now faces a higher evidentiary bar. A paper proposing a new morphology-aware tokenization algorithm in 2025 cannot simply claim that morphological alignment is generally beneficial and cite the pre-2024 consensus. It must either (a) demonstrate a causal benefit through controlled experimentation (training models from scratch with and without the morphological constraint), (b) show that the benefit appears specifically on tasks that directly test morphological processing—tasks this paper did not evaluate, or (c) argue that the benefit operates through a mechanism not captured by the alignment-as-boundary-matching framework (e.g., improved sample efficiency during pretraining, better crosslingual transfer, or benefits that emerge only at specific scales or with specific architectures). The burden of proof has shifted.

Intrinsic tokenizer evaluation reorients toward combined metrics. The paper's suggestion that "combining morphological alignment with other intrinsic tokenizer evaluation metrics, like compression or Rényi efficiency, could potentially be more informative" (Section 5.2) is not throwaway speculation—it is the natural implication of the null result. If morphological alignment alone does not predict performance, but tokenization clearly does affect performance (as established by Ahia et al., 2023; Petrov et al., 2023; Ali et al., 2024), then the relevant dimensions of tokenizer quality must be multidimensional. The paper's framework—parameterized, configurable, releasing all scores for all conditions—makes it straightforward to test combined metrics: does a weighted combination of morphological alignment, compression (CTC), and Rényi efficiency predict performance better than any single metric? This becomes the obvious next step.

The oversegmentation confound is now visible and must be addressed in future work. By decomposing accuracy into precision and recall and showing that Llama2's inflated recall is "likely due to pervasive oversegmentations" (Section 3.3, Table 1), the paper exposes a measurement error that may have affected prior positive findings. Any future study claiming that morphological alignment benefits performance must use precision-recall decomposition (or an equivalent) rather than accuracy, to avoid conflating genuine morphological alignment with aggressive tokenization that happens to hit some morpheme boundaries. This is a methodological standard that the paper establishes by example.

The difficulty-estimation burden for tokenizer evaluation is clarified. Unlike model training, where scaling laws provide a framework for compute allocation, tokenizer evaluation has no equivalent. This paper does not provide that framework, but it clarifies what would be needed: intrinsic metrics must be validated against extrinsic outcomes, and the validation must control for the known confounds (model size, training data) that dominate performance variance. The fact that model size and training data proportion alone explain significantly more variance than the intercept ( $\chi^2(2) = 25.67$ , $p < 0.001$ ; Section 4.2) while morphological alignment adds at most $\sim$ 2.4% $R^2$ demonstrates the scale of the problem—most of what determines crosslinguistic model performance is not tokenization quality, making it genuinely difficult to isolate tokenization effects with correlational designs.

Research directions that become less attractive:

New single-axis morphological alignment metrics without validation. The paper tested four scoring configurations and two metric types; all produced small negative correlations with performance. A new metric that is a minor variant of morphological boundary alignment is unlikely to succeed where these failed, unless it is validated against a fundamentally different set of tasks (morphological processing tasks rather than general NLP benchmarks) or shown to predict training dynamics (sample efficiency, convergence speed) rather than final task accuracy.
Studies claiming morphological alignment benefits based on a handful of languages. The paper's 70-language sample—while still skewed toward European languages—is substantially broader than prior work. A study testing morphological alignment on 5–10 languages, particularly if they are all agglutinative or all fusional, now faces the criticism that the result may be a language-type artifact, since this paper's null result holds across both types (within the concatenative-morphology scope).
Using accuracy as a morphological alignment metric without addressing the oversegmentation confound. The paper demonstrates that "tokenizing words into more tokens is strongly correlated with morphological alignment as measured with accuracy" (Section 2.3). This is not a subtle bias—it is a structural property of accuracy-based metrics that can reverse conclusions. Future work using accuracy must justify why precision-recall decomposition is not appropriate, or risk having its results attributed to oversegmentation artifacts.

Research directions that become more attractive:

Causal experiments manipulating tokenizer properties. The paper's correlational design cannot establish whether morphological alignment would help if properly optimized. The natural follow-up is to train the same model architecture from scratch with tokenizers that differ only in morphological alignment (e.g., standard BPE vs. BPE with morphological constraints vs. character-level), controlling for vocabulary size and training data, and measure performance on both general and morphological tasks. The paper's null result motivates this expensive experiment by showing that the correlational approach has reached diminishing returns.
Tokenizer evaluation focused on compression, efficiency, and fairness rather than linguistic alignment. Since morphological alignment does not predict performance, the factors that do drive performance differences across tokenizers—compression efficiency, inference cost, crosslinguistic fairness—become higher-priority targets for intrinsic metric development. The paper implicitly points toward this by contextualizing its work against compression metrics (fertility, CTC) and Rényi efficiency (Section 1).
Fine-grained analysis of when morphological alignment matters (by POS, by morphological feature, by task type). The paper releases datasets with POS tags and morphological features specifically to enable this (Section 2.1) but does not conduct the analysis. A study showing that morphological alignment predicts performance on verb inflection tasks in agglutinative languages but not on topic classification in fusional languages would both explain the null aggregate result and provide actionable guidance for tokenizer design in specific settings.

Follow-Up Research This Work Enables

1. Per-POS and per-morphological-feature analysis of alignment-performance correlations using the released datasets. The paper explicitly created evaluation datasets with part-of-speech tags and morphological feature annotations from Universal Dependencies (Section 2.1) but analyzed only aggregate alignment scores pooled across all words, all POS, and all morphological features. The released datasets on Hugging Face make it straightforward to compute morphological alignment scores separately for, e.g., verbs versus nouns, or for words with case marking versus those without, and correlate these with performance on tasks that specifically require processing those linguistic categories. A strong follow-up would test whether morphological alignment of verbs (which carry tense, aspect, mood, and agreement morphology critical for sentence interpretation) predicts performance on MultiBLiMP subject-verb agreement better than morphological alignment of nouns, using the UD POS tags already in the dataset. If alignment of verbs but not nouns predicts agreement performance, this would partially rescue the morphological alignment hypothesis by showing it operates in linguistically specific ways that aggregate analysis washes out. Conversely, if per-POS analysis also yields null results, the case against morphological alignment as a useful intrinsic metric becomes substantially stronger.

2. Training models from scratch with tokenizers that differ only in morphological alignment to establish causal evidence. The paper's central limitation is its correlational design—it evaluates off-the-shelf tokenizers that differ on multiple dimensions simultaneously (vocabulary size, training data, pre-tokenization rules). A causal experiment would fix the model architecture (e.g., a 125M-parameter GPT-style transformer), the training data (a fixed multilingual corpus with known language proportions), and the vocabulary size, then vary only the tokenizer training procedure: (a) standard BPE with no morphological constraints, (b) BPE with morphological boundary constraints (e.g., preventing merges that cross gold morpheme boundaries during BPE training, following approaches like MorphBPE or BPE-knockout), and (c) a character-level or byte-level baseline. Train all models from scratch on the same data, evaluate on a battery including both general NLP benchmarks (XNLI, XCOPA for continuity with this paper) and morphological processing tasks (inflection generation, paradigm completion, agreement judgment). If (b) outperforms (a) on morphological tasks but not general tasks, the causal benefit of morphological alignment is confirmed but scoped to morphological processing. If (b) shows no benefit even on morphological tasks, the morphological alignment hypothesis is effectively disproven for this class of models. This experiment is expensive but feasible at small scale (125M parameters, single-digit GPU-weeks) and would provide the causal evidence this paper cannot. The paper's 70-language evaluation datasets provide the gold segmentations needed to train the morphologically constrained BPE tokenizer across many languages.

3. Combining morphological alignment, compression, and Rényi efficiency into a multidimensional intrinsic metric and testing whether the combination predicts performance. Section 5.2 suggests that "combining morphological alignment with other intrinsic tokenizer evaluation metrics, like compression or Rényi efficiency, could potentially be more informative." This is a directly testable hypothesis: compute MorphScore (under the recommended True_True settings), corpus token count (CTC), and Rényi efficiency for each of the five tokenizers on each of the 70 languages, then fit a linear mixed effects model predicting downstream performance from all three metrics simultaneously (plus the standard controls for model size and training data proportion). Does the combined model explain substantially more variance than any single metric alone? Does morphological alignment contribute predictive power when compression and Rényi efficiency are already in the model, or is its (already negligible) contribution entirely redundant? The paper's released MorphScore data (Appendix B) and the public availability of CTC (via the tokenization-scorer package) and Rényi efficiency (via Zouhar et al.'s released code) make this combination analysis computationally trivial—it requires only running existing tools on existing tokenizers and merging the resulting scores. A finding that the combination explains, say, 10% of performance variance (versus 2.4% for morphological alignment alone) would be a significant advance for intrinsic tokenizer evaluation; a finding that the combination still explains very little would indicate that intrinsic evaluation of tokenizers through these metrics is fundamentally limited, and the field needs entirely different approaches.

4. Replicating the analysis on encoder-only and encoder-decoder models. The paper explicitly scopes its analysis to "large, autoregressive LMs" and acknowledges excluding "encoder models or those trained with masked language modeling" (Limitations). This is a substantive scope limitation because encoder models (BERT, XLM-R) and encoder-decoder models (mT5, NLLB) have different training objectives that may interact differently with tokenization quality. A masked language model that must predict tokens based on bidirectional context might benefit more from morphologically transparent tokenization (because morphological regularities are more accessible when the model can attend to both the stem and the affix context) than an autoregressive model that only sees left context. A replication using XLM-R (which has a publicly available tokenizer and reported performance on XNLI, XCOPA, and UD parsing tasks for many languages) would test whether the null result is specific to autoregressive models or generalizes across architectures. Given the paper's strong emphasis on expandability (configurable scoring, released datasets), such a replication could be conducted by a different research group using the released MorphScore datasets and evaluation code, making it a natural community validation effort.

5. Testing whether morphological alignment predicts pretraining sample efficiency rather than final downstream performance. The paper tests whether morphological alignment correlates with final downstream task accuracy after full pretraining. An alternative hypothesis—not tested and not discussed—is that morphological alignment matters for how quickly a model learns, not for how well it performs after convergence. A morphologically aligned tokenizer might allow the model to learn morphological patterns from fewer examples, achieving the same performance with less pretraining compute, even if the final converged performance is similar. This would manifest as a steeper learning curve for morphological tasks during pretraining but identical asymptote. Testing this requires pretraining models from scratch (as in follow-up #2) and tracking validation performance on morphological probe tasks throughout training. If morphological alignment provides a 2× speedup in learning morphological patterns but no final accuracy benefit, that would reconcile the paper's null finding on final performance with the intuition (and some prior evidence) that morphological alignment helps—it helps with efficiency, not capability. This hypothesis is particularly relevant given the paper's framing around "enabling tokenizer selection before model training, leading to significant computational and financial savings" (Section 1). If morphological alignment predicts training efficiency but not final quality, it would still be practically valuable for the use case the paper motivates, just through a different mechanism than the one tested.

6. Developing gold segmentation methods for non-concatenative and isolating languages to extend MorphScore coverage. The paper explicitly identifies the exclusion of Semitic languages (Arabic, Hebrew, Amharic) and isolating languages (Chinese, Vietnamese, Thai) as a limitation arising from the concatenative-only gold segmentation procedure (Section 2.1). For Semitic languages, recent work by Gazit et al. (2025) on tokenization for non-concatenative morphology provides a starting point: root-template patterns can be segmented by identifying the triconsonantal root and the vowel pattern as separate morphological units, even though they interleave rather than concatenate. A gold segmentation standard for Arabic that separates "kataba" into [k-t-b + a_a_a] (root + pattern) rather than attempting concatenative segmentation would provide evaluation items for languages that are currently excluded. For isolating languages, the challenge is different—there are few multi-morpheme words to evaluate—but evaluating whether tokenizers appropriately handle compounding (a productive morphological process in Chinese and Vietnamese) or whether they segment at semantically meaningful boundaries within compounds could provide an alternative evaluation target. Developing these standards would expand MorphScore from its current "fusional and agglutinative only" scope to near-universal language coverage, directly addressing one of the paper's most significant limitations. The released framework (configurable scoring, sentential context for superword tokenizers) is designed to accommodate new gold segmentation standards without architectural changes—only new evaluation datasets are needed.

Practical Applications and Downstream Use Cases

1. Tokenizer selection for multilingual model training can deprioritize morphological alignment. For practitioners deciding between candidate tokenizers (e.g., different vocabulary sizes, different BPE training data mixtures, or different tokenization algorithms) before committing to an expensive multilingual model training run, this paper provides negative guidance: do not use morphological alignment as a selection criterion. Across 70 languages, five tokenizers, and four scoring configurations, morphological alignment explains at most ~2.4% of variance in downstream performance and correlates negatively. A practitioner who selects Tokenizer A over Tokenizer B because A has higher MorphScore is, based on this evidence, making a decision uncorrelated with—or slightly negatively correlated with—the eventual model's performance on XNLI, XCOPA, SIB-200, and MultiBLiMP. This negative guidance has direct computational and financial implications: time spent computing and comparing morphological alignment scores is time not spent on other selection criteria (vocabulary size experiments, compression benchmarks, inference latency profiling) that may better predict model quality. The paper does not say which criteria to use—it only rules out one prominent candidate.

2. Benchmarking tokenizer fairness across languages should not use morphological alignment as a diagnostic. Tokenization has been shown to introduce systematic unfairness: languages with non-Latin scripts or complex morphology require more tokens to encode the same content, increasing inference cost and potentially degrading performance (Ahia et al., 2023; Petrov et al., 2023). A natural diagnostic for this unfairness would be to measure whether a tokenizer "respects" the morphological structure of different languages equally—if a tokenizer segments English words morphologically but fragments Turkish words arbitrarily, that would be evidence of unfairness. This paper's null result suggests that such a diagnostic would not capture the dimension of unfairness that actually affects downstream performance. The negative correlation between alignment and performance (Section 4.2) is particularly cautionary: a tokenizer with more uniform morphological alignment across languages might actually produce worse crosslinguistic performance equity than one with uneven alignment, because the factors that produce high alignment (e.g., aggressive segmentation in non-Latin scripts) may themselves introduce other forms of unfairness. Practitioners auditing tokenizers for fairness should focus on compression equity (tokens per semantic unit across languages), inference cost equity, and downstream performance equity—metrics with established links to practical outcomes—rather than morphological alignment equity.

3. MorphScore's released datasets enable low-cost morphological probing of pretrained models. While the paper shows that MorphScore does not predict downstream task performance, the evaluation datasets themselves—70 languages of words with gold morphological segmentations, sentential context, POS tags, and morphological features—are a valuable resource for a different purpose: analyzing whether pretrained models internally represent morphological structure, independent of whether their tokenizers segment morphologically. A researcher can take a pretrained model (e.g., Llama3 8B), feed it the sentences from the MorphScore datasets, extract hidden-state representations for the tokens corresponding to each evaluation word, and test whether the model's internal representations distinguish morphological features (e.g., does the representation of a verb encode its tense, even if the tokenizer fragmented the tense suffix?). This is a morphological probing task, not a tokenizer evaluation task, and it uses the MorphScore datasets' gold annotations—POS tags and morphological features that were included but not analyzed in the paper (Section 2.1)—to train lightweight probes on model representations. The null result on tokenization actually makes this application more interesting: if a model's tokenizer segments uninformatively but its internal representations nonetheless encode morphological structure, that suggests the model learns morphology despite suboptimal tokenization, which would be evidence for robustness. If the model fails to encode morphological structure, the failure could be attributed either to tokenization or to training data limitations—and comparing models with different tokenizers but similar training data (e.g., Llama2 vs. Llama3) would help disambiguate. This is an immediate, low-cost use of the released resources that does not depend on the paper's negative conclusion about intrinsic evaluation.

4. Guiding vocabulary size selection for low-resource morphologically rich languages. The paper's empirical findings about the relationship between frequency and tokenization behavior—more frequent words are more likely to be stored as single tokens ( $\rho = -0.108$ for token count vs. frequency; Section 3.1) and more likely to be morphemically segmented when they are split ( $\rho = 0.119$ ; Section 3.1)—provide practical guidance for vocabulary size selection in low-resource languages, even though the paper does not frame it as such. For a morphologically rich language with a small text corpus (e.g., a language with only a few million words of training data), the BPE vocabulary size determines a tradeoff: a larger vocabulary stores more common words as single tokens (good for compression, reduces inference cost) but means rare words must be composed from subwords that may not align with morphological boundaries. The paper's frequency-tokenization correlations suggest that for low-resource languages, this tradeoff is particularly sharp: the small corpus means few words reach the frequency threshold to be stored whole, so most words are segmented, and the quality of those segmentations (whether they align with morphology) depends on which subword merges the BPE training data supports. A practitioner could use the MorphScore evaluation datasets—which include gold segmentations and frequency information—to sweep vocabulary sizes for their language of interest, measuring both compression and morphological alignment at each size, and select the vocabulary size that achieves acceptable compression without catastrophic misalignment on the (many) words that must be segmented. The paper's recommended default settings (True_True: frequency-scaled, single-token words excluded) provide the evaluation recipe.

When to Prefer This Method

The paper does not position MorphScore against named alternative intrinsic tokenizer evaluation methods in a way that articulates clear tradeoffs or decision rules. It tests whether MorphScore predicts performance, finds that it does not, and suggests (Section 5.2) that combining it with other metrics may be informative—but it does not claim that MorphScore should be preferred over compression-based metrics, Rényi efficiency, or any other specific alternative for any identifiable use case. The paper's primary contribution is a negative result about morphological alignment's predictive validity, not a new method that practitioners should adopt over existing alternatives. A forced "prefer A when, prefer B when" matrix would be speculative and unsupported by the paper's own analyses and claims.