BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization

🎯 Pitch

Tokenizers like GPT-4o smuggled 874 "Frankenstein" tokens—partial multi-character merges—into their vocabulary, a cascade defect born from greedy byte-pair encoding. The authors kill this bug with a constrained merge rule that simply forbids crossing character boundaries, simultaneously boosting compression without needing a complex regex pretokenizer.

1. Executive Summary

This paper introduces SCRIPT-BPE (Script Category Representation in Pre-Tokenization), a novel tokenization framework that replaces UTF-8 byte conversion with a structured two-token encoding based on Unicode script and supercategory properties, coupled with a rule-based pretokenization strategy that avoids the fragility and language-specific biases of regular expressions. The approach also includes a constrained BPE merging strategy—enforcing that merges respect character boundaries so that tokens never mix partial and full characters, applicable to both SCRIPT-BPE and standard byte-level BPE (e.g., prohibiting a space from merging with only the first byte of a subsequent multi-byte character). Across monolingual tokenizers for 12 languages and a 256k-vocabulary multilingual tokenizer trained on CulturaX, constrained merging universally eliminates partial-UTF-8-sequence tokens while almost always improving compression, and SCRIPT-BPE achieves competitive compression relative to GPT-4 and GPT-4o tokenizers—substantially outperforming them on scripts penalized by diacritic-splitting regex patterns like Thai, Hindi, and Punjabi—establishing that a simpler, script-aware encoding can match or exceed widely deployed tokenizers in compression while eliminating encoding-based penalties for non-Latin scripts.

2. Context and Motivation

The Core Problem: Tokenization Is Broken for Multilingual Text

The fundamental issue this paper addresses is that modern language models use tokenizers that systematically disadvantage non-Western scripts. This isn't a minor inefficiency — it's a design-level bias baked into the most widely deployed tokenization pipelines, and it affects everything from how much text a model can process to how well it handles languages spoken by billions of people.

The problem manifests across three interconnected layers of the tokenization stack, each of which the paper identifies as having distinct failures:

Layer 1: The byte encoding introduces differential costs across scripts. BPE tokenizers typically operate on UTF-8 encoded bytes rather than on characters directly. UTF-8 is a variable-length encoding: basic Latin characters (the ASCII range) take 1 byte each, characters from scripts like Greek or Cyrillic take 2 bytes, and characters from scripts like Chinese, Japanese, Korean (CJK), Devanagari, or Thai take 3–4 bytes. When a BPE tokenizer treats these bytes as its atomic units, a single Chinese character consumes 3 initial tokens before any training, while a single Latin letter consumes only 1. This is what Arnett et al. (2024) termed the "byte premium effect" — a structural encoding tax on non-Latin scripts that exists before the tokenizer even sees any training data. As illustrated in Figure 1A, the word "你好" (Chinese for "hello") requires 6 byte-tokens in UTF-8 encoding, while a 6-character English word like "hello!" requires only 6 byte-tokens — the Chinese text consumes the same raw token budget despite containing only 2 meaningful units versus 6 for English. The consequence is that for a fixed context window, models can process substantially less text written in non-Latin scripts.

Layer 2: UTF-8 byte sequences create semantically meaningless tokens that cross character boundaries. The variable-length nature of UTF-8 means that individual characters span multiple bytes, but standard BPE training has no mechanism to respect character boundaries. The greedy merging process can — and routinely does — create tokens that combine the last byte(s) of one character with the first byte(s) of the next. Figure 1C illustrates a concrete cascading failure: an early merge between a space character (a full character in the ASCII range) and the first byte (<0xE0>) of a subsequent multi-byte Thai character creates a token representing partial UTF-8 sequences. Once this merge is established in the vocabulary, subsequent merges can extend these partial sequences, producing tokens that represent a mix of full and partial characters — tokens that have no linguistic interpretation, risk producing invalid byte sequences if they appear in the wrong context, and as Land and Bartolo (2024) showed, can become severely under-trained because they appear much less frequently in training data than the model's vocabulary size would suggest. The paper quantifies this concretely: the GPT-4o tokenizer contains 874 tokens representing a mix of full and partial characters, a direct artifact of unconstrained byte-level BPE merging. One such token is <0x95>\n\n — a byte from some character's encoding combined with two newline characters, a token with no coherent meaning.

The cascade mechanism is particularly insidious: because BPE is greedy, an early problematic merge can create a byte sequence that then participates in further merges, propagating the problem through the vocabulary. The authors demonstrate this vividly with the Thai dataset, where the standard o200k regex pretokenizer produces 42,831 tokens with full-and-partial character mixing (Table 2) — a massive failure mode driven by a small number of initial merges involving common characters and their leading UTF-8 bytes (<0xE0><0xB8>).

Layer 3: Pretokenization via regular expressions is fragile, language-biased, and difficult to analyze. Before BPE merges even begin, most tokenizers apply a pretokenization step that splits the input text into "pretokens" — smaller units that set the boundaries across which BPE merges are prohibited. The dominant approach, inherited from GPT-2 and refined through GPT-4 (cl100k) and GPT-4o (o200k), uses hand-crafted regular expressions. These regex patterns encode specific heuristics: split on contractions (e.g., 'm in "I'm", 's in "woman's"), handle whitespace in particular ways, and split digits into groups of up to three. The problem is that these heuristics are designed around English and fail in systematic ways when applied to other languages.

The paper provides a compelling example of the contraction-splitting failure: the regex rule that preserves English contractions like 'm (I'm) and 'd (I'd) also splits words in Scottish Gaelic (s'mhath — "(it's) good"), Mi'kmaq (m'sit — "all"), Fulfulde (n'di — "eat"), and Hausa ('dan — "son") at unnatural points. What is a helpful structural preservation for English becomes a destructive word-internal split for these languages.

Furthermore, common regex pretokenizers split diacritics from their base characters for scripts like Tamil, Sinhala, and Hindi (Velayuthan and Sarveswaran, 2025). For these languages, the visual representation of a character often combines a base letter with one or more combining marks, but the regex rules treat these components as separate tokens, breaking apart what is semantically a single unit. The paper's compression results in Table 4 confirm the severity: for Thai, Hindi, and Punjabi, the cl100k regex pattern produces compression ratios that are substantially worse than SCRIPT-BPE, precisely because the regex splits words at diacritic boundaries in ways that are linguistically inappropriate.

Even for English, regex-based pretokenization has documented fragility. Schmidt et al. (2025) showed that the original GPT pretokenizer failed to handle the difference between straight (') and curly (') apostrophes, treating them inconsistently. While a fix was proposed, the deeper point stands: regex-based pretokenizers accumulate edge cases that are hard to discover and harder to fix systematically. They are fundamentally opaque — when a tokenizer produces a surprising segmentation, tracing the behavior back to a specific regex rule and understanding its interaction with other rules is non-trivial.

Why This Matters: Real-World Impact and Theoretical Significance

The practical consequences of these tokenization failures extend well beyond academic concern:

For model deployment budgets: The byte premium effect means that for a model with a fixed context window (e.g., 8K tokens), the same user prompt written in Chinese or Arabic takes up proportionally more of the context budget than the equivalent English prompt. This either increases costs (more tokens = more inference compute) or degrades quality (less available context for the actual reasoning task). In high-volume production deployments, these per-token cost differences compound dramatically.

For language equity: Models are trained on datasets where English and Latin-script languages dominate. The tokenization system compounds this imbalance: not only do non-Latin languages have less training data, they also get fewer "meaningful" tokens per unit of training data because each character consumes more byte-tokens. This creates a feedback loop where models perform worse on non-Latin languages in part because the tokenizer allocates them less effective representational bandwidth.

For robustness and security: Tokens that represent partial UTF-8 sequences create vectors for tokenizer-level vulnerabilities. A malicious input could be constructed to produce unusual byte sequences that trigger these under-trained tokens in ways not seen during training, potentially causing model behavior anomalies. The 874 such tokens in GPT-4o's vocabulary represent an unquantified attack surface.

Theoretical significance: The tokenization pipeline is the first transformation applied to input text, before the model even sees it. Any biases introduced here are pre-linguistic — the model never has the opportunity to learn around them because it never sees the original character-level representation. If the tokenizer systematically fragments certain scripts or assigns them higher token costs, the downstream model inherits these distortions as ground truth. This paper's theoretical contribution is to demonstrate that many of these distortions are not inherent to the task of tokenization but are artifacts of specific design choices (UTF-8 encoding, regex pretokenization, unconstrained BPE) that can be changed.

Prior Approaches and Where They Fall Short

The paper situates its contribution against four existing lines of work:

Byte-level BPE with regex pretokenization (the dominant paradigm). This is the approach used by GPT-2 through GPT-4o, and it is the baseline the paper primarily compares against. The well-documented problems — byte premiums, partial-UTF-8 tokens, script-unaware regex splitting — are not unknown to the field, but prior attempts to address them have been incremental rather than fundamental. Schmidt et al. (2025) proposed more robust regex patterns to handle apostrophe variants, but this is a patch on a patch: it fixes one edge case without addressing the deeper structural biases. The Boundless BPE approach (Schmidt et al., 2025) removes whitespace-based pretokenization to improve compression, but still operates within the byte-level framework and doesn't address the underlying UTF-8 encoding inequities.

Character-level BPE (e.g., SentencePiece). Operating directly on Unicode characters avoids the UTF-8 byte premium problem entirely, since every character — whether Latin 'a' or Chinese '你' — starts as exactly one token. However, this approach encounters a different scaling problem: Unicode contains approximately 150,000 defined codepoints. Training a BPE tokenizer directly on this space is impractical, so implementations like SentencePiece must select a subset of codepoints to use as initial tokens and fall back to UTF-8 byte encoding for any character not in the selected subset. This means the byte premium problem isn't eliminated — it's just pushed to a subset of characters that weren't selected by the coverage parameters, reintroducing the same inequities for less common scripts.

Language-specific or script-specific tokenization improvements. Several recent works have proposed domain-specific fixes: Lee et al. (2025) demonstrated benefits from jamo-level (sub-syllabic) tokenization for Korean Hangul, and Velayuthan and Sarveswaran (2025) proposed grapheme-level tokenization for Tamil, Sinhala, and Hindi to keep characters and their diacritics together. MYTE (Limisiewicz et al., 2024) introduced morphology-driven byte encodings for more equitable cross-lingual representation, showing that linguistically informed byte assignments improve fairness. While these approaches demonstrate genuine improvements, each is inherently limited to specific languages or requires resources (morphological annotations, grapheme inventories) that don't scale easily to the 100+ languages that multilingual models aim to serve. The paper's critique is not that these approaches are wrong, but that they don't provide a unified framework that works across all scripts without requiring per-language engineering.

Whitespace-agnostic tokenization. Recent work (Liu et al., 2025; Schmidt et al., 2025) has questioned whether whitespace-based pretokenization should be abandoned entirely, showing that allowing merges across whitespace boundaries improves compression, reduces latency, and increases throughput. The SCRIPT-BPE approach aligns with this direction: its rule-based pretokenizer doesn't enforce whitespace splitting as a hard constraint, instead using whitespace as one signal among multiple in its grouping algorithm (specifically, single spaces may be merged with following groups under defined conditions). However, the paper doesn't fully abandon structured pretokenization; it argues that script and category properties provide a more principled basis for boundary decisions than regex patterns.

How This Paper Positions Itself

The paper's innovation is architectural rather than incremental: change the fundamental representation layer rather than patching the regex or the BPE merging procedure. The key insight is that Unicode itself provides structured metadata — every character has a defined script property (Latin, Cyrillic, Han, etc.) and a general category (letter, punctuation, number, etc.) — that can serve as a principled basis for encoding and pretokenization, but current tokenizers largely ignore this information.

SCRIPT-BPE makes three interconnected design choices, each motivated by a specific failure of the dominant paradigm:

Replace UTF-8 bytes with a two-token script-category-index encoding. Instead of encoding characters as variable-length UTF-8 byte sequences (1–4 bytes), every character is represented as exactly two tokens: a block token that identifies the script and supercategory of the character, and an index token that identifies the specific character within that block. This simultaneously solves the byte premium problem (every character costs exactly 2 tokens, regardless of script) and eliminates the disconnectedness problem illustrated in Figure 1B (where characters from different scripts can share the same initial UTF-8 byte prefix due to historical Unicode assignment ordering — under SCRIPT encoding, similar characters are in the same block and share the same block token).
Use a rule-based pretokenizer grounded in Unicode properties rather than regex. Instead of maintaining a complex regular expression that captures language-specific heuristics, the pretokenizer groups consecutive characters that share the same script and supercategory, with a small number of additional rules for edge cases (space merging, inherited script handling, Hiragana-Han merging). This is not only simpler and more transparent but also inherently multilingual: it doesn't need to know about English contractions because it doesn't split on contractions at all — characters from the same script and category group together naturally.
Constrain BPE merges to respect character boundaries. This is presented as an independent contribution applicable to both SCRIPT-BPE and byte-level BPE. The constraint is simple: merges are only permitted between tokens that already represent one or more complete characters, or (in the SCRIPT case) between a single block token and a single index token to form a complete character. In byte-level BPE, this means allowing merges within the byte sequence of a single character (processed left-to-right) or between sequences that each represent complete characters, but never between a complete character and a partial byte of an adjacent multi-byte character. The paper explicitly shows that this constraint universally eliminates the partial-UTF-8 token problem while generally improving compression (Table 2), making it a no-downside improvement that the authors recommend for all BPE tokenizers.

The paper's positioning is distinctive: it is not claiming to have discovered that tokenization matters — the literature already establishes that pretokenization can have greater impact on downstream performance than vocabulary size (Wegmann et al., 2025). Rather, it is offering a coherent alternative design that addresses multiple known failure modes simultaneously through a representation change, accompanied by a constraint that improves even the baseline it aims to replace. The emphasis is on robustness and parity over raw compression: SCRIPT-BPE is designed to be competitive on compression while eliminating encoding-level bias, not necessarily to beat regex-based tokenizers on every language. The fact that it outperforms them substantially on languages like Thai, Hindi, and Punjabi — precisely the languages where regex-based pretokenization causes the most damage — serves as validation of the design philosophy rather than as the central claim.

The paper also explicitly acknowledges the limits of its current evaluation. Compression is the primary metric, and the authors note that compression alone "does not necessarily guarantee better downstream model performance" (Section 5), citing Schmidt et al. (2024). This is not a deflection — it's a recognition that the next step (training models with SCRIPT-BPE and evaluating on downstream tasks) is essential to validate that the encoding improvements translate to model quality improvements, and the paper frames this as the primary direction for future work.

3. Technical Approach

3.1 Reader Orientation

This paper proposes a tokenizer design system — a recipe for converting raw text into the integer token sequences that language models consume — called SCRIPT-BPE. The problem it solves is that current tokenizers systematically disadvantage non-Western writing systems by using a byte encoding that charges different "token costs" for different scripts and by using pretokenization rules designed around English. The solution has the shape of a three-part replacement for the standard tokenization pipeline: (1) swap the underlying character representation from variable-length UTF-8 bytes to a fixed-length encoding grounded in Unicode script properties, (2) swap the regex-based pretokenizer for a rule-based pretokenizer that groups characters by those same script properties, and (3) add a simple constraint to the BPE merging process that prevents tokens from ever mixing partial and complete characters.

3.2 Big-Picture Architecture (Diagram in Words)

The SCRIPT-BPE tokenization pipeline has four major components that process text in sequence:

Unicode Property Mapper: Takes raw Unicode text as input, looks up each character's script and general category from the Unicode standard database, and applies a small set of manual reassignments (e.g., treating newlines as separators rather than control characters). The output is a sequence of "(script, supercategory)" tags — one per character — with some characters grouped into sub-blocks if their script-category block is too large. No regex, no pattern matching, no language-specific logic.
SCRIPT Encoder: Maps each character to exactly two integers: a block token (identifying which script-supercategory-sub-block the character belongs to) and an index token (identifying which specific character within that block). Every character, regardless of script, costs exactly two tokens in the initial representation — a Latin 'a' and a Chinese '你' both become a pair of integers. This replaces UTF-8 byte encoding, where those same characters would cost 1 byte and 3 bytes respectively.
Rule-Based Pretokenizer: Groups consecutive characters into "pretokens" based on whether they share the same script and supercategory. Characters with the same (script, category) are grouped together; boundaries are placed where either the script or the category changes. A small set of additional rules handle edge cases: single spaces may merge with the following group (for scripts that use whitespace to separate words), characters with the "Inherited" script (like combining diacritics) merge with their preceding group, and sequences of Han + Hiragana characters merge to avoid splitting Japanese words. BPE merges are prohibited across pretoken boundaries.
Constrained BPE Merger: Trains and applies BPE merges with an additional constraint: merges are only permitted when both sides of the merge represent complete characters or complete sequences of characters. In the SCRIPT encoding, this means an index token can only merge with its corresponding block token (forming a complete character), after which the resulting merged token can merge with other complete-character tokens. In byte-level BPE, this means merges within the byte sequence of a single character are allowed (processed left-to-right), and merges between complete-character sequences are allowed, but merges that would create a token mixing part of one character with another are prohibited. This constraint is enforced during both training and inference.

Information flows linearly: raw text → Unicode property lookup → paired (block token, index token) sequence → grouping into pretokens based on script/category boundaries → iterative constrained BPE merging until the target vocabulary size is reached.

3.3 Roadmap for the Deep Dive

First, the SCRIPT encoding scheme itself: how the block tokens and index tokens are defined, what Unicode properties are used, how large blocks are split into sub-blocks, and the concrete vocabulary sizes that result (468 block tokens, 1448 index tokens). This is the foundational representation change — everything else builds on it.
Second, the rule-based pretokenization algorithm: the four specific rules (initial script-based grouping, space merging, inherited script merging, Hiragana-Han merging), what each rule accomplishes, and how this replaces the complex regex patterns used by GPT-family tokenizers. Understanding the pretokenization is essential because it sets the boundaries that BPE merges cannot cross and has been shown (Wegmann et al., 2025) to have outsized impact on final compression.
Third, the constrained BPE merging strategy: the specific constraint that prohibits cross-character-boundary merges, how it is implemented for both SCRIPT-BPE and byte-level BPE, and why the paper argues this should be adopted universally. This is presented as a separate, independently applicable contribution.
Fourth, the training pipeline and hyperparameters: the datasets, vocabulary sizes, merge counts, and hardware used. These are straightforward but essential for reproducibility and for understanding the scale at which the comparisons operate.
Fifth, the design rationale synthesis: why each design choice was made and how the three components (encoding, pretokenization, constrained merging) interact synergistically rather than being independent fixes.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems design paper whose core idea is that structured metadata already present in the Unicode standard — specifically, script and general category properties — can replace both the UTF-8 byte encoding and the regex-based pretokenization that dominate current tokenizer designs, producing a simpler, more equitable, and comparably efficient tokenization pipeline.

3.4.1 The SCRIPT Encoding Scheme: Replacing UTF-8 Bytes with Script-Category-Index Pairs

The SCRIPT encoding is the paper's central representational innovation. Instead of converting each Unicode character to its variable-length UTF-8 byte sequence (1–4 bytes depending on the script), SCRIPT maps every character to exactly two tokens derived from two Unicode properties that every defined character possesses.

Unicode Properties Used

Unicode Script: Every character in the Unicode standard is assigned to exactly one script — a named writing system such as Latin, Cyrillic, Han, Arabic, Devanagari, Thai, or Hiragana. The script property answers: "what writing system does this character belong to?" There are approximately 160 scripts defined in Unicode. The script property is precise, exhaustive, and maintained as part of the Unicode standard — it does not require any manual annotation or per-language engineering.

Unicode General Category: Every character is also assigned to a general category that describes its linguistic function: letters (Lu, Ll, Lt, Lm, Lo for uppercase, lowercase, titlecase, modifier, and other letters), marks (Mn, Mc, Me for non-spacing, spacing-combining, and enclosing marks), punctuation (Pc, Pd, Ps, Pe, Pi, Pf, Po for various punctuation types), symbols (Sc, Sk, Sm, So for currency, modifier, math, and other symbols), numbers (Nd, Nl, No for decimal digits, letter numbers, and other numbers), separators (Zs, Zl, Zp for spaces, line separators, and paragraph separators), and other (Cc, Cf, Cs, Co, Cn for control, format, surrogate, private use, and unassigned characters). These fine-grained categories capture meaningful functional distinctions.

Supercategory Formation

Rather than using all 30+ fine-grained general categories, the paper groups them into five supercategories. This grouping reduces complexity while preserving the most important functional distinctions:

Letters & Marks (LM): All L* (letter) and M* (mark) categories are merged into a single supercategory. The rationale stated in Appendix A is that marks (diacritics, accents, vowel signs) are "typically attached to or modify letters" — they are functionally part of the same orthographic unit, and splitting them into separate groups would fragment characters that visually and semantically belong together. This is precisely the problem that Velayuthan and Sarveswaran (2025) identified for Tamil, Sinhala, and Hindi, where diacritic-splitting by regex pretokenizers degrades compression; the LM supercategory prevents this splitting at the encoding level.
Punctuation & Symbols (PS): All P* (punctuation) and S* (symbol) categories are combined. The authors note that "the boundary between punctuation and symbols can be ambiguous" — for instance, the common programming operators -> and != consist of a punctuation character followed by a symbol character, but they serve a unified syntactic role. Merging them into one supercategory allows these functionally related characters to be grouped together during pretokenization.
Numbers (N): All Nd, Nl, and No categories — decimal digits, letter-like numbers (e.g., Roman numerals), and other numbers. This preserves the functional distinction between numeric and non-numeric characters.
Separators (Z): All Zs, Zl, and Zp categories — spaces, line separators, paragraph separators. This groups whitespace-like characters together.
Other (C): The remaining categories — control characters, format characters, private use, surrogates, and unassigned codepoints. However, the paper explicitly filters out characters from the Unassigned (Cn), Private Use (Co), and Surrogate (Cs) subcategories, treating them as "not meaningful or generalizable for language modeling" (Appendix A). These are excluded from all tokenizers (both SCRIPT and baselines) during evaluation to maintain a fair comparison.

Manual Reassignments

The paper applies a small number of manual reassignments to improve how specific characters are handled during pretokenization. These are not arbitrary — each addresses a specific edge case where the Unicode standard's default assignment would produce suboptimal grouping:

Newline (\n, U+000A) and tab (\t, U+0009) are reassigned to the Separator (Z) category. By default, Unicode classifies these as Other/Control characters (Cc) "for historical reasons" (Appendix A). If left as Control characters, they would be grouped with other control characters rather than with whitespace — a clearly undesirable outcome for a tokenizer that treats newlines and tabs as word separators. By reassigning them to Z, they become part of the same supercategory as spaces and can participate in the space-merging pretokenization rule.
Katakana-Hiragana prolonged sound marks (U+30FC and U+FF70) are reassigned to the 'Inherited' script. These marks (the long-vowel marker "ー" in Japanese) are used with both Katakana and Hiragana, but Unicode classifies them under the 'Common' script. By reassigning them to 'Inherited', they are allowed to group with whichever script they appear adjacent to — either Katakana or Hiragana — rather than forming their own separate group. This prevents an artificial boundary within Japanese words that contain these marks.
The Tatweel mark (U+0640, used for text justification in Arabic) is reassigned from Common to the Arabic script. This elongated dash-like character stretches text for visual justification in Arabic script; reassigning it to Arabic ensures it groups with the Arabic characters it is visually and functionally part of, rather than forming a separate Common-script group.

These reassignments are not part of the encoding itself (which still uses the original Unicode codepoint) but affect how characters are grouped during the subsequent pretokenization step. They represent a small, explicit set of corrections to Unicode's default categorization — a total of 4 specific characters affected — compared to the hundreds of implicit heuristics embedded in a GPT-style regex pattern.

Block Token and Index Token Formation

The SCRIPT encoding maps each Unicode character (after filtering out Cn, Co, and Cs categories) to a unique pair: (block_token, index_token).

Block tokens: A block is defined by the combination of a script and a supercategory. For example, all Latin letters and marks form the "Latin LM" block; all Arabic punctuation and symbols form the "Arabic PS" block; all Han letters form the "Han LM" block. Each unique (script, supercategory) pair is a potential block.

However, some blocks contain an extremely large number of characters. As shown in Table 1, the largest blocks include Han LM (44,266 characters, primarily Kanji/Hanzi), Latin LM (5,561 characters, including accented letters and various Latin-script extensions), and Hangul LM (11,798 characters, primarily Korean syllable blocks). If each block were assigned a single block token, the index token would need to encode up to 44,266 distinct values for Han LM alone — an enormous and sparse range.

Sub-block splitting: To manage this, any (script, supercategory) block containing more characters than a threshold is split into multiple sub-blocks. The threshold is set to the size of the 'Latin LM' block — specifically, 5,561 characters (the total size of Latin LM, including all its sub-blocks discussed below). The reasoning is implicit but logical: since Latin is the most commonly used script in training data, using its LM block size as the threshold ensures that all blocks are at most as large as the largest commonly-used block, creating a reasonable upper bound on index token ranges.

The splitting procedure works as follows (though the paper does not exhaustively detail the splitting algorithm): characters within a large block are partitioned into sub-blocks of at most 5,561 characters each. For example, Han LM (44,266 characters) would be split into approximately 8 sub-blocks (since 44,266 / 5,561 ≈ 7.96). Each sub-block receives its own distinct block token. The resulting total vocabulary of block tokens is 468 (as stated in Section 3).

Index tokens: Within each block or sub-block, characters are assigned sequential index tokens starting from 0 (or from a base offset determined by the block token). The index token identifies which specific character within the block is being represented. Since the largest sub-block contains at most ~5,561 characters, the maximum index value is bounded by this threshold. The total vocabulary for index tokens is 1,448, as stated in the paper. The relationship between the 468 block tokens and 1,448 index tokens is not one-to-one: some blocks are small (e.g., a script-category combination with only a handful of characters) and contribute a correspondingly small range of index values, while larger sub-blocks contribute larger index ranges. The 1,448 figure represents the union of all index values needed across all blocks.

Note that 1,448 is substantially less than 5,561 (the threshold) times 468 (the number of blocks). This makes sense: most blocks contain far fewer than 5,561 characters — for example, a block for "Thai PS" might contain only a few dozen punctuation characters — so the total index token vocabulary is dominated by the handful of large LM blocks (Han, Hangul, Latin, etc.) and is much smaller than the product of block count times threshold.

Concrete example of the encoding: Suppose the Chinese character "你" (U+4F60) is in the Han script and the LM (Letters & Marks) supercategory. The Han LM block is too large (44,266 characters) and has been split into sub-blocks. "你" is assigned to, say, Han LM sub-block 3 (one of ~8 sub-blocks), and within that sub-block it has index 2,341. Its SCRIPT encoding is the pair (BLOCK_Han_LM_3, 2341). A Latin letter "a" (U+0061) is in the Latin script and LM supercategory. If Latin LM is not split (it's exactly at the threshold of 5,561), it might be in block (BLOCK_Latin_LM, 97) (since 'a' is the 98th character in the block if zero-indexed). Both characters cost exactly two tokens.

Comparison with UTF-8 and Unicode Codepoint Approaches

The design of SCRIPT encoding is motivated by specific deficiencies in the two dominant alternatives:

UTF-8 byte encoding: This is the most common approach, used by OpenAI's tiktoken (GPT-2 through GPT-4o) and many other tokenizers. The fundamental problem is variable length: a Latin 'a' is 1 byte → 1 token, a Greek 'α' is 2 bytes → 2 tokens, a Chinese '你' is 3 bytes → 3 tokens. This is the "byte premium effect" (Arnett et al., 2024): non-Latin scripts pay a structural tax in token count before any BPE training occurs. Additionally, UTF-8 byte sequences for different scripts can share common prefixes purely due to historical assignment ordering in Unicode. Figure 1B illustrates this: characters from entirely different scripts may share the same initial byte pattern, making the byte representation uninformative about script affiliation. A Thai character starting with bytes <0xE0><0xB8> and a CJK character starting with <0xE0><0xB8> would appear related to the tokenizer despite having no linguistic connection — they just happen to occupy adjacent codepoint ranges.

SCRIPT encoding eliminates both problems: every character costs exactly 2 tokens regardless of script (no byte premium), and the block token explicitly encodes script identity (characters from the same script share the same block token, creating meaningful initial groupings).

Direct Unicode codepoint encoding: This is used by SentencePiece in its character-level mode. Every character is a single integer (its codepoint), so the initial cost is 1 token per character for all scripts — no byte premium. However, Unicode has ~150,000 defined codepoints, far too many to use as initial tokens in BPE (which would require tracking merge frequencies across a 150,000 × 150,000 pair matrix). SentencePiece addresses this by selecting a subset of codepoints to include as initial characters (determined by a coverage parameter, e.g., 0.9995) and falling back to UTF-8 byte encoding for any character not in the selected subset. This means the byte premium problem isn't eliminated — it's merely pushed to less frequent characters, reintroducing the very inequities the approach sought to avoid, now in a more unpredictable way (whether a given character pays the byte premium depends on its frequency rank in the training data).

SCRIPT encoding avoids this two-level problem entirely: every character is always exactly two tokens, regardless of frequency. There is no coverage parameter to tune and no fallback encoding. The tradeoff is a slightly higher initial token count for ASCII-range characters (2 tokens versus 1 in byte-level or codepoint-level encoding) but a substantial reduction for multi-byte characters (2 tokens versus 2–4 in byte-level encoding), with the additional benefit of script-structured initial representations.

3.4.2 Rule-Based Pretokenization

The second major component of the SCRIPT-BPE pipeline is a pretokenizer that uses the script and supercategory metadata (already computed for the encoding) to determine where to place boundaries that BPE merges cannot cross. This replaces the complex regular expressions used by the GPT family of tokenizers.

The Four Pretokenization Rules

The pretokenization algorithm is defined by four sequential rules, each applied to the character sequence after SCRIPT encoding. The rules are described in Appendix B with sufficient precision to implement:

Rule 1: Initial Script-Based Grouping. Consecutive characters that share the same Unicode script and the same supercategory are grouped together into an initial "protogroup." The script and supercategory are the same properties used in the encoding step — specifically, the script is the character's assigned Unicode script property (after the manual reassignments discussed in Section 3.4.1), and the supercategory is the LM/PS/N/Z grouping. Sub-blocks within a large script-supercategory combination are ignored for this grouping step; all Latin LM characters form one group regardless of which sub-block they fall into. The grouping rule is: place a boundary wherever either the script or the supercategory changes. For instance, a sequence "Hello 世界!" would initially group as: [H e l l o] (Latin LM), [ ] (Separator Z), [世界] (Han LM), [!] (Common PS). This produces groups that are internally homogeneous in both script and functional category.

Why this works for multilingual text: unlike a regex pattern that hard-codes rules like "split on English contractions," this grouping is content-driven. It naturally keeps Tamil letters and their combining marks together (both are LM, both are Tamil script), naturally separates Latin text from interleaved Han text (different scripts), and naturally separates letters from punctuation (different supercategories). No per-language rules are needed — the Unicode metadata carries the relevant information.

Rule 2: Space Merging. If a protogroup consists of a single space character (specifically, a character in the Separator supercategory that is a space, not a newline or tab), it may be merged with the following protogroup. This merging occurs only when the following group falls into one of two categories:

An LM supercategory group whose script is in a set of scripts that "use whitespace to separate words." The paper's current list (footnote 4) consists of: Latin, Arabic, Devanagari, Hangul, Ethiopic, Cyrillic, Greek, Hebrew, Bengali, Syriac, Oriya, Tamil, Telugu, Gurmukhi, Gujarati, Sinhala, Malayalam, Armenian, Kannada, and Georgian. These are all scripts that conventionally use spaces as inter-word boundaries in their standard orthography.
A 'Common PS' group (punctuation and symbols assigned to the Common script).

The effect of this rule is to attach a preceding space to the word that follows it — precisely the behavior that whitespace pretokenization achieves by default. However, this rule is more selective: it only merges the space when the following group is from a script that conventionally uses spaces, and it doesn't enforce a blanket "always split on whitespace" rule. This selectivity matters: for scripts like Chinese or Japanese that do not use spaces between words, a space appearing in the text is more likely to be a deliberate formatting element (e.g., separating a Latin-script abbreviation from surrounding Han characters) than a word boundary, and this rule correctly leaves such spaces as their own group rather than merging them inappropriately.

The rule also implicitly handles what happens when a space is followed by a Han LM group: since Han is not in the list of scripts that "use whitespace to separate words," the space is not merged and remains as its own protogroup. This respects the orthographic conventions of CJK scripts.

Rule 3: Inherited Script Merging. Some Unicode characters have their script property set to 'Inherited' — this includes combining diacritics, zero-width joiners, variation selectors, and other characters that derive their script from the surrounding context. If a protogroup's script is 'Inherited', the following merging occurs:

The 'Inherited' group is merged with the immediately preceding group (which provides the script context), and any immediately following groups that share the same script and supercategory as that preceding group are also merged into the same combined group. The paper gives a concrete example: a sequence of groups (Arabic LM, Inherited LM, Arabic LM, Inherited LM, Arabic LM) will be merged into a single combined group. This handles the common case in Arabic script (and other scripts with extensive diacritic marking) where base letters and their attached marks alternate: the Inherited characters (marks) attach to their base letters, and the whole word stays together as one pretoken.

Without this rule, every diacritic mark would form its own separate group (since its script is 'Inherited', distinct from 'Arabic'), fragmenting words into base-letter and mark pieces — exactly the behavior that regex-based pretokenizers cause for Tamil, Sinhala, and Hindi.

Rule 4: Hiragana-Han Merging. Sequences of Han (Kanji) and Hiragana characters are merged into a single group. Japanese text commonly alternates between Kanji (for content words) and Hiragana (for grammatical particles, verb endings, and furigana), and these alternations represent a single linguistic unit (a word or phrase) despite the script switching. Without this rule, a Japanese sentence like "私は学生です" (Watashi wa gakusei desu — "I am a student") would be fragmented into separate groups for the Kanji portions (私, 学生) and Hiragana portions (は, です), breaking apart words like "私は" (watashi-wa) and "学生です" (gakusei-desu) at unnatural boundaries. The rule applies specifically to sequences that alternate between Han and Hiragana; it does not merge Katakana (used for loanwords and emphasis) with Han, respecting the distinct orthographic function of Katakana.

This is the one script-specific rule in the pretokenizer — it encodes linguistic knowledge about Japanese orthography that is not directly captured by the script and supercategory properties. It represents an acknowledgment that not all useful pretokenization boundaries can be derived purely from category-level properties, but the rule is explicit, documentable, and localized to a specific script interaction rather than being embedded in an opaque regex pattern.

What the Pretokenizer Does Not Do

Equally important is what the rule-based pretokenizer deliberately omits:

No contraction splitting: Unlike the GPT-family regex patterns that split on 'm, 's, 'd, 'll, 've, etc., SCRIPT pretokenization does not have any contraction-specific rules. An apostrophe character is treated as punctuation (PS supercategory), so it is separated from the surrounding letters (LM supercategory) by the initial script-based grouping rule. This means English contractions like "I'm" become two pretokens: I (Latin LM) and 'm (Common PS, since the apostrophe is Common script and the 'm' is Latin LM — actually, the apostrophe triggers a boundary, and then the 'm' is in a separate LM group). Wait — let's think through this more carefully. The apostrophe (U+0027) is in the Common script and Punctuation supercategory. After it, the letter 'm' is Latin LM. So the initial grouping produces: [I] (Latin LM), ['] (Common PS), [m] (Latin LM). Then, by the space-merging rule: not applicable (no space). By the inherited script rule: not applicable (the apostrophe is not Inherited script). So the final pretokens are I, ', and m — three separate groups. This is actually more fragmented than the GPT approach, which would keep I and 'm separate (since the regex splits before the apostrophe in this case). The paper does not discuss this specific behavior, and it may represent a compression tradeoff: English contractions are slightly fragmented by SCRIPT pretokenization. This is a consequence of the design philosophy — rather than encoding language-specific heuristics, the pretokenizer follows the Unicode categories and accepts that some language-specific compression optimizations are lost.
No digit splitting: The GPT regex splits digits into groups of up to three (e.g., 123456 becomes 123 and 456). SCRIPT pretokenization treats all digits as Numbers (N supercategory), so consecutive digits stay together in one group. Single-digit tokenization — which Schmidt et al. (2024) and others have associated with better arithmetic performance — is not enforced by the SCRIPT pretokenizer. The paper notes this as an area for future refinement (Section 5): "combining refining the handling of digits and leading spaces."
No whitespace splitting mandate: The rule-based pretokenizer does not universally split on whitespace. Spaces form their own group (Separator Z), but the space-merging rule selectively attaches them to following words. Newlines and tabs are also in the Z supercategory but are not included in the space-merging rule. This is a design choice that aligns with recent work questioning whether whitespace-based pretokenization is essential (Liu et al., 2025; Schmidt et al., 2025), while maintaining the practical benefit of treating spaces as word-attached when orthographically appropriate.

3.4.3 Constrained BPE Merging

The third major component is a constraint on the BPE merging process that enforces character integrity: merges are prohibited from creating tokens that mix partial and complete characters. This constraint is presented as applicable to both SCRIPT-BPE and byte-level BPE, and the paper argues it should be adopted universally given that it "universally eliminated tokens representing a mix of full and partial characters and generally improved compression across different base encodings" (Section 5).

The Problem That Constrained Merging Solves

In standard (unconstrained) BPE, the merging algorithm has no awareness of whether the tokens it is merging represent complete characters, partial characters, or mixtures. The algorithm simply tracks the frequencies of adjacent token pairs and greedily merges the most frequent pair. This can produce merges like:

A space character (a full character in the ASCII range, encoded as a single byte 0x20) merging with the first byte (0xE0) of a three-byte CJK character. The resulting token represents one full character (the space) and one-third of a CJK character — a "mix of full and partial characters."
Subsequently, this mixed token can merge with the second byte of the CJK character, creating a token representing a space plus two-thirds of a CJK character. And so on.
The process can cascade: once a mixed token exists in the vocabulary, it can participate in further merges, creating ever more tokens that mix full and partial characters in increasingly complex ways.

The paper quantifies this concretely: the GPT-4o tokenizer has 874 tokens representing a mix of full and partial characters, and the o200k regex pretokenizer on the Thai dataset produces 42,831 such tokens — an extreme case where the problem cascades extensively. These tokens are problematic for several reasons:

They have no linguistic interpretation — they don't correspond to any character, subword, or meaningful unit.
They risk producing invalid byte sequences if they appear in the wrong context (e.g., a token containing partial UTF-8 bytes for a character that needs a different continuation byte).
They can become "under-trained" (Land and Bartolo, 2024): since they don't correspond to any linguistic unit, their occurrence frequency in training data may be much lower than expected for a token in the vocabulary, leading to poorly learned embeddings.

The Constraint for SCRIPT-BPE

In the SCRIPT encoding, every character is represented as exactly two tokens: a block token and an index token. The constrained merging rule is:

Merges are only allowed between tokens that already represent one or more full characters, or between a single block token and a single index token (thereby forming a complete character).

This means the first merge for any character must be between its block token and its index token, fusing the pair into a single token representing the complete character. Once that merge has occurred, the resulting character-representing token can merge with other character-representing tokens to form multi-character tokens (subwords, words, etc.).

The effect is that at every step of the BPE process, every token in the vocabulary represents either (a) a block token alone (not yet merged), (b) an index token alone (not yet merged), (c) exactly one complete character (merged block+index), or (d) a sequence of complete characters (further merges of character tokens). No token ever represents a fractional character or a mix of character fragments from different characters.

Implementation detail: The constraint is enforced during the pair frequency counting step of BPE training. When scanning the corpus to count adjacent token pair frequencies, any pair that would violate the constraint is simply not counted — it is excluded from the set of candidate merges. This means the constrained BPE never considers such merges, so they never enter the vocabulary. The computational cost of checking the constraint is small: each token can be tagged as "complete character" or "partial character" (block or index token alone), and the constraint check is a simple tag comparison.

The Constraint for Byte-Level BPE

For standard byte-level BPE, the constraint is slightly more complex because characters can be 1–4 bytes long, and the encoding is not self-delimiting (you can't tell from a single byte whether it's a complete character or part of a multi-byte sequence without additional information). The constrained merging rule for byte-level BPE is:

Merges are allowed within the byte sequence of a single character (processed strictly left-to-right), or between sequences that each represent complete characters. Merges between a complete character and a partial byte of an adjacent multi-byte character are disallowed.

To enforce this, the tokenizer needs to track, for each token, whether it represents:

A complete single-byte character (ASCII range: byte values 0x00–0x7F)
A partial multi-byte character (the first 1–3 bytes of a character that requires 2–4 bytes)
A complete multi-byte character (all 2–4 bytes merged together)
A sequence of complete characters (further merges)

The constraint then operates as: merges are permitted between two partial-byte sequences of the same character (left-to-right within the character), between a partial-byte sequence that completes a character and the preceding partial-byte sequences of that same character, or between two complete-character sequences. Merges between a complete-character token and a partial-byte token of a different character are prohibited.

The paper notes that implementing this constraint for byte-level BPE is "more complex" than for SCRIPT-BPE "due to the more complex character boundary checks required for UTF-8 encoding compared to SCRIPT." This complexity is reflected in the training time results (Table 3): for byte-based tokenizers, constrained merging increases training time slightly (from 0.87 to 0.93 hours for o200k in the multilingual setting), while for SCRIPT-based tokenizers, constraining merges actually reduces training time (from 0.63 to 0.58 hours for rule-based SCRIPT) — presumably because the reduction in candidate merge pairs outweighs the cost of boundary checks for the simpler SCRIPT encoding.

Why Constrained Merging Improves Compression

The finding that constrained merging "almost universally improves compression" (Section 4.1, Table 2) is initially counterintuitive. Why would restricting the set of possible merges — reducing the algorithm's flexibility — produce better (lower) tokens-per-character ratios?

The explanation, implied by the paper's discussion of the Thai cascade failure, is that the unconstrained merges that cross character boundaries are short-sighted optimizations. They look locally frequent — perhaps a space followed by the first byte of a common Thai character is a very frequent pair — but they produce tokens that are less reusable across contexts because they encode a mixture of unrelated linguistic units. When the BPE algorithm merges a space with the first byte of a Thai character, it commits to a token that can only appear before Thai characters sharing that first byte, rather than keeping the space and the Thai character's bytes as separate, recombinable tokens. The constraint prevents these "premature" cross-boundary merges, keeping the representation more compositional and enabling merges that are genuinely linguistically motivated (e.g., merging the bytes of the multi-byte character into a single token representing that specific character, then merging that character token with adjacent characters to form subword units).

This interpretation is consistent with the finding that the constraint eliminates the cascading failure on Thai (reducing partial-UTF-8 tokens from 42,831 to 0) while also improving compression (from 3.263 to 3.231 tokens/character). The cascade was producing many low-quality tokens that inflated the vocabulary without providing good compression.

Generality of the Constraint

The paper emphasizes that constrained merging is "compatible with all encoding approaches" (Section 5) and recommends its adoption "particularly for massively multilingual tokenizers." The constraint is independent of the encoding scheme (SCRIPT or UTF-8 bytes) and the pretokenization method (regex or rule-based). It is a drop-in modification to the BPE merge selection process, requiring no changes to the training data, the vocabulary size, or the inference procedure.

The universality claim is supported by Table 2, which shows that constrained merging eliminates partial-UTF-8 tokens and generally improves compression for all combinations tested: byte-level BPE with both cl100k and o200k regex patterns, and SCRIPT-BPE with both rule-based and regex pretokenization. The worst-case degradation (where constrained merging slightly increases tokens/character) is small — at most a fraction of a percent — while the best-case improvements are substantial.

3.4.4 Training Pipeline and Experimental Configuration

The paper trains and evaluates tokenizers in two settings, with specific dataset, vocabulary, and hardware configurations that enable reproducibility.

Datasets

Monolingual setting: Tokenizers are trained for each of 12 languages: Japanese, Chinese, Thai, Punjabi, Hindi, Korean, Russian, Arabic, Hebrew, Vietnamese, German, and English. For each language, the training data is a 300 MB subset sourced from Chang et al. (2024) — the Goldfish project, which provides monolingual corpora for 350 languages. The 300 MB figure refers to the size of the text after Unicode normalization (details not specified, though the paper implies consistent preprocessing across all methods).

Multilingual setting: A single multilingual tokenizer is trained on a 35 GB subsample of CulturaX (Nguyen et al., 2023). CulturaX is a large-scale, cleaned multilingual dataset covering 167 languages, drawn from web-crawled sources. The 35 GB subsample is designed to provide broad multilingual coverage. A separate 136 GB validation set is constructed from CulturaX (presumably a disjoint subset) for evaluating compression.

Vocabulary sizes: Monolingual tokenizers are trained to 64,000 merges (vocabulary size = base tokens + 64,000). The multilingual tokenizer is trained to 256,000 merges. These are standard sizes: 64K is common for monolingual models (comparable to the vocabulary size of GPT-2's 50,257), and 256K is standard for large multilingual models (comparable to GPT-4o's 200K vocabulary). The SCRIPT encoding's base vocabulary is 468 block tokens + 1,448 index tokens = 1,916 base tokens (before any merges), compared to 256 base tokens for byte-level BPE. However, this difference in base vocabulary size is negligible compared to the 64K–256K merge budget: for SCRIPT-BPE with 64,000 merges, the final vocabulary is 1,916 + 64,000 = 65,916 tokens; for byte-BPE, it's 256 + 64,000 = 64,256 tokens. The SCRIPT-BPE vocabulary is ~2.6% larger at the monetization level and ~0.65% larger at the multilingual level — a minor difference.

Baselines

The paper compares against two standard tokenizer configurations, both using tiktoken (OpenAI, 2024) as the BPE implementation with UTF-8 byte-level encoding:

cl100k (GPT-4): The pretokenization pattern used by GPT-4. This is a complex regex that handles English contractions, whitespace, digit grouping, and letter-based splitting.
o200k (GPT-4o): The updated pretokenization pattern used by GPT-4o. This is a refinement of the GPT-4 pattern with additional rules (e.g., handling of non-ASCII characters and different apostrophe types).

For SCRIPT-BPE tokenizers, two pretokenization variants are tested:

Rule-based: The four-rule pretokenizer described in Section 3.4.2.
Regex (cl100k or o200k): As an ablation, the SCRIPT encoding is combined with the standard GPT regex pretokenizers, to isolate the effect of the encoding from the pretokenization.

All tokenizers are trained with both unconstrained and constrained BPE merging, as reported in Table 2.

Training Implementation

The tokenizer training experiments use a custom Python implementation (not tiktoken or SentencePiece). The paper notes this in Section 4.2: "Our tokenizer training experiments are conducted using a custom Python implementation." This is significant because it means the constrained merging logic is implemented from scratch, and the training time comparisons in Table 3 reflect this custom implementation rather than an optimized library.

Hardware: The paper states (Section 4.2) that the training times in Table 3 are reported for "a moderately parallel setup with 16 CPUs." All tokenizers train in approximately one hour for the 256K-merge multilingual setting, "ensuring that training time is not a bottleneck." The specific hardware details (CPU model, RAM, disk) are not provided.

Training time results (Table 3, multilingual tokenizer with 256K merges):

Encoding	Pretokenizer	Constrained?	Training time (hours)
Byte	cl100k	No (×)	1.04
Byte	cl100k	Yes (✓)	1.17
Byte	o200k	No (×)	0.87
Byte	o200k	Yes (✓)	0.93
SCRIPT	rule-based	No (×)	0.63
SCRIPT	rule-based	Yes (✓)	0.58
SCRIPT	o200k regex	No (×)	1.01
SCRIPT	o200k regex	Yes (✓)	0.98

These numbers support the claim that training time is not a bottleneck (all under ~1.2 hours for the largest vocabulary size), and they show the interesting asymmetry: constraining merges slows down byte-level tokenizers (due to the more complex UTF-8 boundary checks) but speeds up SCRIPT tokenizers (due to the reduced search space from eliminating cross-character merge candidates).

Evaluation Metric

The primary evaluation metric throughout the paper is tokens per character: the total number of tokens produced by the tokenizer when encoding a text dataset, divided by the total number of Unicode characters in that dataset. Lower values are better — they indicate that the tokenizer compresses the text into fewer tokens, meaning the same amount of text fits into a smaller context window.

The paper also reports number of tokens with partial UTF-8 sequences (Table 2), defined as tokens that represent "a mix of full and partial character sequences" — tokens where some bytes belong to a complete character and some bytes are part of an incomplete character. This is not a compression metric but a quality metric: fewer partial-UTF-8 tokens indicates a cleaner, more linguistically coherent vocabulary.

The initial encoding cost is reported as "initial tokens/character" in Table 4 — the number of tokens per character before any BPE merges are applied. This measures the inherent cost of the encoding scheme, independent of how well BPE compresses the language. For byte-level encoding, this varies by language: English (mostly ASCII, 1 byte per character) has an initial cost near 1.0 tokens/character, while Chinese (3 bytes per character) has an initial cost near 3.0. For SCRIPT encoding, the initial cost is always exactly 2.0 tokens/character for all languages (since every character is 2 tokens). The final compression ratio (after 256K merges) shows how much the BPE process compresses from this initial cost.

3.4.5 Design Rationale: Why These Three Components Together

The three components of SCRIPT-BPE — the encoding, the pretokenization, and the constrained merging — are not independent fixes applied to separate problems. They interact in ways that amplify each other's benefits:

The encoding enables the pretokenization. By representing characters through their script and category properties, the encoding provides exactly the information that the rule-based pretokenizer needs to make boundary decisions. The pretokenizer doesn't need to infer script from byte patterns (which is error-prone, as Figure 1B shows) or maintain a separate lookup — the block token directly encodes script and category, making the pretokenization grouping a simple comparison operation on adjacent block tokens. If the encoding were still byte-based, the rule-based pretokenizer would need to decode the UTF-8, look up Unicode properties, and then group — adding complexity and potential for errors.

The encoding reduces the damage that constrained merging needs to prevent. In the SCRIPT encoding, the worst possible unconstrained merge would be between an index token of one character and the block token of the next — creating a "wrong character" token that combines part of one character with the script identity of another. While this is bad, it's arguably less catastrophic than the UTF-8 case, where a merge between a space and the first byte of a multi-byte character creates a byte sequence that may not correspond to any valid character's start bytes. The SCRIPT encoding's fixed-length, two-token-per-character structure makes the constraint simpler to implement and the failure mode of unconstrained merging less severe.

Constrained merging prevents the encoding from introducing new problems. The SCRIPT encoding uses two tokens per character, which means twice as many initial tokens as character-level encoding and potentially more opportunities for cross-boundary merges (since each character contributes two tokens that could participate in problematic merges). The constrained merging constraint closes this vulnerability: by ensuring that block and index tokens always merge with each other first, it maintains the character-level integrity that the encoding was designed to provide.

The pretokenization determines how much of the encoding's potential is realized. Even with a perfect encoding, if the pretokenization places boundaries at linguistically inappropriate positions (as the cl100k regex does for Thai, Hindi, and Punjabi by splitting at diacritics), the final compression will suffer. The rule-based pretokenizer, by grouping characters with their diacritics (via the LM supercategory and the Inherited script rule) and by respecting script boundaries without imposing English-centric heuristics, allows the encoding's script-aware structure to translate into linguistically coherent pretokens.

This synergy is what distinguishes SCRIPT-BPE from alternative approaches that address only one layer: MYTE improves the byte encoding but doesn't change pretokenization; Boundless BPE removes whitespace pretokenization but keeps byte encoding; jamo-level tokenization improves Korean but doesn't generalize. SCRIPT-BPE addresses encoding, pretokenization, and merging constraints simultaneously through a unified structural change — the introduction of script and category awareness at the representation level — that cascades through the pipeline.

The paper's explicit acknowledgment that the components can be decoupled (constrained merging works with byte-BPE; SCRIPT encoding works with regex pretokenization, as shown in the Table 2 ablation) demonstrates that the design is modular. But the full SCRIPT-BPE system — SCRIPT encoding + rule-based pretokenization + constrained merging — is presented as the combination that best realizes the design goals of robustness, language parity, and competitive compression.

4. Key Insights and Innovations

Innovation 1: Unicode Metadata as Representational Primitive Instead of Post-Hoc Patch

The paper's most distinctive intellectual move is treating Unicode script and category properties not as auxiliary information to be consulted when debugging tokenizer failures, but as the foundational representation from which encoding, pretokenization, and merging constraints all derive. This inverts the relationship between character identity and encoding: instead of starting with a byte-stream encoding (UTF-8) and then struggling to recover linguistic structure through pretokenization heuristics, SCRIPT-BPE starts with linguistically structured metadata and builds the encoding around it.

Prior to this work, the dominant paradigm — byte-level BPE with regex pretokenization — implicitly assumed that the encoding layer's job is to convert characters to bytes efficiently, and that any script-awareness or language-sensitivity should be handled by the pretokenizer sitting on top. The consequence is that the encoding and pretokenization are adversaries: the encoding fragments characters into variable-length byte sequences that obscure script boundaries, and the pretokenizer burns complexity trying to reconstruct those boundaries from byte patterns (witness the o200k regex's elaborate handling of non-ASCII characters). This adversarial relationship produces the failure modes the paper documents — byte premiums, partial-UTF-8 tokens, diacritic splitting — none of which are inherent to the task of tokenization but are artifacts of a design that separates representation from linguistic structure.

SCRIPT-BPE makes the counter-argument that encoding and pretokenization should be allies. By building the encoding directly from Unicode metadata (script + category → block token + index token), the representation makes script identity explicit at the token level. The pretokenizer then doesn't need to infer script from bytes — it simply compares adjacent block tokens. The constrained merging constraint similarly operates on a representation where character boundaries are trivially identifiable (block token + index token = a character; anything else hasn't been merged yet). This unification simplifies every downstream component because the encoding actively encodes the information those components need.

This is a fundamental shift, not a refinement. Prior work on tokenizer equity — MYTE's morphology-driven byte encodings (Limisiewicz et al., 2024), jamo-level Korean tokenization (Lee et al., 2025), grapheme-level tokenization (Velayuthan and Sarveswaran, 2025) — all operate within the existing byte-encoding or codepoint-encoding paradigms and introduce domain-specific linguistic knowledge to compensate for the encoding's deficits. Each requires per-language resources (morphological annotations, grapheme inventories, script-specific splitting rules) that don't scale. SCRIPT-BPE achieves similar or better equity improvements through a representation change that requires zero per-language engineering because the metadata comes from Unicode itself — a resource already maintained, already comprehensive, and already covering essentially all written languages.

The significance of this move extends beyond compression numbers. It demonstrates that the Unicode standard already contains the information needed for script-fair tokenization; the failure of current tokenizers is not a limitation of available metadata but a consequence of choosing an encoding (UTF-8 bytes) that discards that metadata, forcing the system to spend complexity reconstructing it. Table 4 provides the quantitative evidence: SCRIPT-BPE with rule-based pretokenization achieves compression competitive with the far more complex o200k regex pattern while eliminating the byte premium entirely (initial tokens/character is uniformly 2.0 for SCRIPT vs. 1.0–3.1 for byte-level encoding across languages). The initial encoding cost is higher for ASCII-range text but the final compressed representation is comparable, indicating that the BPE process can effectively compensate for the initial cost difference when the encoding provides better structural information to guide merges.

Innovation 2: Character-Integrity Constraints as a Universally Beneficial, Zero-Cost Intervention

The constrained BPE merging strategy is presented as a secondary contribution, but its empirical universality makes it arguably the paper's most actionable finding. The constraint — never merge tokens that would create a mix of partial and complete characters — is a simple rule with no downside: it universally eliminates partial-UTF-8-sequence tokens (reducing them to zero in all configurations tested, Table 2) while generally improving compression (tokens/character decreases in the majority of configurations, with the worst-case being a fraction of a percent degradation). This is rare in systems design: a constraint that simultaneously improves both a quality metric (token integrity) and a performance metric (compression) without requiring any new data, any additional training cost (Table 3 shows training time is comparable, and actually decreases for SCRIPT-based tokenizers), or any inference-time changes.

The field's prior stance on this problem was essentially complacency with a known defect. Land and Bartolo (2024) had documented that partial-UTF-8 tokens exist and can become under-trained, but the implied solution was to detect and handle these tokens post-hoc rather than prevent their creation. The GPT-4o tokenizer has 874 such tokens; the Thai dataset with o200k pretokenization produces 42,831 — orders of magnitude worse, but the default response has been to accept these as an unavoidable consequence of byte-level BPE's flexibility. The paper reframes this: partial-character tokens are not an unavoidable cost of BPE but a preventable design error, eliminated by a constraint that costs nothing and helps compression.

Why does constraining merges improve compression? The paper's data suggests an interpretation that is intellectually significant beyond the empirical result: unconstrained BPE overfits to local byte-pair frequencies at the expense of representational compositionality. When a space merges with the first byte of a common Thai character, the BPE algorithm is optimizing a short-sighted objective — that specific adjacent pair is frequent — but produces a token that is less reusable because it encodes a cross-category mixture (a word boundary separator combined with a fragment of an unrelated character). The constraint prevents these premature merges, keeping the representation more compositional: spaces stay as spaces, character bytes merge within the character to form complete character tokens, and only then can character tokens merge with other character tokens to form linguistically meaningful subword units. The resulting merges are more transferable across contexts, which manifests as better compression. This is a diagnostic finding: it reveals that BPE's greedy optimization objective is misaligned with the goal of producing a vocabulary of reusable, compositionally meaningful tokens, and that a small structural constraint on the optimization space corrects this misalignment.

The practical significance is amplified by the constraint's universality. It works for byte-level BPE (with any regex pretokenizer) and for SCRIPT-BPE (with any pretokenizer). It requires no language-specific knowledge, no additional training data, and no hyperparameter tuning. The paper's recommendation — "adopt constrained merging, particularly for massively multilingual tokenizers" (Section 5) — is backed by comprehensive evidence and represents a directly actionable improvement to the dominant tokenization paradigm that any practitioner can implement. This is not a research contribution that requires follow-up validation; it's a finding that is validated within the paper's own experiments and can be deployed immediately.

Innovation 3: Script-Fair Encoding Without Per-Language Engineering — A Scalability Argument

A thread running through the paper is that language-specific tokenization improvements don't scale, and that this scalability failure is itself a form of inequity. Prior work has demonstrated that linguistically informed tokenization improves performance for specific languages — jamo-level Korean (Lee et al., 2025), grapheme-level Tamil/Sinhala/Hindi (Velayuthan and Sarveswaran, 2025), morphology-driven encodings (Limisiewicz et al., 2024) — but each requires resources (morphological analyzers, grapheme segmenters, syllable decomposers) that exist for only a subset of the world's languages. A tokenizer that works beautifully for Korean but is unusable for Wolof or Quechua because the necessary linguistic resources don't exist is not equitable; it merely shifts the inequity from one set of languages to another.

SCRIPT-BPE's innovation on this dimension is not that it outperforms language-specific approaches for any particular language — it almost certainly doesn't beat a dedicated jamo-level tokenizer on Korean — but that it achieves competitive compression across all languages simultaneously using only information available in the Unicode standard. The Unicode script and category properties are defined for every encoded character in every script; there is no "low-resource script" problem equivalent to the low-resource language problem for morphological analyzers. The total manual intervention required is trivially small: three specific character reassignments (newline and tab to Separator, prolonged sound marks to Inherited, Tatweel to Arabic) and one script-specific pretokenization rule (Hiragana-Han merging). Everything else is derived automatically from Unicode metadata.

This is a reframing of the tokenization equity problem from "how do we design better tokenizers for language X?" to "how do we design a tokenizer that is equally fair to all languages without requiring per-language intervention?" The answer SCRIPT-BPE provides is: build the tokenizer around structured metadata that already exists uniformly across all scripts, rather than around an encoding (UTF-8 bytes) that introduces differential costs and then trying to compensate through language-specific patches.

Table 4 provides the empirical anchor for this argument. SCRIPT-BPE with rule-based pretokenization achieves tokens/character ratios that are within 5% of the best-performing configuration (usually o200k regex) for most languages, while substantially outperforming cl100k on languages where the regex splits diacritics (Thai: 3.60 for SCRIPT vs. 5.72 for cl100k; Hindi: 2.58 vs. 4.29; Punjabi: 2.46 vs. 3.69). The SCRIPT approach doesn't dominate o200k's carefully tuned regex on every language — Chinese and Thai show some compression disadvantage — but it achieves this near-parity without any language-specific tuning, using a pretokenization algorithm whose entire logic fits in a paragraph. The scalability argument is that this approach would work identically for the 150+ languages in CulturaX, including those for which no regex has ever been optimized, because it relies on metadata that exists for all of them.

This insight connects to a broader principle in multilingual NLP: uniformity of representation can be more important than optimality for any specific case. A tokenizer that is slightly suboptimal for English but equally fair across 100 languages may produce a better overall multilingual model than a tokenizer that is optimized for the 20 languages with the most engineering resources and degrades sharply for the remaining 80. The paper doesn't prove this empirically (downstream model training is left to future work), but it provides the representational foundation for testing this hypothesis.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses two dataset configurations. For monolingual tokenizer training, 300 MB subsets are drawn from the Goldfish project (Chang et al., 2024), which provides monolingual corpora for 350 languages, for each of 12 languages: Japanese, Chinese, Thai, Punjabi, Hindi, Korean, Russian, Arabic, Hebrew, Vietnamese, German, and English. For multilingual tokenizer training, a 35 GB subsample of CulturaX (Nguyen et al., 2023) is used — a cleaned, 167-language web-crawled dataset — with a separate 136 GB disjoint subset serving as the validation set for compression evaluation. The 12-language monolingual selection spans scripts with varying UTF-8 byte costs (1-byte Latin/German, 2-byte Russian/Arabic/Hebrew, 3-byte CJK/Thai/Hindi/Punjabi/Vietnamese, and mixed-width Korean) and diverse orthographic conventions, providing coverage of the byte premium effect across its full range.

Base model(s). No language model is trained or evaluated in this paper. The experiments are conducted entirely at the tokenizer level — the "models" being compared are tokenizer configurations (SCRIPT-BPE variants vs. byte-level BPE baselines), and the evaluation measures the properties of the resulting token vocabularies and their compression behavior on text corpora. This is explicitly scoped in Section 5: "This preliminary evaluation focused primarily on compression; however, this metric alone does not necessarily guarantee better downstream model performance." The paper frames downstream model training as the essential next step. The tokenizer training implementation is a custom Python implementation (not tiktoken or SentencePiece), run on a "moderately parallel setup with 16 CPUs" (Section 4.2), with all multilingual 256K-merge tokenizers training in approximately one hour.

Metrics. The primary metric is tokens per character: total tokens produced when encoding a text corpus divided by the total number of Unicode characters in that corpus. Lower values indicate better compression — less context window consumed per unit of text. The secondary metric is initial tokens per character: the token-to-character ratio before any BPE merges are applied, measuring the inherent cost of the encoding scheme independent of how well BPE compresses the language. A quality metric is number of tokens with partial UTF-8 sequences (Table 2), defined as tokens representing "a mix of full and partial character sequences" — tokens where some bytes belong to a complete character and others belong to an incomplete character from an adjacent character. This metric captures a specific known defect of unconstrained byte-level BPE documented by Land and Bartolo (2024). Training time in hours (Table 3) serves as an efficiency metric. All metrics are reported on validation sets (the 136 GB CulturaX split for multilingual tokenizers; the 300 MB training corpora for monolingual), with monolingual training-set performance provided in Appendix C (Table 5) for completeness.

Baselines. Four baseline configurations are tested, all using tiktoken's BPE implementation with UTF-8 byte-level encoding: (1) cl100k base — the GPT-4 pretokenization regex with unconstrained BPE merging, (2) cl100k + constrained — same regex with the character-integrity merging constraint applied, (3) o200k base — the GPT-4o pretokenization regex with unconstrained BPE merging (this is the more recent, more extensively tuned regex pattern), and (4) o200k + constrained — same regex with constrained merging. For SCRIPT-BPE, the baselines tested against these byte-level configurations are: (1) SCRIPT + rule-based pretokenization + unconstrained merging, (2) SCRIPT + rule-based + constrained merging, (3) SCRIPT + o200k regex + unconstrained merging (as an ablation isolating the encoding from the pretokenization), and (4) SCRIPT + o200k regex + constrained merging.

Generation budget / compute accounting. There is no generation budget in this paper — no text is being generated, no model inferences are being performed. The relevant "compute" measure is BPE training time (Table 3), reported in hours for the multilingual 256K-merge configuration on 16 CPUs. This includes all stages after pretokenization and initialization — the BPE merge counting and vocabulary construction. Training time ranges from 0.58 hours (SCRIPT rule-based with constrained merging) to 1.17 hours (byte-level cl100k with constrained merging), establishing that no configuration is computationally prohibitive. The paper explicitly notes that these times are measured for a custom Python implementation and that optimized library implementations (tiktoken in Rust, SentencePiece in C++) would likely be faster.

Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. The evaluation is a direct comparison of compression ratios and partial-UTF-8 token counts on held-out validation sets. For the monolingual setting, compression is reported on the training data itself (Table 5) rather than on a held-out split, which limits the ability to assess overfitting — though tokenizer overfitting is generally less of a concern than model overfitting, since BPE is a deterministic frequency-based algorithm operating on text statistics rather than a learned function. The 136 GB multilingual validation set is large enough that sampling variance in the tokens/character metric is likely negligible, though this is not quantified.

Main Quantitative Results

The experimental results are organized around three axes: (1) the effect of constrained merging on compression and token quality across all encoding/pretokenizer combinations, (2) training time comparisons, and (3) compression performance of the full SCRIPT-BPE system versus byte-level baselines across 12 languages.

Constrained Merging Universally Eliminates Partial-UTF-8 Tokens and Generally Improves Compression

The headline result from Table 2 is that applying the character-integrity constraint to BPE merging reduces the count of tokens with mixed full-and-partial character sequences to zero for every encoding and pretokenizer combination tested, while improving compression in 7 out of 8 configurations and matching it in the 8th.

Table 2 reports mean tokens per character and mean count of partial-UTF-8 tokens across the monolingual training corpora for the 12-language set, comparing constrained (✓) versus unconstrained (×) merging for four base configurations:

Byte-level BPE with cl100k regex: Unconstrained produces a mean of 1,571 partial-UTF-8 tokens and a compression ratio of 2.196 tokens/character. Constrained eliminates all partial-UTF-8 tokens (0) and improves compression to 2.163 tokens/character — a reduction of 0.033 tokens per character, or approximately 1.5% better compression, while simultaneously eliminating the token quality defect.

Byte-level BPE with o200k regex: This is where the cascade failure is most dramatic. Unconstrained produces a mean of 42,831 partial-UTF-8 tokens — more than 27× the cl100k count — driven by the specific interaction of the o200k regex with Thai text. The paper explains that "several early merges between common characters and two leading UTF-8 bytes <0xE0><0xB8>" initiate a cascade, producing tokens mixing spaces and partial Thai characters that then participate in further merges. Constrained merging eliminates all 42,831 such tokens (reducing the count to 0) and improves compression from 3.263 to 3.231 tokens/character — a modest 1.0% compression improvement, but accompanied by the elimination of a massive token quality defect. This is the strongest single piece of evidence for constrained merging: a simple constraint prevents a catastrophic failure mode while slightly improving the primary performance metric.

SCRIPT-BPE with rule-based pretokenization: Unconstrained produces 153 tokens with partial-character mixing (not partial UTF-8, since SCRIPT doesn't use UTF-8 bytes, but tokens that mix fragments of different characters — e.g., an index token from one character merged with the block token of the next character, analogous to the UTF-8 problem). Constrained eliminates all such tokens (0) and improves compression from 2.306 to 2.290 tokens/character.

SCRIPT-BPE with o200k regex: Unconstrained produces 134 partial-character-mixing tokens. Constrained eliminates these (0) and produces essentially identical compression: 2.767 unconstrained vs. 2.763 constrained — a difference of 0.004 tokens/character, which is effectively flat.

The key pattern: constrained merging never makes compression meaningfully worse. The worst observed change across all 8 configurations is the flat result for SCRIPT + o200k regex. Every other configuration shows improvement, ranging from modest (~1%) to substantial when measured against the elimination of a large token quality defect. The paper's claim that constrained merging "universally eliminated tokens representing a mix of full and partial characters and generally improved compression across different base encodings" (Section 5) is fully supported by Table 2 — the improvement is universal, with the caveat that one configuration shows effectively zero change rather than a gain.

An important note on interpreting Table 2: the "Tokens/Char" values are reported as mean compression on the training corpora for the monolingual tokenizers. The paper states "As differences in compression are generally small, we present only the constrained versions in all subsequent results" (Section 4.1). This means Tables 3, 4, and 5 report only the constrained configurations going forward — the unconstrained results in Table 2 serve as the ablation establishing that constrained merging is strictly preferable.

Training Time Is Comparable Across Configurations and Not a Bottleneck

Table 3 reports training time in hours for the multilingual tokenizer with 256K merges, excluding pretokenization and initialization time. The key comparisons:

Byte-level tokenizers: Constrained merging slightly increases training time — from 1.04 to 1.17 hours for the cl100k regex (a 12.5% increase), and from 0.87 to 0.93 hours for the o200k regex (a 6.9% increase). The paper attributes this to "the more complex character boundary checks required for UTF-8 encoding compared to SCRIPT" (Section 4.2) — determining whether a UTF-8 byte sequence represents a complete or partial character requires examining the byte values against the UTF-8 encoding specification, which adds computational overhead.

SCRIPT tokenizers: Constrained merging reduces training time — from 0.63 to 0.58 hours for rule-based pretokenization (an 7.9% decrease), and from 1.01 to 0.98 hours for o200k regex pretokenization (a 3.0% decrease). The paper explains that constraining merges "reduces the size of internal data structures used in training by limiting the search space" (Section 4.2). Since SCRIPT encoding makes character boundaries trivially identifiable (two tokens = one character; the first must be a block token and the second an index token), the constraint check is cheap, and the elimination of cross-character merge candidates reduces the number of token pairs that must be tracked and sorted.

Absolute training times: All configurations train in approximately one hour — ranging from 0.58 to 1.17 hours — for 256K merges on the 35 GB multilingual dataset with 16 CPUs. The paper states this is "ensuring that training time is not a bottleneck" (Section 4.2). While 16 CPUs is a modest compute allocation by modern standards, the custom Python implementation likely understates what an optimized library could achieve; the paper's point is that even an unoptimized implementation completes in ~1 hour, so training cost is not a barrier to adoption.

The training time results are reported for the multilingual 256K-merge configuration only; monolingual 64K-merge times are not provided, though they would presumably be proportionally faster given the smaller corpus and fewer merges.

SCRIPT-BPE Achieves Competitive Compression While Eliminating Byte Premiums

Table 4 is the central compression results table, reporting performance of the multilingual tokenizer (256K merges) on its training set and on the 136 GB multilingual validation set, broken out by language. The table reports both initial tokens/character (encoding cost before any BPE merges) and final tokens/character (after 256K BPE merges) for three byte-level configurations (cl100k bytes, o200k bytes, o200k bytes with constrained merging) and four SCRIPT configurations (rule-based pretokenization, rule-based + constrained, o200k regex, o200k regex + constrained). Values worse than 5%, 10%, and 20% from the best in each row are highlighted with progressively darker shading.

Initial Encoding Cost: The Byte Premium Effect Quantified

The "Initial Tokens/Char" column quantifies the byte premium effect precisely. For byte-level encoding, the initial cost varies dramatically by language, directly reflecting the average UTF-8 bytes per character in each language's script:

English and German (primarily Latin script, mostly ASCII): 1.0 tokens/character — essentially every character is a single byte.
Russian (Cyrillic, 2-byte characters): 2.0 tokens/character.
Arabic and Hebrew (2-byte characters): 2.0 tokens/character.
Vietnamese (Latin-based but with extensive diacritic marks, many multi-byte characters): 1.2 tokens/character — intermediate due to the mix of 1-byte and 2-byte characters.
Chinese, Japanese, Korean (3-byte CJK characters): 3.0, 3.0, and 3.0 tokens/character respectively. The Korean figure is 3.0 despite Hangul syllable blocks being precomposed in Unicode — each syllable is a single 3-byte UTF-8 sequence.
Thai, Hindi, Punjabi (3-byte characters): 3.0, 3.1, and 3.0 tokens/character respectively.
The "All" row reports a weighted average of 1.7 tokens/character across the full multilingual training and validation sets, reflecting the mixture of scripts in CulturaX.

For SCRIPT encoding, the initial cost is uniformly 2.0 tokens/character for every language, since every character is exactly two tokens (block + index) regardless of script, language, or UTF-8 byte length. This means:

For English and German, SCRIPT has a higher initial cost (2.0 vs. 1.0) — it effectively doubles the pre-BPE token count for ASCII-range text.
For Russian, Arabic, and Hebrew, SCRIPT matches the byte-level initial cost (2.0 vs. 2.0) — no penalty, no advantage at initialization.
For Chinese, Japanese, Korean, Thai, Hindi, and Punjabi, SCRIPT has a substantially lower initial cost (2.0 vs. 3.0–3.1) — a 33% reduction in pre-BPE tokens for 3-byte scripts.
For Vietnamese, SCRIPT has a higher initial cost (2.0 vs. 1.2) but a much smaller relative penalty than for English.

This initial cost pattern is the direct manifestation of the byte premium effect: byte-level encoding charges a 3× structural tax on CJK and Indic scripts relative to English before any training occurs. SCRIPT encoding charges a uniform 2× tax on all scripts. Whether this initial cost difference translates to final compression differences depends on how effectively BPE merges can compress from these different starting points.

Final Compression: Script-Aware vs. Regex-Aware Pretokenization

The "Final Tokens/Char (Multilingual Validation)" columns in Table 4 provide the head-to-head comparison after 256K BPE merges. The patterns are language-dependent and reveal where SCRIPT's structural advantages overcome its higher initial cost for Latin scripts.

The paper uses a highlighting scheme: values >5% worse than the best in each row (language) are lightly highlighted, >10% are medium-highlighted, and >20% are dark-highlighted. This makes it easy to identify catastrophic failures. The key comparisons:

English and German (Latin-dominant, byte-level advantage at initialization):

English: o200k + constrained achieves 1.55 tokens/character. SCRIPT rule-based + constrained achieves 1.83 — approximately 18% worse (medium-highlighted, since it exceeds 10% but not 20%). The initial 2× token disadvantage for SCRIPT (2.0 vs. 1.0) is partially but not fully compensated by BPE merges. English is the language where SCRIPT-BPE shows the largest compression disadvantage, which is expected: the byte-level encoding's 1-byte-per-character representation for ASCII text is essentially optimal for English, and SCRIPT's fixed 2-token encoding necessarily has a higher floor that BPE can only partially offset.
German: Similar pattern — o200k + constrained achieves 1.64, SCRIPT rule-based + constrained achieves 1.87 (~14% worse, medium-highlighted). The slightly smaller gap versus English may reflect German's higher frequency of non-ASCII characters (umlauts, ß) that cost 2 bytes in UTF-8, reducing byte-level's relative advantage.

Russian, Arabic, Hebrew (2-byte scripts, equal initial cost):

Russian: o200k + constrained achieves 1.76, SCRIPT rule-based + constrained achieves 1.90 (~8% worse, lightly highlighted). The gap is smaller than for English, reflecting the equal starting point (2.0 vs. 2.0 initial tokens/character).
Arabic: o200k + constrained achieves 1.33, SCRIPT rule-based + constrained achieves 1.33 — identical compression. This is a significant result: for Arabic, SCRIPT-BPE matches the best byte-level configuration despite having no language-specific optimization, while the byte-level tokenizer benefits from the o200k regex's extensive tuning.
Hebrew: o200k + constrained achieves 1.49, SCRIPT rule-based + constrained achieves 1.63 (~9% worse, lightly highlighted). The gap is modest.

Vietnamese (Latin diacritic-heavy, intermediate initial cost):

o200k + constrained achieves 1.27, SCRIPT rule-based + constrained achieves 1.59 (~25% worse, dark-highlighted). Vietnamese is a notable weakness for SCRIPT-BPE. Despite having many multi-byte characters (2 bytes for diacritic-bearing Latin characters like ắ, ệ, ố), the byte-level tokenizer achieves excellent compression (1.27 is among the best final ratios in the table, comparable to English). The SCRIPT encoding's 2.0 initial cost (vs. 1.2 for byte-level) is not adequately compensated by BPE merges, resulting in the largest percentage gap in the table. The paper does not discuss this case specifically.

Chinese and Japanese (3-byte scripts, SCRIPT initial advantage):

Chinese: o200k + constrained achieves 1.58, SCRIPT rule-based + constrained achieves 1.81 (~15% worse, medium-highlighted). Despite starting with a 33% lower initial cost (2.0 vs. 3.0), SCRIPT-BPE ends up with moderately worse final compression. The paper notes that this "may be attributed to mixed Chinese/Latin phrases often found in web data (e.g. spam or advertisements) and the prevalence of non-standard use of spaces" (Section 4.3). The rule-based pretokenizer groups Latin and Han characters separately (different scripts), which may produce fragmented pretokens when Latin and Han characters are interleaved within what is functionally a single Chinese phrase. The o200k regex, by contrast, may handle these mixed-script sequences more gracefully through its whitespace-based splitting rules. This is a genuine tradeoff: the script-aware pretokenization that prevents diacritic splitting on Indic scripts creates fragmentation on mixed-script sequences that are common in Chinese web text.
Japanese: o200k + constrained achieves 2.99, SCRIPT rule-based + constrained achieves 2.60 — SCRIPT-BPE is substantially better (~13% improvement). This is the strongest positive result for SCRIPT-BPE among the high-byte-count scripts. The Hiragana-Han merging rule (Rule 4 of the pretokenizer) likely plays a key role here: by keeping Kanji and Hiragana together within pretokens, SCRIPT-BPE preserves Japanese word boundaries that the regex-based pretokenizer fragments. This is the one place where SCRIPT's script-specific pretokenization rule provides a clear advantage over the general-purpose regex.

Thai, Hindi, Punjabi (3-byte scripts where regex pretokenization catastrophically splits diacritics):

Thai: cl100k (without constrained merging) achieves 5.72 tokens/character — dark-highlighted as >20% worse than the best. The o200k + constrained achieves 1.36. SCRIPT rule-based + constrained achieves 1.96 (~44% worse than o200k, heavily dark-highlighted). However, this comparison is misleading if read as "SCRIPT is bad for Thai." The cl100k result (5.72) represents the catastrophic diacritic-splitting failure mode discussed in Section 2. The o200k regex has been specifically tuned to handle Thai better (achieving 1.36 — excellent compression). SCRIPT-BPE at 1.96 is substantially better than the cl100k failure mode and competitive in absolute terms, but significantly worse than the highly optimized o200k. The paper frames this as the o200k regex being specifically advantaged for Thai, and notes that SCRIPT-BPE's rule-based pretokenization "can lag behind the more complex 'o200k' pretokenization pattern in specific cases such as Chinese and Thai" (Section 4.3).
Hindi: cl100k achieves 4.29 (dark-highlighted catastrophe). o200k + constrained achieves 1.78. SCRIPT rule-based + constrained achieves 2.13 (~20% worse than o200k, medium-highlighted but far better than cl100k). The pattern is similar to Thai: SCRIPT-BPE avoids the regex-induced diacritic-splitting failure but doesn't match the extensively optimized o200k pattern.
Punjabi: cl100k achieves 3.69 (dark-highlighted). o200k + constrained achieves 1.67. SCRIPT rule-based + constrained achieves 1.97 (~18% worse than o200k, but nearly 2× better than cl100k).

Korean (3-byte Hangul syllables):

o200k + constrained achieves 2.28, SCRIPT rule-based + constrained achieves 3.13 (~37% worse, dark-highlighted). Korean is SCRIPT-BPE's worst-performing language in relative terms. The paper does not discuss this result specifically, but the likely explanation is that Korean Hangul syllable blocks — while each being a single Unicode character (costing 3 bytes in UTF-8) — are composed of individual jamo (consonant and vowel components) that follow regular patterns. The byte-level BPE can discover these sub-syllabic patterns through frequent byte-pair merges, achieving good compression despite the 3-byte initial cost. SCRIPT-BPE treats each syllable as an atomic character (one block+index pair), and the subsequent BPE merges must rediscover the jamo-level regularities — but within a representation that doesn't expose the internal structure of the syllables. Lee et al. (2025) demonstrated that explicit jamo-level tokenization improves Korean performance, and SCRIPT-BPE's character-level granularity is actually coarser than optimal for Korean. This is an acknowledged limitation: SCRIPT-BPE's character boundaries, while addressing the byte premium problem, lose the sub-character structure that byte-level BPE exploits for composed scripts like Hangul.

The "All" row: Across the full multilingual validation set, the best overall compression is achieved by o200k + constrained at 1.55 tokens/character (on the training set) and comparable values on the validation set. SCRIPT rule-based + constrained achieves 1.84 on training and 1.90 on validation — approximately 19–23% worse, falling in the medium-highlighted range. However, this aggregate comparison masks the distribution: SCRIPT-BPE is competitive or better on Japanese, Arabic, and (in absolute terms) on the Indic scripts where cl100k fails catastrophically; it is meaningfully worse on English, Vietnamese, Chinese, and Korean.

The Gap Between Training and Validation Compression

Table 4 reports compression on both the training set and the validation set. For most configurations and languages, the training and validation compression ratios are very similar (typically within 0.02–0.05 tokens/character), indicating that the BPE-derived vocabularies generalize well — merges learned from the 35 GB training sample transfer effectively to the 136 GB held-out data. This is expected for BPE, which learns from corpus-level statistics rather than individual examples, but it is a useful sanity check that the CulturaX subsample is representative.

Monolingual Tokenizer Results (Appendix C, Table 5)

Table 5 in Appendix C reports final compression ratios for monolingual tokenizers (64K merges, trained on 300 MB per language) on their training data. The patterns largely mirror those in the multilingual validation results:

English: o200k achieves 1.61, SCRIPT rule-based achieves 1.99 (~24% worse).
Chinese: o200k achieves 1.72, SCRIPT rule-based achieves 2.06 (~20% worse).
Japanese: o200k achieves 2.09, SCRIPT rule-based achieves 1.84 (~12% better) — consistent with the multilingual result where SCRIPT outperforms on Japanese.
Thai: o200k achieves 1.19, SCRIPT rule-based achieves 2.75 (~131% worse) — the monolingual gap is much larger than in the multilingual setting, likely because the 300 MB Thai-specific training data allows the o200k regex to fully optimize for Thai patterns without the interference of mixed-script data.
Hindi: o200k achieves 1.55, SCRIPT rule-based achieves 2.40 (~55% worse).
Korean: o200k achieves 2.19, SCRIPT rule-based achieves 3.11 (~42% worse).

The monolingual results confirm that SCRIPT-BPE's compression disadvantage is consistent across training scales and is largest for Vietnamese, Korean, Thai, and Hindi — all languages where either the byte-level encoding has a substantial initial cost advantage (Vietnamese, 1.2 vs. 2.0) or the byte-level BPE can exploit sub-character regularities that SCRIPT's character-level encoding masks (Korean syllables, Thai character composition).

Ablation Studies and Robustness Checks

Pretokenization choice with SCRIPT encoding (Table 4, comparing rule-based vs. o200k regex within SCRIPT rows): Using the o200k regex for pretokenization with SCRIPT encoding produces generally worse compression than the rule-based pretokenizer for most languages — English (2.07 vs. 1.83), Chinese (2.32 vs. 1.81), Thai (2.25 vs. 1.96), Hindi (2.70 vs. 2.13), Korean (3.18 vs. 3.13), and the aggregate (2.02 vs. 1.84 training, 2.09 vs. 1.90 validation). This ablation demonstrates that the SCRIPT encoding's benefits are partially negated when combined with a regex pretokenizer designed for byte-level encoding: the regex splits characters in ways that conflict with the script-aware structure that the SCRIPT encoding provides, suggesting that the encoding and pretokenization are co-designed and should be used together for optimal results. The rule-based pretokenizer is not just a simpler alternative to regex; it is specifically adapted to leverage the script and category information that the SCRIPT encoding makes explicit.

Constrained merging within SCRIPT-BPE (Table 4, comparing SCRIPT rule-based vs. SCRIPT rule-based + constrained): The differences in compression between unconstrained and constrained merging for SCRIPT-BPE are small across languages. For the multilingual validation set, constrained merging changes compression from 1.84 to 1.90 in the "All" row — a small degradation. However, Table 2 establishes that unconstrained SCRIPT-BPE produces 153 tokens with partial-character mixing, which constrained merging eliminates. The compression change is a tradeoff: eliminating 153 defective tokens at the cost of a 0.06 tokens/character compression increase (~3% relative). This is a favorable tradeoff given that the defective tokens are known to cause under-training issues (Land and Bartolo, 2024), even though the compression impact is slightly negative rather than positive as seen in the byte-level case.

Regex baseline comparison with and without constrained merging (Table 4, comparing o200k bytes vs. o200k bytes + constrained): For most languages, the difference between o200k with and without constrained merging is negligible in the final compression (typically <0.05 tokens/character). The key difference is in token quality: unconstrained o200k produces the massive 42,831 partial-UTF-8 tokens documented in Table 2, while constrained eliminates them. The fact that compression is essentially unchanged while eliminating a known defect is exactly the "no downside" argument the paper makes for universal adoption.

Monolingual vs. multilingual tokenizer compression pattern consistency (Table 4 vs. Table 5): The relative ordering of methods is broadly consistent between the multilingual tokenizer evaluated on monolingual validation sets (Table 4) and monolingual tokenizers evaluated on their training data (Table 5). Japanese shows SCRIPT-BPE outperforming byte-level in both settings; Korean shows SCRIPT-BPE substantially worse in both; English shows SCRIPT-BPE worse in both. This consistency across different training data sizes (35 GB multilingual vs. 300 MB monolingual) and vocabulary sizes (256K vs. 64K merges) suggests the patterns are robust to scale, at least within the range tested. However, the monolingual tokenizers are evaluated only on their training data (not a held-out validation set), so overfitting cannot be ruled out for the 64K monolingual results.

Script-specific effects of the rule-based pretokenizer (Table 4, comparing SCRIPT rule-based constrained across languages): The variation in SCRIPT-BPE's compression across languages reveals which scripts benefit most from the encoding+pretokenization combination. Japanese (2.60), Arabic (1.33), and—in absolute terms despite being worse than o200k—the Indic scripts (Hindi 2.13, Punjabi 1.97, Thai 1.96) achieve compression ratios that, while not always best-in-class, are within a usable range without catastrophic failure. English (1.83), German (1.87), Russian (1.90), and Hebrew (1.63) show moderate compression that is somewhat worse than o200k but not dramatically so. Vietnamese (1.59, but o200k achieves 1.27) and Korean (3.13, o200k achieves 2.28) are the clear weaknesses. This pattern serves as a diagnostic: the scripts where SCRIPT-BPE underperforms most are those where either (a) the byte-level encoding has a very large initial cost advantage (Vietnamese 1.2 vs. 2.0 initial tokens/character) that BPE cannot overcome, or (b) sub-character regularities that byte-level BPE exploits are hidden by SCRIPT's character-level granularity (Korean Hangul syllables).

Critical Assessment

The experiments provide substantial evidence for the paper's operational claims about tokenizer compression and token quality, but they also reveal important boundaries on what has been demonstrated and what remains to be shown.

On the claim that constrained merging is universally beneficial: This is the most robustly supported claim in the paper. Table 2 demonstrates that for all four base configurations × two constraint settings, the constrained variant eliminates partial-character-mixing tokens (reducing them to zero in every case) while either improving compression (7 of 8 configurations) or matching it (1 configuration). The improvement is not always large — the compression difference ranges from 0.001 to 0.033 tokens/character — but the direction is consistently favorable or neutral, and the elimination of thousands of defective tokens provides a quality benefit that compression metrics don't capture. The universality of this finding across different encodings (byte and SCRIPT), different pretokenizers (cl100k, o200k, rule-based), and different languages strengthens it considerably. The caveat is that compression is measured only on the training corpora in Table 2; while the close correspondence between training and validation compression in Table 4 suggests this would generalize, a direct validation-set measurement of the constrained vs. unconstrained comparison would have been more rigorous.

However, the paper's framing often implies that constrained merging "generally improved compression," which is true but somewhat overstates the magnitude. The improvements are small — typically 1–3% relative — and one configuration (SCRIPT + o200k regex) shows essentially zero change. The more honest characterization is "constrained merging eliminates a known defect at negligible or slightly positive compression cost." This is still a strong recommendation for adoption, but the compression improvement is a minor bonus, not the primary motivation.

On the claim that SCRIPT-BPE achieves competitive compression while eliminating encoding-based penalties: The evidence supports this claim with important qualifications that the paper acknowledges but does not fully characterize. "Competitive" needs to be defined relative to baselines: if the baseline is the cl100k regex (GPT-4), SCRIPT-BPE is substantially better on many languages and comparable on others. If the baseline is the o200k regex (GPT-4o), SCRIPT-BPE is worse on most languages by margins of 10–40% relative, with Japanese and Arabic as the notable exceptions where it matches or exceeds. The paper's highlighting scheme in Table 4 makes this visible — SCRIPT rule-based + constrained has 5 languages in the dark-highlighted (>20% worse) category compared to o200k + constrained, and only Japanese is substantially better.

The "eliminating encoding-based penalties" claim is true at the initial encoding level (uniform 2.0 tokens/character vs. 1.0–3.1 for byte-level) but is partially offset by the BPE merging process. Languages where byte-level encoding has the largest initial penalty (Chinese 3.0, Japanese 3.0, Thai 3.0) do not consistently show better final compression under SCRIPT — Chinese is worse, Japanese is better, Korean (also 3.0 initial) is much worse. The relationship between initial encoding equity and final compression is not straightforward; BPE merges interact with pretokenization in language-specific ways that can either compensate for or exacerbate the initial cost differences.

The paper's title and framing emphasize SCRIPT-BPE as achieving "competitive compression," which is accurate if "competitive" means "within a usable range and far better than the catastrophic failures of unoptimized regex tokenizers." It would be misleading if interpreted as "matching or exceeding the best available tokenizers across languages." The paper itself does not make this stronger claim — Table 4 honestly reports the gaps — but casual readers might infer it from the abstract's "competitive compression" language.

What is not tested and limits the strength of conclusions:

Downstream model performance is entirely unmeasured. This is the most significant gap, and the paper is explicit about it: "In future work, we aim to train language models using both the SCRIPT and the constrained merging strategy and evaluate their effects on model performance" (Section 5). Compression is a proxy metric — Schmidt et al. (2024) demonstrated that tokenization choices affecting compression do not necessarily translate to downstream performance changes in the expected direction or magnitude. For example, removing pretokenization entirely achieves better compression but worse downstream performance (as the paper itself notes). The paper cannot claim that SCRIPT-BPE would produce better or even equivalent language models; it can only claim that it produces competitive compression without encoding-level bias, which is a necessary but not sufficient condition for downstream parity. The 10–40% compression gap versus o200k on many languages might or might not translate to a performance gap — it depends on whether the model can effectively use the differently-structured token sequences.

Evaluation is limited to 12 languages (for monolingual) plus aggregate multilingual metrics. While these 12 cover a range of script types and byte costs, they represent only 7% of the 167 languages in CulturaX. The paper's scalability argument — that SCRIPT-BPE works for any language because Unicode metadata exists for all scripts — is logically sound but empirically untested. Languages with complex orthographies not represented in the 12 (e.g., Khmer with its stacked diacritics, Tibetan with its subjoined consonants, Myanmar with its complex syllable structure) might exhibit different compression behavior that the current experiments cannot predict.

Only one vocabulary size is tested per setting (64K merges for monolingual, 256K for multilingual). The relationship between vocabulary size and the relative performance of SCRIPT vs. byte-level BPE is not explored. It's possible that SCRIPT-BPE catches up to o200k at larger vocabulary sizes (its structured representation might enable more efficient use of additional merges) or that the gap widens (byte-level BPE might have more headroom from its lower initial token count). This is not a weakness of the experiments — vocabulary size sweeps are expensive and the paper's scope is already substantial — but it means the "competitive compression" claim is validated only at the specific vocabulary sizes tested.

Training time comparisons are for a custom Python implementation, not for optimized libraries. The absolute times (~1 hour for multilingual 256K) are clearly not a bottleneck, but the relative comparisons (constrained vs. unconstrained speed) might differ in an optimized implementation where the constraint-checking overhead is a larger fraction of a much smaller total time. This doesn't affect the paper's conclusions — training time is demonstrated to be low enough — but limits the generality of the specific speedup claims.

No statistical measures of uncertainty are reported. Compression ratios on a 136 GB validation set have negligible sampling variance, but the monolingual results on 300 MB training sets (Table 5) might have meaningful variance that is not characterized. The paper treats compression ratio differences as deterministic comparisons, which is reasonable for corpus-level metrics but limits the ability to assess whether small differences (<0.05 tokens/character) are meaningful.

The interaction between pretokenization and encoding is not fully disentangled. The paper compares SCRIPT + rule-based vs. SCRIPT + o200k regex (Table 4), showing that rule-based is generally better, but does not compare SCRIPT + rule-based vs. byte-level + rule-based (which would isolate the encoding effect from the pretokenization effect, keeping pretokenization constant). The o200k regex is designed for byte-level encoding and may contain patterns (like digit grouping) that are suboptimal for SCRIPT but that a regex designed for SCRIPT encoding might handle better. This is a minor point — the paper is advocating for the full SCRIPT-BPE system (encoding + rule-based pretokenization + constrained merging) rather than individual components — but a more complete ablation would strengthen the case that the encoding, not just the pretokenization, drives the benefits.

The Hiragana-Han merging rule is a script-specific intervention that partially undermines the paper's "no per-language engineering" argument. While the rule is simple, explicit, and motivated by well-known properties of Japanese orthography, it represents precisely the kind of language-specific knowledge that the paper criticizes regex pretokenizers for encoding opaquely. The difference is one of degree and transparency — one rule vs. hundreds of regex clauses — but the principle is similar. The paper might have tested a version without the Hiragana-Han rule to determine how much of Japanese's strong performance depends on this rule versus the general script-category grouping. The fact that Japanese is SCRIPT-BPE's best-performing language relative to baselines, combined with the existence of a Japanese-specific rule, raises the question of whether the rule is doing most of the work for that result.

Summary assessment: The experiments convincingly demonstrate that (1) constrained BPE merging is a universally beneficial, zero-cost intervention that should be adopted broadly, (2) SCRIPT-BPE eliminates the byte premium effect at the encoding level, and (3) SCRIPT-BPE achieves compression that is generally competitive (within 10–40% of the best regex-based tokenizers) while being dramatically simpler and not requiring per-language regex tuning. The paper does not demonstrate that SCRIPT-BPE produces better or equivalent downstream language models — this is explicitly left to future work. The compression results alone are sufficient to motivate that downstream evaluation, but they do not constitute proof that SCRIPT-BPE should replace existing tokenizers for model training. The paper's contribution is establishing SCRIPT-BPE as a credible and well-motivated alternative whose primary advantages (encoding equity, simplicity, robustness) are clearly demonstrated, while its primary unknown (downstream performance) is honestly acknowledged.

6. Limitations and Trade-offs

Limitation 1: No Downstream Language Model Evaluation — Compression Is Only a Proxy Metric

The assumption or constraint. The paper evaluates SCRIPT-BPE entirely at the tokenizer level, measuring compression ratios (tokens per character) and token quality metrics (partial-UTF-8 token counts). No language model is trained or evaluated using any tokenizer configuration. The authors are explicit about this scope limitation in Section 5:

"This preliminary evaluation focused primarily on compression; however, this metric alone does not necessarily guarantee better downstream model performance (Schmidt et al., 2024)." And in the same section: "In future work, we aim to train language models using both the SCRIPT and the constrained merging strategy and evaluate their effects on model performance. This is essential for understanding their true impact on downstream task performance, generalization, and fairness at scale."

The paper itself cites evidence that compression and downstream performance do not always correlate: Schmidt et al. (2024) showed that "tokenization is more than compression" and that removing pretokenization entirely improves compression but degrades downstream performance. The relationship between how a tokenizer compresses text and how well a model trained on that tokenizer performs on tasks is not monotonic — tokens that produce better compression may fragment linguistic units in ways that make the model's learning problem harder, or conversely, tokens that produce worse compression numbers may present the model with more semantically coherent units that improve generalization.

The consequence. All of the paper's positive claims — competitive compression, elimination of byte premiums, robustness advantages — are validated only on a proxy metric that is known to be imperfectly correlated with the outcome that actually matters: model quality. The 10–40% compression gap between SCRIPT-BPE and o200k on languages like English, Vietnamese, Korean, and Thai could translate to downstream performance differences of unknown magnitude and direction. It is entirely possible that a model trained with SCRIPT-BPE tokenization would underperform an o200k-tokenized model on English benchmarks despite SCRIPT-BPE's structural advantages for non-Latin scripts, or that SCRIPT-BPE's uniform two-token-per-character representation provides representational benefits that outweigh its compression disadvantages when the model is actually trained. Neither outcome can be ruled out from the current experiments.

The fairness argument is similarly unvalidated: SCRIPT-BPE eliminates the byte premium at the encoding level and avoids regex-based diacritic splitting, but whether this translates to more equitable downstream model performance across languages — the stated motivation in the Impact Statement — is an empirical question not addressed by compression metrics. A model might learn to compensate for tokenizer-level inequities in ways that reduce downstream disparities, or tokenizer-level inequities might be amplified through training dynamics. Compression ratios cannot answer this.

What evidence exists in the paper. None. This limitation is entirely unmeasured. The paper provides extensive compression evidence (Tables 2, 4, 5) and training-time evidence (Table 3), but zero bytes of downstream evaluation. The gap between what is measured and what matters is the single largest uncertainty in the paper's contribution — acknowledged transparently but not bridged.

Mitigation status. The paper explicitly defers this to future work, framing it as the essential next step ("This is essential for understanding their true impact..."). The authors note that SCRIPT-BPE is computationally efficient enough that "it does not represent a barrier to scaling these methods to training large language models with SCRIPT-BPE" (Section 5), acknowledging that the barrier is not technical feasibility but the substantial compute cost of training models at scale. This is a reasonable scoping choice for a paper introducing a tokenizer design, but it means the paper's claims should be interpreted as necessary-condition evidence (SCRIPT-BPE's compression is good enough to be viable; it eliminates known tokenizer defects) rather than sufficient-condition evidence (models trained with SCRIPT-BPE will perform well). A practitioner deciding whether to adopt SCRIPT-BPE for model training in 2025 would need to either conduct their own downstream evaluation or wait for the authors' follow-up work.

Limitation 2: The SCRIPT Encoding Carries a 2× Initial Token Cost for ASCII-Range Text That BPE Cannot Fully Compensate

The assumption or constraint. The SCRIPT encoding maps every Unicode character to exactly two tokens (block token + index token), regardless of how many UTF-8 bytes the character requires. For Latin-script text in the ASCII range — which constitutes the vast majority of English text and a substantial fraction of web data in many languages — UTF-8 encoding uses exactly 1 byte per character. SCRIPT-BPE therefore starts with a 2× token count disadvantage on ASCII text before any BPE merges occur (2.0 initial tokens/character vs. 1.0 for byte-level encoding, as shown in the "Initial Tokens/Char" column of Table 4). The paper's design philosophy implicitly assumes that BPE merges will substantially close this gap by combining multiple characters into single tokens, but there is a floor: no matter how aggressively BPE merges, a single character can never be represented in fewer than one token, and SCRIPT requires that token to carry both block and index information.

The consequence. The 2× initial cost penalty on ASCII-range text manifests as a persistent compression disadvantage for English and other Latin-script-heavy languages even after 256K BPE merges. In Table 4, o200k + constrained achieves 1.55 tokens/character on English (validation) while SCRIPT rule-based + constrained achieves 1.83 — approximately 18% worse, placing it in the medium-highlighted category (>10% worse than best). This gap is structural, not a training artifact: the byte-level encoding's single-token-per-character representation for ASCII text is essentially optimal, and SCRIPT-BPE's two-token floor means it can at best approach but never match byte-level compression on English under equivalent merge budgets.

The practical consequence is that SCRIPT-BPE would produce longer token sequences for English text than a comparably-sized byte-level tokenizer, consuming proportionally more of a model's fixed context window for the same English input. In multilingual deployment, this means English queries become more expensive (more tokens = more inference compute) under SCRIPT-BPE than under byte-level tokenization, even as non-Latin-script queries become cheaper (due to the elimination of the byte premium for 3-byte scripts). Whether this is a net win or loss depends on the language distribution of the deployment — an English-dominated production workload would see a token cost increase, while a balanced multilingual workload might break even or benefit. The paper does not analyze this tradeoff in terms of real-world deployment costs.

What evidence exists in the paper. Table 4 quantifies this clearly. The initial tokens/character column shows SCRIPT at 2.0 vs. byte-level at 1.0 for English and German, with the gap narrowing but not closing after 256K merges. The monolingual results in Table 5 confirm the same pattern at 64K merges. The paper does not attempt to hide this — it's visible in the compression numbers — but the framing emphasizes the elimination of the byte premium for non-Latin scripts without equal emphasis on the premium SCRIPT introduces for Latin scripts. The abstract's claim of "competitive compression while eliminating encoding-based penalties for non-Latin-script languages" is technically accurate but omits mention of the encoding penalty SCRIPT introduces for Latin-script languages.

Mitigation status. Not addressed. The paper does not propose any mechanism to reduce the ASCII-range token cost — no single-token representation for ASCII characters, no vocabulary sharing across blocks for frequently co-occurring characters, no hybrid encoding that uses 1 token for ASCII and 2 for non-ASCII. The design is principled in its uniformity (every character costs exactly 2 tokens) but does not acknowledge the practical tradeoff this uniformity imposes on the most widely used script in current LLM training data. The suggestion in Section 5 about "combining refining the handling of digits and leading spaces, or allowing certain punctuation to combine with adjacent script characters" hints at possible optimizations but does not address the fundamental 2× vs. 1× ASCII cost difference.

Limitation 3: Failure on Scripts Where Sub-Character Structure Matters — The Korean Hangul Case

The assumption or constraint. SCRIPT-BPE treats each Unicode character as an atomic unit that maps to a (block token, index token) pair. The constrained merging strategy ensures that character boundaries are always respected — a character's two tokens merge with each other first, forming a complete character token, before that character token can merge with adjacent character tokens. This design implicitly assumes that the Unicode character is the appropriate atomic unit for BPE to build upon — that the BPE process should start from complete characters and then combine them into subwords and words.

This assumption breaks down for scripts where the Unicode character is itself composed of meaningful sub-units that recur across many different characters. The most prominent example is Korean Hangul: each Hangul syllable block is a single Unicode character (encoded in 3 UTF-8 bytes), but each syllable is composed of individual jamo — consonant and vowel components (choseong, jungseong, jongseong) — that follow regular combinatorial patterns. A small set of jamo (19 initial consonants, 21 vowels, 27 final consonants, with some constraints) generates the 11,172 possible Hangul syllables. Byte-level BPE can discover these jamo-level regularities because it operates below the character level — frequent byte sequences corresponding to common jamo components can be merged into tokens that capture sub-syllabic structure, even though no single byte corresponds to a complete jamo.

The consequence. SCRIPT-BPE performs poorly on Korean. Table 4 shows that on the multilingual validation set, SCRIPT rule-based + constrained achieves 3.13 tokens/character for Korean, while o200k + constrained achieves 2.28 — SCRIPT is approximately 37% worse, placing it in the dark-highlighted category (>20% worse). In the monolingual comparison (Table 5), the gap is similar: o200k achieves 2.19 vs. SCRIPT rule-based at 3.11. Korean is SCRIPT-BPE's worst-performing language in relative terms across all 12 tested languages.

The root cause is structural: by treating each Hangul syllable as an atomic character, SCRIPT-BPE forces the BPE process to rediscover jamo-level regularities through merges of complete syllable tokens. But a vocabulary of 64K or 256K merges is insufficient to capture all the productive syllable combinations — there are 11,172 possible syllables, far more than can each receive a dedicated token — and the BPE algorithm has no way to decompose a syllable into its jamo components because the constrained merging strategy prohibits splitting a character token once it's formed. Byte-level BPE, by contrast, can learn tokens corresponding to frequent jamo components (e.g., the byte sequence for a common initial consonant followed by a common vowel) that compose across many different syllables, achieving better compression with fewer vocabulary entries.

This limitation is not specific to Korean — any script with productive sub-character combinatorics would face the same issue — but Korean is the most prominent example in the paper's language sample because Hangul's jamo composition is systematic and well-documented. The paper's silence on this result (Korean is not discussed in Section 4.3's language-by-language analysis) is a notable omission: the dark-highlighted 37% gap is the largest relative failure in the table, and it directly illustrates a fundamental tension between SCRIPT-BPE's character-integrity constraint and the compression benefits of sub-character structure.

What evidence exists in the paper. Table 4 and Table 5 clearly show the Korean gap, but the paper does not analyze it. The related work section (Section 2) cites Lee et al. (2025) on jamo-level tokenization for Korean, demonstrating that the authors are aware of the sub-character structure issue, but the Results section does not connect this prior work to SCRIPT-BPE's Korean performance. The discussion of constrained merging (Section 3.2) frames the character-integrity constraint as universally beneficial — which it is for eliminating partial-character tokens — but does not discuss the tradeoff it imposes for scripts that benefit from sub-character tokenization.

Mitigation status. Not addressed in the paper. The SCRIPT-BPE design as presented has no mechanism for handling sub-character structure — the character is the atomic unit, period. A potential extension (not suggested by the authors) would be a hierarchical SCRIPT encoding for scripts like Hangul: represent a syllable as a sequence of jamo tokens, each of which is a (block, index) pair, with the block token indicating "this is a Hangul jamo component" and the index identifying which jamo. This would preserve the script-aware structure while allowing the BPE process to operate at the jamo level. But such an extension would require per-script engineering of the kind the paper argues against, creating a tension between the universal-design philosophy and the practical need to handle scripts with productive sub-character structure.

Limitation 4: The 2× Initial Token Penalty Persists Across All Scripts and Compresses Less Efficiently Than Expected for Several Major Languages

The assumption or constraint. SCRIPT-BPE gives every character an initial cost of exactly 2 tokens, reducing the worst-case initial cost for 3-byte scripts from 3.0 to 2.0 tokens/character (a 33% reduction) but increasing the initial cost for 1-byte ASCII-range scripts from 1.0 to 2.0 (a 100% increase) and for Vietnamese (1.2 initial) to 2.0 (a 67% increase). The paper's implicit assumption is that the BPE merging process will compress these different starting points toward a similar final compression ratio, because SCRIPT-BPE's more structured representation provides better guidance for which merges are linguistically meaningful.

The consequence. The final compression results in Table 4 show that this assumption holds unevenly. For 2-byte scripts (Russian, Arabic, Hebrew) — where the initial cost is already 2.0 under both SCRIPT and byte-level encoding — SCRIPT-BPE achieves final compression within 0–9% of the best byte-level tokenizer, a strong result. For 3-byte scripts, the outcome varies dramatically: Japanese is better under SCRIPT (2.60 vs. 2.99, a 13% improvement), Chinese is moderately worse (1.81 vs. 1.58, a 15% gap), and Korean is substantially worse (3.13 vs. 2.28, a 37% gap). For the 1-byte-dominant scripts (English, German), the gap is 14–18%. For Vietnamese (1.2 initial), the gap is 25%.

This pattern reveals that the relationship between SCRIPT-BPE's initial cost and its final compression is not simply "higher initial cost → harder to compress." Vietnamese, with an initial cost of 2.0 under SCRIPT vs. 1.2 under byte-level, ends up with final compression 25% worse — the BPE process does not close the gap as effectively as it does for the 2-byte scripts where the initial cost is equal. Korean, with initial costs of 2.0 (SCRIPT) vs. 3.0 (byte-level), ends up with SCRIPT substantially worse — the BPE process not only fails to capitalize on the initial advantage, it produces worse final compression. This suggests that the BPE merging process under SCRIPT encoding is less efficient at compressing certain scripts, independent of the initial token count, because the character-level granularity hides productive sub-character patterns (Korean) or because the script-category grouping interacts poorly with mixed-script text (Chinese, as discussed in Section 4.3).

The practical consequence is that SCRIPT-BPE does not deliver on the promise that uniform initial encoding costs translate to uniform competitive compression. Instead, it trades the byte premium problem (non-Latin scripts paying more tokens per character) for a new set of language-specific compression disparities that are harder to characterize in a simple "fairness" framework. A deployment that switched from byte-level BPE to SCRIPT-BPE would see English token counts increase by ~18%, Chinese by ~15%, Korean by ~37%, and Japanese decrease by ~13% — a redistribution of token costs across languages rather than an equalization.

What evidence exists in the paper. Table 4 provides all of these numbers but does not synthesize them into this pattern. The paper's discussion in Section 4.3 mentions specific language-level explanations (Chinese mixed-script phrases, Thai regex optimization) but does not step back to characterize the overall relationship between initial cost, script structure, and final compression. The "All" row showing an aggregate ~19–23% gap between SCRIPT and o200k masks the wide variance across languages.

Mitigation status. Partially acknowledged through the discussion of specific languages in Section 4.3, but not addressed as a systematic limitation. The paper's future work suggestions in Section 5 — "combining refining the handling of digits and leading spaces, or allowing certain punctuation to combine with adjacent script characters" — hint at pretokenization refinements that could improve compression, but these are incremental adjustments to the pretokenizer, not changes to the encoding that could address the sub-character structure issue (Korean) or the ASCII cost floor (English). A more fundamental mitigation would require either a hybrid encoding (1 token for ASCII, 2 for others — sacrificing the uniformity principle) or a hierarchical encoding that exposes sub-character structure for composed scripts (sacrificing the character-as-atomic-unit principle). The paper does not explore these tradeoffs.

Limitation 5: Constrained Merging Validation Relies on Training-Set Measurements for the Compression Improvement Claim

The assumption or constraint. The paper's strongest operational claim — that constrained BPE merging "universally eliminated tokens representing a mix of full and partial characters and generally improved compression across different base encodings" (Section 5) — is supported by Table 2, which reports mean tokens/character on the training corpora for the 12 monolingual tokenizers. The paper states this explicitly: "Tokens/Char shows mean compression ratio on the training corpora for the monolingual tokenizers." In contrast, the main compression results in Table 4 are reported on the 136 GB multilingual validation set (a disjoint held-out split from CulturaX), and the monolingual tokenizer results in Table 5 are reported on the training data.

The consequence. The claim that constrained merging improves compression is validated only on the same data used to train the tokenizers for the monolingual comparison (Table 2). While BPE is a deterministic frequency-based algorithm applied to corpus statistics rather than a learned function prone to overfitting in the traditional sense, there is a subtle circularity concern: constrained merging changes the set of candidate merges by excluding cross-character-boundary pairs, which changes which merges are selected and in what order. A merge that is frequent in the training data but excluded by the constraint might have been useful for compressing held-out text; conversely, a merge that the constraint permits might be less robust to distribution shift than the merge it replaces. The risk is that constrained merging slightly overfits the training data compared to unconstrained merging, producing a small apparent compression improvement that does not generalize to new text.

This risk is likely minimal — the close correspondence between training and validation compression for the multilingual tokenizer in Table 4 (where constrained merging is the default) suggests that BPE-derived vocabularies generalize well regardless of the constraint. But the specific claim of "constrained merging improves compression" rests on a training-set measurement in Table 2, without a corresponding validation-set measurement for the constrained-vs-unconstrained comparison. The magnitude of the claimed improvement is small (0.001–0.033 tokens/character across configurations) and could plausibly be within the range of training-validation noise. For the SCRIPT + o200k configuration, the difference is 0.004 tokens/character — essentially zero — which is consistent with the interpretation that constrained merging is compression-neutral (eliminating defects without affecting compression either way) rather than compression-improving.

The stronger and more robust claim — that constrained merging eliminates partial-character-mixing tokens — is not affected by this limitation, since those tokens are eliminated by construction (the constraint makes them impossible) and this property holds for any text, training or validation. The weaker "improves compression" claim is the one that depends on training-set measurement.

What evidence exists in the paper. Table 2 is the sole source for the compression improvement claim for constrained merging. Table 4 reports only constrained configurations (the paper states in Section 4.1: "As differences in compression are generally small, we present only the constrained versions in all subsequent results"), so no validation-set comparison of constrained vs. unconstrained compression is available. Table 5 similarly reports only constrained configurations for the monolingual tokenizers.

Mitigation status. The paper acknowledges the small effect size ("differences in compression are generally small") and the primary motivation for the constraint is token quality (eliminating partial-character tokens) rather than compression improvement. The recommendation to adopt constrained merging would be equally well-supported if the compression difference were exactly zero, since eliminating a known defect at zero compression cost is already a strong argument. The paper's framing slightly overstates the compression benefit — "generally improved compression" is true for 7 of 8 configurations in Table 2 but the improvements are tiny and measured only on training data — but the core recommendation does not depend on this overstatement. A practitioner adopting constrained merging should expect to eliminate partial-character tokens at negligible or zero compression cost; the small compression improvement seen in Table 2 is a bonus that may or may not materialize on their specific data.

Limitation 6: The Difficulty Estimation Analog — Language-Specific Compression Patterns Are Not Characterized in Terms of Deployment Impact

The assumption or constraint. The paper evaluates compression across 12 languages and aggregate multilingual data, reporting tokens/character as the primary metric. However, compression ratios are reported as averages over entire validation corpora for each language, without any characterization of how compression varies within a language — for example, across different text domains (web text vs. news vs. code), different registers (formal vs. informal), or different subword frequency distributions. The paper also does not translate compression ratios into concrete deployment metrics: what does an 18% compression gap on English mean in terms of context window utilization, inference latency, or cost for a production LLM serving English-dominant traffic?

The consequence. A practitioner evaluating SCRIPT-BPE for deployment needs to answer questions that the paper's evaluation cannot directly address. If a production workload is 80% English, 10% Chinese, 5% Japanese, and 5% other languages, the per-language compression numbers in Table 4 can be combined into a weighted average — but the paper does not provide this weighted-average analysis, and it's unclear whether the aggregate "All" row in Table 4 reflects the language distribution of CulturaX (which may differ substantially from any specific deployment's distribution). More importantly, compression ratios are a linear proxy for a non-linear cost function: the cost of an additional token is not constant — it depends on whether that token pushes the sequence past a context window boundary (causing truncation), whether it increases a batch past a memory limit, and whether the model's performance degrades smoothly or sharply with increased sequence length. Two tokenizers that differ by 18% in tokens/character on English might differ by much more or much less than 18% in practical deployment cost, depending on these non-linearities.

The paper's fairness framing compounds this issue. The Impact Statement says the work "may help create more equitable language models that better serve diverse linguistic communities," but equity cannot be evaluated from compression ratios alone. If SCRIPT-BPE reduces token counts for Thai by 44% (relative to cl100k's catastrophic 5.72 tokens/character in Table 4) while increasing English token counts by 18% (relative to o200k), is this "more equitable"? The answer depends on the baseline definition of fairness: equal tokens per character across languages? Proportional to information content? Proportional to training data representation? The paper does not engage with this normative question, presenting the elimination of the byte premium as self-evidently fairer without acknowledging that SCRIPT-BPE's redistribution of token costs (less for 3-byte scripts, more for 1-byte scripts) is itself a normative choice about what constitutes equitable representation.

What evidence exists in the paper. None. The paper reports per-language averages and an aggregate "All" row, with no domain breakdown, no within-language variance analysis, and no translation of compression ratios to deployment cost estimates. The highlighting scheme in Table 4 (>5%, >10%, >20% worse than best) provides a relative ranking but not an absolute cost impact. The discussion of specific languages in Section 4.3 offers qualitative explanations for patterns (Chinese mixed-script phrases, Thai regex optimization) but does not quantify their deployment significance.

Mitigation status. Not addressed. This is not a flaw in the paper's experimental design — tokenizer papers traditionally report compression ratios and leave deployment cost analysis to practitioners — but it is a limitation on the actionability of the results. A practitioner cannot read this paper and determine whether adopting SCRIPT-BPE would increase or decrease their total inference costs, because that depends on their language distribution and their model's sensitivity to sequence length, neither of which the paper models. The paper's contribution is establishing SCRIPT-BPE as a technically viable alternative with demonstrated compression properties; the economic case for adoption requires additional analysis that the paper does not provide.

Summary Pattern Across Limitations

These six limitations form a coherent picture of what SCRIPT-BPE has demonstrated and what remains uncertain:

The compression-to-performance gap (Limitation 1) means all claims are provisional until downstream models are trained — the paper establishes viability but not superiority.
The ASCII cost floor (Limitation 2) means SCRIPT-BPE structurally disadvantages the highest-volume language in current LLM deployments, creating an adoption barrier that the paper does not address.
The sub-character structure blindness (Limitation 3) means SCRIPT-BPE fails on scripts where the Unicode character is not the optimal atomic unit — Korean being the clearest example, with potential implications for other composed scripts not in the 12-language sample.
The uneven compression compensation (Limitation 4) means the 2× uniform initial cost does not produce uniform competitive final compression — language-specific factors (sub-character structure, mixed-script prevalence, BPE merge efficiency) create new disparities that the paper does not systematically characterize.
The training-set validation of the compression improvement claim (Limitation 5) is a minor methodological concern that slightly weakens the "constrained merging improves compression" framing, though the core "eliminates defects at negligible cost" argument is unaffected.
The missing deployment cost analysis (Limitation 6) means practitioners cannot determine from this paper alone whether SCRIPT-BPE would reduce or increase their total costs, because compression ratios don't directly translate to economic impact without modeling language distributions and context-window constraints.

The paper is transparent about Limitation 1 (explicitly deferred to future work), partially acknowledges Limitation 4 (through language-specific discussion in Section 4.3), and is largely silent on Limitations 2, 3, 5, and 6. A practitioner considering SCRIPT-BPE for a specific deployment would need to conduct their own downstream evaluation (addressing Limitation 1), analyze their language distribution's interaction with the ASCII cost tradeoff (Limitation 2) and Korean-level sub-character structure issues (Limitation 3), and model the deployment cost implications (Limitation 6) — none of which the current paper provides guidance for.

7. Implications and Future Directions

How This Work Changes the Landscape

SCRIPT-BPE does not introduce a new model architecture or a more efficient training algorithm—it intervenes at the earliest possible stage of the NLP pipeline: the conversion of raw text into token sequences. The paper's contribution is best understood as a reframing of the tokenization problem from "how do we optimize compression for the dominant script?" to "how do we design a representation that is structurally fair across all scripts before any optimization occurs?" This is not a paradigm shift on the scale of the transformer architecture, but within the subfield of tokenizer design, it challenges a foundational assumption: that UTF-8 byte encoding is the appropriate default for representing multilingual text.

The reframing operates on three levels:

First, it demonstrates that Unicode metadata—specifically script and general category properties—can serve as the primary representational layer rather than as auxiliary information consulted when debugging tokenizer failures. Prior work treated script identity as something the pretokenizer had to reconstruct from byte patterns (e.g., the o200k regex's elaborate non-ASCII handling) or as domain-specific knowledge injected per language (jamo-level Korean, grapheme-level Tamil). SCRIPT-BPE inverts this: the block token is the script identity, and the pretokenizer simply compares adjacent block tokens to determine boundaries. This architectural inversion—making structure explicit in the representation rather than recovering it through heuristics—is the paper's central intellectual move and one that generalizes beyond tokenization to any pipeline where structured metadata is available but goes unused.

Second, it shifts the evaluation discourse from "which tokenizer compresses best?" to "which tokenizer is most equitable across languages?" The paper's compression results (Table 4) show that SCRIPT-BPE is generally not the best-compressing tokenizer for any individual language—it is outperformed by o200k on English, Chinese, Thai, Hindi, Korean, and Vietnamese. But this framing misses the point: the o200k regex was tuned over multiple iterations specifically to optimize compression on high-resource languages, with English as the implicit primary target. SCRIPT-BPE achieves competitive compression across all languages simultaneously without per-language tuning, using a pretokenization algorithm whose logic fits in a paragraph versus the hundreds of clauses in the o200k regex. This reframes "compression" from a single-objective optimization (minimize tokens per character on a weighted mix) to a multi-objective constraint satisfaction problem (achieve adequate compression for every script while eliminating structural penalties). The paper does not prove that this reframing produces better downstream models, but it makes the normative case that script-equitable representation is a design goal worth optimizing for, separate from raw compression.

Third, it establishes the character-integrity constraint as a "free lunch" improvement that should be adopted immediately by all BPE tokenizer implementations. The universality of the finding—constrained merging eliminates partial-UTF-8-sequence tokens and generally improves compression across all encoding and pretokenizer combinations tested (Table 2)—is rare in systems research. This is not a tradeoff; it is a constraint that fixes a known defect (Land and Bartolo, 2024) while slightly improving the primary metric. The paper provides sufficient evidence that any tokenizer training pipeline—whether byte-level or SCRIPT-based, regex-pretokenized or rule-based—should adopt this constraint as a default. If widely adopted, this would eliminate a class of tokens that currently number in the thousands for large multilingual tokenizers (874 in GPT-4o, 42,831 in the Thai failure case), removing a known source of under-trained embeddings and potential tokenizer-level vulnerabilities.

Which research directions become more attractive: The paper makes structured-metadata-aware tokenization a credible research direction rather than a niche interest. Prior to this work, a researcher proposing to replace UTF-8 bytes with Unicode-property-based encodings would face the reasonable objection: "why abandon the encoding that works perfectly well for English?" SCRIPT-BPE provides a counterargument: because that encoding systematically penalizes the majority of the world's writing systems, and a structured alternative achieves comparable compression without those penalties. The burden of proof has shifted—it is now incumbent on defenders of byte-level encoding to demonstrate that its compression advantages on English outweigh its structural inequities for other scripts, rather than on proponents of script-aware encoding to justify the cost of change.

Which research directions become less attractive: The paper indirectly argues against the approach of incrementally patching regex pretokenizers to handle newly discovered edge cases. Schmidt et al. (2025) proposed a more robust regex to handle curly vs. straight apostrophes; future proposals might handle additional contraction types or script-specific diacritic patterns. SCRIPT-BPE demonstrates that a fundamentally simpler alternative exists that avoids the entire class of regex-induced failures by using a different representation. Continued investment in regex refinement starts to look like optimizing a local maximum—each fix addresses one language's edge case while leaving the underlying structural bias (byte premiums, script-oblivious representation) untouched. This does not mean regex research is obsolete—the o200k regex still outperforms SCRIPT-BPE on several languages in Table 4—but it suggests that the long-term direction should be toward representation-level solutions rather than pretokenization-level patches.

A reconciliation of prior contradictions: The paper provides a framework for understanding why prior work on tokenization equity produced inconsistent results. MYTE (Limisiewicz et al., 2024) showed that morphology-driven byte encodings improve fairness, but required per-language morphological resources that don't scale. Jamo-level Korean tokenization (Lee et al., 2025) works well for Korean but doesn't generalize. Grapheme-level tokenization (Velayuthan and Sarveswaran, 2025) fixes diacritic splitting for Tamil, Sinhala, and Hindi but requires grapheme inventories for each new script. These are not contradictory findings—they are all successful applications of the same principle ("tokenize at linguistically meaningful boundaries") applied to different languages with different engineering requirements. SCRIPT-BPE unifies these approaches by identifying a source of linguistically meaningful boundaries (Unicode script and category properties) that exists for all scripts without per-language engineering. The prior works are not wrong; they are special cases of a general principle that SCRIPT-BPE operationalizes universally.

The paper also reconciles the tension between compression and simplicity. The o200k regex achieves excellent compression but at the cost of extreme complexity (hundreds of regex clauses, opaque edge cases, undocumented design decisions). Removing all pretokenization achieves high raw compression (the paper notes ~12% higher on average) but degrades downstream performance (Schmidt et al., 2024). SCRIPT-BPE with rule-based pretokenization occupies a middle ground: simpler than regex, more structured than no pretokenization, with compression that is generally competitive (within 10–40% of o200k) rather than dominant. This suggests a simplicity-efficiency Pareto frontier where SCRIPT-BPE trades some compression for substantially reduced complexity and improved cross-lingual equity—a tradeoff that prior work did not explicitly characterize.

Follow-Up Research This Work Enables

Downstream model training with SCRIPT-BPE vs. byte-level BPE on multilingual benchmarks. The paper's most critical open question is whether SCRIPT-BPE's compression properties translate to improved or equivalent downstream model performance. A strong follow-up would train two identically-sized language models (e.g., 1B parameters, same architecture, same training data) with SCRIPT-BPE vs. o200k tokenization, then evaluate on a suite of multilingual tasks: XNLI for natural language inference, Flores for machine translation, TyDi QA for question answering, and MMLU in multiple languages. The hypothesis to test is that SCRIPT-BPE's elimination of byte premiums and diacritic splitting leads to better performance on non-Latin-script languages, potentially at a small cost to English performance. This would also stress-test Limitation 2 (the ASCII cost floor): does the 18% compression gap on English translate to a measurable performance gap, or can the model compensate? The experiment would require approximately 100B training tokens at 1B scale—a non-trivial but feasible compute budget for an academic lab—and would transform SCRIPT-BPE from a tokenizer-level proposal to a validated modeling choice.

Hybrid encoding that reduces the ASCII token cost without sacrificing script-equity. The 2× initial token cost for ASCII-range characters is SCRIPT-BPE's most significant structural disadvantage (Limitation 2). A natural extension would test whether selective single-token encoding for ASCII-range characters—while maintaining the two-token (block, index) encoding for all non-ASCII characters—improves compression for English and other Latin-script languages without reintroducing byte-premium-like penalties. The experiment would define a threshold: characters below U+0080 (the ASCII range) are encoded as a single token whose value directly encodes the character (similar to byte-level encoding), while all characters at U+0080 and above use the SCRIPT two-token encoding. The constrained merging rule would need to handle the heterogeneous representation: single-token ASCII characters can merge with each other and with two-token SCRIPT characters (since both represent complete characters). This hybrid approach sacrifices the conceptual purity of "every character costs exactly 2 tokens" for practical compression gains on English-dominant workloads. The experiment should measure compression across all 12 languages in the paper's sample plus English-heavy code datasets (Python, JavaScript) where ASCII predominates, testing whether the hybrid approach narrows or closes the gap with byte-level BPE on English while preserving SCRIPT-BPE's advantages on non-Latin scripts.

Sub-character decomposition for composed scripts like Korean Hangul. Limitation 3 identified SCRIPT-BPE's poor Korean performance (37% worse compression than o200k) as a failure mode driven by the character-as-atomic-unit assumption. A follow-up could design a hierarchical SCRIPT encoding for scripts with productive sub-character structure: for Hangul, decompose each syllable into its constituent jamo (initial consonant, vowel, optional final consonant), encode each jamo as a (block, index) pair, and allow BPE merges to operate at the jamo level. The block token would indicate "Hangul jamo" rather than "Hangul syllable," and the index would identify the specific jamo. This would reduce the initial token cost for Korean (from 2 tokens per syllable to 2 tokens per jamo, but a syllable of 3 jamo would cost 6 tokens vs. 3 bytes in UTF-8), so the key metric is whether jamo-level BPE merges can efficiently recombine jamo into syllable-level tokens when those syllables are frequent enough to justify dedicated vocabulary entries, while using jamo-level tokens for rare syllables. The experiment would compare three conditions on Korean text: byte-level BPE (current best), SCRIPT-BPE with syllable-level encoding (current worst), and SCRIPT-BPE with jamo-level encoding (proposed), measuring both compression and (ideally) downstream perplexity on a Korean language model. The hypothesis is that jamo-level SCRIPT-BPE matches or exceeds byte-level BPE by combining the script-aware structure of SCRIPT with the sub-character granularity that byte-level BPE exploits. A negative result—jamo-level SCRIPT-BPE still underperforming byte-level—would suggest that the benefit of byte-level encoding for Korean is not solely about sub-character access but about something more fundamental to the byte representation.

Adaptive pretokenization that applies the Hiragana-Han merging rule and similar script-specific rules only when relevant, learning others from data. The paper acknowledges that the Hiragana-Han merging rule (Rule 4) is a script-specific intervention—precisely the kind of per-language engineering that SCRIPT-BPE's philosophy argues against, albeit as a single explicit rule rather than hundreds of opaque regex clauses. A follow-up could investigate whether this rule can be learned from data rather than hard-coded. The approach: during BPE training on a multilingual corpus, track the frequency with which adjacent characters from different scripts co-occur within the same pretoken vs. across pretoken boundaries. If Han and Hiragana characters appear adjacent with high frequency in the raw text (before pretokenization), and a pretokenization that splits them produces worse compression than one that merges them, a data-driven rule could automatically merge Han-Hiragana sequences. More ambitiously, the system could learn a script-adjacency merging matrix: for each ordered pair of scripts, determine whether merging adjacent groups of those scripts improves or degrades validation compression, and apply the merge automatically. This would generalize the Hiragana-Han rule—which is essentially one cell in an adjacency matrix where the value is "merge"—to all script pairs, potentially discovering other beneficial merges (e.g., Latin-Han merging for Chinese web text with mixed-script phrases) without manual specification. The experiment would compare the manually-specified pretokenization rules against a data-driven version on the 12-language sample, measuring both compression and the interpretability of the learned merging decisions. This would address the tension between SCRIPT-BPE's "no per-language engineering" claim and the existence of the Hiragana-Han rule.

Stress-testing constrained merging on out-of-distribution text to verify the training-set compression improvement. Limitation 5 noted that the compression improvement from constrained merging is measured only on training corpora (Table 2). A focused follow-up would evaluate constrained vs. unconstrained merging on deliberately out-of-distribution text to test whether the small compression gains are robust to domain shift. The experiment would take the 12 monolingual tokenizers (trained on Goldfish data) and measure compression on text from different domains: Wikipedia articles in each language, news articles, social media posts, and code (Python, for English). If constrained merging's compression advantage persists across all domains, the "constrained merging improves compression" claim is strengthened. If the advantage disappears or reverses on specific domains (e.g., social media with non-standard orthography where cross-character-boundary merges might be more frequent), the claim should be qualified to "constrained merging eliminates defective tokens with no systematic compression penalty; small improvements observed on in-domain training data may not generalize." This experiment is low-cost (requiring only tokenizer evaluation, not model training) and directly addresses the most significant methodological gap in the paper's empirical claims.

Ablation of the individual pretokenization rules to measure their contribution to compression and token quality. The rule-based pretokenizer has four components (initial script-category grouping, space merging, Inherited script merging, Hiragana-Han merging). A systematic ablation removing each rule one at a time would quantify how much each contributes to the final compression and to the prevention of specific failure modes. Removing the Inherited script merging rule (Rule 3) should cause diacritic splitting on Arabic, Hindi, and other scripts with combining marks—analogous to the cl100k regex's failure mode—and the experiment would measure the resulting compression degradation and count of split-character tokens. Removing the space merging rule (Rule 2) should affect scripts that use whitespace for word separation, potentially fragmenting words on Latin and Cyrillic text. Removing the Hiragana-Han merging rule (Rule 4) should degrade Japanese compression specifically, quantifying how much of SCRIPT-BPE's Japanese advantage depends on this rule versus the general script-category grouping. This ablation would clarify which rules are essential for SCRIPT-BPE's performance and which could be removed or replaced without significant impact, guiding future simplification or generalization of the pretokenization algorithm.

Practical Applications and Downstream Use Cases

Multilingual API tokenizers for production LLM services. A deployment scenario where a single tokenizer serves queries in dozens of languages—such as an LLM API endpoint used by customers worldwide—would benefit directly from SCRIPT-BPE's elimination of regex-induced catastrophic failures. The paper shows that the cl100k regex (GPT-4) produces compression ratios of 5.72 for Thai and 4.29 for Hindi (Table 4), meaning Thai and Hindi queries consume 3–4× more tokens than equivalent English queries under the same pretokenizer. This translates to 3–4× higher inference costs for users writing in those languages, creating an invisible "encoding tax" that most users are unaware of. SCRIPT-BPE reduces Thai from 5.72 (cl100k) to 1.96 tokens/character and Hindi from 4.29 to 2.13—a 66% and 50% reduction respectively. For a service with non-trivial Thai or Hindi traffic, switching from cl100k to SCRIPT-BPE would dramatically reduce the token count (and thus cost) for those users while keeping English token counts within ~18% of the o200k baseline. The practical benefit is not theoretical: it is a measurable reduction in per-query cost for historically disadvantaged language communities, achieved by changing the tokenizer rather than the model. The constraint is that SCRIPT-BPE increases English token counts by ~18% relative to o200k, so the net cost impact depends on the language distribution. For services with >20% non-Latin-script traffic, the savings on those languages would likely outweigh the English cost increase; for English-dominant services, o200k remains more cost-efficient on compression grounds alone.

Pretraining data preprocessing for massively multilingual language models. When assembling training corpora covering 100+ languages, the tokenizer determines how the model's fixed context budget is allocated across languages. A byte-level tokenizer with the cl100k regex effectively allocates 3× more context space per character to English than to Chinese, simply because of UTF-8 byte counts. This means that during pretraining, the model sees proportionally less text from high-byte-count languages for the same number of training tokens, compounding any existing imbalances in the raw data distribution. SCRIPT-BPE eliminates this structural bias: every character costs exactly 2 tokens, so the tokenizer does not artificially amplify or suppress any language's representation in the training data. This is particularly relevant for training data mixtures that are carefully balanced across languages (e.g., 10% English, 5% Chinese, 5% Arabic, etc.)—byte-level tokenization undermines that balance by giving Chinese and Arabic less effective representation per token than English, while SCRIPT-BPE preserves it. A practitioner assembling a multilingual pretraining corpus with explicit per-language sampling weights would get closer to their intended distribution using SCRIPT-BPE than using byte-level BPE, because the tokenizer is not introducing a second, unaccounted-for weighting factor on top of the sampling weights. The tradeoff is the ~18% English compression penalty, which means the overall training corpus will be ~18% larger in token count for the same amount of English text—but this is a known, uniform scaling factor rather than a language-dependent distortion, making it easier to budget for and account for in training compute estimates.

Tokenizer-level fairness auditing for regulatory compliance and transparency reporting. As AI regulations increasingly require documentation of model biases and mitigations (e.g., the EU AI Act's transparency requirements, various proposed algorithmic fairness reporting standards), tokenizer-level inequities represent a pre-model bias that is currently invisible in most fairness audits. A model developer could claim their training data is balanced across languages while using a tokenizer that silently allocates 3× more tokens to English text than to Hindi text for equivalent content—technically true at the character level but misleading at the representation level the model actually sees. SCRIPT-BPE provides a concrete, auditable alternative: the fixed 2-token-per-character encoding means that "tokens per unit of text" is uniform across languages, making it straightforward to verify that the tokenizer is not introducing script-dependent disparities. Even if an organization does not adopt SCRIPT-BPE for training (perhaps because the downstream performance is not yet validated), the framework of measuring and reporting initial tokens per character by language—the "Initial Tokens/Char" column in Table 4—could become a standard component of tokenizer documentation. A fairness report could state: "Our tokenizer uses UTF-8 byte encoding, which produces initial token costs of 1.0 for English, 2.0 for Russian, and 3.0 for Chinese. After BPE training, these compress to 1.55, 1.76, and 1.58 tokens/character respectively." This transparency would allow downstream users to understand and account for tokenizer-level inequities rather than treating the tokenizer as an opaque black box whose biases are invisible. SCRIPT-BPE, by making the initial cost uniform and auditable, sets a standard for what tokenizer equity reporting could look like—even for tokenizers that do not adopt its encoding.

Tokenization for on-device and edge-deployed multilingual models. On-device language models (smartphone keyboards, real-time translation, accessibility tools) face tight constraints on both model size and context window length. The byte premium effect is particularly damaging in this setting: a 2,048-token context window can hold ~2,048 characters of English text (at near-1.0 tokens/character after BPE compression) but only ~682 characters of Chinese text (at 3.0 initial tokens/character, partially compressed by BPE). This means an on-device translation model with a fixed context window simply cannot process long Chinese or Thai sentences that would fit comfortably in the window if they were in English—not because the model lacks capability, but because the tokenizer consumes the window budget with encoding overhead. SCRIPT-BPE's uniform 2.0 initial tokens/character provides a fixed, predictable relationship between character count and token count regardless of language: a developer knows that a 1,024-token context window can hold approximately 512 characters of any language (before BPE compression), rather than 1,024 characters of English but only 341 of Chinese. This predictability is valuable for engineering context-window-sensitive applications where worst-case behavior matters. The compression penalty on English (~18% relative to o200k) may be an acceptable tradeoff in this setting because it buys predictability and eliminates catastrophic overflows for high-byte-count scripts. The developer can budget for a slightly larger context window or a slightly smaller model to compensate for the English cost increase, knowing that this compensation applies uniformly rather than trying to handle script-dependent token budgets.