EuroLLM: Multilingual Language Models for Europe

ArXiv: 2409.16235

🎯 Pitch

A 1.7B-parameter model, EuroLLM, trained on balanced multilingual data, outperforms 7B-parameter models on machine translation and matches 2B models on non-English reasoning benchmarks. A novel joint scaling law reveals that dedicating around 20% of training data to parallel corpora optimally boosts translation without hurting general performance.

---

1. Executive Summary

This paper introduces the EuroLLM project, a multilingual model development effort targeting all official European Union languages plus additional relevant languages, and releases initial models EuroLLM-1.7B and EuroLLM-1.7B-Instruct. The authors develop a training pipeline that balances data mixture decisions through joint scaling laws (predicting language-specific performance as a function of parameter count and data weighting, revealing that 20% parallel data per language improves translation without degrading other domains while exceeding 25% yields diminishing returns) and a trapezoid learning rate scheduler (warmup → constant → linear decay, outperforming cosine scheduling on both general benchmarks and machine translation). The 1.7B-parameter model, trained on 4 trillion tokens with a 128,000-piece multilingual tokenizer, matches or exceeds Gemma-2B and TinyLlama on multilingual Hellaswag across all non-English languages and proves competitive with Gemma-7B on machine translation despite having substantially fewer parameters — establishing that a purpose-built multilingual tokenizer and data mixture enable smaller models to rival much larger general-purpose models on non-English tasks, though the English-only Arc Challenge benchmark reveals a remaining gap attributable to the model's lower non-embedding parameter count (1.133B vs. Gemma-2B's 1.981B).

2. Context and Motivation

The Core Problem: Open-Weight LLMs Are Predominantly English-Centric

The fundamental gap this paper addresses is straightforward but consequential: the vast majority of open-weight large language models are built for and optimized around English, systematically neglecting most of the world's languages — including many official languages of the European Union. While models like LLaMA (Touvron et al., 2023), Mistral (Jiang et al., 2023), and Gemma (Team et al., 2024a) have driven a surge in accessible, reproducible LLM research, their training data, tokenizer design, and evaluation protocols are overwhelmingly English-centric. The paper opens by explicitly identifying this asymmetry:

"despite the growing availability of open-weight LLMs (e.g., LLaMA, Mistral, or Gemma...), these are predominantly limited to English and a few high-resource languages, leaving out many European languages."

This is not merely a coverage issue — it's a structural design problem. When an LLM's tokenizer is optimized for English (e.g., Mistral's 32,000-piece vocabulary heavily skewed toward English morphemes), non-English text becomes tokenization-inefficient: the model must process more tokens to represent the same semantic content, wasting both inference compute and the model's limited context window. When training data is overwhelmingly English, the model never learns the morphological patterns, idiomatic structures, or cultural references of other languages with sufficient depth. The result is a class of models that are nominally "multilingual" — they can produce text in other languages — but perform substantially worse on non-English tasks, with the gap widening for lower-resource languages.

The paper targets a specific set: all 24 official EU languages (Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Irish, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Slovak, Slovenian, Spanish, Swedish) plus 11 additional relevant languages (Arabic, Catalan, Chinese, Galician, Hindi, Japanese, Korean, Norwegian, Russian, Turkish, Ukrainian). This is a deliberately broad scope — 35 languages spanning multiple language families (Romance, Germanic, Slavic, Uralic, Semitic, Sino-Tibetan, etc.) with vastly different levels of digital resource availability. Finnish and Hungarian, for instance, are Uralic languages with complex agglutinative morphology that English-centric tokenizers handle particularly poorly; Maltese is a Semitic language written in Latin script with fewer than 500,000 native speakers, making it a genuine low-resource target even within the EU.

Why This Matters: Real-World Impact and Strategic Significance

The paper's motivation has several layers of importance, ranging from immediate practical concerns to broader strategic considerations:

Immediate practical impact: European institutions and businesses. The European Union operates in 24 official languages with a legal requirement for multilingual communication. EU institutions (the European Commission, Parliament, Court of Justice) produce and translate vast volumes of text daily. A purpose-built multilingual LLM that genuinely works across all EU languages could dramatically reduce translation costs, improve cross-border information access, and enable services (legal document summarization, parliamentary record analysis, citizen inquiry processing) that current English-centric models struggle to deliver reliably. Similarly, European businesses operating across multiple language markets — e-commerce platforms, customer support systems, content moderation tools — need models that perform consistently across languages, not just acceptably in English and poorly elsewhere.

Theoretical significance: understanding how multilingualism interacts with model capacity. At 1.7B parameters, EuroLLM is deliberately small by contemporary standards. Training a model of this size to be competent across 35 languages forces hard tradeoffs: how do you allocate a fixed parameter budget and training token budget across languages to maximize overall utility? The paper's systematic approach — using scaling laws to predict language interference, measuring the impact of parallel data on cross-lingual alignment, and designing a tokenizer that evenly distributes capacity across languages — provides empirical evidence about how multilinguality scales with model size. This is valuable for the broader research community because it helps answer questions like: at what parameter count does adding more languages stop hurting per-language performance? How much parallel data is enough to align representations across languages without wasting capacity? The paper's findings (e.g., that 20% parallel data per language improves translation without degrading monolingual performance, but 25-37.5% yields diminishing returns) provide concrete data points for these questions.

Strategic significance: digital sovereignty and the concentration of AI capability. The paper alludes to this when noting that "the most advanced models are owned by large corporations with a piecemeal commitment to open science." The concentration of frontier LLM development in a handful of US-based corporations (OpenAI, Anthropic, Google, Meta) creates a dependency that extends beyond research access to include language coverage decisions. When the organizations building the most capable models are primarily serving English-speaking markets, languages spoken by smaller populations — even official EU languages like Maltese, Irish, or Estonian — risk being perpetually underserved. The EuroLLM project, by releasing open-weight models specifically designed for European linguistic diversity, represents an investment in digital sovereignty: the capacity for European institutions, researchers, and companies to deploy capable language technology without depending on the language-coverage priorities of foreign corporations.

Where Prior Approaches Fall Short

The paper identifies several categories of prior work and their limitations:

General-purpose open-weight LLMs (LLaMA, Mistral, Gemma). These models represent the state of the art in open-weight LLMs, and the paper uses them as baselines (comparing against Gemma-2B, Gemma-7B, and TinyLlama). Their primary limitation for the paper's goals is language coverage as an afterthought rather than a design constraint. The tokenizers for these models are developed primarily for English efficiency. Mistral's 32,000-piece vocabulary, for instance, achieves low fertility (fewer tokens per word) for English and linguistically similar languages but forces languages with different morphological structures to be represented as sequences of suboptimal subword units. The training data mixtures for these models, while they may include some non-English text, are not systematically balanced to ensure adequate representation across languages. The paper's tokenizer comparison (Figure 5) makes this concrete: Mistral's tokenizer shows significantly higher fertility (more pieces per word) for most European languages compared to the EuroLLM tokenizer, directly translating to wasted compute and worse modeling of non-English text.

Typical general-purpose training strategies for multilingual LLMs fall into several categories:

• Equal sampling or temperature-based sampling: sample uniformly across languages or up-weight lower-resource languages. These are heuristic and don't account for how languages interfere with each other in a shared parameter budget or how the optimal mix changes with model scale.
• Data-volume-proportional sampling: allocate training tokens proportional to how much data is available for each language. This systematically starves lower-resource languages (e.g., Irish, Maltese) in favor of English and other high-resource languages.
• Manual trial-and-error: iterate on data mixtures by training multiple models, which is expensive and doesn't generalize to new model sizes or language sets.

The paper's use of joint scaling laws (Fernandes et al., 2023) — fitting power-law performance predictions as a function of model size and language-specific data weighting by training small proxy models — directly addresses this limitation. It provides a principled, predictive framework for making data mixture decisions without having to train the full target model repeatedly. This is described in more detail in Section 2.2.1 and will be expanded in the Technical Approach section.

Multilingual models from large organizations (NLLB, mT5, MADLAD-400). The machine translation community has produced purpose-built multilingual models, notably Meta's No Language Left Behind (NLLB; Team et al., 2022) which covers 200 languages. However, these are typically encoder-decoder translation models, not general-purpose decoder-only LLMs. They are trained specifically for translation and lack the broad world knowledge, reasoning capability, and instruction-following behavior that characterize LLMs. The paper uses data sources developed in these projects (MADLAD-400, mC4, HPLT) but targets a fundamentally different model class.

Multilingual instruction-tuning datasets (Aya, OpenHermes). The Aya dataset (Singh et al., 2024) and OpenHermes-2.5 (Teknium, 2023) provide multilingual instruction-following examples, which the paper incorporates into its EuroBlocks fine-tuning dataset. However, these datasets are supplements to model training — they don't address the fundamental pretraining challenge of building a base model with genuinely multilingual representations. Fine-tuning an English-centric base model on multilingual instructions can teach it to produce output in other languages, but the underlying representations remain English-optimized, limiting the depth of understanding the model can achieve in non-English languages.

How This Paper Positions Itself

The EuroLLM project is positioned not as a novel architectural contribution or a new training algorithm, but as an engineering and design effort to systematically build multilingual LLMs from the ground up, with multilinguality as a first-class constraint at every stage of the pipeline. The paper's contributions span the entire model development lifecycle:

• Data collection and filtering (§2.1): Assembling a multilingual corpus covering all 35 target languages from diverse sources (web data, parallel corpora, code/math data, high-quality curated data) with language-specific filtering pipelines, rather than applying a one-size-fits-all preprocessing recipe.
• Data mixture optimization (§2.2): Using joint scaling laws to make principled decisions about parallel data proportion, high-quality data repetition, and inter-language token allocation — decisions that would otherwise require expensive trial-and-error or heuristic guessing.
• Tokenizer design (§3): Building a 128,000-piece BPE tokenizer with byte-fallback, deliberately sized to balance vocabulary coverage against embedding parameter cost, and evaluated via fertility comparisons against existing tokenizers.
• Training recipe decisions (§4): Selecting architecture components (GQA, RMSNorm, SwiGLU, RoPE) from the modern LLM design space and empirically comparing learning rate schedulers to find that a trapezoid scheduler outperforms cosine scheduling for multilingual training — a finding that may generalize beyond this specific project.
• Instruction tuning (§5): Creating EuroBlocks, a multilingual instruction dataset that combines existing resources with translation data, and fine-tuning the base model into a conversational assistant.

The paper explicitly frames this as a project report rather than a methods paper — it describes "the work done so far" and releases initial models as a checkpoint toward the larger goal of scaling to larger parameter counts. This honesty about the current state (1.7B parameters is small by 2024 standards, and the English-only Arc Challenge results lag behind Gemma-2B) is appropriate: the paper is establishing the methodology and infrastructure that will enable larger-scale training, not claiming to have solved multilingual LLM development at this parameter count.

The positioning relative to existing models is also specific. The baseline comparisons in Section 6 use Gemma-2B and Gemma-7B (Google's open models), TinyLlama (a 1.1B-parameter open model), and to ensure fair comparison, the authors fine-tune Gemma models on the same EuroBlocks dataset used for EuroLLM-Instruct. This controls for instruction-tuning quality and isolates the effect of the pretraining design choices (tokenizer, data mixture, training recipe). The result — EuroLLM-1.7B outperforming Gemma-2B on non-English benchmarks and competing with Gemma-7B on translation despite having 1.133B vs. 1.981B (Gemma-2B) and ~7B (Gemma-7B) non-embedding parameters — provides evidence that the multilingual-first design pays off in non-English capabilities, even if it comes at some cost to English performance.

The paper also implicitly positions itself within the broader conversation about efficient multilingual modeling. Rather than arguing that more parameters are needed to cover more languages (the "just scale up" approach), it demonstrates that careful data mixture engineering, tokenizer design, and training recipe choices can extract significantly more multilingual capability from a fixed parameter budget. This is practically important because it means smaller, more deployable models can serve linguistically diverse user bases if designed appropriately from the start.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

What is being built: EuroLLM is a suite of open-weight multilingual large language models — specifically, a 1.7B-parameter Transformer trained from scratch — designed to understand and generate text across all 24 official European Union languages plus 11 additional relevant languages.

What problem it solves and the shape of the solution: The core problem is that open-weight LLMs are overwhelmingly English-optimized at every level of their design (tokenizer, training data, evaluation), and simply scaling up model size does not automatically fix the systematic neglect of non-English languages. The solution is a multilingual-first engineering pipeline where every design decision — from vocabulary construction through data mixture optimization to learning rate scheduling — is made with explicit, empirically-grounded consideration of how it affects performance across 35 languages of varying typological diversity and resource availability. The shape is not a single algorithmic innovation but rather an integrated set of principled design choices validated through scaling-law experiments and comparative ablations.

3.2 Big-Picture Architecture (Diagram in Words)

The EuroLLM pipeline has five major stages, each with a distinct responsibility:

1. Data Collection and Filtering Pipeline — ingests raw text from web crawls, parallel corpora, code repositories, mathematics datasets, and curated high-quality sources; applies language-specific deduplication, language identification, perplexity filtering, heuristic filtering, and translation quality filtering; outputs cleaned monolingual text, parallel sentence pairs, and code/math documents organized by language.

2. Data Mixture Optimizer — takes the cleaned dataset sizes and target languages as input; uses joint scaling laws fitted from small proxy models (100M–341M non-embedding parameters) trained on 100B-token corpora with varying parallel-data percentages and data repetition strategies; outputs the optimal allocation of training tokens across languages, data categories, and training phases (main phase vs. annealing phase).

3. Multilingual Tokenizer — takes representative text samples from all target languages; trains a 128,000-piece byte-level BPE tokenizer with byte-fallback using SentencePiece; outputs a vocabulary and merging rules that minimize fertility (tokens per word) across all target languages while keeping embedding parameter count manageable.

4. Pre-training Engine — takes the tokenized training corpus, model architecture hyperparameters, and training recipe; trains a 1.7B-parameter dense Transformer with GQA, SwiGLU, RoPE, and RMSNorm for 4 trillion tokens on 256 H100 GPUs using the Adam optimizer with a trapezoid learning rate scheduler; outputs base model weights (EuroLLM-1.7B).

5. Instruction Fine-Tuning Pipeline — takes the base model and the EuroBlocks dataset (1M multilingual instruction-following examples with chat formatting control tokens); fine-tunes with standard next-token prediction loss for 4 epochs; outputs instruction-following model weights (EuroLLM-1.7B-Instruct).

Information flows sequentially through stages 1→5, but with feedback loops: the data mixture optimizer (stage 2) informs data collection priorities; the tokenizer (stage 3) is designed concurrently with data mixture decisions to ensure vocabulary coverage matches the language distribution; and the pre-training engine (stage 4) evaluates design choices from stages 2–3 through scaling-law experiments before committing to the full training run.

3.3 Roadmap for the Deep Dive

• First, the data collection and filtering pipeline (§3.4.1), because the training data's composition and quality fundamentally constrain everything downstream. I will cover the four data categories (web, parallel, code/math, high-quality), the language-specific filtering decisions, and the annealing-data preparation.

• Second, the data mixture optimization framework (§3.4.2), because deciding how much of each data type to include and how to allocate tokens across languages is the central intellectual contribution of the pretraining design. I will cover the joint scaling law methodology, the parallel-data proportion experiments, the data-repetition analysis, and the inter-language token allocation.

• Third, the multilingual tokenizer (§3.4.3), because the tokenizer determines how efficiently the model can process text in each language. I will cover the BPE+byte-fallback training procedure, the vocabulary size tradeoff, and the fertility comparison against existing tokenizers.

• Fourth, the model architecture and pre-training (§3.4.4), covering the Transformer configuration choices, the rationale behind each architectural component, the trapezoid vs. cosine learning rate scheduler comparison, and the full training hyperparameters.

• Fifth, the instruction fine-tuning (§3.4.5), covering the EuroBlocks dataset construction, the SFT procedure, and the chat formatting system with control tokens.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an engineering and empirical design paper whose core idea is that building a genuinely multilingual LLM requires multilinguality to be a first-class constraint at every design stage — not an afterthought applied to an English-first model — and that principled scaling-law experiments can guide these design decisions for a fraction of the cost of full-scale training runs.

---

3.4.1 Data Collection and Filtering Pipeline

The training corpus is assembled from diverse sources and organized into four categories: web data, parallel data, code/math data, and high-quality data. Each category has distinct collection and filtering procedures, with language-specific decisions made based on data availability and quality requirements.

Web Data Collection and Filtering

Web data constitutes the bulk of the training corpus and is collected through a tiered strategy that applies different sourcing and filtering rigor depending on language resource availability:

For English, the paper uses the FineWeb-edu dataset (Lozhkov et al., 2024), which has already undergone individual dump deduplication (removing duplicate documents within each Common Crawl dump), heuristic filtering (rule-based removal of low-quality text such as documents with excessive punctuation, short length, or high repetition), and model-based educational quality filtering. The educational filter is a classifier that scores documents on their educational value; the paper selects only documents with scores above 2, which preferentially retains Wikipedia-like explanatory text, textbooks, and well-written articles while discarding low-information content such as comment sections, navigation menus, and auto-generated text.

For high-resource languages (specifically German, Spanish, French, and Italian), the paper uses the RedPajama-Data-v2 corpus (Computer, 2023), which has been pre-deduplicated at the document level. The authors apply additional filtering on top: a perplexity filter and a variety of heuristic filters. The perplexity filter works by training or using a language model on clean text in the target language, then scoring each candidate document — documents with anomalously high perplexity (indicating nonsensical, machine-generated, or severely noisy text) are discarded. The heuristic filters are rule-based quality checks (likely including minimum/maximum length thresholds, character repetition ratios, word-to-symbol ratios, and language-specific stop-word coverage checks), though the paper does not enumerate the exact heuristic rules.

For the remaining languages (which include lower-resource EU languages like Irish, Maltese, Estonian, and the additional non-EU languages), the paper concatenates data from four existing multilingual datasets: HPLT (de Gibert et al., 2024), MADLAD-400 (Kudugunta et al., 2023), CulturaX (Nguyen et al., 2023), and mC4 (Xue et al., 2021). These are all pre-existing cleaned multilingual web-crawl datasets. On this concatenated pool, the authors run their own comprehensive pipeline based on CCNet (Wenzek et al., 2019), which applies: (1) deduplication at the paragraph or document level across and within datasets, (2) language identification to ensure each document is actually in the claimed language (critical because web-crawled data frequently contains mislabeled or mixed-language content), (3) perplexity filtering using language-specific models, and (4) a set of heuristic quality filters. The CCNet-based pipeline is a standard in multilingual data preparation because it provides a reproducible sequence of filtering steps that can be applied consistently across many languages.

Design choice: tiered web data strategy. The tiered approach (FineWeb-edu for English → RedPajama with custom filtering for high-resource → concatenated multilingual datasets with CCNet for the rest) reflects a pragmatic reality: English has the most abundant and highest-quality pre-filtered data, high-resource European languages have good coverage in RedPajama, and lower-resource languages require pooling from multiple sources to accumulate enough usable text. This is not a theoretically optimal strategy but a response to the uneven distribution of web text across languages.

Parallel Data Collection and Filtering

Parallel data consists of sentence pairs or documents that are translations of each other across languages. The paper collects parallel data in both directions: into English (xx→en, i.e., non-English language to English) and from English (en→xx, i.e., English to non-English language). The sources are "various public sources" — likely including OPUS, ParaCrawl, Europarl, United Nations Parallel Corpus, and similar standard parallel corpora — though the paper does not enumerate them exhaustively.

Quality filtering for parallel data uses two automated metrics at the sentence-pair level:

• Bicleaner (Sánchez-Cartagena et al.; Ramírez-Sánchez et al., 2020): a classifier trained to distinguish clean translation pairs from noisy ones. The paper applies a threshold of 0.6 for Portuguese and 0.5 for all other languages, discarding pairs below these scores. The higher threshold for Portuguese reflects either higher data availability (allowing more aggressive filtering) or empirical observation that Bicleaner's Portuguese scoring distribution differs from other languages.

• COMETKIWI-22 (Rei et al., 2022b): a reference-free quality estimation model that predicts translation quality without access to a reference translation. The paper applies a threshold of 0.7, discarding pairs below this score. COMETKIWI-22 is a neural metric that captures semantic adequacy and fluency, complementing Bicleaner's focus on surface-level translation cleanliness.

The dual-filter strategy (both Bicleaner and COMETKIWI-22) ensures that retained parallel sentences are both clean (free of OCR errors, encoding issues, misalignments) and semantically adequate (the translation actually conveys the meaning of the source, not just looks superficially like a translation). This is important because parallel data directly teaches the model cross-lingual correspondences — low-quality parallel data would teach incorrect alignments between languages.

Code and Math Data

Code and mathematical data are included to improve the model's reasoning capabilities and to expose it to formal, structured languages that complement natural language. The paper uses three existing datasets:

• The Stack (Kocetkov et al., 2022): a large collection of permissively licensed source code from GitHub, covering numerous programming languages. This provides exposure to programming syntax, algorithmic reasoning patterns, and code documentation written in natural language.

• The Algebraic-stack (Azerbayev et al., 2023): a dataset of mathematical text including formal mathematics, proofs, and mathematical discussions. This is specifically designed to improve mathematical reasoning in language models.

• Open-web-math (Paster et al., 2023): a collection of mathematical content extracted from web pages, providing additional coverage of mathematical notation, problem-solving patterns, and explanatory mathematical writing.

Design choice: including code and math in a multilingual model. This decision reflects the observation (well-established in the LLM literature, e.g., from the LLaMA, Mistral, and Gemini technical reports) that code and mathematical data improve general reasoning capabilities even on non-code, non-math tasks. The structured, logical nature of code appears to transfer to improved performance on tasks requiring multi-step reasoning, instruction following, and structured output generation. For a multilingual model, this is particularly valuable because reasoning capability is largely language-agnostic — improving it in one language tends to transfer across languages.

High-Quality Data

High-quality data consists of curated, reliably accurate text sources that provide clean training signal. The paper uses:

• Wikipedia (Foundation): included for all languages, not just English. This is the only high-quality data source specified as multilingual. Wikipedia articles are human-written, factually oriented, and available in many languages, making them a standard component of multilingual LLM training.

• Arxiv (Clement et al., 2019): scientific papers from arXiv, used for English only. This provides exposure to academic writing, technical terminology, and structured scientific reasoning.

• Books (Zhu et al., 2015): a corpus of books, used for English only. Book-length texts provide long-range narrative coherence and literary language that web-crawled text typically lacks.

• Apollo (Wang et al., 2024): a medical-domain dataset, used for English only. This provides specialized medical knowledge and terminology.

Design choice: English-dominated high-quality data. The high-quality data sources other than Wikipedia are English-only, reflecting the reality that curated, high-quality text corpora are overwhelmingly available in English. This is a genuine limitation: the model receives cleaner, more reliable training signal in English than in other languages, which may contribute to the performance gap on English-only benchmarks that the paper observes. The decision is pragmatic — high-quality curated data in Irish, Maltese, or Estonian simply does not exist at the scale needed for LLM pretraining — but it means the model's factual knowledge and reasoning patterns may be English-centric even though its language generation capabilities are multilingual.

Annealing Data

In the final 10% of pretraining, the paper increases the proportion of high-quality data in the training mixture — a practice known as annealing or data curriculum learning. The intuition is that the final phase of training disproportionately influences the model's final parameter state, so feeding it higher-quality data at the end improves the model's output quality.

The annealing-specific data includes:

• Educational-quality-filtered monolingual data: The paper trains a binary classifier inspired by FineWeb-Edu (Lozhkov et al., 2024) that predicts whether a document has educational value. This classifier is applied to filter the monolingual web data, retaining only documents that score above a threshold. This is analogous to the FineWeb-edu score>2 filtering used for English web data, but extended to all languages. The classifier is trained specifically for this purpose, meaning the filtering criteria for annealing are adapted to the EuroLLM data distribution rather than borrowed from an English-only classifier.

• Cosmopedia-v2 (Ben Allal et al., 2024): a synthetic dataset generated by Mixtral-8x7B-Instruct-v0.1 (Jiang et al., 2024) consisting of textbooks, blog posts, and stories. This provides diverse, well-structured text that resembles human educational content but is machine-generated. Using synthetic data in the annealing phase is an interesting choice: it provides high-quality formatting and structure, but risks introducing artifacts from the teacher model (Mixtral).

• Python-Edu (Ben Allal et al., 2024): a subset of Python code from The Stack filtered specifically for educational value — i.e., well-commented, pedagogically structured Python code rather than arbitrary repository code. This targets improved code reasoning in the final training phase.

• GSM8K training set (Cobbe et al., 2021): Grade School Math 8K, a dataset of grade-school-level math word problems with step-by-step solutions. This provides explicit mathematical reasoning examples.

• MATH training set (Hendrycks et al., 2021): Mathematics Aptitude Test of Heuristics, a dataset of competition-level mathematics problems. This provides more challenging mathematical reasoning examples than GSM8K.

• Document-level parallel data: Europarl (Koehn, 2005), the proceedings of the European Parliament in multiple languages, and ParaDocs (Wicks et al., 2024), a document-level parallel corpus. Unlike the sentence-level parallel data used in the main training phase, these provide cross-lingual correspondences at the document level, potentially teaching the model longer-range cross-lingual alignment.

Data composition summary. Figure 1 in the paper shows the percentage attributed to each data category. In the first training phase (the main 90% of pretraining), the composition by broad category is depicted in the left pie chart (exact percentages are not given in the text but are visually represented). In the annealing phase (the final 10%), shown in the right pie chart, the high-quality data proportion increases and the distribution shifts — the English allocation drops from 50% to 32.5%, while other languages and code/math data increase their share.

---

3.4.2 Data Mixture Optimization

The data mixture — how many tokens to allocate to each language and each data type — is the single most impactful design decision in multilingual LLM training. Allocate too much to English, and other languages suffer; allocate too aggressively to low-resource languages, and the model's overall capacity may be diluted. The paper makes this decision through three interconnected analyses: determining the optimal proportion of parallel data, deciding whether to repeat high-quality data, and allocating tokens across languages.

Joint Scaling Laws for Parallel Data Proportion

The central question: what percentage of each non-English language's training data should be parallel (translated) data rather than monolingual data? Parallel data has two hypothesized benefits: (1) it improves cross-lingual alignment by teaching the model explicit translation correspondences, and (2) it enhances machine translation capability specifically. But it also has a potential cost: parallel data may be less diverse in content than monolingual web data, and over-reliance on it could degrade performance on non-translation tasks.

To answer this question without training the full 1.7B model on multiple data mixtures (which would be prohibitively expensive at 4 trillion tokens), the paper employs joint scaling laws. This methodology, adapted from Fernandes et al. (2023), works as follows:

1. Train small proxy models: Multiple models are trained with varying non-embedding parameter counts $N$ — specifically 100M, 203M, and 341M parameters — on a smaller corpus of 100B tokens (1/40th of the full training budget). Each proxy model uses a different proportion of parallel data for each non-English language: 0%, 25%, and 37.5% of the per-language data budget allocated to parallel sentences.

2. Measure performance across domains: For each trained proxy model, the paper measures loss (perplexity) on held-out test sets from three domains: web data (monolingual web text), Wikipedia data (encyclopedic text), and parallel data (translation pairs). This captures how parallel data proportion affects both monolingual performance (web, Wikipedia) and cross-lingual performance (parallel).

3. Fit a joint scaling law: The core equation is:

$L(N, p) = f(p) \cdot \beta \cdot N^{-\alpha} + L_\infty$

where $L(N, p)$ is the predicted loss for a model with $N$ non-embedding parameters trained with a language that receives proportion $p$ of the total data (or, in this application, where parallel data constitutes proportion $p$ of that language's allocation). $f(p)$ is a ratio function capturing how performance scales with the data proportion, $\beta$ is a scaling coefficient, $\alpha$ is the power-law exponent governing how loss decreases with model size, and $L_\infty$ is the irreducible loss (the minimum achievable loss even with infinite parameters and data).

The ratio function $f(p)$ has a specific parametric form:

$f(p) = p + c_1 p^{c_2} (1-p)^{c_3}$

where $c_1$, $c_2$, and $c_3$ are empirically estimated parameters. This functional form captures several behaviors: when $p = 0$ (no parallel data), $f(p) = 0$, meaning performance on the task measured at that data proportion is at its baseline; as $p$ increases, the $p$ term captures linear improvement from adding data, while the $c_1 p^{c_2} (1-p)^{c_3}$ term captures interaction effects between the specific data type being varied and the remaining data. This allows the law to predict performance for data proportions not seen during proxy model training.

What it computes: For a given model size $N$ and a given parallel-data proportion $p$, the equation predicts the expected loss on a specific test domain. By fitting $\alpha$, $\beta$, $L_\infty$, and $c_{1,2,3}$ from the small proxy model experiments, the law can be extrapolated to the target model size (1.7B parameters) to predict what would happen at different parallel-data proportions — without ever training a 1.7B model on those mixtures.

Why this form: The power-law form $\beta N^{-\alpha} + L_\infty$ is the standard neural scaling law from Kaplan et al. (2020), reflecting the empirical observation that model loss decreases as a power of model size. The ratio function $f(p)$ multiplies this to capture how the language-specific data allocation modifies the effective capacity. The specific form $p + c_1 p^{c_2} (1-p)^{c_3}$ is flexible enough to capture diminishing returns (as $p$ increases, the second term can saturate), while ensuring that $f(0) = 0$ and $f(1)$ is finite. Alternative forms — such as a simple linear function $f(p) = kp$ — would fail to capture the saturating behavior the paper observes (going from 25% to 37.5% parallel data yields diminishing returns).

Results from the scaling law analysis (Figure 2):

The key findings, shown in Figure 2, are:

• On web data test sets: The scaling law curves for 0%, 25%, and 37.5% parallel data are very close together across the range of model sizes. Adding parallel data does not degrade monolingual web text performance — the model seems able to use the parallel data to learn cross-lingual patterns without sacrificing its ability to model monolingual text. This is non-obvious: one might expect that replacing diverse web text with repetitive translation pairs would hurt monolingual fluency, but the results suggest the model has sufficient capacity (even at 100M–341M parameters) to absorb the parallel signal without crowding out monolingual knowledge.

• On Wikipedia test sets: Similar to web data — parallel data does not hurt Wikipedia-domain performance. The curves largely overlap.

• On parallel data test sets: Adding parallel data significantly improves performance on parallel-domain test sets (translation tasks). The 25% curve is clearly below (better than) the 0% curve, and the 37.5% curve is below the 25% curve. However, the gap between 25% and 37.5% is smaller than the gap between 0% and 25% — diminishing returns set in, meaning the marginal benefit of additional parallel data decreases as the proportion increases.

Decision: Based on these scaling law predictions, the paper chooses 20% parallel data for each non-English language in the final training corpus. This is slightly below the 25% tested in the proxy experiments, reflecting a conservative choice given the diminishing returns observed: 20% captures most of the translation benefit while preserving more monolingual data budget for general language modeling. This decision applies per-language — for a language like French that receives X tokens in the final corpus, 0.2X of those tokens are parallel sentence pairs (en→fr and fr→en combined).

Joint Scaling Laws for Data Repetition

A separate but related question: should high-quality datasets like Wikipedia be repeated (seen multiple times during training) or not? Repeating data increases the effective exposure to high-quality text but risks overfitting — the model might memorize Wikipedia rather than learning generalizable patterns.

The methodology mirrors the parallel-data analysis:

1. Train small proxy models on 100B-token corpora, one where Wikipedia data is repeated (oversampled) and one where it is not (each Wikipedia article appears at most once in the training data, with the remaining tokens filled by other data).

2. Measure performance on Wikipedia test sets and web data test sets.

3. Fit scaling laws to extrapolate to larger model sizes.

Results (Figure 3):

• On Wikipedia test sets: Repeating Wikipedia data clearly improves performance (lower loss on Wikipedia-domain text). This is expected — the model sees more Wikipedia, so it models Wikipedia better.

• On web data test sets: Repeating Wikipedia data does not degrade performance on web text. The curves for repeat and no-repeat conditions are very close, indicating that the extra Wikipedia exposure does not crowd out learning of general web text patterns.

Decision: The paper chooses to repeat data from high-quality datasets. This means Wikipedia articles (and likely other high-quality sources) are sampled multiple times over the course of training, rather than being seen exactly once. The rationale is that high-quality data provides disproportionately valuable training signal, and the risk of harmful overfitting is low enough (as evidenced by the web-data test set results) to justify the repetition. This is consistent with practices in other LLM training pipelines (e.g., LLaMA, which repeats high-quality sources), but the paper validates the decision empirically rather than adopting it as a heuristic.

Division Between Languages

The final data mixture decision: how to allocate the total training token budget across the 35 supported languages. This is not determined through scaling laws but through a pragmatic approach that respects both data availability and the desire for a multilingual model:

• English receives 50% of tokens in the main training phase. This is justified because "both high-quality data and web data are predominantly in English" — English has by far the most available clean training data, and much of the high-quality data (Arxiv, Books, Apollo) is English-only. Allocating half the budget to English ensures the model develops strong general reasoning and knowledge capabilities, which can then transfer to other languages through the parallel data and shared multilingual representations.

• Code and math data receive 5% of tokens in the main training phase. This covers The Stack, Algebraic-stack, and Open-web-math. The allocation is small relative to natural language but sufficient to provide reasoning signal.

• The remaining 45% of tokens are distributed among the other 34 languages. The allocation is based on "the amount of data obtained after the collection and filtering processes" — languages with more available clean data (e.g., German, French, Spanish, Italian) receive more tokens than languages with less available data (e.g., Irish, Maltese). This is essentially a data-availability-proportional allocation after carving out the English and code/math budgets, rather than an equal-per-language or scaling-law-optimized allocation.

Annealing phase adjustment: In the final 10% of training, the language distribution shifts:

• English drops from 50% to 32.5% — a significant reduction, freeing 17.5% of the total token budget.

• The freed tokens are distributed across the other languages, increasing their per-language allocation. This directly targets improved multilinguality in the final phase: the model has already learned strong English capabilities in the first 90% of training, and the annealing phase can focus on strengthening non-English performance.

• Code and math data increases from 5% to 7% — a smaller increase that provides additional reasoning signal in the critical final phase.

Figure 4 shows the exact percentage allocated to each language in both phases. The visual makes clear that languages vary substantially in their allocation: high-resource European languages (German, French, Spanish) receive larger slices, while low-resource languages (Maltese, Irish) receive very thin slices. This is an acknowledged limitation — the data mixture is constrained by what data exists, not an optimal allocation that a scaling law might prescribe if unlimited data were available for all languages.

---

3.4.3 Multilingual Tokenizer

The tokenizer converts raw text into sequences of integer token IDs that the Transformer processes. For a multilingual model, the tokenizer design is critical because it determines how efficiently text in each language can be represented. An English-optimized tokenizer will fragment non-English text into many small subword pieces, wasting the model's context window and forcing it to learn from unnecessarily long sequences.

Tokenizer Training Procedure

The paper adopts the approach used by LLaMA-2 (Touvron et al., 2023) and Mistral (Jiang et al., 2023): a byte-level BPE (Byte-Pair Encoding) tokenizer with byte-fallback, trained using the SentencePiece framework (Kudo & Richardson, 2018).

The byte-level BPE algorithm works as follows:

1. Initial vocabulary: Start with the 256 bytes as the base vocabulary. Every Unicode character can be represented as a sequence of bytes, so this ensures the tokenizer can encode any text in any language without unknown tokens — if a character is not covered by any learned subword, it falls back to byte-level encoding.

2. Iterative merging: On a training corpus representative of the target text distribution, repeatedly find the most frequent adjacent pair of tokens (initially bytes) and merge them into a new token, adding it to the vocabulary. Repeat until the vocabulary reaches the target size (128,000 pieces).

3. Byte-fallback: At tokenization time, if the input text contains a character sequence that cannot be represented by any learned subword token, the tokenizer falls back to encoding it as individual bytes. This guarantees that no text is ever "out of vocabulary" — even rare or unseen characters are encodable.

The training corpus for the tokenizer needs to represent all target languages to ensure the vocabulary includes subword units optimized across the multilingual distribution. The paper does not specify the exact composition of the tokenizer training data, but standard practice is to sample proportionally to the intended training data mixture, ensuring languages with more training tokens get more vocabulary representation.

Vocabulary Size Tradeoff

The paper identifies a fundamental tension: larger vocabularies improve tokenization efficiency (lower fertility — fewer tokens per word) but increase the number of embedding parameters. The embedding matrix maps each token ID to a dense vector of size $d_{\text{model}}$. With an embedding size of 2,048 and 128,000 tokens, the embedding parameters account for:

$128,000 \times 2,048 = 262,144,000$

parameters — approximately 262M, which is 15.8% of the total 1.657B parameters. If the vocabulary were doubled to 256,000, the embedding parameters would double to ~524M, consuming nearly a third of the model's parameter budget on just the input embedding table. Conversely, a smaller vocabulary like Mistral's 32,000 would save parameters but increase fertility for non-English languages, making the model less efficient at processing non-English text.

The paper states: "Through experimentation, we reach the conclusion that a vocabulary of 128,000 pieces provides the best trade-off." The experimentation is not detailed, but the reasoning is clear: after accounting for the 262M embedding parameters and a symmetric 262M for the output projection (LM head), the remaining 1.133B non-embedding parameters are the model's "thinking" capacity. Too large a vocabulary would cannibalize this; too small would degrade multilingual efficiency.

Fertility Comparison

Fertility is defined as pieces per word — the average number of tokenizer output tokens needed to represent one whitespace-delimited word. Lower fertility is better because it means the model can process semantic content in fewer tokens, leaving more context window for reasoning.

Figure 5 compares the EuroLLM tokenizer (128,000 pieces) against:

• Mistral (32,000 pieces): a small, English-optimized vocabulary.
• LLaMA-3 (128,256 pieces): a vocabulary of similar size to EuroLLM.
• Gemma (256,000 pieces): a much larger vocabulary.

The fertility comparison across a subset of EuroLLM languages reveals:

• Compared to Mistral: EuroLLM's larger vocabulary yields "significantly lower fertilities" across all shown languages. For languages with complex morphology (Finnish, Hungarian, Estonian), the difference is especially pronounced because Mistral's 32,000-piece vocabulary lacks sufficient subword units to represent the rich inflectional patterns of these languages efficiently. For example, a single Finnish word like "talossanikinko" (meaning "in my house too?") might be 5-7 tokens in Mistral but 3-4 in EuroLLM, directly improving the model's effective context length for Finnish text.

• Compared to LLaMA-3: Despite having similar vocabulary sizes (128,000 vs. 128,256), LLaMA-3 shows the lowest fertility in English — it is optimized for English even with a large vocabulary. For most other languages, EuroLLM achieves lower fertility. This suggests that vocabulary size alone doesn't determine multilingual efficiency; the tokenizer training data composition matters substantially. LLaMA-3's tokenizer was likely trained on English-heavy data even with a large vocabulary, while EuroLLM's tokenizer training data was deliberately balanced across languages.

• Compared to Gemma: Gemma (256,000 pieces) is better for Asian languages but "very similar to EuroLLM for the European ones." This makes sense: Gemma's very large vocabulary provides ample representation for the character sets and morphological patterns of Asian languages (Japanese, Korean, Chinese), which differ substantially from Latin-script languages. For European languages, the additional vocabulary pieces beyond 128,000 provide diminishing returns — most useful subword merges have already been captured.

Design choice: 128,000 vocabulary size. This choice balances three factors: (1) embedding parameter cost (262M parameters, or ~15.8% of the model), (2) fertility across all target languages (demonstrably better than Mistral and competitive with LLaMA-3 for non-English), and (3) training stability (very large vocabularies can slow training due to the large embedding matrix updates, though this is not discussed explicitly). The explicit comparison against multiple baselines provides empirical justification rather than relying on a heuristic choice.

---

3.4.4 Model Architecture and Pre-Training

Architecture Choices

EuroLLM-1.7B uses a standard, dense Transformer architecture (Vaswani et al., 2017) — it is not a mixture-of-experts or any other non-standard variant. The specific components and their justifications:

Grouped Query Attention (GQA) with 8 key-value heads. Standard multi-head attention computes separate key, query, and value projections for each attention head. Multi-query attention (MQA) shares keys and values across all heads, reducing memory and compute at inference but potentially degrading quality. GQA (Ainslie et al., 2023) is a middle ground: it groups query heads to share key-value pairs. With 16 total query heads and 8 key-value heads, each key-value head serves 2 query heads. The paper cites the justification that GQA "has been shown to increase speed at inference time while maintaining downstream performance" (Team et al., 2024b). For a multilingual model that may be deployed in resource-constrained settings, inference efficiency matters.

Pre-layer normalization with RMSNorm. In a Transformer, normalization can be applied before or after each sub-layer (attention or FFN). Pre-norm (Xiong et al., 2020) — normalizing the input before the sub-layer rather than the output after — improves training stability by preventing gradients from exploding through residual connections. RMSNorm (Zhang & Sennrich, 2019) is a simplification of LayerNorm (Ba et al., 2016) that removes the mean-centering step, computing:

$\text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d}\sum_{i=1}^d x_i^2}} \cdot \gamma$

where $x$ is the input vector, $d$ is its dimensionality, and $\gamma$ is a learnable scale parameter. The paper notes RMSNorm is "faster than LayerNorm" — it requires fewer operations per normalization, which adds up over 24 layers and billions of training tokens.

SwiGLU activation function. The feed-forward network in each Transformer layer uses the SwiGLU activation (Shazeer, 2020) instead of the original ReLU or GELU. SwiGLU is a gated variant:

$\text{SwiGLU}(x) = \text{Swish}(xW_1) \odot (xW_2)$

where $\text{Swish}(z) = z \cdot \sigma(z)$ with $\sigma$ being the sigmoid function, $W_1$ and $W_2$ are learned weight matrices, and $\odot$ is element-wise multiplication. The gating mechanism allows the FFN to learn which information to pass through, leading to better downstream performance (Shazeer, 2020; Le Scao et al., 2022). The FFN hidden size is 5,632, which is approximately $4 \times 2,048 \times 2/3$ due to the SwiGLU architecture having three weight matrices (gate, value, output) instead of the standard two.

Rotary Positional Embeddings (RoPE). RoPE (Su et al., 2024) encodes positional information by rotating the query and key vectors in attention based on their positions. The rotation angle for position $pos$ and dimension index $i$ is:

$\theta_{pos,i} = pos \cdot \Theta^{-2i/d}$

where $\Theta = 10,000$ (the base frequency). This means lower dimensions encode coarse positional information (slow rotation), and higher dimensions encode fine positional detail (fast rotation). The paper notes RoPE "allows the extension of the context length" — because RoPE uses continuous rotations rather than learned absolute position embeddings, it can generalize to sequence lengths longer than those seen during training (through techniques like position interpolation). The choice of $\Theta = 10,000$ is standard (matching the original RoPE paper and many subsequent models).

No tied embeddings. The paper explicitly states "Tied Embeddings: No" — the input embedding matrix and output projection (LM head) are separate parameter matrices, each with 262M parameters. Tying embeddings (sharing weights between input and output) would save parameters but can constrain the model, especially for a multilingual model where the input embedding needs to encode a wide variety of scripts and the output needs to predict the next token distribution — the optimal representations may differ.

Full hyperparameter table (Table 1):

The model has approximately $24 \times (4 \times 2048 \times 5632 + 2048^2) \approx 1.1\text{B}$ non-embedding parameters (this back-of-envelope estimate accounts for the attention projections, FFN weights, and layer norms across 24 layers). The remaining ~0.52B parameters are in the embedding and LM head matrices.

Training Configuration

The model is pre-trained on 4 trillion tokens (4T), using 256 Nvidia H100 GPUs on the Marenostrum 5 supercomputer. The batch size is constant at 3,072 sequences, where each sequence is up to 4,096 tokens, giving approximately 12 million tokens per batch ($3072 \times 4096 \approx 12.6\text{M}$). The total number of training steps is therefore roughly $4 \times 10^{12} / 12.6 \times 10^6 \approx 317,000$ steps.

The optimizer is Adam (Kingma & Ba, 2014) with bfloat16 mixed precision. BF16 (Google Brain's 16-bit floating point format) has the same exponent range as FP32 but reduced mantissa precision — this means it handles the wide dynamic range of gradients during training better than FP16, reducing the need for loss scaling, while still being memory-efficient.

Learning Rate Scheduler: Trapezoid vs. Cosine

The paper empirically compares two learning rate scheduler options — a decision evaluated because the scheduler controls how the learning rate evolves over training, directly affecting convergence speed and final model quality.

Option 1: Cosine scheduler. The standard choice in many LLM training recipes. The learning rate follows a cosine curve from the maximum to the minimum over the course of training, with a warm-up phase at the start:

• Warm-up for 10% of the steps: learning rate linearly increases from 0 (or a small value) to the maximum learning rate ($3 \times 10^{-4}$).
• For the remaining 90% of steps: learning rate follows a half-cosine decay from the maximum to the minimum learning rate ($3 \times 10^{-5}$).

The cosine decay means the learning rate drops fastest in the middle of training and slowest at the beginning and end.

Option 2: Trapezoid scheduler (also called Warmup-Stable-Decay; Xing et al., 2018; Hu et al., 2024). This has three distinct phases:

1. Warm-up for 10% of steps: Same as cosine — linear increase to $3 \times 10^{-4}$.
2. Constant learning rate: The learning rate stays at the maximum ($3 \times 10^{-4}$) for the middle 80% of training steps.
3. Linear decay for the final 10% of steps: The learning rate linearly decreases from the maximum to the minimum ($3 \times 10^{-5}$).

The key difference from cosine: the learning rate stays high for most of training rather than continuously declining, then drops sharply in the final annealing phase. This means the model continues to learn aggressively (large updates) through the bulk of training, and the final phase of rapid decay coincides with the introduction of the higher-quality annealing data.

Comparison results (Table 2):

The two scheduler variants are compared on two types of evaluation:

• Multilingual general benchmarks: Hellaswag (commonsense natural language inference) and Arc Challenge (science exam questions), both in their translated multilingual versions from Lai et al. (2023).
• Machine translation: Three test sets — FLORES-200 (Team et al., 2022), WMT-23 (Kocmi et al., 2023), and WMT-24 (Kocmi et al., 2024) — evaluated using COMET-22 (Rei et al., 2022a).

The results show the trapezoid scheduler outperforming cosine on all metrics:

The improvement is consistent: +0.0098 on Hellaswag, +0.0062 on Arc Challenge, +0.27 on FLORES-200, +0.25 on WMT-23, and +0.48 on WMT-24. While these are modest absolute improvements, they are consistent across all evaluation types and come at no additional computational cost — the trapezoid scheduler uses the same number of training steps as the cosine scheduler.

Why the trapezoid scheduler might work better for multilingual training: The paper does not offer a mechanistic explanation, but a plausible hypothesis relates to the interaction between the scheduler shape and the data mixture shift in the annealing phase. With a cosine scheduler, the learning rate is already quite low by the time the annealing data is introduced, meaning the model makes only small updates on the high-quality multilingual data. With the trapezoid scheduler, the learning rate stays high through the main training, then drops sharply during the annealing phase — the model can make relatively larger updates when first exposed to the higher-quality data, potentially learning more from it. Additionally, the constant high learning rate through most of training may help the model explore the loss landscape more broadly, which could be beneficial for a multilingual model that needs to learn representations covering diverse linguistic patterns.

Design choice: trapezoid scheduler selected for future models. The paper states this comparison is to "decide which option to use in future models," and the results clearly favor the trapezoid scheduler. The full EuroLLM-1.7B model (the released version) uses the trapezoid scheduler.

The annealing phase (final 10% of training) serves dual purpose: it coincides with both the learning rate linear decay and the shift to higher-quality, more multilingual data. This coupling of scheduler design and data curriculum is intentional — the model experiences the most rapid learning rate reduction while being exposed to the cleanest training data.

---

3.4.5 Instruction Fine-Tuning

EuroBlocks Dataset Construction

To convert the base EuroLLM-1.7B into an instruction-following conversational model (EuroLLM-1.7B-Instruct), the paper creates EuroBlocks, a multilingual instruction-tuning dataset of approximately 1M samples covering all supported languages and "a variety of tasks."

The dataset combines three types of data:

• Instruction-following conversations from OpenHermes-2.5 (Teknium, 2023): a large, diverse dataset of synthetic and human-written instruction-following examples covering tasks like question answering, text generation, summarization, coding, and creative writing. OpenHermes-2.5 is primarily English, providing high-quality instruction-following signals.

• Multilingual instruction data from Aya (Singh et al., 2024): a dataset specifically designed for multilingual instruction tuning, covering 101 languages. This provides instruction-following examples directly in the target languages, teaching the model to respond to prompts in non-English languages.

• High-quality machine translation examples from four parallel test sets: NTREX-128 (Federmann et al., 2022), FLORES-200-DEV (Team et al., 2022), WMT-21 (Farhad et al., 2021), and WMT-22 (Kocmi et al., 2022). These are human-translated or professionally translated sentence pairs that serve as exemplars of correct translation behavior. By including translation examples directly in the instruction-tuning data, the model learns to perform translation as an instruction-following task (e.g., responding to "Translate to French: The European Commission is..." with the correct French translation).

The inclusion of both general instruction-following data and translation-specific data is important: the OpenHermes and Aya data teach broad conversational and task-following abilities, while the translation data specifically teaches the model to produce accurate translations in a structured format. This mix ensures EuroLLM-1.7B-Instruct can both engage in open-ended multilingual dialogue and perform reliable machine translation when explicitly asked.

Supervised Fine-Tuning (SFT) Procedure

The fine-tuning uses standard next-token prediction with cross-entropy loss, applied only to the model's output tokens (the assistant's responses) — loss on the prompt tokens (user messages, system prompts) is masked. The training configuration:

• Learning rate: $7 \times 10^{-6}$, which is substantially lower than the pretraining maximum ($3 \times 10^{-4}$), as is standard for fine-tuning to avoid catastrophic forgetting.
• Epochs: 4 epochs over the 1M-sample dataset.
• Steps: Approximately 6,000 steps (implying a batch size of roughly $1,000,000 / 6,000 \times 4 \approx 667$ samples per step, though exact batch size is not specified).
• Mixed precision: bfloat16, same as pretraining.
• Packing: enabled — multiple short conversations can be packed into a single training sequence up to the 4,096-token context length, improving training efficiency.
• Loss masking: "only calculate the loss on target tokens" — the model is not penalized for failing to predict the user's prompt, only for its own response.

Chat Formatting with Control Tokens

The instruction-tuned model uses a structured chat format with special control tokens to delineate conversation turns. The tokens are:

• <|im start|>: marks the beginning of a message.
• <|im end|>: marks the end of a message.

These tokens are included in the original tokenizer described in Section 3 — they are added to the vocabulary before pretraining (or during tokenizer construction) rather than during SFT. The paper states "no tokens are added during SFT," which means the control tokens existed in the base model's vocabulary from the start, presumably as reserved special tokens.

One important modification: the end-of-sequence token is overwritten with the <|im end|> token during SFT. This means the model learns to treat <|im end|> as the signal to stop generating, and the standard EOS token is repurposed. This is a design choice that ties the conversation structure to the generation termination — the model learns that generating <|im end|> both ends its turn and signals that it's the user's turn to speak.

The chat template supports three roles:

• System (optional): prefixed with <|im start|>system and ending with <|im end|>. The system prompt provides meta-instructions that steer the model's behavior throughout the conversation (e.g., "Translate all user texts to English.").

• User: prefixed with <|im start|>user and ending with <|im end|>. Contains the user's message or request.

• Assistant (model): prefixed with <|im start|>assistant and ending with <|im end|>. The model's generated response.

Table 3 in the paper provides a concrete example of a multi-turn dialogue where the system prompt instructs the model to translate, and the model correctly translates Portuguese and Spanish texts to English in successive turns. The formatting is consistent and machine-parseable: each turn is clearly bounded by the control tokens, and the model can maintain context across turns.

Design choice: chat format control tokens. Using explicit control tokens rather than relying on natural language markers (like "User:" and "Assistant:") has two advantages: (1) the special tokens are unambiguous and cannot appear in natural text, preventing the model from confusing formatting with content; (2) they are native to the tokenizer, meaning they are single tokens that the model can process efficiently. The choice of <|im start|> and <|im end|> follows conventions from models like ChatML, providing compatibility with existing tooling and making the model easier to deploy with standard inference frameworks.

4. Key Insights and Innovations

Innovation 1: Multilinguality as a First-Class Design Constraint Across the Entire Training Pipeline

The dominant approach to multilingual LLMs in the open-weight ecosystem has been retroactive multilingualism: take an architecture, tokenizer, and training recipe designed for English, add some non-English data to the training mix, and hope the model acquires multilingual capabilities as a byproduct. Models like LLaMA, Mistral, and Gemma are all English-first in this sense — their tokenizers prioritize English fertility, their data mixtures are English-dominant by volume rather than by explicit design tradeoff, and their training recipes are validated on English benchmarks. Multilingual performance is emergent, not engineered.

EuroLLM inverts this. The paper treats multilinguality as a constraint that propagates backward through every design decision. The choice of vocabulary size (128,000 pieces) is justified by fertility parity across languages, not just English efficiency — the explicit comparison in Figure 5 shows that EuroLLM's tokenizer achieves lower fertility than Mistral for all non-English languages and lower fertility than LLaMA-3 for most European languages, at the cost of slightly worse English fertility than LLaMA-3. The data mixture allocates exactly 50% of tokens to English — a deliberate cap, not a floor, reflecting a conscious decision to trade some English capability for non-English representation. The annealing phase further shifts tokens away from English (to 32.5%) and toward other languages, meaning the final, most influential phase of training actively prioritizes non-English improvement. Even the learning rate scheduler choice — trapezoid over cosine — is justified not by English-only benchmarks but by multilingual benchmarks (Hellaswag, Arc Challenge in translation) and machine translation across three test sets.

What makes this conceptually distinctive is not any single decision but the systematicity: every stage of the pipeline is explicitly evaluated for its multilingual impact, and tradeoffs between English and non-English performance are made visible rather than obscured. The fertility comparison (Figure 5) is the clearest diagnostic: it quantifies the tokenizer's language bias and makes it a design target. The scaling-law experiments on parallel data proportion (Figure 2) make the English-vs-other tradeoff explicit — adding parallel data helps translation without hurting monolingual performance, but the benefit saturates, allowing a principled choice of 20% rather than a heuristic. This framing — multilinguality as a constraint to be satisfied across stages, not a feature to be added at training time — is what distinguishes the paper from the "just add more languages to the training data" approach that characterizes most prior open-weight LLM releases.

The significance is broader than the specific model. By making multilinguality a pipeline-level design constraint and showing that this yields competitive non-English performance at small parameter counts (1.7B matching or exceeding Gemma-2B on non-English Hellaswag, competing with Gemma-7B on translation), the paper provides a template for future multilingual model development: treat the tokenizer and the data mixture and the training recipe as a coupled design problem where every choice has language-specific consequences that should be measured, not assumed.

Innovation 2: Joint Scaling Laws as a Practical Tool for Multilingual Data Mixture Decisions

Prior work on data mixture for multilingual models has been largely heuristic: sample proportional to data volume (which starves low-resource languages), sample uniformly (which can dilute the model on high-resource languages where more data is available), or iterate manually through trial-and-error. Fernandes et al. (2023) introduced the theoretical framework of joint scaling laws — predicting language-specific performance as a function of model size and data weighting — but their work focused on neural machine translation models, not general-purpose LLMs, and the framework remained a descriptive tool rather than a design instrument.

This paper operationalizes joint scaling laws as a practical decision-making tool for LLM pretraining design. The methodology is applied to two concrete questions that would otherwise require either expensive trial-and-error or unprincipled heuristics: (1) what proportion of parallel data to include per language, and (2) whether repeating high-quality data helps or hurts. In both cases, the answer is non-obvious before the experiment. For parallel data: one might worry that replacing diverse monolingual web text with repetitive translation pairs would hurt monolingual fluency. The scaling law shows it does not (the web and Wikipedia test set curves in Figure 2 overlap across 0%, 25%, and 37.5% parallel data conditions), while significantly improving translation performance — but with diminishing returns, justifying the conservative choice of 20%. For data repetition: one might worry about overfitting, but the scaling law shows Wikipedia-domain performance improves while web-domain performance is unaffected, justifying repetition.

The intellectual move here is treating scaling laws as a cheap proxy for design iteration. Training even a single 1.7B-parameter model on 4T tokens is computationally expensive (256 H100 GPUs for the full run); training multiple such models to compare data mixture choices would be prohibitive. By training small proxy models (100M–341M parameters) on a smaller corpus (100B tokens), fitting the joint scaling law, and extrapolating to the target model size, the paper can explore the data mixture design space for a fraction of the cost. The extrapolation from 100B tokens and 100M parameters to 4T tokens and 1.7B parameters is a significant leap, and the fact that the resulting model performs well (competitive with larger models on multilingual benchmarks) provides retrospective validation that the scaling-law predictions were informative.

This is an incremental but practically significant contribution: it takes an existing theoretical framework and demonstrates that it can guide concrete engineering decisions in a large-scale LLM project. The finding that parallel data provides diminishing returns beyond ~25% (with the marginal benefit from 25% to 37.5% being smaller than from 0% to 25%) is a specific, quantitative result that other multilingual LLM projects can use as a reference point. The framework itself — train small models, fit scaling laws, extrapolate to target size, make decisions — is generalizable to any multilingual model development effort.

Innovation 3: The Trapezoid Scheduler as a Better Default for Multilingual Training with Data Curriculum

The standard learning rate scheduler in LLM pretraining is a cosine decay with warmup — used in models from GPT-3 to LLaMA to Chinchilla. The trapezoid scheduler (warmup → constant → linear decay) is not new — it has been used in some smaller-scale training efforts (MiniCPM, Hu et al., 2024) — but its systematic comparison against cosine scheduling specifically for multilingual training with a data curriculum shift is novel and produces a finding with practical implications.

The experiment in Table 2 is simple: train two models identically except for the scheduler shape, evaluate on multilingual benchmarks and machine translation, and compare. The trapezoid scheduler wins across all five metrics. But the significance is not the raw performance delta (+0.0098 on Hellaswag accuracy, +0.27 on FLORES-200 COMET) — it is the hypothesized interaction between scheduler shape and the annealing data shift. The trapezoid scheduler keeps the learning rate high through the main training (allowing aggressive optimization on the main data mixture), then decays it linearly during the annealing phase (allowing the model to settle into a good local optimum when exposed to the higher-quality, more multilingual data). A cosine scheduler, by contrast, has already decayed the learning rate substantially by the time annealing begins, meaning the model's response to the higher-quality data is more muted.

This is a diagnostic finding rather than a theoretical one: it suggests that when training with a data curriculum (a shift in data distribution partway through training), the learning rate schedule should be designed to interact with that shift intentionally, not independently. The trapezoid scheduler couples the learning rate decay to the data quality improvement — the model learns aggressively when the data is good. This is a hypothesis that the paper does not prove mechanistically (it does not ablate the annealing phase separately to show that the trapezoid advantage disappears without it), but the consistency of the improvement across diverse metrics (general benchmarks and translation, across multiple test sets) makes it compelling as an empirical observation worth adopting.

The practical implication is clear: future multilingual LLM projects that use data annealing should consider the trapezoid scheduler as a default, not retain cosine scheduling from English-only precedents. The computational cost of this comparison (training two full models or running a smaller-scale proxy experiment) is modest relative to the total training budget, making it a low-risk, potentially high-return design choice to validate.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses three categories of evaluation data. For general benchmarks: Hellaswag (Zellers et al., 2019), a commonsense natural language inference test set, and Arc Challenge (Clark et al., 2018), a set of science exam questions — both originally English-only and used here in their translated multilingual versions from Lai et al. (2023). For machine translation: FLORES-200-TEST (Team et al., 2022), WMT-23 (Kocmi et al., 2023), and WMT-24 (Kocmi et al., 2024) — three standard translation evaluation benchmarks covering diverse language pairs. The paper does not specify the exact number of examples per benchmark or language, but FLORES-200-TEST typically contains ~1,012 sentences per language pair, WMT test sets vary by year and language pair, and the translated Hellaswag/Arc Challenge sets contain the full original test sets (~10,000 and ~1,172 examples respectively) translated into each target language.

• Base model(s). The primary model is EuroLLM-1.7B (1.133B non-embedding parameters, 1.657B total), a dense Transformer trained from scratch on 4T tokens as described in Section 4. For the instruction-tuned variant, the paper uses EuroLLM-1.7B-Instruct, fine-tuned on the EuroBlocks dataset. Baselines include Gemma-2B (Team et al., 2024a) with 1.981B non-embedding parameters, Gemma-7B (Team et al., 2024a), and TinyLlama (Zhang et al., 2024) with approximately 1.1B parameters — all representative open-weight models in a comparable size range. For fair comparison on machine translation, the paper fine-tunes Gemma-2B and Gemma-7B on the same EuroBlocks dataset, creating Gemma-2B-Instruct and Gemma-7B-Instruct.

• Metrics. For general benchmarks (Hellaswag, Arc Challenge), the metric is accuracy — the fraction of questions answered correctly, using 10-shot prompts for Hellaswag and 25-shot prompts for Arc Challenge (matching the standard evaluation protocol for these benchmarks). For machine translation, the paper uses COMET-22 (Rei et al., 2022a), a neural reference-based metric that predicts translation quality on a continuous scale (typically 0–100); higher scores indicate better translations. The paper reports average COMET-22 scores across language pairs within each test set. For the learning rate scheduler comparison (Table 2), the evaluation is conducted on the base (non-instruction-tuned) model using standard prompting rather than fine-tuning.

• Baselines. Four distinct baseline comparisons are used across different experiments: (1) Gemma-2B — a similarly-sized open model from Google with an English-optimized design, compared on general benchmarks in both base and instruction-tuned forms; (2) TinyLlama — a 1.1B-parameter open model trained primarily on English data, compared on general benchmarks; (3) Gemma-7B — a substantially larger model (~7B parameters), compared only on machine translation after EuroBlocks fine-tuning, to assess whether the smaller EuroLLM can compete with larger general-purpose models on multilingual tasks; (4) for the scheduler comparison, the baseline is the cosine-scheduled EuroLLM-1.7B trained identically except for the learning rate scheduler shape.

• Generation budget / compute accounting. The paper does not use a generation budget framework (as in the reference example's search-vs-best-of-N analysis) because the evaluation protocol is fundamentally different: general benchmarks use fixed few-shot prompting (10-shot for Hellaswag, 25-shot for Arc Challenge) with a single greedy generation per example, and machine translation uses 0-shot prompting with greedy decoding — there is no varying compute budget or search strategy being compared. The compute accounting for fairness is instead implicit in the model size comparisons: EuroLLM-1.7B is compared against models of similar size (Gemma-2B, TinyLlama) and one substantially larger model (Gemma-7B), with the instruction-tuned comparisons controlling for fine-tuning data by training all models on the same EuroBlocks dataset.

• Cross-validation / statistical protocol. For the scheduler comparison (Table 2), the paper reports the average COMET-22 score across language pairs within each test set, but does not specify confidence intervals, standard deviations, or statistical significance tests. For the general benchmarks (Figure 6), results are reported per language without error bars or significance testing. For the machine translation comparisons (Figures 7, 8), results are aggregated across language pairs as averages, again without reported variance or statistical testing. The paper does not employ cross-validation or hold-out strategy selection — the scheduler comparison is a direct A/B test between two training runs.

---

Main Quantitative Results

General Benchmarks: Multilingual Commonsense and Science Reasoning

The general benchmark evaluation (Figure 6) compares EuroLLM-1.7B and EuroLLM-1.7B-Instruct against Gemma-2B and TinyLlama on the translated Hellaswag and Arc Challenge benchmarks across multiple languages. The headline findings are:

On Hellaswag (Figure 6, top): EuroLLM-1.7B matches or outperforms Gemma-2B and TinyLlama on all languages except English. The paper states in Section 6.1:

"On Hellaswag, EuroLLM-1.7B matches or outperforms Gemma-2b and TinyLlama on all languages besides English, which showcases its increased multilinguality."

This is the central claim supporting the thesis that purpose-built multilingual design yields better non-English performance. While the exact per-language accuracies are shown only in the bar chart (Figure 6) without tabular values in the text, the visual pattern is clear: for languages like German, Spanish, French, Italian, Portuguese, and others, the EuroLLM bars are at or above the Gemma-2B and TinyLlama bars. For English specifically, EuroLLM-1.7B lags behind — consistent with the deliberate tradeoff of capping English data at 50% to allocate more capacity to other languages.

EuroLLM-1.7B-Instruct shows slightly better results than the base model on both benchmarks (Figure 6, comparing the two EuroLLM bar colors). This suggests that instruction fine-tuning on EuroBlocks does not degrade general knowledge and reasoning — in fact, it provides a small improvement, possibly because the fine-tuning data includes diverse task formats that help the model leverage its pretrained knowledge more effectively.

On Arc Challenge (Figure 6, bottom): EuroLLM-1.7B outperforms TinyLlama on all languages but is worse than Gemma-2B. The paper attributes this gap to parameter count:

"This can be caused by the lower number of parameters (EuroLLM-1.7B has 1.133B non-embedding parameters while Gemma-2B has 1.981B)."

The non-embedding parameter gap is substantial: 1.133B vs. 1.981B, meaning Gemma-2B has approximately 75% more "thinking" parameters than EuroLLM-1.7B, despite both being marketed as ~2B-parameter models. This is a direct consequence of EuroLLM's large vocabulary (128,000 pieces): 262M parameters are consumed by the embedding table and another 262M by the LM head, leaving only 1.133B for the Transformer layers. Gemma-2B, with its 256,000-piece vocabulary, has an even larger embedding cost but compensates with more total parameters. The paper does not provide Gemma-2B's non-embedding parameter count, but the comparison highlights a fundamental tension in multilingual model design: large vocabularies improve tokenization efficiency across languages but consume parameters that could otherwise contribute to reasoning capacity. The Arc Challenge gap suggests that at the 1.7B total parameter scale, this tradeoff tilts against reasoning-heavy benchmarks, though the Hellaswag results show that the multilingual tokenizer advantage offsets it for more language-sensitive tasks.

Language-specific patterns in Figure 6: The bar charts reveal that performance varies substantially across languages for all models. High-resource European languages (German, French, Spanish, Italian) generally show higher accuracy than lower-resource languages, but the gap between EuroLLM and baselines is largest for the non-English languages — exactly what the multilingual-first design targets. The paper does not provide a table of per-language values or discuss which specific languages show the largest improvements, but the visual pattern supports the claim that multilingual design trades English performance for non-English gains.

Machine Translation: Zero-Shot Translation Quality

The machine translation evaluation (Figures 7 and 8) compares EuroLLM-1.7B-Instruct against Gemma-2B-Instruct and Gemma-7B-Instruct — all fine-tuned on the same EuroBlocks dataset — on three standard translation benchmarks using COMET-22 scores.

On FLORES-200 (Figure 7): Results are shown separately for English-to-XX (EN→XX, translating from English into other languages) and XX-to-English (XX→EN, translating into English from other languages). The paper states:

"We can see that EuroLLM-1.7B-Instruct clearly outperforms Gemma-2b-Instruct on all languages pairs and datasets, and is competitive with Gemma-7b-Instruct despite the much lower number of parameters."

The exact COMET-22 scores are not provided as numerical values per language pair in the text; they are displayed as bar charts in Figures 7 and 8. However, Table 2 provides aggregate COMET-22 scores that contextualize the translation performance:

• EuroLLM-1.7B (base, trapezoid) achieves 86.75 on FLORES-200, 83.13 on WMT-23, and 79.35 on WMT-24.
• The instruction-tuned models in Figures 7 and 8 presumably achieve different (likely higher) scores due to the translation-specific fine-tuning data in EuroBlocks, but these values are not directly reported in table form.

The claim of "competitive with Gemma-7B-Instruct" is the most striking finding: a 1.7B-parameter model rivaling a ~7B-parameter model on translation. If the comparison is fair (both fine-tuned on identical data, both evaluated zero-shot with greedy decoding), this suggests that the combination of multilingual tokenizer, parallel data in pretraining, and balanced language allocation enables the smaller model to develop translation capabilities that a larger English-centric model cannot match without explicit multilingual pretraining design. However, the paper does not quantify "competitive" — does EuroLLM match Gemma-7B on most language pairs? Come within a few COMET points? The Figure 7 bar charts provide the visual comparison but the text lacks precise numeric comparisons.

On WMT-23 and WMT-24 (Figure 8): The same pattern holds — EuroLLM-1.7B-Instruct outperforms Gemma-2B-Instruct and is competitive with Gemma-7B-Instruct. WMT test sets are annual competition benchmarks with carefully curated test data; the consistent advantage across both WMT-23 and WMT-24 adds robustness to the finding that the multilingual-first design pays off specifically for translation, where cross-lingual alignment from parallel pretraining data and tokenizer efficiency directly improve performance.

Context from the scheduler comparison (Table 2): The base EuroLLM-1.7B (trapezoid) achieves COMET-22 scores of 86.75 on FLORES-200, 83.13 on WMT-23, and 79.35 on WMT-24 without any instruction fine-tuning. These are strong baseline translation scores for a 1.7B model (COMET-22 scores above 80 indicate high-quality translation), and the instruction-tuned variant likely improves on these through the explicit translation examples in EuroBlocks. The fact that the base model already achieves competitive translation scores suggests that the pretraining data mixture — specifically the 20% parallel data per language — successfully bakes translation capability into the model without requiring task-specific fine-tuning.

Scheduler Comparison: Trapezoid vs. Cosine

The learning rate scheduler experiment (Table 2) compares two EuroLLM-1.7B base models trained identically except for the scheduler shape. The numerical results:

The trapezoid scheduler outperforms cosine on all five metrics. The absolute improvements are small: +0.0098 on Hellaswag (+2.1% relative), +0.0062 on Arc Challenge (+1.9% relative), +0.27 on FLORES-200 (+0.3% relative), +0.25 on WMT-23 (+0.3% relative), and +0.48 on WMT-24 (+0.6% relative). While modest in absolute terms, the consistency across diverse evaluation types (two general reasoning benchmarks and three translation test sets) makes random noise an unlikely explanation — the trapezoid scheduler genuinely appears to produce a slightly better model at the same computational cost.

The paper presents this comparison to "decide which option to use in future models" (Section 4.1), and the results justify selecting the trapezoid scheduler for the released EuroLLM-1.7B and for future scaling efforts. However, the comparison has limitations: it is a single run per scheduler (no multiple seeds or confidence intervals), the test sets are fixed and relatively small (500 questions for Arc Challenge, ~10,000 for Hellaswag), and the experiment does not isolate why the trapezoid scheduler helps — is it the constant high-learning-rate phase, the coupling with the annealing data shift, or some other factor? The paper does not run the intermediate ablation (e.g., trapezoid scheduler without data annealing) that would disentangle these effects.

---

Ablation Studies and Robustness Checks

Tokenization efficiency: fertility comparison across tokenizers (Figure 5): The paper compares the EuroLLM tokenizer's fertility (pieces per word) against Mistral (32K vocab), LLaMA-3 (128K vocab), and Gemma (256K vocab) on a subset of EuroLLM languages. EuroLLM achieves significantly lower fertility than Mistral for all shown non-English languages (due to its 4× larger vocabulary), lower fertility than LLaMA-3 for most European languages (suggesting the training data composition matters beyond vocabulary size), and similar fertility to Gemma for European languages (indicating diminishing returns beyond 128K pieces for Latin-script languages). This is not an ablation in the traditional sense (it compares across model families, not within the EuroLLM design space), but it serves as a robustness check that the tokenizer design achieves its stated goal of improving multilingual efficiency. The finding that LLaMA-3's tokenizer has the lowest English fertility despite having the same vocabulary size as EuroLLM is a non-obvious result that highlights the importance of tokenizer training data balance.

Data repetition: repeat vs. no-repeat Wikipedia (Figure 3): The scaling-law analysis comparing models trained with and without repeated Wikipedia data shows that repeating Wikipedia improves performance on Wikipedia-domain test sets without degrading performance on web-domain test sets. This justifies the decision to repeat high-quality data in the final training corpus. The experiment is conducted at small scale (100M–341M parameters on 100B tokens) and extrapolated, not validated at the full 1.7B/4T scale, but the consistency of the pattern across model sizes in Figure 3 suggests the extrapolation is reliable. An important limitation: only Wikipedia repetition is tested — the conclusion that "high-quality data can be repeated safely" is assumed to generalize to other high-quality sources (Books, Arxiv, Apollo) without direct empirical validation.

Parallel data proportion: 0% vs. 25% vs. 37.5% (Figure 2): The scaling-law analysis of parallel data proportion shows that adding parallel data (25%) significantly improves performance on parallel-domain test sets without degrading web or Wikipedia performance. Increasing from 25% to 37.5% yields diminishing returns — the marginal benefit is smaller than the initial gain from 0% to 25%. This directly informs the choice of 20% parallel data per language. A robustness limitation: the scaling law is fitted on only three parallel-data proportions (0%, 25%, 37.5%) and three model sizes (100M, 203M, 341M), meaning the extrapolation to 20% parallel data at 1.7B parameters relies on the fitted functional form being accurate outside the observed range — the paper does not validate with, for example, a model trained at 20% parallel data at an intermediate scale.

Instruction fine-tuning effect on general benchmarks (Figure 6): Comparing the base EuroLLM-1.7B against EuroLLM-1.7B-Instruct on Hellaswag and Arc Challenge shows that instruction fine-tuning slightly improves performance on both benchmarks. This serves as a robustness check that the EuroBlocks fine-tuning does not cause catastrophic forgetting of general knowledge — a common failure mode when fine-tuning LLMs on narrow task datasets. The improvement is small but consistent, suggesting the diverse, multilingual EuroBlocks dataset provides complementary training signal rather than conflicting with pretrained knowledge.

Gemma baseline fine-tuning for fair comparison (Figures 7, 8): To ensure fair comparison on machine translation, the paper fine-tunes Gemma-2B and Gemma-7B on the same EuroBlocks dataset used for EuroLLM-1.7B-Instruct. This controls for instruction-tuning data quality and isolates the effect of pretraining design choices (tokenizer, data mixture, training recipe). Without this control, one might attribute EuroLLM's translation advantage to the EuroBlocks data rather than the multilingual pretraining. The fact that EuroLLM-1.7B-Instruct outperforms the similarly fine-tuned Gemma-2B-Instruct strengthens the claim that pretraining design matters.

Negative result: English performance tradeoff (Figure 6): On the Hellaswag benchmark, EuroLLM-1.7B underperforms Gemma-2B and TinyLlama specifically on English, even while matching or exceeding them on other languages. This is a genuine negative result that the paper does not hide: the multilingual-first design trades English performance for non-English gains. On Arc Challenge, EuroLLM-1.7B underperforms Gemma-2B across all languages, which the paper attributes to the non-embedding parameter gap (1.133B vs. 1.981B). These findings highlight the parameter-budget tension inherent in multilingual model design — the large vocabulary that enables efficient multilingual processing consumes parameters that could otherwise contribute to reasoning capacity, and the deliberate English data cap reduces English-specific knowledge.

Negative result: Arc Challenge gap despite multilingual design (Figure 6, bottom): EuroLLM-1.7B trails Gemma-2B on Arc Challenge across all languages, including non-English ones. This is informative because it suggests that for reasoning-heavy tasks (science exam questions), the non-embedding parameter count matters more than tokenizer efficiency or data mixture balance — Gemma-2B's ~75% larger non-embedding parameter budget outweighs EuroLLM's multilingual design advantages. This implies that while multilingual-first design improves language-sensitive tasks (Hellaswag, translation), it does not compensate for raw parameter count on reasoning-intensive tasks, at least at the 1.7B scale.

Missing ablation: tokenizer vocabulary size comparison. The paper states that 128,000 pieces "provides the best trade-off" but does not report experiments comparing alternative vocabulary sizes (e.g., 64K, 96K, 160K) on downstream task performance. The fertility comparison (Figure 5) provides efficiency data but not end-task performance data. A convincing ablation would train models with different vocabulary sizes (at small scale using the scaling-law methodology) and measure both fertility and benchmark accuracy.

Missing ablation: language allocation alternatives. The 50% English allocation in the main phase and 32.5% in annealing are stated choices but not empirically justified against alternatives (e.g., 40% English, 60% English, equal-per-language). The scaling-law framework could have been applied to language allocation (as in Fernandes et al., 2023) but was not — the paper uses scaling laws for parallel data proportion and data repetition, but language allocation is determined by data availability and pragmatic judgment.

Missing robustness check: multiple training seeds. All reported experiments appear to be single training runs. The trapezoid-vs-cosine comparison (Table 2) reports results from two runs (one per scheduler) without replication. The differences are small enough (~0.25–0.48 COMET points on translation) that run-to-run variance could plausibly account for some of the difference. Training even a small number of seeds (3–5) and reporting means with standard deviations would substantially strengthen confidence in the scheduler comparison.

---

Critical Assessment

Claim from Section 1: "EuroLLM-1.7B matches or outperforms Gemma-2B and TinyLlama on all languages besides English on Hellaswag."

The evidence for this claim comes from Figure 6 (top), which shows per-language Hellaswag accuracies as a bar chart. The claim is supported visually but lacks numerical precision in the text — the paper does not report the actual accuracy values per language, the size of the gaps, or whether the gaps are statistically significant. Moreover, the "all languages besides English" qualification is important: the multilingual advantage comes at the cost of English performance, and the paper is transparent about this tradeoff. The Hellaswag evaluation uses 10-shot prompting, which means the model's few-shot learning capability is being tested alongside its underlying language understanding — for non-English languages, the multilingual tokenizer's efficiency advantage (processing more text in the same context window) may contribute to the few-shot performance gain, not just the pretrained knowledge. The claim is adequately supported for a project report but would be stronger with per-language numerical values, confidence intervals, and analysis of whether the gains are concentrated in high-resource languages or distributed across all target languages.

Claim from Section 1: "Competitive with Gemma-7B on machine translation despite much lower number of parameters."

The evidence comes from Figures 7 and 8, comparing EuroLLM-1.7B-Instruct against Gemma-7B-Instruct (both fine-tuned on EuroBlocks). The bar charts show the two models as roughly comparable, with EuroLLM sometimes slightly higher, sometimes slightly lower, depending on the language pair and test set. However, the paper never quantifies "competitive" — does EuroLLM achieve COMET scores within 1 point of Gemma-7B? Within 2 points? For which language pairs is it competitive and for which does it lag? The FLORES-200 results (Figure 7) split by direction (EN→XX and XX→EN) and show multiple language pairs, but without numerical labels on the bars or a summary table, the reader cannot assess the magnitude or consistency of the competitiveness claim. The WMT results (Figure 8) show aggregated scores per test set, not per language pair. This claim is suggestive but imprecise — the visual evidence supports that EuroLLM-1.7B is in the same ballpark as Gemma-7B for translation, but the lack of quantification weakens the claim. Additionally, translation is evaluated with 0-shot greedy decoding — a single deterministic output per input. This does not capture whether Gemma-7B would pull ahead with sampling or beam search, or whether the comparison would hold for more creative/adversarial translation examples beyond standard benchmarks.

Claim from Section 1: "The trapezoid scheduler outperforms cosine scheduling on both general benchmarks and machine translation."

This claim is supported by Table 2, which shows the trapezoid scheduler achieving higher scores on all five metrics. The evidence is straightforward and the comparison is well-controlled (identical architecture, data, and training budget). However, the experiment's strength is also its limitation: it is a single run per condition. The improvements are small (+0.0062–0.0098 on accuracy benchmarks, +0.25–0.48 on COMET scores), and without multiple seeds, we cannot distinguish a genuine scheduler effect from random seed variation. The paper also does not ablate why the trapezoid scheduler helps — is it the constant high learning rate phase, the coupling with the annealing data shift, or something else? The hypothesis that the trapezoid scheduler interacts favorably with the data curriculum is plausible but untested. If the annealing phase were removed, would the trapezoid advantage disappear? If the cosine scheduler were modified to delay its decay so the learning rate is higher when annealing begins, would it match trapezoid performance? These follow-up experiments would transform the finding from an empirical observation into a mechanistic understanding, but they are absent.

Genuine weakness: very limited evaluation breadth. The paper evaluates on only two general benchmarks (Hellaswag, Arc Challenge) and machine translation (three test sets). There is no evaluation on: open-ended generation quality (fluency, coherence in non-English languages), factual knowledge (e.g., multilingual MMLU or similar), summarization, question answering beyond science exams, code generation in non-English contexts, or safety/toxicity across languages. The omission of factual knowledge benchmarks is particularly notable — the paper includes Wikipedia in training data for all languages, so evaluating whether the model actually acquired factual knowledge in non-English languages would be a direct test of the multilingual pretraining's effectiveness. The translation evaluation, while thorough (three test sets across many language pairs), only measures one specific capability that the model was explicitly trained for (with 20% parallel data). The paper's central claim is that EuroLLM is a strong multilingual model across tasks, but the experimental evidence is narrow — it demonstrates improved performance on commonsense reasoning, science QA, and translation, but leaves open whether these gains generalize to the broader range of capabilities expected of an LLM.

Missing baseline: LLaMA-3 or Mistral at comparable size. The paper compares against Gemma-2B, Gemma-7B, and TinyLlama, but not against LLaMA-3 or Mistral models of similar scale. LLaMA-3's tokenizer (128K vocabulary) is the closest direct comparison to EuroLLM's tokenizer, and Mistral-7B is a widely-used open model that would provide a stronger baseline for the "competitive with larger models" claim. The omission may reflect practical constraints (LLaMA-3 did not release a ~2B model, and fine-tuning Mistral-7B on EuroBlocks for fair comparison requires computational resources), but it limits the strength of the claim that EuroLLM's design is specifically what drives the multilingual advantage rather than just being a well-trained model on multilingual data.

Missing evaluation: instruction-following quality across languages. EuroLLM-1.7B-Instruct is presented as an instruction-following conversational model, but the only evaluation of the instruction-tuned variant is on the same benchmarks as the base model (Hellaswag, Arc Challenge) plus translation quality. There is no evaluation of whether the model actually follows instructions reliably across languages — does it refuse inappropriate requests appropriately? Does it maintain the requested output format? Does it handle multi-turn conversations in non-English languages without losing coherence? The EuroBlocks dataset construction (Section 5) and chat formatting (Table 3) suggest these capabilities were targeted, but they are not evaluated. This is a significant gap for a model described as an "instruction-following conversational model."

The scaling-law methodology is applied post-hoc to justify decisions, not pre-registered as a design tool. The paper uses scaling laws to determine optimal parallel data proportion and data repetition strategies, but the final model (1.7B on 4T tokens) is substantially larger and trained on orders of magnitude more data than the proxy models (100M–341M on 100B tokens). The extrapolation assumes the fitted power-law relationship holds across this scale gap, which is a strong assumption. The paper does not validate the scaling-law predictions at an intermediate scale (e.g., training a 500M-parameter model on 500B tokens and verifying that its performance matches the scaling-law prediction). The fact that the final EuroLLM-1.7B performs well is consistent with the scaling-law predictions being informative, but does not confirm their accuracy — a poorly-predicted mixture might still produce a decent model due to the sheer scale of training. This is not a fatal weakness (training multiple models at intermediate scale to validate scaling laws would consume budget that could go toward the final model), but it means the scaling-law guidance should be understood as heuristic rather than rigorously validated.

The French high-resource advantage and low-resource evidence gap. The paper targets 35 languages but evaluates on a subset. The machine translation evaluation (Figures 7, 8) shows results for multiple language pairs, but the general benchmark evaluation (Figure 6) shows a limited set of languages. For the lowest-resource EU languages (Maltese, Irish, Estonian, Latvian, etc.), it is unclear whether the multilingual design actually delivers meaningful improvements — these languages received very small slices of the training data budget (as Figure 4 shows), and they may not be represented in the translated Hellaswag/Arc Challenge evaluation sets. The claim that EuroLLM is capable "in all official European Union languages" is thus broader than what the evaluation evidence supports — the model has been trained on all these languages, but its performance on most of them is not directly measured. The translation evaluation partially addresses this (FLORES-200 covers many languages), but for general reasoning and knowledge, the evidence is thin for the lowest-resource languages.

6. Limitations and Trade-offs

6.1 Parameter Budget Tension: Large Multilingual Vocabulary Consumes Non-Embedding Capacity

The constraint. EuroLLM's 128,000-piece vocabulary — deliberately chosen to minimize fertility across 35 languages — consumes 262M embedding parameters and 262M LM head parameters, together representing ~31.6% of the model's total 1.657B parameters. This leaves only 1.133B non-embedding parameters for the Transformer layers that perform the actual reasoning. As the paper acknowledges when discussing Arc Challenge results:

"This can be caused by the lower number of parameters (EuroLLM-1.7B has 1.133B non-embedding parameters while Gemma-2B has 1.981B)."

The consequence. The vocabulary-size-to-reasoning-capacity tradeoff creates a direct performance ceiling on tasks that depend primarily on non-embedding parameter count. On Arc Challenge (Figure 6, bottom), EuroLLM-1.7B trails Gemma-2B across all languages — including non-English ones where the multilingual tokenizer should provide an advantage. This suggests that for reasoning-intensive benchmarks, the ~75% larger non-embedding parameter budget of Gemma-2B outweighs any tokenizer efficiency gains. The tradeoff grows more severe as the target language count increases: supporting 35 languages with low fertility requires a large vocabulary, which consumes parameters that would otherwise contribute to model depth, width, or both. At the 1.7B total parameter scale, this tension is particularly acute — the embedding parameters are a substantial fraction of the total budget. At larger scales (7B, 13B, 70B), embedding parameters become a smaller fraction, so the tradeoff naturally relaxes, but the paper does not characterize how the optimal vocabulary size scales with total parameter count in a multilingual setting.

Evidence in the paper. The Arc Challenge results (Figure 6, bottom) are the primary evidence: EuroLLM-1.7B underperforms Gemma-2B across all languages despite matching or exceeding it on Hellaswag (Figure 6, top). The non-embedding parameter comparison (1.133B vs. 1.981B) is available from Table 1 and the Gemma-2B specification. The fertility comparison (Figure 5) demonstrates that the 128K vocabulary does achieve its multilingual efficiency goal, but the Arc Challenge gap shows the downstream cost.

Mitigation status. Not addressed. The paper does not explore vocabulary sizes other than 128,000, does not ablate the effect of vocabulary size on downstream task performance (only on fertility), and does not discuss whether the tradeoff would change at larger model sizes where embedding parameters are a smaller fraction. The statement that 128,000 "provides the best trade-off" is based on unreported experimentation. A practitioner deciding on vocabulary size for a multilingual model at a specific parameter budget would need to run their own tradeoff analysis — this paper does not provide transferable guidance.

---

6.2 Narrow Evaluation: Only Three Task Families Measured, No Open-Ended or Factual Assessment

The constraint. The evaluation covers exactly three task categories: commonsense reasoning (Hellaswag), science exam questions (Arc Challenge), and machine translation (FLORES-200, WMT-23, WMT-24). There is no evaluation of:

• Open-ended generation quality in non-English languages (fluency, coherence, stylistic appropriateness)
• Factual knowledge retrieval (e.g., multilingual MMLU, cross-lingual question answering)
• Summarization, paraphrasing, or other generation tasks
• Code generation in non-English contexts
• Safety, toxicity, or bias across languages
• Instruction-following reliability in multilingual conversational settings

This is particularly notable given the paper's framing: EuroLLM is positioned as a general-purpose multilingual LLM "capable of understanding and generating text in all official European Union languages," and EuroLLM-1.7B-Instruct is released as a conversational model. Yet none of the evaluations measure open-ended generation quality or instruction-following behavior — the model's primary intended use case.

The consequence. The paper's central claims about multilingual capability are supported only for a narrow slice of LLM abilities. A practitioner cannot determine from the reported results whether EuroLLM-1.7B-Instruct can reliably summarize a document in Estonian, answer factual questions about Polish history, or generate safe responses to harmful prompts in Bulgarian. The strong translation results (Figures 7, 8) are encouraging but expected given the 20% parallel data — translation is the task most directly supported by the training data design. The general benchmark results (Figure 6) demonstrate that commonsense reasoning transfers across languages, but do not indicate whether the model acquired factual knowledge in non-English languages (the high-quality data is predominantly English-only, as Section 2.1 notes). This creates a substantial evidence gap: the model may produce fluent, plausible-sounding text in low-resource languages while being factually unreliable or incoherent over multiple turns.

Evidence in the paper. The evaluation sections (§6.1, §6.2) contain only the benchmarks listed above. The paper does not claim to have evaluated broader capabilities, but it also does not discuss the evaluation gap as a limitation or outline what additional evaluations are planned. The omission of safety and bias evaluation is standard for a technical project report but notable given the model's public release and intended use across EU institutions.

Mitigation status. Not addressed. The paper does not acknowledge the evaluation narrowness as a limitation, does not report results on standard multilingual evaluation suites (e.g., MMLU in translation, XQUAD, XNLI, Flores for generation quality), and does not outline future evaluation plans beyond "scaling up the number of model parameters and improving further the quality of our data" (§7). For a model released as an initial project milestone, this narrowness is understandable but represents a significant gap between the claimed capabilities ("understanding and generating text in all official European Union languages") and the demonstrated evidence.

---

6.3 The English Performance Tradeoff Is Real and Its Magnitude Is Not Quantified

The constraint. EuroLLM's data mixture caps English at 50% of training tokens (main phase) and 32.5% (annealing phase), deliberately trading English training volume for non-English representation. The consequence, visible in Figure 6 (top), is that EuroLLM-1.7B underperforms both Gemma-2B and TinyLlama on English Hellaswag. The paper acknowledges this explicitly: "EuroLLM-1.7B matches or outperforms Gemma-2b and TinyLlama on all languages besides English."

The consequence. For any deployment where English performance matters alongside multilingual capability (which describes the vast majority of real-world use cases — multilingual customer support, EU document processing, cross-border business communication), the model presents a genuine tradeoff. The paper does not quantify how much English performance is sacrificed for the multilingual gain. The Hellaswag bar chart (Figure 6, top) shows the English gap visually, but without numerical values, the reader cannot assess whether the English drop is marginal (e.g., 1-2 percentage points) or substantial (e.g., 5-10 points). On Arc Challenge (Figure 6, bottom), the English gap is compounded by the non-embedding parameter deficit, making it unclear whether the English performance loss is primarily from data allocation or parameter constraints. A practitioner choosing between EuroLLM-1.7B and Gemma-2B for a multilingual deployment would need to weigh the non-English gains against the English loss, but the paper provides no quantitative basis for this tradeoff — no per-language accuracy table comparing the two models across all evaluated languages, no ablation separating the effect of English data cap from the effect of vocabulary size on English performance.

Evidence in the paper. Figure 6 provides the English comparison visually but without numerical values. The data mixture allocation (Section 2.2.3, Figure 4) shows English receiving 50% and 32.5% of tokens in the two training phases, compared to what would presumably be a much higher percentage in an English-centric model like Gemma-2B (whose exact data mixture is not reported by the paper but is known from the Gemma technical report to be heavily English-weighted). The Arc Challenge results further compound the English deficit with the non-embedding parameter gap.

Mitigation status. The paper is transparent that the English tradeoff exists — the "besides English" qualification in Section 6.1 is an honest acknowledgment — but it does not quantify the magnitude, provide per-language comparison tables, or suggest strategies for mitigating the English loss (e.g., whether instruction fine-tuning partially recovers English performance, whether the annealing phase's English reduction is necessary or could be softened, whether a slightly larger English allocation would still preserve the multilingual gains). The paper frames the English tradeoff as an acceptable cost of multilinguality rather than a design parameter to be optimized, leaving future practitioners to determine the right balance through their own experimentation.

---

6.4 Low-Resource Language Performance Is Largely Unevaluated

The constraint. The paper targets 35 languages including genuinely low-resource ones: Maltese (~500K speakers), Irish (~170K daily speakers), and several others with limited digital presence (Estonian, Latvian, Lithuanian). The data mixture allocates tokens proportionally to available data (Section 2.2.3, Figure 4), meaning these languages receive very small slices of the training budget. The evaluation, however, does not report per-language results for the lowest-resource targets, making it impossible to assess whether the model actually acquired meaningful capabilities in these languages.

The consequence. The paper's claim that EuroLLM is "capable of understanding and generating text in all official European Union languages" is broader than what the evidence supports. For high-resource European languages (German, French, Spanish, Italian), the Hellaswag and translation results provide positive evidence. For mid-resource languages (Polish, Dutch, Greek, Czech), the translated benchmarks may provide some signal, but results are not broken out by language. For the lowest-resource languages (Maltese, Irish), the translated Hellaswag and Arc Challenge benchmarks may not even exist — Lai et al. (2023) covers a subset of languages, and the paper does not specify which of the 35 target languages are included in the translated benchmark set. The FLORES-200 and WMT test sets do cover many of these languages, so translation quality can be partially assessed, but general reasoning, knowledge, and generation quality in Maltese or Irish are unevaluated. A practitioner building a system for Maltese-language document processing or Irish-language education tools would find no direct evidence that EuroLLM-1.7B meets their needs.

Evidence in the paper. Figure 6 shows Hellaswag and Arc Challenge results for a limited set of languages (the visual bar chart shows perhaps 8-12 languages, far fewer than the 35 supported). The paper does not specify which languages are included in the translated benchmarks or provide a table of per-language results. Figure 4 shows the token allocation per language, confirming that low-resource languages receive very small fractions of the training budget. The FLORES-200 translation evaluation (Figure 7) covers many languages but shows results for the instruction-tuned model only as part of aggregated bar charts — per-language COMET scores for all supported languages are not tabulated.

Mitigation status. Not addressed. The paper does not discuss the low-resource evaluation gap as a limitation, does not report per-language results broken out by resource level, and does not analyze whether the data-availability-proportional allocation provides sufficient training signal for the lowest-resource languages. The conclusion mentions "improving further the quality of our data" but does not specifically target low-resource language improvement. This is a significant omission for a project whose stated goal is coverage of all EU languages — the model may technically support these languages (the tokenizer can encode them, and some training data exists), but whether it actually performs usefully in them remains unknown from the reported evidence.

---

6.5 Scaling-Law Extrapolation Is Unvalidated at the Target Scale

The constraint. The joint scaling-law methodology (Section 2.2.1, 2.2.2) fits performance predictions using small proxy models (100M, 203M, 341M non-embedding parameters) trained on 100B-token corpora, then extrapolates to guide decisions for a 1.7B-parameter model trained on 4T tokens — a 5–17× scale gap in parameters and a 40× gap in training tokens. The extrapolation assumes that the fitted power-law relationship $L(N, p) = f(p) \beta N^{-\alpha} + L_\infty$ holds across this range, and that the ratio function $f(p) = p + c_1 p^{c_2} (1-p)^{c_3}$ captures the interaction between data proportion and model scale accurately at points far from the fitted region.

The consequence. If the scaling-law extrapolation is inaccurate, the key data mixture decisions — 20% parallel data per language, repeat high-quality data — may be suboptimal for the target model scale. The paper does not validate the predictions at any intermediate scale (e.g., training a 500M model on 500B tokens and verifying that the predicted and observed losses match). This means the reported decisions, while guided by a principled methodology, are effectively heuristic rather than rigorously proven for the 1.7B/4T scale. The consequences are difficult to assess in the absence of a counterfactual: would a model trained with 30% parallel data (which the scaling law predicts would yield diminishing returns) actually perform meaningfully worse? Would a model trained without Wikipedia repetition show a measurable degradation at 1.7B scale? The paper's positive results are consistent with the scaling-law predictions being informative, but they do not confirm them — a model trained on any reasonable data mixture at 4T tokens would likely achieve non-trivial performance, and the specific mixture choices could be suboptimal without the paper being able to detect it from the final model's evaluation alone.

Evidence in the paper. The scaling-law experiments are described in Sections 2.2.1 and 2.2.2, with results in Figures 2 and 3. The proxy model sizes (100M–341M non-embedding) and training budget (100B tokens) are stated explicitly. The final model (1.7B total, ~1.133B non-embedding, 4T tokens) represents a substantial extrapolation. No intermediate-scale validation is reported. The paper does not discuss the extrapolation uncertainty or the risk that the functional form $f(p)$ might change at larger scales (e.g., if the model develops new capabilities that alter the optimal data mixture).

Mitigation status. The paper does not acknowledge the extrapolation gap as a limitation. The scaling-law methodology is presented as a decision-making tool, and the decisions it produced are implemented, but the reader is given no indication of how much trust to place in the extrapolation. In fairness, validating the scaling-law predictions would require training additional models at intermediate scales, consuming compute that the project budget may not have supported. The paper could have addressed this by: (1) reporting prediction intervals or uncertainty estimates for the fitted scaling laws, (2) training a single intermediate-scale validation model (e.g., 500M parameters) and comparing its observed performance against the scaling-law prediction, or (3) explicitly discussing the extrapolation risk and framing the decisions as scaling-law-informed rather than scaling-law-validated. None of these are done. This is the most consequential methodological gap in the paper, because the scaling-law experiments represent the paper's primary intellectual contribution beyond the model release itself — if the extrapolation is unreliable, the paper's main design methodology is unvalidated.

---

6.6 No Accounting for Inference Latency or Deployment Practicality in the Multilingual Tokenizer Design

The constraint. The 128,000-piece vocabulary achieves lower fertility (fewer tokens per word) for non-English languages than smaller vocabularies like Mistral's 32K (Figure 5), which improves the model's effective context length for non-English text. However, the paper does not discuss the inference-time implications of this design choice: with 128K vocabulary, the embedding lookup and final softmax projection each operate over 128,000 token candidates, which is computationally more expensive per generated token than a 32K vocabulary model. The total parameters (1.657B) are comparable to other ~2B models, but the time per token at inference depends on operations in both the Transformer layers and the output projection — the large vocabulary increases the output projection cost.

The consequence. A practitioner deploying EuroLLM-1.7B for real-time multilingual applications (e.g., interactive translation, chat) may find that while the model processes non-English text in fewer tokens, each token takes more time to generate compared to a model with a smaller vocabulary. The net effect on inference latency depends on the balance between token count reduction and per-token cost increase, which varies by language: for languages where EuroLLM's fertility advantage is large (e.g., Finnish, Hungarian, with highly agglutinative morphology that fragments heavily in English-centric tokenizers), the token-count reduction likely dominates and inference is faster. For languages close to English morphologically (e.g., German, Dutch), the token-count reduction may be smaller, and the per-token overhead of the large vocabulary may result in similar or slightly higher total latency. The paper provides fertility comparisons (Figure 5) but no inference latency or throughput measurements, leaving deployers to benchmark this themselves.

Additionally, the large vocabulary increases memory requirements: the embedding matrix (262M parameters × 2 bytes in bf16 ≈ 524MB) and the LM head (another 524MB) must be resident in GPU memory during inference, regardless of the batch size. For deployment on memory-constrained edge devices (a motivation the paper does not discuss but which is relevant for on-device multilingual applications), this ~1GB overhead for embedding parameters alone could be prohibitive.

Evidence in the paper. The vocabulary size (128K) and embedding parameter counts (0.262B each for embeddings and LM head) are specified in Table 1. The fertility comparison (Figure 5) demonstrates the tokenization efficiency benefit. The paper does not report inference latency, throughput, or memory benchmarks, and does not discuss the deployment implications of the large vocabulary.

Mitigation status. Not addressed. The paper does not discuss inference efficiency as a design consideration, does not compare generation speed against baselines (even a simple tokens-per-second measurement on a standard GPU would be informative), and does not mention deployment scenarios or constraints. This is understandable given the paper's focus on pretraining methodology and model quality, but for a project that releases models intended for community use, omission of practical deployment characteristics is a meaningful gap. The abstract positions the models as "open-weight" releases for broad accessibility, but the largest deployment cost — inference latency and throughput — is not characterized. Given that the 128K vocabulary was explicitly chosen as a "trade-off" between fertility and embedding parameter count, quantifying the inference consequences of that trade-off would be natural and valuable.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper represents an incremental but practically significant refinement in how the field approaches multilingual LLM development. It does not introduce a new architecture, a novel training objective, or a theoretical breakthrough. Rather, it demonstrates that treating multilinguality as a first-class design constraint across every stage of the LLM pipeline — tokenizer construction, data mixture optimization, training recipe selection, and instruction tuning — yields non-English performance that rivals or exceeds much larger English-centric models, even at modest parameter counts (1.7B).

The magnitude of this contribution is best understood as a template and a diagnostic, not a paradigm shift. Prior open-weight LLM releases (LLaMA, Mistral, Gemma) treated multilinguality as an emergent property: design for English, add some non-English data, and hope transfer learning fills the gaps. EuroLLM inverts this by asking, at each design decision, "what does this choice mean for each of our 35 target languages?" and making the tradeoffs visible rather than implicit. The fertility comparison (Figure 5) is the most concrete diagnostic: it quantifies tokenizer bias across languages and makes it an optimization target. The data mixture decisions — capping English at 50% (main phase) and 32.5% (annealing), allocating 20% parallel data per non-English language based on scaling-law extrapolation — are explicit English-vs-multilingual tradeoffs that prior work either ignored or made without empirical justification. The finding that a trapezoid learning rate scheduler outperforms cosine scheduling for multilingual training with a data curriculum (Table 2) is a small but actionable discovery that challenges the default recipe inherited from English-centric pretraining.

The paper also reconciles a tension in the multilingual modeling literature between two competing intuitions: (1) that larger vocabularies are necessary for efficient multilingual processing, and (2) that vocabulary parameters consume capacity that could otherwise contribute to reasoning. The Arc Challenge results (Figure 6, bottom) show that at 1.7B total parameters, the tradeoff is real and costly — EuroLLM's 1.133B non-embedding parameters trail Gemma-2B's 1.981B on reasoning-heavy tasks, even in non-English languages. Yet on language-sensitive tasks (Hellaswag, translation), the multilingual tokenizer and data mixture advantages outweigh the parameter deficit — EuroLLM-1.7B matches or exceeds Gemma-2B on Hellaswag across non-English languages (Figure 6, top) and competes with Gemma-7B on translation (Figures 7, 8). This resolution is practically important: it tells practitioners that multilingual-first design pays off for tasks where language processing is the bottleneck, but for reasoning-intensive tasks at small parameter scales, the non-embedding parameter count remains the binding constraint. The implication is that the optimal vocabulary size for a multilingual model should grow with total parameter count — at 1.7B, 128K consumes too large a fraction of the budget; at 7B or 13B, the same vocabulary would be a smaller fraction and the non-embedding capacity deficit would naturally shrink.

Several research directions become more attractive in light of this work. Vocabulary-size scaling laws for multilingual models — analogous to the data-mixture scaling laws the paper already employs — become an obvious target: at what total parameter count does a 128K vocabulary stop being a meaningful constraint on reasoning capacity? Data mixture optimization as a function of total compute budget (not just model size) is another: the paper's scaling-law methodology optimizes for a fixed model size, but the real design problem is to allocate a total FLOPs budget across model size, data volume, and data mixture jointly. Most importantly, evaluating multilingual models on reasoning benchmarks beyond translation and commonsense — factual knowledge, multi-step reasoning, code generation — becomes essential to characterize where multilingual-first design helps and where it simply cannot overcome parameter constraints.

One research direction becomes less attractive: the approach of training English-first models with minimal non-English data and expecting multilingual behavior to emerge through transfer. The paper's results, while not definitive, suggest that this approach systematically underperforms purpose-built multilingual design on non-English tasks, at least at the 1.7B–2B scale. The finding that EuroLLM-1.7B lags on English Hellaswag but leads on non-English Hellaswag (Figure 6, top) implies that multilingual capability and English capability are partially in tension at fixed parameter budgets — you cannot simply add languages for free. This challenges the implicit assumption in many open-weight LLM releases that the English-optimized base will serve as an adequate multilingual foundation with minimal modification.

Follow-Up Research This Work Enables

Vocabulary-size scaling laws for multilingual models: at what parameter count does the embedding-capacity tradeoff become negligible? The paper's central tension — a 128K-piece vocabulary consuming ~31.6% of total parameters, leaving only 1.133B non-embedding parameters for reasoning — is scoped to the 1.7B scale. A natural extension is to adapt the joint scaling-law methodology (Section 2.2.1) to predict how vocabulary size interacts with total parameter count for multilingual performance. The experiment would train proxy models (100M–500M parameters) with varying vocabulary sizes (32K, 64K, 128K, 256K) and fit scaling laws that predict downstream task performance as L(N, V) where V is vocabulary size. The key question: does the vocabulary-size penalty on reasoning tasks diminish as N grows, and at what N does a 128K vocabulary become "free" (i.e., the non-embedding parameters are sufficient for reasoning regardless of vocabulary size)? The paper's fertility comparison (Figure 5) provides the tokenizer-efficiency half of the tradeoff; this experiment would provide the reasoning-capacity half, enabling practitioners to choose vocabulary size as a function of target model scale and language count.

Data mixture optimization over the full pretraining compute budget, not just model size. The paper uses scaling laws to choose parallel data proportion and data repetition given a fixed model size (1.7B) and fixed training budget (4T tokens). But the real design problem is higher-dimensional: given a total FLOPs budget, how should one allocate compute across model size, training tokens, and data mixture? The paper's methodology could be extended by training proxy models that vary both N (non-embedding parameters) and D (training tokens) alongside the data mixture proportions, and fitting a joint scaling law L(N, D, p) where p is the parallel-data proportion or the per-language token fraction. This would enable compute-optimal multilingual design — answering, for example, whether it is better to train a 3B model on 2T tokens with 20% parallel data or a 2B model on 6T tokens with 30% parallel data. The paper already has the infrastructure (proxy model training, scaling-law fitting) and the evaluation pipeline (Hellaswag, Arc Challenge, FLORES, WMT); the extension is to add D as a dimension to the existing N and p grid.

What happens when PRM tree-search or revision models (from the reference paper's framework) are applied to multilingual generation tasks? This is a cross-pollination direction. The reference paper on compute-optimal test-time scaling shows that difficulty-conditioned search strategies (beam search against a process reward model, sequential revisions) can yield 4× efficiency gains on math reasoning. Would similar strategies improve multilingual generation — specifically, machine translation quality or cross-lingual factual consistency? A strong follow-up would train a process reward model on EuroLLM's translation outputs (using COMET-22 scores as the reward signal, since reference translations exist for FLORES/WMT), then apply beam search with the PRM guiding which translation hypotheses to expand. The question is whether verifier-guided search can close the remaining gap between EuroLLM-1.7B and Gemma-7B on translation (Figures 7, 8) — i.e., whether test-time compute can substitute for pretraining scale in the multilingual domain, analogous to the reference paper's finding for math reasoning. The failure mode would be if the COMET-based verifier over-optimizes (producing translations that score highly but are actually worse), directly testing whether the reference paper's over-optimization findings generalize beyond math.

Measuring factual knowledge in non-English languages: does the English-dominated high-quality data in pretraining create a factual knowledge gap? The paper's high-quality data (Section 2.1) is almost entirely English-only — Arxiv, Books, Apollo, GSM8K, MATH — with Wikipedia being the only multilingual high-quality source. This implies the model's factual knowledge about the world may be English-centric even though its language generation is multilingual. A direct test would evaluate EuroLLM-1.7B on multilingual factual knowledge benchmarks: translated MMLU (Hendrycks et al., 2021) across all supported languages, or cross-lingual question answering datasets like XQUAD (Artetxe et al., 2020) and MLQA (Lewis et al., 2020). The hypothesis is that EuroLLM's factual accuracy will be highest in English (where it received high-quality factual data) and lower in non-English languages, even after controlling for language understanding — the model may know the answer but struggle to express it, or conversely, it may produce fluent-sounding but factually incorrect text in languages where it never saw reliable factual training data. This experiment would quantify a limitation that the current evaluation (only Hellaswag, Arc Challenge, and translation) does not capture, and would inform whether multilingual LLM training needs multilingual high-quality data — not just multilingual web text — to achieve factual reliability across languages.

Does the trapezoid scheduler advantage persist without a data curriculum, and is it specific to multilingual training? The paper's scheduler comparison (Table 2) is a single A/B test without mechanistic ablation. A follow-up would train models with both schedulers under three conditions: (1) with the data curriculum shift (annealing), replicating the paper's experiment with multiple seeds to establish statistical significance; (2) without the data curriculum (constant data mixture throughout training), testing whether the trapezoid advantage disappears when there is no late-training distribution shift; (3) on English-only data, testing whether the trapezoid advantage is specific to multilingual training or generalizes. If the trapezoid advantage persists without the curriculum shift, it suggests the constant-high-learning-rate phase is inherently beneficial — perhaps because it prevents premature convergence to sharp minima. If it disappears, the advantage is specifically from coupling learning rate decay with data quality improvement, providing a design principle that other multilingual projects can adopt. The experiment is computationally modest (training small proxy models at 100M–200M parameters on 10B–50B tokens would likely suffice to detect the interaction) and would transform an empirical observation into a mechanistic understanding.

How do the lowest-resource EU languages actually perform? A focused evaluation on Maltese, Irish, and Estonian. The paper claims coverage of all 24 official EU languages but evaluates primarily on high- and mid-resource languages (the translated Hellaswag and Arc Challenge sets cover a subset, and FLORES covers many but only for translation). A targeted stress-test would evaluate EuroLLM-1.7B (and the instruction-tuned variant) specifically on Maltese, Irish, and Estonian across multiple task types: machine translation (already partially covered by FLORES), open-ended generation evaluated by native speakers or through reference-based metrics, and a small curated factual QA set in each language (e.g., "Who is the current President of Malta?", "What year did Ireland join the EU?", etc.). The question is whether the data-availability-proportional token allocation (Figure 4) — which gives these languages very small fractions of the training budget — is sufficient to acquire even basic competence, or whether the model produces superficially fluent but substantively nonsensical text in these languages. This is a negative-result experiment that would define the lower bound of the multilingual-first approach: it clarifies whether "supporting all EU languages" requires a minimum per-language data threshold that some languages may not meet with current data availability.

Practical Applications and Downstream Use Cases

Cross-border EU institutional document processing and translation. The European Union operates in 24 official languages with legal requirements for multilingual communication. EuroLLM-1.7B-Instruct, which the paper shows competes with Gemma-7B on machine translation (Figures 7, 8) while being approximately 4× smaller, offers a deployable translation engine that can run on more modest hardware than models requiring datacenter-scale GPUs. The COMET-22 scores on FLORES-200 (86.75 for the base model, likely higher for the instruction-tuned variant based on Figure 7 trends) indicate translation quality in the range of strong dedicated MT systems — at 1.7B parameters, this could be deployed on a single consumer GPU or even quantized for CPU inference, enabling EU institutions, law firms, and NGOs to run in-house translation without depending on cloud APIs or large-scale infrastructure. The model's ability to follow system prompts (Table 3, where a system instruction "Translate all user texts to English" is followed correctly) further enables customization for specific terminology or style guides without retraining.

Multilingual customer support for European e-commerce and services. A company operating across German, French, Italian, and Spanish markets (widely spoken EU languages representing hundreds of millions of consumers) needs support systems that work consistently across these languages. The paper's Hellaswag results (Figure 6, top) show EuroLLM-1.7B matching or exceeding Gemma-2B on all non-English languages, meaning it can handle commonsense reasoning tasks — answering product questions, resolving shipping inquiries, understanding customer complaints — with comparable or better accuracy in the local language. The multilingual tokenizer's fertility advantage (Figure 5) means that for morphologically rich languages like German (with compound nouns that fragment heavily in English-centric tokenizers), the model processes the same semantic content in fewer tokens, reducing per-query inference cost and latency. A deployment using EuroLLM-1.7B-Instruct with the chat format shown in Table 3 could maintain context across multi-turn support conversations in multiple languages without switching models per market.

Multilingual data generation for self-improvement and distillation pipelines. The paper's finding that EuroLLM-1.7B approaches Gemma-7B on translation (Figures 7, 8) despite having far fewer parameters suggests that the multilingual-first pretraining recipe produces a strong "teacher" for knowledge distillation into even smaller, specialized models. A concrete pipeline: use EuroLLM-1.7B-Instruct to generate high-quality translations, summaries, or Q&A pairs in all 35 supported languages, then distill these outputs into task-specific models (e.g., a 300M-parameter translation model for mobile deployment). The scaling-law methodology (Section 2.2.1) provides the principled framework for determining how much synthetic data to generate per language, and the 20% parallel data in pretraining ensures the generation quality is high for translation tasks specifically. For low-resource languages (Maltese, Irish), where human-annotated data is scarce, this synthetic data generation could be the primary path to building functional NLP systems — the model serves as a data factory.

On-device multilingual assistants for European smart devices. A voice assistant or smart speaker targeting the European market must handle 24+ languages while running on-device for privacy and latency reasons. EuroLLM-1.7B, at 1.657B total parameters (approximately 3.3GB in bfloat16, or ~1.7GB in 8-bit quantized form), is within the storage and memory budget of high-end mobile devices. The trapezoid scheduler and annealing recipe produce a base model that the paper shows achieves strong commonsense reasoning (Hellaswag) across non-English languages — precisely the capability needed for answering general knowledge questions, setting reminders, and controlling smart home devices in the user's native language. The 4,096-token context window (Table 1) is sufficient for multi-turn interaction, and the instruction-tuned variant's chat format (Table 3) provides the structured conversation interface that a voice assistant backend requires. The key practical advantage over using a larger cloud-based model is latency elimination (no network round-trip) and privacy (user speech never leaves the device), both enabled by the model's small size relative to its multilingual capability.

When to Prefer This Method

The paper does not articulate a formal decision framework comparing its approach against named alternatives for a specific practitioner choice. It presents EuroLLM as a model and a design methodology, and the comparisons in Section 6 are post-hoc benchmarks rather than an explicit "use our approach when X, use their approach when Y" prescription. The paper acknowledges that the English performance tradeoff exists (EuroLLM lags Gemma-2B on English Hellaswag, Figure 6), and that the non-embedding parameter deficit hurts reasoning (Arc Challenge, Figure 6), but it does not frame these as part of a deployment decision rule. The scaling-law methodology is presented as a general tool rather than as one side of a tradeoff against specific alternative methodologies. A decision matrix would therefore be an interpolation rather than a faithful representation of the paper's positioning, so this section is omitted.