Read More — Paper Graph

🎯 Pitch

A 12.8B-parameter Korean model trained on 863 GB of curated data achieves a 0-shot COPA F1 of 0.7937, decisively outperforming same-sized multilingual models and showing that dedicated monolingual pretraining can resolve the performance gap that English-centric multilingual models fail to bridge.

1. Executive Summary

This technical report introduces Polyglot-Ko, a family of four open-source Korean monolingual language models at 1.3B, 3.8B, 5.8B, and 12.8B parameters trained on a 863 GB curated Korean corpus using EleutherAI's GPT-NeoX framework. The authors evaluate these models against existing Korean and multilingual billion-scale alternatives—ko-gpt-trinity-1.2B, KoGPT-6.0B, and XGLM-7.5B—on the KOBEST benchmark across four tasks (COPA, HellaSwag, BoolQ, SentiNeg), with the 12.8B model achieving the highest F1 scores in nearly every shot setting, including a 0-shot COPA score of 0.7937 and a 50-shot SentiNeg score of 0.972. The results demonstrate that a dedicated monolingual Korean model outperforms comparably sized multilingual models on Korean-language tasks, establishing that language-specific pretraining provides a substantial advantage over multilingual approaches only when sufficient monolingual training data is available (863 GB in this work).

2. Context and Motivation

The Core Problem: Korean and Non-English Languages Are Underserved by Multilingual Models

The fundamental gap this paper addresses is straightforward but consequential: multilingual language models, despite their name, perform substantially worse on non-English languages than on English, and publicly available monolingual models for languages other than English are scarce. This creates a practical barrier for researchers and practitioners working in non-English linguistic contexts who need high-quality language models for downstream applications.

The paper frames this problem explicitly in its opening paragraph (Section 1):

"While numerous large language models for English have been publicly released... the availability of such models for non-English languages remains limited. Although several multilingual large language models have also been released, they are typically trained on English-centric corpora, resulting in lower performance on other languages."

This is not merely an observation about linguistic equity — it reflects a structural property of how multilingual models are built. The training data for models like XGLM (Lin et al., 2022), BLOOM (Scao et al., 2022), and mBERT (Devlin et al., 2019) skews heavily toward English text. As a result, the models allocate a disproportionate share of their representational capacity to English at the expense of other languages. For a language like Korean — which has distinct morphology, syntax, and writing system (Hangul) — this imbalance means that even large multilingual models may produce substandard results on Korean-specific tasks.

The paper cites the practical consequence: "researchers and developers often resort to building monolingual models in their respective languages due to the dissatisfaction with the current multilingual models' non-English language capabilities" (Abstract). The Polyglot-Ko project exists precisely because existing options were inadequate. Rather than waiting for better multilingual models to emerge, the authors took the direct path: build a purpose-built Korean model with carefully curated Korean data.

Why Korean Specifically, and Why This Matters

The paper's focus on Korean is motivated by both practical and strategic considerations. The authors are primarily Korean speakers, and they had "a readily available dataset for training purposes" (Section 1). But the significance extends beyond convenience. Korean represents an interesting test case for monolingual model development for several reasons:

Linguistic properties. Korean is an agglutinative language with a rich morphological system. Words are formed by chaining morphemes (meaning-bearing units), producing complex word forms that encode grammatical relationships English expresses through word order or separate function words. This morphological richness poses challenges for tokenization, representation learning, and generation that may not surface in English-centric training. A model trained predominantly on English data — even a multilingual one — is unlikely to capture these Korean-specific patterns well.

Availability constraints. At the time of this work, the landscape of publicly available Korean language models at billion-parameter scale was extremely sparse. The paper identifies only three comparable models: ko-gpt-trinity-1.2B (from SKT), KoGPT-6.0B (from KakaoBrain), and XGLM-7.5B (from Meta, a multilingual model). This small set of options — and the fact that the largest publicly available Korean model suitable for commercial use was relatively modest in scale — meant that Korean NLP practitioners faced a significant resource gap compared to their English-focused counterparts, who could choose from dozens of open-source models spanning multiple architectures, training paradigms, and size classes.

Commercial relevance. The authors emphasize that their 12.8B model represents "the largest publicly available Korean language model suitable for commercial applications" (Section 1). This framing — emphasizing commercial usability — signals that the paper's contribution is not purely academic. Korean companies building NLP products (chatbots, search, content moderation, translation) need models they can deploy without restrictive licensing terms. By releasing Polyglot-Ko openly, the authors aim to fill this practical void.

Where Existing Approaches Fall Short

The paper identifies limitations in two categories of prior work: multilingual models and existing Korean-specific efforts.

Multilingual models: English-centric training data. The paper names XGLM (7.5B parameters) as a representative multilingual model included in the evaluation. XGLM was trained on a corpus covering multiple languages but with a heavy English skew — a common pattern across multilingual models. The consequence is visible in the paper's results: XGLM-7.5B, despite having more parameters than several Polyglot-Ko variants, consistently underperforms on Korean tasks. On COPA 0-shot, XGLM scores 0.672 versus 0.720 for Polyglot-Ko-1.3B — a smaller model outperforming a larger one by a substantial margin purely due to language-specific training data. On SentiNeg, XGLM's performance is particularly poor (0.358 in 0-shot versus 0.912 for Polyglot-Ko-12.8B), suggesting that sentiment analysis in Korean requires linguistic knowledge that the multilingual model's English-heavy training did not adequately provide.

The paper does not claim that multilingual models are fundamentally flawed — only that their current incarnations do not serve non-English languages well enough to obviate the need for monolingual alternatives. The implicit argument is that building a monolingual model on high-quality, language-specific data is a more reliable path to strong performance than hoping that a multilingual model's limited non-English capacity will suffice.

Existing Korean models: limited scale and availability. The paper compares against two Korean-specific models:

ko-gpt-trinity-1.2B (SKT): At 1.2B parameters, this model is in the same size class as Polyglot-Ko-1.3B. The evaluation shows Polyglot-Ko-1.3B outperforming it on most tasks (e.g., COPA 0-shot: 0.720 vs. 0.670), suggesting that Polyglot-Ko's training data and process yield better results at comparable scale.
KoGPT-6.0B (KakaoBrain): This is the closest comparison to Polyglot-Ko's mid-range models (5.8B). On BoolQ, KoGPT achieves scores quite competitive with Polyglot-Ko-5.8B (e.g., 5-shot: 0.598 vs. 0.570; 10-shot: 0.550 vs. 0.519, with KoGPT slightly ahead in some settings). On HellaSwag, the models are broadly comparable. This suggests that the performance gap between Polyglot-Ko and existing Korean models is real but not enormous — the primary contribution is providing a new, larger-scale option in a resource-constrained ecosystem, not claiming revolutionary improvements over prior Korean-specific work.

The critical missing piece that Polyglot-Ko provides is scale combined with open availability. Prior to this work, no publicly available Korean model exceeded roughly 6B parameters in a commercially usable form. The 12.8B model directly addresses this gap, and the paper's results show that this additional scale translates to improved benchmark performance.

The Data Quality Dimension

The paper's motivation is not solely about scale — it is equally about data quality. The authors devote considerable attention (Section 2.1) to their data preprocessing pipeline, which addresses issues like personally identifiable information (PII) removal, deduplication, HTML tag cleaning, and filtering of excessively short or malformed texts. This focus suggests an implicit argument: one reason existing multilingual models underperform on Korean is that their Korean training data likely includes noisy, poorly filtered web text that provides a weak learning signal. By carefully curating 863 GB of Korean text from diverse sources (blog posts, news articles, patent documents, Q&A datasets, fiction, Wikipedia, and more — see Table 1), the authors aim to maximize the learning signal per training token.

The paper also candidly acknowledges data preprocessing errors — specifically, "the data was incorrectly stripped of newlines, leading to a loss of document structure" (Section 5). This admission is revealing: it demonstrates that even with careful curation, data quality issues can slip through, and it signals to the community that document structure preservation matters for Korean language modeling. The fact that the models achieved strong results despite this error suggests that the underlying data was rich enough to partially compensate.

How This Paper Positions Itself

The Polyglot-Ko paper positions itself not as a methodological contribution — there are no novel architectures, training algorithms, or evaluation techniques — but as a resource contribution that fills a specific, well-defined gap. The paper's value proposition is:

We built a large, high-quality Korean dataset (1.2 TB raw, 863 GB preprocessed) with diverse sources and careful filtering.
We trained and released four models across a range of parameter scales (1.3B to 12.8B), all using the same tokenizer and training framework (GPT-NeoX), enabling fair scaling comparisons.
We benchmarked these models against existing alternatives on a standard Korean evaluation suite (KOBEST), providing empirical evidence that language-specific training yields better performance than multilingual approaches at comparable scale.
We made everything open-source and commercially usable, lowering the barrier to entry for Korean NLP research and development.

The paper is, in essence, a technical report documenting these contributions — similar in spirit to the GPT-NeoX-20B report (Black et al., 2022) or the Pythia scaling suite (Biderman et al., 2023), both from the same research collective (EleutherAI). It does not claim to advance the state of the art in language modeling architecture or training methodology. Instead, it claims to advance the state of availability for Korean language models, which is a different but equally legitimate form of contribution in a field where access to models often determines who can participate in research and development.

The paper also gestures toward a broader vision — the Polyglot project's ultimate goal of building multilingual models that genuinely serve non-English languages — but makes clear that the Korean models are a deliberate first step: "We made a deliberate decision to prioritize the development of Korean models before venturing into multilingual models. This choice was motivated by multiple factors: firstly, the Korean models facilitated performance comparisons with existing multilingual models; and finally, they catered to the specific needs of Korean companies and researchers" (Abstract). This framing positions Polyglot-Ko as a proof of concept for a monolingual-first approach to improving multilingual model quality: get each language right individually before combining them.

The Broader Significance

While this paper is narrowly focused on Korean, the problem it addresses — the performance gap between English and non-English languages in large language models — is global. The approach it demonstrates (curate high-quality monolingual data, train dedicated models at multiple scales, release openly) is replicable for any language with sufficient digital text resources. The paper's existence implicitly argues that the NLP community should invest more heavily in non-English model development rather than relying on English-centric multilingual models to serve all languages. For speakers of languages that receive even less attention than Korean in the global NLP research landscape, this paper provides both a template and a motivating example of what dedicated language-specific efforts can achieve.

3. Technical Approach

3.1 Reader Orientation

This paper presents a family of four autoregressive language models trained to predict the next token in Korean text, ranging from 1.3 billion to 12.8 billion parameters, all built using a standard transformer decoder architecture with no architectural novelty. The core problem it solves is the absence of large-scale, publicly available, commercially usable Korean language models — the Polyglot-Ko system provides these models and the training pipeline (data curation, preprocessing, tokenization, distributed training) that produced them, enabling Korean NLP researchers and practitioners to fine-tune or directly use these models on downstream tasks.

3.2 Big-Picture Architecture (Diagram in Words)

The Polyglot-Ko system has five major stages, arranged in a linear pipeline:

Raw Data Collection — 1.2 TB of Korean text gathered from 15 diverse sources (blog posts, news, patents, Q&A forums, Wikipedia, etc.) in collaboration with TUNiB.
Data Preprocessing and Filtering — a cleaning pipeline that removes HTML tags, PII, duplicate documents, empty or excessively short texts, and malformed content, producing 863 GB of clean training text.
Tokenizer Training — a morpheme-aware Byte-Level BPE tokenizer trained on the cleaned Korean corpus using MeCab for morphological analysis, producing a shared vocabulary of 30,003 tokens used across all four model sizes.
Model Training — four separate GPT-NeoX transformer decoder models (1.3B, 3.8B, 5.8B, 12.8B parameters) trained on the preprocessed data using distributed data- and model-parallel training on 256 A100 GPUs provided by Stability AI, with varying token budgets (167B–219B tokens) due to computational constraints and early stopping decisions.
Evaluation — few-shot and zero-shot evaluation on the KOBEST benchmark (COPA, HellaSwag, BoolQ, SentiNeg, WiC) using EleutherAI's lm-evaluation-harness, comparing against existing Korean and multilingual models.

Information flows in a single direction: raw text enters the preprocessing pipeline → clean text trains the tokenizer and serves as model input → the tokenizer converts text to token IDs → the transformer models are trained via next-token prediction → trained checkpoints are evaluated on Korean downstream tasks using standardized prompts with varying numbers of in-context examples.

3.3 Roadmap for the Deep Dive

First, the data collection and composition — what sources were used, how much data each contributed, and why this diversity matters for a general-purpose Korean model.
Second, the data preprocessing pipeline — the quality issues identified, the cleaning steps applied, and the design principle of prioritizing long, well-formed text while removing PII and noise.
Third, the tokenizer — the vocabulary size, the Byte-Level BPE algorithm, and the critical design choice of using MeCab-based morphological analysis to build a tokenizer suited to Korean's agglutinative morphology.
Fourth, the model architectures and training configurations — the hyperparameter tables for all four model sizes, the distributed training setup, the training token budgets, and the early stopping decisions triggered by broken generation near epoch boundaries.
Fifth, the evaluation protocol — the KOBEST benchmark tasks, the few-shot prompting methodology, the evaluation harness, and the prompt modification for SentiNeg that substantially improved zero-shot results.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a resource contribution paper whose core idea is that carefully curated monolingual Korean data, trained at multiple parameter scales using a standard transformer architecture with a Korean-optimized tokenizer, yields models that outperform comparably sized multilingual alternatives on Korean-language tasks — and that making these models publicly and commercially available fills a critical gap in the Korean NLP ecosystem.

Data Collection and Composition

The training corpus originates from a collaboration with TUNiB, a Korean AI company, who collected the raw data. The total raw dataset size is 1.2 TB, which after preprocessing reduces to 863 GB of clean text used for training. Table 1 in the paper enumerates 15 distinct sources, but the distribution is highly skewed — a small number of sources dominate the total volume, while many contribute less than 1 GB each.

The single largest source is Korean blog posts at 682.3 GB, accounting for approximately 79% of the total 863 GB training corpus. This dominance means that blog-style writing — informal, conversational, opinionated, and covering a wide range of everyday topics — constitutes the bulk of the model's exposure to Korean text. This is both a strength (the model learns colloquial Korean and diverse topical vocabulary) and a potential weakness (formal, technical, or domain-specific registers may be underrepresented relative to their importance in certain downstream applications).

The second-largest source is a Korean news dataset at 87.0 GB (about 10% of the corpus). News text provides formal, fact-focused prose with different syntactic patterns and vocabulary distributions compared to blog content. This source helps balance the informal skew of the blog data and exposes the model to the kind of objective, declarative writing that is important for tasks like question answering and factual generation.

The remaining sources, in order of contribution:

Modu corpus (26.4 GB): A corpus from the National Institute of Korean Language, likely containing curated, high-quality Korean text across various registers. This provides a quality anchor — text that has been professionally vetted for linguistic correctness.
Korean patent dataset (19.0 GB): Technical and legal Korean, exposing the model to domain-specific terminology and highly structured prose.
Korean Q&A dataset (18.1 GB): Question-answer pairs from Korean online platforms, providing conversational patterns and explicit answer structures that are directly relevant to downstream QA tasks.
KcBert dataset (12.7 GB): The training corpus associated with KcBERT, a Korean BERT model, likely consisting of comments and social media text.
Korean fiction dataset (6.1 GB): Literary Korean with narrative structure, dialogue, and creative language use.
Korean online comments (4.2 GB): Short, informal user comments — high in colloquialisms, slang, and internet-native Korean.
Korean Wikipedia (1.4 GB): Encyclopedia-style Korean with formal, definitional prose and structured factual knowledge.
Six additional sources each under 1 GB: Clova call (spoken dialogue transcripts), Naver Sentiment Movie Corpus (movie reviews with sentiment labels), a Korean hate speech dataset, Open Subtitles (movie/TV subtitles), AIHub various tasks datasets (collections for specific NLP tasks like NER and text classification), and the Standard Korean dictionary (definitions and lexical entries).

This source diversity is deliberate. The paper states that the model should have "versatility across different domains and styles" (Section 1). A model trained only on news text would produce formal prose ill-suited for chatbot applications; a model trained only on blog posts might struggle with factual accuracy. By including legal (patents), literary (fiction), conversational (comments, Q&A, Clova call), factual (Wikipedia, news), and task-specific data, the training corpus exposes the model to the full breadth of Korean language use.

However, the extreme skew toward blog data (79%) means that blog-style writing dominates the training signal. The model's internal language model is fundamentally shaped by how Korean bloggers write, with other sources providing supplementary exposure to different registers rather than equal representation. This is not necessarily a flaw — it reflects the natural distribution of available Korean text on the internet — but it means the model may be more fluent in informal Korean than in, say, legal argumentation or technical patent language, regardless of the token count from those specialized sources.

Data Preprocessing and Quality Control

Section 2.1 describes a systematic effort to identify and remediate data quality issues before training. The authors categorize the types of problems they found into four broad classes:

Data available for training without modification. This category contains "predominantly news and Wikipedia data, which provide substantial information with sufficiently long text sequences." The implication is that these sources arrived relatively clean — well-formed paragraphs, minimal markup, few duplicates — and could be used with minimal intervention.

Data requiring contextual information. This category includes "blog data and news data" containing "numerous short texts that were incorrectly scraped, necessitating the inclusion of contextual information during the training process." This is a partial admission of a preprocessing failure. Some web scraping produced text fragments — single sentences or even phrases — that lost their surrounding document context. For language model training, such fragments are problematic because the model learns from truncated sequences that lack the surrounding discourse needed to infer meaning. The paper does not specify how they attempted to restore context or whether these fragments were simply excluded via the short-text filter. The phrase "necessitating the inclusion of contextual information during the training process" is ambiguous — it could mean they attempted to re-scrape the surrounding text for these fragments, or it could mean they concatenated short texts from the same source to form longer sequences.

Data containing hate speech. The authors note "a significant presence of hate speech in datasets sourced from certain community websites." This is a documentation of risk rather than a cleanup action — they do not describe a hate speech filter, only that they observed the problem. Section 5 (Limitations) later acknowledges that "Polyglot-Ko may generate content that is socially unacceptable or offensive" and recommends human curation or filtering mechanisms for deployments. This suggests that hate speech was not systematically removed at the preprocessing stage, which is a notable omission for a model intended for commercial use.

NLP task-specific data. Datasets designed for specific NLP tasks (text classification, named entity recognition) require "separate handling during model evaluation" because including them in pretraining would leak evaluation labels into the training data. The paper acknowledges this concern but does not detail how they handled it — presumably these datasets were excluded from the training corpus or included only as text (stripping labels) if that was feasible.

Beyond these categories, the paper itemizes eight specific quality issues addressed in the preprocessing pipeline:

Empty text: Instances with zero text content — simply discarded. These likely arise from failed web scrapes or parsing errors.
Unnecessary spaces: Korean text, like many languages, does not use spaces to separate words within compound expressions or certain grammatical constructions (unlike English). "Unnecessary spaces" likely refers to formatting artifacts — multiple consecutive spaces, trailing spaces, or spaces inserted in positions that break normal Korean spacing rules.
De-identification: "Identification and removal of personally identifiable information within the data instances." This is a crucial step for a publicly released model trained on web-scraped data. PII in the training data creates privacy risks (the model could memorize and regurgitate phone numbers, resident registration numbers, email addresses, or names) and legal liability. The paper does not specify the PII detection method — whether rule-based (regex patterns for Korean ID numbers, phone formats, email addresses) or model-based (a trained PII classifier).
Uncleaned HTML tags: "Removal of HTML tags that had not been properly cleaned." Web scraping often leaves residual HTML markup in the extracted text — <div>, <br>, <a href="...">, and similar elements. These are noise for language model training because they teach the model to generate HTML rather than natural language text. The paper describes this as handling tags that survived an initial cleaning pass, suggesting a multi-stage HTML stripping process.
Deduplication: "Identification and removal of duplicated data instances based on exact matches." This is exact deduplication (not fuzzy or near-duplicate detection), meaning it catches identical documents that appear in multiple web crawls or across different sources. Exact deduplication is simpler than semantic deduplication (e.g., MinHash), but it is effective at removing the most egregious repetition — such as the same news article appearing on multiple aggregation sites.
Broken code: "Handling instances where only fragments of HTML or Markdown were present." This is distinct from full HTML pages — these are text snippets that are essentially code debris, such as a lone </div> tag or a Markdown table fragment. These are unlikely to help language modeling and may teach the model to produce code-like artifacts in Korean text.
Short text: "Detection and handling of excessively short data instances." The paper does not specify the minimum length threshold, but the rationale is clear: very short sequences provide limited context for training a next-token predictor. A single word or short phrase teaches the model little about Korean syntax or discourse structure. The "handling" could mean either discarding them or concatenating multiple short texts into longer sequences (the latter would be consistent with the "contextual information" concern discussed earlier).
Repeated characters: "Identification and addressing of instances with repeated characters." This likely refers to keyboard-mashing artifacts (e.g., "ㅋㅋㅋㅋㅋㅋㅋ") or formatting errors (repeated punctuation or spaces). While some repeated characters in Korean are legitimate — "ㅋㅋㅋ" is Korean internet slang for laughter, analogous to "lol" — excessive repetition beyond typical usage patterns is likely data noise.

The newline stripping error. Section 5 discloses a significant preprocessing mistake:

"we discovered mistakes in the data preprocessing phase during our experiments. Specifically, the data was incorrectly stripped of newlines, leading to a loss of document structure."

This means that paragraph breaks, section boundaries, and other structural cues signaled by newlines were removed from the training data. In Korean text, as in English, newlines carry important information: they separate paragraphs (which typically correspond to topic boundaries), mark dialogue turns, and distinguish headings from body text. By stripping them, the preprocessing pipeline turned documents into long, undifferentiated blocks of text where the model could not learn to generate paragraph breaks appropriately. The paper frames this as an acknowledged limitation and a priority for future iterations, implying that the model's ability to produce well-structured documents is degraded relative to what better preprocessing would enable.

The overall preprocessing philosophy is pragmatic: remove clear noise (HTML, empty texts, exact duplicates, PII, broken code), filter out low-information sequences (very short texts), and accept that some issues (hate speech, newline structure loss, contextual fragmentation) will persist and impact model behavior. This is not a rigorous data curation pipeline by the standards of, say, the Pile (Gao et al., 2020) or RefinedWeb (Penedo et al., 2023), but it represents a substantial investment in data quality for a language where high-quality, pre-cleaned training corpora are scarce.

Tokenizer Design

The paper uses a single tokenizer across all four model sizes, trained on the same preprocessed Korean corpus. The key design decisions are:

Byte-Level BPE (Byte-Pair Encoding). BPE is a subword tokenization algorithm that starts with a vocabulary of individual characters (or bytes) and iteratively merges the most frequent adjacent pairs to create new subword tokens, up to a target vocabulary size. The "byte-level" variant operates on the raw byte representation of text rather than on Unicode characters, which has two advantages: (1) it guarantees that any Unicode character can be represented (avoiding the <unk> token problem), and (2) it handles non-Korean text (Latin characters, emoji, special symbols) gracefully since the byte encoding is universal. For a Korean model that may encounter English loanwords, URLs, or code snippets in training data, byte-level encoding provides robustness against out-of-vocabulary inputs.

Vocabulary size of 30,003. The vocabulary size is specified as "30,003" in the model configuration table (Table 2), with a parenthetical note "30,003 / 30,080" — the 30,080 likely refers to the actual embedding matrix dimension (padded to a convenient multiple), while 30,003 is the number of active tokens. This is a standard vocabulary size for models in the 1–13B parameter range — GPT-NeoX-20B uses 50,432, GPT-3 uses 50,257, and OPT-175B uses 50,272. A vocabulary of 30,003 is moderate: smaller vocabularies produce longer token sequences (less compression), while larger vocabularies increase the embedding matrix memory footprint. For a 5.8B model with a model dimension of 4,096, the embedding layer alone requires approximately $30,003 \times 4,096 \approx 123$ million parameters, or about 2% of total model parameters.

Morpheme-aware tokenization via MeCab. This is the most important design choice for Korean specifically. The paper states:

"We performed morpheme analysis using MeCab, a widely-used morphological analysis tool for Korean text."

To understand why this matters, recall that Korean is an agglutinative language. A single Korean word can encode information that English expresses through multiple words and grammatical particles. For example, the Korean word "가르쳐드렸습니다" (I taught [to someone]) contains the verb root "가르치-" (teach), the benefactive auxiliary "-드리-" (doing for someone's benefit), the past tense marker "-었-", and the formal declarative ending "-습니다". This single word corresponds to a multi-word English phrase.

Standard BPE applied directly to Korean text would tokenize based purely on character co-occurrence statistics, which may split words at suboptimal boundaries — for instance, splitting a verb root from its grammatical endings in ways that obscure morphological relationships. By running MeCab first to segment text into morphemes (meaning-bearing units), and then applying BPE to those morphemes, the tokenizer learns subword units that respect Korean morphological structure. This means that related words sharing the same root will share overlapping token representations, helping the model learn morphological patterns.

The practical workflow is: (1) run MeCab on all training text to identify morpheme boundaries, (2) insert boundary markers (likely whitespace) between morphemes, (3) run byte-level BPE on the morpheme-segmented text to learn subword merges up to a vocabulary of 30,003 tokens. At inference time, input text undergoes the same MeCab segmentation before tokenization.

The paper does not report metrics on tokenizer fertility (average tokens per Korean character or word) or out-of-vocabulary rates, so the quantitative impact of morpheme-awareness cannot be assessed from the paper alone. However, this design choice reflects a thoughtful engagement with Korean's linguistic properties — it is not a generic "throw BPE at raw text" approach, but an adaptation of the tokenization pipeline to the morphology of the target language.

Shared tokenizer across all model sizes. Using the same tokenizer for all four models enables direct comparison: any performance difference between the 1.3B and 12.8B models is attributable to model capacity and training volume, not to differences in how text is tokenized. This is consistent with the Pythia scaling suite methodology (Biderman et al., 2023), which similarly uses a single tokenizer across model sizes to isolate the effects of scale.

Model Architecture and Configurations

All four models share the same fundamental architecture: a GPT-NeoX transformer decoder (Andonian et al., 2021), which is EleutherAI's implementation of a standard autoregressive transformer closely following the GPT-3 architecture but with specific design choices inherited from the GPT-NeoX-20B codebase. The architecture uses:

Rotary Positional Embeddings (RoPE). RoPE encodes positional information by rotating the query and key vectors in the attention mechanism by an angle proportional to their position. Unlike learned absolute position embeddings (used in GPT-2 and GPT-3), RoPE is applied at each attention layer rather than added once at the input. It has the property that the dot product between query at position $i$ and key at position $j$ depends only on the relative distance $j - i$ , which means the model naturally handles sequences longer than those seen during training (extrapolation) and learns translation-invariant attention patterns. For Korean text, where word order within clauses is relatively free but clause boundaries carry important discourse information, relative position encoding is theoretically well-suited. The RoPE dimension is fixed at 64 across all model sizes, meaning each attention head applies rotary embeddings to only 64 of its dimensions (the first 64 dimensions of the query and key vectors are rotated, the remainder are passed through unchanged — this is a standard RoPE partial application).

Model Parallelism for Larger Models. The paper describes using model parallelism for the 3.8B, 5.8B, and 12.8B models (the 1.3B model fit on a single GPU). Model parallelism splits the model's layers across multiple GPUs, with each GPU responsible for computing and storing only a subset of layers — activations are forwarded between GPUs as the computation proceeds through the network. This is distinct from data parallelism, where each GPU holds a full copy of the model and processes different batches of data. The 12.8B model uses "a scale of 2 times larger than the 5.8B model" in its model parallelism configuration, meaning it was distributed across twice as many GPUs along the pipeline dimension.

Gradient Accumulation Steps (GAS). For the 12.8B model, the paper notes that "the overall batch size was maintained through the use of gradient accumulation steps." Gradient accumulation allows simulating a large effective batch size when GPU memory constraints prevent fitting that many samples at once. Instead of updating weights after each micro-batch, the model computes gradients for several micro-batches, accumulates them (sums them), and then applies a single optimizer step for the accumulated total. This preserves the statistical properties of large-batch training (more stable gradient estimates) while respecting memory limits. The tradeoff is that gradient accumulation increases the number of forward-backward passes per optimizer step, effectively trading memory for computation time.

Model Configuration Table (Table 2). The paper provides a complete hyperparameter table for all four models. Let's walk through each parameter and what it means:

Hyperparameter	1.3B	3.8B	5.8B	12.8B
$n_{parameters}$	1,331,810,304	3,809,974,272	5,885,059,072	12,898,631,680
$n_{layers}$	24	32	28	40
$d_{model}$	2,048	3,072	4,096	5,120
$d_{ff}$	8,192	12,288	16,384	20,480
$n_{heads}$	16	24	16	40
$d_{head}$	128	128	256	128
$n_{vocab}$	30,003 / 30,080	30,003 / 30,080	30,003 / 30,080	30,003 / 30,080
Positional Encoding	Rotary (RoPE)	Rotary (RoPE)	Rotary (RoPE)	Rotary (RoPE)
RoPE Dimensions	64	64	64	64

Parameter count ( $n_{parameters}$ ). These are non-embedding parameter counts, consistent with how GPT-NeoX reports model sizes. The 1.3B model has 1,331,810,304 parameters; the 12.8B model has 12,898,631,680 parameters — approximately a 9.7× increase in capacity. The four models cover a nearly 10× range in scale, enabling within-family scaling analysis.

Number of layers ( $n_{layers}$ ). Layer count increases with model size but not monotonically with parameter count: 24 (1.3B), 32 (3.8B), 28 (5.8B), 40 (12.8B). Note that the 5.8B model has fewer layers (28) than the 3.8B model (32) despite being larger — it compensates with a much wider model dimension. This means the 5.8B model is a "shallower but wider" architecture compared to the 3.8B, which has different implications for what kinds of computation the model can perform. Deeper models can compose more sequential transformations, while wider models can represent more information at each layer. The choice to make the 5.8B model shallower-and-wider is an architectural design decision whose motivation is not explained in the paper — it may reflect empirical tuning for Korean text, hardware efficiency considerations (wider models are more parallelizable than deeper ones), or simply an exploratory choice in the parameter sweep.

Model dimension ( $d_{model}$ ). This is the width of the residual stream — the size of the hidden state vector that passes through each layer. It increases from 2,048 (1.3B) to 5,120 (12.8B), representing a 2.5× expansion. The residual stream is the model's "working memory" — it carries token representations through the network, accumulating information from each layer's computations. Wider residual streams can represent more features in parallel, which is especially important for representing the complex morphological and syntactic properties of Korean text.

Feed-forward dimension ( $d_{ff}$ ). This is the inner dimension of the position-wise feed-forward network in each transformer layer, typically $4 \times d_{model}$ . The values in Table 2 confirm this ratio: 8,192 = 4 × 2,048; 12,288 = 4 × 3,072; 16,384 = 4 × 4,096; 20,480 = 4 × 5,120. The feed-forward network is where the model stores factual knowledge and performs non-linear transformations — a 4× expansion ratio means the FFN has four times the capacity of the attention mechanism at each layer, which is the standard choice in GPT-style architectures.

Number of attention heads ( $n_{heads}$ ). Attention heads control how the model splits its attention computation across parallel pathways. The 1.3B model uses 16 heads (each of dimension 128, so total attention dimension = 16 × 128 = 2,048 = $d_{model}$ — a consistent split). The 3.8B uses 24 heads of dimension 128 (24 × 128 = 3,072). The 5.8B uses 16 heads of dimension 256 (16 × 256 = 4,096). The 12.8B uses 40 heads of dimension 128 (40 × 128 = 5,120). The 5.8B model stands out: it uses fewer heads (16) than the 3.8B model (24) but with double the per-head dimension (256 vs. 128). This means each attention head in the 5.8B model sees a higher-dimensional subspace of the representation, which may allow it to capture more complex attention patterns per head at the cost of fewer parallel attention pathways. The paper does not explain this choice.

Head dimension ( $d_{head}$ ). Per-head dimension — the size of the query, key, and value vectors for each attention head. Stays at 128 for the 1.3B, 3.8B, and 12.8B models, but increases to 256 for the 5.8B model. A head dimension of 128 is standard in many transformer implementations (GPT-3, OPT) and is likely chosen for hardware efficiency on A100 GPUs (128-dimensional dot products map well to 128-byte cache lines and tensor core operations).

Vocabulary size ( $n_{vocab}$ ). As discussed in the tokenizer section: 30,003 active tokens, with 30,080 being the effective embedding matrix size (padded). Note the consistency — all four models share the identical tokenizer and embedding configuration.

RoPE dimensions: 64 across all models. The dimension of the rotary embedding subspace is constant regardless of model scale, meaning that as models get larger, a smaller fraction of the attention computation is modulated by relative position. In the 12.8B model with 40 heads of dimension 128, only 64 of each head's 128 dimensions carry position information — the remaining 64 dimensions attend without positional modulation. This partial RoPE application is a deliberate design: it allows each head to blend positional (rotated) and non-positional (unrotated) attention, which may be beneficial for tasks where content-based attention is more important than position-based attention (e.g., semantic matching of key phrases).

The paper does not report several hyperparameters that are typically important for transformer training: the optimizer (likely AdamW, consistent with GPT-NeoX defaults), the learning rate schedule (warmup steps, decay type), the dropout rate (if any), the activation function (likely GeLU, consistent with GPT-NeoX), the weight initialization scheme, or the sequence length used during training. These omissions are consistent with the "technical report" framing — the paper assumes readers are familiar with GPT-NeoX defaults and does not claim novelty in the training methodology.

Training Process and Token Budgets

The paper reports training details for each model individually, and they reveal a pattern of computational constraints shaping the training process:

1.3B model. Trained on 213B tokens with a total batch size of 1,024. At a batch size of 1,024, each optimizer step processes 1,024 sequences (of unspecified length — likely 2,048 tokens, based on GPT-NeoX defaults). The model was trained without model parallelism (all layers on a single GPU). The paper reports that "broken generation was observed as the loss sharply dropped around the 100,000 steps." At a batch size of 1,024 and 100,000 steps, the model would have processed approximately $100,000 \times 1,024 = 102.4$ million sequences. With a sequence length of 2,048 tokens, this is approximately 210B tokens — close to the reported 213B total tokens, suggesting the loss drop occurred very near the end of the first training epoch. The model checkpoint was taken before this loss drop, meaning the final model represents a state shortly before the collapse rather than at the point of minimum training loss.

3.8B model. Trained on 219B tokens with the same total batch size of 1,024, but with model parallelism enabled. The paper reports that it "experienced same symptoms at around 100,000 steps" — the same sharp loss drop and broken generation near the epoch boundary. As with the 1.3B model, "the decision was made to halt the model training process" before the collapse. This symmetry — both models exhibiting degraded generation at the same approximate step count (100,000) despite different sizes — suggests it is not an overfitting issue per se (if it were, the larger model might be expected to handle more training before overfitting). The authors speculate "it was overtrained or overfitting, but there are not any scientific evidence" — the honest acknowledgment of uncertainty is notable. The broken generation symptom (the model "generates same tokens repeatedly") could indicate training instability, an optimizer issue near the epoch boundary where the model sees repeated data, or a learning rate that was too high for the late stages of the first epoch.

5.8B model. Trained on 172B tokens with model parallelism and a batch size reduced by 1/4 compared to the 1.3B and 3.8B models — so a total batch size of 256. The smaller batch size means each training step sees fewer examples, producing noisier gradient estimates but allowing the model to take more steps for the same token budget (the training lasted "a total of 320,000 steps" to process 172B tokens). The paper reports that "the model's performance consistently improved as the number of training steps increased" — crucially, this model did not exhibit the loss collapse that affected the smaller models. The smaller batch size (and correspondingly different optimization dynamics) may explain this. With a batch size of 256, the gradient noise is higher, which can act as implicit regularization that prevents the sharp loss drops observed at batch size 1,024. Alternatively, the 5.8B model's shallower-but-wider architecture may simply be more stable under the training conditions.

12.8B model. Trained on 167B tokens with model parallelism "at a scale of 2 times larger than the 5.8B model" and gradient accumulation to maintain effective batch size. It was trained "for a total of 301,000 steps." The training duration (301K steps) and token count (167B) imply an effective batch size around $167 \times 10^9 / (301,000 \times 2,048) \approx 271$ sequences per step — similar to the 5.8B model's batch size of 256, which is consistent with the description of "the overall batch size was maintained through the use of gradient accumulation steps." The paper does not report whether the 12.8B model experienced the epoch-boundary collapse issue.

Training infrastructure. All models were trained on 256 A100 GPUs (organized as 8 GPUs × 32 nodes on an HPC cluster) provided by Stability AI. A100 GPUs have 40 GB or 80 GB of VRAM and are optimized for mixed-precision training (FP16/BF16), which halves memory usage for activations and gradients compared to FP32. For the 12.8B model with a model dimension of 5,120 and 40 layers, the model parameters alone require approximately $12.9 \times 10^9 \times 2$ bytes (FP16) = 25.8 GB of GPU memory. Adding optimizer states (AdamW stores two moments per parameter, each FP32, so 8 additional bytes per parameter) brings the total to roughly $12.9 \times 10^9 \times 10$ bytes = 129 GB — far exceeding a single A100's memory. This is why model parallelism and gradient accumulation were necessary: the model was split across multiple GPUs (pipeline parallelism), with each GPU storing only a fraction of the layers and their associated optimizer states.

Training tokens and the Chinchilla scaling law. For context, the Chinchilla scaling law (Hoffmann et al., 2022) suggests that compute-optimal training uses approximately 20 tokens per parameter. For a 12.8B model, compute-optimal training would require approximately 256B tokens. The actual training budget of 167B tokens is about 65% of the compute-optimal amount, meaning the 12.8B model is significantly undertrained relative to what the Chinchilla formula would prescribe. The 1.3B model at 213B tokens is conversely overtrained (Chinchilla would prescribe ~26B tokens, the model received 8× that). This variability in training budgets across model sizes — driven by the computational constraints and stability issues — means that the scaling comparisons in Section 4 are not "pure" scaling comparisons where all models are trained to compute-optimality or to equal under-training ratios. The 1.3B model likely benefits from extensive training on a rich corpus, while the 12.8B model likely has untapped capacity that additional training tokens would unlock.

Evaluation Protocol and Benchmark Tasks

The paper evaluates on the KOBEST benchmark (Kim et al., 2022), a Korean-language evaluation suite covering five tasks. All evaluations use the lm-evaluation-harness from EleutherAI (Gao et al., 2021), specifically the "polyglot" branch which was adapted for Korean language evaluation. The metric is F1 score for all tasks except WiC (where accuracy is used to highlight near-random performance). The evaluation uses few-shot prompting: the model receives $n$ input-output examples from the task before being asked to produce an answer for a test instance, where $n$ is varied across {0, 5, 10, 50}. The specific prompts used are those provided with the KOBEST dataset, except for SentiNeg where a modified prompt was developed (discussed below).

The tasks are:

COPA (Choice of Plausible Alternatives). A causal reasoning task: given a premise sentence and two alternatives, the model must select which alternative is the cause or effect of the premise. For example, given "The man fell asleep" (premise), is the cause "He was tired" or "He was hungry"? This tests the model's understanding of causal relationships expressed in Korean. The F1 score measures the model's accuracy in selecting the correct alternative.

HellaSwag. A commonsense natural language inference task: given a context paragraph with the final sentence truncated, the model must select the most plausible completion from four options, where three are adversarial distractors (plausible-sounding but incorrect completions designed to fool language models). For example, given "A person is playing guitar on stage. They..." the completions might include "...play a song" (correct) and "...cook a meal on the guitar" (adversarial). This tests commonsense world knowledge as expressed in Korean. HellaSwag is particularly challenging for models trained on non-English data because the adversarial distractors often exploit English-centric commonsense that may transfer poorly.

BoolQ. A boolean question answering task: given a passage and a yes/no question about the passage, the model must answer correctly. For example, given a passage about the history of Seoul and the question "Is Seoul the capital of South Korea?" the answer is "Yes." This tests reading comprehension and factual reasoning in Korean. The F1 score is computed over the binary yes/no predictions.

SentiNeg. A sentiment analysis task specifically focused on negation detection. Negation in Korean is expressed through specific verbal endings and auxiliary verbs that can reverse the polarity of a sentence. For example, "The movie was good" vs. "The movie was not good" — the negation marker changes the sentiment from positive to negative. This task tests whether the model can recognize these morphological negation cues and understand their semantic impact.

WiC (Word in Context). A lexical semantics task: given a target word and two sentences containing it, determine whether the word is used with the same meaning in both contexts. For example, the Korean word "배" can mean "pear" (fruit), "ship" (vessel), or "stomach" (body part). Given two sentences, the model must decide if they use "배" in the same sense. The paper reports that "all models demonstrated random performance in this task" (accuracy around 0.48–0.51, where chance is 0.50), indicating that word-sense disambiguation from context is a hard problem that none of the evaluated models (including the largest Polyglot-Ko) successfully solve.

Prompt modification for SentiNeg. The paper reports that the default KOBEST prompt for SentiNeg was problematic:

"the prompt used in the KoBEST paper employed a simple classification task. However, the range of questions encompassed a wide spectrum and often exhibited ambiguity. Consequently, we made arbitrary modifications to the prompt in order to obtain results."

This is a significant methodological detail. The default prompt apparently produced unstable or low-quality results, particularly in zero-shot settings — the paper's Table 4 shows the default SentiNeg prompt producing F1 scores as low as 0.339 for Polyglot-Ko-5.8B in zero-shot, while the modified prompt (Table 5) achieves 0.878 for the same model. This 2.6× improvement from prompt modification alone demonstrates how sensitive few-shot evaluation is to prompt design, especially for classification tasks where the model needs to understand the output format expected. The paper provides the modified prompt details in an appendix, making the evaluation reproducible.

Comparison models. The paper compares against:

ko-gpt-trinity-1.2B: A Korean GPT model from SK Telecom, the most comparable model to Polyglot-Ko-1.3B in scale.
KoGPT-6.0B: A Korean GPT model from KakaoBrain, close in scale to Polyglot-Ko-5.8B.
XGLM-7.5B: Meta's multilingual GPT model, the largest publicly available multilingual model at the time. This comparison directly tests the paper's central claim: a smaller, monolingual Korean model (e.g., Polyglot-Ko-1.3B) can outperform a larger multilingual model (XGLM-7.5B) on Korean tasks.

The evaluation protocol does not include fine-tuning — all results are from zero-shot or few-shot prompting, meaning the models are evaluated on their pretrained Korean capabilities without task-specific adaptation. This is consistent with the paper's framing of Polyglot-Ko as a general-purpose Korean language model that should perform well across diverse tasks "out of the box."

Summary of Design Choices and Their Justifications

Morpheme-aware BPE with MeCab over standard character-level BPE: respects Korean's agglutinative morphology, producing subword units that align with meaning-bearing morphemes rather than arbitrary character sequences. This allows the model to learn morphological patterns that would be obscured by character-level tokenization.
Byte-level encoding over character-level: guarantees no <unk> tokens, handles mixed-script text (Korean + English + symbols), and is standard practice in modern LLMs.
Single shared tokenizer across all model sizes: isolates scaling effects from tokenization differences, enabling clean within-family comparisons.
RoPE with partial application (64 dimensions) over learned absolute position embeddings: relative position encoding is better suited for languages with flexible word order (Korean can reorder constituents more freely than English), and partial application allows each head to blend positional and content-based attention.
Variable layer counts with 5.8B model being shallower-and-wider: the paper does not justify this choice explicitly, but wider models benefit more from data parallelism (because each layer has more parameters that can be computed in parallel), while deeper models require more sequential computation. The 5.8B architecture may represent a hardware-efficiency-motivated design.
Early stopping before epoch boundary for 1.3B and 3.8B models: a pragmatic response to generation quality collapse, prioritized model usability over training loss minimization. This is a common real-world tradeoff in LLM training — selecting checkpoints based on qualitative evaluation rather than validation loss.
Reduced batch size for 5.8B and 12.8B models (256 vs. 1,024): likely driven by GPU memory constraints (larger models need smaller micro-batches) and the observation that the larger models did not exhibit the epoch-boundary collapse, possibly because smaller batches provide implicit regularization through gradient noise.
Multiple training token budgets (167B–219B): a consequence of computational resource limitations rather than a deliberate experimental design. The paper is transparent that models were trained with "varying numbers of tokens" due to practical constraints, which is a more honest portrayal of the training process than many technical reports provide.
F1 score as primary metric for most tasks: appropriate for tasks with imbalanced classes (COPA alternatives, BoolQ yes/no, SentiNeg positive/negative) where simple accuracy could be misleading.
Exclusion of WiC from main results due to random performance: a transparent admission that the models (and all compared models) fail on this task, avoiding the common pitfall of burying negative results.

4. Key Insights and Innovations

Innovation 1: The Monolingual-First Strategy as a Principled Alternative to Multilingual Models

The paper's most significant conceptual contribution is not any technical invention but rather a strategic proposition: that building high-quality monolingual models is a more reliable path to strong non-English performance than waiting for multilingual models to improve. This flips the dominant assumption in the post-BLOOM, post-XGLM landscape, where the prevailing belief has been that scaling multilingual models to ever-larger sizes and more languages will eventually solve the non-English performance gap. Polyglot-Ko argues implicitly — through its existence and its results — that this assumption deserves scrutiny.

The evidence for this proposition is direct and interpretable. The paper compares Polyglot-Ko-1.3B (130M parameters, trained on 863 GB of Korean) against XGLM-7.5B (a multilingual model with ~5.8× more parameters). On COPA 0-shot, the smaller monolingual model scores 0.720 versus XGLM's 0.672 (Table 3, left). On SentiNeg 0-shot with the modified prompt, Polyglot-Ko-1.3B scores 0.889 versus XGLM's 0.835 (Table 5, left). On BoolQ, the gap is smaller but consistent (Table 4). Across nearly every task and shot setting, a Korean-specific model outperforms a much larger model that was asked to learn Korean alongside dozens of other languages — often from what was likely a much smaller Korean subcorpus within the multilingual training mixture.

What makes this more than a simple "more data helps" story is the mechanism implied by the comparison. XGLM's parameter count suggests it has substantial representational capacity — capacity that could, in principle, be allocated to Korean. But the allocation is determined by the training data distribution, which skews heavily English. The result is that XGLM's ~7.5B parameters are effectively amortized across many languages, with Korean receiving only a fraction of the model's total capacity. Polyglot-Ko's 1.3B parameters, by contrast, are entirely dedicated to Korean. The paper demonstrates that dedicated capacity on high-quality monolingual data beats shared capacity on English-centric multilingual data, at least for the languages and scales tested.

This finding echoes arguments made in the machine translation literature — where dedicated bilingual systems often outperform massively multilingual ones for specific language pairs — but extends them to the general-purpose language model setting. It also prefigures the later "language-specific fine-tuning" approach that would become common (e.g., fine-tuning LLaMA on Arabic data to produce Jais), but at a more fundamental level: Polyglot-Ko argues for language-specific pretraining, not just fine-tuning. The model's representations of Korean were built from scratch on Korean data alone, free from the interference of English-centric optimization signals.

The paper's transparency about its broader vision reinforces this insight. The authors describe the Korean models as a deliberate first step in a larger Polyglot project aimed at multilingual models, but the ordering matters: "We made a deliberate decision to prioritize the development of Korean models before venturing into multilingual models" (Abstract). The implication is that building good multilingual models requires first understanding what good monolingual models look like for each constituent language — a "get each language right first" philosophy rather than a "train on everything and hope it works" approach. This is a methodological reframing of how to approach the multilingual model problem, not just a resource release.

This innovation is incremental in its mechanism (monolingual pretraining is the oldest paradigm in language modeling) but fundamentally important as a strategic argument because it challenges the field's momentum toward ever-larger multilingual models as the default answer to non-English NLP. The paper effectively says: before scaling to 100 languages at once, make sure you can do one language well. The strong comparative results against XGLM — a model from a major industrial lab with vastly more resources — lend credibility to this position.

Innovation 2: Morpheme-Aware Tokenization as a Key Design Decision for Agglutinative Languages

The tokenizer design choice — using MeCab morphological analysis as a preprocessing step before byte-level BPE — is the paper's most linguistically motivated technical contribution. While not entirely novel (morpheme-aware tokenization has been used in prior Korean NLP systems), the paper elevates this from a preprocessing detail to a first-order architectural decision that distinguishes Polyglot-Ko from a naive "download Korean text and run BPE" approach, and from multilingual models that apply a single tokenizer across languages with vastly different morphological structures.

To appreciate why this matters, consider what happens when a standard BPE tokenizer is trained on Korean text without morphological preprocessing. Korean's agglutinative morphology means that a single "word" (more precisely, an eojeol — a whitespace-delimited unit) can contain a content root plus multiple grammatical suffixes. For example, the verb form "먹었다" (ate) contains the root "먹-" (eat), the past tense marker "-었-", and the declarative ending "-다". A frequency-based BPE tokenizer sees this as a sequence of characters and merges them based purely on co-occurrence statistics. It might learn "먹었" as a token (root + partial suffix) and "다" as another token, obscuring the fact that "먹-" is the semantic core that appears in "먹고" (eat-and), "먹으면" (eat-if), "먹어서" (eat-because), etc.

By running MeCab first to segment text at morpheme boundaries, Polyglot-Ko's tokenizer sees "먹 + 었 + 다" as separate input units before BPE applies. BPE then learns to merge frequent morpheme sequences, but the fundamental tokenization respects the morphological structure. This means that the embedding for "먹" (the root) is shared across all its inflected forms, allowing the model to learn that these forms are semantically related — exactly what a speaker of Korean intuitively knows.

The paper does not provide a controlled ablation comparing morpheme-aware BPE against standard BPE on Korean text, which is a limitation. However, the decision is theoretically well-motivated from linguistic principles, and the strong downstream performance — particularly on tasks like SentiNeg (negation detection, which requires recognizing morphological negation markers like "안" and "-지 않-") and COPA (causal reasoning expressed through specific verbal endings) — is consistent with the hypothesis that morphological awareness improves Korean language understanding.

This innovation is incremental technically (the tools — MeCab and Byte-Level BPE — both existed before) but conceptually important because it identifies tokenization as a language-specific adaptation that should be part of the model design process, not an afterthought. For practitioners building language models for other morphologically rich languages (Turkish, Finnish, Hungarian, Japanese, Swahili), this design choice serves as a template: invest in language-specific tokenization before investing in scale.

The contrast with multilingual models is again instructive. A multilingual tokenizer trained on 50+ languages must allocate its vocabulary budget across diverse scripts and morphological systems. It cannot be optimized for Korean morphology specifically. This means that even if a multilingual model had equal Korean training data to Polyglot-Ko, its tokenizer would likely produce longer, less semantically coherent token sequences for Korean text — a built-in disadvantage that no amount of model scaling can fully overcome. The morpheme-aware tokenizer is thus not just a quality improvement for Polyglot-Ko; it is an argument that tokenization is a language-specific problem that multilingual approaches systematically undersolve.

Innovation 3: Scaling Monolingual Models as a Diagnostic for Data Quality and Training Stability

The paper's training of four models across a ~10× parameter range serves a purpose beyond the obvious "bigger is better" benchmarking. By training at multiple scales, the authors surface scale-dependent training phenomena that would be invisible in a single-model technical report, and that carry implications for anyone attempting to train large Korean (or other non-English) language models.

The most striking of these phenomena is the epoch-boundary generation collapse observed in the 1.3B and 3.8B models. Around 100,000 training steps — near the end of the first epoch over the training data — both models experienced a sharp loss drop accompanied by "broken generation (e.g. generate same tokens repeatedly)." The paper is admirably honest about this: "We suspect it was overtrained or overfitting, but there are not any scientific evidence" (Section 3, footnote 4). This is not a confident diagnosis; it is a documentation of an observed failure mode with uncertain causes.

What makes this a genuine insight — albeit a negative one — is its diagnostic value for the Korean NLP community. A practitioner reading this paper learns: if you train a Korean autoregressive LM at batch size 1,024 on ~200B tokens of Korean web data, you may encounter a catastrophic degradation in generation quality near the first epoch boundary, and you should checkpoint frequently and evaluate generation quality (not just loss) to catch it. The fact that the 5.8B model (trained at 1/4 batch size) did not exhibit this collapse, while the 1.3B and 3.8B models (at full batch size) did, hints at a possible role for batch size in the phenomenon — perhaps the larger-batch training converges to a sharper minimum that collapses when the model sees repeated data at the epoch boundary, while the noisier small-batch optimization avoids this regime.

This is not a universal finding about LLM training — it is specific to the Korean data, the model architectures, and the training hyperparameters used. But it is precisely this specificity that makes it valuable. The field's understanding of training dynamics comes disproportionately from English-language models (GPT-3, Chinchilla, LLaMA, Pythia). Phenomena that emerge when training on non-English data — with different text distributions, different morphological properties, different data quality characteristics — are underdocumented. The Polyglot-Ko paper's willingness to report these failures, hypothesize about causes, and describe the mitigation (early stopping before the collapse) is a contribution to collective knowledge about non-English LLM training that goes beyond the models themselves.

A second, related diagnostic emerges from the variable training token budgets. The 1.3B model received 213B tokens; the 12.8B model received 167B tokens. By the Chinchilla scaling law (20 tokens per parameter), the 1.3B model is massively overtrained (~164 tokens/param) while the 12.8B model is significantly undertrained (~13 tokens/param). This is not a deliberate experimental design — it reflects computational constraints — but it produces an interesting natural experiment: what scaling behavior do you observe when larger models are undertrained relative to smaller ones? The answer in this paper is that the 12.8B model still outperforms the smaller models across most tasks (Figure 2, Tables 3–4), suggesting that scale can partially compensate for undertraining on Korean downstream tasks. However, the gap between the 12.8B and 5.8B models on some tasks (e.g., HellaSwag 50-shot: 0.621 vs. 0.612) is small enough that one wonders what a properly Chinchilla-trained 12.8B Korean model might achieve. This is a diagnostic finding about the scale-vs-training-budget tradeoff for Korean — it suggests that the performance curve may not be far into diminishing returns at these scales, and that additional training tokens for the larger models could yield substantial further gains.

Both of these contributions are incremental in their specifics but collectively significant because they provide the kind of granular, language-specific training experience that is typically lost in polished publications. They make the paper useful to practitioners as a field guide to Korean LLM training, not just as a model release announcement.

Innovation 4: Prompt Sensitivity as a First-Class Evaluation Concern for Non-English Benchmarks

The paper's handling of the SentiNeg prompt is easy to overlook, but it encodes an important methodological insight: prompt design for non-English benchmarks is under-explored relative to English benchmarks, and default prompts inherited from dataset papers may substantially underestimate model performance.

The specific finding is stark. Using the default KoBEST prompt for SentiNeg, Polyglot-Ko-5.8B achieves an F1 score of 0.339 in zero-shot (Table 4, left), which is essentially non-functional. After the authors made "arbitrary modifications to the prompt" (Appendix A), zero-shot performance jumps to 0.878 — a 2.6× improvement from prompt engineering alone. This is not a small tweak; it is the difference between "the model cannot do this task" and "the model is reasonably good at this task." The 12.8B model similarly goes from 0.912 (default prompt, already good) to 0.893 (modified prompt — slightly lower, but the 5-shot jumps to 0.985 and the 50-shot to 0.982, which are near-ceiling scores).

What does this mean? First, it demonstrates that the SentiNeg task itself is solvable by these models — the capability exists in the pretrained representations. The default prompt was simply failing to elicit it. This is a prompt sensitivity problem, and it is likely more severe for non-English benchmarks because:

Fewer researchers have iterated on prompt design. English benchmarks like HellaSwag, BoolQ, and MMLU have been studied by dozens of papers, each refining prompts and discovering what formats work best. Korean benchmarks have received a tiny fraction of that attention. The default KoBEST prompts may be suboptimal in ways that have not yet been discovered.
Translation artifacts in multilingual benchmarks. KoBEST is not a translation of English benchmarks — it is a collection of Korean-native tasks — but the format of prompts (classification instruction, few-shot examples, test instance) follows conventions developed for English models. Whether these conventions transfer optimally to Korean — a language with different discourse patterns, politeness levels, and syntactic structure — is an open question that the SentiNeg prompt modification highlights.
Task ambiguity interacts with language-specific expression. The paper notes that the SentiNeg task's "range of questions encompassed a wide spectrum and often exhibited ambiguity" (Appendix A). Ambiguity in sentiment expression differs across languages and cultures. A prompt that works for English sentiment analysis may not adequately disambiguate Korean sentiment expressions, where negation can be expressed through multiple morphological and syntactic mechanisms, and where cultural norms around expressing negative sentiment differ.

The second implication is methodological: when evaluating non-English models on benchmarks designed for those languages, prompt sensitivity analysis should be a standard part of the evaluation protocol. The paper's approach — identify instability, modify the prompt, report both results — is a model of transparency that should be adopted more widely. Without it, zero-shot results on less-studied benchmarks can be misleadingly low, and inter-model comparisons can reflect prompt compatibility rather than genuine capability differences.

This insight is incremental in that prompt sensitivity is a known issue (English NLP has extensive literature on prompt engineering, prompt sensitivity, and prompt robustness), but it is fundamentally important for the non-English evaluation ecosystem because it identifies a systematic quality gap: non-English benchmarks are likelier to have poorly optimized prompts, and this fact can distort the research record until it is recognized and corrected. The paper's elevation of this issue from an appendix note to a main-paper finding (the modified SentiNeg results are discussed in the main text, Section 4.4) is a contribution to evaluation methodology for non-English NLP.

The WiC results — where all models perform at random — reinforce the broader point about evaluation quality. If every model, regardless of architecture, scale, or training data, achieves ~0.50 accuracy (chance level) on a binary task, the most likely explanation is not that all models are equally bad, but that the evaluation setup (task format, prompt, scoring) is not measuring what it intends to measure. The paper's decision to exclude WiC from the main results while reporting it transparently in the appendix is a methodological choice that signals: benchmark results should be interpreted critically, especially for less-vetted non-English benchmarks, and reporting that a task doesn't work is as valuable as reporting strong performance.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the KOBEST benchmark (Kim et al., 2022), a Korean-language evaluation suite consisting of five downstream tasks: COPA (causal reasoning), HellaSwag (commonsense NLI), BoolQ (boolean question answering), SentiNeg (negation-aware sentiment analysis), and WiC (word-in-context disambiguation). The paper does not specify a train/validation/test split for KOBEST — the evaluation is conducted on the standard KOBEST evaluation instances using few-shot prompting, meaning the models are evaluated in a zero-shot or in-context learning setup without task-specific fine-tuning or a held-out validation set for hyperparameter tuning.
Base model(s). Four Polyglot-Ko models are evaluated: 1.3B, 3.8B, 5.8B, and 12.8B parameters, all using the GPT-NeoX transformer decoder architecture with Rotary Positional Embeddings and a shared morpheme-aware Byte-Level BPE tokenizer (vocabulary size 30,003). These models were chosen to span a roughly 10× parameter range, enabling within-family scaling analysis. For comparison, three external models are evaluated: ko-gpt-trinity-1.2B (SKT, a Korean-specific model at comparable scale to the smallest Polyglot-Ko variant), KoGPT-6.0B (KakaoBrain, a Korean-specific model close in scale to Polyglot-Ko-5.8B), and XGLM-7.5B (Lin et al., 2022, Meta's multilingual autoregressive model, representing the multilingual baseline). These are the only publicly available billion-scale Korean or multilingual models identified by the authors at the time of writing.
Metrics. F1 score is used as the primary evaluation metric for all tasks except WiC, where accuracy is used to highlight near-random performance. The F1 score — the harmonic mean of precision and recall — is appropriate for tasks with potentially imbalanced answer distributions (e.g., COPA alternatives, BoolQ yes/no labels, SentiNeg positive/negative classes). For WiC, the paper explicitly notes that "all models demonstrated random performance in this task" and switches to accuracy "as it provided a more straightforward evaluation of performance, particularly for random performance" (Appendix B). This choice makes the chance-level baseline (0.50 for binary classification) immediately interpretable. Answers are evaluated using the lm-evaluation-harness framework (Gao et al., 2021), specifically the "polyglot" branch adapted for Korean evaluation, which standardizes prompt formatting, few-shot example selection, and metric computation across all models.
Baselines. Three comparison models are used:
- ko-gpt-trinity-1.2B (SKT): a Korean autoregressive language model at 1.2B parameters, directly comparable to Polyglot-Ko-1.3B.
- KoGPT-6.0B (KakaoBrain, Kim et al., 2021): a Korean autoregressive language model at 6.0B parameters, directly comparable to Polyglot-Ko-5.8B.
- XGLM-7.5B (Lin et al., 2022): Meta's multilingual autoregressive language model at 7.5B parameters, representing the performance of a large multilingual model on Korean tasks. XGLM is the critical baseline for the paper's central claim — that a monolingual Korean model outperforms a larger multilingual model on Korean-language tasks.
Generation budget / compute accounting. This paper does not use a "generation budget" framework like the test-time compute scaling literature. Instead, all models are evaluated in a fixed few-shot prompting setup: the model receives $n$ in-context examples (where $n \in \{0, 5, 10, 50\}$ ) followed by a test instance, and produces a single greedy or temperature-sampled prediction. There is no search, no multiple sampling, and no verifier-guided selection. The "compute" in this evaluation is the forward pass through the model for each test instance — a cost that scales with model size but is not varied as an experimental axis. The paper does not report inference FLOPs or wall-clock time, and does not compare models under equal-compute constraints. The comparison is strictly accuracy-at-scale: how well does each model perform given its parameter count and (implicitly) its pretraining compute budget?
Cross-validation / statistical protocol. None reported. The paper does not describe any cross-validation, bootstrap confidence intervals, statistical significance testing, or multiple-run averaging. Results are reported as point estimates (single F1 scores per model, per task, per shot count). The 500-instance test set for KOBEST (if the standard split is used) provides reasonable per-task sample sizes, but the absence of any variance estimates means that small differences between models (e.g., Polyglot-Ko-5.8B vs. KoGPT-6.0B on HellaSwag 5-shot: 0.600 vs. 0.583) cannot be assessed for statistical reliability. The paper uses the polyglot branch of lm-evaluation-harness, which standardizes evaluation but does not itself provide uncertainty quantification unless explicitly configured.

Main Quantitative Results

COPA (Causal Reasoning)

The COPA results are presented in the left portion of Table 3 and visualized in Figure 1 (top left). The headline finding is that Polyglot-Ko-12.8B achieves the highest F1 scores across all shot settings, with a 0-shot score of 0.794, a 5-shot score of 0.811, a 10-shot score of 0.804, and a 50-shot score of 0.837. This represents the best performance on this task among all evaluated models in every shot regime.

The scaling trend within the Polyglot-Ko family is monotonic and clear: for 0-shot COPA, scores progress from 0.720 (1.3B) → 0.760 (3.8B) → 0.775 (5.8B) → 0.794 (12.8B). Each parameter increase yields a consistent improvement, with the largest absolute jump occurring between 1.3B and 3.8B (+0.040), and diminishing returns thereafter. At 50-shot, the progression is 0.721 → 0.779 → 0.789 → 0.837, with the 12.8B model showing a substantially larger benefit from additional in-context examples (+0.043 over 5.8B at 50-shot vs. +0.019 at 0-shot), suggesting that larger scale confers an enhanced ability to leverage few-shot examples for causal reasoning in Korean.

The comparison against external models is favorable across the board. Polyglot-Ko-1.3B outperforms the similarly-sized ko-gpt-trinity-1.2B at all shot settings: 0.720 vs. 0.670 (0-shot), 0.721 vs. 0.651 (50-shot). The gap is substantial — roughly 7 percentage points in 0-shot — and consistent, indicating that Polyglot-Ko's training data or tokenization provides an advantage over SKT's model even at matched scale. Polyglot-Ko-5.8B outperforms the larger KoGPT-6.0B at all shot settings (0-shot: 0.775 vs. 0.735; 50-shot: 0.789 vs. 0.748), though the gap narrows at higher shot counts. Polyglot-Ko-1.3B outperforms the much larger XGLM-7.5B by substantial margins (0-shot: 0.720 vs. 0.672; 50-shot: 0.721 vs. 0.712), directly supporting the paper's central claim that monolingual Korean training beats multilingual training for Korean-language tasks, even at a ~5.8× parameter disadvantage for the smallest Polyglot-Ko variant.

A notable pattern: XGLM-7.5B shows minimal improvement from few-shot examples on COPA (0.672 → 0.673 → 0.677 → 0.712), while all Korean-specific models benefit substantially from additional in-context examples. This suggests that XGLM's Korean in-context learning capability is fundamentally weaker — it struggles to extract useful patterns from Korean-language examples, likely because its representations of Korean are less structured and less amenable to pattern extraction from prompts.

HellaSwag (Commonsense Reasoning)

The HellaSwag results appear in the right portion of Table 3 and Figure 1 (top right). The headline is more nuanced than COPA: Polyglot-Ko-12.8B achieves the best scores in the 5-shot (0.631), 10-shot (0.610), and 50-shot (0.612) settings, but is outperformed in 0-shot by the 5.8B model (0.598 vs. 0.595) — a marginal difference. In the 50-shot setting, Polyglot-Ko-5.8B actually achieves the highest score (0.621), slightly surpassing the 12.8B model (0.612).

The scaling progression within Polyglot-Ko on HellaSwag is less clean than on COPA. For 0-shot: 0.525 (1.3B) → 0.571 (3.8B) → 0.598 (5.8B) → 0.595 (12.8B) — the 12.8B model plateaus relative to the 5.8B. For 5-shot: 0.526 → 0.583 → 0.600 → 0.631 — here the 12.8B shows a clear advantage. For 50-shot: 0.543 → 0.579 → 0.621 → 0.612 — the 12.8B model actually degrades relative to the 5.8B model when given the maximum number of in-context examples. This non-monotonic scaling behavior is unusual and not discussed by the authors. Possible explanations include: (1) the 12.8B model, being undertrained (167B tokens vs. a Chinchilla-optimal ~256B), has not fully converged its higher-level reasoning capabilities, while the 5.8B model (172B tokens on fewer parameters, closer to compute-optimal) has better-learned representations for commonsense tasks; or (2) the HellaSwag task's adversarial distractors are more sensitive to specific training data patterns, and the 12.8B model's additional capacity leads to overthinking — preferring artificially plausible distractors — in a way that additional in-context examples do not correct.

Against external models: Polyglot-Ko-1.3B (0.525 0-shot) and ko-gpt-trinity-1.2B (0.524) are essentially tied at 0-shot, with Polyglot-Ko pulling slightly ahead at 50-shot (0.543 vs. 0.535). Polyglot-Ko-5.8B and KoGPT-6.0B are broadly comparable, with Polyglot-Ko ahead at 0-shot (0.598 vs. 0.559) and 50-shot (0.621 vs. 0.591), but KoGPT slightly ahead at 10-shot (0.583 vs. 0.598). XGLM-7.5B underperforms the Korean-specific models — its 0-shot (0.566) is better than Polyglot-Ko-1.3B (0.525) but well below Polyglot-Ko-3.8B (0.571), and its 50-shot (0.562) trails all Polyglot-Ko models except the 1.3B variant. HellaSwag is the task where the multilingual model comes closest to competing — the advantage of Korean-specific training is smaller here than on COPA or SentiNeg, perhaps because commonsense reasoning has more universal elements that transfer across languages, or because the HellaSwag format (selecting plausible completions) is less dependent on Korean-specific linguistic knowledge.

BoolQ (Boolean Question Answering)

Results are in the left portion of Table 4 and Figure 1 (bottom right). Polyglot-Ko-12.8B achieves the highest F1 scores at all shot settings: 0-shot: 0.482, 5-shot: 0.604, 10-shot: 0.629, 50-shot: 0.645. The improvement from scaling is dramatic — the 12.8B model's 50-shot performance (0.645) is nearly double the 1.3B model's 50-shot (0.404), representing a 0.241 absolute improvement or roughly a 60% relative gain.

BoolQ shows the strongest evidence for scale-dependent few-shot learning within the Polyglot-Ko family. The 1.3B model actually degrades at 50-shot (0.404) compared to 5-shot (0.475), suggesting that for small models, too many in-context examples confuse rather than help — a known phenomenon in few-shot learning where smaller models struggle to maintain coherent attention over long prompts. The 3.8B model shows a similar pattern (5-shot: 0.526; 50-shot: 0.404, a sharp decline). The 5.8B model is more stable (5-shot: 0.570; 50-shot: 0.524 — a modest decline). Only the 12.8B model shows consistent improvement as shot count increases (0.482 → 0.604 → 0.629 → 0.645), demonstrating that the capacity to effectively use long in-context prompts for boolean reasoning emerges only at larger scales.

Against external models: Polyglot-Ko-12.8B outperforms KoGPT-6.0B at all shot settings, with the largest gaps at 10-shot (0.629 vs. 0.550) and 50-shot (0.645 vs. 0.520). This is a substantial margin — roughly 12–13 percentage points — and suggests that Polyglot-Ko's training data or architecture provides particular advantages for Korean reading comprehension. Polyglot-Ko-1.3B (0-shot: 0.355) outperforms ko-gpt-trinity-1.2B (0.336) at 0-shot and 5-shot, but the pattern reverses at higher shot counts where both small models struggle. XGLM-7.5B shows a bizarre pattern: it achieves 0.446 at 0-shot (second only to Polyglot-Ko-12.8B), but then collapses to 0.332 at 5-shot, 10-shot, and 50-shot — identical scores across all few-shot settings. This is suspicious and suggests a formatting or parsing failure in the evaluation harness for XGLM at shot counts above 0. The uniformity of the collapse (exactly 0.332 at three different shot settings) is a strong indicator that XGLM was not actually performing the BoolQ task in these settings — it may have been producing outputs that the evaluation harness could not parse into valid yes/no answers, defaulting to a fixed score. This makes the BoolQ comparison against XGLM unreliable for shot counts above 0.

SentiNeg (Sentiment Analysis with Negation)

The default-prompt results are in the left portion of Table 4 and Figure 1 (bottom left); modified-prompt results are in the left portion of Table 5 and Figure 3a. The paper presents both because the default prompt introduced "significant instability, particularly in zero-shot performance" (Section 4.4). I discuss both, as the comparison reveals important evaluation dynamics.

With the default prompt (Table 4), Polyglot-Ko-12.8B achieves the highest F1 scores at all shot settings: 0-shot: 0.912, 5-shot: 0.902, 10-shot: 0.934, 50-shot: 0.972. This is the highest absolute performance on any task in the paper — the 12.8B model is near ceiling on SentiNeg even with zero examples. However, the Polyglot-Ko family shows erratic scaling behavior with the default prompt. The 1.3B model scores 0.679 (0-shot), 0.626 (5-shot), 0.551 (10-shot), 0.785 (50-shot) — performance drops as shot count increases from 0 to 10, then recovers dramatically at 50-shot. The 3.8B model is even more erratic: 0.486 (0-shot), 0.795 (5-shot), 0.732 (10-shot), 0.785 (50-shot). The 5.8B model shows extreme variance: 0.339 (0-shot) → 0.884 (5-shot) → 0.881 (10-shot) → 0.952 (50-shot) — a 0.545 jump from 0-shot to 5-shot. These wild swings strongly suggest that the default prompt was not reliably eliciting SentiNeg capability from the models, especially at intermediate shot counts, and that the zero-shot setting in particular was highly sensitive to minor phrasing differences.

With the modified prompt (Table 5, left), the instability largely disappears. Polyglot-Ko-12.8B achieves: 0-shot: 0.893, 5-shot: 0.985, 10-shot: 0.982, 50-shot: 0.982. These scores are extremely high — the 5-shot and above results are near-perfect (F1 > 0.98), suggesting the task is essentially solved for the 12.8B model. The scaling behavior becomes cleaner: 1.3B (0-shot: 0.889, 50-shot: 0.907), 3.8B (0.942, 0.952), 5.8B (0.878, 0.960), 12.8B (0.893, 0.982). The 5.8B model's 0-shot score (0.878) is actually higher than the 12.8B's (0.893) — a small reversal — but the 12.8B pulls ahead decisively at 5-shot and beyond. The 3.8B model notably achieves the highest 0-shot score (0.942), outperforming both larger variants, which suggests that zero-shot SentiNeg performance is not a simple function of scale — the 3.8B model's particular training dynamics may have produced better representations for negation detection in the absence of in-context examples.

Against external models: KoGPT-6.0B is highly competitive with Polyglot-Ko-12.8B on the modified SentiNeg prompt, achieving 0.927 (0-shot), 0.955 (5-shot), 0.955 (10-shot), and 0.952 (50-shot). These scores are within 2–3 percentage points of the 12.8B model at higher shot counts, and KoGPT's 0-shot (0.927) actually exceeds Polyglot-Ko-12.8B's 0-shot (0.893). This is the one task where an external Korean model matches or exceeds Polyglot-Ko's performance, suggesting that KakaoBrain's training approach was particularly effective for sentiment-related Korean language understanding. ko-gpt-trinity-1.2B performs substantially worse (0-shot: 0.767, 50-shot: 0.836), trailing all Polyglot-Ko variants. XGLM-7.5B is consistently weak (0-shot: 0.835, 50-shot: 0.727 — note the decline with more shots), reinforcing the pattern that multilingual models struggle with Korean sentiment analysis, likely because negation markers and sentiment expressions in Korean are morphologically encoded in ways that XGLM's tokenizer and training data do not adequately capture.

The SentiNeg results collectively demonstrate two things: (1) at sufficient scale, Korean-specific models can achieve near-ceiling performance on sentiment analysis with negation, and (2) prompt design matters enormously for eliciting this capability in few-shot settings, with the modified prompt producing substantially higher and more stable results than the default KOBEST prompt, particularly at intermediate shot counts.

WiC (Word-in-Context Disambiguation)

Results are in the right portion of Table 5 and Figure 3b. The headline is simple: all models perform at random on WiC. Accuracy scores cluster tightly around the 0.50 chance baseline: Polyglot-Ko-1.3B: 0.489 (0-shot), 0.486 (5-shot), 0.506 (10-shot), 0.487 (50-shot). Polyglot-Ko-12.8B: 0.493, 0.494, 0.488, 0.487. KoGPT-6.0B: 0.484, 0.495, 0.479, 0.475. XGLM-7.5B: 0.488, 0.490, 0.498, 0.514 (slightly above chance at 50-shot, but still effectively random). There is no trend with model scale, no trend with shot count, and no model scores accuracy above 0.514 — barely distinguishable from the 0.50 random baseline.

This is a clear negative result. The paper's decision to report WiC in the appendix rather than the main text, and to switch the metric from F1 to accuracy "to show random performance" (Appendix B), is transparent and appropriate. The WiC failure indicates that word-sense disambiguation — determining whether a polysemous Korean word like "배" (pear/ship/stomach) is used in the same sense across two contexts — is an unsolved problem for models in this parameter range, regardless of whether they are Korean-specific or multilingual. The failure occurs because WiC requires fine-grained lexical-semantic knowledge that next-token prediction pretraining at 167–219B tokens does not adequately encode: the model must compare two contextualized representations of the same word form and judge whether they refer to the same underlying sense. This is a fundamentally different capability from the causal reasoning, commonsense inference, or sentiment detection tested by the other KOBEST tasks, and it appears to require either more targeted training data (e.g., word sense disambiguation corpora), larger scale, or architectural modifications beyond what the Polyglot-Ko models provide.

Scaling Trends Within the Polyglot-Ko Family

Figure 2 aggregates 5-shot performance across all four tasks for the Polyglot-Ko models only, plotting F1 score against model scale. The paper describes a "clear trend that as the compute increases, the performance improves" (Section 4.4). This claim is partially supported:

COPA: F1 increases from ~0.72 (1.3B) to ~0.81 (12.8B) — a clear, monotonic improvement.
HellaSwag: F1 increases from ~0.53 (1.3B) to ~0.63 (12.8B) — also monotonic in the 5-shot setting.
SentiNeg: F1 increases from ~0.85 (1.3B) to ~0.99 (12.8B) with the modified prompt — dramatic improvement, but note the 5.8B model's score (0.927) is intermediate, and the 3.8B model (0.894) is actually below the 5.8B, so the scaling is not perfectly monotonic.
BoolQ: F1 increases from ~0.48 (1.3B) to ~0.60 (12.8B) in 5-shot — clear improvement, though the 5.8B (0.570) and 12.8B (0.604) are close.

The overall trend is real: larger models perform better on average. However, the non-monotonicities (5.8B outperforming 12.8B on HellaSwag 50-shot; 3.8B outperforming 5.8B on SentiNeg 0-shot modified) and the small gap between 5.8B and 12.8B on several tasks suggest that the scaling curve may be approaching diminishing returns in the 6–13B parameter range, at least given the training token budgets used (which, as noted in Section 3, leave the 12.8B model significantly undertrained). A fully Chinchilla-optimal 12.8B model (trained on ~256B tokens rather than 167B) might show larger gains over the 5.8B variant.

Ablation Studies and Robustness Checks

This paper does not contain formal ablation studies in the conventional sense. There are no controlled experiments where a single component (tokenizer, architecture choice, training data source, hyperparameter) is varied while holding all else constant. The paper's "ablation" information comes instead from natural variation across the four model sizes and the comparison with external models, which provides some indirect evidence about the importance of specific design choices.

Morpheme-aware tokenization (indirect evidence): The paper does not train a Polyglot-Ko variant with a standard (non-morpheme-aware) BPE tokenizer and compare performance. The only evidence for the value of morpheme-aware tokenization comes from the comparison with XGLM-7.5B, which uses a multilingual tokenizer not optimized for Korean morphology. Polyglot-Ko-1.3B substantially outperforms XGLM-7.5B on most tasks despite the latter's ~5.8× parameter advantage — but this comparison confounds tokenizer design with training data language composition (monolingual Korean vs. multilingual) and data quality (curated Korean corpus vs. English-centric web crawl). The ablation one would want — "Polyglot-Ko-1.3B with standard BPE vs. Polyglot-Ko-1.3B with morpheme-aware BPE, all else equal" — does not exist.

Model scale (natural variation): The four Polyglot-Ko sizes serve as an informal scaling study, but because training token budgets, batch sizes, and architectural parameters (layer count, head dimension, model width) all vary across sizes, the scaling curves in Figure 2 conflate model capacity with training duration and architecture. A proper scaling law analysis — holding architecture type constant (e.g., constant aspect ratio) and training to equal Chinchilla-optimality — is not possible with this data.

Batch size and training stability (natural variation): The observation that the 1.3B and 3.8B models (batch size 1024) experienced epoch-boundary generation collapse while the 5.8B and 12.8B models (batch size 256) did not is a natural experiment, but it is not a controlled ablation — the models differ in scale, architecture, and training token count simultaneously with the batch size change. The finding that smaller batch sizes may mitigate training instability near epoch boundaries is suggestive but not conclusively demonstrated.

Prompt design (explicit variation): The modified vs. default SentiNeg prompt comparison is the closest the paper comes to an ablation. The results (Table 4 default vs. Table 5 modified) show that prompt modification improves zero-shot performance from erratic scores (as low as 0.339 for 5.8B, 0.486 for 3.8B) to high and stable scores (0.878+ for all models above 1.3B). This is a robustness check on the evaluation methodology, not on the model itself — it demonstrates that the SentiNeg capability exists in the models and can be elicited with appropriate prompting, but that the default KOBEST prompt is unreliable for zero-shot evaluation. This is an important negative result for the benchmark: it indicates that KOBEST SentiNeg scores reported with the default prompt may not accurately reflect model capability, especially in low-shot regimes.

Data preprocessing error (acknowledged, not ablated): The paper discloses that "the data was incorrectly stripped of newlines, leading to a loss of document structure" (Section 5). There is no ablation comparing the released model to a version trained with correctly preserved newlines, so the impact of this error on downstream task performance is unknown. The fact that the models achieve strong results despite this error suggests that document structure (paragraph breaks, section boundaries) may be less critical for the KOBEST tasks — which focus on sentence-level and short-passage understanding — than it would be for long-form generation tasks. This is a robustness check in reverse: the models are surprisingly robust to a known preprocessing flaw, but the limits of that robustness are untested.

Critical Assessment

The paper's central claims — as articulated in the Abstract and Section 1 — can be evaluated against the experimental evidence as follows:

Claim: Polyglot-Ko achieves "competitive results across various benchmark datasets" and "our 12.8B model outperforms the other models across all scenarios" on COPA (Section 4.1).

This claim is supported by the reported numbers but requires careful qualification. On COPA, Polyglot-Ko-12.8B does outperform all comparison models (ko-gpt-trinity, KoGPT, XGLM) at all shot settings in Table 3. The margins are substantial: 0.794 vs. 0.735 (KoGPT-6.0B, 0-shot), 0.837 vs. 0.748 (KoGPT-6.0B, 50-shot). However, "across all scenarios" is an overstatement when looking at the full set of tasks. On HellaSwag 50-shot, Polyglot-Ko-5.8B (0.621) outperforms Polyglot-Ko-12.8B (0.612) — the largest model is not the best in that specific setting. On SentiNeg 0-shot (modified prompt), Polyglot-Ko-3.8B (0.942) outperforms Polyglot-Ko-12.8B (0.893), and KoGPT-6.0B (0.927) also outperforms the 12.8B model. On BoolQ 5-shot, KoGPT-6.0B (0.598) slightly edges Polyglot-Ko-5.8B (0.570). The 12.8B model is the best overall, but it does not sweep every task-shot combination. The more precise statement is: the 12.8B model achieves the highest or near-highest scores on most tasks, with occasional reversals against the 5.8B model and KoGPT-6.0B that are small in magnitude but real.

Claim: The models demonstrate that "language-specific pretraining provides a substantial advantage over multilingual approaches" (implied by the comparison with XGLM).

This claim is well-supported but the BoolQ and WiC results introduce important nuance. On COPA, Polyglot-Ko-1.3B (the smallest monolingual model) outperforms XGLM-7.5B (5.8× larger) by 7 percentage points in 0-shot. On SentiNeg with the modified prompt, Polyglot-Ko-1.3B (0.889) outperforms XGLM-7.5B (0.835). These are clean comparisons — same task, same evaluation protocol, smaller monolingual model beats much larger multilingual model. On HellaSwag, the advantage is smaller but consistent. However, on BoolQ at 0-shot, XGLM-7.5B (0.446) actually outperforms Polyglot-Ko-1.3B (0.355), and on WiC, all models perform at random, so no advantage can be claimed. The BoolQ exception is important because it shows that the monolingual advantage is task-dependent: for tasks requiring factual knowledge and reading comprehension (BoolQ), XGLM's larger parameter count and broader training data may partially compensate for its Korean-language disadvantage. For tasks requiring Korean-specific linguistic knowledge (causal reasoning via Korean verbal endings, sentiment analysis with Korean negation morphology), the monolingual advantage is decisive. The claim should be qualified as: monolingual Korean pretraining provides a substantial advantage for tasks that require Korean-specific linguistic processing, but this advantage narrows or reverses for tasks where general reasoning capacity and factual knowledge are more important, particularly at 0-shot where in-context learning cannot compensate.

A significant weakness in this comparison is the unreliability of XGLM's BoolQ results for shot counts above 0. The identical 0.332 score at 5-shot, 10-shot, and 50-shot suggests an evaluation failure — XGLM may not have been producing parseable outputs. This means the 0-shot comparison (where XGLM achieves 0.446, competitive with Polyglot-Ko models) is the only reliable BoolQ data point for the multilingual comparison, and it shows XGLM outperforming the smallest Polyglot-Ko model. This weakens the paper's claim of universal monolingual superiority.

Claim: The 12.8B model is "the largest publicly available Korean language model suitable for commercial applications" (Section 1).

This claim is about availability and licensing, not performance, and the paper provides no experimental evidence related to it. It is a factual claim about the model landscape at the time of release. I do not assess it against the experiments.

Claim: "Performance improves with increasing compute" (Section 4.4, Figure 2).

This claim is broadly supported but overstated in its simplicity. Figure 2 shows a generally upward trend in 5-shot F1 scores as model size increases from 1.3B to 12.8B across four tasks. However, the curves are not monotonic on close inspection: the 5.8B → 12.8B step on HellaSwag shows a tiny gain (from roughly 0.60 to 0.63 — the actual numbers are 0.600 at 5-shot for 5.8B and 0.631 for 12.8B, a 3 percentage point improvement for a ~2.2× parameter increase), and on SentiNeg the 3.8B model (0.894 modified 5-shot) actually trails the 5.8B model (0.927) but the 1.3B model (0.850) is closer to the 3.8B than parameter count alone would suggest. The scaling is real but noisy, and the phrase "improves with increasing compute" implies a cleaner relationship than the data supports.

More importantly, the paper conflates "compute" (total FLOPs used in training) with "model size" (parameter count). The four models were trained with different token budgets, different batch sizes, and different architectures — their total training FLOPs are not a simple function of parameter count. The 1.3B model (213B tokens) used approximately $6 \times 1.33 \times 10^9 \times 213 \times 10^9 \approx 1.7 \times 10^{21}$ FLOPs. The 12.8B model (167B tokens) used approximately $6 \times 12.9 \times 10^9 \times 167 \times 10^9 \approx 1.29 \times 10^{22}$ FLOPs — roughly 7.6× more FLOPs for the larger model, but with significant undertraining. The relationship between "compute" (training FLOPs) and performance is confounded with the relationship between "scale" (parameters) and performance, and this is not cleanly separable in the reported results. A cleaner scaling analysis would require either holding training tokens constant across model sizes, or training all models to Chinchilla-optimality, or reporting performance as a function of total FLOPs rather than parameter count.

Missing experiments that would have strengthened the paper:

Comparison against fine-tuned models: All results are zero-shot or few-shot. How do Polyglot-Ko models perform when fine-tuned on KOBEST tasks? This would provide an upper bound on task performance and indicate whether the few-shot gap between Polyglot-Ko and external models persists after task-specific adaptation.
Comparison against larger multilingual models at matched Korean data volume: XGLM-7.5B is the multilingual baseline, but its Korean training data volume is unknown. A comparison where the multilingual model is trained on the same 863 GB of Korean data (perhaps as a continued pretraining experiment) would isolate the effect of language-specific vs. multilingual architecture from the effect of training data quantity.
Ablation on the tokenizer: Train a Polyglot-Ko-1.3B with standard Byte-Level BPE (no MeCab morphological preprocessing) and compare against the morpheme-aware version on KOBEST. This would quantify the contribution of the tokenizer design choice.
Ablation on data composition: Train a variant on the 682 GB of blog data only (the dominant source) and compare against the full 863 GB corpus to assess whether the diverse sources beyond blogs contribute meaningfully to benchmark performance.
Evaluation on generation tasks: KOBEST is entirely discriminative (classification, multiple choice). Evaluating on Korean generative tasks (summarization, translation, open-ended QA) would provide a more complete picture of model capability, especially given that the authors note potential issues with document structure (newline stripping) that would most affect generation quality.
Confidence intervals: The complete absence of any variance estimates makes it impossible to assess whether differences between models (e.g., 0.600 vs. 0.583 on HellaSwag 5-shot between Polyglot-Ko-5.8B and KoGPT-6.0B) are statistically meaningful or within sampling noise.

Overall assessment: The experimental results support the paper's primary message — that purpose-built Korean monolingual models, trained on a carefully curated Korean corpus with a Korean-optimized tokenizer, achieve strong performance on Korean-language understanding benchmarks and generally outperform comparably sized multilingual alternatives. The evidence for this is clear on COPA and SentiNeg, moderate on HellaSwag, and mixed on BoolQ (where XGLM's 0-shot score is competitive). The WiC results demonstrate a clear capability ceiling that no model in this scale range overcomes. The experiments do not cleanly isolate why Polyglot-Ko works — the contributions of data curation, tokenizer design, model architecture, and scale are all confounded. The paper is best understood as a model release and benchmark report that demonstrates the viability of the monolingual-first approach for Korean; it is not a controlled study that identifies which specific design decisions matter most. For the Korean NLP practitioner deciding whether to use Polyglot-Ko, these results provide strong evidence that the models work well on the tested tasks. For the researcher trying to understand how to build better non-English models, the paper provides suggestive evidence but leaves key ablations to future work.

6. Limitations and Trade-offs

6.1 The Models Are Significantly Undertrained Relative to Compute-Optimal Scaling Laws

The assumption or constraint. The Polyglot-Ko models were trained with variable and, for the larger models, substantially suboptimal token budgets due to computational resource constraints. The paper states directly: "Due to the limitations of available computational resources, the models were trained with varying numbers of tokens" (Section 3). The 1.3B model received 213B tokens (~164 tokens per parameter), the 3.8B received 219B tokens (~57 tokens per parameter), the 5.8B received 172B tokens (~29 tokens per parameter), and the 12.8B received 167B tokens (~13 tokens per parameter). By the Chinchilla scaling law (Hoffmann et al., 2022), which prescribes approximately 20 tokens per parameter for compute-optimal training, the 12.8B model is significantly undertrained — it would require roughly 256B tokens to reach compute-optimality, meaning it received only about 65% of the optimal token budget.

The consequence. The performance of the 12.8B model relative to the 5.8B and 3.8B models is not a clean measure of what scale alone provides for Korean language understanding. The 12.8B model's untapped capacity means that (a) the gap between it and smaller models is smaller than it would be with full training, and (b) the scaling trends observed in Figure 2 and Section 4 — where the 12.8B model sometimes barely outperforms the 5.8B (HellaSwag 50-shot: 0.612 vs. 0.621, where the smaller model wins) or shows non-monotonic behavior — may reflect undertraining rather than genuine diminishing returns from scale. A practitioner considering whether to use the 12.8B model versus the 5.8B cannot assess the true value of the additional parameters because they are comparing models at different points on their respective training trajectories. The 1.3B model, conversely, is massively overtrained relative to Chinchilla-optimality (~8× the optimal token count), which means the performance gap between it and the larger models may be smaller than it would be if all models were trained to compute-optimality — the 1.3B model benefits from extensive training that the larger models did not receive.

What evidence exists in the paper. The non-monotonic scaling behavior is visible across multiple task-shot combinations: HellaSwag 50-shot (5.8B: 0.621 vs. 12.8B: 0.612), SentiNeg 0-shot with modified prompt (3.8B: 0.942 vs. 12.8B: 0.893), and BoolQ 5-shot (5.8B: 0.570 vs. 12.8B: 0.604 — a small gain for a ~2.2× parameter increase). The paper explicitly acknowledges the variable token budgets (Section 3) and the computational limitation that caused them, but does not quantify how much performance is left on the table for the larger models. There is no Chinchilla-optimal version of any model to compare against, so the magnitude of the undertraining penalty is unknown from this paper alone.

Mitigation status. The paper does not attempt to address this limitation — it is an inherent constraint of the available compute budget. The authors are transparent about it, which is appropriate for a technical report. The consequence for practitioners is that the 12.8B model should be viewed as a lower bound on what a properly trained Korean model at that scale could achieve. For researchers studying scaling laws for non-English languages, the confound between scale and training budget makes Polyglot-Ko unsuitable as a clean scaling benchmark without additional controlled experiments.

6.2 The Training Instability Near Epoch Boundaries Is Undiagnosed and Could Recur in Derivative Work

The assumption or constraint. The 1.3B and 3.8B models both experienced a catastrophic degradation in generation quality near the first epoch boundary (~100,000 training steps at batch size 1,024), characterized by "broken generation (e.g generate same tokens repeatedly)" that occurred "as the loss sharply dropped" (Section 3). The paper explicitly states: "We suspect it was overtrained or overfitting, but there are not any scientific evidence" (Section 3, footnote 4). The 5.8B and 12.8B models, trained with a 4× smaller batch size (256), did not exhibit this collapse, but the causal mechanism — whether batch size, model scale, architecture (the 5.8B is shallower-and-wider), or some interaction — is unknown.

The consequence. A practitioner who fine-tunes or continues training Polyglot-Ko models — especially the 1.3B and 3.8B variants — cannot predict whether the epoch-boundary collapse will recur. If a downstream user continues training on new Korean data and approaches a full epoch over their dataset, they may encounter the same generation quality degradation with no clear mitigation strategy beyond "monitor generation quality and early-stop if it degrades." More broadly, the undiagnosed nature of the collapse means that the Korean NLP community does not know whether this is a general property of training autoregressive LMs on Korean text (perhaps related to data distribution, tokenizer properties, or morphological complexity), or specific to the GPT-NeoX implementation with these hyperparameters on this particular corpus. This limits the reusability of the training methodology for building future Korean models.

What evidence exists in the paper. The paper reports the collapse for the 1.3B and 3.8B models (Section 3), notes its absence for the 5.8B and 12.8B models (which used different batch sizes), and includes the candid footnote admitting the lack of scientific evidence for the cause. There is no analysis of loss curves, gradient norms, activation statistics, or other diagnostic metrics that might shed light on the mechanism. There is no controlled experiment varying batch size while holding model size constant, which would be the minimum needed to attribute the collapse to batch size versus model scale.

Mitigation status. The paper's mitigation is purely pragmatic: "the decision was made to select and early stop the model checkpoints prior to the epoch boundary, as they exhibited better generation performance" (Section 3). This produces usable models but does not solve the underlying problem. The paper does not propose diagnostic experiments for future work, nor does it provide guidance to practitioners on how to detect or avoid the collapse beyond what they themselves did (early stopping). The lack of root-cause analysis means this limitation carries forward to anyone who builds on these models or this training methodology.

6.3 Evaluation Is Limited to a Single Discriminative Benchmark Suite with No Generative Assessment

The assumption or constraint. All evaluation in the paper uses the KOBEST benchmark, which consists entirely of discriminative tasks: multiple-choice selection (COPA, HellaSwag), binary classification (BoolQ, SentiNeg, WiC). The paper states that "Polyglot-Ko has been primarily trained to optimize next token prediction, which makes it suitable for a wide range of tasks" (Section 5), but the evaluation does not include any open-ended generative tasks — no summarization, no translation, no question answering with free-form generation, no dialogue, no long-form text completion.

The consequence. The evaluation tells a practitioner how well Polyglot-Ko discriminates between pre-defined answer choices, but says nothing about its ability to produce fluent, coherent, factually correct Korean text — which is the primary use case for an autoregressive language model. The preprocessing error disclosed in Section 5 — that "the data was incorrectly stripped of newlines, leading to a loss of document structure" — is particularly concerning for generative use cases. A model trained on text with all paragraph breaks and section boundaries removed is likely to produce long, undifferentiated blocks of text when asked to generate Korean documents, because it never learned that newlines carry structural meaning. This limitation is especially acute for applications like email generation, report writing, or dialogue systems where document formatting and structure are important. The paper provides no evidence about whether this structural degradation affects generation quality, and a practitioner considering Polyglot-Ko for a generation task has no benchmark data to inform their decision.

What evidence exists in the paper. The paper acknowledges the newline stripping error (Section 5) and the lack of generative evaluation implicitly — the evaluation section (Section 4) covers only KOBEST discriminative tasks. The paper does not report any generative metrics (perplexity on held-out Korean text, BLEU/ROUGE for summarization, human evaluation of generation quality, or even qualitative generation examples). The WiC task, where all models perform at random (Table 5, right), indirectly hints at a generative limitation — word-sense disambiguation is a prerequisite for generating text with the correct sense of polysemous words — but this is not explored.

Mitigation status. Not addressed. The paper notes the newline stripping as something "to address this issue in future iterations to ensure the preservation of document structure and minimize information loss" (Section 5), but does not evaluate the current models' generation quality or provide generation examples that would allow users to assess the impact. The "wide range of tasks" claim in Section 5 is unsupported by evidence from generative evaluation. A practitioner must either assume generation quality is adequate (based on the strong discriminative results) or conduct their own generation evaluation before deploying the model.

6.4 The Data Preprocessing Does Not Adequately Address Hate Speech and Toxic Content

The assumption or constraint. The paper's data analysis in Section 2.1 identifies "a significant presence of hate speech in datasets sourced from certain community websites" as one of four data categories requiring attention. However, the preprocessing pipeline described in the same section does not include hate speech filtering or toxicity classification — it focuses on structural issues (empty text, HTML tags, deduplication, short text, repeated characters, broken code, PII removal, unnecessary spaces). Section 5 (Limitations) acknowledges the consequence: "Polyglot-Ko may generate content that is socially unacceptable or offensive. To mitigate this risk, we strongly recommend implementing a human curator or employing other filtering mechanisms to censor sensitive or inappropriate content."

The consequence. A practitioner deploying Polyglot-Ko in a user-facing application — particularly a chatbot, content generation tool, or comment moderation system — faces a nontrivial risk that the model will produce hate speech, derogatory language, or offensive content when prompted in certain ways. The recommendation to use "a human curator or other filtering mechanisms" places the burden of safety on the downstream user, without providing any safety evaluation, toxicity benchmarks, or guidance on what kinds of offensive content the model is likely to produce. For commercial applications — which the paper explicitly targets by emphasizing that the 12.8B model is "suitable for commercial applications" (Section 1) — this is a significant gap. Many organizations considering deployment have safety requirements that cannot be met by an unmoderated model with unknown toxicity characteristics.

What evidence exists in the paper. The paper reports observing hate speech in the training data (Section 2.1) and acknowledges the generation risk (Section 5), but provides no quantitative measurement of the problem: no toxicity scores on standard Korean hate speech benchmarks (despite including a Korean hate speech dataset in the training sources — Table 1), no red-teaming results, no examples of problematic outputs, and no comparison of Polyglot-Ko's toxicity against other Korean or multilingual models. The hate speech dataset listed in Table 1 (< 1.0 GB) was apparently included in training rather than used for evaluation — meaning the model may have been exposed to hate speech examples without any corresponding mitigation training.

Mitigation status. The paper suggests downstream filtering as a mitigation (Section 5) but does not implement it or evaluate its effectiveness. This is an acknowledged but unaddressed limitation. The paper does not propose future work on safety-specific fine-tuning, toxicity benchmarking, or controlled generation techniques for Polyglot-Ko. For a model released for public and commercial use, this places an unusually heavy burden on users to discover and mitigate safety issues themselves.

6.5 The WiC Failure Reveals a Hard Capability Ceiling That Scale Alone Does Not Overcome

The assumption or constraint. The Word-in-Context (WiC) task requires determining whether a polysemous Korean word is used in the same sense across two different contexts — a task that demands fine-grained lexical-semantic knowledge. The paper reports (Appendix B, Table 5 right) that all models, regardless of scale or training regime (Korean-specific or multilingual), perform at random on WiC: Polyglot-Ko-12.8B achieves 0.493 accuracy at 0-shot, 0.494 at 5-shot, and 0.487 at 50-shot — indistinguishable from the 0.50 chance baseline. No model exceeds 0.514 (XGLM-7.5B at 50-shot, which is still effectively random given the sample size). The paper acknowledges this by excluding WiC from the main results and switching the metric to accuracy "to show random performance" (Appendix B).

The consequence. WiC represents a class of tasks that require comparing contextualized representations of the same surface form — a fundamentally different capability from the causal reasoning, commonsense inference, or sentiment detection tested by the other KOBEST tasks. The random performance indicates a hard capability ceiling: pretraining on 167–219B tokens of Korean text with a standard autoregressive objective, at scales up to 12.8B parameters, does not produce models that can perform lexical-semantic disambiguation. This is not a failure of Polyglot-Ko specifically — all tested models fail — but it reveals a boundary condition that practitioners need to know about. If a downstream application requires word-sense disambiguation (e.g., a Korean dictionary tool, a translation system that must correctly translate polysemous words, or a reading comprehension system where word sense affects answer correctness), Polyglot-Ko in its current form cannot provide this capability, and scaling to 12.8B parameters did not move the needle. This suggests that architectural innovations, different training objectives (e.g., contrastive learning on word senses), or substantially more training data may be needed.

What evidence exists in the paper. Table 5 (right) and Figure 3b provide the evidence. The flat accuracy across all models (range: 0.475–0.514, tightly clustered around 0.50) and the absence of any trend with model scale or shot count is the statistical signature of a capability floor — the models genuinely cannot do this task, and no amount of in-context examples helps. The paper's decision to report this transparently (rather than burying the result) is a strength of the evaluation, but the paper does not analyze why WiC fails or what it implies about the models' representational limitations.

Mitigation status. Not addressed. The paper does not propose experiments to diagnose the WiC failure (e.g., probing whether the models learn distinct representations for different senses of the same word, or testing whether fine-tuning on a Korean word-sense disambiguation dataset can recover performance). The failure is reported and left as an open problem. For practitioners, this means that WiC-like capabilities should not be assumed, and applications requiring word-sense disambiguation should either avoid relying on Polyglot-Ko for this purpose or plan for task-specific fine-tuning with appropriate training data. The paper does not provide guidance on how much fine-tuning data might be needed or whether fine-tuning can close the gap.

6.6 The Model Family Cannot Be Used for Clean Scaling Analysis Due to Confounded Architectural and Training Choices

The assumption or constraint. The paper frames the four Polyglot-Ko models as a size family spanning a ~10× parameter range, and Figure 2 and the discussion in Section 4.4 treat performance as a function of model scale ("as the compute increases, the performance improves"). However, the models differ along multiple dimensions simultaneously, making it impossible to attribute performance differences to scale alone. Specifically: (a) training token budgets vary from 167B to 219B, (b) batch sizes differ (1,024 for 1.3B and 3.8B; 256 for 5.8B and 12.8B), (c) architectural aspect ratios differ — the 5.8B model is shallower-but-wider (28 layers, $d_{model} = 4096$ ) than the 3.8B model (32 layers, $d_{model} = 3072$ ), and the 5.8B model uses a non-standard head dimension (256 vs. 128 for all other models), and (d) the 1.3B and 3.8B models were early-stopped before an epoch-boundary collapse while the 5.8B and 12.8B models were not, meaning the selected checkpoints represent different points on their training stability curves.

The consequence. A researcher who wants to study how Korean language understanding scales with model size cannot isolate the effect of parameter count from the effects of training duration, batch size, architecture, or training stability. If the 12.8B model barely outperforms the 5.8B on HellaSwag (0.631 vs. 0.600 at 5-shot), is that because scaling provides diminishing returns, or because the 12.8B model is undertrained (167B tokens vs. a Chinchilla-optimal ~256B), or because its 40-layer architecture with 128-dimension heads is suboptimal compared to the 5.8B's 28-layer architecture with 256-dimension heads? The paper provides no basis for choosing among these explanations. The scaling narrative in Section 4.4 ("a clear trend that as the compute increases, the performance improves") is therefore an oversimplification that overlooks the confounded experimental design.

What evidence exists in the paper. Table 2 demonstrates the architectural variation — the non-monotonic layer count (24 → 32 → 28 → 40) is directly visible, as is the varying head dimension. Section 3 documents the differing token budgets and batch sizes. The non-monotonic performance results (HellaSwag 50-shot, SentiNeg 0-shot modified) in Tables 3 and 5 provide the empirical signature of confounding — if scale were the dominant factor and other variables were controlled, performance would be expected to increase monotonically with parameter count, which it does not consistently.

Mitigation status. Not addressed as a limitation. The paper presents the four models as a family without acknowledging that the confounded design weakens scaling interpretations. The statement that "the performance improves with increasing compute" (Section 4.4), citing Kaplan et al. (2020), implies a cleaner relationship than the experimental design supports. This is not a failure of the models themselves — they are useful artifacts regardless — but it is a limitation of the paper's scaling analysis. A practitioner choosing which Polyglot-Ko model to use should not assume that the 12.8B is universally better than the 5.8B based on scale alone; task-specific evaluation is necessary given the confounded training conditions. A researcher designing a follow-up scaling study for Korean (or another non-English language) should control for training budget and architecture to produce interpretable scaling curves — the Polyglot-Ko paper does not provide a template for this.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not shift the field conceptually — monolingual pretraining is the oldest paradigm in language modeling, and the finding that dedicated Korean training data produces better Korean-language models than English-centric multilingual pretraining is unsurprising to anyone familiar with the data composition problem. The paper's contribution is not a new method, a new architecture, or a new theoretical insight that would cause researchers to rethink how they build language models.

What the paper does change is the resource landscape for Korean NLP. Before Polyglot-Ko, a Korean NLP practitioner who wanted a publicly available, commercially usable autoregressive language model at billion-parameter scale had approximately three options: ko-gpt-trinity-1.2B (too small for many applications), KoGPT-6.0B (larger, but from a single provider with specific licensing terms), or XGLM-7.5B (multilingual and underperforming on Korean). After Polyglot-Ko, that practitioner has four additional options spanning a ~10× parameter range, all released under open and commercially permissive terms, with transparent documentation of their training process, data composition, and known limitations. This is a substantial expansion of available infrastructure, not a conceptual advance.

The magnitude of the landscape change is best understood through the specific comparison that the paper makes possible: Polyglot-Ko-12.8B at 0.794 COPA 0-shot vs. KoGPT-6.0B at 0.735 (Table 3). The previously largest available Korean model achieved 0.735; the new model achieves 0.794 — roughly a 6 percentage point absolute improvement, or approximately an 8% relative gain. On SentiNeg with the modified prompt, the 12.8B model reaches 0.985 at 5-shot (Table 5), effectively solving the task. These are concrete capability improvements that Korean NLP applications can now access, and they exist because someone built and released the models, not because anyone invented a new technique.

The paper also changes the landscape in a methodological sense, but this contribution is implicit rather than argued. By (a) documenting the epoch-boundary generation collapse in 1.3B and 3.8B models, (b) disclosing the newline stripping preprocessing error, (c) reporting the WiC failure transparently, and (d) providing a detailed comparison of default vs. modified SentiNeg prompts, the paper establishes a norm for what a non-English model release report should include. The field has numerous English model reports (GPT-NeoX-20B, Pythia, OPT, LLaMA, BLOOM) that document training dynamics and known issues, but far fewer non-English releases with comparable transparency. Polyglot-Ko implicitly argues — through its structure and content — that Korean NLP deserves the same level of technical documentation as English NLP, and that claims about model capability should be supported by evaluation on language-specific benchmarks with acknowledged limitations.

This is not a paradigm shift. It is an infrastructure contribution that makes Korean NLP research and development more practical for a wider community, combined with a transparency contribution that provides a template for future non-English model reports. The paper does not resolve prior contradictions (there were no competing claims about Korean LLM training to reconcile), nor does it make any research direction less attractive. It does make one research direction more attractive: the "monolingual-first" approach to multilingual model development, which the paper's Conclusion explicitly endorses by describing plans for East-Asian and Romance multilingual models built on Korean-model expertise. The implicit argument is: before attempting to serve dozens of languages in one model, get one language right first, understand its training dynamics and data requirements, and then use that knowledge to inform multilingual development. This is a methodological reframing of the multilingual model problem rather than a technical innovation.

One area where the paper may reduce enthusiasm for a specific approach is in the viability of English-centric multilingual models for Korean. The consistent underperformance of XGLM-7.5B — a model with 5.8× more parameters than Polyglot-Ko-1.3B — on Korean-specific tasks (COPA, SentiNeg) provides concrete evidence that the current generation of multilingual models does not adequately serve Korean. A researcher who was hoping that scaling up multilingual models would eventually close the non-English performance gap for Korean might, after reading this paper, consider investing instead in dedicated Korean training data and models. The paper does not prove that multilingual models cannot work for Korean — only that the specific multilingual model tested (XGLM) does not work as well as a purpose-built Korean model at a fraction of the scale. This is evidence against a particular implementation of the multilingual approach, not against the general concept, but it shifts the burden of proof: advocates of multilingual models for Korean now need to demonstrate that their approach can match or exceed monolingual performance, rather than assuming it will.

Follow-Up Research This Work Enables

Quantifying the cost of the newline stripping error on Korean generation quality. The paper discloses that "the data was incorrectly stripped of newlines, leading to a loss of document structure" (Section 5) but provides no evaluation of its impact. A direct follow-up would train a Polyglot-Ko-1.3B replica on the same data with newlines correctly preserved, then compare the two models on Korean generative tasks: summarization (using the Korean summarization datasets from AIHub, which the paper lists in Table 1), long-form text completion, and structured document generation (e.g., formatted emails, reports with section headings). The comparison would measure whether newline stripping primarily affects surface-level formatting (paragraph breaks, document readability) or deeper discourse coherence (topic transitions, argument structure). This experiment is newly tractable because Polyglot-Ko provides a documented training recipe and data composition — a researcher can replicate the preprocessing pipeline, fix the newline stripping step, and train a comparable model without reverse-engineering the entire data curation process. A negative result (no measurable impact of newline preservation on downstream generation metrics) would be valuable too, because it would tell the community that document structure may be less critical for Korean generation quality than intuition suggests, reducing the pressure to perfectly preserve formatting in future Korean training pipelines.

Diagnosing the epoch-boundary generation collapse through controlled batch size and architecture experiments. The 1.3B and 3.8B models (batch size 1,024) experienced generation quality collapse near the first epoch boundary, while the 5.8B and 12.8B models (batch size 256) did not. This natural experiment suggests a batch size effect but is confounded with model size, architecture, and training token count. A clean follow-up would train multiple Polyglot-Ko-1.3B replicas on the same data, varying only batch size (256, 512, 1,024, 2,048) while holding architecture and training tokens constant, and monitor both perplexity and generation quality (via human evaluation or automated metrics like MAUVE) through the first epoch boundary. A second axis would test whether the collapse is architecture-dependent by comparing the 3.8B's configuration against the 5.8B's shallower-but-wider design at matched batch size, to determine whether depth, width, or some interaction mediates training stability. The paper makes this tractable by providing the exact step counts (100,000) and batch sizes (1,024) at which the collapse occurred, giving future researchers a clear target to reproduce or avoid. A strong result would identify the minimum batch size or architectural constraint that prevents the collapse, providing concrete guidance for anyone training Korean autoregressive LMs with GPT-NeoX on web-scraped Korean text.

Ablation on the morpheme-aware tokenizer against standard BPE for Korean downstream tasks. The paper's tokenizer design — MeCab-based morphological analysis followed by Byte-Level BPE — is linguistically well-motivated for Korean's agglutinative morphology but has no controlled ablation supporting its value. A direct follow-up would train two Polyglot-Ko-1.3B variants on identically preprocessed data: one with the morpheme-aware tokenizer described in Section 3, and one with standard Byte-Level BPE applied directly to raw Korean text (no MeCab preprocessing). Both would use the same vocabulary size (30,003) and the same training recipe. The comparison would measure (a) tokenizer fertility — average tokens per Korean character and per eojeol (whitespace-delimited unit) — to quantify the compression efficiency difference, (b) downstream performance on KOBEST tasks, particularly SentiNeg (where morphological awareness of negation markers like "안" and "-지 않-" should matter most) and COPA (where verbal endings encode causal relationships), and (c) performance on a Korean morphological reinflection task (e.g., given a root form and target grammatical features, produce the correctly inflected word) to directly measure whether the tokenizer helps the model learn Korean morphology. The SentiNeg results in Table 5 provide the motivation: the 12.8B model's near-ceiling scores (0.982+ at 5-shot and above) suggest that negation detection is well-handled by the current tokenizer, but without the ablation, we cannot know whether standard BPE would achieve similar performance or whether the morpheme-aware design is load-bearing.

Fine-tuning Polyglot-Ko on Korean word-sense disambiguation to test whether the WiC failure is fixable with task-specific data. The WiC results in Table 5 (right) show all models performing at random (accuracy ~0.50). This could mean that Korean word-sense disambiguation is fundamentally beyond the capability of autoregressive models at this scale, or that the pretraining data and objective do not produce lexical-semantic representations that can distinguish word senses, but that task-specific fine-tuning can recover the capability. A direct follow-up would fine-tune Polyglot-Ko-12.8B on a Korean word-sense disambiguation dataset (if one exists — the paper lists an AIHub dataset in Table 1 that may include word-sense annotations) and evaluate on WiC. Even a small fine-tuning dataset (a few thousand annotated instances) could determine whether the WiC failure reflects a representational deficit (the model never learned separable embeddings for different word senses) or a task-format mismatch (the model has the knowledge but cannot express it in the WiC binary-choice format). If fine-tuning succeeds, it would establish WiC as a task that requires explicit supervision rather than a fundamental capability ceiling, and would guide practitioners toward fine-tuning for word-sense-sensitive applications. If fine-tuning fails even with thousands of examples, it would establish WiC as a genuinely hard problem for current Korean models and motivate research into contrastive pretraining objectives or architectural modifications for lexical semantics.

Evaluating whether the monolingual advantage persists when multilingual models receive comparable Korean data volume through continued pretraining. The paper's comparison with XGLM-7.5B shows that a dedicated Korean model outperforms a larger multilingual model, but this comparison confounds training data composition with training data volume: XGLM's Korean subcorpus is almost certainly much smaller than Polyglot-Ko's 863 GB. A controlled follow-up would take XGLM-7.5B and continue pretraining it on the full Polyglot-Ko training corpus (or a subset thereof) until the Korean token count matches Polyglot-Ko-1.3B's 213B tokens, then evaluate on KOBEST. This experiment isolates the effect of multilingual pretraining architecture and tokenizer from the effect of Korean data volume. If the Korean-continued XGLM matches or exceeds Polyglot-Ko-1.3B, it would demonstrate that the multilingual approach can work for Korean given sufficient language-specific data — shifting the problem from "multilingual models can't do Korean" to "current multilingual models don't include enough Korean data." If the Korean-continued XGLM still underperforms, it would provide stronger evidence that the multilingual tokenizer or architecture imposes a fundamental ceiling on Korean performance, and that monolingual models are genuinely necessary. The paper makes this experiment tractable by providing the corpus and the training infrastructure description, and by establishing Polyglot-Ko-1.3B as a clear monolingual baseline with known performance on KOBEST.

A scaling law study for Korean that controls for training budget and isolates parameter count effects. The paper's four-model family spans a ~10× parameter range but confounds scale with training budget, batch size, and architecture. A rigorous follow-up would train a series of Korean autoregressive LMs using a fixed architecture (constant aspect ratio, head dimension, and layer count scaling formula) and fixed training budget per parameter (e.g., Chinchilla-optimal 20 tokens/param), then measure downstream performance as a function of parameter count. This would produce a Korean-specific scaling law analogous to Hoffmann et al. (2022) for English. The experiment is computationally expensive but newly tractable because Polyglot-Ko provides a validated Korean data pipeline (863 GB of text, documented preprocessing steps) and a known tokenizer, removing the data-engineering barrier that would otherwise make such a study impractical for researchers without Korean NLP expertise. The key measurement would be whether the Chinchilla scaling exponent — which relates parameter count to optimal training tokens and predicts performance as a function of compute — transfers from English to Korean, or whether Korean's morphological complexity and different data distribution change the compute-optimal tradeoff. The WiC and HellaSwag results in the current paper — where scaling from 5.8B to 12.8B sometimes yields minimal or negative gains — hint that Korean scaling behavior may differ from English, but the confounded training budgets prevent a clean test.

Practical Applications and Downstream Use Cases

Korean customer service chatbots and virtual assistants. Polyglot-Ko-12.8B's strong performance on sentiment analysis — 0.972 F1 on SentiNeg at 50-shot with the default prompt (Table 4), 0.982 with the modified prompt (Table 5) — makes it directly applicable to Korean-language customer service systems that need to detect customer sentiment and respond appropriately. A Korean e-commerce company deploying a chatbot can use Polyglot-Ko-12.8B as the base language model, fine-tuned on their specific product catalog and support dialogues, with confidence that the underlying model already achieves near-ceiling performance on negation-aware sentiment detection. The importance of negation detection for customer service is concrete: a customer saying "I'm not unhappy with the product" expresses a different sentiment than "I'm unhappy," and a model that fails to detect the negation marker "not" would misclassify the sentiment and potentially escalate unnecessarily or respond inappropriately. The 0.972–0.982 F1 scores suggest that this class of error will be rare, making Polyglot-Ko suitable for production sentiment-sensitive applications without extensive additional safety engineering for negation-related failures.

Korean content moderation and hate speech detection fine-tuning. The paper acknowledges that hate speech exists in the training data (Section 2.1) and that "Polyglot-Ko may generate content that is socially unacceptable or offensive" (Section 5). Rather than using the model for generation in high-stakes settings, a practical deployment would fine-tune Polyglot-Ko on a Korean hate speech classification dataset (the paper lists a Korean hate speech dataset in Table 1, though it is < 1.0 GB and was apparently included in training rather than used for evaluation). The strong discriminative performance on other KOBEST classification tasks (BoolQ: 0.645 F1 at 50-shot for 12.8B; SentiNeg: 0.972) indicates that the model's representations support accurate binary and multi-class classification after task-specific fine-tuning. A Korean social media platform could fine-tune Polyglot-Ko-5.8B (to balance performance and inference cost) on their proprietary hate speech labels, leveraging the model's pretrained understanding of Korean sentiment, negation, and discourse structure to build a content moderation classifier. The BoolQ results (Table 4) provide evidence that the model can handle boolean classification with reading comprehension, which is structurally similar to hate speech detection (read a passage, determine whether it violates a policy). A concrete deployment would measure the fine-tuned model's precision-recall curve against a baseline of keyword-based filtering or a smaller BERT-based Korean classifier, quantifying the benefit of Polyglot-Ko's larger pretrained representations.

Low-resource Korean NLP task development via few-shot prompting. The paper's evaluation demonstrates that Polyglot-Ko models benefit from in-context examples across multiple tasks — COPA improves from 0.794 (0-shot) to 0.837 (50-shot) for the 12.8B model (Table 3), BoolQ improves from 0.482 to 0.645 (Table 4). For a Korean NLP researcher who needs to classify or extract information from Korean text but lacks a large labeled dataset, Polyglot-Ko-12.8B enables few-shot prototyping: provide 5–50 labeled examples in the prompt, and the model can perform classification or reasoning on new instances without any fine-tuning infrastructure. This is particularly valuable for niche Korean domains — legal document classification, medical text annotation, dialect identification — where labeled data is scarce and fine-tuning a full model is impractical. The BoolQ results suggest that tasks requiring reading comprehension and factual reasoning benefit strongly from few-shot examples (0.482 → 0.645, a 34% relative improvement), while SentiNeg is already strong at 0-shot (0.893–0.912, depending on prompt). A Korean legal tech startup could, for example, use Polyglot-Ko-12.8B with 10–50 example case summaries and rulings to classify new case documents by legal area, without collecting thousands of labeled examples or training a specialized model — the COPA and BoolQ results provide empirical justification that the model's few-shot learning capacity is sufficient for this class of task.

Pretraining initialization for Korean-specific domain adaptation. The Polyglot-Ko models, particularly the 1.3B and 3.8B variants that are small enough for continued pretraining on modest hardware, serve as strong initializations for domain-specific Korean models. A Korean medical AI company could take Polyglot-Ko-3.8B, continue pretraining on a corpus of Korean medical textbooks, research papers, and clinical notes, and obtain a medical-domain Korean model with substantially less compute than training from scratch. The paper's documentation of training infrastructure (256 A100s, GPT-NeoX codebase, morpheme-aware tokenizer) provides a recipe that the company can follow for continued pretraining. The 3.8B model's strong COPA performance (0.760 0-shot, 0.779 50-shot — Table 3) indicates solid base reasoning capability that domain adaptation would build upon. The key practical benefit is cost reduction: training a Korean medical language model from scratch would require assembling a large Korean medical corpus (expensive), designing a tokenizer (the paper's MeCab+BPE design can be reused as-is), and running pretraining on hundreds of GPUs for weeks. Starting from Polyglot-Ko-3.8B reduces this to a continued pretraining task on ~10–50B tokens of domain-specific text, which is feasible on a single 8-GPU node within days to weeks, depending on corpus size. The paper does not provide continued pretraining guidelines, but the publicly released model weights and tokenizer make this workflow possible for any organization with modest GPU resources.