CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean

ArXiv: 2403.06412

🎯 Pitch

Even Korea’s best language models fail on over 30% of a new Korean culture and language benchmark—and simply scaling up model size or adding more Korean data doesn’t help. This benchmark reveals a stubborn gap where current LLMs can’t acquire the deep cultural and linguistic knowledge that Korean speakers take for granted.

1. Executive Summary

This paper introduces CLIcK, a benchmark dataset for evaluating Cultural and Linguistic Intelligence in Korean, comprising 1,995 QA pairs sourced from official Korean exams and textbooks and organized into eleven subcategories across language and culture. Using CLIcK, the authors evaluate 13 language model configurations spanning open-source Korean-specialized models (Polyglot-Ko, KULLM, KoAlpaca, LLaMA-Ko) and proprietary LLMs (GPT-3.5, Claude-2), finding that open-source models with fewer than 13B parameters struggle with over 60% of the dataset—with a shared difficulty spanning 35.5% across models—while GPT-3.5 and Claude-2 still face challenges on over 30% of the data (sharing 12.6% of problems). The analysis reveals that neither scaling model size nor fine-tuning with additional Korean corpora reliably improves cultural and linguistic comprehension, establishing that current pretraining-centric approaches are insufficient for non-English cultural intelligence and only 0.6% of the dataset was universally answered correctly by all models tested.

2. Context and Motivation

The Core Problem: Korean LLMs Have No Culturally Authentic Benchmark

The fundamental gap this paper addresses is deceptively simple to state but deeply consequential in practice: there exists no comprehensive evaluation dataset that tests whether language models actually understand Korean culture and language on their own terms, rather than simply performing well on Korean translations of English benchmarks. This is not merely a matter of academic completeness—it represents a systematic blind spot in how the field evaluates and develops LLMs for non-English languages, particularly those with cultural contexts that diverge meaningfully from the Anglosphere.

The paper frames this gap through a specific, telling comparison in Section 1: KoGPT3-39B underperforms on the Korean HellaSwag task by approximately 20 percentage points compared to the similarly-sized English Falcon40B model on the original English HellaSwag, even though human annotators achieve comparable performance in both languages. This gap cannot be attributed to the inherent difficulty of the Korean language—Korean speakers find both versions equally solvable. Rather, it signals that something about the knowledge required to answer these questions—knowledge that is culturally situated, linguistically nuanced, and not directly translatable—is absent from current models.

This is the paper's central insight: culture and language are not interchangeable surface forms over universal knowledge. They encode distinct ways of knowing, distinct factual landscapes, and distinct social practices that must be learned specifically—they cannot be approximated through translation or scale alone.

Why This Problem Matters: Real-World and Theoretical Stakes

The paper's motivation extends beyond benchmark construction into several interconnected concerns that make this problem urgent:

1. The deployment gap for non-English LLMs. Korean LLMs are actively being developed and deployed—the paper references multiple model families including Polyglot-Ko, KULLM, KoAlpaca, and LLaMA-Ko—yet without culturally authentic evaluation, developers have no way to know whether their models actually serve Korean users effectively. A model might score well on translated benchmarks while being fundamentally incompetent at answering questions that any Korean speaker would consider basic cultural knowledge. This creates a dangerous illusion of capability: models appear competent according to available metrics but fail in ways that are invisible to those metrics.

2. The cultural erasure problem. When Korean benchmarks are primarily derived from English through translation (as the paper notes is common practice, citing the Open Ko-LLM Leaderboard in Section 2.3), the evaluation framework itself enforces an Anglocentric standard of what constitutes important knowledge. Questions about Shakespeare might be translated into Korean and used to evaluate Korean models, but questions about traditional Korean poetry, Korean kinship terminology, Korean legal systems, or Korean historical events are absent—not because they are less important, but because they don't exist in the English source datasets. The paper argues this systematically disadvantages evaluation of cultural competencies that matter to Korean users.

3. The pretraining-limits hypothesis. The paper's experimental finding that neither model scale nor additional Korean fine-tuning reliably improves performance on CLIcK (Section 4.1, Table 6) has profound practical and theoretical implications. If current scaling approaches plateau on cultural knowledge, then the field's dominant paradigm—"collect more data, train bigger models"—may be fundamentally insufficient for cultural intelligence. This connects to broader concerns about whether LLMs can truly acquire deep cultural understanding from statistical patterns in text, or whether something more structured (curated data, explicit knowledge integration, cultural alignment training) is needed.

4. Human-level comparison reveals stark deficits. The paper's comparison to human test-takers on actual Korean examinations (Section 5.2) provides concrete, interpretable evidence of the problem's severity. GPT-3.5, which scores in the top 13th percentile on the English SAT, scores in the lowest 11th percentile on the Korean CSAT. Claude-2 achieves TOPIK level 6 (the highest attainable level for non-native speakers, indicating professional proficiency), but all open-source Korean-specialized models fall below the measurable range. On the Kedu exam for teaching Korean as a foreign language, all models score below the human average, with proprietary models lagging by ~10 percentage points and open-source models by >20. These are not marginal deficiencies—they represent fundamental gaps in cultural knowledge.

Where Prior Approaches Fall Short

The paper identifies specific, structural limitations in existing Korean evaluation resources across three dimensions:

Limitation 1: Korean benchmarks are predominantly translations, not authentically Korean. The paper explicitly critiques the dominant approach to Korean LLM evaluation (Section 2.3): most benchmark tasks on the Open Ko-LLM Leaderboard are "mainly derived from English benchmarks" through translation, "failing to capture the key aspects of the Korean language or culture." This critique is not merely about cultural representation—it has technical substance. Translated benchmarks carry several problems:

• Factual mismatch: English benchmarks may ask about Thanksgiving traditions or U.S. presidential history; translated, these remain about American culture but in Korean text. They test translation quality and general reasoning, not Korean cultural knowledge.
• Linguistic mismatch: Certain linguistic phenomena—Korean honorifics (존댓말), the complex system of kinship terms, or topic-comment sentence structures—have no English equivalent and are thus absent from translation-derived benchmarks.
• Difficulty mismatch: The paper notes that existing Korean tasks are "either too simple"—referring to early Korean NLP benchmarks like KorNLI, KorSTS (Ham et al., 2020a), and KorQuAD (Lim et al., 2019)—or "mainly derived from English benchmarks." The simple tasks are insufficient for evaluating LLM-level reasoning; the translated tasks are linguistically Korean but culturally English.

Limitation 2: Korean-native datasets exist but are task-narrow. The paper acknowledges prior efforts to create authentically Korean datasets but identifies their scope as insufficient for comprehensive evaluation. The Korean Language Understanding Evaluation (KLUE) benchmark (Park et al., 2021) introduced eight downstream tasks for Korean, but the paper notes these "lacked tasks for advanced reasoning, which is inadequate for LLM evaluation." Other Korean-native datasets focus on specific problems:

• Korean Offensive Language Dataset (KOLD) (Jeong et al., 2022): targets toxic speech detection
• Korean Bias Benchmark for Question Answering (KoBBQ) (Jin et al., 2023): targets bias detection, building on the English BBQ dataset but with cultural adaptation (classifying translations into Simply-Translated, Sample-Removed, and Target-Modified categories to reflect Korean cultural context)
• HAE-RAE Benchmark (Son et al., 2023): provides Korean reading comprehension from original Korean corpora, but remains constrained to reading comprehension as a task

The paper's critique is that these datasets, while valuable, are "constrained to specific tasks and may not be suitable for evaluating diverse topics related to Korean cultural and linguistic knowledge." A model might excel at hate speech detection and reading comprehension while knowing nothing about Korean history, geography, law, traditional culture, or popular culture—and no existing benchmark would reveal this gap.

Limitation 3: The human-examination gap in Korean LLM evaluation. The paper draws inspiration from benchmarks like MMLU (Hendrycks et al., 2021) and AGIEval (Zhong et al., 2023), which use questions from actual human examinations (professional certifications, university entrance exams, standardized tests) to evaluate models at human-relevant difficulty levels. These benchmarks established that human examination questions provide (a) clear difficulty calibration, (b) direct comparability between model and human performance, and (c) broad domain coverage drawn from real educational curricula. However, no such resource existed for Korean cultural and linguistic knowledge specifically—AGIEval includes some Korean exams but is not focused on Korean cultural content as a primary evaluation axis.

Limitation 4: Cultural knowledge cannot be acquired through scaling alone—but nobody has systematically shown this. The paper positions itself against an implicit assumption in the LLM development community: that training on larger, more diverse datasets will eventually subsume cultural knowledge. This assumption is challenged by theoretical work the paper cites: Lee et al. (2023a) demonstrated that "language models fail to capture biases in different languages due to their cultural insensitivity," and Ma et al. (2022) showed that "integrating cultural knowledge can improve model performance" above and beyond what general pretraining provides. However, prior to CLIcK, there was no comprehensive Korean benchmark that could systematically test whether scaling and additional Korean pretraining actually improve cultural understanding across diverse domains.

How This Paper Positions Itself

The paper positions CLIcK as filling a specific, well-defined gap in the evaluation landscape. Unlike prior Korean benchmarks that are either (a) English-derived through translation, (b) task-narrow, or (c) insufficiently challenging for LLMs, CLIcK is explicitly designed to be:

1. Culturally authentic by construction. All 1,995 QA pairs are sourced from original Korean materials—six official Korean examinations (CSAT, TOPIK, PSE, PSAT, KHB, Kedu) and the KIIP government textbook for immigrant integration. The paper obtains official permission from the relevant institutions and emphasizes this sourcing choice as a deliberate methodological decision: rather than translating English cultural questions or generating questions from a model's internal Korean knowledge, CLIcK uses questions that were designed by Korean educators for Korean test-takers to assess Korean cultural and linguistic competence. This ensures that the knowledge being tested is genuinely what Korean society considers important cultural knowledge.

2. Broad in coverage with fine-grained categorization. The 11 subcategories—eight for cultural intelligence (Society, Tradition, Pop Culture, Politics, Economy, Law, History, Geography) and three for linguistic intelligence (Textual Knowledge, Functional Knowledge, Grammatical Knowledge)—provide the first taxonomy of Korean cultural-linguistic knowledge for LLM evaluation. This taxonomy is grounded in established frameworks: the cultural categories follow the KIIP textbook's curriculum structure, while the linguistic categories follow Bachman and Palmer's (1996) framework of communicative language ability. This is not an ad-hoc grouping but a principled categorization that enables fine-grained analysis of exactly where models fail.

3. Demonstrating that scale doesn't solve culture. The paper's experimental contribution is as much about the negative finding as the positive resource. By testing 13 model configurations across five model families with varying parameter counts and Korean training data exposure, and finding no statistically significant effect of model scale (F₁,₄₃ = 0.33, p = .57) or Korean corpus supplementation (F₁,₅₄ = 0.30, p = .59) on CLIcK performance, the paper makes a specific empirical claim: the dominant paradigm for improving LLMs does not transfer to cultural knowledge acquisition. This finding is positioned as motivating "more tailored methods in further research" (Section 6)—not just better Korean pretraining data, but fundamentally different approaches to instilling cultural knowledge.

4. Releasing a public resource to enable future work. The paper emphasizes that CLIcK is publicly available and provides not just questions and answers but also per-instance annotations of the cultural or linguistic knowledge category required. This design choice is deliberate: it enables future researchers to study not just whether models fail but what kind of cultural knowledge they fail on, enabling targeted interventions (e.g., augmenting training data specifically for categories where models underperform).

The paper's position, in essence, is that the LLM evaluation community has been asking the wrong questions about non-English languages. The relevant question is not "Can this Korean-tuned model perform well on Korean translations of English benchmarks?" but rather "Does this model actually understand Korean culture and language as a Korean speaker would?" CLIcK is designed to answer the latter question for the first time at scale, and the answer it provides—no, current models do not, and scaling won't fix it—carries implications for the entire field's approach to multilingual and multicultural LLM development.

3. Technical Approach

3.1 Reader Orientation

CLIcK is an evaluation benchmark—a collection of 1,995 carefully curated multiple-choice questions designed to measure whether language models possess the cultural and linguistic knowledge that any fluent Korean speaker would be expected to have. The paper constructs this benchmark from scratch using authentic Korean source materials (government exams and textbooks) rather than translating English benchmarks, then uses it to test 13 different language model configurations, revealing that current LLMs—including those specifically fine-tuned for Korean—systematically fail on questions requiring deep Korean cultural knowledge, and that neither model scale nor additional Korean pretraining reliably fixes this gap.

3.2 Big-Picture Architecture (Diagram in Words)

The CLIcK system has four major components, arranged as a pipeline from raw source material to final model evaluation:

1. Source Material Collection — the acquisition of raw Korean-language examination questions and textbook content from seven distinct Korean government sources (six exams plus the KIIP textbook), each representing authentic Korean cultural or linguistic knowledge domains that Korean society has deemed important enough to test formally.

2. Data Extraction and Generation — two parallel processes: (a) optical character recognition (OCR) is applied to exam materials to convert physical or PDF-format multiple-choice questions into structured text, while (b) GPT-4 is prompted with textbook passages to generate novel multiple-choice questions strictly based on the provided text. This component produces the raw pool of candidate questions with answers.

3. Human Validation and Categorization — a multi-round annotation process where four native Korean speakers verify each question against four validity criteria (textual grounding, temporal stability, Korean relevance, objectivity), discard invalid questions, revise ambiguous ones, and assign each surviving question to one of eleven fine-grained subcategories of cultural or linguistic knowledge. This component transforms an unverified question pool into the final, validated CLIcK dataset of 1,995 questions.

4. Model Evaluation Framework — a standardized testing protocol that administers CLIcK questions to language models using controlled prompts (with cyclic permutation of answer choices to neutralize position bias), collects model predictions (via log-probability ranking for open-source models or constrained generation for API-based models), and computes accuracy metrics per category, with human-level comparisons enabled by official exam statistics.

Information flows sequentially: raw Korean materials → extracted/generated questions → human-validated and categorized dataset → model predictions → accuracy scores per category and model.

3.3 Roadmap for the Deep Dive

• First, the data collection pipeline—how questions are sourced from exams (via OCR) and generated from textbooks (via GPT-4 prompting)—because understanding the provenance of each question type is essential for evaluating the benchmark's authenticity claims.
• Second, the human validation protocol—the multi-round annotation criteria and reconciliation process—because this is where CLIcK's quality guarantee comes from: the 37.1% question rejection rate and the specific validity criteria directly shape what the final dataset tests and what it excludes.
• Third, the categorization taxonomy—the eleven subcategories across Cultural Intelligence and Linguistic Intelligence—because this taxonomy is the framework for all downstream analysis, enabling fine-grained diagnosis of model failures.
• Fourth, the evaluation protocol—prompts, cyclic permutation, acceptance criteria, and accuracy computation—because the paper's quantitative claims about model performance depend entirely on these methodological choices, and subtle details like cyclic permutation and out-of-option handling matter for reproducibility.
• Fifth, the design rationale—why certain choices were made (authentic exam sourcing over translation, GPT-4 generation over manual authoring, specific categorical frameworks) and what the tradeoffs are.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a dataset and evaluation paper whose core technical contribution is the construction methodology for the CLIcK benchmark and the evaluation framework applied to it. The technical depth lies not in a novel algorithm but in the careful, multi-stage process of creating a culturally authentic, linguistically precise, and evaluation-ready dataset from scratch, along with the controlled experimental methodology for testing models against it.

3.4.1 Data Collection: Sourcing Authentic Korean Examination Questions

The first stage of the pipeline acquires raw multiple-choice questions from seven distinct Korean sources—six official examinations and one government textbook—selected because they represent formal, institutional assessments of Korean cultural and linguistic knowledge designed by Korean subject-matter experts for Korean test-takers. The paper describes this sourcing in Section 3.1 and provides source details in Table 3.

Examination-based sourcing. The six examinations, described in Appendix A, are:

• CSAT (College Scholastic Ability Test of Korea, 대학수학능력시험): Korea's national university entrance examination, administered annually by the Korean Ministry of Education, assessing high school curriculum knowledge including Korean language and geography. The paper uses questions from the Korean language and geography sections.
• TOPIK (Test of Proficiency in Korean, 한국어능력시험): The standardized test measuring Korean language proficiency for non-native learners, covering levels 1–6 from beginner to advanced.
• PSE (National Public Service Examination - Grade 9, 9급 국가공무원 공개경쟁채용시험): The civil service examination for entry-level government positions. The paper uses questions from the history section and the language section covering grammar, vocabulary, and reading comprehension.
• PSAT (Public Service Aptitude Test, 공직적격성평가): A higher-level civil service aptitude test; the paper uses only the Korean Constitution section.
• KHE (Korean History Exam, 한국사능력검정시험): A standardized test measuring historical literacy of Korean citizens, covering Korean history from ancient to modern.
• Kedu (Test of Teaching Korean as a Foreign Language, 한국어교육능력검정시험): A certification exam for individuals who wish to teach Korean to overseas Koreans or foreigners. This exam covers both Korean language and Korean culture.

For each exam, the paper obtains the test materials and then applies Clova OCR, a commercial optical character recognition service from Naver, to extract text from the exam documents, explicitly excluding images and tables (Section 3.1: "We use Clova OCR to extract text from the exams, excluding images and tables"). This extraction converts the physical or PDF-format exam questions into structured text records containing the question text, the multiple choice options, and the correct answer as specified by the exam's answer key.

Why OCR rather than manual transcription? The paper does not explicitly justify this choice, but the practical rationale is evident: six different examinations spanning multiple years each contain hundreds of questions. Manual transcription by human annotators would be prohibitively expensive and slow, and exam documents are structured enough that OCR accuracy for Korean text is high (Korean's Hangul writing system, being featural and regularly spaced, is generally well-handled by modern OCR systems). The explicit exclusion of images and tables is a necessary limitation—OCR systems struggle with complex formatting, and the knowledge embedded in diagrams, maps, or statistical tables cannot be captured in a text-only multiple-choice format.

The critical design decision: questions are taken verbatim from actual exams. This means CLIcK's questions were not designed by the paper's authors; they were designed by the Korean testing institutions themselves. Each question reflects what Korean educators and policymakers have decided constitutes important knowledge about Korean language, history, geography, law, politics, and culture. This is a deliberate methodological choice that distinguishes CLIcK from benchmarks where questions are authored by researchers or crowdworkers—the difficulty calibration and domain validity come from the exam design process, not from post-hoc researcher judgment.

3.4.2 Data Collection: Generating Novel Questions from Textbook Content Using GPT-4

For the KIIP (Korean Immigration and Integration Program) textbook—a government-produced textbook used to teach Korean language and culture to immigrants applying for Korean citizenship—the paper takes a different approach. Rather than extracting existing questions, it generates new multiple-choice questions from the textbook content using GPT-4 (Section 3.1: "Generating Questions Using GPT-4").

Why generate rather than extract? The KIIP textbook is a prose educational resource, not an examination with pre-existing multiple-choice questions. To convert its content into evaluation items, the authors need to create questions that test comprehension of the textbook material. Manual question authoring would be extremely labor-intensive given the textbook's length and the need to cover diverse topics; GPT-4 generation is used to produce questions at scale while constraining the model to the textbook content.

The generation procedure (described in Appendix B):

1. Text extraction and chunking: The full text of each KIIP textbook chapter is extracted. Because GPT-4 has a context window limitation and question quality degrades with very long passages, the extracted text is split into smaller, manageable chunks using a RecursiveCharacterTextSplitter from the Langchain library. The paper states that splitter parameters were "chosen based on initial experiments to balance between maintaining textual coherence and ensuring manageable chunk sizes for processing."

2. Structured prompting: Each text chunk is fed to GPT-4 with a specific prompt that instructs the model to generate exactly 10 multiple-choice questions. The prompt (provided in both original Korean and English translation in Appendix B) enforces a strict JSON output format with five required fields per question:

   • cite: The exact sentence from the passage used to create the question (this serves dual purpose: it grounds the question in the source text and enables human validators to verify correctness)
   • question_id: A unique sequential identifier starting from current_cnt
   • question: The question text
   • choices: The multiple choice options
   • answer: The correct answer

   The prompt also instructs GPT-4 to ensure that "the content of each question does not overlap"—preventing the generation of near-duplicate questions that would artificially inflate the question count without testing distinct knowledge.

3. Format verification: Any generated instances that fail to conform to the specified JSON format are identified and removed as a final automated quality check.

The critical constraint: GPT-4 must use only the provided text. The prompt explicitly instructs GPT-4 to base questions on the provided passage, and the required cite field forces the model to identify the specific source sentence. This is reinforced by the first validation criterion in Section 3.2: "Questions are solely based on the given text." The intent is to prevent GPT-4 from drawing on its own (potentially incorrect or culturally inappropriate) internal knowledge about Korea—the questions must be answerable from the textbook content alone.

What types of knowledge does the KIIP textbook cover? The textbook spans the cultural intelligence categories: Society, Culture (Tradition and Pop Culture), Politics, Economy, Law, History, and Geography. These are the domains that the Korean government has determined immigrants need to understand to integrate into Korean society. By using this textbook as source material, CLIcK inherits this scope—the generated questions test precisely the cultural knowledge that Korean institutions consider essential for cultural membership.

3.4.3 Data Validation: The Multi-Round Human Annotation Protocol

After data collection (exam OCR extraction and textbook GPT-4 generation), the paper subjects all candidate questions to a rigorous human validation process described in Section 3.2. This is the quality-control stage that transforms an unverified pool of candidate questions into the final CLIcK dataset.

The four validity criteria. Every question is evaluated against four explicit criteria:

1. "Questions are solely based on the given text." For GPT-4-generated questions, this means the answer must be derivable from the cited textbook passage without requiring external knowledge. For exam-derived questions, this criterion is automatically satisfied since exam questions were designed to be answerable from the exam's own context.

2. "Information in Questions remains consistent over time." This criterion excludes questions whose answers could change—for instance, questions about current government officials, recent election results, or time-sensitive statistics. CLIcK aims to test stable cultural knowledge, not knowledge of current events that would become outdated.

3. "Questions should centrally relate to Korea." This is the cultural authenticity filter. A question that happens to be in Korean but is about universal mathematics or a generic English-translated concept would be excluded. The benchmark must test Korean cultural and linguistic knowledge specifically.

4. "Questions should be objective and free from bias." This criterion excludes questions "connected to subjective beliefs or biases, like gender, politics, or international relations" (Section 3.2). The paper explicitly acknowledges that this criterion helps "maintain dataset objectivity"—CLIcK should test factual cultural knowledge, not controversial opinions or politically sensitive topics.

The multi-round annotation protocol. Four of the paper's authors, all Korean native speakers, serve as annotators through a three-phase process:

Phase 1: Initial independent review. Three annotators independently review each candidate question. The paper describes a specific decision rule: "If two or more annotators considered a sample invalid, it was discarded." For samples that need modification rather than outright rejection, one annotator performs the revision. The paper reports that 15.9% of the data was labeled invalid by at least one annotator in this phase.

Phase 2: Reconciliation of disagreements. For the samples that remain invalid or modified after Phase 1, a second round of annotation is conducted by three annotators. After this second round, 3.9% of the initial set still had discrepancies—meaning the annotators could not agree on validity.

Phase 3: Consensus discussion. The four annotators (the three from previous phases plus a fourth) discuss the remaining disagreements. Only samples with "unanimous agreement between all four annotators" are included in the final dataset.

The rejection rate and what it means. The paper reports that the initial dataset of 1,985 candidate questions was reduced to 1,245 questions after validation—62.9% of the original pool, meaning 37.1% was rejected. This substantial rejection rate serves as evidence of the rigor: the authors are not simply accepting automatically generated or OCR-extracted content at face value. The rejected questions likely include:

• GPT-4-generated questions that introduced external knowledge not present in the source text
• Questions with OCR errors that rendered them nonsensical or ambiguous
• Questions touching on politically sensitive topics or subjective interpretations
• Questions with culturally irrelevant content despite being in Korean
• Questions where the correct answer was ambiguous even to native speakers

The final dataset composition. The validated 1,245 questions from exams and GPT-4 generation are combined with questions from exams where the subject is inherently categorical (e.g., History exam questions are automatically classified under History). The final dataset size is reported as 1,995 QA pairs (Section 3, opening paragraph). Examining Table 1 reveals the distribution: the vast majority come from exams (1,811 from exams vs. 184 from the KIIP textbook). The textbook-generated portion is smaller, reflecting both the GPT-4 generation being more constrained and the higher rejection rate from validation.

3.4.4 Data Categorization: The Cultural and Linguistic Intelligence Taxonomy

Each question in CLIcK is annotated with a fine-grained category label indicating what kind of cultural or linguistic knowledge is required to answer correctly. This taxonomy (Section 3.3, Table 1) is not arbitrary—it is grounded in established frameworks for both cultural curriculum design and linguistic competence modeling.

Cultural Intelligence categories (8 subcategories). These are adopted directly from the KIIP textbook's chapter structure (Section 3.3: "We adopt eight subcategories based on the KIIP textbook"). The primary chapters of the KIIP basic textbook cover: Society, Culture, Politics, Economy, Law, History, and Geography. Within the Culture chapter, there are subsections on Tradition and Pop Culture, which the paper preserves as separate subcategories. This yields the eight cultural intelligence categories:

• Society (사회): Questions about Korean social structures, institutions, customs, and everyday social practices (e.g., kinship terminology, housing policies, social etiquette).
• Tradition (전통문화): Questions about traditional Korean culture, including historical customs, traditional arts, ceremonies, and heritage.
• Pop Culture (대중문화): Questions about contemporary Korean popular culture, including entertainment, media, modern arts, and current cultural phenomena.
• Politics (정치): Questions about Korean political systems, government structure, civic processes, and political institutions.
• Economy (경제): Questions about Korean economic systems, financial institutions, economic policies, and economic terminology in the Korean context.
• Law (법률): Questions about Korean legal systems, specific Korean laws, legal procedures, and legal terminology.
• History (역사): Questions about Korean historical events, periods, figures, and historical processes.
• Geography (지리): Questions about Korean physical geography, administrative divisions, regional characteristics, and geographic features.

Why use the KIIP textbook as the organizing framework? The paper's choice is deliberate: the KIIP textbook represents the Korean government's official curriculum for what immigrants need to know about Korean culture. By adopting its chapter structure as the taxonomy, CLIcK's cultural categories reflect an institutional—not researcher-arbitrary—partitioning of cultural knowledge domains. This gives the categorization external validity: when the paper reports that models perform poorly on "Law" or "Society" questions, it refers to the same categories that the Korean government uses to structure cultural education.

Linguistic Intelligence categories (3 subcategories). The paper adopts the framework from Bachman and Palmer's (1996) model of communicative language ability, a well-established theoretical framework in language testing and assessment. The three categories are:

• Textual Knowledge (텍스트 지식): "Concerns organizing utterances into coherent texts with cohesion and rhetorical structures" (Section 3.3). This includes understanding how Korean texts are structured—paragraph organization, discourse coherence, rhetorical patterns—and the ability to comprehend extended prose passages as unified wholes rather than isolated sentences. In the CSAT-Korean exam, this category corresponds to the literature (문학) and reading (독서) sections, which test comprehension of extended Korean texts including fiction, non-fiction, and poetry.

• Functional Knowledge (기능적 지식): "Focusing on the communicative roles of language, especially ideational, manipulative, heuristic, and imaginative functions" (Section 3.3). This tests understanding of what language is doing in context—the pragmatic and sociolinguistic functions of utterances. In the CSAT-Korean exam, this corresponds to the speaking (화법) and writing (작문) sections, which test the ability to understand communicative intent, register, and appropriate language use for different contexts. The paper notes that ideational functions involve expressing ideas and experiences, manipulative functions involve affecting the world through language (requests, commands), heuristic functions involve using language to learn and explore, and imaginative functions involve creative and aesthetic language use.

• Grammatical Knowledge (문법 지식): "Addressing the organization of utterances with an emphasis on vocabulary, syntax, and phonology/graphology" (Section 3.3). This tests formal linguistic knowledge: Korean grammar rules, correct vocabulary usage, syntactic structures, and orthographic conventions. In the CSAT-Korean exam, this corresponds to the language (언어) section.

What the paper excludes from Bachman and Palmer's framework. The original framework includes a fourth category: Socio-linguistic Knowledge, which concerns sensitivity to language use conventions determined by context (register, dialect, cultural references). The paper explicitly excludes this category because "it is largely subsumed by our annotations in the Cultural Intelligence category" (Section 3.3). This is a reasonable design choice: in a Korean-specific benchmark, the distinction between "socio-linguistic knowledge" and "cultural knowledge" becomes blurry—knowing which honorific level to use with a superior versus a friend is both a linguistic skill and cultural knowledge. Rather than forcing a separation, CLIcK captures this intersection through its cultural categories.

The categorization procedure. The paper describes a stratified approach (Section 3.3: "Specific Procedures"):

• Textbook-generated questions are automatically aligned to cultural categories because they were generated from specific textbook chapters (e.g., a question from the Economy chapter is labeled Economy).
• Single-subject exams need no additional categorization: all questions from the Korean History Exam (KHE) are History; all questions from the CSAT-Geography exam are Geography, etc.
• Multi-subject exams require manual mapping. The CSAT-Korean exam is the primary example: it contains six problem types (speaking, writing, language, media, literature, reading). The paper maps these to the linguistic framework: speaking and writing → Functional Knowledge; language (grammar section) → Grammar Knowledge; literature and reading → Textual Knowledge.
• Other Korean language exams (TOPIK, PSE language section, Kedu Korean language section) provide solution guides that detail each problem's category. "Based on this information, a single annotator validates the label assignment."

This hybrid approach—automatic categorization where possible, manual mapping with single-annotator validation where needed—balances efficiency with accuracy. The paper acknowledges the single-annotator validation for certain categories as a limitation, but given that the mapping is from exam-provided category labels (not from annotator judgment), the risk of misclassification is low.

3.4.5 Evaluation Protocol: Prompting, Scoring, and Accuracy Computation

The evaluation of language models on CLIcK uses a standardized protocol described in Section 4, designed to ensure fair comparison across models with different architectures, tokenization schemes, and output formats.

Prompt design. The paper uses two prompt templates depending on whether the question requires reading background context (Type 1) or can be answered from the question alone (Type 2). Both prompts are provided in Korean with English translations:

Type 1 (with context):

주어진 맥락을 천천히 읽고, 질문에 대한 적절한 정답을 A, B, C, D 중에 골라 알파벳 하나로 답하시오.
(Read the given context slowly, and choose the correct answer to the question from options A, B, C, or D. Respond with a single alphabet.)
맥락 (Context): {CONTEXT}
질문 (Question): {QUESTION}
보기 (Options):
A: {A}, B: {B}, C: {C}, D: {D}
정답 (Answer):

Type 2 (without context):

주어진 질문을 천천히 읽고, 적절한 정답을 A, B, C, D 중에 골라 알파벳 하나로 답하시오.
(Read the given Question slowly, and choose the correct answer from options A, B, C, or D. Respond with a single alphabet.)
질문 (Question): {QUESTION}
보기 (Options):
A: {A}, B: {B}, C: {C}, D: {D}
정답 (Answer):

The prompts follow a format derived from Jin et al. (2023), the KoBBQ paper. The key design features are:

• The model is instructed to "Read slowly" (천천히 읽고) and to respond with a single alphabet—this is a soft constraint intended to discourage verbose responses that complicate answer extraction.
• The options are presented with both the alphabet identifier and the option text (e.g., "A: 국민임대주택"), giving the model both the symbolic label and the semantic content.
• The prompt ends with "정답 (Answer):" to prime the model to output the answer directly.

Cyclic permutation to neutralize position bias. A known issue in multiple-choice evaluation is that language models often exhibit position bias—they prefer certain answer positions (typically A or the first position) regardless of content. To mitigate this, the paper adopts cyclic permutation from Izacard et al. (2023):

"Following Izacard et al. (2023), we apply cyclic permutation for each question to mitigate option's order effects in the prompt to the language model."

In cyclic permutation, the options are rotated through the A/B/C/D positions. For a question with 4 options (N=4), the model is tested N times with the options appearing in different positional slots. The paper reports results averaged over $3N$ experiments per question: "we prompt the model 3 times and adopt cyclic permutation for each instance, the total number of experiments per instance is $3N$, where $N$ denotes the number of options." The factor of 3 comes from using three different prompt wordings (likely paraphrases of the instruction), though the paper states only that prompts are "derived from Jin et al. (2023)" and that they report "the average over three different wordings of the prompt."

Why cyclic permutation rather than just randomizing? Cyclic permutation ensures that each option appears in each position exactly once across the N permutations, providing balanced coverage. The additional factor of 3 prompt wordings further reduces variance from prompt sensitivity. This design means that for a typical 4-option question, the model makes 12 predictions (3 prompt wordings × 4 cyclic permutations), and the reported accuracy for that question is the fraction of those 12 predictions that are correct.

Scoring for open-source models. For open-source models (Polyglot-Ko, KULLM, KoAlpaca, LLaMA-Ko, LLaMA variants), the evaluation uses a log-probability ranking approach standard in the MMLU and EleutherAI lm-evaluation-harness frameworks:

"For open-source models, we examine the output probabilities of option ID tokens (A/B/C/D or A/B/C/E) concatenated with the option string, selecting the most probable answer as the model prediction."

This means the model is not asked to generate an answer; instead, the evaluation computes the probability the model assigns to each complete option sequence (identifier letter followed by option text), and the option with the highest probability is selected as the model's prediction. This approach avoids issues with generative decoding (e.g., the model outputting an option letter that doesn't correspond to any choice, or outputting verbose text) and provides a deterministic score given the model's probability distribution.

Scoring for API-based models. For GPT-3.5-turbo and Claude-2, the evaluation uses a generative approach:

"For API-based LLMs (GPT-3.5-turbo and Claude-2), the evaluation involves comparing the generated response with the labeled answer. Here, the decoding temperature is set to 0."

The temperature of 0 means the models use greedy decoding, producing the most likely output deterministically. However, because these models may "at times produce verbose responses" despite the prompt asking for a single alphabet, the paper defines specific acceptance criteria for answer extraction, adopted from Jin et al. (2023):

The model's response is accepted as a valid answer if it:

• i) mentions only one alphabet from the given options
• ii) exactly matches a term provided in the options
• iii) includes specific expressions clearly intended to convey the answer, such as 'the answer is -'
• iv) presents the answer distinctly as per conditions i) to iii), followed by further explanation

Responses that don't meet any of these conditions are classified as "out-of-option answers"—effectively treated as incorrect, since they cannot be mapped to a choice. This is a conservative approach: ambiguous responses are counted against the model rather than being excluded from accuracy computation.

Accuracy computation. The paper defines accuracy per question using the following formula:

$p_{\text{accuracy}} = \frac{\text{count}(\text{correct answers})}{3N}$

where $N$ denotes the number of options for the question, the count of correct answers is the number of individual trials (out of $3N$ total) where the model selected the correct answer, and $p_{\text{accuracy}}$ is the per-question accuracy as a fraction between 0 and 1.

What it computes: For a single question with, say, 4 options, the model is tested 12 times (3 prompt wordings × 4 cyclic permutations). If the model answers correctly on 6 of those 12 trials, the per-question accuracy is $6/12 = 0.5$. The overall accuracy for a category or the full dataset is the average of these per-question accuracies across all questions in that subset.

Why this form: The cyclic permutation and multiple prompt wordings convert a single question into multiple quasi-independent trials, providing a more robust estimate of the model's knowledge than a single trial would. A model that truly knows the answer should answer correctly regardless of answer position and minor prompt variations, yielding accuracy near 1.0. A model that is guessing should show accuracy near $1/N$ (the random baseline) with high variance across trials and positions. A model with position bias might score well only when the correct answer happens to appear in the first position—but cyclic permutation ensures this advantage is averaged out, so position-biased models are penalized appropriately.

Model selection and experimental setup. The paper tests 13 model configurations spanning five open-source families and two proprietary APIs, detailed in Table 4:

• Polyglot-Ko (Ko et al., 2023): a family of Korean-pretrained autoregressive language models at four scales—1.3B, 3.8B, 5.8B, and 12.8B parameters—trained on a Korean-specific corpus of 863GB.
• KULLM (Lee et al., 2023b): Korean instruction-tuned models built by fine-tuning Polyglot-Ko with additional Korean instruction-following data, available at 5.8B and 12.8B.
• KoAlpaca: Korean instruction-tuned variants, also based on Polyglot-Ko, at 5.8B and 12.8B.
• LLaMA-Ko: The LLaMA model additionally pretrained on Korean data to create a Korean-specialized version, available at 7B and 13B parameters (though Table 4 lists only the 13B variant as evaluated—footnote: the paper's Table 4 actually shows LLaMA-Ko at 13B only, with LLaMA (base) at 7B and LLaMA-2-chat at 13B).
• LLaMA (base): the original LLaMA model at 7B for comparison.
• LLaMA-2-chat: the chat-optimized version at 13B.
• GPT-3.5-turbo and Claude-2: proprietary API models, evaluated between September and October 2023.

Why test across model families and scales? The experimental design enables two specific analyses:

1. Scale analysis within families: Comparing Polyglot-Ko 1.3B → 3.8B → 5.8B → 12.8B tests whether simply increasing model parameters within the same pretraining regime improves Korean cultural knowledge (it doesn't, as the ANOVA results in Table 6 confirm).
2. Korean corpus effect analysis: Comparing Polyglot-Ko (Korean-pretrained) against KULLM and KoAlpaca (same base but additional Korean instruction-tuning) tests whether additional Korean-specific training improves cultural knowledge (it doesn't, and in some cases accuracy decreases). Comparing LLaMA (mostly English-pretrained, only 0.06% Korean data) against Korean-specialized models tests whether Korean-language pretraining proportion matters for cultural knowledge (it doesn't, with LLaMA-2-chat outperforming most Korean-specialized models despite minimal Korean training data).

3.4.6 Design Rationale: Why This Specific Construction Methodology?

The paper makes several explicit and implicit design choices that warrant analysis, as they reflect tradeoffs between authenticity, scalability, coverage, and evaluation rigor.

Choice 1: Use authentic Korean exams rather than create questions from scratch or translate. This is the most fundamental methodological commitment. By sourcing questions from official Korean examinations, CLIcK inherits:

• Domain validity: The questions test knowledge that the Korean educational system has formally identified as important. This is particularly valuable for human comparisons—when the paper reports that GPT-3.5 scores in the lowest 11th percentile on the CSAT, the percentile is computed against actual Korean students who took the same exam, providing a concrete, interpretable benchmark.
• Difficulty calibration: Exam questions are designed with specific difficulty levels in mind, filtered through the test-design expertise of Korean educational institutions. The paper doesn't need to independently validate question difficulty—it's baked in.
• Cultural grounding: Questions about Korean kinship terminology (e.g., "What does a husband call his wife's older brother?") emerge naturally from the Korean language and society exam, not from researchers trying to think of "culturally Korean" questions.

The tradeoff is that exam questions are limited to what Korean institutions test, which may not cover all aspects of cultural knowledge (e.g., informal cultural practices, regional dialects, internet culture not yet formalized in exams).

Choice 2: Use GPT-4 to generate questions from textbook content rather than manually authoring. This approach provides scalability—the KIIP textbook contains multiple chapters spanning diverse cultural domains, and manual question authoring by Korean native speakers would require subject-matter expertise across all these domains. The cite field requirement (linking each question to a specific source sentence) provides a verification mechanism: human validators can check whether the question is actually answerable from the cited text and whether GPT-4 introduced hallucinated content.

The tradeoff is that GPT-4-generated questions may have different characteristics from human-authored exam questions—different linguistic patterns, different distractor quality, different difficulty distributions. The paper mitigates this through the rigorous validation protocol, but residual differences may exist. Notably, only 184 of 1,995 questions (9.2%) come from the textbook GPT-4 generation, making this a minor component of the overall benchmark.

Choice 3: The multi-round, multi-annotator validation protocol with a 37.1% rejection rate. This is the quality-vs-quantity tradeoff made explicit. By rejecting over a third of candidate questions, the paper prioritizes benchmark reliability over benchmark size. The specific protocol—requiring two-of-three annotator agreement for rejection in the first round, then full consensus for final inclusion—is designed to balance efficiency (not requiring all annotators to review every question) with stringency (no question enters the final dataset without unanimous agreement from all four annotators).

Choice 4: Categorization using established frameworks (KIIP curriculum structure, Bachman and Palmer's model) rather than emergent or data-driven categories. This provides external validity: the categories are grounded in established educational and linguistic theory, not in post-hoc clustering of question content. This means that model deficiencies can be interpreted within these existing frameworks—for instance, poor performance on "Functional Knowledge" questions has a specific, theoretically-grounded meaning related to understanding communicative intent, not just "questions about speaking and writing." The tradeoff is that some questions may not fit neatly into these categories, and the exclusion of socio-linguistic knowledge from the linguistic framework (subsumed into cultural categories) may blur boundaries.

Choice 5: Cyclic permutation and multiple prompt wordings for evaluation robustness. This addresses two well-known evaluation pitfalls simultaneously: position bias (models preferring certain answer positions) and prompt sensitivity (model performance varying with subtle wording changes). The $3N$ trials per question provides a more reliable accuracy estimate than a single-trial approach would, at the cost of $3N$ times the computation. For the 1,995-question dataset with an average of 4 options and 13 models, this means approximately 1,995 × 12 × 13 = 311,220 individual model inferences—a substantial but manageable computational cost.

Choice 6: Temperature 0 for API models with lenient acceptance criteria. Setting temperature to 0 means the models use greedy decoding, removing stochasticity from the evaluation. The lenient acceptance criteria (accepting verbose responses that clearly indicate the answer) is a practical necessity—proprietary models often produce explanatory text even when asked for a single letter—but introduces a subtle asymmetry: open-source models are evaluated on probability ranking (which always produces a valid option), while API models might have responses that fail the acceptance criteria and are counted as incorrect. This makes the open-source evaluation somewhat more "generous" in that it always selects the most probable option, while API models can "refuse to answer" in a way that counts against them.

Choice 7: Accuracy as the sole metric with no calibration or confidence reporting. The paper reports only accuracy—the fraction of trials where the model selects the correct answer. It does not report calibration metrics (whether model probability estimates align with actual correctness frequencies) or confidence intervals on accuracy estimates. The uncertainty analysis in Section 5.1 (Shannon entropy of model predictions) provides some insight into model confidence, but this is a post-hoc analysis, not an evaluation metric. For a 500-question evaluation, accuracy estimates have binomial confidence intervals that should be reported but aren't—for a category with 100 questions, an observed accuracy of 40% has a 95% confidence interval of approximately ±9.6 percentage points, which is large enough to affect the interpretation of differences between models.

Summary of the technical approach. CLIcK's construction methodology is characterized by five key principles: (1) authentic Korean sourcing from official examinations and government textbooks, ensuring cultural validity through institutional provenance; (2) scalable question generation using GPT-4 for textbook content, constrained by a citation requirement that enables human verification; (3) rigorous multi-round human validation with explicit, documented rejection criteria that eliminated 37.1% of candidate questions; (4) theory-grounded categorization into 11 subcategories using established frameworks from Korean cultural education (KIIP curriculum) and linguistic competence modeling (Bachman and Palmer, 1996); and (5) robust evaluation through cyclic permutation and multiple prompt wordings that neutralize position bias and prompt sensitivity, yielding reliable per-question accuracy estimates at the cost of $3N$-fold inference. The evaluation framework tests 13 model configurations spanning parameter scales from 1.3B to proprietary frontier models, enabling statistical analysis of whether scale and Korean-specific training improve cultural understanding—an analysis that reveals they do not.

4. Key Insights and Innovations

Innovation 1: Cultural Knowledge Is Not a Byproduct of Scale—And the Field Has Been Testing for the Wrong Thing

The paper's most intellectually disruptive contribution is a negative result with positive implications: the empirical demonstration that neither increasing model parameters nor adding more Korean pretraining data reliably improves performance on questions requiring Korean cultural and linguistic knowledge. The ANOVA results in Table 6 tell a stark story: model scale has no statistically significant effect (F₁,₄₃ = 0.33, p = .57), and Korean corpus supplementation similarly fails to move the needle (F₁,₅₄ = 0.30, p = .59). This flatlines the dominant assumption in multilingual LLM development—that cultural competence will emerge naturally from larger models trained on larger, more diverse datasets.

What the field assumed before this work. The implicit logic driving most non-English LLM development has been: (1) pretrain a large model on a multilingual corpus, (2) optionally fine-tune on language-specific data, (3) evaluate on translated benchmarks, and (4) assume that good performance on those benchmarks indicates genuine linguistic and cultural competence. This assumption is baked into the evaluation infrastructure itself—the Open Ko-LLM Leaderboard, which the paper explicitly critiques (Section 2.3), evaluates Korean models primarily on tasks derived from English through translation. Under this paradigm, cultural knowledge is treated as a special case of general knowledge—just more facts to absorb from more text.

What CLIcK reveals instead. The paper's finding that LLaMA-2-chat—trained on a corpus where Korean constitutes only 0.06% of the data—outperforms most Korean-specialized models (Polyglot-Ko, KULLM, KoAlpaca) despite their orders-of-magnitude greater Korean data exposure is the clearest empirical refutation of the "more Korean data = more Korean cultural knowledge" assumption. If additional Korean pretraining reliably taught cultural knowledge, the Korean-specialized models should dominate. They don't. If larger models within the same Korean-training regime reliably acquired more cultural knowledge, the 12.8B Polyglot-Ko should consistently outperform the 1.3B version. It doesn't.

This is not a marginal finding—it's a category error in how the field conceptualizes cultural knowledge acquisition. The paper's results suggest that cultural competence is not simply a matter of absorbing more text in the target language. Something structural is missing: either the training data itself doesn't contain the right kind of cultural information (because it's drawn from web text skewed toward English-centric content even when in Korean), or the training objective (next-token prediction) doesn't extract cultural knowledge in a way that supports answering explicit knowledge questions, or both.

Why this is a fundamental contribution, not an incremental refinement. Prior work had theorized that language models lack cultural sensitivity (Lee et al., 2023a) and that integrating cultural knowledge improves performance (Ma et al., 2022). But no prior work had built a broad-coverage, culturally authentic benchmark that could systematically test whether the scaling-and-data paradigm works for cultural knowledge across diverse domains. CLIcK provides the instrument that converts these theoretical concerns into falsifiable empirical claims, and the results falsify the scaling assumption. This shifts the burden of proof: the field can no longer assume that bigger models with more data will eventually solve cultural competence. Researchers must now demonstrate that their approaches actually work on benchmarks like CLIcK, or propose fundamentally different methods for instilling cultural knowledge.

Innovation 2: The Cultural-Authenticity-by-Construction Principle for Non-English Benchmarks

The paper's second conceptual contribution is a methodological principle for non-English benchmark construction that challenges the translation-centric approach dominating the field. Rather than asking "How well does this Korean model perform on Korean translations of English benchmarks?", CLIcK asks "Does this model possess the cultural and linguistic knowledge that Korean society itself has deemed important enough to test formally?" The distinction is not cosmetic—it reflects a fundamental shift in what the benchmark is evaluating.

The dominant paradigm: translation as evaluation. Most non-English benchmarks follow a pattern: take an existing English benchmark (HellaSwag, MMLU, GLUE tasks), translate it into the target language (or generate translations via machine translation), and evaluate models on the translated version. The multilingual benchmarks the paper reviews in Section 2.2—XGLUE (Liang et al., 2020), XTREME (Hu et al., 2020), MEGA (Ahuja et al., 2023)—all operate within this paradigm. Even Korean-specific efforts like the Open Ko-LLM Leaderboard rely heavily on translated tasks.

The problem, which the paper diagnoses clearly, is that translation preserves linguistic surface form while potentially carrying over culturally-specific content that is alien to the target culture. A Korean translation of a question about U.S. Thanksgiving traditions tests whether the model can reason about Thanksgiving in Korean—but Thanksgiving is not part of Korean cultural knowledge, and a Korean speaker should not be expected to know about it as cultural commonsense. Conversely, questions about Korean cultural practices (Chuseok holidays, kinship terminology, Korean legal procedures) are systematically absent from translation-derived benchmarks because they don't exist in the English source materials.

CLIcK's alternative: institutional provenance as cultural validation. By sourcing questions from official Korean examinations and government textbooks—materials designed by Korean educators for Korean test-takers—CLIcK adopts what we might call cultural-authenticity-by-construction: the benchmark's cultural validity derives not from researcher judgment about what constitutes "Korean culture," but from the institutional processes of the Korean educational and governmental systems. The CSAT is designed by the Korean Ministry of Education to assess what Korean high school graduates should know. The KIIP textbook is designed by the Korean immigration service to teach immigrants what they need to know to integrate into Korean society. TOPIK is the official Korean language proficiency certification. When CLIcK tests models on these materials, it is testing them against the same standard that Korean society applies to humans.

This principle has a specific, measurable consequence: it enables direct model-human comparison on culturally meaningful scales. The paper's finding that GPT-3.5 scores in the lowest 11th percentile on the Korean CSAT while scoring in the top 13th percentile on the English SAT (Section 5.2) is interpretable precisely because the CSAT is a real exam with a known human score distribution. If CLIcK were a researcher-designed quiz, "11th percentile" would be meaningless—there is no population to compare against. The institutional sourcing provides the reference distribution that makes the performance gap legible as a real-world deficiency, not just a benchmark score.

The KoBBQ precedent and how CLIcK extends it. The paper acknowledges Jin et al.'s (2023) KoBBQ dataset as a prior effort that recognized the translation problem. KoBBQ classified translations into three categories (Simply-Translated, Sample-Removed, Target-Modified) based on whether cultural adaptation was needed, then selectively modified content to align with Korean cultural context. This was an adaptation methodology—it took an existing English benchmark and culturally adjusted it. CLIcK goes further: it constructs the benchmark from Korean-native sources entirely, abandoning the translation pipeline altogether. The cultural authenticity is inherent in the source material rather than achieved through post-hoc modification.

Why this is more than a dataset contribution. The paper isn't simply releasing another Korean dataset—it's providing a template for cultural benchmark construction that could be replicated for other languages and cultures. The principle—identify official institutional assessments of cultural knowledge in the target culture, source questions from them, validate and categorize—is transferable to any language with formal examination systems or cultural education curricula. This makes CLIcK not just a product but a process innovation with implications for how the field builds non-English evaluation resources.

Innovation 3: Fine-Grained Cultural Knowledge Taxonomy as a Diagnostic Instrument

The third contribution is the categorization framework itself—the decomposition of "Korean cultural and linguistic knowledge" into eleven operationally defined subcategories—and the demonstration that this decomposition enables diagnosis of where models fail, not just that they fail. Prior Korean benchmarks evaluated models on aggregate task performance; CLIcK provides the first instrument for identifying which specific domains of cultural knowledge are systematically absent from language models.

What the taxonomy enables that aggregate scores obscure. If CLIcK reported only an overall accuracy of, say, 35% for open-source models, the finding would be important but undiagnostic—it tells developers the model has a problem but not what kind of problem. The category-level breakdown in Table 5 transforms this into actionable intelligence:

• On Pop Culture questions, GPT-3.5 achieves 68.61% accuracy—dramatically outperforming its performance on History (30.75%) and Functional Knowledge (32.62%). This pattern is interpretable: K-pop, K-dramas, and Korean cinema have massive international reach, meaning GPT-3.5's English-heavy pretraining data likely contains substantial discussion of Korean popular culture. In contrast, Korean history and Korean functional linguistics (knowing the communicative intent behind specific Korean expressions) are domains with almost no English-language coverage.
• On Law questions, both proprietary and open-source models perform relatively well (GPT-3.5 at 55.31%, Claude-2 at 57.09%, open-source models in the 38-45% range)—higher than most other cultural categories. The paper doesn't explore this in detail, but it suggests that legal knowledge may transfer more readily across languages (legal systems share structural features) or that Korean legal terminology appears more frequently in the models' training data.
• On Textual Knowledge (comprehending Korean discourse structure, rhetorical patterns, literary texts), GPT-3.5 achieves 53.19% while open-source models hover around 23-26%—a gap of nearly 30 percentage points in the linguistic category. This suggests that Korean textual comprehension—understanding how Korean texts are organized as coherent wholes—is a capability that scales with model capacity and English-pretraining exposure in ways that factual cultural knowledge does not.
• On Functional Knowledge (understanding communicative roles and pragmatic functions of Korean utterances), all models struggle deeply—even GPT-3.5 reaches only 32.62%, barely above the 25% random baseline. This is the category most resistant to both scaling and Korean-specific training, suggesting that pragmatic competence in Korean (knowing what a speaker is doing with their words) may require interactive language experience that statistical pretraining cannot provide.

The theoretical grounding and why it matters. The paper's decision to ground its linguistic categories in Bachman and Palmer's (1996) model of communicative language ability—a well-established framework in language testing—gives the taxonomy external validity beyond the paper's own analysis. When the paper reports that models fail on Functional Knowledge, it is using a term with specific theoretical content (knowledge of ideational, manipulative, heuristic, and imaginative language functions), not an ad-hoc label. This means future researchers can build on CLIcK's findings using the same theoretical vocabulary, connecting LLM evaluation to decades of work in language assessment.

Similarly, the cultural categories following the KIIP textbook's structure provide a taxonomy that reflects how Korean institutions themselves partition cultural knowledge. This is not the paper's authors deciding what "counts" as Korean cultural knowledge—it's the Korean government's official curriculum for cultural integration, adopted as the evaluation framework.

The unanticipated diagnostic: model uncertainty patterns by category. The paper's entropy analysis in Figure 3 provides a complementary diagnostic dimension: not just whether models get questions wrong, but how they get them wrong. Smaller models (Polyglot 1.3B, 3.8B) show high entropy—they guess randomly across options. Larger models within the same family show lower entropy on wrong answers—they consistently select the same wrong answer, suggesting they have learned systematic but incorrect associations. This is a more dangerous failure mode: a model that guesses randomly is obviously unreliable; a model that confidently selects the wrong answer with high consistency appears knowledgeable while being systematically wrong. The shift from high-entropy random guessing to low-entropy confident errors as model scale increases is an important diagnostic pattern that the categorization framework makes visible across different knowledge domains.

Innovation 4: Human-Examination Anchoring as a Legibility Mechanism for Model Deficits

The fourth contribution is a methodological innovation in evaluation reporting: using existing human examination statistics to translate model accuracy scores into percentile ranks against actual test-takers, thereby making model deficits legible to non-technical audiences in terms they already understand.

Why raw accuracy numbers mislead. Reporting that GPT-3.5 achieves 45% accuracy on CSAT-Korean questions is a technically accurate but informationally impoverished statement. Is 45% good? Bad? What does it mean? Without a human reference point, accuracy scores on researcher-constructed benchmarks are abstract numbers whose real-world significance is unclear. A model achieving 80% on a benchmark might be near-perfect if humans average 82%, or abysmal if humans average 98%.

CLIcK's solution: institutional score distributions as interpretability infrastructure. By sourcing questions from exams with publicly available human score statistics, CLIcK enables statements like:

• GPT-3.5 scores in the lowest 11th percentile on the CSAT Korean exam—meaning it performs worse than 89% of Korean high school seniors on their national university entrance exam.
• Claude-2 achieves TOPIK level 6, the highest attainable level for non-native Korean speakers, indicating it "can perform language functions required for specialized research or professional tasks relatively accurately and fluently" while still noting it "doesn't reach the proficiency of a native speaker."
• All open-source Korean models tested fall below the measurable range on TOPIK, meaning their Korean language proficiency is too low to be assigned a level.
• On the Kedu exam (certification for teaching Korean as a foreign language), all models score below the human average of 49.9%, with proprietary models approximately 10 percentage points below and open-source models more than 20 points below.

These statements have immediate practical meaning. "GPT-3.5 performs at the 11th percentile on the Korean CSAT" tells a Korean parent or educator exactly what that means in a way that "47% accuracy on CLIcK" never could. The institutional anchoring converts abstract model performance into the same evaluation framework that Korean society uses to assess human competence, making the results interpretable to policymakers, educators, and the Korean public—not just NLP researchers.

Why this is distinct from prior work. Benchmarks like MMLU (Hendrycks et al., 2021) and AGIEval (Zhong et al., 2023) also source questions from human examinations, and AGIEval specifically includes some Korean exams. But CLIcK's contribution is the systematic use of exam-specific human score distributions for cross-comparison across multiple exam types and difficulty levels, combined with the cultural-authenticity principle. AGIEval reports model-vs-human accuracy comparisons; CLIcK provides exam-specific percentile rankings that account for the actual difficulty distribution of each exam as experienced by human test-takers. A question that 90% of CSAT test-takers answer correctly and one that 30% answer correctly are different kinds of failures for a model, and the percentile anchoring captures this in a way raw accuracy cannot.

The limitation and its productive tension. The paper acknowledges a limitation of this approach: "our dataset does not encompass the actual score distribution for the problems, a simple score conversion, the ratio of correctly answered questions, is applied" (Section 5.2). The percentile rankings are approximated by assuming that accuracy percentage maps linearly to percentile within the exam distribution—a simplification, since actual exam scoring involves item-level difficulty calibration that the paper doesn't replicate. But this limitation actually highlights the innovation's significance: even with this simplification, the human-anchored comparisons provide substantially more insight than context-free accuracy reporting. A future benchmark that incorporated actual item response theory parameters from the source exams would strengthen this approach further.

5. Experimental Analysis

Evaluation Methodology

• Dataset. CLIcK comprises 1,995 QA pairs sourced from six official Korean examinations and the KIIP government textbook, partitioned into 11 subcategories across Cultural Intelligence (8 categories: Society, Tradition, Pop Culture, Politics, Economy, Law, History, Geography) and Linguistic Intelligence (3 categories: Textual Knowledge, Functional Knowledge, Grammatical Knowledge). The dataset statistics are reported in Table 1 and Table 2. There is no explicit train/validation/test split because CLIcK is purely an evaluation benchmark—all 1,995 questions are used for testing only, with no training or fine-tuning conducted on the dataset.

• Base models. The paper evaluates 13 model configurations across five open-source families and two proprietary APIs. Open-source models include: Polyglot-Ko at four parameter scales (1.3B, 3.8B, 5.8B, 12.8B), Korean-instruction-tuned variants KULLM (5.8B, 12.8B) and KoAlpaca (5.8B, 12.8B) built on Polyglot-Ko, Korean-specialized LLaMA-Ko (13B), and the general LLaMA family (LLaMA-7B base, LLaMA-2-chat-13B). Proprietary models include GPT-3.5-turbo and Claude-2, evaluated between September and October 2023. The selection spans parameter scales from 1.3B to >100B and includes models with varying degrees of Korean-language training exposure, enabling analysis of both scale effects and Korean corpus effects on cultural knowledge.

• Metrics. The primary metric is per-question accuracy, computed as $p_{\text{accuracy}} = \frac{\text{count}(\text{correct answers})}{3N}$ where $N$ is the number of options (Equation 1, Section 4). For each question, the model is tested $3N$ times (3 prompt wordings × $N$ cyclic permutations of answer choices), and accuracy is the fraction of these trials where the correct answer is selected. Overall accuracy for a category or the full dataset is the average of per-question accuracies. A secondary analysis in Section 5.1 uses normalized Shannon entropy to quantify model uncertainty: $\text{Uncertainty score} = -\frac{1}{\log N}{\sum_{i\in\text{options}}p_{i}\log p_{i}}$ where $p_i = \frac{\text{count}(i)}{3N}$, producing a score between 0 (perfect consistency, always selecting the same option) and 1 (uniform random guessing).

• Baselines. The paper does not employ traditional algorithmic baselines since this is an evaluation benchmark, not a method-improvement paper. The implicit baseline is random chance: accuracy of $1/N$ (typically 25% for 4-option questions, 20% for 5-option questions). All model performances are interpretable relative to this random baseline. The paper also uses human exam-taker performance as a natural reference point via the official score distributions of the source examinations (Section 5.2): CSAT percentile rankings, TOPIK proficiency levels, and Kedu average scores provide human-calibrated comparisons.

• Generation budget / compute accounting. There is no generation budget in the traditional sense—testing is a fixed-cost evaluation where each model answers each question $3N$ times. For open-source models, predictions are derived from output token probabilities (the model computes log-probabilities for option identifier tokens concatenated with option strings), so inference cost is a single forward pass per question per cyclic permutation. For API models, predictions are generated with temperature 0 (deterministic greedy decoding), producing a single token or short response per prompt. The total inference count across 13 models and 1,995 questions with an average of 4 options is approximately 311,220 individual model inferences.

• Cross-validation / statistical protocol. The paper does not employ cross-validation (consistent with an evaluation-only benchmark where no model training occurs). For the ANOVA analyses in Table 6 testing whether model scale or Korean corpus size significantly affects accuracy, the paper uses standard F-tests: F₁,₄₃ = 0.33, p = .57 for model scale effect and F₁,₅₄ = 0.30, p = .59 for Korean corpus effect, both failing to reject the null hypothesis of no effect at conventional significance levels. The uncertainty analysis in Figure 3 reports per-model entropy distributions on challenging samples using box-and-whisker plots with mean and median markers. For human-comparable exams (Section 5.2), the paper uses officially reported human score distributions averaged across multiple years: 2017–2020 for CSAT (five years), a five-year period for TOPIK, and 2014–2018 for Kedu (five years).

Main Quantitative Results

The paper's experimental results are organized around three axes of analysis: (1) overall model accuracy by category, (2) the effects of model scale and Korean corpus training on performance, and (3) human-level comparisons using exam-derived score distributions. There is no separate result grouping for search, revisions, or FLOPs-matched comparison since CLIcK is a pure evaluation benchmark without proposed methods to ablate or optimize.

Overall Model Performance by Category

Table 5 presents the central quantitative result: the accuracy of all 13 model configurations broken down across the 8 Cultural Intelligence categories and 3 Linguistic Intelligence categories. The headline findings are:

Open-source models systematically underperform. Across the Cultural Intelligence categories, open-source models (Polyglot-Ko, KULLM, KoAlpaca, LLaMA-Ko, LLaMA variants) achieve accuracies predominantly in the 22–45% range, with considerable variation by category. The Linguistic Intelligence categories show even lower performance, with open-source models consistently scoring in the 19–31% range—close to the 25% random baseline for 4-option questions. The highest individual open-source performance in any category is 46.27% (LLaMA-2-chat-13B on Economy), and the lowest is 19.38% (KULLM-12.8B on Functional Knowledge, effectively below random chance for 5-option questions).

Proprietary models substantially outperform open-source models but remain far from ceiling. GPT-3.5-turbo achieves 49.30% average on Cultural Intelligence and 42.32% on Linguistic Intelligence across all categories. Claude-2 achieves 51.72% on Cultural Intelligence and 45.39% on Linguistic Intelligence. These are substantial improvements over open-source models (roughly 15–20 percentage point gaps in most categories), but they represent accuracies far below what would be expected of a system with genuine cultural competence—a Korean high school student scoring 50% on the CSAT would be in the lowest performance tiers.

Category-level performance reveals systematic knowledge disparities. Several patterns in Table 5 are noteworthy:

• Pop Culture is a relative strength for proprietary models. GPT-3.5 achieves 68.61% on Pop Culture—its highest score in any category—while Claude-2 achieves 59.56%. This is the only category where GPT-3.5 outperforms Claude-2 across all examined categories. Open-source models also perform relatively well on Pop Culture (27–36% range), though the gap with proprietary models is the largest of any category (roughly 30 percentage points). This pattern is consistent with Korean popular culture's global reach through K-pop, K-dramas, and Korean cinema, which likely provide substantial English-language training signal that proprietary models can leverage but smaller Korean-specialized models, trained on narrower Korean corpora, capture less effectively.

• Economy is the highest-scoring category for open-source models. All open-source models achieve their best Cultural Intelligence performance on Economy questions, with scores in the 42–46% range (Polyglot-Ko 12.8B: 45.03%; LLaMA-2-chat-13B: 46.27%). This is notable because Economy questions presumably require domain-specific Korean terminology (financial terms, policy names, economic concepts), yet models perform better here than on ostensibly more "everyday" categories like Society (32–41%) or Tradition (32–36%). The paper does not analyze this pattern in detail, but it may reflect that economic terminology is more internationally standardized and thus more represented in multilingual training data than culturally-specific social or traditional concepts.

• History is paradoxically difficult for all models. Despite Korean history being a well-documented field with presumably substantial Korean-language textual coverage, all models perform poorly on History questions: open-source models score 24–26%, GPT-3.5 scores 30.75%, and Claude-2 scores 31.32%. This is the second-lowest Cultural Intelligence category for GPT-3.5 (after Pop Culture, where it excels). The paper does not explore why History underperforms relative to Economy or Law, but the finding suggests that historical knowledge—which requires temporal reasoning about specific events, figures, and causal relationships—may be fundamentally harder for LLMs to acquire from pretraining than more structural knowledge like legal or economic frameworks.

• Functional Knowledge is the hardest Linguistic Intelligence category. Across all models, Functional Knowledge—understanding the communicative roles and pragmatic functions of Korean utterances—yields the lowest scores of the three linguistic categories. GPT-3.5 achieves only 32.62%, Claude-2 achieves 32.88%, and open-source models range from 19.38% (KULLM-12.8B) to 30.50% (LLaMA-13B base). These scores are barely above random guessing (25% for 4-option questions, 20% for 5-option questions), suggesting that pragmatic competence in Korean—knowing what a speaker is doing with their words in context—is almost entirely absent from current models regardless of scale or training data composition.

• Textual Knowledge shows the largest proprietary-vs-open-source gap in Linguistic Intelligence. GPT-3.5 achieves 53.19% and Claude-2 achieves 55.86% on Textual Knowledge, while open-source models cluster at 22–27%. This ~30 percentage point gap is the largest in the Linguistic Intelligence categories and suggests that comprehending extended Korean discourse—understanding rhetorical structure, cohesion, and literary texts—is a capability that scales with model capacity and broad pretraining in ways that more discrete knowledge (grammar rules, vocabulary) does not.

Claude-2 outperforms GPT-3.5 in 9 of 11 categories. The exceptions are Geography (45.42% vs. 43.30% for Claude-2) and Pop Culture (68.61% vs. 59.56% for Claude-2), both categories where GPT-3.5 holds an advantage. Across the remaining nine categories (History, Law, Politics, Society, Tradition, Economy, Textual, Functional, Grammar), Claude-2 consistently scores 1–13 percentage points higher. The paper does not report whether these differences are statistically significant given the per-category sample sizes.

The "best open-source" model varies by category. No single open-source model dominates across all categories. LLaMA-2-chat-13B achieves the highest score in Economy (46.27%) and is competitive in Society (40.93%) and Pop Culture (36.41%). The LLaMA-7B base model achieves the highest score in Geography (33.21%) and Functional Knowledge (30.50%). KULLM-5.8B achieves the highest score in Tradition (35.86%) and Functional Knowledge (24.76%). This inconsistency suggests that no open-source model has robust Korean cultural knowledge—each may have incidentally absorbed different subsets of cultural information from its training data.

Scale and Training Data Analysis

Table 6 reports the results of two ANOVA tests designed to answer the central questions: does making models larger within the same architecture family improve Korean cultural and linguistic knowledge, and does additional Korean-language training data improve performance?

Model scale has no statistically significant effect. Testing across Polyglot-Ko (1.3B through 12.8B), KULLM (5.8B and 12.8B), KoAlpaca (5.8B and 12.8B), and LLaMA variants, the paper finds F₁,₄₃ = 0.33 with p = .57. This means that for models within the same family and training regime, going from 1.3B to 12.8B parameters does not reliably increase CLIcK accuracy. This finding is visible in the raw numbers: Polyglot-Ko 1.3B achieves 32.71% average on Cultural Intelligence, while Polyglot-Ko 12.8B achieves 33.40%—a difference of less than one percentage point. Similarly, KULLM-5.8B achieves 33.79% and KULLM-12.8B achieves 33.51%—actually a slight decrease.

Additional Korean corpus training has no statistically significant effect and sometimes hurts performance. Comparing base Polyglot-Ko models against their instruction-tuned derivatives (KULLM, KoAlpaca), which received additional Korean-language fine-tuning, the paper finds F₁,₅₄ = 0.30 with p = .59. More strikingly, the paper notes that "the accuracy is even lower than that of their base models" for some comparisons. For example, Polyglot-Ko 12.8B achieves 33.40% on Cultural Intelligence; its instruction-tuned derivative KULLM-12.8B achieves 33.51% (effectively flat), while KoAlpaca-12.8B drops to 33.80%—still within noise range. In Linguistic Intelligence, the pattern is more pronounced: Polyglot-Ko 12.8B achieves 23.78%; KULLM-12.8B drops to 22.42%; KoAlpaca-12.8B drops to 22.24%.

LLaMA-2-chat-13B outperforms Korean-specialized models despite minimal Korean training data. The paper highlights a specific comparison: LLaMA-2-chat, with only 0.06% of its training data from Korean sources, outperforms most models that were either pretrained on Korean corpora (Polyglot-Ko, LLaMA-Ko) or additionally fine-tuned with Korean instruction data (KULLM, KoAlpaca). On Cultural Intelligence, LLaMA-2-chat-13B achieves 36.22% vs. Polyglot-Ko-12.8B's 33.40% and LLaMA-Ko-13B's 33.26%. On Linguistic Intelligence, LLaMA-2-chat-13B achieves 26.71% vs. Polyglot-Ko-12.8B's 23.78%. The paper interprets this as evidence that "simply amassing more data and enlarging the model size (currently common practices in LM) may not be the optimal solution for enhancing the cultural intelligence of Language Models in non-English languages" (Section 4.1).

Difficulty and Uncertainty Analysis

Figure 2 reports the proportion of the CLIcK dataset that each model finds challenging, defined as questions where model accuracy falls below the random selection threshold (1/N). The proportional analysis reveals:

• Open-source models struggle with over 60% of CLIcK. Polyglot-Ko, KULLM, KoAlpaca, LLaMA-Ko, and LLaMA-2-chat all have difficulty with more than 60% of the 1,995 questions. The paper identifies a shared difficulty set spanning 35.5% of the dataset—meaning more than a third of CLIcK questions are challenging for all open-source models simultaneously.

• Proprietary models still struggle with over 30% of the dataset. GPT-3.5 and Claude-2 face challenges on more than 30% of CLIcK questions, with a shared difficulty set of 12.6%—questions that neither proprietary model can reliably answer.

• Only 0.6% of CLIcK is universally unsolvable. Questions that no model—open-source or proprietary—could answer correctly constitute just 0.6% of the dataset. This means the benchmark is not impossibly difficult; the knowledge exists somewhere in the model ecosystem, but it is fragmented—distributed across different models rather than concentrated in any single one.

Figure 3 provides the uncertainty analysis for challenging samples, showing per-model entropy distributions:

• Smaller models exhibit high uncertainty (random guessing). Polyglot-Ko 1.3B and 3.8B have median uncertainty scores near 1.0, indicating they select answers essentially randomly on questions they get wrong—they don't know the answer and their predictions reflect this uncertainty.

• Larger models exhibit low uncertainty (confident errors). As model scale increases within the Polyglot-Ko and LLaMA families, median uncertainty scores shift toward 0—models become increasingly consistent in their wrong answers. A Polyglot-Ko 12.8B model getting a question wrong does so with high confidence, consistently selecting the same incorrect answer across prompt variations and position permutations. This is a more dangerous failure mode: a model that guesses randomly is obviously unreliable, while a model that confidently selects wrong answers appears knowledgeable but is systematically incorrect.

• GPT-3.5 and Claude-2 show ambiguous uncertainty patterns. Both proprietary models have mean and median uncertainty scores approximating 0.5, indicating a mix of confident errors and uncertain guesses on their challenging samples. The interpretation is less clear than for the open-source models, but the 0.5 median suggests these models are not uniformly confident in their wrong answers—they exhibit more variation in their error patterns.

Human-Level Comparison Results

Section 5.2 compares model performance to actual human test-taker statistics on three examinations for which official score distributions are publicly available. These comparisons use a simple score conversion—the percentage of correctly answered questions—mapped onto the exam's existing grading framework.

CSAT Korean results. The CSAT uses a 9-tier stanine system where tier 1 represents the top 4% of test-takers and tier 9 represents the bottom 4%. Averaged across 2017–2020 statistics (five years of data):

• Polyglot-Ko 12.8B, KULLM 5.8B, KULLM 12.8B, and KoAlpaca 12B all score at the 9th level—the lowest 4% of Korean high school seniors. These models perform worse than 96% of college-bound Korean students on their own national language examination.
• The remaining open-source models score at the 8th level, representing the lowest 11% of test-takers.
• GPT-3.5 scores in the lowest 11th percentile, which the paper explicitly contrasts with its top 13th percentile performance on the English SAT, demonstrating the stark asymmetry in cultural competence across languages for the same model.

TOPIK results. TOPIK uses an absolute proficiency scale with levels 1–6, where higher levels indicate greater Korean language mastery. The proficiency descriptions are cumulative: level 6 indicates near-native functional ability, level 5 indicates appropriate register differentiation, and levels below 3 indicate basic survival Korean:

• Claude-2 achieves TOPIK level 6, the highest attainable level. The paper describes this as indicating the model "can perform language functions required for specialized research or professional tasks relatively accurately and fluently. While it doesn't reach the proficiency of a native speaker, it doesn't face difficulties in functional performance or expressing meanings."
• GPT-3.5 achieves TOPIK level 5, indicating it "can appropriately differentiate language use depending on formal and informal, as well as spoken and written contexts."
• All open-source models fall below the measurable range. Their Korean language proficiency is too low to be assigned any TOPIK level, meaning they cannot reliably perform even the basic communicative functions assessed at level 1.

This is a striking asymmetry: Claude-2, an English-native model, achieves the highest possible Korean proficiency certification for non-native speakers, while models purpose-built for Korean (Polyglot-Ko, KULLM, KoAlpaca, LLaMA-Ko) cannot even qualify for the lowest level. The TOPIK evaluation measures functional language use (production and comprehension), not just factual knowledge, which may explain why broad-capability proprietary models trained on massive multilingual data outperform narrower Korean-specialized models—language proficiency may transfer across languages in ways that cultural factual knowledge does not.

Kedu for Korean Culture results. The Kedu exam certifies individuals to teach Korean as a foreign language and covers both Korean language and Korean culture content. Using five-year averaged statistics from 2014–2018:

• The human average score is 49.9% correct answers.
• Claude-2 scores 39.1% and GPT-3.5 scores 37.0% — both approximately 10 percentage points below the human average.
• All open-source models score more than 20 percentage points below the human average (i.e., below ~30% accuracy).

Every model tested falls below the mean performance of human test-takers on this exam, with proprietary models substantially closer but still deficient, and open-source models catastrophically behind. This is the exam that most directly tests the ability to teach Korean language and culture—a task requiring robust, integrated cultural and linguistic knowledge rather than isolated facts.

Ablation Studies and Robustness Checks

CLIcK is an evaluation benchmark, not a method-improvement paper, so traditional ablation studies (varying model architecture components, training data composition, hyperparameters) are absent. The paper instead provides several forms of robustness analysis that probe the reliability of its findings:

Prompt wording variation. The paper uses three different prompt wordings for each question (Section 4: "prompts are derived from Jin et al. (2023)... we report the average over three different wordings of the prompt"). This effectively serves as a prompt sensitivity check—if model performance varied wildly across prompt phrasings, the benchmark would be measuring prompt engineering skill rather than underlying knowledge. The paper does not report variance across the three prompt wordings, so the reader cannot assess whether some models are more prompt-sensitive than others. However, the fact that per-question accuracy is aggregated across three wordings means the reported scores are robust to moderate prompt variation—a model that only answers correctly with one specific phrasing will have its accuracy diluted by the other two, penalizing prompt-sensitive behavior.

Cyclic permutation for position bias mitigation. The $N$-fold cyclic permutation of answer choices (Section 4) serves as a position bias robustness check. A model with strong position bias (always selecting option A regardless of content) would score at approximately $1/N$ after cyclic permutation—the correct answer only appears in the preferred position once per $N$ trials. This means the reported accuracies are conservative estimates of knowledge: a model that genuinely knows the answer but also has position bias overlapping the correct position will be penalized on permutations where the correct answer moves to a non-preferred position. The paper does not report per-position accuracy breakdowns that would reveal the magnitude of position bias, which is a missed diagnostic opportunity.

Out-of-option handling for API models. The acceptance criteria for API model responses (Section 4) serve as an extraction robustness check. By defining four specific conditions under which a verbose response is mapped to a valid option, and counting all other responses as incorrect, the paper adopts a conservative approach: models that fail to follow the output format are penalized. The paper does not report the frequency of out-of-option responses for GPT-3.5 or Claude-2, so readers cannot assess whether poor API model performance in some categories might be partly attributable to answer extraction failures rather than knowledge gaps. This is a meaningful reporting gap—if, for example, 10% of Claude-2's responses on Textual Knowledge questions fail the acceptance criteria, the reported 55.86% accuracy understates its knowledge of those questions compared to an open-source model whose log-probability ranking always produces a valid option.

Cross-source consistency. The paper's dataset draws from seven distinct sources (six exams plus KIIP textbook). While not presented as a formal ablation, the per-source breakdown in Table 2 implicitly tests whether the difficulty and knowledge type varies systematically by source. Questions from different exams test different populations (high school students for CSAT, language learners for TOPIK, public servants for PSE/PSAT, history exam takers for KHE, prospective teachers for Kedu) and different knowledge domains. The fact that models show category-level patterns (e.g., poor History performance across all sources containing History questions) rather than source-level patterns (e.g., poor performance specifically on KHE-sourced questions) suggests the categorization framework is capturing genuine knowledge distinctions rather than exam-specific artifacts.

Multi-annotator validation as quality control. The three-phase annotation process with a 37.1% rejection rate (Section 3.2) functions as a data quality robustness mechanism rather than an experimental ablation. However, the paper does not report whether excluding the rejected 37.1% of candidate questions altered the difficulty distribution of the benchmark. If the rejected questions were systematically harder (because they contained ambiguities that caused annotator disagreement), the remaining benchmark would be artificially easier; if they were systematically easier but poorly formatted, the remaining benchmark would be artificially harder. Without this analysis, readers cannot assess whether the benchmark's difficulty is representative of the original source material or has been shaped by the validation process.

Temporal stability criterion. The second validation criterion—"Information in Questions remains consistent over time"—is a robustness commitment rather than a check. The paper excludes questions whose answers could change (current officials, recent events, time-sensitive statistics). This means CLIcK should remain valid over time without requiring updates—a model tested in 2024 and a model tested in 2028 should face the same correct answers. However, the paper does not verify empirically that all included questions satisfy this criterion beyond the annotator judgment, and some cultural knowledge (e.g., Pop Culture references) may age in ways that annotators did not anticipate.

Missing robustness analyses. Several checks that would strengthen the paper's claims are absent:

• Inter-annotator agreement metrics are not reported. The paper describes a multi-round validation process but provides only rejection rates (15.9% invalid by one annotator in Phase 1, 3.9% discrepancies after Phase 2) without standard inter-rater reliability statistics (e.g., Fleiss' kappa, Krippendorff's alpha). Without agreement metrics, the reader cannot assess whether the categories are applied consistently or whether the validation criteria are interpreted uniformly.

• No confidence intervals on accuracy scores. All accuracy numbers in Table 5 are point estimates without error bounds. For category-level scores based on sample sizes ranging from roughly 50 to several hundred questions, binomial confidence intervals would be informative. A difference of 2–3 percentage points between models in a 100-question category is likely within sampling error, but the paper treats all numerical differences as meaningful without qualifying their statistical reliability.

• No analysis of distractor quality. In multiple-choice evaluation, the plausibility of incorrect options (distractors) strongly affects difficulty. Questions from different exams may have different distractor design philosophies (TOPIK distractors designed for language learners vs. CSAT distractors designed for native-speaking high schoolers). The paper does not analyze whether model errors cluster on particular distractor types or whether some categories have systematically more plausible distractors, which would affect the interpretation of cross-category accuracy differences.

• No per-source calibration of difficulty. While Table 2 provides per-source statistics, the paper does not test whether some exam sources are systematically harder than others for models. If, for example, CSAT questions are harder than TOPIK questions (because CSAT targets native speakers while TOPIK targets learners), then a category dominated by CSAT questions would show lower model accuracy for reasons unrelated to the knowledge domain, complicating cross-category comparisons.

Critical Assessment

The experimental results provide genuine support for the paper's central descriptive claims—that current LLMs perform poorly on Korean cultural and linguistic knowledge, that proprietary models outperform open-source ones, and that scale and Korean-specific training do not reliably help. However, the strength of evidence varies across claims, and several interpretative limitations deserve careful scrutiny.

Claim from the executive summary: "Open-source models with fewer than 13B parameters struggle with over 60% of the dataset."

This claim is directly supported by Figure 2, which shows the proportion of challenging samples per model. The operationalization is clear: a "challenging" question is one where model accuracy falls below random chance ($1/N$). The >60% figure is visually evident in the figure and consistent with the per-category accuracy scores in Table 5, where most open-source models score 22–45% on Cultural Intelligence and 19–31% on Linguistic Intelligence. The claim is well-supported within the paper's own framework.

However, the "below random chance" threshold is a low bar—it identifies questions where the model is worse than guessing, but a model could be "not struggling" (above random chance) while still performing inadequately (e.g., 35% accuracy on a 25%-chance baseline). The paper's framing may inadvertently make open-source performance seem less concerning than it is: the >60% figure counts only questions where models are actively worse than random, but an additional fraction of questions where models score only marginally above random (e.g., 30–40%) would still represent substantial knowledge deficits.

Claim from the executive summary: "GPT-3.5 and Claude-2 still face challenges on over 30% of the data."

Supported by Figure 2. The shared difficulty of 12.6% between GPT-3.5 and Claude-2 is a revealing statistic—it identifies questions that even the most capable models cannot answer, potentially representing knowledge that is genuinely absent from their training data rather than knowledge that exists but is unreliably retrieved. The paper does not analyze what characterizes these 12.6% of questions (are they concentrated in specific categories? specific sources? specific difficulty types?), which limits the diagnostic value of this finding. A breakdown of the shared difficulty set by category would reveal whether the proprietary models' knowledge gaps are systematic (e.g., they both lack Korean historical knowledge) or idiosyncratic (each model lacks different subsets of knowledge).

Claim from the executive summary: "Neither scaling model size nor fine-tuning with additional Korean corpora reliably improves cultural and linguistic comprehension."

This is the paper's most consequential empirical claim, supported by the ANOVA results in Table 6 (model scale: F₁,₄₃ = 0.33, p = .57; Korean corpus: F₁,₅₄ = 0.30, p = .59). The statistical analysis is appropriate for the question being asked, but several considerations qualify the interpretation:

First, the scale range tested (1.3B to 12.8B for Polyglot-Ko, with proprietary models implied to be much larger) may be insufficient to detect scale effects that would emerge at larger sizes. The gap from 1.3B to 12.8B is roughly one order of magnitude; if cultural knowledge acquisition requires two or three orders of magnitude in scale (which would be consistent with emergent abilities observed in other domains), this experiment would not detect it. The paper's claim is that scale does not reliably improve performance within the tested range, which is accurate; it is not a claim that no amount of scale would ever help, which would require testing at GPT-3.5 and Claude-2 scales within a controlled architecture family.

Second, the "Korean corpus supplementation" analysis compares base models against instruction-tuned variants. Instruction tuning typically adds a relatively small amount of curated training data (often tens of thousands to hundreds of thousands of examples) compared to the pretraining corpus (hundreds of billions of tokens). The finding that instruction tuning with Korean data does not improve cultural knowledge is not equivalent to a finding that pretraining on more Korean data would not help—the instruction-tuning datasets may simply not contain cultural knowledge content (they are typically designed for task-following ability, not factual knowledge acquisition). The paper's conclusion that "simply amassing more data... may not be the optimal solution" overstates what the experiment tested: the experiment tested additional instruction-tuning data, not additional pretraining data at scale.

Third, the finding that LLaMA-2-chat outperforms Korean-specialized models despite minimal Korean training data is striking but has an alternative interpretation the paper does not explore: LLaMA-2-chat is a much larger model trained on a much larger and more diverse corpus than Polyglot-Ko, and its superior performance may reflect general reasoning ability and cross-lingual transfer rather than specifically Korean cultural knowledge. A model that is better at reasoning in general might perform better on Korean multiple-choice questions even with minimal Korean-specific knowledge, simply by eliminating implausible distractors or recognizing question patterns. This would make the LLaMA-2-chat advantage an artifact of general intelligence rather than evidence against the value of Korean-specific training.

Claim from the executive summary: "Only 0.6% of the dataset was universally answered correctly by all models tested."

Supported by Figure 2 and directly stated in Section 5.1. This is a striking figure that the paper uses to argue CLIcK's difficulty is well-calibrated—it's not impossibly hard (since some model somewhere can answer almost every question) and not trivially easy (since no single question is answerable by all models). The 0.6% figure represents questions where even the best models fail, suggesting genuine knowledge gaps rather than benchmark artifacts.

However, the interpretation of "universally unsolvable" depends on the model set tested. The 13 models include several that perform poorly; if the model set were restricted to GPT-3.5 and Claude-2, the universally unsolvable percentage would be higher than 0.6% (the shared difficulty set for proprietary models is 12.6%). Conversely, if the model set included only the worst-performing open-source models, the universally unsolvable percentage would be lower (they share difficulty on 35.5% of the dataset, meaning 64.5% of questions are solvable by at least one open-source model). The 0.6% figure is a function of the specific model set chosen, not an intrinsic property of CLIcK.

Human-level comparisons: strong framing but imperfect methodology.

The CSAT, TOPIK, and Kedu comparisons in Section 5.2 provide the paper's most interpretable results, but the methodology has limitations the paper partially acknowledges. The percentile mappings are based on accuracy percentage alone, without item-level difficulty weighting. Real exam scoring systems (particularly for CSAT with its stanine system) use item response theory or equating procedures that account for question difficulty—a 50% score on a difficult exam form might place a student in a higher percentile than a 50% score on an easy form. The paper's comparison assumes all CLIcK CSAT questions represent the average difficulty of the CSAT questions used to compute the human distribution, which may not hold.

Additionally, the human comparison is between models taking an extracted subset of exam questions and humans taking the full exam under standardized conditions. The extracted subset may not be representative of the full exam's difficulty distribution—OCR processing excluded images and tables, meaning visually-dependent questions (maps, charts, diagrams) are absent from CLIcK. If these excluded questions were systematically harder or easier, the model-vs-human comparison would be biased. The paper does not report what fraction of source exam questions were excluded due to image/table content, nor how this exclusion might affect difficulty calibration.

Missing experiments that would strengthen the paper.

Several analyses would substantially strengthen the paper's claims but are absent:

• No analysis of whether model performance correlates with the frequency of tested knowledge in training data. The paper claims that cultural knowledge is absent from models, but never verifies that the specific facts tested in CLIcK actually appear in the models' training data. It's possible that the facts are present in training data but models fail to retrieve them; it's also possible the facts are genuinely absent. These have different implications—retrieval failure suggests improvements to inference or knowledge integration, while genuine absence suggests data curation is needed.

• No few-shot or chain-of-thought evaluation. All experiments use zero-shot prompting. Testing whether few-shot examples (providing similar Korean questions with answers) or chain-of-thought reasoning (asking models to explain their reasoning in Korean) improves performance would distinguish between knowledge that is accessible but not elicited by the zero-shot format and knowledge that is genuinely absent. The paper's finding that scale doesn't help might partially reflect that larger models need different prompting strategies to access their knowledge, not that they lack the knowledge entirely.

• No analysis of whether models trained on Korean data show different uncertainty patterns than models trained primarily on English. Figure 3 shows that larger models tend toward confident errors (low entropy), but the paper doesn't disaggregate this by training data composition. Do Korean-specialized models show different confidence calibration than general models? A finding that Korean-specialized models are more uncertain (closer to random guessing) while general models are confidently wrong would suggest different failure modes requiring different interventions.

• No human performance ceiling on CLIcK itself. The paper compares models to human exam-takers using official statistics, but never reports how well Korean native speakers perform on CLIcK's specific question set. A ceiling study with even a small number of native Korean speakers would establish whether CLIcK's questions are answerable by humans at high rates and whether any questions contain residual ambiguity or errors missed by the validation process.

Causal claims vs. correlational evidence.

The paper states that "simply amassing more data and enlarging the model size... may not be the optimal solution for enhancing the cultural intelligence of Language Models in non-English languages" (Section 4.1 and echoed in Section 6). This is a causal claim—that increasing data and scale does not cause improved cultural intelligence—supported only by correlational evidence (comparing existing models at different scales and training data compositions). The models tested differ along multiple dimensions simultaneously: architecture, pretraining data composition, tokenization, training objectives, and scale. Attributing performance differences (or the lack thereof) specifically to scale or Korean data exposure requires assuming these other dimensions are irrelevant, which the paper does not justify empirically. A proper causal test would hold all other factors constant and vary only scale or only Korean data proportion—something the paper's model selection does not enable.

Despite these limitations, the paper's core empirical contribution—that 13 current LLMs, including those specifically developed for Korean, perform poorly on a culturally authentic Korean benchmark, and that the simple remedies of scale and instruction-tuning do not obviously fix this—is robustly supported. The magnitude of the performance deficits (open-source models barely above random chance, proprietary models 10–20 percentage points below human averages on professional exams) is large enough that reasonable methodological critiques would not overturn the qualitative conclusion: current LLMs lack Korean cultural and linguistic intelligence as measured by the standard Korean society itself uses to assess such knowledge. The paper's value lies more in establishing this fact convincingly and providing the instrument for measuring it than in definitively explaining why it occurs or what would fix it.

6. Limitations and Trade-offs

Limitation 1: Evaluation restricted to a single benchmark format (multiple-choice) on a single language, with no demonstration of cross-linguistic or cross-task generalisability

The assumption or constraint. CLIcK is exclusively a multiple-choice QA benchmark in Korean, sourced from Korean examinations and textbooks. The paper evaluates 13 model configurations on this single dataset and draws conclusions about "cultural and linguistic intelligence in Korean" broadly. The evaluation methodology follows MMLU and AGIEval conventions (Section 4), but never tests whether CLIcK performance correlates with performance on other Korean tasks (generation, summarisation, dialogue, open-ended QA), other cultural knowledge modalities (images, audio, situated interaction), or other languages with similar cultural distance from English (Japanese, Chinese, Arabic). The paper makes no claim about cross-linguistic generalisability, but the implication—that CLIcK reveals something fundamental about LLMs' inability to acquire non-English cultural knowledge through scaling—invites the question of whether the findings are specific to Korean or to the multiple-choice format.

The consequence. A model that performs poorly on CLIcK might nonetheless possess substantial Korean cultural knowledge that the multiple-choice format fails to elicit—knowledge that would emerge in generative tasks, in interactive dialogue, or with different prompting strategies. Conversely, a model that performs well on CLIcK might be exploiting shallow pattern-matching on exam-style questions without genuine cultural understanding, and would fail on more naturalistic assessments. The paper cannot distinguish these possibilities because it provides only one measurement instrument. A practitioner deciding whether to deploy a Korean-language system cannot know whether poor CLIcK performance means "this model will fail at any Korean task requiring cultural knowledge" or "this model is bad at exam-style multiple choice but fine at conversational Korean." The single-task, single-format design precludes that calibration.

Additionally, the paper's finding that "simply amassing more data and enlarging the model size... may not be the optimal solution for enhancing the cultural intelligence of Language Models in non-English languages" (Section 6) generalises from Korean to "non-English languages" writ large without testing even one other language. Cultural knowledge in Japanese, Arabic, Hindi, or Swahili may scale differently with model size and training data than Korean cultural knowledge—Korean has specific linguistic features (honorifics, agglutinative morphology, a unique writing system) and cultural features (a relatively homogeneous society with strong institutional cultural education) that may make it an outlier rather than a representative case.

What evidence exists in the paper. None. The paper provides no correlation between CLIcK performance and any other Korean benchmark or task, no human validation that CLIcK scores predict real-world Korean language competence, and no replication in another language. The evaluation is entirely self-contained: models are tested on CLIcK, and conclusions are drawn about Korean cultural intelligence based solely on CLIcK scores.

Mitigation status. The paper does not address this limitation. Section 6 concludes that "models find it challenging to understand non-English linguistic and cultural intelligence, highlighting the need for more tailored methods in further research"—a claim that extends beyond Korean without supporting evidence. The paper does not suggest future multi-task or cross-linguistic validation studies. The contribution is explicitly a benchmark dataset, not a general theory of cultural knowledge acquisition, and the paper's claims should accordingly be read as specific to CLIcK-format evaluation of Korean cultural knowledge. However, the discussion language frequently blurs this boundary, and the absence of any external validation limits the benchmark's practical utility for model selection: a developer choosing between two Korean models cannot determine whether the one with higher CLIcK scores will actually serve Korean users better in deployed applications.

Limitation 2: Human comparisons rely on unvalidated accuracy-to-percentile mappings without item-level difficulty calibration

The assumption or constraint. Section 5.2 compares model accuracy on CLIcK subsets to human performance on the source examinations using "a simple score conversion, the ratio of correctly answered questions" because "our dataset does not encompass the actual score distribution for the problems." The paper maps the percentage of CLIcK questions a model answers correctly directly onto the percentile distribution of human test-takers who took the full, original examinations. This assumes that (a) the CLIcK-extracted subset of each exam is representative of the full exam's difficulty distribution, and (b) accuracy percentage maps linearly to percentile rank within the exam's scoring system.

The consequence. Both assumptions are likely violated, and the direction of bias is unknowable from the paper's data. The OCR extraction process explicitly "exclud[ed] images and tables" (Section 3.1), meaning any exam questions that relied on maps, charts, photographs, diagrams, or other visual elements were omitted from CLIcK. If these visual-dependent questions were systematically harder or easier than text-only questions—which is plausible, since visual reasoning and textual reasoning tap different competencies—then the CLIcK subset is not a random sample of the exam's difficulty distribution. A model might score 45% on CLIcK's text-only CSAT subset while a human scoring 45% on the full CSAT (with visual questions) might represent a different ability level entirely, because the human faced a different mix of question types.

Furthermore, real examination scoring systems—especially for high-stakes exams like the CSAT—use standardised scoring, item response theory, or equating procedures that weight questions by difficulty and adjust for exam form. A raw accuracy percentage does not straightforwardly map to a stanine or percentile. The paper's CSAT analysis, which places GPT-3.5 in "the lowest 11th percentile" and open-source models at the "8th grade level, representing the lowest 11%" or "9th level... the lowest 4%," is an approximation that conflates percentage-correct with norm-referenced ranking. A student could answer 50% of CSAT questions correctly and place in the 30th percentile on an unusually difficult exam form, or the 60th percentile on an easy form—the mapping depends on the cohort's performance distribution for that specific exam administration, not just the raw score.

What evidence exists in the paper. The paper provides the official score statistics used for comparison (Section 5.2): CSAT tiers from 2017–2020 averaged, TOPIK absolute level descriptions, Kedu five-year average participant scores. However, the paper provides no analysis of whether the CLIcK-extracted question subsets match the difficulty of the full exams—no comparison of CLIcK question difficulties to full-exam question difficulties, no item-level analysis, and no reporting of what fraction of source exam questions were excluded (particularly due to images and tables). The paper acknowledges the limitation explicitly—"our dataset does not encompass the actual score distribution for the problems"—but does not quantify its magnitude.

Mitigation status. The paper acknowledges the limitation in a single sentence (quoted above) but treats the simple score conversion as adequate for comparative analysis. The multi-year averaging (five years of CSAT data, five years of Kedu data) provides some stability against year-to-year difficulty variation, but does not address the fundamental problem that the CLIcK subset may not represent the full exam. The paper does not suggest obtaining item-level difficulty parameters from the testing institutions or conducting a human baseline study on the CLIcK subset itself as a calibration. A practitioner reading the percentile comparisons should treat them as rough ordinal indications ("GPT-3.5 performs substantially worse than Korean high school students on the CSAT") rather than precise quantitative claims ("GPT-3.5 is at exactly the 11th percentile"), but the paper's presentation does not consistently make this distinction.

Limitation 3: The ANOVA analyses rule out scale and Korean data effects only within narrow, confounded parameter ranges—not across the full capability spectrum

The assumption or constraint. The paper's headline finding—that "neither scaling model size nor fine-tuning with additional Korean corpora reliably improves cultural and linguistic comprehension" (Section 4.1)—rests on ANOVA tests reported in Table 6. The model scale test (F₁,₄₃ = 0.33, p = .57) compares Polyglot-Ko at 1.3B through 12.8B, KULLM at 5.8B and 12.8B, KoAlpaca at 5.8B and 12.8B, and LLaMA variants. The Korean corpus test (F₁,₅₄ = 0.30, p = .59) compares base Polyglot-Ko models against their instruction-tuned derivatives KULLM and KoAlpaca. From these tests, the paper concludes that current scaling approaches are insufficient for cultural knowledge.

The consequence. The parameter range tested—roughly 1.3B to 13B—spans one order of magnitude. The paper then contrasts these models with GPT-3.5 and Claude-2 (likely 100B+ parameters, though exact sizes are undisclosed), which substantially outperform the open-source models on CLIcK (Table 5). This creates a paradox in the paper's own data: within the 1.3B–13B range, scale does not help; but jumping to proprietary scale produces large improvements. The paper cannot distinguish between two interpretations: (a) cultural knowledge has a scale threshold above 13B, meaning the ANOVA fails to detect an effect that does exist at larger scales, or (b) GPT-3.5 and Claude-2's advantages come from factors other than scale—different training data composition, different training objectives (RLHF, instruction tuning at scale), different architectural choices, or different evaluation dynamics (generative vs. log-probability scoring). The experimental design conflates scale with model family, training procedure, and evaluation method, making it impossible to attribute the proprietary models' superior performance to any specific factor.

The Korean corpus finding has a related confound: "Korean corpus supplementation" is operationalised as instruction-tuning Polyglot-Ko with additional Korean data (KULLM, KoAlpaca). Instruction-tuning datasets are typically small (tens or hundreds of thousands of examples) compared to pretraining corpora (hundreds of billions of tokens), and they are curated for instruction-following ability rather than factual knowledge coverage. Finding that instruction-tuning does not improve cultural knowledge is not equivalent to finding that additional pretraining on Korean data would not help—the instruction-tuning data may simply not contain the cultural facts being tested. The paper's conclusion that "simply amassing more data... may not be the optimal solution" overstates what the experiment tested.

What evidence exists in the paper. The ANOVA results in Table 6, the per-model accuracy breakdown in Table 5, and the comparison with proprietary models are the primary evidence. Table 5 shows the scale effect within Polyglot-Ko: Cultural Intelligence accuracy is 32.71% (1.3B), 32.90% (3.8B), 33.14% (5.8B), 33.40% (12.8B)—a total improvement of 0.69 percentage points across an order of magnitude of scale. The proprietary models score 49.30% (GPT-3.5) and 51.72% (Claude-2). The gap from 12.8B to proprietary scale is 16–18 percentage points—larger than the entire 1.3B-to-12.8B range—but the paper attributes this to factors other than scale without being able to isolate them.

Mitigation status. The paper does not address this confound. The ANOVA results are reported as evidence that scale does not matter, and the proprietary model results are reported separately as evidence that even the best models still struggle. The possibility that scale matters but only above 13B, or that proprietary models benefit from scale in ways that interact with other factors (training data diversity, RLHF, instruction following), is not discussed. The paper's framing—"simply amassing more data and enlarging the model size... may not be the optimal solution"—is stated as a conclusion rather than a hypothesis requiring testing at larger scales with controlled comparisons. A controlled scale comparison within a single architecture family from 1B to 100B+ parameters, holding training data composition constant, would be needed to test the scale hypothesis properly, and the paper acknowledges no such experiment exists or is planned.

Limitation 4: The benchmark provides no mechanism for distinguishing knowledge absence from knowledge access failure

The assumption or constraint. CLIcK evaluates models exclusively in a zero-shot, multiple-choice format with brief prompts that ask for a single-letter answer (Section 4). The paper does not test whether model performance improves under conditions that are known to affect knowledge access in LLMs: few-shot prompting (providing example Korean questions with answers before the test question), chain-of-thought reasoning (asking the model to explain its reasoning in Korean before selecting an answer), translation-based strategies (asking the model to translate the question to English, reason, and translate back), or retrieval-augmented generation (providing relevant Korean text passages as context). The assumption is that zero-shot multiple-choice accuracy reflects the underlying presence or absence of knowledge.

The consequence. The paper cannot distinguish between two fundamentally different failure modes: (a) the model genuinely lacks the tested Korean cultural knowledge—the facts, concepts, or linguistic patterns were not present in its training data and cannot be reconstructed—versus (b) the model possesses the knowledge but cannot access it through the specific evaluation format used. These two failure modes have entirely different implications. If the knowledge is genuinely absent, the remedy is data curation: adding Korean cultural content to training corpora. If the knowledge is present but inaccessible, the remedy is prompt engineering, decoding strategy changes, or fine-tuning for better knowledge elicitation—the training data itself may be adequate.

The paper's own qualitative example (Problem 1 vs. Problem 2 in Section 5.1) hints at this ambiguity. Problem 1 (about Korean housing policy) was answered correctly by all 13 models; Problem 2 (about Korean kinship terminology—what a husband calls his wife's older brother) was answered correctly by none. The paper presents this as evidence that "cultural and linguistic intelligence contexts are challenging for the model to comprehend." But the kinship question (형님, hyeongnim) tests knowledge that is ubiquitous in Korean-language text—Korean kinship terms appear regularly in everyday conversation, K-dramas, and written Korean. It is plausible that the models have encountered this term but cannot map the specific question format ("How does a husband call his wife's older brother?") to the stored knowledge. A few-shot prompt providing similar kinship questions with answers might elicit the correct response, which would change the interpretation from "models lack Korean kinship knowledge" to "models struggle to retrieve Korean kinship knowledge in zero-shot exam format."

What evidence exists in the paper. None. The paper conducts no experiments varying the prompting strategy, number of examples, reasoning format, or language of reasoning. The entropy analysis in Figure 3 partially addresses the access question by showing that larger models make confident errors rather than random guesses, which is consistent with models having learned some systematic association (even if wrong) rather than having no knowledge at all—but this does not test whether the correct association is present and inaccessible.

Mitigation status. The paper does not address this limitation. The evaluation protocol is described as following "the evaluation methodology from MMLU, which also aligns with prevalent LLM evaluation frameworks" (Section 4). The choice of zero-shot evaluation is standard practice and reasonable for an initial benchmark study, but the absence of any knowledge-access experiments means the paper's findings should be interpreted as lower bounds on model knowledge: models might know more Korean cultural content than CLIcK reveals. The paper's strong claim that "current pretraining-centric approaches are insufficient for non-English cultural intelligence" (Section 2 framing) assumes that the zero-shot scores reflect knowledge absence, not access failure, without providing evidence for this assumption. This is a particularly salient limitation given recent work showing that prompting strategy dramatically affects LLM performance on knowledge-intensive tasks, sometimes doubling accuracy between zero-shot and well-designed few-shot or chain-of-thought formats.

Limitation 5: No human ceiling study establishes whether CLIcK's specific questions are answerable at high rates by Korean native speakers

The assumption or constraint. The paper evaluates models against CLIcK and compares model performance to human test-taker statistics from the original examinations (Section 5.2), but never reports how well Korean native speakers actually perform on CLIcK's specific extracted question set. The assumption is that CLIcK questions are answerable by fluent Korean speakers at rates consistent with the source exams' human score distributions, and that the multi-round validation by four native Korean annotators (Section 3.2) is sufficient to guarantee this.

The consequence. Without a human ceiling study, the paper cannot determine whether residual errors in CLIcK—OCR artifacts, questions whose answers depend on removed context (images, tables, preceding questions), cultural knowledge that has genuinely shifted since the exam was administered, or answer-key errors—depress model scores in ways that would also affect humans. If native Korean speakers achieve only, say, 85% accuracy on CLIcK (rather than the 95%+ that would be expected for "basic cultural knowledge"), then the benchmark's ceiling is lower than assumed, and model scores should be evaluated relative to that ceiling rather than relative to 100%. The paper's interpretation of open-source models scoring 22–45% as "poor" depends on the implicit assumption that competent humans would score near 100%—an assumption that is untested.

This matters particularly for the qualitative examples the paper uses to illustrate model failures. Problem 2 (the kinship question) is described as a question that "all 13 models... none correctly answered" (Section 5.1), implying this is a damning indictment of model capability. But if the question contains ambiguity that would also cause some humans to answer incorrectly—for instance, if the term 형님 (hyeongnim) has regional or generational variation in usage—then the model failure is less informative. The human validation process (four Korean native speakers reviewing questions) may not catch ambiguities that affect answer selection, since validators were checking for correctness of the provided answer, not independently answering the question under test conditions.

What evidence exists in the paper. The paper provides extensive evidence about human performance on the source examinations (CSAT tiers, TOPIK levels, Kedu averages) but no evidence about human performance on CLIcK itself. The multi-round validation process (Section 3.2, with a 37.1% rejection rate) is a quality-control mechanism, not a performance ceiling study—annotators were verifying that questions met validity criteria, not taking the test under evaluation conditions.

Mitigation status. The paper does not address this limitation. The human comparisons in Section 5.2 use exam statistics rather than CLIcK-specific human data. The paper acknowledges that "our dataset does not encompass the actual score distribution for the problems" but treats this as a limitation of the score conversion methodology rather than a gap in ceiling calibration. A small-scale human study (even 10–20 native Korean speakers answering a random subset of CLIcK questions) would establish whether the benchmark's questions are answerable at ceiling rates and would calibrate the interpretation of model scores. The paper does not suggest this as future work. Given that CLIcK involves OCR extraction from exam documents—a process that can introduce text corruption, missing context, and formatting artifacts—the absence of human baseline data on the extracted questions themselves is a significant gap in the benchmark's validation.

Limitation 6: The benchmark provides no diagnostic power for distinguishing between cultural knowledge domains that are genuinely absent from training data vs. domains where models fail for other reasons

The assumption or constraint. CLIcK categorises questions into 11 subcategories (8 Cultural Intelligence, 3 Linguistic Intelligence) and reports per-category accuracy scores (Table 5). The paper interprets category-level performance differences as revealing "which specific domains of cultural knowledge are systematically absent from language models" (paraphrasing the motivation for fine-grained categorisation in Section 3.3). The implicit assumption is that lower accuracy in a category reflects less training data coverage for that category's knowledge, and that cross-category accuracy comparisons reveal which domains are better or worse represented in training corpora.

The consequence. Category-level accuracy conflates multiple factors that could produce performance differences independently of training data coverage. A category might show low accuracy because (a) the facts are genuinely absent from training data, (b) the questions in that category are systematically harder in ways unrelated to cultural knowledge (longer passages, more complex reasoning, more plausible distractors, more obscure vocabulary), (c) the category contains more questions from harder exam sources (e.g., CSAT vs. KIIP textbook), or (d) the knowledge type in that category is inherently harder for language models to represent regardless of training data volume (e.g., pragmatic Functional Knowledge vs. declarative factual knowledge).

The paper's finding that History is the lowest-scoring Cultural Intelligence category for GPT-3.5 (30.75%) while Economy is the highest (47.59%) illustrates this ambiguity. It might be that Korean economic content is more represented in English-language training data (through international business reporting, economic analyses, trade discussions) while Korean history appears primarily in Korean-language sources that GPT-3.5 sees less of. Or it might be that Korean history questions require temporal reasoning about specific events and causal relationships that are inherently harder for LLMs, while economic questions involve more transferable conceptual frameworks. Or it might be that the History questions in CLIcK come disproportionately from harder exams (KHE targets native Korean historical literacy, while Economy questions come from the KIIP textbook designed for immigrants). The paper provides no analysis to distinguish these explanations, limiting the diagnostic value of the per-category breakdown.

Similarly, the paper's finding that Functional Knowledge is the hardest Linguistic Intelligence category across all models (GPT-3.5: 32.62%, Claude-2: 32.88%, open-source: 19–31%) is interpreted as evidence that "pragmatic competence in Korean... is almost entirely absent from current models." But Functional Knowledge questions in CLIcK come from the CSAT-Korean exam's speaking and writing sections (Section 3.3), which test nuanced understanding of communicative intent in Korean—these might be simply harder questions by human standards, even for native speakers, independent of training data coverage. Without item-level difficulty calibration or human ceiling data per category, the paper cannot determine whether low Functional Knowledge scores reflect absence of pragmatic knowledge or higher intrinsic question difficulty.

What evidence exists in the paper. Table 5 provides per-category accuracy breakdowns. Table 2 provides per-source question counts, but the paper does not cross-tabulate category accuracy with source difficulty. The difficulty analysis in Figure 2 treats questions dichotomously (challenging vs. not challenging based on the random baseline threshold) without analysing whether challenging questions cluster in particular sources or categories. The qualitative analysis in Section 5.1 explicitly notes the difficulty interpretation problem: "both inquire about Korean-specific matters... Problem 2 asks about the everyday life of Korean society, whereas problem 1 seeks more professional information, making problem 1 appear more challenging. However, all 13 models achieved 100% accuracy on Problem 1, yet none correctly answered Problem 2." The paper uses this to argue that "there is less alignment between the model's perceived difficulty and those by humans" but does not extend the analysis to ask whether category-level patterns reflect actual knowledge gaps or difficulty misalignment.

Mitigation status. The paper does not address this limitation systematically. The per-category breakdown is presented as a diagnostic tool, but the paper provides no analysis that validates the categories as measuring training data coverage rather than question difficulty or format effects. The paper's recommendation that future work should develop "more tailored methods" for cultural knowledge acquisition implicitly assumes the categories identify genuine knowledge gaps, but this assumption is untested. A training data coverage analysis—measuring how frequently the specific facts, terminology, and concepts tested in each CLIcK category appear in the pretraining corpora of the evaluated models—would be needed to validate the categories as diagnostics for knowledge absence, and the paper neither conducts nor proposes such an analysis.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new diagnostic instrument rather than a paradigm shift—but it is a diagnostic that makes visible a problem the field has systematically overlooked, and in doing so, it changes what counts as evidence in multilingual LLM development. The conceptual shift is from evaluating non-English models on how well they perform on translations of English tasks to evaluating them on whether they possess the cultural and linguistic knowledge that the target culture itself considers important. This is a reframing of the evaluation question, not a new method for improving models, but reframing the question has downstream consequences for what the field optimizes toward.

What changes: the burden of proof for multilingual capability claims. Before CLIcK, a Korean LLM developer could reasonably claim their model "understands Korean" if it performed well on translated benchmarks like Korean HellaSwag or Korean MMLU. CLIcK demonstrates that this inference is invalid: models can perform adequately on translated tasks while being catastrophically deficient on authentically Korean cultural knowledge. The paper's finding that LLaMA-2-chat—with only 0.06% Korean training data—outperforms Korean-specialized models on CLIcK (Section 4.1) is a concrete refutation of the assumption that Korean-language benchmark performance implies Korean cultural competence. After CLIcK, the field can no longer claim cultural understanding without testing it directly on culturally authentic instruments. This shifts evaluation from a translation-centric paradigm toward a cultural-authenticity paradigm, where the question is not "Can this model answer English questions in Korean?" but "Does this model know what Korean society expects its members to know?"

Reconciling prior contradictions. The paper does not explicitly resolve contradictions in prior literature, but its findings provide a unifying explanation for the persistent gap between Korean models' performance on translated benchmarks (where they appear competitive) and their well-known deficiencies in actual Korean-language deployment (where they make errors that any Korean speaker would find baffling). The gap exists because translated benchmarks test general reasoning and linguistic pattern-matching in Korean—capabilities that transfer from English pretraining—while culturally specific factual knowledge (kinship terms, historical events, legal systems, traditional practices) does not transfer. CLIcK isolates the latter, revealing it as the missing component. This explains why scaling and Korean instruction-tuning do not help (Table 6): these approaches improve the general linguistic surface, not the culturally specific content that CLIcK targets.

Research directions that become more attractive:

• Culturally grounded data curation becomes a higher priority than scaling. If additional Korean instruction-tuning does not improve CLIcK performance, the problem is not data quantity but data relevance—the Korean data used for training may not contain the specific cultural facts that Korean society considers important. This directs attention toward curating training corpora that explicitly cover institutional cultural knowledge (textbooks, examination materials, government publications) rather than assuming web-scale Korean text will contain it incidentally.
• Evaluation-driven model development for non-English languages becomes feasible. CLIcK provides the first comprehensive diagnostic for Korean cultural knowledge, enabling developers to measure whether architectural changes, training data modifications, or fine-tuning strategies actually improve cultural competence rather than just general Korean language performance.
• Cross-cultural evaluation as a standard practice becomes more defensible. CLIcK demonstrates that a single-language, culturally authentic benchmark can reveal performance dimensions invisible to translation-based multilingual benchmarks. This creates a template for other languages to follow.

Research directions that become less attractive:

• Assuming that more parameters will eventually solve cultural knowledge gaps. The ANOVA results (F₁,₄₃ = 0.33, p = .57 for scale effect) suggest that within the 1.3B–13B range, scale does not help. While the paper cannot rule out a threshold effect at proprietary scales (GPT-3.5 and Claude-2 do outperform open-source models, but the paper cannot attribute this to scale specifically), the flat scaling curve within the tested range makes "just train a bigger model" a less credible default strategy for Korean cultural competence than it was before CLIcK existed.
• Sole reliance on translated benchmarks for non-English model evaluation. CLIcK provides empirical evidence that translated benchmarks are not adequate proxies for cultural knowledge. A Korean model developer who evaluates only on the Open Ko-LLM Leaderboard (which the paper critiques as translation-derived) has no way to detect the cultural knowledge deficits that CLIcK reveals. This makes translation-only evaluation pipelines harder to justify.

Follow-Up Research This Work Enables

1. Training data coverage analysis: do the specific facts tested in CLIcK actually appear in Korean LLM pretraining corpora? The paper's central claim—that models lack Korean cultural knowledge—rests on the assumption that CLIcK tests knowledge that is genuinely absent from training data. But this assumption is untested. A direct follow-up would audit the pretraining corpora of Polyglot-Ko, LLaMA, and (to the extent possible) GPT-3.5 for the presence or absence of the specific facts, concepts, and terminology tested in CLIcK. For a stratified sample of CLIcK questions (say, 100 questions drawn proportionally from the 11 categories), researchers could search the training corpora for the key entities, events, and relationships needed to answer each question. This would distinguish between two failure modes: (a) the information genuinely does not appear in training data (suggesting data curation is needed) versus (b) the information appears but models cannot retrieve or reason with it in the zero-shot format (suggesting knowledge access, not knowledge absence, is the bottleneck). The paper's finding that models make confident errors on challenging questions (low entropy in Figure 3 for larger models) suggests they have learned some systematic associations, which is more consistent with (b) than (a), but a coverage analysis would resolve this definitively.

2. Cross-lingual knowledge transfer experiments: does Korean cultural knowledge transfer from English-language sources? The paper's finding that LLaMA-2-chat (0.06% Korean training data) outperforms Korean-specialized models raises a specific, testable question: is the advantage coming from (a) general reasoning ability enabling better use of limited Korean knowledge, (b) English-language coverage of Korean cultural topics (e.g., English Wikipedia articles about Korean history, English-language K-pop coverage, English news about Korean politics), or (c) other factors like instruction-tuning or RLHF? A controlled experiment would compare model performance on CLIcK questions where the tested knowledge has high English-language coverage (e.g., questions about Korean War history, which is extensively documented in English) versus questions where the knowledge is almost exclusively in Korean sources (e.g., questions about specific Korean kinship terminology, local administrative divisions, or traditional practices with limited English documentation). If LLaMA-2-chat's advantage is concentrated in the high-English-coverage questions, it suggests that cultural knowledge can transfer cross-lingually when English-language sources exist, and the bottleneck for other categories is monolingual Korean data coverage. If the advantage is uniform, it suggests general reasoning capability is the driver. This experiment requires constructing a coverage-annotated CLIcK subset—a tractable annotation task given the dataset's existing categorization and citation structure.

3. Prompting strategy ablation: can few-shot examples, chain-of-thought reasoning, or language-switching strategies unlock knowledge that zero-shot evaluation misses? The paper's zero-shot evaluation protocol (Section 4) provides a lower bound on model knowledge, but the paper explicitly does not test whether alternative prompting strategies improve access to stored cultural knowledge. A systematic ablation would test CLIcK performance under: (a) few-shot prompting with 1–5 Korean-language example questions and answers, (b) chain-of-thought prompting asking models to explain their reasoning in Korean before selecting an answer, (c) English-language reasoning (translating the question to English, reasoning, and answering in English, then mapping back to the Korean option), and (d) retrieval-augmented generation with relevant Korean text passages provided as context. The paper's qualitative example—all models fail on a kinship question that tests ubiquitous Korean knowledge—is a prime candidate for this kind of experimentation: if few-shot examples with other kinship questions enable correct answers, it demonstrates that the knowledge is present but inaccessible in zero-shot format, which would substantially change the interpretation of CLIcK scores from "models lack Korean cultural knowledge" to "models cannot retrieve Korean cultural knowledge without appropriate prompts." The 13-model evaluation framework the paper establishes makes this ablation tractable: the same models, same questions, same cyclic permutation protocol, varying only the prompt structure.

4. Human ceiling study on CLIcK: what is the achievable accuracy for native Korean speakers on this specific question set? The paper's human comparisons rely on official exam statistics from the original examinations (Section 5.2), not on human performance on CLIcK's extracted question subset. A direct human ceiling study would recruit 20–50 native Korean speakers to answer a random subset of CLIcK questions under test-like conditions, establishing (a) whether any CLIcK questions are genuinely ambiguous or error-containing at rates that would depress human scores, (b) what the effective ceiling is for each category (do humans score near 100% on all categories, or are some categories intrinsically harder even for native speakers?), and (c) whether the OCR extraction process introduced any systematic errors (text corruption, missing context from excluded images and tables) that make some questions unanswerable. The paper reports a 37.1% question rejection rate during validation (Section 3.2), indicating substantial quality concerns in the candidate pool; a human ceiling study would verify that the retained 62.9% of questions are answerable at high rates by the intended audience. This study would also provide the per-category human baselines needed to interpret whether, for example, the paper's finding that Functional Knowledge is the hardest linguistic category reflects genuine model deficits or intrinsically harder questions.

5. Replication in another culturally distant language: does the "scale and data don't help" finding generalise beyond Korean? The paper's conclusion that "models find it challenging to understand non-English linguistic and cultural intelligence" (Section 6) generalises from a single language. A replication study constructing an analogous benchmark for a language with similar cultural distance from English—Japanese (complex honorifics, distinct cultural traditions, strong institutional examination culture), Arabic (different script, rich cultural and religious knowledge, diglossia), or Hindi (different cultural context, complex kinship systems, strong examination tradition through competitive exams like UPSC)—would test whether the paper's core finding is specific to Korean or reflects a general property of how LLMs acquire (or fail to acquire) non-Anglophone cultural knowledge. The CLIcK construction methodology—source questions from official examinations and government textbooks, validate with native speakers, categorize using local institutional frameworks—provides a transferable template. A replication would use the same evaluation protocol (cyclic permutation, multiple prompt wordings, accuracy metric) and compare models at multiple scales within the target language, enabling direct comparison of scaling behavior across languages. If the flat scaling curve replicates, it strengthens the paper's claim that current pretraining approaches are fundamentally limited for cultural knowledge; if scaling helps in some languages but not others, it reveals language-specific factors (data availability, cultural representation in web corpora, linguistic distance from English) that moderate the relationship.

6. Category-level diagnostic validation: do per-category accuracy differences reflect genuine knowledge gaps or confounding difficulty factors? The paper's 11-category taxonomy (Section 3.3) is presented as a diagnostic tool for identifying which cultural knowledge domains models lack, but the paper provides no validation that lower accuracy in a category reflects less training data coverage rather than harder questions, more plausible distractors, or differences in source exam difficulty. A validation study would measure: (a) per-category human accuracy (via the ceiling study described above) to establish whether some categories are intrinsically harder, (b) per-category question features that might affect difficulty independent of cultural knowledge—passage length, vocabulary rarity, syntactic complexity, number of reasoning steps required, distractor plausibility (measured by how often humans select each distractor), and (c) per-category representation in the pretraining corpora of the tested models (via the coverage analysis described above). If categories that show low model accuracy also show low human accuracy, are linguistically more complex, or draw from harder source exams, then category-level patterns reflect question difficulty rather than knowledge domain gaps. If categories show low model accuracy despite high human accuracy and comparable linguistic complexity, then the category genuinely measures deficient cultural knowledge. This validation would transform CLIcK's categories from descriptive labels into calibrated diagnostic instruments, enabling researchers to use per-category scores to target specific knowledge domains for data augmentation or fine-tuning with confidence that they are addressing genuine gaps, not artifacts of question design.

Practical Applications and Downstream Use Cases

1. Auditing Korean-language deployment readiness for commercial LLM products. A company considering deploying an LLM-powered chatbot, search engine, or content moderation system for Korean users needs to know whether the model understands Korean culture well enough to avoid embarrassing or harmful errors. CLIcK provides a standardised, culturally authentic evaluation that can be run before deployment. The paper's human-exam anchoring makes the results interpretable to non-technical stakeholders: "This model scores in the lowest 11th percentile on the Korean CSAT" communicates the severity of cultural knowledge gaps more effectively than "accuracy of 35% on a benchmark." The per-category breakdown enables targeted risk assessment—a model scoring 68.61% on Pop Culture (as GPT-3.5 does) but 30.75% on History may be acceptable for entertainment applications but unsuitable for educational or informational products requiring accurate historical knowledge. The finding that Claude-2 achieves TOPIK level 6 (professional Korean proficiency) while open-source models fall below the measurable range provides a concrete threshold: for applications requiring near-native Korean language production, only proprietary-scale models currently meet the bar.

2. Guiding data curation for Korean LLM pretraining. The paper's finding that additional Korean instruction-tuning does not improve CLIcK performance (ANOVA: F₁,₅₄ = 0.30, p = .59) suggests that current Korean fine-tuning datasets—typically curated for instruction-following ability, not cultural knowledge coverage—are insufficient for cultural competence. A Korean LLM developer can use CLIcK's category taxonomy as a coverage checklist: for each of the 11 categories, audit whether the pretraining corpus contains substantial content about Korean society, tradition, pop culture, politics, economy, law, history, geography, and the three linguistic knowledge types. The paper's source materials themselves—the KIIP textbook, CSAT Korean exams, TOPIK materials, Korean history exams—represent high-quality, institutionally validated Korean cultural content that could be incorporated into pretraining corpora. A developer could measure whether adding these specific sources to training data improves CLIcK scores, using the benchmark both as a target and as a diagnostic for identifying remaining coverage gaps. The paper's finding that only 0.6% of questions were universally unsolvable (Section 5.1) means almost all CLIcK knowledge exists somewhere in the model ecosystem; the data curation challenge is consolidating it into individual models.

3. Benchmarking progress in Korean cultural AI over time. CLIcK's construction from official examinations with temporal stability as a design criterion (validation criterion 2 in Section 3.2: "Information in Questions remains consistent over time") means the benchmark can serve as a longitudinal evaluation instrument. As new Korean LLMs are developed, their CLIcK scores can be compared against the 13-model baseline established in this paper, tracking whether the field is making genuine progress on cultural knowledge or merely improving on translation-derived benchmarks while cultural competence stagnates. The paper's institutional anchoring—comparing models to human test-takers on real Korean exams—provides a stable reference point: even as models improve, the human performance distributions on CSAT, TOPIK, and Kedu remain meaningful benchmarks. A model that moves from the 11th percentile to the 50th percentile on the CSAT Korean exam represents genuine, interpretable progress. The public availability of CLIcK enables community-wide tracking, similar to how MMLU and HellaSwag serve as standard progress indicators for English LLMs.

4. Informing multilingual model development strategy at AI labs. For organisations developing multilingual LLMs (OpenAI, Anthropic, Meta, Google), CLIcK provides evidence that the dominant strategy—pretrain on massive multilingual web corpora and assume cultural knowledge will emerge—has a specific, measurable failure mode for Korean. The paper's comparison showing that LLaMA-2-chat (0.06% Korean data) outperforms purpose-built Korean models (Polyglot-Ko, KULLM) on CLIcK suggests that general reasoning capability from large-scale English-centric pretraining transfers better than Korean-specific pretraining when the Korean-specific training data does not contain the right cultural content. This has a concrete implication for resource allocation: rather than investing in larger Korean web crawls (which may not contain institutionally validated cultural knowledge), organisations might achieve better Korean cultural competence by investing in (a) curating high-quality Korean cultural content (textbooks, examinations, government publications), (b) improving cross-lingual knowledge transfer from English sources that discuss Korean topics, or (c) developing fine-tuning strategies specifically designed to inject cultural knowledge rather than general instruction-following ability. CLIcK provides the measurement instrument that makes these resource allocation decisions evidence-based rather than speculative.