MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

ArXiv: 2406.01574

🎯 Pitch

Leading AI models have hit a wall on the popular MMLU benchmark, with GPT-4o eking out only a 1% gain despite major reasoning leaps, suggesting the test is broken. This paper introduces MMLU-Pro, where simply expanding multiple-choice options and cutting trivial questions causes accuracy to plunge by up to 33%, yet intriguingly, Chain-of-Thought reasoning provides a major boost—an effect absent on the original benchmark.

---

1. Executive Summary

This paper introduces MMLU-Pro, an enhanced benchmark designed to extend the mostly knowledge-driven MMLU benchmark by integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options while eliminating trivial and noisy questions. Evaluating over 50 LLMs—including GPT-4o, Claude-3-Opus, Gemini-1.5-Pro, and Llama-3-70B-Instruct—the benchmark causes a significant accuracy drop of 16% to 33% compared to MMLU, with the leading model GPT-4o achieving only 72.6%, leaving substantial room for improvement. The benchmark demonstrates greater stability under varying prompts, reducing score sensitivity from 4–5% in MMLU to just 2% in MMLU-Pro across 24 prompt styles, and reveals that Chain of Thought (CoT) reasoning boosts performance by up to 19% on MMLU-Pro while hurting performance on the original MMLU, establishing that MMLU-Pro includes more complex reasoning questions that necessitate deliberate reasoning rather than knowledge recall, though the benchmark remains constrained to the multiple-choice format and does not assess multi-modal models.

2. Context and Motivation

The Core Problem: MMLU Is No Longer Fit for Purpose

The central problem this paper addresses is the performance saturation and instability of the MMLU benchmark, which had become the de facto standard for evaluating large language models' multi-task language understanding. The paper identifies a concrete inflection point: since GPT-4 achieved 86.4% on MMLU in March 2023, no significant progress has been registered on the benchmark. As the authors state in Section 1:

"Most recent frontier models like GPT-4-Turbo, Gemini-1.5-Pro, Claude, and LLaMA-3-400B (all published in early-mid 2024) all settle at an accuracy between 86% - 87%."

This clustering is a symptom of a deeper problem. When the best models in the world all score within a 1–2% band, the benchmark loses its ability to discriminate among them. A benchmark that cannot distinguish between models cannot tell us which innovations matter, which architectures produce genuine improvements, or whether the field is actually making progress toward more capable systems. The paper frames this as a direct challenge to the goal of tracking progress toward "expert-level intelligence," citing Morris et al. (2023)'s definition of expert-level AI as performance that matches or surpasses the top 10% of skilled adults across diverse tasks.

The specific case of GPT-4o makes the saturation problem vivid: the model achieved 10+% improvements on MATH and significant gains on the Chatbot Arena leaderboard, yet only improved by 1% on MMLU (reaching 87.4%). If a major model release produces dramatic gains on reasoning benchmarks but MMLU barely budges, the benchmark is no longer sensitive to the capabilities that are advancing—suggesting it measures something orthogonal to the frontier of model progress.

Why This Matters: The Benchmark Crisis in LLM Evaluation

The saturation problem matters for both practical and scientific reasons:

Practical impact: Leaderboards drive resource allocation. When benchmarks cluster scores tightly, small prompt variations or evaluation choices can reorder model rankings, making leaderboards unreliable guides for practitioners deciding which models to deploy. The paper explicitly notes that MMLU performance is "highly sensitive to the prompt and scoring function, which causes significant order changes in the leaderboard" (Section 1). If a production team evaluates two candidate models and the ranking flips depending on which of two reasonable prompts they use, the benchmark provides no actionable signal.

Scientific impact: Progress becomes unmeasurable. The ML community relies on benchmarks as yardsticks for methodological progress. When a benchmark saturates, new techniques that genuinely improve language understanding may show zero or negligible gains, while techniques that exploit benchmark-specific shortcuts may appear to win. This creates perverse incentives: researchers optimize for the benchmark rather than for the underlying capability it was designed to measure. The paper's observation that MMLU questions "are mostly knowledge-driven without requiring too much reasoning, especially in the STEM subjects" (Section 1) implies that models have been optimized for knowledge retrieval—a capability that saturates quickly—rather than the deeper reasoning that would continue to differentiate stronger models.

Broader field implications: The expert-level intelligence milestone. The community's stated goal of "expert-level intelligence" requires benchmarks that can measure progress in the upper percentiles of human performance. A benchmark with a ceiling at ~88% cannot distinguish a model performing at the 90th percentile from one at the 99th. MMLU-Pro's construction—with its explicit goal of creating a "proficient-level" benchmark where even the best model leaves a 27.4% gap to perfect accuracy—is a direct response to this measurement gap.

Root Causes: Three Specific Failures in MMLU

The paper doesn't merely observe that MMLU is saturated; it provides three specific diagnoses of why (Section 1). This diagnostic framing is critical because it motivates each design choice in MMLU-Pro's construction:

1. Insufficient distractor depth. MMLU provides only four answer options (one correct, three distractors). For a language model with strong prior knowledge, this creates a shallow decision space: the model can often identify the correct answer through elimination of obviously wrong options, or by exploiting surface-level associations, without genuinely understanding the reasoning chain. The random guessing baseline on four-option multiple-choice is 25%, which is already high. But more importantly, when the correct answer is the only plausible option among three clearly wrong distractors, the model's task reduces to recognition rather than reasoned selection. The paper conjectures that this leads to "an overestimate of LLMs' true performance"—the model appears to understand more than it does.

This is not merely a theoretical concern. The paper's own experiments demonstrate the effect empirically: when the same content is expanded to ten options, model accuracy drops precipitously (16–33% across models, Figure 4), suggesting that a substantial portion of MMLU's reported accuracy was attributable to the ease of the distractor structure rather than genuine comprehension.

2. Knowledge-driven rather than reasoning-driven questions. The paper makes a direct claim that MMLU questions—particularly in STEM subjects—primarily test factual recall rather than multi-step reasoning. The evidence for this comes from the Chain of Thought (CoT) inversion pattern: on MMLU, using CoT prompting actually decreases performance for several models (GPT-4-Turbo drops 0.2%, Llama-3-8B drops 3.9%, Gemma-7B drops 3.6%, per Table 3). This is telling because CoT prompting is designed to improve multi-step reasoning—if a benchmark's scores go down when models think more carefully, it strongly suggests the questions reward quick pattern matching over deliberation.

The authors frame this as a limitation of the benchmark, not of CoT. On knowledge-recall questions, intermediate reasoning steps can introduce errors (a calculation mistake, a misremembered fact) that wouldn't occur in direct answering, degrading performance. The benchmark therefore measures a mix of knowledge retrieval and reasoning, but the knowledge component dominates and saturates quickly, while the reasoning component is too sparse to differentiate frontier models.

3. Dataset noise creating a low ceiling. The paper identifies a non-trivial portion of MMLU questions as "either unanswerable or mistakenly annotated" (Section 1). When a benchmark contains questions where the labeled "correct" answer is actually wrong, or where insufficient information is provided to determine any answer, the maximum achievable score is capped below 100%. If enough noise exists to cap the ceiling at ~91% and models are scoring 86–87%, the remaining headroom is just 4–5 percentage points—an extremely narrow band that makes differentiation impossible. This is the "ceiling effect" problem: models hit the noise floor, not their true capability limits.

The paper quantifies this in the expert review process (Table 1), documenting 350 incorrect answers in the MMLU subset alone, plus 1,953 false negative distractors (options incorrectly marked as wrong that were actually correct). These errors directly constrain the maximum possible score, compressing the range in which models can be ranked.

Where Existing Benchmarks Fall Short

The paper situates its contribution relative to a landscape of existing benchmarks, each with documented limitations:

MMLU (Hendrycks et al., 2020): The primary predecessor and baseline. As described above, it suffers from saturation, prompt sensitivity, and a knowledge-heavy composition that fails to distinguish reasoning capabilities. The paper's initial filtering process (Table 4) reveals that 42% of MMLU's questions (5,886 of 13,937) are answered correctly by more than four of eight tested small models (Llama-2-7B, Mistral-7B, Gemma-7B, etc.), qualifying them as "too easy." In disciplines like Business (60% filtered), History (57% filtered), and Psychology (55% filtered), more than half the questions were trivial for relatively modest models—making them useless for evaluating frontier systems.

AGIEval (Zhong et al., 2023): Focuses on general-exam questions from standardized tests (SAT, Gaokao, GRE). While these exams are designed to discriminate among human test-takers, they share MMLU's limitations: they typically use four-option formats and test a blend of knowledge and reasoning that may not stress-test frontier LLMs. The paper does not evaluate AGIEval directly, but the structural parallel is clear—any four-option, knowledge-weighted benchmark faces the same saturation trajectory.

ARC (Clark et al., 2018): A science-focused benchmark that the paper acknowledges as valuable but notes is domain-constrained. MMLU-Pro explicitly aims for broader multi-discipline coverage (14 domains spanning STEM, humanities, and social sciences) while raising difficulty, distinguishing it from single-domain reasoning benchmarks.

BBH (Suzgun et al., 2023): Tests hard synthetic reasoning tasks drawn from BIG-Bench. The paper acknowledges BBH as targeting complex reasoning, but MMLU-Pro's positioning is different: rather than synthetic hard tasks, it targets naturalistic expert-level problems drawn from real college exams and professional curricula. This matters because synthetic benchmarks can introduce distributional artifacts that test model generalization to odd patterns rather than real-world competence.

BigBench (Srivastava et al., 2022): The broadest existing benchmark suite, spanning 204 tasks. However, its breadth comes at the cost of difficulty calibration for frontier models. Many BigBench tasks were designed when model capabilities were lower, and some have been effectively solved. The suite's diversity also makes it harder to use as a single headline metric—MMLU's success as a standard was partly due to its simplicity as a unified score.

HellaSwag (Zellers et al., 2019): Tests commonsense reasoning through sentence completion, but the paper positions it as measuring a narrower capability than the multi-discipline language understanding that MMLU-Pro targets.

The paper's key contribution to this landscape is a targeted upgrade rather than a completely from-scratch benchmark. MMLU-Pro preserves MMLU's broad-coverage philosophy and multi-discipline structure while addressing its three specific failure modes: shallow distractors → expanded to 10 options per question; knowledge-dominated → increased proportion of reasoning-intensive STEM problems; noisy data → two-round expert review to eliminate errors. This "upgrade" strategy preserves comparability with MMLU (enabling direct before/after analysis like Figure 4), which a completely new benchmark would lose.

How This Paper Positions Itself

The paper frames MMLU-Pro as a discriminative, stable, and reasoning-focused benchmark that specifically targets the expert-level performance regime where MMLU saturates. Several design principles reflect this self-positioning:

Discrimination over raw accuracy. The paper explicitly values separating models more than measuring absolute knowledge. The finding that the gap between GPT-4o and GPT-4-Turbo widens from 1% on MMLU to 9% on MMLU-Pro (Section 1, Finding 2) is presented as a feature, not a bug—the benchmark reveals differences that MMLU masks. Similarly, the range of scores among top models expands from ~4% on MMLU to ~10% on MMLU-Pro (Section 6.1), allowing finer-grained ranking.

Stability as a benchmark quality metric. The paper treats prompt sensitivity as a benchmark defect, not just a model property (Section 6.3). While the authors acknowledge that prompt sensitivity is "generally attributed to models' lack of robustness," they argue that "a high-quality benchmark should aim to minimize the impact of prompt variability on scores." By testing 24 different prompt styles and showing that score variation shrinks from 4–5% on MMLU to ~2% on MMLU-Pro, the paper demonstrates that deeper distractors and harder questions produce more stable leaderboards—the best model is the best model regardless of exactly how you ask.

CoT as a capability signal, not just a prompting technique. The paper uses the CoT vs. Direct Answer comparison (Table 3) diagnostically: if CoT helps substantially, the benchmark is testing reasoning; if CoT hurts, it's testing knowledge recall. On MMLU-Pro, CoT provides gains of 3.9% to 19.1%, establishing that the benchmark's difficulty comes from multi-step reasoning rather than obscure facts. This positions MMLU-Pro in a different evaluation regime from MMLU, where CoT either doesn't help or actively hurts—implying that future model releases that improve reasoning should show gains on MMLU-Pro even if they plateau on MMLU.

Proficient-level, not impossible. The paper is careful not to overshoot the difficulty target. Unlike adversarial benchmarks designed to be unsolvable, MMLU-Pro aims for questions that are hard but answerable for experts—the analogy is to college-level exam problems that require deliberate problem-solving rather than PhD-level research questions. GPT-4o's 72.6% accuracy (Table 2) demonstrates that the benchmark is challenging but tractable, leaving realistic room for improvement without being demoralizing or random. The error analysis (Section 5.3) further validates this: of GPT-4o's errors, 39% are reasoning failures, 35% are knowledge gaps, and 12% are calculation mistakes—all categories an improved model could address—rather than annotation errors or genuinely insoluble problems.

Practical construction with LLM assistance. The paper is transparent about using LLMs (GPT-4-Turbo for option generation, Gemini-1.5-Pro for false negative detection) in the dataset construction pipeline (Section 3.2, Figure 2). This is not presented as a limitation but as a pragmatic choice that scales: generating six plausible distractors for 12,000+ questions without LLM assistance would be prohibitively expensive. The paper explicitly verifies that GPT-4-Turbo "does not gain additional advantage from such an augmentation procedure" (Section 3.2), addressing the concern that LLM-generated options might be systematically easier for the generating model. This is a self-positioning as a benchmark that uses modern LLM capabilities for construction while remaining a fair and challenging test for those same models.

3. Technical Approach

3.1 Reader Orientation

This paper presents a benchmark dataset construction pipeline—a systematic procedure for taking existing multiple-choice questions from various sources and upgrading them into a harder, more robust, and reasoning-focused evaluation suite. The system solves the problem of benchmark saturation by transforming four-option knowledge-recall questions into ten-option reasoning-demanding problems while simultaneously filtering out noisy and trivial items, producing a dataset where even the strongest available models score well below ceiling (72.6% for GPT-4o), leaving substantial measurable headroom for future progress.

3.2 Big-Picture Architecture (Diagram in Words)

The pipeline has five major stages, executed sequentially:

1. Initial Filtering — removes questions that are too easy for current models to discriminate among frontier systems. Questions answered correctly by a majority of small reference models are discarded.

2. Question Collection and Integration — aggregates new questions from external sources (STEM Website, TheoremQA, SciBench) and normalizes them into a uniform four-option multiple-choice format compatible with the filtered MMLU core.

3. Option Augmentation — expands every question from four to ten answer choices using GPT-4-Turbo to generate six additional plausible but incorrect distractors, dramatically reducing the probability of correct guessing and increasing the reasoning required for discrimination.

4. Expert Review (Two-Phase) — Phase 1: human experts verify answer correctness and remove questions unsuitable for multiple-choice format. Phase 2: a frontier LLM (Gemini-1.5-Pro) re-evaluates all options to flag potential "false negatives" (correct options mislabeled as distractors), followed by targeted human verification of these flagged items.

5. Evaluation Protocol — defines a standardized 5-shot Chain-of-Thought prompting strategy with answer extraction heuristics to ensure consistent and reproducible model assessment.

Information flows linearly: source questions enter → filtered for difficulty → normalized to four-option format → expanded to ten options → expert-verified for correctness and distractor validity → stored as final benchmark items → deployed with standardized evaluation prompts.

3.3 Roadmap for the Deep Dive

• First, the initial filtering criterion—how "too easy" is operationalized using a committee of small models and why this particular threshold was chosen over alternatives like human difficulty ratings.
• Second, the question collection and integration process—how questions from heterogeneous sources (raw STEM problems with solutions, TheoremQA's short-answer format, SciBench's college exam questions) are normalized into a consistent multiple-choice structure, including the critical answer extraction step.
• Third, the option augmentation procedure—the prompt engineering for GPT-4-Turbo to generate plausible distractors, the verification that this does not advantage the generating model, and why ten options specifically (not eight, not twelve).
• Fourth, the dual-phase expert review—the categorization of detected issues (incorrect answers, false negative options, bad questions) with concrete counts and the role of Gemini-1.5-Pro as a second-pass verifier.
• Fifth, the evaluation protocol—the 5-shot CoT prompt design, the two-stage regular expression answer extraction, the fallback random selection, and how this protocol enables fair comparison across models with different output formatting tendencies.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a dataset construction and evaluation methodology paper whose core idea is that benchmark difficulty, stability, and discriminative power can be systematically engineered through three interventions—expanding distractors, increasing reasoning content, and eliminating noise—applied to an existing broad-coverage benchmark.

---

Initial Filtering: Operationalizing "Too Easy"

The first stage of the pipeline removes questions that fail to challenge current language models. Rather than relying on human judgments of difficulty (which might not align with what models find easy), the paper operationalizes "too easy" through a committee-of-models approach.

The filtering criterion. A question is classified as too easy and removed if it is answered correctly by more than four out of eight reference models (Section 3.2). The reference committee consists of: Llama-2-7B, Llama-2-7B-Chat, Llama-2-13B, Llama-2-13B-Chat, Mistral-7B, Gemma-7B, Yi-6B, and Yi-6B-Chat. These are deliberately chosen as relatively small, accessible models (6–13 billion parameters) rather than frontier models, establishing a conservative threshold—if even these modest models collectively get a question right, it is almost certainly trivial for GPT-4-class systems.

The threshold of "more than four" (i.e., five or more) means a majority of the eight-model committee must answer correctly. This is a stricter bar than requiring unanimous agreement (which might filter too aggressively, removing questions that are actually challenging but happen to be well-handled by all tested models). It is also stricter than a simple majority of a smaller set, since eight models provide more statistical power. The choice of specifically eight models and five-model agreement is a design parameter without explicit ablation in the paper, but the rationale is implicit: it identifies questions that are consistently easy across diverse model architectures and training recipes.

Scale of filtering. Across all subjects, 5,886 questions out of the original 13,937 MMLU items (42.23%) are filtered out (Table 4). The per-discipline filtering rates vary dramatically: Business loses 60.18% of its questions, History loses 56.56%, and Psychology loses 54.50%, while Math loses only 16.18% and Chemistry only 26.73%. This variation reveals which domains MMLU was most saturated in: knowledge-heavy, fact-recall-oriented disciplines (Business, History, Psychology) had the highest proportion of questions that even small models could answer, while computation-heavy disciplines (Math, Chemistry, Physics) retained more challenging items. This pattern validates the paper's diagnosis that MMLU's ceiling was driven by knowledge questions saturating while reasoning questions still provided some differentiation.

Why this approach over alternatives. The paper could have used human difficulty labels (e.g., the MMLU dataset's own subject categories or external difficulty ratings), but model-measured difficulty has a crucial advantage: it directly captures what models find hard, which may differ from human intuitions. A question about a obscure historical fact might seem difficult to a human annotator but be trivially retrievable for a model trained on Wikipedia. Conversely, a question requiring multi-step arithmetic might seem easy to a human but trip up models that make calculation errors. Using model performance as the filtering criterion ensures that the retained questions genuinely test current model limitations.

A subtle implication: because the filtering committee consists of models from early-to-mid 2023 (Llama-2 era), questions that were borderline in 2023 might become too easy in 2024 as models improve. The paper does not discuss periodic re-filtering, which would be necessary to maintain the benchmark's difficulty over time. This is an implicit assumption that the option augmentation and reasoning-content increase (subsequent stages) provide sufficient difficulty headroom that base-question easiness becomes less relevant.

---

Question Collection and Integration: Normalizing Heterogeneous Sources

After filtering, the pipeline incorporates new questions from three external sources to increase both the volume and the reasoning intensity of the dataset. Each source presents a different integration challenge.

Source 1: STEM Website. These are problems from online STEM education platforms, provided in the format of a problem statement paired with a full solution (not just an answer). To convert these into multiple-choice questions, the paper uses GPT-4-Turbo with a structured extraction prompt (Table 6, "STEM Website Prompt Instruction"). The model is instructed to produce a JSON output with five fields:

• image_question: boolean indicating whether the question requires a figure
• image_answer: boolean indicating whether the answer requires a figure
• short_answer: boolean indicating whether the answer can be expressed as a short phrase
• answer: the extracted short answer phrase
• plausible answers: three additional incorrect but plausible options

The image_question and image_answer fields serve as filters: questions requiring non-textual elements are excluded from the dataset since MMLU-Pro is text-only. The short_answer field identifies questions amenable to multiple-choice format—problems whose answers are long derivations or proofs are excluded. Questions where the extracted answer is "incomplete or incorrect" are removed through manual comparison of the extracted answer against the original solution (Section 3.2).

Source 2: TheoremQA (Chen et al., 2023). These are human-annotated questions requiring application of specific theorems for resolution, provided as question-answer pairs without distractors. The paper uses a one-shot prompt (Table 6, "TheoremQA Prompt Instruction") that demonstrates the desired output format by showing how a simple geography question gets transformed: the model receives a question and its correct answer, and must produce the same question with the correct answer plus three plausible distractors.

The one-shot example—"What is the capital city of France?" with answer "Paris" transformed to options including Paris, Madrid, Rome, and Berlin—trains the model to generate distractors that are plausible (other European capitals) rather than random (arbitrary city names). The key design principle is that distractors must be from the same semantic category as the correct answer, requiring domain knowledge to distinguish.

Source 3: SciBench (Wang et al., 2023). These are advanced science questions from college exams, already in a format compatible with MMLU-Pro (questions with answer choices). They require no format conversion but do undergo the same option augmentation and expert review as all other questions.

Distribution of sources. The final dataset draws 56.60% of questions from original MMLU (filtered), 33.93% from the STEM Website, 4.97% from TheoremQA, and 4.50% from SciBench (Figure 3b, Table 5). The heavy reliance on STEM Website questions for Engineering (93.08%), Chemistry (65.46%), and Biology (69.46%) reflects the paper's goal of increasing reasoning intensity—STEM problems from these platforms typically require formula derivation, multi-step calculation, and principled application of concepts rather than fact recall.

Subject restructuring. The original MMLU's 57 fine-grained subject categories are merged into 14 broader disciplines (Section 3.2, Figure 3a). For example, MMLU's separate categories for "High School Mathematics," "College Mathematics," and "Elementary Mathematics" are all folded into a single "Math" discipline. This coarsening serves two purposes: it makes per-discipline sample sizes large enough for meaningful score comparisons (the smallest discipline, History, has 381 questions; the largest, Math, has 1,351), and it reduces the temptation to over-interpret small score differences in niche subcategories.

---

Option Augmentation: From Four to Ten Choices

This is the single most impactful design intervention in MMLU-Pro. Expanding the answer choice set from four to ten options transforms the statistical properties of the benchmark, the cognitive demands it places on models, and its robustness to prompt variations.

The augmentation procedure. For each question that arrives with four options (from filtered MMLU, from TheoremQA conversion, or from STEM Website extraction), GPT-4-Turbo is prompted with a one-shot example (Table 6, "Option Augmentation Prompt Instruction") to generate six additional options labeled E through J. The prompt includes an example where a United Nations Security Council question with four options (one correct, three incorrect) is expanded with six new distractors, all of which are structurally similar to the original options (different configurations of permanent and rotating membership, veto powers) but factually incorrect.

The one-shot example demonstrates the desired properties for generated distractors: they should be plausible (a model with imperfect knowledge might select them), structurally similar (same format and level of detail as correct options), and unambiguously wrong (experts can definitively rule them out). Generating distractors that satisfy all three properties simultaneously is non-trivial—too obvious distractors don't increase difficulty, while ambiguous distractors create annotation errors where the "incorrect" option might actually be correct under some interpretation.

Scale and exceptions. The ideal output is ten options per question, but the paper acknowledges that "83% have ten options, 17% have fewer, and the average options count per question is 9.47" (Section 3.2, Expert Review). Options get removed during expert review when they are identified as false negatives (actually correct) or when they are judged too similar to the correct answer to be clearly distinguishable. The paper does not discard questions that end up with fewer than ten options, prioritizing answer quality over rigid format adherence.

Why ten options specifically. The paper does not provide an explicit ablation comparing eight, ten, or twelve options, but the choice of ten follows from clear statistical reasoning. With four options, the random guessing baseline is 25%—a model with zero knowledge gets one in four questions right by chance. With ten options, the random baseline drops to 10%. This 15-percentage-point reduction in the chance floor means that measured accuracy becomes a more faithful reflection of actual capability: a model scoring 40% on the ten-option version is substantially more competent than one scoring 40% on the four-option version, because far less of that score is attributable to lucky guesses.

More subtly, the deeper distractor set changes the cognitive strategy required. With four options, a model can often succeed through elimination: rule out two obviously wrong choices, then guess between the remaining two. With ten options, elimination alone rarely narrows the field to a single answer—the model must positively identify the correct answer through reasoning or knowledge. This shift from "which of these is right?" to "do I know enough to pick the right one from this large set?" is the mechanism by which option expansion increases the benchmark's sensitivity to genuine understanding.

Verification against self-advantage. A critical concern when using an LLM to generate benchmark content is that the same model (or a similar one) might have an inherent advantage on the generated items—for instance, if GPT-4-Turbo generates distractors that are systematically less plausible to GPT-4-class models than to other models. The paper explicitly checks this: "In experiments, we found that GPT-4-Turbo does not gain additional advantage from such an augmentation procedure" (Section 3.2). This is presumably tested by comparing GPT-4-Turbo's relative performance on questions with GPT-generated distractors versus questions with human-written distractors, though the specific methodology for this verification is not detailed.

---

Expert Review: Two-Phase Quality Assurance

The expert review process is the human-in-the-loop stage that catches errors introduced or inherited in earlier pipeline stages. It operates in two sequential phases, each targeting different error types (Section 3.2, Figure 2).

Phase 1: Verification of Correctness and Appropriateness. Human experts—the paper describes them as "expert annotators" but does not specify their qualifications beyond domain expertise—review every question for three criteria:

1. Answer correctness: Is the designated correct answer actually correct? This catches both pre-existing errors in the original MMLU (the paper documents these but does not provide a breakdown of MMLU-only error rates) and errors introduced during the STEM Website answer extraction step, where GPT-4-Turbo might extract an incomplete or incorrect short answer from a longer solution.

2. Format suitability: Is the question appropriate for a multiple-choice format? The paper specifically excludes "proof problems, true or false questions, and open-ended questions" (Section 3.2, Expert Review). These question types require free-form responses or binary judgments that do not fit the ten-option selection paradigm.

3. Information completeness: Does the question provide all necessary information to derive an answer? Questions requiring "non-textual elements like images or tables" are excluded since MMLU-Pro is a text-only benchmark. Questions with insufficient textual information to reach a conclusive answer are also removed.

Phase 2: Ensuring Distractor Validity. This phase targets a specific failure mode of the option augmentation process: false negatives—distractor options that were generated as "incorrect" but are actually correct or arguably correct under a reasonable interpretation. The phase works as follows:

1. LLM re-evaluation: Gemini-1.5-Pro is prompted to answer every question (Table 6, "False Negative Options Recall Prompt Instruction"). When Gemini-1.5-Pro selects an option that is labeled as a distractor (not the designated correct answer), that option is flagged as a potential false negative. The choice of Gemini-1.5-Pro rather than GPT-4-Turbo for this stage is strategic: using a different model family reduces the risk that the same systematic errors present in the distractor generation model (GPT-4-Turbo) would be replicated in the verification model, providing independent error detection.

2. Targeted human review: Human experts rigorously review only the flagged options. This is a cost-efficiency measure—rather than having humans verify all 72,000+ generated distractors (12,032 questions × 6 generated distractors each), they focus on the subset that an independent strong model identifies as suspicious. The experts determine whether each flagged option is genuinely a false negative (in which case it is removed, reducing the option count for that question) or whether Gemini's selection reflects model error rather than a genuine annotation problem.

Quantified error distribution. Table 1 reports the distribution of identified issues across the four data sources:

• Incorrect Answers: 350 from MMLU (pre-existing errors in the original dataset), 483 from STEM Website (primarily from flawed answer extraction), 11 from SciBench, 0 from TheoremQA. The asymmetry between TheoremQA (zero incorrect answers) and STEM Website (483) highlights the risk in the automated answer extraction step—TheoremQA's answers were human-annotated from the start, while STEM Website answers passed through GPT-4-Turbo.

• False Negative Options: 1,953 from MMLU, 293 from STEM Website, 15 from SciBench, 5 from TheoremQA. The large MMLU count (1,953) likely reflects errors in the original MMLU distractors that went undetected in prior work, as well as new errors introduced during the 4→10 option expansion where generated distractors were accidentally correct. The smaller counts for TheoremQA and SciBench are expected since these sources had fewer total questions, but the per-question rates are comparable.

• Bad Questions: 385 from MMLU, 862 from STEM Website, 15 from SciBench, 1 from TheoremQA. The STEM Website dominates here because its source material—raw exam problems—often includes figures, tables, or problem setups unsuitable for text-only multiple-choice adaptation.

Why two phases. The phased design separates two conceptually distinct quality concerns. Phase 1 addresses whether the benchmark items are valid as questions—can they be answered correctly with the provided information, and are the designated answers correct? Phase 2 addresses whether the benchmark items are valid as multiple-choice items—do the distractors actually distinguish knowledgeable respondents from guessers, or are there "secret" correct answers hiding among the wrong options? A benchmark could pass Phase 1 (all designated correct answers are correct) but fail Phase 2 (some distractors are also correct, inflating scores for models that happen to select them), or vice versa. The two-phase structure ensures both dimensions of validity are independently verified.

---

Evaluation Protocol: Standardized Prompting and Answer Extraction

The pipeline concludes with a standardized evaluation protocol designed to ensure consistent and comparable measurements across diverse models with different output formatting tendencies.

5-shot Chain-of-Thought prompting. The primary evaluation uses 5-shot CoT prompting adapted from the Chain-of-Thought Hub (Fu et al., 2023). For each of the 14 disciplines, five demonstration examples are selected from a validation subset of MMLU-Pro (Section 4). Each demonstration consists of a question, its ten options, and a step-by-step reasoning chain ending with the answer formatted as "The answer is (X)" where X is the letter of the chosen option.

The paper introduces two enhancements to the standard Chain-of-Thought Hub approach. First, the demonstration examples are updated to reflect the expanded ten-option format—the original Chain-of-Thought Hub examples assumed four options. This is a mechanical but necessary change: if the demonstrations showed four-option selection, models might be confused when presented with ten options at test time. Second, the five examples per discipline are "representative"—the paper states they were selected for this property but does not detail the selection criteria. The implication is that they cover the typical reasoning patterns required in that discipline (e.g., formula application in Physics, case law interpretation in Law), providing the model with a template for the expected reasoning style.

Table 7 provides the full prompt template for the Physics discipline, showing the initial instruction ("The following are multiple-choice questions (with answers) about physics. Think step by step and then finish your answer with 'The answer is (X)' where X is the correct letter choice."), followed by five worked examples, and ending with the target question. Each demonstration example includes a "Let's think step by step." preamble in the answer, reinforcing the CoT behavior.

Why 5-shot. The paper does not ablate shot count, but five examples is a standard choice in the LLM evaluation literature—enough to establish the format and reasoning pattern without exceeding context window limits for smaller models. The CoT Hub origin of the demonstrations provides some standardization with prior work, though the adaptation to ten options means direct comparison with legacy 5-shot MMLU results is confounded by the demonstration content change.

Comparison with Direct Answering. The paper also evaluates models using Direct Answering (DA), where the prompt asks for immediate answer selection without intermediate reasoning. The DA prompt format is not explicitly shown but is presumably identical to the CoT prompt minus the reasoning chain in the demonstrations and the "Think step by step" instruction. This enables the CoT vs. DA comparison in Table 3, which serves as the diagnostic for whether MMLU-Pro successfully tests reasoning.

Two-stage answer extraction. Extracting a clean letter answer from model outputs is non-trivial because models may deviate from the requested format. The paper implements a two-stage regular expression extraction with a fallback (Section 4):

1. Primary regex: answer is \(?\([A-J]\)?\) — This matches the format explicitly requested in the prompt: "The answer is (X)." The regex is flexible enough to handle optional parentheses and whitespace variations (the \(? and \)? make the parentheses optional), but expects a single letter from A through J.

2. Secondary regex: \.*\[aA\]nswer:\s*\([A-J]\) — If the primary regex fails (e.g., because the model wrote "Answer: (C)" with a colon instead of "is"), this catches the alternative format. The \.* at the start matches any preceding text, [aA] handles case variation, and \s* absorbs variable whitespace before the parenthesized letter.

3. Fallback random selection: If both regexes fail to extract a valid option (A–J), a random choice is made from the available options. The paper does not report the frequency of fallback activation, which is a notable omission—if a substantial fraction of answers require random selection, the reported accuracies would be biased toward the chance rate of 10% for those questions.

Why two-stage regex rather than one comprehensive regex. Regex engineering for LLM evaluation involves a tradeoff between recall (extracting an answer whenever the model intended to provide one) and precision (not extracting false answers from text that happens to contain a letter in parentheses). A single comprehensive regex risks either being too strict (missing valid answers in unexpected formats) or too permissive (extracting letters from reasoning text that happen to match the pattern). The two-stage approach is a pragmatic compromise: the first regex targets the explicitly requested format (high precision, moderate recall), the second catches common deviations (lower precision since it matches anywhere in the text, but higher recall), and the fallback handles the remainder without crashing the evaluation pipeline.

Why the CoT format specifically. The prompt instructs models to produce step-by-step reasoning before answering. This serves both an evaluation purpose (assessing whether models can reason through problems) and a diagnostic purpose (the error analysis in Section 5.3 relies on reading the model's reasoning to categorize mistakes). Without the reasoning trace, classification of errors as "reasoning errors" versus "lack of knowledge" versus "calculation errors" would be impossible—the evaluators would only see the final wrong answer, not the process that produced it.

The extraction protocol's reliance on format compliance (models must output an answer in a parseable format) creates a potential bias: models with better instruction-following capabilities may score higher not because they know more, but because their answers are more reliably extractable. The "No Selection Made" error category in the error analysis (5% of GPT-4o's errors) and "Answer Extraction Errors" (1%) confirm that extraction failures do occur even for strong models, though the rates are low enough to not dominate the overall accuracy figures.

---

Design Choices: Summary of Strategic Decisions

The construction pipeline embodies several strategic choices that distinguish MMLU-Pro from prior benchmark efforts:

Committee-based difficulty filtering instead of human difficulty ratings. Human ratings capture intended difficulty; model performance captures realized difficulty for the target population. Since MMLU-Pro evaluates LLMs, LLM-measured difficulty is the more relevant metric. The specific choice of small reference models (6–13B parameters) rather than frontier models for the filtering committee ensures the retained questions are hard for state-of-the-art systems, not just for weak baselines.

LLM-assisted distractor generation with cross-model verification. Using GPT-4-Turbo to generate six distractors per question is a scalability decision—manual distractor authoring for 12,000+ questions would be prohibitive. Using Gemini-1.5-Pro (a different model family) for the verification pass reduces the risk that systematic errors in distractor generation go undetected. This creates a two-model adversarial dynamic: one model generates, a different model attempts to detect flaws, humans adjudicate.

Ten options as the sweet spot. Ten reduces the random baseline from 25% to 10% without making the multiple-choice format unwieldy (20 or 50 options would be cognitively overwhelming even for human test-takers). The paper does not provide an ablation, but the choice reflects an implicit assumption that the marginal benefit of additional distractors diminishes beyond some point—each additional plausible distractor is harder to generate and verify, while providing progressively less additional discrimination.

Discipline-level demonstration examples. Providing five CoT examples per discipline rather than a single set of five for all questions ensures the demonstrations are domain-appropriate—a Physics reasoning chain looks different from a Law reasoning chain, and showing the model the right template per discipline should improve its ability to produce format-compliant answers.

Preserving MMLU compatibility. By using MMLU as the core dataset (56.6% of questions) and maintaining the same broad-coverage, multi-discipline structure, MMLU-Pro enables direct before/after comparisons (Figure 4) that would be impossible if the benchmark were built entirely from scratch. This compatibility is a practical advantage: researchers familiar with MMLU's format and evaluation conventions can adopt MMLU-Pro with minimal workflow changes.

4. Key Insights and Innovations

Innovation 1: The "Ceiling Effect" as an Engineering Problem, Not an Inevitability

The most conceptually significant move in this paper is reframing benchmark saturation from an observation about model progress into a solvable benchmark design problem. Prior to MMLU-Pro, the dominant narrative treated benchmark saturation as a natural consequence of models getting better—the implicit assumption was that when GPT-4 scores 86.4% on MMLU, that's because GPT-4 truly understands 86.4% of the tested material, and the remaining gap reflects genuine model limitations. This paper argues something fundamentally different: a substantial fraction of that ceiling is artifactual, created by benchmark construction choices that can be systematically reversed.

The evidence for this reframing is the diagnostic decomposition in Section 1. The authors do not simply assert that MMLU is too easy—they identify three distinct, addressable failure modes: insufficient distractors (making correct answers too identifiable), knowledge-heavy question composition (rewarding retrieval over reasoning), and annotation noise (capping the maximum achievable score below 100%). Each is a benchmark property, not a model property. The implication is radical: by changing the benchmark rather than the models, you can recover substantial discriminative headroom. The 16–33% accuracy drop on MMLU-Pro relative to MMLU (Figure 4) is not simply "the questions are harder"—it is a quantitative measurement of how much of MMLU's reported accuracy was benchmark artifact rather than genuine capability.

This reframing connects to a broader methodological shift in the evaluation literature. Prior work on benchmark limitations (Alzahrani et al., 2024; Zheng et al., 2023) focused on robustness—showing that scores fluctuate under perturbations—but largely treated those fluctuations as inevitable noise. MMLU-Pro goes further by demonstrating that benchmark properties (option count, reasoning content, annotation quality) directly control both the score level and the stability, making them engineering levers rather than fixed constraints. This transforms benchmark design from a curation exercise into an optimization problem: given a target discrimination range and stability budget, how should you set distractor depth, reasoning ratio, and noise tolerance?

The practical payoff of this reframing is methodological permission: if benchmark ceilings are artifacts, then the community should expect to periodically rebuild its evaluation suites as models improve, rather than treating any single benchmark as a permanent yardstick. MMLU-Pro's explicit design as an "upgrade" to MMLU—preserving structure while changing difficulty parameters—models this iterative approach.

---

Innovation 2: CoT Performance Inversion as a Diagnostic for Reasoning Content

The paper introduces a simple but powerful diagnostic for distinguishing knowledge-driven from reasoning-driven benchmarks: compare Chain-of-Thought performance against Direct Answering. If CoT improves scores, the benchmark tests multi-step reasoning; if CoT hurts, the benchmark tests knowledge recall where intermediate steps introduce error without providing benefit.

This diagnostic is not the paper's main contribution—the CoT vs. Direct Answer comparison is a standard evaluation technique (Wei et al., 2022; Suzgun et al., 2023). What makes this paper's use of it innovative is the inversion it reveals as a property of the benchmark rather than the method. On MMLU, CoT decreases performance for several models: GPT-4-Turbo drops 0.2%, Llama-3-8B drops 3.9%, Gemma-7B drops 3.6% (Table 3). On MMLU-Pro, the same models gain substantially: GPT-4-Turbo improves 15.3%, Llama-3-8B improves 3.9%, Gemma-7B improves 6.7%. The same prompting technique, the same models, dramatically opposite effects depending on which benchmark is being evaluated.

This inversion serves as an existence proof for the paper's central claim about MMLU's composition. If MMLU were a genuine reasoning benchmark, CoT should help—the original CoT paper (Wei et al., 2022) demonstrated exactly this for reasoning tasks. The fact that CoT hurts on MMLU means the questions are structured such that reasoning introduces more error than it resolves, which is characteristic of fact-recall tasks where the model either knows the answer or doesn't. The inversion on MMLU-Pro—where CoT provides gains of 3.9% to 19.1%—validates that the benchmark construction pipeline (particularly the STEM Website questions and option augmentation) successfully shifted the cognitive demand from retrieval to reasoning.

The broader significance is that the CoT inversion provides a portable diagnostic: any benchmark can be classified along the knowledge-reasoning spectrum by measuring whether CoT helps or hurts relative to direct answering. This is more informative than relying on question source labels (which may not reflect what models actually do) or human difficulty judgments (which may not align with model failure modes). The paper does not explicitly propose this diagnostic as a general tool, but its use here demonstrates the principle.

A subtle implication: if CoT performance relative to Direct Answering is a benchmark property rather than (solely) a model property, then comparing models on a benchmark where CoT hurts is measuring something different from comparing them on a benchmark where CoT helps. Two models might rank identically on knowledge recall but diverge substantially on reasoning—or vice versa. MMLU-Pro's explicit reasoning focus means its leaderboard ranks a different capability profile than MMLU's leaderboard, which may explain why model orderings differ between the two benchmarks for some model pairs.

---

Innovation 3: Option Count as a Stability Mechanism, Not Just a Difficulty Lever

The expansion from four to ten options is the most visible design change in MMLU-Pro, and the paper could have framed it purely as a difficulty intervention—more options means harder questions, simple as that. What makes the treatment of option count innovative is the empirical demonstration that it also stabilizes model scores against prompt variation, and the implicit argument that deeper distractor sets reduce the impact of surface-level benchmark gaming.

The evidence is Figure 5 and the accompanying analysis in Section 6.3. Across 24 different (but reasonable) prompt styles, model score variation on MMLU-Pro is consistently around 2%, with a maximum of 3.74%. On MMLU, the same models show variation of 4–5%, with peaks up to 10.98%. This is not a small effect—the benchmark's sensitivity to how you ask the question is cut by more than half. The paper attributes this to the reduced role of guesswork: with four options, a model uncertain between two plausible answers can flip its selection based on subtle prompt cues, causing score swings. With ten options, the model either knows the answer with sufficient confidence to select it consistently, or it's guessing among many options with roughly equal (low) probability, making the outcome less sensitive to prompt-induced biases.

This finding has implications that go beyond the specific benchmark. It suggests that benchmark stability is partially controllable through distractor design—a claim the evaluation literature has not systematically explored. Prior work on prompt sensitivity (Zheng et al., 2023; Alzahrani et al., 2024) treated it primarily as a model robustness problem, recommending improvements to model training or inference. MMLU-Pro demonstrates that benchmark construction choices can achieve some of the same stability gains without changing the models at all. This shifts part of the responsibility for leaderboard consistency from model developers to benchmark designers.

The mechanism likely operates through the statistical properties of the multiple-choice format. With four options, the decision boundary between "model knows the answer" and "model is guessing" is relatively narrow—the model's internal probability distribution over options can shift significantly with small prompt perturbations, potentially crossing the threshold where a different option wins. With ten options, the probability mass is spread more thinly, so small perturbations are less likely to change the argmax. This is a direct consequence of expanding the choice set, but the paper deserves credit for measuring and documenting it rather than treating it as an obvious side effect.

The practical consequence is that MMLU-Pro scores are more actionable for model comparison. When two models differ by 2% on MMLU-Pro, that gap is more likely to reflect genuine capability differences than when the same two models differ by 2% on MMLU, because the MMLU gap could easily be reversed by a different prompt choice. This makes MMLU-Pro a more reliable tool for tracking progress, which is the paper's stated goal.

---

Innovation 4: Error-Taxonomy as Benchmark Validation, Not Just Model Diagnosis

The error analysis in Section 5.3 is superficially a standard component of any benchmark paper—analyze where the best model fails, categorize the errors, suggest improvement directions. But the paper uses this analysis in a more sophisticated way: as validation that the benchmark's difficulty comes from the intended sources rather than from annotation errors or format artifacts.

The taxonomy of GPT-4o's errors—39% reasoning errors, 35% knowledge gaps, 12% calculation errors, plus small fractions of extraction failures (1%), annotation errors (2%), and generation issues (2%)—tells a specific story. The dominant error categories (reasoning and knowledge) are genuine capability limitations. The annotation error rate (2%) confirms that the expert review process successfully eliminated most benchmark noise—if annotation errors were 10% or 15%, the benchmark would be measuring dataset quality as much as model quality. The extraction error rate (1%) validates that the answer extraction protocol works reliably for strong models, though the paper's silence on extraction failure rates for weaker models is a limitation.

This use of error analysis as benchmark self-diagnosis is methodologically important. A poorly constructed benchmark might show high difficulty (models score low) but for the wrong reasons—unanswerable questions, ambiguous wording, annotation errors. By decomposing why the best model fails, the paper provides evidence that MMLU-Pro's difficulty is "legitimate"—it comes from problems that are solvable in principle (a human expert could answer them) but that stress-test current model capabilities in interpretable ways.

The distinction between reasoning errors (39%) and calculation errors (12%) is particularly notable. The paper explicitly separates these "to aid model developers, as many AI systems can effectively utilize calculators or Python for complex, multi-step calculations." This acknowledges that calculation is a tool-use problem, not a reasoning problem—GPT-4o could likely get many of those 12% correct if given access to a code interpreter. The 39% reasoning errors are the deeper challenge, requiring architectural or training improvements rather than better tool integration. This decomposition helps model developers prioritize: fixing calculation yields 12 percentage points, but closing the remaining gap to human expert performance requires addressing the 39% reasoning failures.

The error analysis also provides indirect evidence about the benchmark's difficulty calibration. If the error distribution were dominated by "lack of specific knowledge" (e.g., 70% knowledge gaps), that would suggest the benchmark tests obscure facts rather than generalizable skills—hard in a trivial sense. The roughly even split between reasoning and knowledge errors, with a substantial minority of calculation errors, indicates that MMLU-Pro's difficulty is multi-factorial, testing a combination of recall, logical processing, and numerical competence. This aligns with the paper's framing of the benchmark as measuring "proficient-level multi-discipline language understanding and reasoning" rather than any single narrow capability.

5. Experimental Analysis

Evaluation Methodology

• Dataset. MMLU-Pro consists of 12,032 questions across 14 disciplines, sourced from filtered MMLU (56.6%), STEM Website (33.9%), TheoremQA (4.97%), and SciBench (4.5%). The dataset is evaluated as a single test set; there is no mention of a separate held-out validation or training split—the full 12,032 questions constitute the evaluation corpus. Disciplines range from computationally intensive subjects (Math: 1,351 questions, Physics: 1,299) to knowledge-intensive subjects (Law: 1,101, History: 381), with the distribution detailed in Figure 3a and Table 5.

• Base model(s). The paper evaluates over 50 LLMs spanning both closed-source and open-source families. Closed-source models include GPT-4o, GPT-4-Turbo, Claude-3-Opus, Claude-3-Sonnet, Gemini-1.5-Pro, Gemini-1.5-Flash, and Yi-Large. Open-source models span Llama-3 (8B, 70B, 70B-Instruct, 8B-Instruct), Llama-2-70B, Phi-3 (medium-4k-instruct at 14B parameters, mini-4k-instruct at 3.8B), DeepSeek-V2-Chat, Qwen1.5 (110B, 72B-Chat), Yi (34B, 1.5-34B-Chat), Mixtral (8x7B, 8x7B-Instruct), Mistral-7B, Gemma (7B, 2B), MAmmoTH2-8x7B-Plus, InternMath (20B-Plus, 7B-Plus), Starling-7B, c4ai-command-r-v01, OpenChat-3.5-8B, Zephyr-7B-Beta, Neo-7B-Instruct, and Llemma-7B. These were chosen to represent "widely-used models in current production environments" (Appendix A.3), spanning the range from compact 2B-parameter models to frontier proprietary systems.

• Metrics. The primary metric is accuracy—the percentage of questions for which the model's selected answer matches the ground-truth correct option. Answers are extracted via the two-stage regex pipeline described in Section 3. If both regexes fail, a random option is selected as fallback. The paper reports overall accuracy (across all 12,032 questions) and per-discipline accuracy for selected representative subjects (Math, Physics, Engineering, History, Law, Psychology in Table 2; full results on the project leaderboard). For the MMLU vs. MMLU-Pro comparison, MMLU accuracy is reported using the original four-option MMLU evaluation protocol, enabling direct score comparison (Figure 4). For prompt sensitivity analysis, the metric is the range of accuracy scores across 24 prompt styles (Figure 5, Figure 1 center panel).

• Baselines. MMLU (Hendrycks et al., 2020) serves as the primary comparison benchmark, with model scores on the original MMLU reported alongside MMLU-Pro scores for the same models (Figure 4, Table 3). The paper does not construct alternative benchmark baselines (e.g., MMLU with eight options but no reasoning-content increase, or MMLU with only noise removed but option count unchanged) that would isolate individual design choices. Direct Answering (DA) serves as a prompting baseline to quantify the benefit of Chain-of-Thought reasoning (Table 3). For the prompt robustness analysis, 24 "different but reasonable prompts" (Section 6.3) serve as the perturbation set, though the paper does not enumerate these prompts in the main text.

• Generation budget / compute accounting. For closed-source API models, the paper reports processing "approximately 20M input tokens and 5M output tokens" for the full MMLU-Pro evaluation (Appendix A.4). For open-source models, evaluation is conducted on NVIDIA A100 GPUs with vLLM acceleration, taking approximately 20–30 minutes for a 7B-parameter model on the full dataset. There is no generation budget constraint in the evaluation protocol—models produce one full response (reasoning chain plus answer) per question, with no search, sampling multiplicity, or best-of-N aggregation. This makes the evaluation a pure pass@1 measurement under the specified prompting regime.

• Cross-validation / statistical protocol. The paper does not report confidence intervals, standard deviations, or statistical significance tests for accuracy scores. The prompt robustness analysis uses 24 prompt variants per model per benchmark but reports only the score ranges (Figure 5), not the full distribution statistics. For the error analysis (Section 5.3), 120 erroneous GPT-4o predictions are randomly sampled and manually categorized by expert annotators, but inter-annotator agreement is not reported. For the CoT vs. Direct Answer comparison (Table 3), each number is a single accuracy measurement with no error bars.

Main Quantitative Results

Overall Performance and Model Ranking

Table 2 presents the headline results for a representative subset of evaluated models across six disciplines plus overall accuracy. The paper reports these under 5-shot Chain-of-Thought prompting (except Gemini-1.5-Pro and Gemini-1.5-Flash, which use 0-shot due to API limitations noted in the table header).

Frontier model performance. GPT-4o achieves the highest overall accuracy at 72.6%, establishing the benchmark's ceiling for current technology. This leaves a 27.4-percentage-point gap to perfect accuracy, compared to an estimated 11.3-percentage-point gap on MMLU (Section 6.1). Claude-3-Opus (68.5%) and Gemini-1.5-Pro (69.0%) form a tight cluster approximately 3.6–4.1 percentage points below GPT-4o. GPT-4-Turbo (63.7%) trails GPT-4o by 8.9 percentage points, a dramatically larger gap than the ~1% difference between these same two models on MMLU (where GPT-4o scores 88.7% and GPT-4-Turbo scores 86.5%, per Figure 4).

Open-source vs. closed-source gap. The strongest open-source model in Table 2 is Llama-3-70B-Instruct at 56.2%, placing it 16.4 percentage points below GPT-4o—a substantially wider gap than on MMLU, where the same model achieves 82.0% (Figure 4), only 6.7 points below GPT-4o. Phi-3-medium-4k-instruct (55.7%), DeepSeek-V2-Chat (54.8%), and Llama-3-70B (52.8%) cluster within a 3.5-percentage-point band, forming a clear open-source frontier tier. The gap from the best open-source model to the worst model in Table 2 (Gemma-2B at 15.9%) spans 56.7 percentage points, confirming the benchmark's wide dynamic range.

Discipline-level differentiation. Across disciplines, the score range among top models varies substantially. In Math, GPT-4o scores 76.1% while Mixtral-8x7B-Instruct scores 36.3%—a 39.8-point spread. In Law, the spread is narrower: GPT-4o at 51.0% versus Gemma-2B at 12.3% (38.7 points). Notably, Engineering emerges as the hardest discipline: GPT-4o scores only 55.0%, and Claude-3-Opus drops to 48.4%. The paper attributes this to the high proportion of STEM Website questions in Engineering (93.08%, per Table 5) requiring "complex formula derivations and multi-step calculations" (Section 5.2). In contrast, Psychology and History show higher floors—the lowest-scoring models still achieve 16.1% and 15.4% respectively, suggesting these knowledge-intensive disciplines are more accessible even to small models.

MMLU vs. MMLU-Pro Difficulty Comparison

Figure 4 presents a side-by-side comparison of ten representative models on both MMLU and MMLU-Pro. This is the paper's central quantitative argument for the benchmark's increased difficulty and discriminative power.

Score compression and expansion. On MMLU, four mid-tier models—Gemini-1.5-Flash (78.9%), Llama-3-70B-Instruct (82.0%), Phi-3-medium-4k-instruct (78.0%), and Qwen1.5-110B (80.2%)—fall within a 4-percentage-point band (78–82%). On MMLU-Pro, the same four models span a 10-percentage-point range (49.9% to 59.1%), more than doubling the spread. The paper frames this as evidence that MMLU-Pro "is more discriminative than MMLU in distinguishing the nuances between models" (Section 6.1).

Top-tier differentiation. The gap between the three frontier models—GPT-4o (72.5%), Claude-3-Opus (68.5%), and GPT-4-Turbo (63.7%)—spans approximately 8.8 percentage points on MMLU-Pro, compared to roughly 2.2 percentage points on MMLU (88.7%, 86.8%, 86.5% respectively). The paper highlights this as a key finding: "the gap between GPT-4o and GPT-4-Turbo is 1% on MMLU, while it becomes 9% on MMLU-Pro" (Section 1, Finding 2).

Model-specific drops. The accuracy drop from MMLU to MMLU-Pro is not uniform across models. GPT-4o drops 16.2 percentage points (88.7% → 72.5%), while Llama-2-70B drops 32.2 points (69.7% → 37.5%). Gemma-7B drops 32.3 points (66.0% → 33.7%). This non-uniform drop is evidence that the benchmark's added difficulty differentially affects models—some are relying more heavily on the shallow distractor structure and knowledge-recall content of MMLU that MMLU-Pro eliminates. The paper does not systematically analyze which model properties predict larger drops, but the pattern suggests that weaker reasoning capabilities (as distinct from knowledge breadth) are penalized more heavily on MMLU-Pro.

CoT vs. Direct Answering: The Reasoning Diagnostic

Table 3 presents the CoT versus Direct Answering (DA) comparison for five models on both MMLU and MMLU-Pro. This is the paper's evidence for the claim that MMLU-Pro tests reasoning while MMLU tests knowledge recall.

MMLU: CoT hurts or barely helps. On MMLU, GPT-4o improves by only 1.5% with CoT (87.2% DA → 88.7% CoT). GPT-4-Turbo actually declines by 0.2% (86.7% → 86.5%). Llama-3-8B drops by 3.9% (66.6% → 62.7%), and Gemma-7B drops by 3.6% (66.0% → 62.4%). Only Phi-3-medium-4k-instruct shows a modest gain of 1.4% (78.0% → 79.4%). The paper interprets this pattern as evidence that MMLU's questions are "mostly knowledge-driven without requiring too much reasoning" (Section 1).

MMLU-Pro: CoT substantially helps. On MMLU-Pro, the CoT advantage is large and consistent across models. GPT-4o improves by 19.1% (53.5% DA → 72.6% CoT), nearly a one-fifth uplift. GPT-4-Turbo improves by 15.3% (48.4% → 63.7%), Phi-3-medium-4k-instruct by 8.2% (47.5% → 55.7%), Llama-3-8B by 3.9% (31.5% → 35.4%), and Gemma-7B by 6.7% (27.0% → 33.7%). The paper presents this as validation that MMLU-Pro "includes more complex reasoning questions" (Section 6.2) where step-by-step deliberation genuinely aids in reaching the correct answer.

Asymmetric patterns across models. The CoT benefit is not uniform—GPT-4o gains 19.1% while Llama-3-8B gains only 3.9%. This likely reflects the weaker model's lower baseline reasoning ability: if a model struggles to produce coherent multi-step reasoning, CoT may not help as much, or may even introduce compounding errors. The paper does not analyze this interaction between model capability and CoT benefit, but the pattern is consistent with the interpretation that MMLU-Pro's difficulty is reasoning-dependent and that reasoning quality varies across models.

Prompt Robustness Analysis

Figure 5 presents the performance variability of six models under 24 different prompt styles on both MMLU and MMLU-Pro. The data is displayed as horizontal bars showing the full range from minimum to maximum accuracy achieved across prompts.

Reduced score variability. On MMLU, the six models show prompt-induced score ranges between approximately 4–5 percentage points, with the paper noting that the maximum observed range across all tested models reaches 10.98% (Section 6.3). On MMLU-Pro, the same models show ranges "generally around 2%, with a maximum of 3.74%." For Llama-3-8B, the MMLU range spans approximately 59.9% to 64.8% (4.9-point spread), while the MMLU-Pro range spans approximately 25.2% to 28.7% (3.5-point spread). For Llama-3-8B-Instruct, the MMLU range is approximately 61.4% to 66.0% (4.6 points), while MMLU-Pro is approximately 31.5% to 35.2% (3.7 points).

Mechanism interpretation. The paper attributes this stabilization to the reduced role of lucky guesses. With four options, a model that is uncertain between two plausible answers can have its selection flipped by small prompt-induced shifts in token probabilities. With ten options, the model either selects the correct answer with sufficient confidence to be robust to minor perturbations, or it is guessing among many options where the probability of selecting any specific distractor is low and the particular distractor chosen is less sensitive to prompt details. The paper states this as a benchmark quality property: "a high-quality benchmark should aim to minimize the impact of prompt variability on scores, ensuring more consistent and reliable evaluations" (Section 6.3).

Missing analysis. The paper does not report whether the specific prompts that produce high versus low scores are consistent across models—if GPT-4o and Llama-3-8B both score highest on the same prompt variants, that would suggest the prompt format itself is systematically easier (e.g., better matching pre-training patterns) rather than random variation. Similarly, the paper does not report means or medians alongside the ranges, making it impossible to assess whether the distribution of scores across prompts is symmetric or skewed.

Error Analysis of GPT-4o

The paper conducts a detailed manual review of 120 randomly selected erroneous predictions from GPT-4o, the best-performing model (Section 5.3). Expert annotators classified each error into one of eight categories based on the model's generated reasoning trace and the ground-truth answer.

Reasoning Errors (39%): The largest single category. Even when GPT-4o recalls correct domain knowledge, it "frequently encounters difficulties with logical reasoning" (Section 5.3). The paper provides a concrete example in Table 10: when calculating the pressure difference inside and outside a container, the model erroneously added internal and external pressures together rather than subtracting them. The paper attributes these failures to the model's "dependence on recognizing patterns in training data rather than engaging in a true understanding of the problem." This is consistent with the CoT benefit observed in Table 3—CoT helps the model structure its reasoning, but it doesn't eliminate reasoning errors entirely, even for the strongest model.

Lack of Specific Knowledge (35%): Domain-specific knowledge gaps, such as incorrect financial calculations (Table 8: GPT-4o used the full product price of $1,650 as the principal for interest calculation instead of subtracting the down payment to get$1,600) and misapplication of optical principles (Table 9: the model subtracted refractive indices rather than computing their ratio when applying the lens maker's formula across media). These errors reflect genuine gaps in the model's training distribution rather than failures to apply known information.

Calculation Errors (12%): The paper distinguishes these from reasoning errors "to aid model developers, as many AI systems can effectively utilize calculators or Python for complex, multi-step calculations." A concrete example in Table 11: when applying Graham's law of effusion, GPT-4o correctly identified the formula but miscalculated the molecular weight of the compound C₁₈H₃₄O₃ as 218 g/mol instead of 298 g/mol, leading to an incorrect diffusion rate. The paper implies these 12% of errors represent relatively "easy" gains—they could be addressed through tool integration rather than architectural improvements.

Other Errors (14%): The remaining errors include No Selection Made (5%, the model responded but failed to output a parseable answer choice), Question Understanding Errors (4%, such as Table 13 where GPT-4o interpreted Singer's equality principle as applying to all beings rather than specifically to humans), Generation Issues (2%, such as Table 12 where the model entered a repetitive loop and repeated the same sentence until hitting the token limit), Annotation Errors (2%, where the ground-truth label was incorrect), and Answer Extraction Errors (1%, where the model's answer format defeated the regex extraction pipeline).

Small annotation error rate as validation. The 2% annotation error rate on GPT-4o's 120 analyzed errors translates to approximately 2–3 annotation errors out of 120 sampled questions. Extrapolated to the full 12,032-question dataset, this suggests a residual noise rate of roughly 2%, which the paper implicitly presents as evidence that the two-phase expert review successfully eliminated most benchmark noise. However, the paper does not independently verify the annotation error rate across the full dataset—the 120-sample estimate has substantial uncertainty, and annotation errors may be concentrated in specific disciplines or source types.

Ablation Studies and Robustness Checks

The paper does not conduct traditional ablations (e.g., varying the number of options, testing different filtering thresholds, comparing different distractor generation strategies). The primary comparative analyses—MMLU vs. MMLU-Pro (Figure 4, Table 3) and CoT vs. Direct Answering (Table 3)—serve as the main mechanism-validation experiments rather than ablations in the conventional sense. However, several design choices are implicitly validated through data patterns:

• Discipline-level filtering rates as implicit difficulty validation. Table 4 reports the percentage of MMLU questions filtered out per discipline as "too easy." The variation—from 16.18% in Math to 60.18% in Business—validates the filtering criterion: disciplines with more reasoning content (Math, Chemistry, Physics) retained more questions, while knowledge-heavy disciplines (Business, History, Psychology) had higher filtering rates. This is not an ablation but provides internal consistency evidence that the committee-of-models filter successfully identifies knowledge-recall questions as easier for current models.

• Source distribution as difficulty mechanism. Table 5 shows that disciplines with high proportions of STEM Website questions (Engineering: 93.08%, Chemistry: 65.46%, Biology: 69.46%) tend to produce lower model scores (Engineering is the hardest discipline overall per Section 5.2). This is not an ablation in the controlled sense—the paper does not compare questions from different sources within the same discipline while holding difficulty constant—but the pattern is consistent with the claim that externally sourced STEM problems increase reasoning difficulty.

• Option count variation as a natural experiment. Since 17% of questions have fewer than ten options due to false-negative removal during expert review (with an average of 9.47 options per question), there exists a natural variation in option count within the dataset. The paper does not analyze whether model accuracy varies systematically with the number of options per question, which would have been a direct test of the distractor-depth hypothesis.

• The CoT vs. DA comparison across models is an implicit ablation of reasoning content. The fact that CoT helps more on MMLU-Pro than on MMLU for every tested model (Table 3) is evidence that the benchmark change (not model-specific factors) drives the CoT benefit. However, this comparison confounds the option-count change and the reasoning-content change—it doesn't isolate which of the two interventions is responsible for the increased CoT benefit.

Missing ablations that would have strengthened the paper:

• Option count ablation. The paper could have constructed an MMLU-Pro-4 variant with the same questions but only four randomly selected options (one correct plus three distractors from the full ten). Comparing MMLU-Pro-4 vs. MMLU-Pro-10 would isolate the effect of distractor depth from the effect of question content changes.

• Filtering threshold ablation. Varying the "more than four out of eight" criterion (e.g., "more than two," "more than six") and measuring the resulting dataset's difficulty and discrimination would validate the specific threshold choice.

• Distractor generation method comparison. Comparing GPT-4-Turbo-generated distractors against human-written distractors (or distractors from a different model) on a subset would directly test the "no additional advantage" claim.

• Shot count ablation. Comparing 0-shot, 1-shot, 5-shot, and 10-shot CoT performance would reveal whether the 5-shot choice is optimal and whether different models show different scaling with demonstration count.

• Per-discipline prompt sensitivity. The paper reports aggregate prompt sensitivity (Figure 5) but does not report whether prompt stability varies by discipline—reasoning-heavy disciplines might show different sensitivity patterns than knowledge-heavy ones.

Critical Assessment

The experiments in this paper are structured to validate four central claims: (1) MMLU-Pro is substantially harder than MMLU, causing a 16–33% accuracy drop; (2) MMLU-Pro is more discriminative, spreading models over a wider score range; (3) MMLU-Pro tests reasoning rather than knowledge recall, as evidenced by CoT helping rather than hurting; and (4) MMLU-Pro is more robust to prompt variation, with reduced score fluctuation across prompt styles.

Claim 1 (increased difficulty): Supported, but with important confounds.

The accuracy drops in Figure 4—ranging from 16.2 points for GPT-4o to 32.3 points for Gemma-7B—unambiguously demonstrate that MMLU-Pro is harder than MMLU for the tested models. However, this comparison conflates three simultaneous changes: (a) removal of 42% of MMLU's easiest questions via filtering, (b) addition of new, harder questions from STEM Website/TheoremQA/SciBench, and (c) expansion from four to ten options. The 16–33% drop is the combined effect, and the paper cannot attribute how much of the drop comes from each intervention. It is entirely possible that simply removing MMLU's easiest 42% of questions (without adding new ones or changing option counts) would produce a substantial fraction of the observed accuracy drop. The paper would be stronger if it had reported scores on the filtered MMLU subset alone (the 8,051 retained questions, still with four options) to isolate the contribution of filtering versus the other two interventions.

Additionally, the difficulty numbers in Figure 4 compare MMLU-Pro (evaluated with 5-shot CoT) against MMLU (evaluated with the standard MMLU protocol, which may differ in shot count and prompting style). If the MMLU baselines use direct answering or a different few-shot configuration, some portion of the accuracy difference could be attributable to evaluation protocol differences rather than benchmark content differences. The paper does not fully standardize the evaluation protocols between the two benchmarks for the Figure 4 comparison.

Claim 2 (increased discrimination): Supported, contingent on the models evaluated.

The expansion of score spread from ~4% on MMLU to ~10% on MMLU-Pro for mid-tier models (Figure 4), and from ~2% to ~9% for frontier models, is clear empirical evidence that MMLU-Pro differentiates models that MMLU clusters. However, this evidence comes from a specific set of model pairs. The paper does not establish whether the discrimination comes primarily from the reasoning-heavy questions selectively penalizing models with weaker reasoning (while knowledge-heavy questions would still cluster), or whether the discrimination is uniform across disciplines. A model achieving 72.6% on MMLU-Pro might be differentiated from a 63.7% model primarily by performance on the externally sourced STEM questions, with both models performing similarly on the filtered MMLU-derived questions. Without per-source or per-discipline breakdowns of the score differences, the nature of the discrimination is unclear.

Furthermore, the paper's discrimination claim is implicitly about ability to separate models that are close on MMLU. But the evidence only shows that models far apart on MMLU-Pro happen to be close on MMLU—it does not test the inverse: whether models that are close on MMLU-Pro would be far apart on MMLU, or whether the rank ordering is actually different between the two benchmarks. The only explicit rank-order comparison given is that GPT-4o > GPT-4-Turbo on both (with a larger gap on MMLU-Pro), which doesn't show a ranking inversion.

Claim 3 (reasoning focus): Supported, with strong internal evidence from the CoT inversion.

The CoT vs. Direct Answer comparison in Table 3 provides the cleanest evidence for this claim. The inversion—CoT hurting on MMLU (for three of five models) while helping substantially on MMLU-Pro (for all five)—is difficult to explain except through the reasoning-content hypothesis. On knowledge-recall questions, intermediate reasoning steps can introduce errors (misremembered facts, calculation mistakes) without providing benefit, explaining why CoT degrades performance. On reasoning-requiring questions, structured decomposition helps, explaining the gains of 8–19% on MMLU-Pro. The magnitude of the effect (GPT-4o gains 19.1% on MMLU-Pro vs. 1.5% on MMLU) makes the signal unambiguous.

However, the paper's interpretation that this demonstrates MMLU-Pro's questions "require deliberate reasoning" is an inference, not a direct measurement. An alternative explanation is that MMLU-Pro's questions are harder in a way that happens to benefit from CoT for reasons unrelated to reasoning depth—for instance, longer question stems might benefit from step-by-step parsing, or the ten-option format might benefit from explicit elimination reasoning even for knowledge-recall questions. The paper does not provide a control where only option count is changed (keeping the same question content) to test whether CoT helps simply because there are more options to choose among. The error analysis partially addresses this by showing concrete reasoning failures in GPT-4o's outputs (Table 10), but the connection between "models make reasoning errors" and "CoT helps because the benchmark tests reasoning" is correlational, not causal.

Claim 4 (prompt robustness): Supported in magnitude, but the analysis is incomplete.

The reduction in score range from 4–5% on MMLU to ~2% on MMLU-Pro (Figure 5) is well-documented. However, the paper does not enumerate the 24 prompts used, describe how they were selected, or characterize whether they represent a systematic exploration of the prompt design space (e.g., varying instruction wording, demonstration ordering, output format instructions) or an ad-hoc collection. Without this information, it is unclear whether the reduced variability reflects genuine robustness or limited exploration—perhaps the 24 prompts vary along dimensions that MMLU is sensitive to but MMLU-Pro is not, while other prompt dimensions (not tested) might affect MMLU-Pro equally.

The paper also does not report whether the score rankings of models are stable across prompts—a model might score consistently but be ranked differently relative to peers under different prompts, which would undermine the benchmark's utility for leaderboard construction. Two models with overlapping score ranges under different prompts might swap rankings even if their individual ranges are narrow.

Genuine weaknesses in the experimental design:

• The 120-sample error analysis extrapolates to the full dataset with no error bars. With 12,032 questions and a 2% measured annotation error rate, the 95% confidence interval based on 120 samples would span roughly 0% to 5%—too wide to confidently claim the benchmark is "clean." A larger verification sample or systematic spot-checking across all disciplines would strengthen this claim.

• Extraction failure rates are not reported. The paper describes a fallback random selection mechanism but does not report how often it is invoked per model. If some models trigger the fallback on 5–10% of questions, their reported accuracies are biased toward the random baseline of 10%—artificially depressing scores for models with poor format compliance rather than poor reasoning. This is particularly concerning for smaller or less instruction-tuned models.

• The "no additional advantage" claim for GPT-4-Turbo on GPT-generated distractors is stated without evidence. The paper says "In experiments, we found that GPT-4-Turbo does not gain additional advantage from such an augmentation procedure" (Section 3.2) but does not show these experiments. Did GPT-4-Turbo score differently on questions with GPT-generated distractors versus questions with human-authored distractors from the original MMLU? Without seeing the data, this claim—which is critical to the benchmark's fairness—cannot be evaluated.

• Single evaluation protocol for the main results. All Table 2 results use 5-shot CoT with a specific prompt format and demonstration set. The paper does not test whether model rankings change under different shot counts, different demonstration examples, or different answer format instructions. If model A outperforms model B under 5-shot CoT but the ranking reverses under 0-shot, the benchmark's leaderboard is contingent on the specific evaluation protocol—a concern the paper acknowledges (prompt sensitivity is one of MMLU's failures) but does not fully test for MMLU-Pro beyond the 24-prompt analysis.

• No human baseline. The paper frames MMLU-Pro as targeting "proficient-level" performance and mentions the goal of "expert-level intelligence" (Section 1), but does not report human accuracy on the benchmark. Without a human ceiling, the 27.4% gap between GPT-4o and perfect accuracy is difficult to interpret—is that gap bridgeable in principle, or does some fraction of it represent genuinely ambiguous or expert-only questions that no AI is expected to solve? MMLU-Pro's predecessor MMLU had established human performance baselines (Hendrycks et al., 2020 reported human expert accuracy around 89.8% on MMLU), providing context for model scores. The absence of a comparable human baseline for MMLU-Pro weakens the interpretation of the accuracy numbers.

• Conflated interventions prevent attributing effects to specific design choices. The three simultaneous changes (filtering, new questions, option expansion) are all reasonable and well-motivated, but the paper cannot disentangle their individual contributions to the improved discrimination, increased difficulty, or enhanced stability. A cleaner experimental design would have evaluated MMLU-Pro variants that apply these changes incrementally, but the paper's construction pipeline makes such ablation difficult—the filtering and option expansion were applied to the integrated dataset, not independently controllable.

Experiments that would have strengthened the paper:

• Ablation of option count: Evaluate the same MMLU-Pro questions with only four randomly selected options (one correct plus three distractors) versus the full ten options. This would directly test the hypothesis that option count alone drives the difficulty increase and stability improvement.

• Comparison with noise-cleaned MMLU: Evaluate models on the 8,051 filtered MMLU questions (before option expansion) to measure how much of the observed score drop comes purely from removing easy/noisy questions.

• Per-source performance breakdown: Report accuracy separately for MMLU-derived questions, STEM Website questions, TheoremQA questions, and SciBench questions to reveal whether model differentiation is driven primarily by externally sourced reasoning problems or by the expanded-option MMLU core.

• Human performance on MMLU-Pro: Even a small-scale human evaluation (e.g., 100 questions across disciplines) would provide a critical interpretability anchor for the model scores.

• Model ranking stability analysis: For the 24-prompt robustness experiment, report rank correlations between prompts to test whether model ordering (not just score magnitude) is stable.

• Systematic extraction failure analysis: Report per-model extraction failure rates and compare results with and without the fallback random selection to quantify bias from format non-compliance.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Completely Externalized

The assumption or constraint. The entire construction pipeline assumes that difficulty assessment—deciding which MMLU questions are "too easy" and should be filtered—can be performed once during benchmark creation and will not need repetition. The paper uses a committee of eight small models tested on the full MMLU dataset to identify and remove 5,886 questions (Table 4), a process consuming substantial computation. The paper does not address what happens when model capabilities advance beyond the point where this static filtering remains appropriate—questions that were appropriately difficult for 2024 models may become trivial for 2026 models, recreating exactly the saturation problem MMLU-Pro was designed to solve.

This is not a hypothetical concern. The filtering committee consists of models from the Llama-2 era (Llama-2-7B/13B, Mistral-7B, Gemma-7B, Yi-6B), and the filtering threshold of "answered correctly by more than four of eight" was calibrated against these specific models. If a future generation of small models (say, Llama-4-7B) can answer 90% of the same questions correctly, the retained questions would no longer be "hard"—they would be the new "too easy." The benchmark would need periodic re-filtering against updated reference models, but the paper provides no protocol, schedule, or criteria for when or how to do this.

The consequence. MMLU-Pro inherits the same temporal fragility that doomed MMLU, just with a higher initial difficulty ceiling. The saturation clock is ticking from the moment of release—the only question is how many years the benchmark remains discriminative before its difficulty decays below the threshold where frontier models cluster again. The paper cannot answer this question because it does not model the rate of capability improvement or provide a mechanism for benchmark difficulty to track model progress. A practitioner evaluating whether to adopt MMLU-Pro as a long-term evaluation standard would need to know its expected useful lifetime, which is unknown.

Additionally, the filtering committee's composition introduces a dependency on the specific model families chosen. If the eight reference models share architectural biases (all are dense transformers trained on similar data distributions), questions that are easy for that family might be hard for models with different architectures (e.g., state-space models, retrieval-augmented systems), and vice versa. The filtering would systematically remove questions that are easy for dense transformers while retaining questions that are hard for them—but these retained questions may be differentially easy or hard for other architectures, creating a benchmark with architecture-specific difficulty that advantages or disadvantages certain model families.

What evidence exists in the paper. The paper reports the filtering rates per discipline (Table 4) and the reference models used (Section 3.2) but provides no analysis of whether the difficulty ranking of retained questions is stable across model architectures, no plan for periodic re-filtering, and no estimate of how quickly the benchmark's difficulty will degrade. The paper's own data shows that the filtering was aggressive—42% of MMLU questions removed—which means the retained questions were genuinely difficult for the reference models. But this tells us nothing about their difficulty for the next generation of models.

Mitigation status. Not addressed. The paper does not discuss temporal robustness of the filtering step, propose a versioning scheme for the benchmark (e.g., MMLU-Pro-2024, MMLU-Pro-2026), or suggest mechanisms for dynamic difficulty adjustment. This is a fundamental limitation of any static benchmark but is particularly acute for MMLU-Pro given the explicit motivation of solving MMLU's saturation—solving it once does not solve it permanently. The paper's silence on this point means the burden of determining when the benchmark has saturated again falls entirely on the community, with no guidance from the benchmark's creators.

---

The Benchmark Is Evaluated Entirely Under a Single, Unvalidated Prompting Protocol

The assumption or constraint. All primary results in Table 2 use 5-shot Chain-of-Thought prompting with a specific set of five demonstration examples per discipline, drawn from the Chain-of-Thought Hub (Fu et al., 2023) and adapted for ten options. The paper asserts these demonstrations are "representative" (Section 4) but does not describe the selection criteria, validate that they are unbiased, or test whether alternative demonstration sets produce different model rankings. The demonstrations are drawn from a "validation subset" of MMLU-Pro, but this subset is not described—its size, composition, and relationship to the full 12,032-question test set are unspecified.

This is a significant omission because few-shot prompting is known to be sensitive to demonstration choice. The specific examples a model sees can influence its output format, its reasoning style, and even its factual recall (through in-context learning effects). Two different sets of five demonstrations—both "representative" under some reasonable criterion—could produce systematically different accuracy estimates and potentially different model rankings. The paper acknowledges prompt sensitivity as one of MMLU's failures (Section 1, citing Zheng et al. 2023 and Alzahrani et al. 2024) and demonstrates that MMLU-Pro reduces score magnitude variation across 24 prompts (Figure 5), but never tests whether model rankings are stable across reasonable variations in the evaluation protocol—different demonstration sets, different shot counts, different answer format instructions.

The consequence. A model developer comparing their system against the Table 2 leaderboard cannot know whether a 2% accuracy difference reflects genuine capability superiority or an artifact of the specific five demonstrations used. If the paper's demonstration set happens to favor certain reasoning patterns—e.g., examples heavy on algebraic manipulation will advantage models trained on math-heavy corpora, while examples emphasizing conceptual explanation will advantage models trained on explanatory text—then the leaderboard rankings are contingent on the demonstration selection. A model that would outperform GPT-4o under a different (equally reasonable) set of demonstrations might appear inferior under the paper's chosen set.

The 24-prompt robustness experiment (Figure 5) partially addresses this by showing that score magnitudes are stable, but stability of magnitudes does not imply stability of rankings. Two models could have narrow, non-overlapping score ranges that produce consistent rankings, or narrow, overlapping ranges that produce inconsistent rankings. The paper reports only the range of scores for each model individually, not the joint distribution across prompts, so pairwise ranking stability cannot be assessed from Figure 5.

For the CoT vs. Direct Answer comparison (Table 3), the specific demonstration content is even more consequential. If the CoT demonstrations are poorly constructed—e.g., containing reasoning errors, using atypical reasoning styles, or being too long and distracting—they could artificially depress CoT performance relative to Direct Answering for some models, inflating the apparent CoT benefit. The paper provides one example prompt (Table 7, Physics) showing five demonstrations, but does not provide the demonstrations for other disciplines or analyze their quality.

What evidence exists in the paper. Table 7 shows the full Physics prompt with five demonstrations—these appear well-constructed and error-free for that discipline. The 24-prompt experiment in Figure 5 tests prompt wording variation (different instructions, different phrasing) but does not test demonstration content variation (different example questions, different reasoning chains per example). The paper reports that prompts were selected from "24 different but reasonable prompt styles" (Section 6.3) without enumeration, so we cannot assess whether the tested variation covers the dimensions most likely to affect model rankings. The per-model score ranges in Figure 5 are narrow (~2% on MMLU-Pro), which is encouraging, but this only addresses score magnitude stability, not ranking stability.

Mitigation status. Partially addressed by the narrow score ranges in Figure 5, which suggest that the specific wording of the instruction and the prompt structure does not dramatically shift individual model scores. However, the paper does not test the effect of changing the demonstration examples themselves—the five few-shot exemplars are held constant across the 24 prompt variants. This leaves open the possibility that demonstration choice is a more impactful variable than instruction wording, and that the benchmark's reported stability may not extend to this dimension. The paper does not discuss this as a limitation or propose a standardization for demonstration selection in future evaluations.

---

Hardest Questions Remain Unanswerable for All Models, With No Clear Path to Improvement

The assumption or constraint. The paper constructs MMLU-Pro to be more difficult than MMLU, and it succeeds—the best model (GPT-4o) achieves 72.6%, leaving a 27.4% gap to ceiling. However, the paper provides no analysis of whether the hardest questions in the benchmark (those that no model answers correctly) are answerable in principle for language models, or whether they represent a fundamentally different category of difficulty that test-time scaling, better prompting, or larger models cannot address. The error analysis (Section 5.3) examines only GPT-4o's errors—120 samples from a single model—which tells us about that model's failure modes but nothing about whether the questions no model gets right share common characteristics that make them categorically harder.

The paper's FLOPs-matched analysis (referenced in Section 7 of the original MMLU-Pro paper? Actually, this paper does not contain FLOPs-matched analysis—this is a benchmark paper, not a methods paper. The limitation is about characterizing difficulty, not about allocation strategies.) The benchmark paper's own error analysis reveals that 35% of GPT-4o's errors stem from lack of specific knowledge and 39% from reasoning failures. But these percentages are for GPT-4o's errors specifically—they do not tell us about questions that all models get wrong. If the set of universally-failed questions is dominated by annotation errors that survived expert review (the paper documents a 2% residual annotation error rate), then the benchmark's ceiling is a noise floor, not a capability frontier. If they are dominated by questions requiring reasoning beyond current architectures, then the benchmark provides a meaningful stretch target. The paper cannot distinguish these cases.

The consequence. The 27.4% gap between GPT-4o and perfect accuracy cannot be interpreted without knowing its composition. If 5% of questions are genuinely unanswerable (annotation errors, ambiguous wording, missing prerequisite information), the effective ceiling is 95%, and the true headroom is 22.4% rather than 27.4%. If another 10% require multi-modal reasoning or external knowledge access that text-only LMs cannot perform, the ceiling drops further. A model developer using MMLU-Pro to track progress cannot distinguish between "we improved reasoning by 3%" and "we got lucky on 3% of the ambiguous questions," because the benchmark does not characterize which questions are reliably answerable.

The error analysis in Section 5.3 acknowledges 2% annotation errors (on GPT-4o's sample) plus 5% "No Selection Made" (extraction failures where the model didn't choose an option) and 4% "Question Understanding Errors"—categories that may reflect benchmark ambiguity rather than model capability. Together, these categories account for 11% of GPT-4o's errors. If a similar fraction applies to the hardest questions that no model answers, the benchmark's effective ceiling may be substantially below 100%, compressing the room for measurable progress.

What evidence exists in the paper. The error analysis (Section 5.3, Tables 8–13) provides detailed case studies of GPT-4o's specific errors, showing that the benchmark catches genuine capability limitations—reasoning errors like adding instead of subtracting pressures (Table 10), knowledge gaps like incorrect optical formula application (Table 9). These examples demonstrate that some difficult questions are legitimately testing model capabilities. However, the paper does not separately analyze questions that zero or near-zero models answer correctly. It does not report the distribution of per-question accuracy (what fraction of questions stump all models, what fraction are answered correctly by 50%+ of models), which would characterize the benchmark's difficulty structure. It does not test whether the hardest questions are consistently hard across model architectures or are selectively hard for specific model families.

The paper also does not report human performance on MMLU-Pro, which would establish the benchmark's effective ceiling. For MMLU, Hendrycks et al. (2020) reported human expert accuracy of approximately 89.8%, providing a meaningful calibration point—models scoring 86–87% were approaching the human ceiling, explaining the saturation. Without a comparable human baseline for MMLU-Pro, we cannot assess whether GPT-4o's 72.6% means "models are 72.6% of the way to human expert performance" or "models are 72.6% of the way to a 95% answerable ceiling"—very different interpretations of the 27.4% gap.

Mitigation status. Not addressed. The paper does not suggest future work on characterizing unanswerable questions, establishing human baselines, or identifying which benchmark items genuinely test capabilities that models can eventually acquire versus those that require fundamentally different architectures. The error analysis is a first step—showing that GPT-4o's errors have interpretable causes—but it does not address the harder question of whether the benchmark's ceiling is achievable in principle.

---

The Benchmark Remains Constrained to the Multiple-Choice Format, Limiting Ecological Validity

The assumption or constraint. MMLU-Pro is exclusively a multiple-choice benchmark. Every question, regardless of source discipline or reasoning depth, is evaluated through the same mechanism: select one option from up to ten choices. The paper explicitly acknowledges this in Section 7:

"The MMLU-Pro dataset, while enhancing the complexity of MMLU by incorporating more challenging, reasoning-focused questions, remains constrained by the limitations of the multiple-choice format. This format may not capture the depth of comprehension and creative response generation as effectively as open-ended answers, which better reflect real-world scenarios."

This acknowledgement is honest but understates the practical consequence. The multiple-choice format imposes a specific cognitive structure on evaluation that may systematically advantage or disadvantage certain reasoning strategies. A model can succeed on a multiple-choice reasoning question by (a) independently deriving the correct answer and matching it to an option, (b) eliminating implausible distractors through partial knowledge, or (c) recognizing surface-level patterns that correlate with correct answers across the benchmark's distribution. Only strategy (a) reflects genuine reasoning competence; strategies (b) and (c) are benchmark-specific skills that may not transfer to real-world tasks where no options are provided.

The paper's ten-option design partially addresses this by making elimination harder (more distractors to rule out), but it does not eliminate the fundamental difference between selecting among provided answers and generating an answer from scratch. A model that can identify the correct pressure calculation among ten numerical options may still fail to compute that pressure when asked in an open-ended format, because the multiple-choice format provides answer verification (the model can check whether its calculation matches any option) that open-ended problems do not.

The consequence. MMLU-Pro scores may overestimate models' genuine reasoning capabilities in deployment scenarios where answers must be generated, not selected. The 19.1% CoT benefit for GPT-4o (Table 3) reflects improved performance on multiple-choice reasoning with answer options visible—the reasoning chain can be guided by checking intermediate conclusions against the available options. In an open-ended setting, the same reasoning chain might produce an answer that cannot be verified against a closed set, and errors that would be caught by "none of the options match my calculation" in multiple-choice would propagate uncorrected.

Additionally, the multiple-choice format excludes entire categories of important language understanding capabilities: explanation generation, argument construction, creative problem-solving where the solution approach is not predetermined, and tasks requiring structured outputs (code, proofs, plans). A model that achieves 90% on MMLU-Pro would be very good at the specific skill of selecting correct answers from provided options under time-unconstrained conditions—but this tells us little about whether it can teach a concept, debug a codebase, or write a legal brief. The paper's positioning of MMLU-Pro as a measure of "proficient-level multi-discipline language understanding" (Section 1) overclaims relative to what the multiple-choice format can actually assess.

What evidence exists in the paper. The paper provides no comparison between multiple-choice and open-ended performance on the same underlying questions. The STEM Website questions were originally open-ended (problem statements with solutions) and were converted to multiple-choice via GPT-4-Turbo answer extraction and distractor generation (Section 3.2), but the paper does not report whether models that get these questions right in multiple-choice format would have produced the correct answer in open-ended format. The error analysis (Section 5.3) relies on reading model-generated reasoning chains—which are available precisely because the CoT format asks models to produce reasoning, not because multiple-choice inherently requires it—but does not test whether the reasoning quality differs when options are absent.

The paper also does not acknowledge that the multiple-choice format may interact with the specific difficulty interventions in unintended ways. Adding six distractors (from four to ten options) makes elimination harder, but it also provides more surface-level cues—a model might learn to recognize that "the correct answer is usually the one with specific numerical precision" or "distractors generated by GPT-4-Turbo tend to have certain linguistic patterns." The paper's verification that GPT-4-Turbo "does not gain additional advantage" from its own generated distractors (Section 3.2) is reassuring but limited to one model and one potential bias; other models might exploit different statistical regularities in the option generation process.

Mitigation status. The paper acknowledges this limitation in Section 7 but treats it as inherent to the benchmark format rather than a design choice that could be addressed through supplementary open-ended evaluation components. The acknowledgement is brief (two sentences) and does not discuss the magnitude of the multiple-choice/open-ended gap, propose future work to characterize it, or suggest that users interpret MMLU-Pro scores as measuring answer selection capability specifically rather than general language understanding.

---

The Dataset Construction Pipeline Introduces Unmeasured Dependencies on Specific LLMs, Creating Potential Unfairness

The assumption or constraint. The MMLU-Pro construction pipeline relies on GPT-4-Turbo for two critical stages: extracting short answers from STEM Website solutions and generating three initial distractors (Stage 2: Question Collection and Integration), and expanding all questions from four to ten options by generating six additional distractors (Stage 3: Option Augmentation). It also uses Gemini-1.5-Pro for the false negative detection pass in Phase 2 of expert review. The paper's verification of fairness is a single sentence:

"In experiments, we found that GPT-4-Turbo does not gain additional advantage from such an augmentation procedure" (Section 3.2).

This claim is neither supported by presented data nor explained methodologically. We do not know: what experiments were conducted (comparison of GPT-4-Turbo's accuracy on GPT-generated vs. human-generated options? Comparison of GPT-4-Turbo's relative ranking vs. other models on these subsets?); what metric was used to assess "additional advantage"; how large the effect would need to be to count as an "advantage"; or whether the finding generalizes to other models from the same provider (GPT-4o, which shares architecture and training data with GPT-4-Turbo, might also benefit from option patterns the generator model inadvertently embeds).

The choice of Gemini-1.5-Pro for false negative detection introduces an asymmetry: a model from Google's model family is used to verify options that may have been generated by OpenAI's model family. This cross-family verification is smart design (it reduces risk of correlated errors), but it means that options flagged by Gemini-1.5-Pro and then removed by human reviewers were removed through a process that involved a specific model's judgment. If Gemini-1.5-Pro has systematic biases—e.g., it is more likely to flag distractors that are plausible under a particular reasoning style as "potentially correct"—the final benchmark's distractor set is shaped by those biases. Models that share Gemini's reasoning biases might find the remaining distractors easier to dismiss, while models with different reasoning styles might find them harder.

The consequence. The benchmark may not be equally fair to all model families. If GPT-4-Turbo's distractor generation produces options that are systematically more plausible to non-OpenAI models (because GPT-4-Turbo's own training makes certain patterns salient to it but not to others), then the benchmark advantages OpenAI models. Conversely, if the generation process inadvertently makes certain distractors easier for GPT-family models to reject (because they recognize their own "tells" in generated text), the benchmark disadvantages OpenAI models. The paper's single-sentence claim cannot distinguish these possibilities.

The Gemini-1.5-Pro verification pass introduces a related concern. Options that survive Gemini review plus human verification may be those that Gemini found unambiguously wrong—but "unambiguously wrong to Gemini" means "Gemini assigned low probability to this option being correct." A model with similar training to Gemini might similarly assign low probability, making the distractor easy to reject. A model with different training might find the distractor more plausible. The verification process therefore potentially calibrates distractor plausibility to Gemini's own internal judgments, making the benchmark differentially difficult for models that differ from Gemini.

The issue extends beyond fairness to reproducibility. If a future researcher wants to recreate MMLU-Pro or extend its methodology to a new domain, they would need access to the same GPT-4-Turbo and Gemini-1.5-Pro model versions. API models are updated over time (GPT-4-Turbo from April 2024 may behave differently from a future version), making exact reproduction impossible. The paper specifies the model version (GPT-4-turbo-2024-04-09) but cannot guarantee its continued availability.

What evidence exists in the paper. Extremely limited. The "no additional advantage" claim (Section 3.2) is the only evidence presented, and it is an assertion without supporting data. The paper does not provide a table comparing GPT-4-Turbo's accuracy on GPT-generated vs. human-generated distractors, does not report relative performance of other models on these subsets, and does not discuss the experimental design that generated this conclusion. Table 1 reports the distribution of issues identified during expert review, which includes Gemini-1.5-Pro's false negative detections, but does not break down these detections by model family of origin or analyze whether the verification process introduced systematic biases.

The error analysis (Section 5.3) provides case studies of GPT-4o's errors but does not separately analyze errors on questions with GPT-generated distractors versus questions with human-authored distractors (e.g., original MMLU questions that retained their human-written wrong options). Such an analysis could partially address the fairness concern by showing whether error patterns differ between these question subsets.

Mitigation status. Not addressed beyond the single-sentence assertion. The paper does not propose a methodology for auditing benchmark fairness with respect to the construction LLMs, does not provide the raw outputs of the distractor generation process for independent verification, and does not discuss this as a limitation that users should consider when interpreting model rankings. The use of LLMs for benchmark construction is a pragmatic choice that the paper is transparent about, but the downstream consequences for benchmark fairness are not explored. A researcher comparing a new model against the Table 2 leaderboard cannot assess whether their model's score is affected by unrecognized dependencies on the construction pipeline's specific model choices.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new model architecture, training objective, or prompting technique. Its contribution is methodological infrastructure, but of a kind that reshapes how the field thinks about evaluation itself. The shift is from treating benchmark difficulty as an emergent property of question collection toward treating it as an engineerable parameter—something you can dial up or down through specific, reproducible design choices: distractor count, reasoning-to-knowledge ratio, and noise elimination.

This reframing matters because it changes the evaluator's job description. Before MMLU-Pro, benchmark creation was largely curation: find good questions, ensure diverse coverage, verify answers. After MMLU-Pro, it is also optimization: what combination of option count, question source, and filtering threshold maximizes discriminative power for a target model population while minimizing prompt sensitivity? The paper doesn't fully solve this optimization—it makes specific, somewhat ad-hoc choices (ten options, five-quintile filtering via an eight-model committee) without ablating alternatives—but it makes the optimization problem visible. Future benchmark designers can now ask, "What is the right distractor depth for my target difficulty range?" rather than accepting four options as a default.

The work also recalibrates expectations around benchmark lifetime. MMLU's saturation—models clustering at 86–87%—was treated as a sign of progress. MMLU-Pro reframes it as a measurement failure: the benchmark ran out of dynamic range before models ran out of capability. The implication is that benchmarks are consumable resources. They have a finite useful life determined by the gap between the best model's score and the noise ceiling, and when that gap narrows to a few percentage points, the benchmark stops providing useful signal. This is obvious in retrospect but was not the operational assumption of the field, which treated benchmarks like MMLU as permanent fixtures. MMLU-Pro models an alternative: iterative benchmark replacement, where each generation of evaluation suite is designed to restore discriminative headroom by adjusting the difficulty parameters that the previous generation's saturation revealed.

The paper also introduces—implicitly, through its CoT vs. Direct Answer comparison—a portable diagnostic for benchmark composition. The pattern of CoT hurting on MMLU but helping on MMLU-Pro (Table 3, gains of 8–19%) is more than evidence for the paper's specific claims. It demonstrates a general principle: the direction and magnitude of the CoT benefit tells you whether a benchmark tests reasoning or knowledge. A benchmark where CoT decreases accuracy is likely dominated by fact-recall questions where intermediate reasoning steps introduce error without providing discriminative value. A benchmark where CoT substantially increases accuracy measures multi-step reasoning. This diagnostic can be applied to any existing or proposed benchmark without modifying it, giving the community a lightweight tool for characterizing what their evaluations actually measure—as opposed to what they claim to measure.

The identification of option count as a stability mechanism (Figure 5, score range dropping from 4–5% on MMLU to ~2% on MMLU-Pro) is another conceptual contribution that transcends this specific benchmark. Prior work on prompt sensitivity (Zheng et al., 2023; Alzahrani et al., 2024) treated it primarily as a model robustness problem. MMLU-Pro demonstrates that benchmark design choices—specifically, making the correct answer harder to identify through elimination—can achieve substantial stability improvements without touching the models. This shifts part of the responsibility for leaderboard consistency from model developers to benchmark designers. A benchmark with high prompt sensitivity isn't just measuring model capability plus noise; it's poorly engineered in a dimension that can be controlled.

The practical upshot for the evaluation landscape is a shift in what counts as a "good" benchmark. Coverage and answer correctness remain necessary but are no longer sufficient. A good benchmark must also demonstrate: (1) sufficient discriminative range among the target model population (not just "models score lower than on the old benchmark"), (2) low score sensitivity to reasonable prompt variations, (3) a CoT benefit that reflects genuine reasoning content, and (4) a noise floor low enough to leave meaningful headroom above the best model's score. MMLU-Pro succeeds on most of these dimensions—discrimination is substantially improved (gap between GPT-4o and GPT-4-Turbo widens from 1% to 9%), stability is better (2% score range vs. 4–5%), CoT helps strongly (19.1% for GPT-4o), and the residual annotation error rate appears low (2% in the error analysis sample). Its primary failure is establishing a human ceiling to contextualize the remaining headroom, a gap the paper acknowledges only implicitly.

Follow-Up Research This Work Enables

1. Directly measuring the contribution of each design intervention through controlled ablation. The paper applies three simultaneous changes to MMLU—filtering easy/noisy questions, adding new reasoning-heavy questions, and expanding from four to ten options—and reports their combined effect (16–33% accuracy drop). A follow-up study should construct MMLU-Pro variants that apply these interventions incrementally: (a) MMLU-Filtered (original MMLU with the 5,886 easiest questions removed, still with four options), (b) MMLU-Filtered+Reasoning (filtered MMLU plus the new STEM Website/TheoremQA/SciBench questions, still with four options), (c) MMLU-Pro-4 (the full MMLU-Pro question set but with only four randomly sampled options per question), and (d) the full MMLU-Pro. Evaluating the same model set on all four variants would decompose the 16–33% drop into components attributable to noise removal, reasoning-content increase, and distractor depth expansion. This would answer whether the benchmark's improved discrimination comes primarily from harder questions or from the ten-option format, with direct implications for future benchmark design—if distractor count alone explains most of the benefit, adding distractors to existing benchmarks is an easy upgrade path; if new question content is the dominant factor, more expensive curation is required.

2. Establishing a human performance baseline to anchor model scores and measure the effective ceiling. MMLU-Pro reports GPT-4o at 72.6% but provides no human comparison. A follow-up should recruit domain experts (graduate students or professionals in each of the 14 disciplines) to answer a stratified random sample of MMLU-Pro questions—say, 100 questions per discipline for 1,400 total evaluations—under conditions comparable to model evaluation (same 5-shot CoT examples visible? Probably not—humans would likely take the test zero-shot or with minimal instructions). The output would be a discipline-level and overall human accuracy figure. This anchors model scores: if human experts achieve 85%, GPT-4o's 72.6% represents 85% of the way to human performance; if humans achieve 95%, the gap is larger and the benchmark's ceiling is higher. The human baseline would also identify questions that experts find ambiguous or unanswerable, directly measuring the benchmark's true noise floor (as opposed to the model-estimated 2% annotation error rate from 120 GPT-4o samples, which has wide confidence intervals). This is the single most important missing piece for interpreting all MMLU-Pro results. A variant could also measure human performance with the 5-shot CoT examples to test whether the demonstration format advantages models over humans.

3. Characterizing which questions no model gets right and whether they form a distinct difficulty class. The paper reports overall and per-discipline accuracies but does not analyze the per-question difficulty distribution. A follow-up should compute, for each of the 12,032 MMLU-Pro questions, the fraction of evaluated models (or a representative subset of strong models) that answer correctly. This would produce a histogram of question difficulty, revealing whether the benchmark has a smooth difficulty curve or a bimodal distribution with a cluster of questions that stump all current models. For questions in the "no model gets right" cluster, the study should manually examine a sample (100–200 questions) to categorize why: are they annotation errors that survived expert review? Questions requiring external knowledge not in any model's training data? Problems requiring multi-modal reasoning impossible for text-only models? Genuinely hard reasoning problems that future models could solve? This analysis would characterize the benchmark's effective ceiling more precisely than a human baseline alone, and would tell model developers which of the remaining 27.4 percentage points of headroom are achievable through better training versus which require fundamentally different model capabilities. The error analysis in Section 5.3 already provides the template—it just needs to be applied to questions rather than to a single model's errors.

4. Testing whether model rankings on MMLU-Pro generalize to open-ended versions of the same questions. MMLU-Pro is exclusively multiple-choice, and the paper acknowledges this constrains ecological validity (Section 7). A follow-up should select a subset of MMLU-Pro questions—particularly from reasoning-heavy disciplines like Math, Physics, and Engineering where the original STEM Website questions had open-ended solutions—and evaluate models on these questions in open-ended format: the question is presented without options, the model generates a free-form answer, and correctness is judged by comparing the generated answer to the ground-truth solution (using automated equivalence checking where possible, human judgment where necessary). The key comparison is whether model rankings on the open-ended subset match rankings on the multiple-choice version. If GPT-4o outperforms Claude-3-Opus by 4.1 percentage points on multiple-choice MMLU-Pro but the gap shrinks or reverses on open-ended versions, the multiple-choice format is inducing ranking artifacts. If rankings are preserved, the benchmark's convenience (easy automated scoring) does not come at the cost of validity. This also addresses the concern that models might solve multiple-choice questions through elimination strategies that don't reflect genuine understanding—open-ended evaluation removes elimination as a strategy.

5. Periodic re-benchmarking to measure difficulty decay and establish a re-filtering schedule. The paper's filtering step used a committee of 2023-era small models (Llama-2, Mistral-7B, Gemma-7B) to identify questions as "too easy" if answered correctly by more than four of eight models (Section 3.2). As models improve, questions that were appropriately difficult in 2024 will become too easy, gradually recreating the saturation problem. A follow-up study should, on an annual or biannual cadence, re-evaluate MMLU-Pro with an updated committee of contemporary small models (e.g., the strongest 6–13B parameter models available at each time point). This would measure the rate of difficulty decay: what fraction of MMLU-Pro questions become "too easy" (by the same >4/8 criterion) each year? With two or three data points, the community could fit a decay curve and predict when MMLU-Pro will reach the same saturation point as MMLU. More importantly, this would inform a benchmark versioning protocol: when the "too easy" fraction exceeds some threshold (say, 20% of questions), release MMLU-Pro v2 with re-filtering, new questions, and potentially further distractor expansion (twelve options? fifteen?). This transforms MMLU-Pro from a one-time upgrade into a maintained evaluation infrastructure.

6. Stress-testing benchmark fairness by measuring per-source performance disparities across model families. The paper's construction pipeline used GPT-4-Turbo for distractor generation and Gemini-1.5-Pro for false negative detection (Section 3.2), with a single unsupported sentence claiming GPT-4-Turbo gains no advantage. A follow-up should systematically test fairness by computing per-model accuracy separately on (a) MMLU-derived questions with human-authored distractors (the original MMLU sub-questions where only the original four options are used, before any GPT-4-Turbo expansion), (b) questions where all ten options were human-authored or come from the original source, and (c) questions where GPT-4-Turbo generated the additional six distractors. If GPT-4-Turbo's accuracy is anomalously high on subset (c) relative to its performance on (a) and (b)—compared to the same ratios for other model families—that's evidence of self-advantage. If Gemini-1.5-Pro's accuracy is anomalously high on questions that survived Gemini-based false-negative detection (compared to questions where false negatives were caught by human review alone), that's evidence of verification bias. The paper's error analysis provides a starting template: the 120 GPT-4o errors could be re-analyzed by source to test whether error rates differ between GPT-distractor and human-distractor questions. A null result (all models show similar relative performance across subsets) would validate the paper's fairness claim with actual evidence. A non-null result would reveal a previously unrecognized limitation requiring mitigation in future benchmark construction efforts.

Practical Applications and Downstream Use Cases

Selecting between frontier models for reasoning-heavy enterprise deployments. Organizations deploying LLMs for tasks requiring multi-step reasoning—financial analysis, legal document review, scientific literature synthesis, engineering design assistance—currently have limited tools for discriminating among top-tier models. MMLU saturates: GPT-4o, Claude-3-Opus, and GPT-4-Turbo all score within ~2% of each other. MMLU-Pro reveals a 9-percentage-point spread among these same three models (72.6%, 68.5%, 63.7%), with GPT-4o showing particular strength in reasoning-intensive disciplines (Math: 76.1%, Physics: 74.7%). A CTO evaluating API providers can use these discipline-level breakdowns to match model selection to their specific workload: a math tutoring platform should strongly prefer GPT-4o over GPT-4-Turbo (13.3-point gap in Math), while a general-knowledge Q&A system might find the 3.6-point overall gap less compelling relative to cost differences. The 24-prompt robustness result (Figure 5) provides additional confidence that these differences are not artifacts of prompt engineering—the model that scores higher does so consistently across prompt variations, making the evaluation actionable for production decisions where prompt formats will vary across use cases.

Calibrating model improvement claims in research papers and technical reports. When a new model release claims "10% improvement on reasoning benchmarks," the claim's meaning depends heavily on which benchmarks are cited. A 10% improvement on MMLU (where models cluster in the 80–87% range) may reflect gains on knowledge recall that saturate the benchmark's remaining headroom, not genuine reasoning progress. MMLU-Pro provides a higher-ceiling, reasoning-weighted evaluation where the same 10% improvement—say, from 63.7% (GPT-4-Turbo) to 73.7%—would represent a substantively different achievement: closing one-third of the gap between the previous state-of-the-art and GPT-4o. Research labs can use MMLU-Pro as part of a standardized evaluation suite alongside MMLU, reporting both scores to demonstrate whether improvements are concentrated in knowledge retrieval (large MMLU gain, small MMLU-Pro gain) or reasoning (small MMLU gain—because it's near ceiling—but large MMLU-Pro gain). The CoT vs. Direct Answer comparison (Table 3) extends this: a model that shows large CoT benefit on MMLU-Pro (like GPT-4o's 19.1%) is genuinely leveraging reasoning for its score, and improvements to reasoning capability should further widen this gap, providing a second dimension of evaluation beyond raw accuracy.

Targeted data collection and fine-tuning for reasoning skill gaps. The per-discipline breakdowns in Table 2 and the error analysis in Section 5.3 provide specific diagnoses that can guide training data investment. If a model underperforms on Engineering (GPT-4o: only 55.0%, the hardest discipline) primarily due to calculation errors (12% of GPT-4o's errors overall, likely higher in computation-heavy Engineering) and reasoning failures (39%), two distinct mitigation strategies are indicated: integrate a code execution tool to eliminate calculation errors (addressing the 12%), and fine-tune on structured reasoning traces from STEM problem-solving to address the 39%. Without MMLU-Pro's discipline-level granularity and error taxonomy, a practitioner seeing a single aggregate accuracy number cannot distinguish between "the model lacks engineering knowledge" (requiring domain-specific pretraining data) and "the model knows the concepts but can't reliably compute answers" (requiring tool integration, a much cheaper fix). The benchmark's construction from diverse sources—particularly the STEM Website questions that dominate Engineering (93.08%, Table 5) and require "complex formula derivations and multi-step calculations"—makes it directly actionable for prioritizing which types of additional training data would most improve real-world STEM application performance. The error analysis template also provides a reusable methodology: evaluate your model on MMLU-Pro, sample 100–200 errors, classify them using the paper's taxonomy, and let the error distribution determine whether to invest in better pretraining data (knowledge gaps), tool use (calculation errors), or reasoning-focused fine-tuning (reasoning errors).