OLMES: A Standard for Language Model Evaluations

ArXiv: 2406.08446

🎯 Pitch

Simply changing how a multiple-choice question is presented to an LLM can slash error rates by nearly 5×, yet most reported benchmarks use undocumented, incompatible setups that make model comparisons meaningless. OLMES is a fully documented open standard that resolves this chaos by standardizing both 'cloze' and multiple-choice formulations, backed by empirical recommendations for every evaluation choice from normalization to prompting.

1. Executive Summary

This paper introduces OLMES (Open Language Model Evaluation Standard), a fully documented, practical, open standard for reproducible LLM evaluations that identifies and standardizes the varying factors in evaluation practices—such as prompt formatting, choice of in-context examples, probability normalizations, and task formulation—across 10 popular MCQA benchmarks and 15 diverse base models. OLMES resolves the core tension between the "cloze" formulation (CF, where each answer choice is scored separately via token probabilities) needed by smaller/weaker models and the multiple-choice formulation (MCF, where answer labels are predicted directly) that stronger models exploit, by standardizing that both formulations are evaluated and the best-performing one is used, establishing that MCF scores can yield nearly 5× lower error rates than CF for strong models (e.g., Llama3-70B achieves 93.7% MCF vs. 69.0% CF on ARC-Challenge). The standard provides justified, per-task recommendations for CF probability normalization—selecting among none, character, token, and pmi based on empirical win rates across 15 models—and standardizes curated 5-shot prompts and instance formatting, delivering a fully reproducible evaluation protocol where previously a statement like "model X scores Y on ARC-Challenge" was uninterpretable due to undocumented variations across references.

2. Context and Motivation

The Core Problem: LLM Evaluation Is Not Reproducible

The central problem OLMES addresses is deceptively simple but pervasive: when someone says "model X scores Y% on ARC-Challenge," that number is often meaningless because there is no standard for what "evaluating on ARC-Challenge" actually entails. The same model, evaluated by different groups on the same dataset, routinely produces different—sometimes dramatically different—performance numbers because the evaluation procedure itself varies in undocumented ways.

This is not a hypothetical concern. Table 1 in the paper demonstrates the problem concretely using ARC-Challenge, a widely-used benchmark for measuring scientific reasoning: Llama2-7B's reported score varies from 43.2% to 53.7% across six different references—a 10.5 percentage point spread. More importantly, the relative ordering of models changes depending on which evaluation setup you consult. Looking at Ref1 (the Hugging Face Open LLM Leaderboard), one would conclude that Llama2-13B (59.4%) and Llama3-8B (60.2%) perform similarly. But Ref6 (the Llama3 model card) reveals a gap of over 10% between them (67.6% vs 78.6%). A researcher deciding which model to build upon would reach opposite conclusions depending on which reference they happened to consult—not because the models changed, but because the measurement instruments were different.

Table 14 in the appendix extends this analysis to OpenBookQA, showing the same pattern: Falcon-7B scores range from 26.0% to 55.2% across five references, and Llama2-7B ranges from 48.4% to 58.6%. The problem is systematic, not isolated to a single dataset or model.

The authors identify two specific failures in current evaluation practice (Section 1):

Failure 1: Comparing against previously reported results is flawed. When a team releases a new model and claims it outperforms existing models on benchmark X, that comparison is only valid if the new evaluation exactly replicates the old evaluation setup. Since prior work often leaves critical details unspecified—as evidenced by the "?" entries in Table 1 denoting undocumented or mixed approaches—these comparisons are based on fundamentally different measurement procedures. A model that appears to "beat" a prior result may simply have been evaluated under a more favorable configuration.

Failure 2: Existing standardization efforts lack documented justification and are not followed. Efforts like the Hugging Face Open LLM Leaderboard (Beeching et al., 2023) and HELM (Liang et al., 2023) have tackled reproducibility by evaluating many models under consistent setups. However, the rationale behind their specific choices—why a particular prompt format, why a particular number of shots, why a particular normalization—is not clearly documented. As a result, subsequent model releases (Touvron et al., 2023a,b; Biderman et al., 2023; Jiang et al., 2023; Groeneveld et al., 2024; AI@Meta, 2024) use their own evaluation setups rather than replicating these standards. The community lacks not just a standard, but a standard with documented justification that researchers can trust and therefore adopt.

Why This Problem Matters

The stakes are substantial, spanning both scientific and practical concerns:

Scientific credibility depends on reproducibility and well-considered comparisons. If two papers report different numbers for the same model on the same task, and neither documents why, the field cannot build cumulative knowledge. Progress claims become unfalsifiable—a model developer can always attribute an apparent regression to "differences in evaluation setup" rather than a genuine degradation. The paper cites Sclar et al. (2023), which showed that "as much as an 80% difference in accuracy on a given task just from varying formatting and in-context examples" can occur. When measurement noise is this large relative to genuine capability differences, the signal-to-noise ratio of published results collapses.

Model development decisions rely on evaluation signals. Evaluations are not just for final leaderboard rankings—they are used during training to decide whether a model is on track, to compare model architectures, to tune hyperparameters, and to decide when to stop training. If the evaluation procedure changes between checkpoints (intentionally or accidentally), these decisions become unreliable. The paper emphasizes that evaluating base models of different strengths is "critical for the community when picking between alternate base models for further training or tuning for their application" (Section 2.3).

Resource allocation depends on performance comparisons. Organizations choosing between Llama2-13B and Llama3-8B (or between training a larger model vs. deploying a smaller one) base these multi-million-dollar decisions partly on benchmark performance. If the benchmarks tell different stories depending on how they're run, the economic consequences of evaluation ambiguity are real.

The problem compounds as models improve. The paper observes that stronger models increasingly use the MCF (multiple-choice formulation) rather than CF (cloze formulation), but report these as identical task names: "25-shot ARC-Challenge." The Llama3 model card reports 78.6% MCF, while the Hugging Face leaderboard reports 60.2% CF for the same model on the "same" task. As models approach human-level performance, these formulation differences produce larger absolute gaps in reported accuracy, making comparisons increasingly misleading.

Where Existing Approaches Fall Short

The paper identifies specific limitations in prior evaluation practices and standardization efforts:

Ad-hoc, undocumented choices in individual model evaluations. The legacy approach—each model paper running its own evaluation with its own setup—is typified by the inconsistencies in Table 1. Different references use different numbers of shots (0 vs. 25), different normalization schemes (char vs. pmi vs. none), different task formulations (CF vs. MCF, sometimes mixed within the same paper), and different sources of in-context examples (curated vs. randomly sampled). Some entries are marked "?" because the evaluation setup was simply not documented. This is not necessarily negligence—the sheer number of decisions required to go from a dataset to a final accuracy number means that exhaustively documenting every choice is burdensome without a standard to reference.

Standardization efforts lack documented rationales. HELM (Liang et al., 2023) and the Hugging Face Open LLM Leaderboard (Beeching et al., 2023) represent important progress toward consistent evaluation—the same setup is used for all models, so relative comparisons within that framework are valid. However, the paper identifies a critical gap: the choices made in these frameworks are presented as defaults without justification. Why 5-shot for some tasks and 10-shot for others? Why character normalization rather than pmi? Without documented reasoning, other researchers have no basis for trusting these choices over alternatives, and thus no motivation to adopt them in their own work. The result is fragmentation: the leaderboard uses one setup, HELM uses another, and individual model papers use their own.

The CF vs. MCF formulation tension is unresolved. This is perhaps the most consequential unstandardized choice. The paper explains that the CF (completion/cloze) formulation was introduced with GPT-3 (Brown et al., 2020) because weaker models could not handle the natural MCF (multiple-choice) format—they lacked the instruction-following ability to understand "choose A, B, C, or D" as a task. CF scores each answer choice independently by substituting it into the prompt and measuring token probabilities, which is unnatural but works with weaker models.

The MCF formulation is how multiple-choice questions are actually posed to humans—presenting the options and asking the model to predict which label is correct. As models have improved, MCF has gradually replaced CF for many evaluations. But the transition has been inconsistent: Llama2-13B was evaluated on ARC-Challenge using CF (scoring ~56%), while Llama3-8B was evaluated using MCF (scoring ~79%). These numbers appear in different papers under the same task name, inviting apples-to-oranges comparisons.

The unresolved question is: which formulation should be used? CF gives better signal for weaker models (where MCF is near random), but MCF gives higher and more representative scores for stronger models. Using only one formulation would either artificially penalize strong models (if using CF, where performance plateaus below true capability) or fail to differentiate weak models (if using MCF, where they hover at random chance). Figure 2 in the paper demonstrates this starkly: on MMLU, the weakest 5 models score essentially at random (around 25%) under MCF, while the same models show meaningful differentiation under CF. Conversely, Llama3-70B achieves 93.7% under MCF but only 69.0% under CF—a 24.7 percentage point gap, representing nearly a 5× difference in error rate.

Probability normalization for CF is an unresolved open question. When using the CF formulation, the model returns $P(a_i|q)$, the probability of an answer choice $a_i$ given the question prompt $q$. Ranking solely by this raw probability favors shorter answers. The paper identifies four approaches used in the literature, each with different implications:

• none: Use raw log-probability $\ln P(a_i|q)$
• token: Normalize by number of tokens: $\ln P(a_i|q) / \text{num\_tokens}(a_i)$ (Brown et al., 2020)
• character: Normalize by number of characters: $\ln P(a_i|q) / \text{num\_characters}(a_i)$ (used by Llama evaluations and Eleuther LM Harness)
• pmi: Pointwise mutual information: $\ln P(a_i|q) / P(a_i|u)$ where $u$ is an unconditional prompt like "Answer:" (Holtzman et al., 2021)

Prior work studied these approaches but left the choice unresolved. HELM settled on different defaults per task without full documentation. The Eleuther LM Harness reports both acc (using none) and acc_norm (using character), leaving the user to decide. Individual model papers used different schemes—sometimes even mixed within the same evaluation (e.g., Brown et al., 2020 selectively used pmi for ARC and OpenBookQA while using token for other tasks). The paper notes a particularly problematic edge case: on BoolQ, where the only answer choices are "yes" and "no" (single tokens, so no length normalization is needed), the character normalization accidentally advantages "yes" because it has one more character than "no"—a normalization scheme that changes the probability of "yes" vs "no" simply because of character count is clearly undesirable.

Few-shot example selection varies arbitrarily. The number of shots used for a given task has varied widely across evaluations: 25-shot for ARC-Challenge on one leaderboard, 0-shot in some model papers, 5-shot for MMLU in most work. When shots are randomly sampled, the distribution of answer labels in those examples can be skewed, potentially biasing the model toward certain answers. The paper notes that random sampling could produce "4 A's and 1 B among the 5 answers," which would not fairly represent the task.

Instance formatting details introduce arbitrary variation. The paper catalogs numerous formatting choices that differ across evaluations: "Question:" vs. "Q:" as question prefix, "A." vs. "(A)" vs. "<mc>A</mc>" as answer label formatting, whether to include a general instruction (common for MMLU, sometimes done for OpenBookQA), and so on. Even within a single paper (e.g., Brown et al., 2020), formatting can vary across tasks. The paper also identifies the tokenization of answer labels as a subtle but consequential issue: without a leading space before answer labels (e.g., "\n A. <choice>" rather than "\nA. <choice>"), the token for the answer label in the prompt may differ from the token in the final answer position, causing the model to receive a token it has never seen in that context.

How This Paper Positions Itself

OLMES positions itself not as a new leaderboard or an alternative to HELM or the Eleuther LM Harness, but as a documented standard with justified choices that can be implemented within existing evaluation frameworks. The key distinction is that OLMES does not merely specify what the choices are—it documents why each choice was made, with empirical evidence and principled reasoning.

The standard is specifically designed to be:

• Reproducible: Every detail is specified, from dataset processing to prompt formatting to probability normalization to output interpretation. There are no ambiguities—"evaluated using OLMES" is a fully-specified claim.
• Practical: Choices account for computational cost (sampling 1000 instances when datasets exceed 1500, using 5-shot rather than 25-shot to limit prompt length, avoiding unnecessary permutation tests for option ordering).
• Documented: Each decision includes its justification, drawing on existing literature and new experiments that resolve open questions (Section 3 provides these justifications across every choice axis).
• Open: All prompts, code, and examples are released, enabling direct adoption by evaluation frameworks like the Eleuther LM Evaluation Harness and HELM.

The paper explicitly emphasizes that OLMES targets base model evaluation during the full development cycle—"from small to large models as well as early to late training stages." This is a deliberate scope choice. Multiple-choice tasks are emphasized because they "present a compelling way of evaluating models and humans alike" and "tend to provide less useful signals" from other task types (generative tasks, math reasoning, coding) for early-stage and smaller base models.

The CF/MCF resolution—evaluating with both formulations and using the best result—is the most consequential methodological contribution. This approach acknowledges that neither formulation is universally superior; rather, CF is necessary to extract meaningful signal from weak models (where MCF scores are at random), while MCF is necessary to accurately assess strong models (where CF scores plateau below true capability). Figure 1 provides the empirical motivation: during the training of OLMo-7B-0424, CF provides better signal early in training (where MCF is random), but around 400 billion training tokens, the model acquires the ability to use the MCF format, and MCF then provides a stronger signal than CF. OLMES's dual-evaluation approach captures this transition automatically, using whichever formulation gives the better result for each model-task combination.

The paper's relationship to prior work is constructive rather than competitive: HELM and the Eleuther LM Harness provided the infrastructure and the awareness of the problem; OLMES provides the documented, justified standard that gives researchers a reason to converge on a common evaluation protocol. The authors frame this as "the first work of its kind to unify practices for evaluating base models throughout the full development cycle" (Section 1).

3. Technical Approach

3.1 Reader Orientation

OLMES is not a new algorithm or model—it is a decision protocol: a fully specified set of choices, backed by empirical evidence and principled reasoning, that takes a dataset, a model, and a compute budget as inputs and produces a reproducible accuracy score as output. The problem it solves is that going from "evaluate model M on dataset D" to a single number like "54.2% on ARC-Challenge" requires dozens of decisions (prompt format, number of shots, which shots, normalization, formulation), and different researchers make different decisions, yielding different numbers for the same model-dataset pair. OLMES resolves this by making every decision once, justifying it with experiments across 15 diverse models, and publishing the complete specification so that "evaluated using OLMES" is a fully unambiguous claim.

3.2 Big-Picture Architecture

OLMES can be understood as a pipeline with five major stages:

1. Dataset Standardization — takes a raw benchmark dataset (e.g., ARC-Challenge from its original release) and produces a standardized set of instances with consistent formatting, a fixed evaluation split, and (if the dataset is large) a fixed random subsample of 1000 instances. This stage eliminates variation in which data points are evaluated and how they are structured.

2. Prompt Construction — takes the standardized instances and wraps them in a templated prompt format that includes a consistent question prefix, answer choice labels with leading spaces, and (crucially) a fixed set of 5 hand-curated in-context examples drawn from the training set. This stage eliminates variation in prompt wording, label tokenization, and few-shot example selection.

3. Model Inference (CF path) — presents each instance to the model using the completion/cloze formulation: each answer choice is individually substituted into the prompt, the model's log-probability of the answer string is extracted, and a normalization scheme (chosen per task from none, character, token, or pmi) is applied to produce a score per answer choice. The highest-scoring choice is the CF prediction.

4. Model Inference (MCF path) — presents each instance to the model using the multiple-choice formulation: all answer choices are listed with labels, and the model's log-probability of each answer label token is extracted. The highest-probability label is the MCF prediction.

5. Score Aggregation — the CF and MCF accuracies are computed separately across all instances, and the higher of the two is reported as the final OLMES score for that model-task combination. For MMLU specifically, the macro-average over 57 subjects is used rather than the micro-average over all instances.

Information flows sequentially: dataset → standardized instances → (CF prompt, MCF prompt) → (CF scores, MCF scores) → best accuracy. There is no feedback loop or model training—OLMES is purely an inference-time evaluation protocol.

3.3 Roadmap for the Deep Dive

• First, dataset standardization: how splits are chosen, when and how subsampling is performed, and why capping at 1000 instances matters for practical adoption.
• Second, prompt construction: the exact formatting templates, the tokenization-motivated rule for answer label spacing, the rationale for consistent question prefixes, and the curation procedure for the 5-shot examples.
• Third, the CF formulation and normalization: the four normalization schemes, their mathematical definitions, the empirical process by which OLMES selects one normalization per task, and the edge cases (BoolQ, WinoGrande) that require special handling.
• Fourth, the MCF formulation and the CF-vs-MCF resolution: why both formulations are evaluated, the empirical evidence that CF is necessary for weak models while MCF is necessary for strong ones, and the decision rule (best-of-two) that produces the final score.
• Fifth, other implementation details: the choices around token limits, model precision, MMLU macro-averaging, BOS token insertion, and the separation of in-context examples.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical standardization paper whose core idea is that the many degrees of freedom in LLM evaluation can be collapsed to a single, justified set of choices via systematic experimentation across a diverse model set, producing a protocol that is simultaneously reproducible (every detail is specified), practical (compute budgets are respected), and principled (each choice has documented reasoning).

Dataset Standardization: Splits, Subsampling, and Instance Counts

The first layer of variation OLMES eliminates is which data points are evaluated. Different references have used different splits (test vs. validation) and different numbers of instances for the same benchmark, making direct comparison impossible even before prompt formatting enters the picture.

Split selection rule. OLMES standardizes a simple rule stated in Section 3.1: use the test split of a dataset if the labels are publicly available; otherwise, use the validation split. This follows existing practice from HELM and the Hugging Face Open LLM Leaderboard. The reasoning is practical: test splits are preferred because they are the intended evaluation target, but for datasets where test labels are hidden (to prevent overfitting), the validation split is the only accessible option. Table 2 lists the specific split used for each of the 10 OLMES tasks—for example, ARC-Challenge uses "Test," while BoolQ, CommonsenseQA, HellaSwag, PIQA, Social IQa, and WinoGrande use "Val."

Subsampling rule. If a dataset has more than 1500 instances, OLMES samples exactly 1000 instances to evaluate. This is explicitly modeled on HELM, which also caps evaluation instances at 1000. The reasoning (stated in Section 3.1) is that "the potential extra statistical signal from more instances would generally be dominated by other sources of score variations, like prompt formatting, so this is a practical consideration to avoid unnecessary computation resources." In other words: when prompt formatting can swing scores by 10+ percentage points, the marginal benefit of evaluating on 5000 vs. 1000 instances is negligible, while the computational cost difference is substantial.

The sampling uses a fixed random seed: Random(1234).sample(all_instances, 1000) in Python. By publishing the seed, OLMES makes the subsample deterministic and therefore reproducible—anyone running OLMES will evaluate on exactly the same subset of instances.

Table 2 specifies which tasks are subsampled and the total dataset size:

• Not subsampled (fewer than 1500 instances): ARC-Challenge (1172 total, all used), ARC-Easy (2376 total but only 1000 used), BoolQ (3270 total, 1000 used), OpenBookQA (500 total, all used), WinoGrande (1267 total, all used).
• Subsampled to 1000: CommonsenseQA (1221 total), HellaSwag (10042 total), PIQA (1838 total), Social IQa (1954 total).
• MMLU is a special case with 14,042 total instances across 57 subjects; all are used because evaluating across all subjects is necessary for the macro-average computation (see Section 3.5), and the per-subject instance counts are modest.

Instance count rationale. The 1000-instance cap is not arbitrary. Statistical power for comparing two models' accuracy on a binary-outcome task scales roughly with $\sqrt{n}$, so increasing from 1000 to 5000 instances shrinks the standard error by a factor of $\sqrt{5} \approx 2.2$. However, if the measurement noise from unstandardized prompt formatting is on the order of 5–10 percentage points, the statistical noise from 1000 instances (standard error $\approx 1.6\%$ for accuracy near 50%) is already dominated by systematic variation. The cap is therefore a well-motivated efficiency tradeoff—it eliminates the primary source of irreproducibility (formatting variation) while accepting a small amount of statistical noise that is dwarfed by the gains from standardization.

Prompt Construction: Templates, Tokenization, and Few-Shot Curation

Once the dataset instances are standardized, OLMES wraps each one in a prompt template. This is where the majority of cross-reference variation originates, and OLMES makes explicit, justified choices for every element.

Instance formatting templates. Each task has two prompt templates—one for CF and one for MCF—which differ in how answer choices are presented. The general pattern for MCF is:

Question: <question text>
A. <choice A>
B. <choice B>
...
Answer:

The model is then scored on its probability of generating the correct answer label token (e.g., "A", "B", "C", "D") in the position immediately following "Answer:".

The general pattern for CF is:

Question: <question text>
Answer: <answer choice text>

Each answer choice is separately substituted into the <answer choice text> position, the model's log-probability of that choice text is extracted, and the choices are ranked.

Consistent prefix/suffix rules with explicit exceptions. OLMES standardizes that most tasks use "Question: <question>" as the prefix and "Answer:" as the suffix, because this "clarifies the question-answering task in a natural way, without relying on verbose instruction understanding" (Section 3.1). The paper identifies three justified exceptions:

1. PIQA uses "Goal: <goal>" instead of "Question:" because the dataset's original semantics frame instances as goals rather than questions (e.g., "how do you stab something?"). Consistency with the dataset's intended framing takes priority over mechanical uniformity.

2. HellaSwag (MCF) skips the question prefix entirely and instead uses "Choose the best continuation:" before presenting the options, because HellaSwag instances are not questions—they are partial sentences where the model must select the most natural completion. Forcing a "Question:" prefix would be semantically inappropriate. For the CF version, both prefixes and suffixes are removed so the task is "closer to pure language modeling"—the model simply sees the context text followed by each candidate continuation, and the highest-probability continuation is selected.

3. WinoGrande (MCF) uses "Fill in the blank:" as the prefix because the task involves selecting which of two entities fills a blank in a sentence. For the CF version, the prompt structure is different from all other tasks: rather than varying the answer choice text in a fixed prompt, WinoGrande varies the prompt itself (e.g., "John moved the couch... The garage" vs. "John moved the couch... The backyard") while keeping the completion identical ("is small."). Since the continuation length is identical across choices, normalization becomes irrelevant—hence OLMES specifies none normalization for WinoGrande.

Answer label formatting and the leading-space rule. This is one of the most subtle but consequential choices in OLMES. For MCF answer choices, OLMES uses the format "\n A. <choice>"—note the space between the newline and the letter "A". The alternative "\nA. <choice>" (no leading space) would create a tokenization mismatch: most tokenizers treat a letter at the start of a line as a different token from the same letter following a space.

The paper provides a concrete example in Appendix C.3 using the Llama tokenizer. With the string "What is 3+4?\n A. 7\nA. 7\nAnswer: A", the Llama tokenizer produces tokens that include _A (the underscored version representing a space-prefixed A) for the answer choice label in the prompt and for the final answer label. Without the leading space, the tokenizer would produce a bare A token for the choice label in the prompt but _A for the answer label position, so the model would never have seen the token it needs to predict.

The OLMES rule—always use a leading space before answer labels—ensures that the token for the answer label in the prompt (e.g., the "A" in "A. choice text") is identical to the token the model needs to generate after "Answer:". This is not a matter of preference; it is a correctness condition for the MCF formulation to function as intended. Without it, model accuracy would be artificially depressed because the model would be asked to predict a token that never appeared in its context.

Few-shot example standardization. OLMES standardizes on exactly 5-shot prompts for every task, using manually curated (not randomly sampled) examples from the training set. This choice is justified by two empirical observations:

• Going beyond 5 shots "generally does not provide meaningful differences in scores" (Section 3.2), citing Brown et al. (2020) and Barton (2024). Additional shots increase prompt length and computational cost without improving measurement quality.
• Random sampling of few-shot examples can produce unbalanced label distributions (e.g., "4 A's and 1 B among the 5 answers"), which would unfairly bias the model toward or against certain answer labels. Manual curation ensures that the examples "cover the label space in a balanced way" and are "of good quality."

The curation procedure is described in Appendix G and follows a systematic process:

1. Start from the beginning of the training set, examining batches of 10 instances at a time.
2. Skip instances that are ambiguous, that "hint at discrimination or otherwise deemed inappropriate," or that would cause consecutive examples to have the same gold label (e.g., 4 consecutive instances with label 'C').
3. If instances are grouped by topic, select diverse instances across topics rather than clustering on one topic.
4. If fewer than 7 suitable instances are found in the first batch, examine the next batch of 10.
5. Finally, reorder the selected instances so the first 5 cover the space of answer labels in a balanced way.

This procedure is performed once per task, and the resulting 5-shot prompts are published as part of the OLMES standard. Any evaluation using OLMES uses exactly these curated examples, eliminating the variation that would arise from random sampling or different curators making different judgment calls.

No additional instructions (with one exception). OLMES deliberately avoids adding verbose instructions beyond the few-shot examples and the basic question/answer formatting. The one exception is MMLU, where the original instruction line from Hendrycks et al. (2021) is retained: "The following are multiple choice questions (with answers) about <subject>." This is included because it has become standard practice for MMLU evaluation and provides useful subject context. For all other tasks, no extra instructions are added, based on the finding that "subject information from instructions makes little changes to model ranking" (Alzahrani et al., 2024) and to "reduce additional sources of variation in the prompt."

In-context example separation. OLMES uses "the standard approach of two newlines to separate each in-context example." This is specified in Section 3.5 as one of the "other implementation details" and serves to visually and structurally separate the examples from each other and from the target instance.

The CF Formulation and Probability Normalization

The completion/cloze formulation (CF) is one of the two parallel evaluation paths in OLMES. It is conceptually simpler than MCF—the model is not asked to "choose" among options, but simply to produce the most probable continuation of a prompt—but it introduces a complication: the raw probability of an answer string is confounded by its length and the model's prior probability of the words it contains.

The raw probability problem and why normalization is needed. When using CF, the model returns $P(a_i \mid q)$, the probability of answer choice $a_i$ given question prompt $q$. Ranking choices solely by this raw probability systematically favors shorter answers with fewer tokens. Intuitively, each additional token in an answer multiplies the (sub-unity) probability by another factor less than 1, so longer answers accumulate lower probabilities independent of their correctness. The normalization step exists to remove this length confound.

OLMES considers four normalization schemes, each transforming the raw probability into a score that can be compared across answer choices of different lengths:

1. none normalization:

$\text{score}_{\text{none}}(a_i \mid q) = \ln P(a_i \mid q)$

where $P(a_i \mid q)$ is the model's probability of the answer choice string given the question prompt.

What it computes: The raw log-probability of the answer string conditioned on the question. No adjustment is made for answer length or prior probability.

Why this form: This is the simplest possible scoring rule—it trusts the model's conditional probability directly. It is appropriate only when all answer choices have similar token counts and similar prior probabilities, or when the model's conditional probability is well-calibrated enough that length effects are negligible relative to correctness signal.

2. token normalization:

$\text{score}_{\text{token}}(a_i \mid q) = \frac{\ln P(a_i \mid q)}{\text{num\_tokens}(a_i)}$

where $\text{num\_tokens}(a_i)$ is the number of tokens in the answer string under the model's tokenizer.

What it computes: The average log-probability per token of the answer string. This is analogous to how language model training loss is computed—the total log-probability is divided by the number of tokens to get a per-token measure.

Why this form: Brown et al. (2020) used this normalization when evaluating GPT-3, reasoning that it mirrors the training objective and therefore should produce well-calibrated scores. The theoretical justification is that if each token's probability is roughly independent and identically distributed, dividing by token count removes the length bias. However, the paper notes a counterargument from the Eleuther LM Evaluation Harness: "normalizing per token is problematic since it depends on the tokenizer." Since different tokenizers segment the same string into different numbers of tokens, scores are not comparable across models with different tokenizers—though within a single model, ranking answer choices is still valid.

3. character normalization:

$\text{score}_{\text{char}}(a_i \mid q) = \frac{\ln P(a_i \mid q)}{\text{num\_characters}(a_i)}$

where $\text{num\_characters}(a_i)$ is the number of characters in the answer string (including the leading space).

What it computes: The average log-probability per character of the answer string. This removes the tokenizer-dependence of token normalization—character counts are deterministic and identical across all tokenizers.

Why this form: This normalization is used by the Llama evaluations (Touvron et al., 2023a) and is the default acc_norm metric in the Eleuther LM Evaluation Harness. It eliminates tokenizer-specific artifacts while still addressing length bias. The paper notes that OLMES specifies including the leading space in the character count, which is a detail that could vary across implementations if not explicitly standardized.

4. pmi normalization (pointwise mutual information):

$\text{score}_{\text{pmi}}(a_i \mid q) = \ln \frac{P(a_i \mid q)}{P(a_i \mid u)}$

where $u$ is an unconditional prompt, specifically u = "Answer:" (the suffix without the question), so $P(a_i \mid u)$ is the model's prior probability of the answer string given only the answer prefix, with no question context.

What it computes: The log-ratio of the conditional probability to the unconditional prior probability. This measures how much more (or less) likely the answer choice becomes when the question is added to the context, compared to its baseline likelihood given just the answer prompt format.

Why this form: This normalization addresses a different confound than length—it corrects for the model's prior preference for certain answer strings independent of the question. If "Whirlpool bath" has a much lower unconditional probability than "Bathtub" simply because it appears less frequently in the training data, the pmi normalization subtracts this baseline difference so that the model is rewarded only for knowledge that comes from the question. Brown et al. (2020) used this selectively for ARC and OpenBookQA, noting that those datasets contain answer choices with unusual vocabulary that models assign low prior probability. Holtzman et al. (2021) studied this approach more extensively as a way to address "surface form competition"—the phenomenon where the most probable answer under the model's language modeling distribution is not necessarily the correct one.

The paper notes (Section 3.3) that computing $P(a_i \mid u)$ "incurs some computation overhead" because it requires an additional forward pass through the model for each answer choice without the question context. This is the reason OLMES avoids pmi for tasks where there is "no strong empirical or theoretical reason" to prefer it.

The task-by-task normalization selection process. Rather than choosing one normalization for all tasks, OLMES makes a per-task recommendation based on empirical comparison across 15 models. Table 3 summarizes the results, and Tables 10–12 in the appendix provide the full per-model, per-task, per-normalization accuracy scores.

For each task, the paper computes which normalization scheme gives the highest accuracy for each of the 15 models, then reports the "win percentage"—the fraction of models for which each normalization is best. The OLMES recommendation is the normalization with the highest win percentage, except where principled reasoning overrides empirical performance (as with BoolQ, discussed below).

The per-task recommendations and their justifications:

• ARC-Challenge, CommonsenseQA, OpenBookQA → pmi: These three tasks have the highest pmi win percentages (66.7%, 53.3%, and 100.0% respectively). The paper explains that the answer choices in these datasets "tend to contain unexpected words or phrases that are less likely for models to generate," giving the example of "Whirlpool bath" compared to "Bathtub." The pmi normalization adjusts for the a priori likelihood of these unusual answer strings, allowing the model's question-conditional knowledge to be measured without being swamped by prior probability differences. This is consistent with Brown et al. (2020), who selectively used this normalization for ARC and OpenBookQA, and with Touvron et al. (2023a,b), who used it for OpenBookQA.

• ARC-Easy, HellaSwag, PIQA, Social IQa, MMLU → character: These five tasks have character win percentages of 86.7%, 100.0%, 46.7% (tied with token), 86.7%, and 46.7% respectively. Even where character is not the absolute winner (PIQA and MMLU are tied or closely behind pmi), OLMES selects it because it "is already implemented (as acc_norm) in the Eleuther LM Evaluation Harness, where it is generally available for multiple-choice tasks," and because it avoids the computational overhead of pmi. The paper notes that for MMLU, pmi has a slightly higher win percentage (53.3% vs. 46.7% for character), but the practical benefit of using an already-implemented, computationally cheaper method sways the recommendation.

• BoolQ → none: BoolQ has only two answer choices: "yes" and "no." Both are single tokens in all standard tokenizers, so there is no length difference to normalize. The raw log-probability is sufficient. The paper explicitly notes that character normalization has a slightly higher win percentage on BoolQ (46.7% vs. 46.7% for none, with 6.7% for pmi), but argues this is "an accidental side effect of 'yes' having one more character than 'no'"—the normalization changes the relative probability of the two answers based on a length difference that has nothing to do with the model's knowledge. The authors argue that "models should be capable of producing such common words (also indicated in the 5-shot examples) without any such corrections," making none the principled choice even though it is not empirically dominant.

• WinoGrande → none: WinoGrande is structurally unique among the OLMES tasks. In the CF formulation, the completion (the text whose probability is scored) is identical across answer choices—e.g., "is small."—while the prompt varies. Since the completion length is identical for all choices, any normalization scheme would divide by the same constant and leave the ranking unchanged. Therefore, normalization is irrelevant, and none is specified for simplicity.

The near-oracle property of OLMES recommendations. The "diff oracle" column in Table 3 computes, for each model and task, the difference between the accuracy under the OLMES-recommended normalization and the accuracy under the best possible normalization for that specific model-task combination. These differences are generally small (average 0.2% for ARC-Challenge, 0.1% for ARC-Easy, 0.4% for MMLU), suggesting that the OLMES recommendations achieve performance very close to what could be obtained by per-model optimization while avoiding the circularity of choosing the evaluation setup based on the evaluation target.

The MCF Formulation and the CF-vs-MCF Resolution

The multiple-choice formulation (MCF) is the second parallel evaluation path in OLMES. Unlike CF, which scores each answer choice independently, MCF presents the model with a single prompt containing all answer choices and asks it to predict which label is correct.

How MCF scoring works. For a prompt like:

Question: What is 3+4?
A. 5
B. 6
C. 7
D. 8
Answer:

The model's probability of generating each answer label token (A, B, C, D) in the position immediately following "Answer:" is extracted. The predicted answer is the label with the highest probability. Crucially, MCF scores only the single token representing the answer label, not the entire answer choice text. This means length normalization is not a concern for MCF—the model is simply choosing among single-token labels, and its probability distribution over those labels directly determines the answer.

Why MCF is more natural but requires "understanding." The paper argues that MCF is "a more realistic formulation for models that can 'understand' this format, yielding higher and more representative scores" (Section 3.4). This is because MCF mirrors how multiple-choice questions are posed to humans: the options are presented together, and the respondent selects among them. CF, by contrast, is an artifact of language model evaluation—no human takes a multiple-choice test by considering each option in isolation and choosing the one that "sounds most natural" as a sentence completion.

However, MCF requires the model to understand the task structure: it must recognize that the A/B/C/D labels correspond to answer choices and that it should "select" one. Weaker models (smaller models, models early in training, or models without instruction tuning) lack this meta-cognitive ability. They may assign probability to answer labels based on superficial patterns (e.g., always preferring "B") rather than based on the content of the question and choices, resulting in near-random performance even when the model possesses the underlying knowledge to answer correctly.

This is the central tension OLMES resolves: CF extracts knowledge from models that cannot handle the MCF format, but MCF gives more accurate assessments for models that can.

Empirical evidence for the CF-early, MCF-late pattern. The paper provides two complementary pieces of evidence:

Figure 1 (MMLU during training of OLMo-7B-0424): This figure plots accuracy on the MMLU validation set as a function of training data processed (in billions of tokens), with separate curves for CF and MCF. Early in training (0–200B tokens), CF accuracy rises steadily from random to about 37%, while MCF accuracy hovers near the random baseline of 25%—the model clearly possesses increasing task knowledge (as measured by CF) but cannot express it in the MCF format. Around 400B tokens, the model "starts gaining the ability on the MCF format, becoming a stronger signal than CF." After this transition, MCF accuracy rises while CF accuracy levels off, and MCF eventually provides a substantially higher absolute score.

This is a single-model, single-task example, but it illustrates the general phenomenon: the ability to "understand" the MCF format is a capability that emerges during training, and before its emergence, CF is the only useful evaluation signal.

Figure 2 (MCF vs. CF across 15 models on all 10 tasks): This figure plots the 15 models along the x-axis ordered by their overall performance across all tasks, with separate bars for MCF and CF accuracy on each task. The pattern is consistent:

• For the weakest models (leftmost on x-axis), MCF scores are near random (e.g., ~25% for 4-choice tasks), while CF scores are substantially above random. On ARC-Challenge, the weakest 8 models have MCF scores near 25% but CF scores ranging from 31.4% to 49.7%.
• For the strongest models (rightmost), MCF scores dramatically exceed CF scores. On ARC-Challenge, Llama3-70B achieves 93.7% MCF vs. 69.0% CF—a 24.7 percentage point gap.
• On some tasks (HellaSwag, WinoGrande, BoolQ), MCF and CF scores are close even for strong models, and CF sometimes remains slightly higher. The paper speculates that this is because these tasks in the CF format are "exactly like the language modeling task of finding the most natural continuation of a running piece of text," so CF is inherently well-suited to them.

The OLMES resolution: evaluate both, use the best. OLMES standardizes a simple decision rule (Section 3.4): "evaluate each model using both the MCF and CF formulations, and the best performing one is used." This is implemented as computing accuracy under both formulations on the same set of instances and reporting the maximum.

This rule has several desirable properties:

• It is automatic. No human judgment is needed to decide whether a model is "strong enough" for MCF. If MCF is better for a given model-task pair, it will naturally produce the higher score and be selected. If CF is better, it will be selected instead.
• It provides meaningful comparisons across the full capability spectrum. A weak model evaluated with CF can be compared to a strong model evaluated with MCF, because both are measured in the formulation that best captures their capabilities. Without this rule, weak models would appear artificially worse (if only MCF were used) or strong models would appear artificially worse (if only CF were used).
• It is transparent. The final OLMES score is marked with a † symbol in Table 4 to indicate when MCF was used, so readers can distinguish which formulation produced the reported number for each model-task pair.

The decision rule operates at the per-model, per-task level, not globally. A model might use MCF for ARC-Challenge (where it has acquired the format understanding) but CF for HellaSwag (where the language modeling nature of the task makes CF inherently superior even for strong models). Table 4 shows this pattern: Llama3-8B uses MCF for ARC-Challenge (79.3%), ARC-Easy (92.4%), MMLU (66.6%), OpenBookQA (77.2%), and Social IQa (70.2%), but uses CF for BoolQ (87.5%), CommonsenseQA (73.9% MCF? no—Table 7 shows 73.9% MCF vs. 69.9% CF, so MCF), HellaSwag (81.8% CF vs. 63.8% MCF, so CF), PIQA (81.6% CF vs. 77.3% MCF, so CF), and WinoGrande (76.2% CF vs. 61.6% MCF, so CF).

The error rate perspective. The paper emphasizes that for strong models, the CF-vs-MCF difference is best understood in terms of error rates rather than raw accuracy. Llama3-70B's ARC-Challenge scores are 93.7% MCF (6.3% error) vs. 69.0% CF (31% error). The error rate is "nearly 5x"—the MCF score provides a much more precise measure of the model's remaining deficiencies. As models approach human-level performance, this precision becomes increasingly important: a 1% improvement from 93% to 94% (a 1% error reduction) is more meaningful than a 1% improvement from 68% to 69% (a 1/32 error reduction), yet the latter might be reported if CF were the only formulation used.

MMLU-Specific Choices: Macro-Averaging and Subject Instructions

MMLU is the largest and most complex dataset in OLMES, consisting of 14,042 questions across 57 subjects ranging from abstract algebra to world religions. It requires two special choices not needed for other tasks.

Macro-average over subjects, not micro-average over instances. OLMES specifies computing accuracy separately for each of the 57 subjects, then averaging these 57 accuracies with equal weight (macro-average). The alternative—computing a single accuracy over all 14,042 instances (micro-average)—would give more weight to subjects with more questions (e.g., professional law with ~274 questions vs. high school statistics with ~216 questions).

The paper justifies the macro-average choice by stating it "better represents the diversity of fields in the dataset" and notes that it follows AI@Meta (2024). Table 8 in Appendix B provides evidence that the choice is not highly consequential: across 15 models, the difference between macro and micro averages is small (usually less than 1 percentage point). For example, Llama3-8B achieves 66.6% macro vs. 65.4% micro under MCF, and 51.1% macro vs. 50.8% micro under CF. The differences are small because MMLU's subject-level instance counts are not wildly imbalanced—the largest subjects have only modestly more questions than the smallest.

Subject instructions retained. Unlike other OLMES tasks, MMLU prompts include the original subject-specific instruction line from Hendrycks et al. (2021): "The following are multiple choice questions (with answers) about <subject>." For example, the abstract algebra prompt begins with this line specifying the subject. This instruction is retained because it has become standard practice in MMLU evaluation and provides useful context about the domain of the questions.

Other Implementation Details: Token Limits, Model Precision, BOS Tokens, and Output Separation

Section 3.5 enumerates several additional standardizations that, while individually minor, collectively ensure reproducibility across different hardware setups and evaluation frameworks.

Token limit: 2048 tokens. All inputs (including few-shot examples, the target question, and the completion) are restricted to 2048 tokens. This is the standard context window for many models in the evaluation set and ensures consistency across models with different native context lengths. The paper notes that "for current tasks this is only exhausted for a few MMLU instances"—most OLMES prompts are well under this limit even with 5-shot examples.

Model precision: default. OLMES specifies using the default model precision when evaluating and explicitly says to "avoid options like load_in_8bit unless it produces identical results." The reasoning is that reduced-precision inference can introduce small floating-point differences that, while usually negligible, could in principle flip the ranking of two answer choices if their probabilities are extremely close.

BOS token insertion. When a model requires it, the appropriate beginning-of-sequence token must be added at the start of the prompt. The paper gives the example of Gemma, which needs its specific BOS token prepended. This is model-specific behavior that could easily be overlooked in ad-hoc evaluations but is specified as part of the OLMES standard.

In-context example separation. Two newlines separate each in-context example from the next, providing clear visual and structural delineation. This is described as "the standard approach" in the literature.

No extra instructions beyond MMLU's subject line. The paper explicitly avoids adding general instructions (e.g., "Please answer the following multiple-choice question") to any task other than MMLU's existing subject instruction. This choice is motivated by previous work finding that "subject information from instructions makes little changes to model ranking" (Alzahrani et al., 2024) and by the desire to "reduce additional sources of variation in the prompt."

Floating-point non-determinism acknowledged but not fully resolved. The paper notes a remaining challenge: "computational details, like batch size and type/state of GPU, can affect floating point operations such that answer choice decisions can flip if they are very close." This is a fundamental limitation of floating-point arithmetic on parallel hardware—the order of operations can affect the least significant bits of the result, and if two answer choices have nearly identical probabilities, the ranking can depend on these bits. The paper suggests that future work could consider "ties" when answers are sufficiently close in confidence, but does not implement tie-breaking in the current standard. This means that OLMES scores are reproducible at the level of approximately ±a fraction of a percentage point, which is sufficient for meaningful model comparison given the larger variation that prompt formatting would otherwise introduce.

Summary of Design Choices and Their Justifications

The OLMES standard can be understood as a sequence of design decisions, each backed by empirical evidence or principled reasoning:

• 1000-instance subsampling over using full datasets: practical efficiency; marginal statistical gains from more instances are dwarfed by formatting-induced variation.
• Curated 5-shot over random N-shot: ensures balanced label distribution and example quality; beyond 5 shots yields diminishing returns.
• Leading space in answer labels (e.g., "\n A." over "\nA."): ensures tokenization consistency between prompt and answer positions—a correctness condition for MCF.
• Per-task CF normalization selection over one-size-fits-all: different tasks have different answer choice characteristics (length, vocabulary unusualness) that favor different normalization schemes.
• Dual CF/MCF evaluation with best-of-two selection over single-formulation: CF is necessary for weak models (where MCF is random); MCF is necessary for strong models (where CF plateaus). The automatic selection rule handles the transition without human intervention.
• Macro-averaging for MMLU over micro-averaging: better represents the diversity of subjects; in practice produces only minor differences from micro-averaging.
• No extra instructions (beyond MMLU's subject line) over verbose prompting: reduces variation; prior work shows minimal impact on model ranking.

4. Key Insights and Innovations

Innovation 1: Evaluation Is a Measurement Protocol Design Problem, Not a Benchmark Collection Problem

The dominant framing in the LLM evaluation community has been that building better benchmarks—new datasets, harder questions, more diverse domains—is the primary path to better model assessment. Efforts like HELM (Liang et al., 2023) advanced this by aggregating many benchmarks under a unified infrastructure, while the Hugging Face Open LLM Leaderboard (Beeching et al., 2023) consolidated a fixed set of popular tasks. Both implicitly treated the benchmark itself as the evaluation, with the implementation details as secondary implementation concerns.

OLMES reframes the problem entirely. The paper's central conceptual move is to treat evaluation as a measurement protocol design problem rather than a benchmark selection problem. The insight is that a statement like "model X scores Y on ARC-Challenge" is not a measurement at all—it is an underspecified claim that could correspond to dozens of different actual measurements depending on undocumented choices about formatting, normalization, few-shot examples, and task formulation. The paper demonstrates this reframing empirically through Table 1, which is not merely an illustration of variance but a diagnostic tool: by showing that the same model produces scores ranging from 43.2% to 53.7% on the same dataset across different references, the paper establishes that the dataset is not the measurement instrument—the protocol is.

This reframing has implications beyond the specific choices OLMES makes. It implies that the field's practice of reporting "SOTA on benchmark X" without specifying the full evaluation protocol is scientifically equivalent to reporting a physical measurement without specifying the instrument's calibration. It also implies that efforts to build "harder" benchmarks (e.g., MMLU-Pro, GPQA) are incomplete solutions to the evaluation crisis: a harder benchmark measured with an unstandardized protocol still produces uninterpretable numbers, just lower ones.

What distinguishes this from prior standardization efforts is the documented justification requirement. HELM and the Eleuther LM Harness already provided consistent evaluation—the same setup applied across models—but they presented their choices as defaults without explicit reasoning. OLMES argues that defaults without justification cannot function as a standard because researchers have no basis for trusting them over alternatives and thus no motivation to adopt them. The paper's insight is that a standard is not a default; a standard is a default plus a documented argument for why the default is correct, backed by empirical evidence that the default is not arbitrary but optimal (or near-optimal) given principled criteria.

This is a fundamental shift in how the community should think about evaluation, not an incremental improvement over existing tooling. The paper's own framing captures this: "We believe this work is the first of its kind to unify practices for evaluating base models throughout the full development cycle, from small to large models as well as early to late training stages" (Section 1).

Innovation 2: The CF-vs-MCF Transition as a Capability Diagnostic, Not a Formatting Preference

Prior to OLMES, the choice between the cloz formulation (CF) and the multiple-choice formulation (MCF) was treated as a matter of convention, model era, or researcher preference. GPT-3 (Brown et al., 2020) used CF because it was the only option that worked; later models increasingly used MCF because it produced higher scores (Robinson et al., 2023; OpenAI, 2024; AI@Meta, 2024). The transition was gradual and inconsistent—some papers used CF for some tasks and MCF for others, sometimes without documenting which, as evidenced by the "?" entries in Table 1. The implicit assumption was that these are two ways to measure the same thing, and that as models improve, MCF becomes the "better" measurement.

OLMES's key insight is that CF and MCF do not measure the same underlying capability. Instead, CF measures a model's knowledge of the content of the task (does the model assign higher probability to correct answer strings than incorrect ones?), while MCF additionally measures a model's meta-cognitive ability to understand the multiple-choice format as a task structure. The gap between CF and MCF performance is not measurement noise—it is a diagnostic signal about whether a model has acquired the format-understanding capability.

Figure 1 provides the clearest empirical evidence for this interpretation. During the training of OLMo-7B-0424, CF accuracy on MMLU rises steadily from random to ~37% over the first 200B training tokens, while MCF accuracy remains at the random baseline of ~25%. This is not because the model lacks knowledge—CF shows it is acquiring knowledge. It is because the model has not yet learned that the A/B/C/D labels in the prompt are answer choices to be selected among. The format-understanding capability emerges around 400B tokens, at which point MCF accuracy begins to rise and eventually surpasses CF. The two curves are measuring the acquisition of different capabilities, with MCF's capability being learned later in training.

This reframes the practical question of "which formulation should I use?" into a theoretical question of "what am I trying to measure?" If the goal is to track the emergence of content knowledge during training, CF is the appropriate instrument during early training; if the goal is to measure the model's full capability including its ability to follow the multiple-choice format, MCF is appropriate later. The OLMES solution—evaluate both, report the maximum—is a pragmatic resolution, but the deeper contribution is establishing that the CF-vs-MCF gap is itself a meaningful signal about model development, not an annoyance to be eliminated.

This insight also explains why prior work found contradictory results about whether models "understand" multiple-choice questions. A model early in training or a smaller architecture might possess the knowledge to answer a question but fail on MCF, leading to the conclusion that it "doesn't understand the format." A stronger model might succeed on MCF, leading to the conclusion that the format is fine. Both conclusions are correct for their respective models, but neither captures the underlying reality: format understanding is a capability that emerges along a training trajectory, and different models at different points on that trajectory will show different relationships between CF and MCF performance.

The practical significance is substantial: the CF-vs-MCF gap can serve as a cheap diagnostic during model development. If a training run shows CF accuracy rising while MCF remains flat at random, developers know that the model is acquiring content knowledge but has not yet crossed the format-understanding threshold. This is actionable information that can guide decisions about continued training versus architectural or data changes, without requiring separate probing tasks.

Innovation 3: Per-Task Probability Normalization as a Principled, Empirically-Grounded Choice Rather Than a Universal Default

Prior work on probability normalization for the CF formulation fell into two camps. The first camp provided universal defaults: Brown et al. (2020) used token-length normalization as the general approach with selective exceptions for specific datasets; the Llama evaluations (Touvron et al., 2023a) defaulted to character-length normalization across all tasks; the Eleuther LM Harness reported both acc (no normalization) and acc_norm (character normalization) and left the choice to the user. The second camp studied normalization comparatively but stopped short of making specific recommendations, treating the choice as an open question (Liang et al., 2023; Holtzman et al., 2021).

The implicit assumption across both camps was that normalization is a methodological detail that, while important, should ideally be resolved with a single universal scheme applied across all tasks. The differences between normalization methods were seen as artifacts to be minimized, not as informative signals about task structure.

OLMES's insight is that different normalization schemes correct for different confounds, and which confound matters depends on the nature of the task's answer choices. Length-based normalizations (token and character) correct for the fact that longer answers accumulate lower total probabilities independent of correctness. The pmi normalization corrects for a different confound entirely: the model's prior probability of the answer string independent of the question, which matters when answer choices contain unusual vocabulary that the model assigns low baseline probability. The none normalization is appropriate only when neither confound is present—when answer choices have identical length and similar prior probabilities.

This reframing transforms normalization from a nuisance parameter to be eliminated into a task-informed modeling choice. The empirical evidence supports this: on ARC-Challenge, CommonsenseQA, and OpenBookQA, the pmi normalization dominates (win percentages of 66.7%, 53.3%, and 100.0% across 15 models), consistent with the observation that these datasets contain unusual answer vocabulary. On HellaSwag, character normalization wins 100.0% of the time, consistent with the task being essentially a language-modeling continuation task where answer length is the primary confound. On BoolQ, where answers are single-token "yes"/"no" with no length difference and both are common words, the principled choice is none, even though character normalization has a slightly higher empirical win rate (an artifact of "yes" having one more character than "no").

The "diff oracle" analysis in Table 3 provides the crucial validation: the OLMES-recommended normalization is typically within 0.1–0.4% accuracy of the best possible per-model normalization. This demonstrates that the task-level recommendations are near-optimal without requiring per-model optimization, which would be circular (choosing the evaluation setup based on the evaluation target).

This innovation is significant because it resolves an open methodological question—which normalization should be used?—with a principled, empirically-validated framework rather than an arbitrary convention. It also provides a template for how future evaluation standards should handle similar methodological choices: identify what confound each option corrects for, determine which confounds are present in each task, and select the option that addresses the relevant confound for that task. This is a systematic approach to measurement protocol design that extends beyond normalization to other evaluation choices.

Innovation 4: Curated Few-Shot Examples as a Standardization Mechanism, Not a Performance Optimization

The use of few-shot examples in LLM evaluation was popularized by Brown et al. (2020) and has since become standard practice, but the selection of those examples has been treated as a low-stakes implementation detail. Most evaluations use randomly sampled examples from the training set, with the number of shots varying arbitrarily: 25-shot for ARC-Challenge on some leaderboards, 10-shot for HellaSwag on others, 5-shot for MMLU, 0-shot for many HELM evaluations. The implicit assumption is that random sampling is "fair" because it introduces no human bias, and that the number of shots is a hyperparameter to be tuned for performance.

OLMES challenges this assumption on two grounds. First, random sampling can be unfair in ways that systematically bias model performance: a random 5-shot sample might contain 4 examples with answer label "A" and 1 with "B," teaching the model a spurious prior that "A" is the default answer. Second, the variation introduced by random sampling—different evaluation runs using different randomly selected examples—is itself a source of irreproducibility, even when all other protocol details are identical.

The insight is that few-shot examples are part of the measurement instrument, not part of the model's input. In a properly standardized evaluation, the prompt—including which examples are used—should be as fixed as the questions themselves. Curated examples that "cover the label space in a balanced way" and are "of good quality" produce a measurement instrument with known, stable properties. Randomly sampled examples produce an instrument with unknown, variable properties that change each time the evaluation is run.

The appendix FAQ (Table 5) provides empirical evidence that the OLMES choice is robust: when the curated examples are varied (changing 3 out of 5 to different but similarly curated examples) and the prompt wording is perturbed slightly, the resulting accuracy scores differ by typically less than 1% from the standard OLMES setup. This demonstrates that the curation procedure, while fixed, is not brittle—similar choices produce similar results. The robustness comes from the principles of curation (label balance, example quality, topic diversity), not from the specific examples chosen.

This innovation is conceptually significant because it redefines what "standardization" means for few-shot evaluation. Rather than standardizing the procedure for selecting examples (e.g., "random sample with seed 42"), which still admits run-to-run variation, OLMES standardizes the examples themselves. The 5-shot prompts are published artifacts, like the test questions, and anyone running OLMES uses the identical prompts. This is a stronger form of standardization that eliminates an entire class of reproducibility failures.

The choice of 5 shots specifically is justified by the observation that "going beyond 5 shots generally does not provide meaningful differences in scores" (Brown et al., 2020; Barton, 2024), making it a practical sweet spot: enough examples to convey the task format, few enough to limit computational overhead and prompt length. This is an engineering tradeoff that balances measurement quality against practical constraints, consistent with OLMES's overall philosophy of being "practical" for adoption.

5. Experimental Analysis

Evaluation Methodology

• Dataset. OLMES standardizes evaluation across 10 multiple-choice question answering (MCQA) benchmarks: ARC-Challenge, ARC-Easy, BoolQ, CommonsenseQA, HellaSwag, MMLU, OpenBookQA, PIQA, Social IQa, and WinoGrande (Table 2). These cover science, commonsense reasoning, factual knowledge, and span a range of difficulty—MMLU alone encompasses 57 subjects. The selection mirrors tasks frequently used in the Hugging Face Open LLM Leaderboard, Llama papers, HELM, and the OLMo evaluation suite, ensuring relevance to existing community practice.

• Base model(s). The paper develops OLMES using 15 openly available pretrained base (not instruction-tuned) LLMs spanning 1B to 70B parameters: Pythia-1B, Pythia-6.7B, OLMo-1B, OLMo-7B, OLMo-7B-0424, TinyLlama-1.1B, StableLM2-1.6B, RPJ-INCITE-7B, MPT-7B, Falcon-7B, Llama2-7B, Llama2-13B, Mistral-7B-v0.1, Llama3-8B, and Llama3-70B. This range is deliberately chosen to "reflect our goal of providing an evaluation standard that suits a range of model capabilities, with the flexibility to apply the same methodology during model development as well as when comparing final powerful base models" (Section 2.3). The focus on base models is motivated by the practical need for evaluation signals during training and before instruction-tuning.

• Metrics. The primary metric is accuracy—the fraction of instances where the model's predicted answer matches the ground truth. For MMLU specifically, accuracy is computed as the macro-average over 57 subjects (each subject weighted equally) rather than the micro-average over all 14,042 instances, because this "better represents the diversity of fields in the dataset" (Section 3.5). For the CF formulation, the predicted answer is the choice with the highest normalized log-probability; for the MCF formulation, it is the answer label token with the highest probability following "Answer:". The final OLMES score for each model-task pair is the maximum of the CF and MCF accuracies.

• Baselines. The paper does not compare OLMES against a single fixed baseline in the traditional sense. Instead, the "baseline" is the status quo of unstandardized evaluation: the varying scores reported across different references for the same model on the same task, as documented in Tables 1 and 14. The paper's central empirical claim is that OLMES produces reproducible scores—scores that can be independently verified and meaningfully compared—whereas the status quo produces scores that vary arbitrarily across references. A secondary form of baseline comparison is the per-task analysis of CF normalization schemes, where the paper compares none, character, token, and pmi against each other (Tables 3 and 9–12) and against an "oracle" that chooses the optimal normalization per model.

• Generation budget / compute accounting. OLMES is an inference-only evaluation protocol—no model training is involved. The paper accounts for computational cost in its design choices rather than metering it as part of the evaluation. The key cost-saving measures are: (a) capping evaluation instances at 1000 for datasets with more than 1500 instances, explicitly justified by the observation that "the potential extra statistical signal from more instances would generally be dominated by other sources of score variations, like prompt formatting" (Section 3.1); (b) standardizing on 5-shot prompts rather than larger numbers of shots, since "going beyond 5 shots generally does not provide meaningful differences in scores" (Section 3.2); (c) avoiding pmi normalization for tasks where there is no strong empirical or theoretical reason to prefer it, because it requires an additional unconditional forward pass per answer choice. The paper reports using approximately 400 GPU hours (NVIDIA RTX A6000) for all inference across all models (Appendix F).

• Cross-validation / statistical protocol. OLMES does not perform cross-validation or statistical significance testing in the traditional machine learning sense—it is a measurement protocol, not a learning algorithm. The paper addresses reproducibility concerns by: (a) using a fixed random seed (Random(1234)) for subsampling, making the instance selection deterministic; (b) publishing the exact curated 5-shot examples, making prompt construction deterministic; (c) specifying every implementation detail (token limits, precision, BOS token insertion, label spacing), eliminating sources of hardware-dependent variation. The appendix FAQ (Appendix A, Table 5) includes a robustness check where three variants of OLMES (minor prompt wording changes, different curated few-shot examples, and both combined) are compared against the standard setup across five models and three tasks, with results showing that differences are "generally minimal, typically less than 1 percent." Standard errors are reported following the EleutherAI method (std err column in Table 5), providing "a statistical bound on the degree of variation in reported numbers" and showing that standard errors range from approximately 0.8 to 2.2 percentage points depending on task and model.

Main Quantitative Results

The experimental results in OLMES are organized around validating two distinct claims: (1) that the OLMES protocol produces well-justified normalization choices that are near-optimal across models, and (2) that OLMES enables meaningful model comparison across the capability spectrum by handling the CF-MCF transition automatically. These are evaluated through the normalization comparison experiments (Section 3.3, Tables 3 and 9–12), the CF-vs-MCF analysis (Section 3.4, Figure 2, Tables 6–7), and the final OLMES score table (Section 4, Table 4).

CF Normalization: Per-Task Recommendations and Their Near-Oracle Performance

The paper evaluates all four normalization schemes (none, token, character, pmi) across all 15 models on all 10 tasks, producing per-task win percentages that determine the OLMES recommendation. Table 3 presents the aggregate win percentages, while Tables 9–12 (Appendix C.2) provide the full per-model accuracy breakdown.

Headline finding: Task-specific normalization achieves near-oracle performance. The "diff oracle" column in Table 3 shows the average difference between the OLMES-recommended normalization's accuracy and the empirically best normalization for each model-task pair. These differences are small: 0.2% for ARC-Challenge, 0.1% for ARC-Easy, 1.1% for BoolQ, 0.6% for CommonsenseQA, 0.0% for HellaSwag, 0.4% for MMLU, 0.0% for OpenBookQA, 0.2% for PIQA, 0.1% for Social IQa, and 0.0% for WinoGrande. In plain terms, the OLMES recommendation is typically within a fraction of a percentage point of the best possible per-model normalization, while being a single fixed choice that enables fair comparison.

Per-task win percentage results (Table 3):

• ARC-Challenge: pmi wins on 66.7% of models (10 out of 15). character wins on 33.3% (5 out of 15). none and token win on 0%. The OLMES recommendation is pmi.

• ARC-Easy: character dominates with an 86.7% win rate (13 out of 15 models). none and token each win on 6.7% (1 model each). pmi wins on 0%. The OLMES recommendation is character.

• BoolQ: none and character each win on 46.7% of models (7 out of 15). pmi wins on 6.7% (1 model). Despite character having an equal empirical win rate, OLMES recommends none because character normalization's advantage is "an accidental side effect of 'yes' having one more character than 'no'"—a normalization that changes the relative probability of "yes" vs. "no" based on character count is "problematic" for a task with single-token, equal-length answers (Section 3.3).

• CommonsenseQA: pmi has a 53.3% win rate (8 models). character has 33.3% (5 models). none and token each have 6.7% (1 model each). The OLMES recommendation is pmi.

• HellaSwag: character achieves a 100.0% win rate—every single model performs best with character normalization. The OLMES recommendation is character. This is consistent with HellaSwag being essentially a language modeling continuation task where answer length is the primary confound.

• MMLU: pmi and character are closely matched, with pmi winning on 53.3% of models (8 out of 15) and character on 46.7% (7 out of 15). OLMES selects character despite pmi's slight edge, citing the fact that character normalization "is already implemented (as acc_norm) in the Eleuther LM Evaluation Harness" and avoids the computational overhead of computing unconditional likelihoods (Section 3.3).

• OpenBookQA: pmi achieves a 100.0% win rate—all 15 models perform best with pmi normalization. This is the strongest empirical signal for any normalization on any task.

• PIQA: character and token are tied at 46.7% win rate each (7 models each). none wins on 6.7% (1 model). pmi wins on 0%. OLMES selects character, consistent with the practical advantage of existing implementation support.

• Social IQa: character wins on 86.7% of models (13 out of 15). none, token, and pmi each win on 6.7% (1 model each). The OLMES recommendation is character.

• WinoGrande: none achieves a 100.0% win rate. All other normalizations win on 0%. This is expected: WinoGrande's CF formulation uses identical completions across choices with varying prompts, so normalization (which divides by completion length or unconditional probability) is mathematically irrelevant to ranking. none is selected for simplicity.

Detailed per-model results (Tables 9–12). These tables reveal that the OLMES recommendation is not always the best for every individual model, but the differences are consistently small. For ARC-Challenge (Table 9), the pmi score is best for 10 of 15 models, with the "diff" column showing that for models where character is better (e.g., MPT-7B: char 46.3% vs. pmi 45.7%, diff 0.6%), the gap is typically under 1%. For MMLU (Table 11), where pmi and character are closely matched, the per-model differences (diff column) range from 0.0% to 1.5%, confirming that the choice between them does not significantly affect any individual model's score.

The BoolQ edge case. Table 10 shows that for BoolQ, character normalization produces slightly higher accuracy than none for several models (e.g., Pythia-1B: 61.1% vs. 56.6%). The OLMES recommendation of none despite character's equal empirical win rate is perhaps the most controversial choice in the standard. The paper's justification—that a normalization which changes the probability of "yes" vs "no" based on character count is "problematic"—is a principled argument that prioritizes measurement validity over raw accuracy optimization. The implicit claim is that an evaluation standard should not exploit accidental properties of answer strings to inflate scores, even if those properties happen to correlate with correctness for some models.

CF vs. MCF: The Capability Spectrum and the Automatic Transition

The paper evaluates all 15 models on all 10 tasks under both the CF and MCF formulations, producing the comprehensive comparison in Tables 6–7 and Figure 2. The central empirical finding is that CF and MCF measure fundamentally different capabilities, with CF necessary for weak models and MCF necessary for strong ones, and that OLMES's best-of-two rule navigates this transition automatically.

Headline finding from Figure 2: CF extracts signal from weak models; MCF extracts signal from strong models. Across all 10 tasks, the weakest models (leftmost on x-axis) show MCF performance at or near the random baseline while CF performance is substantially above random. This pattern is most pronounced on ARC-Challenge, where the 8 weakest models have MCF scores between 24.1% and 28.1% (random for 4-choice is 25%) but CF scores between 31.4% and 47.3%. On MMLU, the 7 weakest models have MCF scores between 25.4% and 29.0% (random for 4-choice is 25%), while CF scores range from 31.1% to 40.5%. The CF formulation demonstrably extracts meaningful knowledge from models that cannot handle the MCF task structure.

For the strongest models, the pattern reverses: MCF scores dramatically exceed CF scores. On ARC-Challenge (Table 6), Llama3-70B achieves 93.7% MCF vs. 69.0% CF (a 24.7 percentage point gap). Mistral-7B-v0.1 achieves 78.6% MCF vs. 59.6% CF (19.0 point gap). On ARC-Easy, Llama3-70B reaches 97.7% MCF vs. 89.6% CF (8.1 point gap). On OpenBookQA (Table 7), the strongest models show gaps of 20+ points: Llama3-70B 93.4% MCF vs. 69.0% CF, Mistral-7B-v0.1 80.6% MCF vs. 61.0% CF. The MCF formulation provides substantially more accurate assessments of these models' capabilities.

Error rate analysis for strong models (Section 3.4). The paper emphasizes that these gaps are best understood through error rates rather than raw accuracy. For Llama3-70B on ARC-Challenge: MCF error rate = 6.3%, CF error rate = 31.0%. The ratio is approximately 4.9×—the CF score leaves nearly five times as much residual error unexplained. For models approaching human-level performance, this difference in measurement precision is critical: a 1% genuine improvement from 93.7% to 94.7% under MCF would be nearly invisible under CF.

The training-time transition: Figure 1 (MMLU during OLMo-7B-0424 training). This figure shows that the CF-MCF gap is not static but emerges along the training trajectory. At 0–200B training tokens, CF accuracy rises from ~25% to ~37% while MCF remains flat at ~25% (random). Around 400B tokens, MCF accuracy begins rising and eventually surpasses CF. The paper interprets this as the model acquiring the meta-cognitive ability to understand the multiple-choice format as a task structure—a capability that is learned later in training than the underlying content knowledge measured by CF. By the end of training, MCF provides the stronger signal.

Per-task exceptions to the CF-MCF gap pattern. Table 7 reveals three tasks where MCF does not clearly surpass CF even for strong models: HellaSwag, WinoGrande, and to a lesser extent BoolQ. On HellaSwag, Llama3-70B achieves 89.1% MCF vs. 89.5% CF—essentially tied. On WinoGrande, Llama3-70B achieves 79.6% MCF vs. 84.1% CF—CF remains higher. The paper notes (Appendix C.2.1) that these tasks in the CF format are "exactly like the language modeling task of finding the most natural continuation of a running piece of text," which may give CF an inherent advantage that even strong models cannot overcome through MCF format understanding. The paper speculates that it will be "interesting to monitor if as even more capable models are developed, the MCF scores will eventually surpass that of the CF scores."

Final OLMES Performance Scores: Reproducible Cross-Model Comparisons

Table 4 presents the fully reproducible OLMES scores for all 15 models across all 10 tasks, along with the average across tasks. Scores marked with † indicate the use of the MCF formulation (where it outperformed CF). The table demonstrates that OLMES produces a coherent ranking of models that aligns with known capability differences.

Average OLMES scores across all 10 tasks (Table 4, "average" column): Pythia-1B: 49.0%, OLMo-1B: 55.1%, TinyLlama-1.1B: 55.4%, Pythia-6.7B: 59.1%, RPJ-INCITE-7B: 62.8%, StableLM2-1.6B: 65.1%, OLMo-7B: 65.3%, MPT-7B: 65.6%, Falcon-7B: 66.9%, Llama2-7B: 69.0%, Llama2-13B: 74.0%, OLMo-7B-0424: 75.5%, Llama3-8B: 78.7%, Mistral-7B-v0.1: 79.1%, Llama3-70B: 88.4%.

Key observations from the ranking:

• Model scale correlates with performance within families but is not the whole story. Pythia-6.7B (59.1%) substantially outperforms Pythia-1B (49.0%). Llama2-13B (74.0%) outperforms Llama2-7B (69.0%). However, Llama3-8B (78.7%) outperforms Llama2-13B (74.0%) by 4.7 points despite having fewer parameters, reflecting architectural and training data improvements.

• The OLMo training iteration shows clear progress. OLMo-7B-0424 (75.5%) improves over the original OLMo-7B (65.3%) by 10.2 percentage points on average, consistent with the training improvements documented in the OLMo release.

• The highest-scoring model achieves 88.4% average. Llama3-70B leads across nearly all tasks, with particularly strong MCF scores on ARC-Challenge (93.7%), ARC-Easy (97.7%), and OpenBookQA (93.4%). The remaining error is concentrated on the hardest tasks: MMLU (79.8%), CommonsenseQA (83.2%), and Social IQa (78.9%).

The extended results table (Table 13, Appendix D) expands the evaluation to 40 models, including more recent releases like Qwen2, Gemma2, DeepSeek, and Llama3.1 variants. This table demonstrates that OLMES scales to new models without modification—the same protocol produces meaningful comparisons as the field advances. The highest-scoring model in this extended set is Qwen2.5-72B at 91.7% average, with near-ceiling performance on ARC-Easy (98.8%) and OpenBookQA (97.4%).

MMLU macro vs. micro comparison (Table 8, Appendix B). Across 15 models, the difference between macro-averaging (over 57 subjects) and micro-averaging (over all instances) is consistently small, typically less than 1 percentage point. For Llama3-8B under MCF: 66.6% macro vs. 65.4% micro, a 1.2 point gap. For Llama2-7B under CF: 44.4% macro vs. 44.3% micro, a 0.1 point gap. This validates that the macro-averaging choice, while principled (better representing subject diversity), does not substantively affect model rankings.

Ablation Studies and Robustness Checks

Cross-reference score variation for the same model-task pairs (Tables 1 and 14): The paper documents the magnitude of the problem that OLMES addresses by tabulating published scores for the same model on the same benchmark across different references. On ARC-Challenge (Table 1), Llama2-7B's reported score ranges from 43.2% to 53.7% across six references—a 10.5 percentage point spread. Llama2-13B ranges from 48.8% to 67.6%. The "Num shots" row reveals that references use anywhere from 0 to 25 shots; "Formulation" shows that some use CF, some use MCF, and some are undocumented ("CF?"). Table 14 extends this analysis to OpenBookQA, where Falcon-7B scores range from 26.0% to 55.2% and Llama2-7B from 48.4% to 58.6%. The paper does not claim these variations are "ablations" in the controlled sense, but rather uses them as empirical evidence that without OLMES, reported benchmark scores are not meaningfully comparable.

Robustness to minor prompt variations and few-shot example changes (Table 5, Appendix A): The paper evaluates five models (Pythia-1B, Llama2-7B, DeepSeek-7B, Gemma2-2B, Llama3-8B) on three tasks (ARC-Easy, OpenBookQA, PIQA) under four conditions: the standard OLMES setup ("orig"), a variant with minor prompt wording changes ("var1": label separator changed from "." to ")", extra newline before answer descriptor, "Answer" changed to "Correct answer"), a variant with different curated few-shot examples ("var2": 3 of 5 examples replaced following the same curation principles), and both variants combined ("var3"). The "avg" column averages across all four conditions. The "diff" column computes |orig − avg|—the absolute difference between the standard OLMES score and the average of all variants. These differences are: for ARC-Easy, 0.4%, 0.0%, 0.0%, 0.8%, 0.2% across the five models; for OpenBookQA, 1.4%, 0.9%, 0.2%, 0.8%, 0.5%; for PIQA, 0.2%, 0.0%, 0.6%, 0.0%, 0.3%. Every difference is under 2 percentage points, and most are under 1 point. The standard errors ("std err") range from 0.8% to 2.2%, confirming that the observed differences are within the expected statistical noise of the evaluation. This demonstrates that OLMES scores are stable under small perturbations to prompt wording and few-shot example selection, as long as the general principles (natural formatting, balanced label coverage) are maintained. The paper explicitly notes that "larger differences would be expected when diverging from OLMES recommendations such as by using unnatural prompts e.g., using rare symbols as answer labels, or randomly sampled few-shot examples which could run into skewed label distribution."

MMLU macro vs. micro averaging (Table 8): Across all 15 models and both formulations, the difference between macro-averaging (over 57 subjects) and micro-averaging (over 14,042 instances) is consistently modest. The largest gaps occur under MCF for stronger models: Llama3-8B shows 66.6% macro vs. 65.4% micro (1.2 point gap), and OLMo-7B-0424 shows 54.4% macro vs. 52.8% micro (1.6 point gap). Under CF, the gaps are smaller: Llama2-13B shows 47.6% macro vs. 47.1% micro (0.5 point gap). These differences are small enough that the choice does not meaningfully affect model rankings—using micro-averaging would not change the conclusions about which model performs better on MMLU. The paper's selection of macro-averaging is therefore a principled choice (better representing subject diversity) that comes at essentially no cost in terms of altered conclusions.

Normalization scheme comparisons across all model-task pairs (Tables 9–12): The detailed per-model, per-task breakdown of all four normalization schemes serves as an implicit ablation of OLMES's task-level recommendations. For ARC-Challenge (Table 10), the pmi score equals or exceeds the next-best normalization for 10 of 15 models. The models where character performs better (e.g., MPT-7B: char 46.3% vs. pmi 45.7%, diff 0.6%; Llama2-13B: char 57.1% vs. pmi 56.2%, diff 0.9%) show gaps under 1.5 percentage points. No model shows a gap larger than 2 percentage points between the OLMES recommendation and the oracle-best normalization. For CommonsenseQA (Table 11), pmi is strictly best for 8 of 15 models, with the largest gap between pmi and the next-best being 3.5 points for OLMo-1B (62.2% pmi vs. 60.0% char). For OpenBookQA (Table 12), pmi achieves a 100% win rate, with gaps between pmi and the next-best normalization ranging from 3.6 points (Falcon-7B: 55.2% pmi vs. 45.8% char) to 23.0 points (Pythia-1B: 40.4% pmi vs. 20.2% none). The latter represents the most dramatic case where normalization choice alone can produce wildly different conclusions about model capability.

HELM MMLU reproduction analysis (Figure 4, Appendix E): The paper includes external evidence from HELM's reproduction of MMLU scores across multiple models. Figure 4 shows a scatter plot where each point represents a model, the x-axis is the self-reported MMLU score, and the y-axis is the difference between the self-reported score and the HELM-reproduced score. Points above y=0 indicate self-reported scores that are higher than the reproduction. The trend line has a positive slope (R²=0.26), indicating "that on average, the higher the self-reported score, the more they overestimate performance compared to the reproduction." This is not an OLMES-controlled experiment but serves as corroborating evidence that the evaluation irreproducibility problem is widespread and systematic—higher-performing models (as self-reported) tend to have larger discrepancies when their evaluations are reproduced under a consistent protocol.

Critical Assessment

Does OLMES Achieve Reproducible Evaluation?

The paper's central claim is that OLMES provides "a completely documented, practical, open standard for reproducible LLM evaluations" (Abstract) where previously "a statement like 'model X scores Y on ARC-CHALLENGE' is generally uninterpretable" (Section 2.1). The evidence presented supports this claim with important qualifications.

What the experiments demonstrate. Tables 1 and 14 convincingly establish that existing evaluation practices produce irreproducible results—the same model on the same dataset yields scores spanning 10+ percentage points across references. Table 4 demonstrates that OLMES produces a specific, fully documented set of scores for 15 models that anyone can reproduce by following the published protocol. Table 5 shows that these scores are robust to small perturbations in prompt wording and few-shot example selection, with variations typically under 1 percentage point and within statistical error bounds. The extended Table 13 shows OLMES scales to new models without modification, demonstrating that the standard is not overfit to the initial 15-model set.

What the experiments do not demonstrate. The paper does not provide a direct head-to-head reproduction study where multiple independent teams evaluate the same models using OLMES and verify that they obtain identical scores. Such a study would be the gold standard for demonstrating reproducibility—showing not just that the protocol can be specified, but that different groups following that specification actually converge on the same numbers. The Table 5 robustness check is a single-team experiment with controlled perturbations, not an independent reproduction. The actual reproducibility of OLMES scores across different hardware, different software stacks, and different teams remains an open empirical question, though the level of specification detail strongly suggests reproducibility should be achievable.

The floating-point non-determinism caveat (Section 3.5) is worth taking seriously. The paper acknowledges that "batch size and type/state of GPU can affect floating point operations such that answer choice decisions can flip if they are very close." If two answer choices have probabilities that differ by less than floating-point epsilon, different GPU configurations could produce different rankings. The paper does not implement tie-breaking for such cases, meaning that OLMES scores could diverge by small amounts (roughly the fraction of instances where probabilities are near-tied) across hardware configurations. The magnitude of this effect is likely small (probably well under 1%), but it means OLMES scores are reproducible up to this floating-point tolerance, not bit-for-bit identical.

Does OLMES Enable Meaningful Comparisons Across the Full Capability Spectrum?

The paper claims that OLMES "supports meaningful comparisons between smaller base models that require the unnatural 'cloze' formulation of multiple-choice questions against larger models that can utilize the original formulation" (Abstract). The evidence supports this claim with important caveats about what "meaningful" means.

What the experiments demonstrate. Figure 2 and Tables 6–7 clearly show that CF is necessary to extract above-random signal from weak models (where MCF is near random) and that MCF provides substantially higher scores for strong models (where CF plateaus). The OLMES best-of-two rule correctly identifies the better formulation for each model-task pair, producing scores that are the best available measurement given the chosen evaluation methodology. The 15-model ranking in Table 4 is coherent: larger models within the same family outperform smaller ones, and known improvements (OLMo-7B vs. OLMo-7B-0424, Llama2 vs. Llama3) are reflected in the scores.

What the experiments do not demonstrate—and the conceptual limitation. The paper does not establish that CF and MCF scores are calibrated on a common scale. When OLMES reports that Pythia-1B scores 31.4% on ARC-Challenge (using CF) and Llama3-70B scores 93.7% (using MCF), the inference that Llama3-70B is ~3× better is based on the assumption that these numbers measure the same underlying construct on comparable scales. But the paper's own analysis demonstrates that CF and MCF measure different capabilities: CF measures content knowledge + language modeling, while MCF measures content knowledge + format understanding. These are different measurement instruments applied to different models, and the fact that both produce numbers between 0% and 100% does not guarantee they are comparable.

Consider a hypothetical: a model with perfect content knowledge but zero format understanding would score near 100% on CF and 25% on MCF. OLMES would report 100% (using CF). Another model with imperfect content knowledge but perfect format understanding might score 80% on CF and 90% on MCF. OLMES would report 90% (using MCF). Which model is "better" depends on the use case: the first model has superior knowledge, the second model has superior format-following ability. OLMES treats them as comparable because both numbers are "accuracy on ARC-Challenge," but they are accuracy on different tasks with different difficulty characteristics. The paper does not address this commensurability problem—whether it exists, how large it is, or how users should interpret cross-formulation comparisons. This is a genuine limitation, though it may be practically acceptable: the alternative (using only CF or only MCF) would produce even more misleading comparisons by forcing one model to be evaluated with an inappropriate instrument.

Does the Normalization Selection Procedure Produce Optimal Recommendations?

The paper claims that OLMES "provides justified recommendations on all aspects of task setups" (Abstract) and that the normalization choices are "guided by results from existing literature as well as new experiments resolving open questions" (same). The evidence partially supports this claim, with some recommendations being more strongly justified than others.

Strongly justified recommendations. The selections of pmi for OpenBookQA (100% win rate across 15 models) and character for HellaSwag (100% win rate) are unambiguously the correct choices given the empirical evidence. The selection of none for WinoGrande is mathematically inevitable—normalization cannot affect ranking when completions are identical across choices. The "diff oracle" analysis in Table 3 shows that for most tasks, the OLMES recommendation is within 0.0–0.6% of the oracle-best per-model normalization, confirming that per-task recommendations capture most of the available benefit of task-aware normalization.

Recommendations with some tension between empirical and principled criteria. The BoolQ recommendation (none over character) is the clearest case where the paper overrides empirical evidence (46.7% win rate for both) with principled reasoning (character normalization accidentally advantages "yes"). This is intellectually honest but raises the question: if principled reasoning can override empirical evidence here, what prevents others from making different principled arguments that lead to different normalization choices? The paper argues that "one should note that this is an accidental side effect of 'yes' having one more character than 'no' and indeed a normalization which changes the probability of 'yes' vs 'no' simply because the 'no' token has fewer characters seem problematic" (Appendix A). This is a compelling argument, but it is an argument about measurement validity, not about empirical performance. A different evaluator might prioritize empirical performance (always choosing whatever gives the highest score for each task) and select character for BoolQ. The OLMES standard resolves this by documenting its reasoning, but the underlying tension between empirical optimization and principled measurement remains.

The MMLU recommendation (character over pmi, despite pmi having a 53.3% win rate vs. 46.7%) is justified on practical grounds: character is already implemented in the Eleuther LM Harness and avoids computational overhead. This is a reasonable tradeoff given that the performance difference between the two is small (0.4% "diff oracle"), but it means the recommendation is not purely empirical—practical convenience plays a role. Researchers who care more about extracting every possible fraction of a percentage point of performance (e.g., for competitive leaderboard rankings) might prefer pmi. OLMES's documentation of this tradeoff enables informed disagreement, which is a strength of the approach even if it means the recommendation is not uniquely determined by the data.

What would strengthen the recommendations. The paper evaluates 15 models, which is a reasonable sample but not exhaustive. The normalization win percentages are computed on this set, and there is no out-of-sample validation showing that the same recommendations generalize to models with radically different architectures, training objectives, or tokenization schemes. A systematic study of how normalization effectiveness varies with model properties (size, vocabulary, training data distribution) would strengthen confidence that the OLMES recommendations are not artifacts of the specific 15-model set. The paper also does not explore normalization robustness across different prompt formats—does pmi remain optimal for ARC-Challenge if the few-shot examples or question prefix are changed? The current evaluation fixes all other protocol choices while varying only normalization, so the interaction between normalization and other protocol choices is unexplored.

What Major Experiments Are Missing?

Several experiments would have strengthened the paper's claims but are absent:

1. Independent reproduction study. As noted above, having 2–3 independent teams evaluate a subset of models using OLMES and confirming score agreement would be the strongest possible evidence for reproducibility. The paper's robustness check (Table 5) is a single-team perturbation study, which demonstrates insensitivity to small changes but not cross-team reproducibility.

2. CF-to-MCF calibration study. The paper identifies the CF-vs-MCF commensurability problem but does not attempt to quantify it. An experiment where human annotators are given the same questions in CF and MCF formats (as sentence completions vs. as multiple-choice) and their accuracy is compared across formats would establish a human baseline for the CF-MCF gap. If humans show a similar gap on certain tasks, the gap in model scores is partly a property of the task rather than a model deficiency. If humans show no gap, the model gap is a genuine capability deficit.

3. Instruction-tuned model comparison. The paper focuses on base models, which is a deliberate and justified scope choice, but the community also evaluates instruction-tuned models extensively. An ablation showing how OLMES scores compare for base vs. instruction-tuned versions of the same model would help users understand whether the standard applies equally well to tuned models or requires modification (e.g., different prompt formatting, inclusion of system messages).

4. Difficulty estimation beyond CF-vs-MCF. The paper uses the CF-MCF gap as an implicit difficulty diagnostic, but does not systematically analyze whether OLMES scores correlate with other measures of task difficulty (e.g., human accuracy, question length, required reasoning steps). Such analysis would help validate that OLMES is measuring capability rather than surface-form artifacts.

5. Adversarial prompt robustness. Table 5 tests minor, benign variations. Testing more extreme perturbations—adversarially chosen to confuse the model while preserving the task's semantic content—would establish the limits of OLMES's robustness and identify which protocol choices are load-bearing vs. cosmetic.

Summary Assessment

OLMES achieves its stated goal of providing a documented, practical, open standard that eliminates the specific sources of irreproducibility identified in Tables 1 and 14. The normalization recommendations are well-supported by the empirical win-rate analysis, and the CF-MCF dual-evaluation approach is a principled resolution to a genuine tension in the field. The standard's main limitation is conceptual rather than empirical: it does not fully address whether CF and MCF scores measure the same underlying construct on comparable scales, and it relies on a fixed set of 15 models for its empirical validation without out-of-sample testing on architecturally dissimilar models. The documented justifications are genuinely useful and represent a meaningful advance over the status quo of undocumented defaults, even where the recommendations involve tradeoffs between empirical optimization and principled measurement design. The standard's practical value for the community will depend on adoption—and the paper's transparency about its choices and their justifications is the strongest argument for why that adoption should occur.

6. Limitations and Trade-offs

The CF-to-MCF Commensurability Problem Is Unresolved

The assumption or constraint. OLMES's core decision rule—evaluate with both CF and MCF, report whichever scores higher—implicitly assumes that CF and MCF scores measure the same underlying construct on comparable scales, such that a 70% CF score for one model and a 70% MCF score for another represent equivalent capability. The paper's own analysis demonstrates this assumption is false: CF and MCF measure different capabilities, with CF capturing content knowledge expressed through language modeling and MCF additionally requiring meta-cognitive format understanding. Section 3.4 establishes this explicitly, showing that the CF-MCF gap varies dramatically across models and that the MCF capability "starts gaining the ability... around 400B tokens" of training (Section 3.4). The paper does not claim CF and MCF are equivalent measurements—in fact, it argues the opposite—but the best-of-two aggregation treats them as interchangeable for the purpose of computing a final score.

The consequence. When OLMES reports that Pythia-1B scores 31.4% on ARC-Challenge (using CF) and Llama3-70B scores 93.7% (using MCF), these numbers come from fundamentally different measurement instruments. The 62.3 percentage point gap conflates two separate effects: genuine capability improvement (the larger model possesses more scientific knowledge) and measurement instrument change (MCF is a more sensitive instrument that can detect capabilities CF cannot). A researcher comparing these numbers cannot distinguish how much of the gap reflects the model's superior knowledge versus the superior measurement properties of MCF for strong models. This matters for practical decision-making: if a model developer sees their model's OLMES score jump from 65% to 75% between training checkpoints, they cannot tell whether the model genuinely improved or simply crossed the MCF format-understanding threshold and got measured with a different yardstick. The paper's own Figure 1 shows exactly this phenomenon—MMLU accuracy jumps when MCF overtakes CF around 400B tokens, but some of that jump is a measurement artifact, not a capability discontinuity.

What evidence exists in the paper. The paper thoroughly documents the CF-MCF divergence but does not measure its impact on cross-model comparability. Tables 6–7 provide per-model CF and MCF scores, showing gaps ranging from near-zero (HellaSwag for strong models) to 24.7 percentage points (ARC-Challenge for Llama3-70B). Figure 2 visualizes the systematic relationship between model strength and the CF-MCF gap. The paper notes in the appendix FAQ (Appendix A) that MCF is "a better indicator of performance later on" while "CF gives a clearer signal early in training," acknowledging the instruments serve different purposes at different stages. However, no experiment calibrates these instruments against a common reference—for example, human performance on the same questions in both formulations, or the correlation between CF and MCF scores within models that can handle both—that would allow users to interpret cross-formulation score differences.

Mitigation status. The paper does not attempt to resolve this limitation. The † annotation in Table 4 provides transparency about which formulation produced each score, which is a partial mitigation—readers can at least identify when cross-model comparisons cross formulation boundaries. But the standard provides no guidance on how to interpret those comparisons, leaving users to make their own judgments about whether a 5-point gap that crosses the CF-MCF boundary is more or less meaningful than a 5-point gap within a single formulation. The paper's framing that OLMES "supports meaningful comparisons between smaller base models that require the unnatural 'cloze' formulation... against larger models that can utilize the original formulation" (Abstract) makes a claim about meaningfulness that the calibration evidence does not fully substantiate.

Difficulty Estimation Overhead Is Not Accounted for in the Evaluation Cost

The assumption or constraint. OLMES requires running every model-task pair through both the CF and MCF formulations to determine which scores higher. This doubles the inference cost relative to evaluating with a single formulation. Additionally, the pmi normalization for CF (used for ARC-Challenge, CommonsenseQA, and OpenBookQA) requires an unconditional forward pass for each answer choice—computing $P(a_i \mid u)$ where $u =$ "Answer:" with no question context—adding approximately 1× the answer-choice-count in extra forward passes per instance. The paper acknowledges this cost for pmi specifically (Section 3.3: "Computing the extra unconditional likelihood incurs some computation overhead, thus OLMES avoids this normalization for other datasets where there is no strong empirical or theoretical reason"), and the FAQ (Appendix A) addresses the general question of "trying a range of settings" by arguing that OLMES saves researchers from having to do that exploration themselves.

The consequence. The total inference cost for a full OLMES evaluation across all 10 tasks is roughly 2× (for dual CF/MCF) plus the pmi overhead for 3 tasks (approximately 4–5 unconditional forward passes per instance per task with pmi, given 4–5 answer choices). The paper reports using "around 400 GPU hours" (Appendix F) on NVIDIA RTX A6000 GPUs, but this figure is for the initial development experiments—running all 15 models on all tasks with all normalization variants to select the per-task recommendations. The ongoing cost for a model developer evaluating a single new model using OLMES, having already accepted the per-task recommendations, would still be roughly double the cost of a single-formulation evaluation. For large models (70B parameters), this could represent substantial GPU expenditure. The paper's justification for practical efficiency—capping instances at 1000, using 5-shot rather than 25-shot—offsets some of this cost, but the doubling from dual formulation remains unaddressed. A model developer training a 70B model and evaluating at multiple checkpoints would pay this overhead at each checkpoint.

What evidence exists in the paper. The paper does not quantify the per-model inference cost or break down the cost between CF, MCF, and pmi components. The 400 GPU-hour figure in Appendix F is a total for all experiments, not a per-model estimate. Table 3 addresses the pmi cost indirectly by recommending character normalization whenever pmi is not strongly empirically justified—this is the paper's primary cost-saving measure, and it affects 7 out of 10 tasks. For the three pmi tasks, the overhead is accepted as the price of measurement validity. The FAQ (Appendix A) argues that "rather than having to... run the different settings to choose a normalization scheme, or delve into the same literature again to study the variants, the community can now directly build upon the various choices in OLMES." This is a comparative argument—OLMES is cheaper than individual exploration—but it does not address whether 2× the inference cost of a single-formulation evaluation is itself well-spent for the measurement improvement.

Mitigation status. The paper partially mitigates this through the per-task normalization recommendations, which eliminate the need to evaluate all four normalizations per task (a 4× cost for CF alone). The dual CF/MCF requirement is not mitigated—it is a core feature of the standard, justified by the finding that neither formulation alone works across the full capability spectrum. The paper does not suggest any future work on reducing this cost, such as using a lightweight heuristic to predict whether CF or MCF will score higher for a given model-task pair without running both, or sharing the CF forward passes between pmi and non-pmi tasks where answer strings overlap.

All Results Are from a Single Task Family (MCQA) and a Single Model Distribution

The assumption or constraint. OLMES is designed exclusively for multiple-choice question answering tasks, evaluated on 10 benchmarks that share the same fundamental structure: a question or context, 2–5 answer choices, and a single correct answer that can be scored by exact match. The paper is explicit about this scope: "We specifically focus on evaluation using these tasks to provide useful guidance during and after base model training" (Section 2.1). The 15 models used to develop the standard are all autoregressive transformer-based language models trained on web-scale text corpora, with similar architectures and tokenization strategies. While the set spans 1B to 70B parameters and includes models from multiple organizations (Meta, Mistral, AI2, EleutherAI, Technology Innovation Institute), all share the fundamental pretraining paradigm of causal language modeling on internet text.

The consequence. The normalization recommendations, the CF-vs-MCF transition patterns, and even the optimal prompt formatting could be specific to this task family and this model distribution. The paper provides no evidence that pmi remains the best normalization for ARC-Challenge-style tasks when evaluated on models with different pretraining objectives (e.g., encoder-decoder architectures like T5, mixture-of-experts models with different token-level probability calibration, or models trained primarily on code rather than natural language). The CF-MCF transition—where models "acquire" MCF capability around a certain training point—may not generalize to models with different training curricula, different data mixtures, or architectural inductive biases that affect instruction-following ability independently of content knowledge.

More practically, the OLMES standard does not address generative tasks (where scoring requires output parsing rather than probability comparison over fixed choices), math reasoning tasks (where chain-of-thought prompting is standard practice), coding benchmarks, or open-ended generation evaluation. These tasks represent a large fraction of how LLMs are evaluated in practice, and the OLMES principles (curated few-shot examples, dual-formulation evaluation, per-task normalization) do not directly transfer. For a generative task, questions like "how to extract the final answer from the model's output" and "how to handle multiple valid phrasings of the same answer" are the evaluation bottlenecks, and OLMES provides no guidance on these.

What evidence exists in the paper. The paper acknowledges this limitation explicitly. Section 2.1 states the focus on MCQA: "Multiple-choice tasks are the most common type of benchmarks for evaluating base LLMs... where the evaluation seems straightforward (did the model predict the right answer?), but in practice... is generally uninterpretable without a clear evaluation standard like OLMES." The "Future work and limitations" section of the Discussion (Section 6) states: "Future work includes adding more tasks to OLMES, covering tasks beyond MCQA such as generative tasks and chain-of-thought prompting." The appendix FAQ addresses "Including a broader range of datasets?" by arguing that the fundamental principles generalize even though the current implementation is MCQA-focused. However, no experiments on non-MCQA tasks are presented, and no evidence supports the claim that the normalization or formulation recommendations would transfer.

The paper also does not test whether the normalization recommendations depend on the model distribution. All 15 models are decoder-only transformers trained on similar data at similar scales. There is no encoder-decoder model (e.g., T5, UL2), no retrieval-augmented model, no model trained primarily on non-English text, and no model with fundamentally different tokenization (e.g., byte-level tokenization without subword merging). The normalization win percentages in Table 3 could shift substantially for models with different probability calibration properties.

Mitigation status. The paper explicitly flags this as future work and invites community contributions: "We welcome the community to contribute to OLMES, extending the principles of OLMES to new tasks" (Section 6). The extended results table (Table 13) includes 40 models, some of which (Qwen2, Gemma2, DeepSeek-MoE) have different architectures and training data than the original 15, providing some evidence that the standard generalizes within the autoregressive transformer family. However, this is incremental extension within the same paradigm, not evidence of generalization to fundamentally different model classes or task types. The limitation is scoped rather than resolved—OLMES is explicitly an MCQA evaluation standard for base models trained with causal language modeling objectives, and its principles have unknown applicability outside that scope.

The 15-Model Empirical Basis May Not Yield Recommendations That Generalize to Radically Different Models

The assumption or constraint. The per-task normalization recommendations, the conclusion that 5-shot prompts are sufficient, and the decision to use the best of CF and MCF are all empirically derived from experiments on 15 specific models. These models, while diverse in size and origin, all belong to the same broad class: autoregressive transformer language models trained with a causal language modeling objective on large-scale web text. The paper implicitly assumes that the empirical patterns observed in this set—which normalization wins most often, where the CF-MCF transition occurs, how many shots saturate performance—will hold for future models with possibly different architectures, training objectives, or data distributions.

The consequence. There are two related failure modes. First, a model with fundamentally different probability calibration—for instance, a diffusion language model that does not produce token-level probabilities in the standard way, or a model trained with a contrastive objective that affects unconditional token probabilities—could break the normalization recommendations entirely. The pmi normalization, in particular, relies on the ratio of conditional to unconditional probability being a meaningful signal; if the model's unconditional distribution is pathological (e.g., assigns near-zero probability to all answer strings because they are out-of-distribution), pmi could amplify noise rather than correct for prior bias. Second, a model that acquires format-understanding capability much earlier or later in training than the 15 studied models—for instance, a model pretrained on instruction-formatted data rather than raw web text—could show a different CF-MCF transition pattern, making the best-of-two rule less effective or even counterproductive (if MCF is always better, the CF evaluation is wasted compute; if MCF is never acquired, the MCF evaluation is wasted compute).

The paper acknowledges the possibility of model-specific peculiarities in the FAQ (Appendix A): "If a model is peculiar in any specific way (e.g., only able to do multiple choice questions with one type of answer label like '1.' or '2.'), it is not the goal of OLMES to tailor to such peculiarities as this standard is intended to be applied across a range of models and to encourage the development of models that produce reasonable outputs given any reasonable input." This is a reasonable position—a standard cannot accommodate every possible model design—but it means OLMES's validity for models outside the empirical validation set is assumed, not demonstrated.

What evidence exists in the paper. The paper does not test whether the recommendations generalize to architecturally dissimilar models. The extended Table 13 adds 25 more models to the original 15, but these are within the same autoregressive transformer family (Qwen2, Gemma2, DeepSeek, Llama3.1, etc.). The 100% win rate for character normalization on HellaSwag and pmi on OpenBookQA across all 15 original models is strong evidence that these recommendations are robust within the studied model class, but not evidence that they generalize outside it. The stability analysis in Table 5 demonstrates robustness to prompt variation, not to model distribution shift.

The paper also does not analyze whether the normalization win percentages correlate with model properties (size, training data, tokenizer type). Such analysis could reveal whether the recommendations are driven by properties that are likely to hold for future models (e.g., pmi works better when answer choices contain rare words, which is a property of the dataset, not the model) or by contingent properties of the 15-model set (e.g., pmi works better for models trained on particular data distributions). The paper's argument for pmi on ARC-Challenge, CommonsenseQA, and OpenBookQA is primarily task-based (these datasets "tend to contain unexpected words or phrases") rather than model-based, which provides some theoretical grounding—the pmi correction addresses a known confound that exists in the dataset regardless of which model is evaluated. But the empirical validation of this theoretical argument comes entirely from the 15-model set.

Mitigation status. The paper partially mitigates this by grounding recommendations in task-level reasoning rather than pure empirical optimization. The BoolQ decision to use none despite character having equal empirical win rate (Section 3.3) demonstrates that the paper does not blindly follow the empirical win percentages—it considers principled arguments about what each normalization corrects for and whether that correction is appropriate for the task. This increases confidence that the recommendations would generalize: pmi is recommended for tasks with unusual answer vocabulary, character for tasks where answer length is the primary confound, and none for tasks where neither confound is present. These are task properties that hold regardless of the model being evaluated. However, the underlying assumption that these confounds are the only ones that matter, and that the normalization schemes correctly address them for all model types, remains empirically untested for models outside the autoregressive transformer class.

The Standard Does Not Account for Known Option-Position and Answer-Label Biases in MCF

The assumption or constraint. The MCF formulation evaluates models by scoring the probability of answer label tokens (A, B, C, D) following the prompt. This implicitly assumes that models assign probability to these labels based on the content of the question and answer choices, not based on superficial biases toward particular labels or positions. The paper is aware of this issue—the appendix FAQ notes that "for MCF it is indeed a confounder that some (especially weaker) models might highly prefer a given label (like B)"—but OLMES does not implement any correction for label bias or position bias beyond relying on the CF formulation to outscore MCF when such biases dominate.

The consequence. Recent work (Zheng et al., 2024; Li et al., 2024; Wang et al., 2024), cited in the paper's Related Work section, has demonstrated that LLM performance on MCF can be highly sensitive to "option position changes in multiple-choice questions... choice symbols, re-ordering of answer options, changing number of answer options." A model that systematically prefers answer label "B" regardless of content will score above random on MCF simply because the correct answer is "B" roughly 25% of the time for 4-choice tasks. This creates a perverse incentive: a model can improve its MCF score by learning a spurious label preference rather than by improving its content knowledge. The OLMES best-of-two rule would use the MCF score whenever it exceeds the CF score, meaning the reported OLMES score could be partially inflated by label bias.

The practical consequences are twofold. First, for weak models, MCF scores that are slightly above random (e.g., 28% rather than 25% on a 4-choice task) could reflect label bias rather than genuine capability—but OLMES would still use the MCF score if it exceeds the CF score by even a small margin, potentially rewarding models for spurious biases. Second, for model comparison, two models with identical content knowledge but different label biases could receive different OLMES scores, creating a ranking that reflects evaluation artifacts rather than capability differences.

What evidence exists in the paper. The paper does not measure the magnitude of label bias in the 15 evaluated models under the OLMES MCF format. The FAQ suggests that CF will generally outscore MCF when label bias dominates: "The benchmarks in OLMES are generally balanced such that such a model would not be much better than random. Further, if this happens, CF would generally get a better score in such cases and OLMES would use that score in its final output." This is partially reassuring—if a model has strong label bias, its MCF score will be near the label's frequency in the benchmark (~25% for balanced 4-choice tasks), and CF will typically exceed that for any model with non-trivial content knowledge. But there is an intermediate regime where a model has weak content knowledge plus moderate label bias, producing MCF scores of 30–40% that exceed CF scores of 25–35% because the label bias provides a small boost. In this regime, OLMES would report the MCF score, treating the label bias as genuine capability.

The paper considered but rejected more rigorous debiasing measures: "We considered applying more rigorous measures (like running all cyclic permutations of answer choices) but decided for practical reasons, the extra processing time and complexity were not worth the minor improvements in robustness (as one consideration of OLMES is also to be a practical standard that does not take unnecessarily more compute than is needed)" (Appendix A FAQ). The phrase "minor improvements in robustness" is an assertion, not an empirical finding—the paper does not measure how much label bias affects MCF scores under OLMES to justify calling the improvement "minor."

Mitigation status. The dual CF/MCF evaluation provides partial mitigation: CF is unaffected by label bias (since each answer choice is scored independently by its content, not by a label), so any inflation in MCF scores from label bias would need to push MCF above CF to affect the final OLMES score. For the practical impact to be significant, label bias would need to push MCF scores substantially above CF scores for models with non-trivial CF performance—a scenario that is plausible for early-training checkpoints but unlikely for strong models where CF already provides high scores. The FAQ's acknowledgment of the issue and the decision not to implement permutation-based debiasing are transparent tradeoffs between measurement accuracy and computational practicality. However, the standard provides no guidance on how users should interpret MCF scores for models where label bias is suspected, and no diagnostic for detecting when label bias is inflating reported performance.

7. Implications and Future Directions

How This Work Changes the Landscape

OLMES does not introduce a new model, algorithm, or benchmark. Its contribution is methodological infrastructure—and that is precisely why its potential impact is substantial. The paper reframes LLM evaluation from a benchmark selection problem to a measurement protocol design problem, and this reframing has cascading implications for how the field produces and consumes performance claims.

The reframing: evaluation as measurement protocol design. Prior to OLMES, the dominant mental model in the community was roughly: "to evaluate a model, run it on a set of benchmarks and report the accuracy." The benchmarks themselves were treated as the measurement instruments. But Table 1 demonstrates that the same model on the same benchmark produces scores spanning 10+ percentage points depending on how the benchmark is administered. The benchmark is not the instrument—the protocol is. OLMES makes this explicit by treating every evaluation choice (prompt format, few-shot examples, normalization, formulation) as a parameter of the measurement instrument that must be specified and justified, not as an implementation detail to be swept under the rug.

This is a conceptual shift comparable to the difference between saying "we measured the temperature" and "we measured the temperature using a mercury thermometer calibrated at 0°C and 100°C, with readings taken at eye level after 3 minutes of equilibration." The former is uninterpretable; the latter is reproducible. OLMES provides the latter for LLM evaluation, and in doing so, it establishes a new floor for what counts as a credible performance claim. A paper that reports "our model scores 82% on ARC-Challenge" without specifying the evaluation protocol is, after OLMES, demonstrably incomplete—Table 1 shows that the number could be anywhere in a wide range depending on unspecified choices. The standard provides a concrete reference: "82% on ARC-Challenge using OLMES" is a fully specified claim that anyone can reproduce and meaningfully compare against.

The magnitude of this shift should not be overstated: OLMES is not a paradigm shift in the Kuhnian sense. It does not challenge the validity of MCQA benchmarks as evaluation instruments or propose a new theory of model capability. It is a standardization effort—an incremental but essential improvement to the measurement infrastructure of the field, analogous to the introduction of standardized effect size reporting in psychology or standardized benchmark suites in computer vision (e.g., ImageNet). The paper's value lies not in theoretical novelty but in the systematic, documented resolution of a measurement crisis that the field has long acknowledged but never systematically addressed.

Reconciling prior contradictions. One of the paper's most valuable contributions is providing a framework for understanding why prior work produced contradictory findings about model performance. The key reconciliation comes from the CF-vs-MCF analysis (Section 3.4, Figure 2): weak models perform near-random on MCF but show meaningful signal on CF; strong models show the reverse. A researcher evaluating only weak models with MCF would conclude the models "don't understand the task"; a researcher evaluating only strong models with CF would conclude the models "plateau below their true capability." Both conclusions are artifacts of using the wrong instrument for the model being measured. OLMES resolves this contradiction by documenting when each formulation is appropriate and providing a decision rule (best-of-two) that automatically selects the right instrument.

Similarly, the normalization analysis (Tables 3, 9–12) reconciles the diversity of normalization practices in prior work. Brown et al. (2020) used pmi selectively for ARC and OpenBookQA; Llama evaluations defaulted to character normalization; HELM settled on per-task defaults without full documentation. OLMES provides the missing connective tissue: a principled framework for normalization selection based on what confound each scheme corrects for (length vs. prior probability) and whether that confound is present in each task. The "diff oracle" analysis (Table 3) demonstrates that task-level recommendations achieve near-optimal performance—within 0.0–0.6% of per-model oracle selection for most tasks—validating that the principled approach does not sacrifice empirical performance.

Research directions that become more attractive. OLMES makes several research directions newly tractable or newly credible:

• Evaluation protocol meta-analysis. Before OLMES, comparing performance numbers across papers required guesswork about which differences were genuine capability improvements and which were evaluation artifacts. With a documented standard, researchers can audit published results for protocol compliance and quantify how much of the "progress" in a given year is attributable to evaluation methodology changes rather than genuine model improvements. Figure 4 provides a preview: HELM's reproduction of MMLU scores found that self-reported scores systematically overestimate performance compared to a consistent reproduction, with the overestimation increasing for higher-performing models (R²=0.26). A systematic OLMES-based audit of the literature could quantify this gap across all major benchmarks.

• Training-time evaluation as a first-class research problem. Figure 1 demonstrates that the CF-MCF transition is a training-time phenomenon—models acquire MCF capability partway through pretraining. This makes evaluation during training not merely a monitoring task but a diagnostic tool for understanding capability emergence. Researchers can now ask: at what point in training does format understanding emerge? Does it correlate with specific data mixtures or architectural choices? Is the transition sudden or gradual? OLMES provides the standardized instrumentation needed to study these questions systematically across training runs.

• Verifier and evaluator robustness. The paper's demonstration that small prompt variations produce score changes typically under 1 percentage point (Table 5) establishes OLMES as a stable measurement baseline. This enables adversarial evaluation research: how do OLMES scores degrade under deliberately confusing prompt formats, label reordering, or distractor insertion? With a stable baseline, the degradation can be attributed to model brittleness rather than measurement noise.

Research directions that become less attractive. OLMES also implicitly deprecates certain practices:

• Ad-hoc, per-paper evaluation protocol design. If OLMES is widely adopted, the practice of designing a custom evaluation setup for each model release becomes scientifically indefensible—it is analogous to designing a custom thermometer for each temperature measurement. The field would converge on OLMES (or an alternative documented standard) as the default, and deviations would need explicit justification.

• Over-optimizing for leaderboard performance through formatting tricks. The paper's documentation of how normalization choice, few-shot curation, and prompt formatting affect scores makes it harder to claim leaderboard gains that are artifacts of protocol choices rather than genuine capability improvements. If everyone uses the same documented standard, the remaining variation reflects model differences—not evaluation gaming.

• Treating "benchmark difficulty" as a property of the dataset alone. OLMES demonstrates that difficulty is partly a property of the evaluation protocol: the same dataset can be "hard" under MCF for a weak model (near-random performance) but "easier" under CF (above-random performance). This complicates the common claim that "Benchmark X is saturated" or "Benchmark Y is too hard"—saturation and difficulty depend on how the benchmark is administered, not just on the questions it contains.

Follow-Up Research This Work Enables

Independent cross-team reproduction study—the gold standard for OLMES's core claim. OLMES's central promise is reproducibility: anyone following the published protocol should obtain the same scores. The paper demonstrates robustness to minor prompt perturbations (Table 5) but does not provide the definitive evidence: 2-3 independent teams, using different hardware, different software stacks, and different implementations of the protocol, evaluating the same 5-10 models on the same 10 tasks and confirming agreement within statistical bounds (standard errors of approximately 0.8-2.2%, as reported in Table 5). Such a study would transform OLMES from a well-documented proposal to a verified measurement standard. The specific test: do independently obtained scores for Llama2-7B on ARC-Challenge using OLMES agree to within, say, ±1.5 percentage points across three independent labs? Any systematic discrepancies would identify remaining undocumented degrees of freedom (e.g., floating-point non-determinism, batch size effects, tokenizer version differences) that need further specification.

CF-to-MCF calibration via human baseline—resolving the commensurability question. The paper identifies but does not resolve the problem that CF and MCF measure different capabilities on potentially incomparable scales. A direct experiment: administer a subset of OLMES tasks (e.g., 200 questions from ARC-Challenge and OpenBookQA) to human participants (e.g., 50-100 Mechanical Turk workers or university students) in both CF format (as sentence completion: "The answer is ___") and MCF format (as standard multiple-choice). Measure human accuracy in both formats. If humans show no CF-MCF gap (or a small, consistent gap), then the large CF-MCF gaps observed for strong LLMs (e.g., Llama3-70B's 24.7 percentage point gap on ARC-Challenge) represent a genuine model deficiency in format understanding—the model is underperforming relative to human capability. If humans show a substantial CF-MCF gap on the same tasks, then part of the gap is task-inherent and should be calibrated out when comparing across formulations. This experiment would directly inform whether OLMES's best-of-two rule produces scores that are comparable across the CF-MCF boundary or whether cross-formulation comparisons require a calibration adjustment.

Generalization to non-autoregressive and retrieval-augmented architectures. All 15 models used to develop OLMES are autoregressive transformer LMs trained on web text. The normalization win percentages, the CF-MCF transition point, and the optimal prompt formatting were all derived from this model class. A necessary stress test: evaluate OLMES on (a) an encoder-decoder model like T5 or UL2, (b) a retrieval-augmented model like RETRO or Atlas, (c) a model with non-standard tokenization (e.g., byte-level ByT5), and (d) a model trained primarily on non-English text (where tokenization and answer-choice prior probabilities may differ). For each, measure whether the OLMES normalization recommendations hold (does pmi still dominate on OpenBookQA? Does character still achieve 100% win rate on HellaSwag?) and whether the CF-MCF transition follows the same pattern. A negative result—e.g., pmi fails for retrieval-augmented models because retrieved context affects unconditional probabilities—would refine our understanding of when OLMES applies and motivate architecture-specific extensions.

Label and position bias quantification under OLMES MCF. The paper acknowledges but does not measure the magnitude of label bias (model preference for "B" over "A," etc.) and position bias (model preference for the first or last option) in the OLMES MCF format. Specific experiment: run OLMES MCF evaluation on all 15 models, but with three variations: (1) standard OLMES label order, (2) all answer choices cyclically permuted (A→B→C→D→A) to measure label bias, and (3) answer choices presented in reverse order to measure position bias. For each model, compute the bias-corrected MCF score (average across all permutations) and compare it to the standard OLMES MCF score. The research question: does label/position bias inflate OLMES scores by a meaningful amount, and if so, for which models? If the inflation is consistently under, say, 2 percentage points for models that select MCF over CF, the current practical decision to skip permutation debiasing is validated. If inflation exceeds 5 points for some models, OLMES should incorporate a bias correction or at minimum report bias diagnostics alongside scores.

Training-time emergence of evaluation-relevant capabilities. Figure 1 shows the CF-MCF transition during OLMo-7B-0424 training on MMLU, but this is a single model on a single task. A systematic study: instrument a large-scale training run (e.g., OLMo-7B or a comparable open model) to log OLMES scores at dense intervals (e.g., every 10B tokens) across all 10 tasks, separately tracking CF and MCF performance. Analyze when MCF overtakes CF for each task—does the transition point vary by task? Does it correlate with task properties (e.g., longer answer choices, more abstract content, higher reading level)? Does the transition occur simultaneously across tasks or sequentially (e.g., ARC-Easy before MMLU)? This would provide a capability emergence atlas that informs model developers about when during training different evaluation signals become reliable. It would also test whether OLMES's best-of-two rule is genuinely robust across the training trajectory or whether there are intermediate phases where neither CF nor MCF provides a clean signal (e.g., when both are noisy and the maximum is an overestimate).

OLMES-based evaluation protocol audit of the published literature. The paper documents that published ARC-Challenge and OpenBookQA scores vary wildly across references (Tables 1, 14). A systematic extension: for the 10 most-cited LLM papers of 2023-2024, attempt to reproduce their reported benchmark scores using OLMES. For each paper-model-benchmark triple, compute the gap between the published score and the OLMES-reproduced score. This would produce an estimate of how much of the field's perceived progress is attributable to genuine model improvement vs. evaluation protocol variation. As a concrete baseline: HELM's MMLU reproduction (Figure 4) found that self-reported scores overestimate reproduction scores with a positive slope (R²=0.26), meaning higher-reported models have larger gaps. An OLMES-based audit across 10+ benchmarks would quantify whether this pattern is MMLU-specific or general, and would provide an empirical basis for recommending that venues require OLMES or comparable standardized evaluation in submitted papers.

Practical Applications and Downstream Use Cases

Model development: training-time evaluation with consistent instrumentation. The paper's most immediately actionable contribution for practitioners is the ability to evaluate base models during training using a consistent, documented protocol that works across the full capability range. A team training a new 7B model from scratch can run OLMES evaluation at intervals (e.g., every 50B tokens) and track both CF and MCF scores across all 10 tasks. The CF curves provide clean signal early in training when the model cannot yet handle MCF (Figure 1: CF rises steadily while MCF stays at random for the first 200-400B tokens). The MCF curves take over when the model acquires format understanding. Because OLMES standardizes the prompts and normalization, comparing checkpoint 500B to checkpoint 200B is genuinely meaningful—differences reflect capability changes, not evaluation protocol drift. The practical benefit: earlier and more confident detection of training issues (plateauing, regression on specific capabilities) without the confounding of inconsistent evaluation setups across the training run. The specific protocol is: run OLMES on a fixed subset of 1000 instances per task at each evaluation checkpoint, tracking both CF and MCF accuracy separately, and use the maximum as the summary metric for model selection decisions.

Model selection: choosing base models for downstream fine-tuning with comparable numbers. Organizations deciding between multiple candidate base models for instruction-tuning or domain adaptation currently face the problem illustrated in Table 1: published numbers are not comparable across references. Using OLMES, a practitioner can evaluate all candidates (e.g., Llama2-7B, Mistral-7B, OLMo-7B, Falcon-7B) under the identical protocol and obtain numbers that genuinely reflect capability differences rather than evaluation variation. The practical workflow: download each candidate model, run the OLMES evaluation script (released at the paper's GitHub repository) on the 10-task suite, and compare scores directly—a 5-point gap on ARC-Challenge under OLMES means the same thing regardless of which model produced it, because the measurement instrument is identical. The computational cost is approximately 400 GPU-hours / 15 models ≈ 27 GPU-hours per model (based on the paper's total compute usage), which is modest relative to the cost of downstream fine-tuning experiments. The specific benefit: avoid selecting a model that appears stronger in published results but actually underperforms under standardized evaluation, and vice versa.

Leaderboard and benchmark maintenance: OLMES as a reference implementation for evaluation frameworks. The paper explicitly notes that OLMES is "ready to be incorporated into evaluation code bases for broad usage" (Section 6) and cites the Eleuther LM Evaluation Harness and HELM as natural integration targets. The practical scenario: the Hugging Face Open LLM Leaderboard or HELM adopts OLMES as its default evaluation protocol for MCQA tasks, replacing the current underspecified defaults with the fully documented OLMES choices (curated 5-shot examples, per-task normalization, dual CF/MCF with best-of-two selection, leading-space label formatting). The benefit: every model submitted to the leaderboard is evaluated under identical conditions with publicly documented justification for every protocol choice, eliminating the "different setups for different models" problem that Tables 1 and 14 document. Model developers who previously ran their own evaluations with different setups would have a strong incentive to adopt OLMES in their own papers to ensure their published numbers match the leaderboard numbers. The specific numbers: Table 4 provides the reference OLMES scores for 15 widely-used models, establishing a baseline that new submissions can be compared against directly.

Standardized evaluation in model cards and documentation. The paper's emphasis on "the rationale behind each decision" (Section 1) makes OLMES a natural fit for model card documentation (e.g., the Llama3 model card already reports MMLU and ARC-Challenge scores). A model developer releasing a new model can include a table analogous to Table 4 with OLMES scores for all 10 tasks, with the † annotation indicating which tasks used MCF. This communicates transparently: readers know exactly what protocol produced the numbers, can reproduce them independently, and can compare them directly to any other model with published OLMES scores. The specific benefit: replacing the current practice of reporting "score on ARC-Challenge (25-shot)"—which, as the paper demonstrates, is underspecified to the point of being uninterpretable—with "score on ARC-Challenge (OLMES)," which is fully specified and reproducible. The computational cost for the model developer is a one-time evaluation run of approximately 27 GPU-hours (scaling linearly with model size), which is a minor fraction of the cost of training the model.

When to Prefer This Method

OLMES is not competing against an alternative standard—the paper's premise is that no fully documented, justified standard exists. Rather, OLMES provides a reference protocol that supersedes ad-hoc, undocumented evaluation practices. The decision is therefore not "OLMES vs. alternative standard X" but "OLMES vs. designing your own evaluation protocol from scratch." The paper explicitly positions OLMES as the default choice for MCQA evaluation of base models, and the documented justifications provide the argument for why deviating from it requires explicit justification.

The paper does articulate specific conditions under which OLMES's design choices are appropriate vs. when they would need modification:

• Use OLMES as-is when evaluating autoregressive transformer base models on MCQA tasks, particularly during training and for model comparison. The standard is validated on 15 models spanning 1B to 70B parameters (and extended to 40 models in Table 13), covering the capability range from near-random to near-ceiling performance. The dual CF/MCF evaluation automatically handles the formulation transition without requiring per-model configuration.

• OLMES is not directly applicable as-is when evaluating instruction-tuned or chat models, where prompt formatting conventions differ (system messages, chat templates, conversational framing). The paper scopes to base models explicitly (Section 2.3) and states that "for chat models how to split the prompt into messages" is future work (Section 6). Similarly, OLMES does not address generative tasks, chain-of-thought prompting, or coding benchmarks—the recommendations around normalization, few-shot curation, and dual formulation are specific to MCQA.

• The OLMES principles generalize even when the specific prompts do not. The paper argues in the FAQ (Appendix A) that "the fundamental principles of OLMES as introduced, generalize to any dataset of interest." The principles—standardize prompt formatting, curate few-shot examples with balanced label coverage, select normalization based on what confound is present, evaluate with the best-performing formulation—can be applied to new tasks by following the same empirical validation procedure: test candidate choices across a diverse model set, select the configuration that optimizes a principled criterion (not just empirical performance), and document the justification. The paper invites community contributions to extend OLMES to new tasks following these principles.

• Modify OLMES if evaluating models where the normalization recommendations may fail due to fundamentally different probability calibration (e.g., non-autoregressive models, energy-based models, or models with non-standard tokenization). In such cases, the principled framework still applies—determine what confound each normalization corrects for, assess whether that confound is present for the model-task pair, and select accordingly—but the specific per-task recommendations derived from the 15-model empirical validation may not transfer. A mini-replication of the normalization comparison (Table 3) on the new model class would be advisable before adopting the OLMES defaults.