ArXiv: 2211.09110

🎯 Pitch

HELM's massive multi-metric benchmark reveals that the most accurate models also display the worst social biases, challenging the assumption that improving accuracy alone yields safer AI. By standardizing evaluation across 30 models on 42 scenarios, it exposes critical trade-offs between capabilities and trustworthiness that previous fragmented benchmarks missed.

1. Executive Summary

This paper introduces HELM (Holistic Evaluation of Language Models), a benchmarking framework that evaluates 30 prominent language models across 42 scenarios spanning question answering, information retrieval, summarization, and toxicity detection, measuring not just accuracy but also calibration, robustness, fairness, bias, toxicity, and efficiency—achieving 87.5% coverage of all possible scenario-metric pairs. The core organizing mechanism is a top-down taxonomy of scenarios (structured as (task, domain, language) triples) and metrics (drawn from desiderata across major AI venues), which makes explicit what the benchmark covers and what it misses, coupled with a standardized adaptation procedure (5-shot prompting with fixed in-context examples across all models) that enables direct head-to-head comparison. The evaluation surfaces 25 empirical findings, including that instruction-tuned models dominate accuracy (text-davinci-002 wins >90% of head-to-head comparisons), that accuracy and fairness are strongly correlated across scenarios, and that the most accurate models on BBQ are precisely those exhibiting the most concerning social biases in ambiguous contexts—establishing that holistic multi-metric measurement reveals trade-offs and phenomena invisible to accuracy-only benchmarks, though the approach is limited to English and to black-box prompting as the sole adaptation method.

2. Context and Motivation

The Core Problem: Language Model Evaluation Is Fragmented and Narrow

The fundamental problem this paper tackles is that we don't really know how good language models are — not because they haven't been evaluated, but because existing evaluation practices are too fragmented and too narrow to support meaningful understanding. The paper identifies three specific failures in the status quo:

First, evaluation coverage is uneven and incomplete. The authors document a striking empirical finding: prior to HELM, the 30 language models they study had been evaluated on only 17.9% of the 16 core scenarios on average (Figure 4). Some models, like T5 (11B) and Anthropic-LM v4-s3 (52B), had zero datasets in common in their original publications, despite both being major language models studied by the community. Others, like J1-Grande v1 (17B), Cohere xlarge v20220609 (52.4B), and YaLM (100B), had no publicly reported evaluation results at all. This means the community lacks the ability to make apples-to-apples comparisons across models — you cannot say whether model A is better than model B on a given task if they've never been tested on the same scenarios under the same conditions.

Second, even when models are evaluated on nominally the same scenario, evaluation conditions are inconsistent. The paper notes that on HellaSwag, some prior work reports fine-tuned accuracies (e.g., T5) while others report prompting accuracies (e.g., davinci), even though these represent fundamentally different adaptation strategies with different computational costs and different assumptions about model access. As the paper puts it, even when works report results through few-shot prompting, "the exact details can vary, which in §8.2 we show leads to wild swings in accuracies (e.g. 30% to 80% for the same (model, scenario))." This means that apparent differences between models may actually reflect differences in evaluation protocol rather than genuine capability differences.

Third, and most critically, evaluation is overwhelmingly accuracy-centric, ignoring other societal desiderata. The paper observes that most benchmarks "primarily center accuracy and often relegate other desiderata to their own bespoke datasets." That is, if you want to know whether a model is fair, you test it on a fairness-specific dataset (like BBQ). If you want to know whether it generates toxic text, you test it on a toxicity-specific dataset (like RealToxicityPrompts). But you almost never see fairness or toxicity measured on the same question-answering or summarization tasks where the model is actually deployed. This separation makes it impossible to study tradeoffs: does improving accuracy on summarization come at the cost of increased toxic generation? Does a model that performs more accurately on question answering also exhibit larger performance disparities across demographic groups? Without measuring multiple metrics on the same scenarios, these questions cannot be answered.

Why This Problem Matters

The paper argues that this fragmentation has consequences beyond academic inconvenience. Language models are already deployed in consequential settings: commercial APIs (OpenAI playground), products (GitHub Copilot), and content moderation systems at major platforms. Several of the models in the study "are having direct social impact at present," yet the absence of a common evaluation standard means we lack systematic understanding of their capabilities, limitations, and risks.

The paper identifies several specific stakes:

For societal impact: Toxicity detection failures have been implicated in serious human rights violations (the Rohingya genocide in Myanmar, as cited in §3.7), where automated content moderation failed in part because of inadequate coverage of languages and cultural contexts. Without holistic evaluation that measures fairness and robustness alongside accuracy on the same toxicity detection scenarios, such failures are harder to anticipate and mitigate.

For scientific understanding: The rapid development of language models — from ELMo and BERT to 530B-parameter models in roughly four years — has outpaced the community's ability to characterize what these models can and cannot do. The paper argues that evaluation can "pinpoint specific phenomena to advance scientific understanding," but only if it is systematic and comprehensive rather than ad hoc and scattered.

For power dynamics and access: The paper notes that some of the most capable models are closed (Anthropic-LM v4-s3, TNLG v2) or limited-access (davinci, text-davinci-002), while open models (OPT, BLOOM, GPT-NeoX) are increasingly available. Without standardized evaluation, it is difficult to track the gap between open and closed models (§8.1, Figure 28), which the paper argues is "crucial for tracking the accessibility (or lack thereof) and ultimately the power dynamics associated with language models."

For resource allocation: Training state-of-the-art language models costs millions of dollars and substantial energy (the paper estimates 1,287 MWh for davinci and 1,703 MWh for TNLG v2). Without clear evidence about which models perform well on which desiderata under which conditions, it is impossible to make informed decisions about whether the additional cost is worthwhile.

Where Prior Approaches Fall Short

The paper situates its critique within the broader history of NLP benchmarking, tracing a lineage from single-task benchmarks (SQuAD, SNLI) to small collections (GLUE, SuperGLUE) to large collections (GPT-3 evaluation suite, EleutherAI LM Harness, BIG-Bench). While acknowledging that each generation of benchmarks expanded coverage, the paper identifies specific limitations that persist even in the most recent efforts:

1. No top-down design framework. Prior benchmarks are collections, not taxonomies. They select datasets based on availability, popularity, or precedent rather than through a systematic process that starts with a definition of the design space and then makes deliberate, justified choices about coverage. The paper contrasts this explicitly in Figure 2: prior work is a "collection of datasets, each with a standard task framing and canonical metric, usually accuracy," while HELM "take[s] a top-down approach of first explicitly stating what we want to evaluate (i.e. scenarios and metrics) by working through their underlying structure." Without such a taxonomy, it is impossible to know what is missing — a benchmark might cover 50 datasets but have enormous blind spots (e.g., no coverage of African American English, no measurement of calibration) that remain invisible because the space was never mapped.

2. Multi-metric measurement is absent from core scenarios. The paper distinguishes between "core scenarios" (user-facing tasks like question answering, summarization) where models are deployed, and "targeted evaluations" (specific probes like BBQ for bias, RealToxicityPrompts for toxicity) that isolate specific phenomena. Prior work almost entirely separates metrics across these categories: accuracy is measured on core scenarios, while fairness, bias, and toxicity are measured on their own isolated benchmarks (Figure 3). The paper argues this is a mistake because it "makes explicit potential trade-offs and helps to ensure these desiderata are not treated as second-class citizens to accuracy." A model might be highly accurate on summarization but generate horribly biased text — but if bias is only measured on BBQ and not on the summarization scenario itself, we will never know.

3. Standardization is absent. The paper's analysis of 33 prior language model evaluations (Appendix F, Table 12) reveals that across all 405 datasets used in major language modeling works, the overlap between different model evaluations is surprisingly sparse. Models are evaluated under different adaptation procedures (fine-tuning vs. prompting), different numbers of in-context examples, different prompt formats, and different decoding parameters — all of which the paper shows can produce "wild swings" in accuracy (§8.2). This means that the field has no common reference point: if you read that model A achieves 80% on HellaSwag and model B achieves 75%, you cannot conclude A is better, because those numbers might come from completely different evaluation protocols.

4. No explicit recognition of incompleteness. The paper makes a pointed observation about benchmarks as instruments of power: "Benchmarks set the agenda and orient progress." When a benchmark is presented as comprehensive without acknowledging its limitations, it can inadvertently define the field's priorities in ways that exclude important considerations. The paper argues that "holistic evaluation should provide a top-down taxonomy and make explicit all the major scenarios and metrics that are missing" — not as an afterthought but as a first-class design principle. Prior benchmarks do not systematically document what they omit (e.g., which English varieties, which languages, which metrics were considered but not implemented), making it harder for the community to identify and fill gaps.

How This Paper Positions Itself

The paper frames HELM not as "the definitive benchmark" but as a living artifact with explicit scope and explicit gaps, built through a two-step process that distinguishes aspiration from implementation:

Step 1 — Taxonomy: Define the full space of what could be evaluated. For scenarios, this means decomposing evaluation into (task, domain, language) triples, with tasks drawn from the ACL 2022 taxonomy of NLP subareas and domains further decomposed into what (genre), who (demographic group of speaker/subject), and when (time period). For metrics, this means enumerating all desiderata that appear across major AI venues (ACL, NeurIPS, FAccT, KDD, etc.) and classifying them by what is required to measure them (e.g., whether they require white-box access, knowledge of training data, or understanding of the broader social context — Table 3).

Step 2 — Selection: From this space, deliberately select a subset based on three principles: (i) coverage of the space, (ii) minimality of the selected set, and (iii) prioritization of user-facing tasks. Crucially, the paper makes these principles explicit: the choice to prioritize user-facing tasks is a judgment, and the gaps it creates in the taxonomy are openly acknowledged in §10 ("What is missing").

The paper positions its contribution as a framework for transparency rather than a leaderboard to climb. It rejects the traditional single-score ranking (like ImageNet accuracy or GLUE score) in favor of a score matrix — for each model, a matrix of scenarios × metrics — that makes trade-offs explicit. This is a deliberate philosophical stance: "While it is possible for model A to be strictly better on every metric for every scenario than model B (i.e. strict Pareto dominance), in almost all cases A is sometimes better and B is sometimes better when one is sufficiently holistic/expansive. To designate A or B as better requires making a judgment that (implicitly/explicitly) weighs the circumstances in which each is better than the other."

The paper also explicitly builds on and acknowledges its debt to prior work, including GLUE and SuperGLUE (for the idea of multi-task benchmarks), the EleutherAI LM Harness and BIG-Bench (for broad dataset coverage), and NL-Augmenter (for perturbation-based evaluation of robustness). What distinguishes HELM is not any individual dataset or metric, but the systematic design process (taxonomy → selection → standardized evaluation → explicit gap analysis) and the multi-metric, scenario-integrated measurement that makes it possible to study trade-offs across desiderata in realistic deployment contexts.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

This paper constructs a large-scale benchmarking framework rather than a single model or algorithm. The system is a standardized evaluation pipeline that takes any language model (accessed as a black-box text-in/text-out interface), adapts it to dozens of different tasks using a uniform few-shot prompting procedure, and then measures not just whether the model gets the right answer, but also how well-calibrated its confidence is, how robust it is to typos and perturbations, how fair it is across demographic groups, how biased or toxic its generated text is, and how computationally efficient it is to run.

3.2 Big-Picture Architecture (Diagram in Words)

The HELM framework organizes evaluation around four major abstractions:

  1. Scenarios (the "what to test") — each scenario is a (task, domain, language) triple that defines what we want the model to do. A scenario is operationalized as a collection of instances, each with an input string and a list of reference strings (possibly with metadata about which references are correct).
  2. Adaptation (the "how to test") — a procedure that transforms a raw language model (which only takes text and generates text) into a system that makes predictions on a specific scenario. HELM uses 5-shot prompting with fixed in-context examples, standardized prompt formatting, and specified decoding parameters.
  3. Metrics (the "how good is the result") — functions that take model completions (and their probabilities) and produce quantitative scores. HELM implements 7 categories: accuracy, calibration, robustness, fairness, bias/stereotypes, toxicity, and efficiency, and applies them uniformly across scenarios to generate a score matrix per model.
  4. Targeted Evaluations (the "deeper dives") — additional scenarios that isolate specific capabilities (language understanding, knowledge, reasoning) or risks (copyright memorization, disinformation generation, bias, toxicity) beyond the core user-facing tasks.

Information flows through these components in a fixed pipeline: a scenario defines instances → the adaptation procedure constructs prompts from those instances (using training examples and formatting rules) → the language model generates completions (with associated log probabilities) → metrics compute scores from those completions → results are stored and visualized in an interactive web interface.

3.3 Roadmap for the Deep Dive

  • First, the scenario taxonomy and selection (§3.1-3.8 in paper, Section 3.4 in this analysis): how HELM defines the full space of possible evaluation scenarios, how it decomposes them into tasks/domains/languages, and how it deliberately selects a subset to prioritize coverage, user-facing tasks, and feasibility. Understanding the taxonomy is essential because it is the mechanism that makes explicit what HELM covers and what it misses.

  • Second, the metric taxonomy and selection (§4.1-4.2): how HELM enumerates desiderata from across major AI venues, classifies them by what is required to measure them, and selects the subset that can be measured given black-box access and no assumptions about model internals. This connects directly to the paper's core argument that multi-metric measurement on the same scenarios is essential for understanding trade-offs.

  • Third, the core scenario instantiations (§3.3-3.8): the concrete datasets chosen for question answering, information retrieval, summarization, sentiment analysis, toxicity detection, and miscellaneous text classification, including the specific choices about data splits and pre-processing.

  • Fourth, the general metrics in detail (§4.3-4.9): how each metric is operationally defined and computed, including perturbation-based robustness and fairness, count-based bias measurement, PerspectiveAPI-based toxicity detection, and hardware-aware efficiency estimation.

  • Fifth, the targeted evaluations (§5.1-5.7): the additional 26 scenarios that probe language, knowledge, reasoning, memorization/copyright, disinformation, bias, and toxicity at a finer granularity than the core scenarios.

  • Sixth, the adaptation procedure (§7 and Appendix J): the exact mechanics of prompt construction, in-context example selection, decoding parameter choices, and the handling of edge cases like multiple-choice scenarios and truncation.

This order mirrors the paper's own structure and builds from abstract design (what are we measuring and why) to concrete implementation (how exactly do we measure it on these datasets with these prompts).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an evaluation framework paper whose core idea is that holistic evaluation of language models requires (1) a systematic taxonomy of what to evaluate, (2) multi-metric measurement on the same scenarios, and (3) standardized adaptation procedures that enable direct model comparison. The technical contribution is not a new algorithm but a design methodology plus a concrete implementation that instantiates that methodology across 42 scenarios, 7 metric categories, and 30 models.

Scenario Taxonomy and Selection

The scenario space is structured as (task, domain, language) triples.

The paper decomposes evaluation scenarios into three orthogonal dimensions:

  • Task: what the model should do (e.g., question answering, summarization, sentiment analysis).
  • Domain: what kind of text the model operates on, further decomposed into three sub-dimensions:
    • What (genre): the type of text — Wikipedia, social media, news, scientific papers, fiction.
    • Who (demographic group): who generated the text or who the text is about — e.g., Black/White speakers, men/women, children/elderly.
    • When (time period): when the text was created — e.g., 1980s, pre-Internet, present day.
  • Language: the specific language or language variety (e.g., Spanish, African American English, Indian English).

This decomposition is not presented as exhaustive or atomic ("Tasks, domains, and languages are not atomic or unambiguous constructs: they can be made coarser and finer"), but as "intuitive structure for the space of scenarios" that enables systematic reasoning about coverage.

Tasks are enumerated from the ACL 2022 taxonomy of NLP subareas.

Rather than inventing a task taxonomy from scratch, the paper leverages the track structure of a major NLP conference (ACL 2022), which "reflect[s] the 'relevant topics' of study in NLP at the time of writing." For each track, the authors expand it into "canonical tasks associated with that track" (Table 1). For example, the "Question Answering" track maps to "question answering and reading comprehension," while "Sentiment Analysis, Stylistic Analysis, and Argument Mining" maps to "sentiment analysis, style transfer, argument mining, stance detection, opinion mining, text simplification." The paper acknowledges subjectivity in this mapping but argues that starting from an existing community-validated taxonomy is better than ad hoc enumeration.

From this full set of tasks, HELM selects a subset by filtering for user-facing tasks — those that "confer much of the direct social impact of language models." This is an explicit prioritization choice, not a claim about importance. The selected tasks are: question answering, information retrieval, summarization, sentiment analysis, toxicity detection, and miscellaneous text classification (to "provide some coverage of the long tail of tasks").

Domains are selected for coverage during dataset choice rather than enumerated explicitly.

The paper notes that "it was more complicated to arrive at an explicit enumeration of domains compared to tasks" and instead focuses on domain coverage during the specific dataset selection process (detailed in §3.3-3.8). Similarly, while the paper acknowledges that language coverage is crucial, it restricts evaluation to English (with coverage of English varieties like African American English and multiple national dialects through targeted evaluations), stating that "we will focus on predominantly evaluating English-only models." The paper explicitly flags the exclusion of non-English languages as a limitation in §10.1.

The selection process is governed by three principles and a constraint.

The principles are: (i) coverage — "aim for coverage of tasks, domains, and languages each independently" (rather than coverage of all possible combinations, which would be combinatorially infeasible), (ii) minimality — prefer a smaller set of scenarios to avoid overwhelming users with results, and (iii) prioritization of user-facing tasks — as discussed above. The practical constraint is feasibility given limited engineering resources, computational budget, and API costs.

The paper emphasizes that this selection process is explicit and documented to make the benchmark's limitations transparent: "by stating this taxonomy, we can select systematically from this space, which makes explicit both our priorities in benchmark design and the limitations in the benchmark at present."

Metric Taxonomy and Selection

The metric space is enumerated from desiderata across major AI venues.

To compile a comprehensive list of what properties a useful AI system should have, the authors survey the calls for papers and lists of areas of study across major AI venues: ACL, EMNLP, NAACL (NLP), NeurIPS, ICML, ICLR (machine learning), AAAI (broad AI), SIGIR (information retrieval), FAccT (fairness and accountability), WWW, ICWSM (web and social media), WSDM, KDD (data mining), and COLT, UAI, AISTATS (theoretical/statistical ML). For each venue, they map listed areas to specific desiderata (Table 2). The union across all venues yields a list of 46 desiderata including accuracy, calibration, robustness, fairness, bias, toxicity, efficiency, privacy, security, interpretability, transparency, causality, and many others.

Desiderata are classified by what is required to measure them (Table 3).

The key insight is that not all desiderata can be measured given the constraints of HELM's evaluation setup. The paper classifies each desideratum based on the knowledge and access required:

  • Requires knowledge of how model was created: causality, environmental impact, linguistic plausibility, memory efficiency, participatory design, privacy, sample efficiency, training efficiency, theoretical guarantees. These are excluded because HELM treats models as black boxes.
  • Requires the model have specific structure: credibility/provenance, explainability. Excluded because HELM does not assume model architecture.
  • Requires more than black-box access: interpretability. Excluded.
  • Requires knowledge about the broader system: maintainability, reliability, security, transparency. Excluded.
  • Requires knowledge about the broader social context: accessibility, accountability, creativity, emotional intelligence, legality, morality, oversight, trustworthiness, user experience/interaction. Excluded.
  • Satisfies HELM's conditions (none of the above): accuracy, bias, fairness, inference efficiency, robustness, toxicity, uncertainty/calibration. These seven are selected for measurement.

To this list, the paper adds training efficiency and environmental impact because "their measurement relies on information that is partially available for some models (i.e. reported in associated papers)." It also addresses some forms of legality through the copyright memorization evaluation and some forms of credibility through the disinformation evaluation, both as targeted evaluations rather than general metrics.

The selection strategy is: take everything that can be measured given black-box access and no assumptions about model internals or context.

This is a pragmatic choice that enables dense multi-metric measurement (87.5% of all possible scenario × metric pairs are actually computed, Table 4) while also making explicit why other important desiderata are excluded — they require access or knowledge that HELM does not assume. The paper argues that foregrounding these exclusions is a feature, not a bug: it tells the community which metrics have been historically neglected because they are hard to measure, and where investment in measurement methodology is most needed.

Core Scenario Instantiations

Each core scenario is a specific dataset chosen to instantiate a (task, domain, language) triple within the broader taxonomy. The paper describes the problem setting, dataset, selection rationale, pre-processing steps, and data access for each scenario. I will summarize the key design choices rather than replicating every detail.

Question Answering (9 scenarios, §3.3).

The QA scenarios cover three axes of diversity: domain, question format, and required skills. NaturalQuestions (open-book and closed-book) covers web search queries with Wikipedia contexts, testing factual knowledge. NarrativeQA covers long-form stories (books and movie scripts) with free-form answers, testing comprehension of extended narrative. QuAC covers conversational QA with dialogue context, testing the ability to track information across a multi-turn interaction. HellaSwag tests commonsense inference using adversarial filtering to create hard distractors. OpenBookQA tests scientific commonsense through questions paired with a corpus of elementary science facts. TruthfulQA tests truthfulness through adversarially-generated questions that probe common human misconceptions. MMLU is a meta-benchmark of 57 constituent datasets spanning humanities, social sciences, and STEM, testing broad knowledge. BoolQ tests binary yes/no questions with passage context.

The selection rationale emphasizes domain coverage (web search queries, stories, conversational questions, books, movie scripts) and skill coverage (factual knowledge, commonsense reasoning, reading comprehension, truthfulness). Specific choices include using the dev set rather than the test set for NaturalQuestions (since the test set is not publicly labeled), randomly sampling one question per document for NarrativeQA (to avoid prohibitively expensive evaluation), and providing the long answer rather than the full Wikipedia page as context in NaturalQuestions open-book (due to context window constraints).

Information Retrieval (2 scenarios, §3.4).

The IR scenarios focus on passage ranking: given a query and a set of candidate passages, rank the passages by relevance. Both scenarios use the MS MARCO dataset with its collection of ~9M web passages: the regular track has sparse relevance judgments (~1 positive passage per query) while the TREC 2019 Deep Learning track has richer graded relevance assessments (0-3 scale with many passages annotated per query). The adaptation is unconventional for LMs: each query-passage pair is formatted as a binary classification prompt ("Does the passage answer the query?" → "Yes"/"No"), and passages are ranked by the model's probability of "Yes." This is "computationally intensive in its naive implementation" but serves as a proof of concept for using LMs for IR.

The paper reports both vanilla metrics (re-ranking BM25's top-30 passages) and boosted metrics (also scoring all passages with explicit relevance annotations, even if not in BM25's top-30). The boosted setting establishes "an intuitive upper-bound on model quality" by guaranteeing the relevant passages are scored.

Summarization (2 scenarios, §3.5).

CNN/DailyMail (largely extractive reference summaries) and XSUM (largely abstractive reference summaries) are selected to cover both ends of the extractive-abstractive spectrum. Both are news domain. The paper pays special attention to faithfulness (whether generated summaries are factually consistent with the source document), measuring it using SummaC and QAFactEval as reference-free evaluation metrics. The adaptation uses 5-shot prompting with training articles between 50-150 tokens (to fit within context windows), and truncates input articles at 512 tokens following prior work.

Sentiment Analysis (1 scenario, §3.6).

IMDB is selected from many available sentiment datasets specifically because it has a contrast set (Gardner et al., 2020), which enables measuring robustness to semantics-altering perturbations (equivariance). The paper acknowledges that "sentiment analysis is more diverse and can be more complex" and that coverage of a single dataset is limited.

Toxicity Detection (1 scenario, §3.7).

CivilComments is selected over other toxicity detection datasets because it includes metadata about the demographic groups mentioned in the text (male, female, LGBTQ, Christian, Muslim, other religions, Black, White). This enables measuring performance disparities across demographic subgroups, which the paper argues is "especially important given the subjective nature of toxicity" and the "potential disparate impact of censoring the voices of the already-marginalized on Internet platforms."

Miscellaneous Text Classification (1 scenario, §3.8).

RAFT is a collection of 11 real-world text classification tasks (banking customer service queries, semiconductor organization classification, detection of overruling in legal statements, etc.) selected to represent the "long and growing tail of miscellaneous text classification tasks" that are not captured by standard NLP benchmarks but represent practical deployment scenarios. The full test set labels are private, so HELM uses a held-out subset of the public training set for evaluation.

General Metrics in Detail

Accuracy (§4.3).

Accuracy is the umbrella term for "the standard accuracy-like metric for each scenario," which varies by task: exact-match accuracy for text classification and multiple-choice QA, F1 score for word overlap in free-form QA, RR@10 for MS MARCO regular (reciprocal rank of first relevant passage), NDCG@10 for MS MARCO TREC (normalized discounted cumulative gain with graded relevance), ROUGE-2 for summarization (bigram overlap), and bits-per-byte for language modeling. The paper explicitly notes the implicit assumption that "accuracy is measured averaged over test instances" and that "minority subpopulations could experience low accuracy despite a high average accuracy."

Calibration and Uncertainty (§4.4).

Calibration is measured only for classification tasks. Two metrics are reported:

Expected Calibration Error (ECE). Given the model's top predicted probability $p_{\text{max}}$ for its predicted class, the ECE measures:

ECE=E[∣pmax−E[1y=y^∣pmax]∣]\text{ECE} = \mathbb{E}\left[ \left| p_{\text{max}} - \mathbb{E}[\mathbb{1}_{y = \hat{y}} \mid p_{\text{max}}] \right| \right]

where $p_{\text{max}} = \max_j p_j$ is the model's confidence in its predicted label $\hat{y}$, $\mathbb{1}_{y = \hat{y}}$ is the indicator that the prediction is correct, and $\mathbb{E}[y = \hat{y} \mid p_{\text{max}}]$ is the expected accuracy conditional on the predicted confidence.

What it computes: the expected absolute difference between the model's reported confidence and its actual probability of being correct. If the model predicts "toxic" with confidence 0.7 on 1000 examples, we expect roughly 700 to actually be toxic; deviation from this expectation is calibration error. The continuous expectation is binned into 10 equal-mass bins in practice, with the finite-sample estimator:

ECE^k=∑i=1mnin∣p^i−a^i∣\widehat{\text{ECE}}_k = \sum_{i=1}^{m} \frac{n_i}{n} \left| \hat{p}_i - \hat{a}_i \right|

where the data is sorted by $p_{\text{max}}$, partitioned into $m = 10$ equal-mass bins, $n_i$ is the number of examples in bin $i$, $\hat{p}_i$ is the average predicted probability in that bin, and $\hat{a}_i$ is the actual accuracy in that bin.

Why this form: ECE is the standard calibration metric in the literature (Naeini et al., 2015; Guo et al., 2017), directly interpretable as the average miscalibration in probability units. Equal-mass binning (Kumar et al., 2019) is chosen over equal-width binning because it ensures every bin has sufficient data for reliable estimation and doesn't overweight the extremes of the probability range.

Selective Classification. Two additional metrics capture whether the model can identify which examples are difficult:

Selective accuracy at 10% coverage (acc@10%): the accuracy on the 10% of examples for which the model is most confident, allowing it to abstain on the remaining 90%. Formally:

acc@C%=a(⌊nC/100⌋)\text{acc@}C\% = a\left(\lfloor nC/100 \rfloor\right)

where $a_r$ is the accuracy on the $r$ most confident examples after sorting in descending order of $p_{\text{max}}$.

Selective Coverage-Accuracy Area (SCAA): the average accuracy as coverage varies from 0% to 100%, equivalent to the area under the selective accuracy vs. coverage curve:

SCAA^=1n∑i=1nai\widehat{\text{SCAA}} = \frac{1}{n} \sum_{i=1}^{n} a_i

where $a_i$ is the accuracy on the $i$ most confident examples.

Why these metrics: they capture something different from calibration — "many models can accurately assess which examples are more difficult even if the raw probability values are incorrect." A model might have high ECE (poorly calibrated probabilities) but still correctly identify the hardest examples, enabling practical applications like routing uncertain cases to human review.

Robustness (§4.5).

Robustness is operationalized through perturbation-based evaluation: given a set of transformations $T_1, \ldots, T_k$ that map an instance $x$ to a perturbed version $T_j(x)$, robustness is the worst-case performance across transformations:

Robustness=1n∑i=1nmin⁡jm(model(Tj(xi)),yi)\text{Robustness} = \frac{1}{n} \sum_{i=1}^{n} \min_{j} m(\text{model}(T_j(x_i)), y_i)

where $m$ is the scenario-specific accuracy metric (exact match, F1, etc.) and the identity transformation is always included (so robustness is upper-bounded by standard accuracy).

Two types of perturbations are applied:

Invariance (semantics-preserving): These test whether the model's predictions are stable under benign corruptions that do not change the correct answer. The perturbation recipe is applied as a composition (not individually): (1) lowercase all text, (2) apply contraction perturbation (e.g., "it is" → "it's"), (3) misspell each word with probability 0.1 using common misspellings from NL-Augmenter, (4) replace each space with 1-3 space characters. The composition is used rather than individual perturbations to provide a scalable, single-pass robustness measurement rather than requiring separate evaluations for each perturbation type.

Equivariance (semantics-altering): These test whether the model changes its prediction when the input is altered in a way that should change the correct answer. This is measured using Contrast Sets (Gardner et al., 2020) — human-authored counterfactual modifications where the target label is known to change. This is only available for BoolQ and IMDB.

Why worst-case over perturbations? It captures the idea that a robust model should maintain performance even in the presence of the most challenging corruption, rather than just on average across perturbations. This is more conservative than reporting mean performance and aligns with the robustness literature (Madry et al., 2018).

Fairness (§4.6).

Fairness is operationalized through two complementary approaches:

Counterfactual fairness (perturbation-based): Similar to robustness, but perturbations correspond to social group membership — either the speaker (e.g., African American English vs. Standard American English via lexical substitutions from Ziems et al., 2022) or the subject of the text (e.g., gender pronouns and gendered terms, first names associated with different races/genders, last names associated with different races). The perturbations are applied deterministically whenever possible, which the paper acknowledges "should overestimate any performance disparities, which help to highlight fairness concerns (rather than underestimating these effects)."

The word lists for these perturbations are drawn from prior work: gender terms from Garg et al. (2018) and Bolukbasi et al. (2016), first names from Caliskan et al. (2017) (derived from Greenwald et al., 1998), last names from Garg et al. (2018) (derived from Chalabi & Flowers, 2017). The paper explicitly acknowledges limitations: the name-demographic associations are based on US Census statistics, which "may also not be invariant across time" and "reify a particular set of associations that are likely to change as a function of time."

Performance disparities (metadata-based): For scenarios where test instances are annotated with group-level metadata (e.g., CivilComments has demographic annotations for subjects mentioned in comments), HELM reports accuracy separately for each subgroup. The paper does not aggregate these into a single fairness score, arguing that "the appropriate choice of metric should depend on the context of the use case" and deferring aggregation to future work.

Why both approaches? Counterfactual methods are scalable (can be applied to any scenario) but may fail to capture fairness issues when "the data distributions across groups differ in more complex ways" than simple lexical substitution. Performance disparity methods are more ecologically valid but are only available when demographic metadata exists. Together they provide complementary signals.

Bias and Stereotypes (§4.7).

Unlike fairness (which is about task-specific accuracy across groups), bias is about properties of model-generated text, independent of task correctness. Two types are measured:

Demographic representation: The extent to which different demographic groups are mentioned at different rates. Using word lists $A_i$ for each group $i \in [k]$ (e.g., male/female for binary gender, White/Hispanic/Asian for race), the count for group $i$ is:

C(i)=∑w∈Ai∑y∈YC(w,y)C(i) = \sum_{w \in A_i} \sum_{y \in \mathcal{Y}} C(w, y)

where $\mathcal{Y}$ is the set of all model generations and $C(w, y)$ is the count of token $w$ in generation $y$. The observed distribution $P_{\text{obs}}$ is the normalized count vector, and demographic representation bias is the total variation distance from the uniform distribution:

demographic representation=TVD(Pobs,Pref)\text{demographic representation} = \text{TVD}(P_{\text{obs}}, P_{\text{ref}})

where $P_{\text{ref}}$ is uniform (all groups equally represented). For binary gender, the maximum possible bias is 0.5; for three-class race, it is 2/3.

Stereotypical associations: The same measurement, but conditioned on the presence of a specific term $t$ (e.g., "mathematician," "nurse"):

Ct(i)=∑w∈Ai∑y∈YC(w,y)⋅1[C(t,y)>0]C_t(i) = \sum_{w \in A_i} \sum_{y \in \mathcal{Y}} C(w, y) \cdot \mathbb{1}[C(t, y) > 0]

That is, only count demographic words in generations that also mention the target term $t$. The stereotypical association for term $t$ is the TVD between the observed co-occurrence distribution and uniform. The overall stereotypical association score is the average across a predefined list of professions $T$ (e.g., accountant, actor, nurse, mathematician, etc.).

Why this form: count-based measures are simple and transparent, and the paper explicitly provides the complete word lists (Appendix C.5) following the recommendation of Antoniak & Mimno (2021) that bias measurement studies should disclose their word lists for reproducibility. The uniform reference distribution encodes equal representation as the normative baseline. The paper acknowledges that "such count-based measures can be brittle in several ways" and that the validity of these measures has "largely not been verified."

Toxicity (§4.8).

Toxicity is measured using the Perspective API (Lees et al., 2022), which assigns a toxicity probability $p \in [0, 1]$ to generated text. A generation is classified as toxic if $p \geq 0.5$, and the toxicity metric is the fraction of generations that are toxic:

Toxicity=E[1[PerspectiveAPI(y)≥0.5]]\text{Toxicity} = \mathbb{E}[\mathbb{1}[\text{PerspectiveAPI}(y) \geq 0.5]]

where $y$ is a model generation. The paper deliberately chooses PerspectiveAPI over newer state-of-the-art toxicity detectors because it "has been strenuously analyzed and its limitations are well-acknowledged," preferring a known-quantity detector to a potentially better but unproven alternative. Since toxicity is measured on generated text, it is only reported for scenarios involving text generation (not classification-only scenarios).

Efficiency (§4.9).

Efficiency is measured across two dimensions:

Training efficiency: Energy cost (kWh) and CO2 emissions (kg). When reported by model creators, these numbers are used at face value. Otherwise, they are estimated using the formula:

e=nGPU⋅WGPU⋅ttrain⋅PUEe = n_{\text{GPU}} \cdot W_{\text{GPU}} \cdot t_{\text{train}} \cdot \text{PUE}

eCO2=e⋅cregione_{\text{CO2}} = e \cdot c_{\text{region}}

where $n_{\text{GPU}}$ is the number of GPUs, $W_{\text{GPU}}$ is the average power draw per GPU (in kW), $t_{\text{train}}$ is training time (in hours), PUE (Power Usage Effectiveness) is set to 1.1, and $c_{\text{region}}$ is the carbon intensity of the datacenter region (using U.S. national average when location is unknown). These are acknowledged as approximate but "of the right order of magnitude."

Inference efficiency: Two metrics are introduced to separate model-intrinsic efficiency from deployment-specific factors:

  • Denoised inference runtime: The runtime using the model provider's own hardware/software but factoring out noise from performance variation (contention, queuing). This represents what end users actually experience in the best case.
  • Idealized inference runtime: The runtime on uniform optimized hardware (NVIDIA A100 GPUs) and software (Megatron; Shoeybi et al., 2019), enabling direct comparison across models independent of deployment infrastructure.

For each model request, the total runtime is modeled as:

Total time=F(num_prompt_tokens)+g×num_output_tokens\text{Total time} = F(\text{num\_prompt\_tokens}) + g \times \text{num\_output\_tokens}

where $F$ is the runtime for encoding the prompt (which depends on prompt length) and $g$ is the per-token generation cost. The idealized runtimes are estimated for all models with publicly available architecture information; denoised runtimes are reported for all models.

Targeted Evaluations

The paper supplements the core scenarios with 7 targeted evaluations covering 26 additional scenarios that probe specific capabilities or risks at finer granularity.

Language (§5.1).

Language modeling scenarios: WikiText-103 (English Wikipedia), The Pile (5 corpora: arXiv, BookCorpus2, Enron Emails, PubMed Central, Wikipedia), TwitterAAE (tweets labeled as African American-aligned or White-aligned), and ICE (International Corpus of English, 10 national/regional varieties). Bits-per-byte (BPB) is used as the metric because it is "invariant to tokenization schemes" (Gao et al., 2021a). These scenarios enable measuring performance disparities across English varieties: the paper finds "a clear and consistent trend" that all models perform worse on African American English than White English on TwitterAAE, and that models perform differently across national varieties in ICE.

Minimal pairs (BLiMP): 67 paradigms × 1000 synthetic minimal pairs each, covering syntax, morphology, and semantics. For each pair of sequences differing by a single token (one acceptable, one unacceptable), the model is correct if it assigns higher probability to the acceptable sequence. This probes fine-grained linguistic knowledge: can the model distinguish "Craig explored that grocery store" (acceptable) from "Craig explored that grocery stores" (unacceptable)?

Knowledge (§5.2).

Knowledge-intensive QA: HellaSwag, OpenBookQA, TruthfulQA, and MMLU (already core scenarios) are re-examined through the lens of knowledge rather than general accuracy.

Fact completion (WikiFact): A new dataset constructed from Wikidata, covering 12 domains (art, law, philosophy, politics, geography, economics, math, computer science, physics, chemistry, biomedicine, and other general facts) with 86 relation types total. For each relation (e.g., "P36" = capital), a template converts a (subject, predicate, object) triple into a natural language completion: e.g., "The capital of France is __" → "Paris." For each relation, 1000 triples are sampled from the January 2022 Wikidata dump (removing entities without Wikipedia pages), split 100/50/850 train/dev/test. The metric is Accuracy@K (K=1,5), where any alias of a permissible object entity is considered correct. This design isolates specific factual knowledge from broader language understanding and reasoning.

Reasoning (§5.3).

Primitives: Synthetic reasoning tasks probe core reasoning capacities:

  • Pattern matching and variable substitution (abstract symbols): test whether the model can identify which pattern is exhibited in a string and apply substitution rules.
  • Deduction in natural language: conditional statements about attributes (e.g., "If a cow is weak, then the cow is small") followed by facts and a question about what can be deduced. Three difficulty levels: easy (exact term matching), medium (subject generalization, e.g., "animal" instead of "cow"), hard (attribute generalization, e.g., "cold" instead of "frigid").
  • Dyck languages (D3): the model must generate the closing brackets for partially-completed well-nested strings over three bracket pairs. This tests the ability to process hierarchical structure, which is believed essential for both natural language syntax and core reasoning.
  • bAbI: 20 tasks testing transitive inference, coreference resolution, logical reasoning (negation, conjunction, disjunction), spatial/temporal reasoning, deduction, and induction through short stories about characters moving objects between rooms.

Realistic reasoning: More complex, domain-specific scenarios that compose primitives:

  • Mathematical reasoning: GSM8K (grade-school math word problems) and MATH (competition mathematics with difficulty levels 1-5 and 7 topics). For MATH, a chain-of-thought variant includes intermediate solution steps.
  • Code synthesis: HumanEval (164 handwritten programming problems) and APPS (10,000 problems from coding competition sites at varying difficulties), evaluated with pass@k. Only code-specialized models (code-davinci-002, code-cushman-001) are evaluated.
  • Legal reasoning (LegalSupport): A new dataset mined from US court opinions. Given an argument and the legal conclusions of two cases, the task is to determine which case most persuasively supports the argument. The evaluation set is balanced (equal number of A and B as correct answers). Split 13,862/3,125/3,047 train/dev/test with average input 137 tokens.
  • Logical reasoning (LSAT): Analytical reasoning questions from the Law School Admission Test — verbal constraint satisfaction problems about assignments, groupings, or orderings. 1,630/231/230 split with average input 168 tokens.
  • Structured data reasoning: Entity matching (determining if two rows from different tables refer to the same entity, using Beer, Abt-Buy, and iTunes-Amazon datasets) and data imputation (filling in missing cell values, using Restaurant and Buy datasets). Data is serialized into natural language prompts following Narayan et al. (2022).

Memorization and Copyright (§5.4).

Three sources of copyrighted or licensed content: (1) 1,000 randomly sampled books from BooksCorpus (copyrighted), (2) 20 books from BooksCorpus that also appear on bestseller lists (copyrighted, likely repeated in training data), and (3) 2,000 randomly sampled functions from the Linux kernel source (GPL-licensed). For (1) and (2), prompts are varying numbers of tokens from the beginning of randomly sampled paragraphs (for bestsellers, only the first paragraph is used). For (3), prompts are varying numbers of lines from the top of each function. One completion per prompt is sampled (due to cost). Metrics include longest common subsequence (normalized by prefix length) for exact regurgitation, and edit distance/edit similarity (normalized) for near-exact reproduction. The paper emphasizes that results are "underpowered" because ideally many samples per prefix would be generated to approximate worst-case behavior, and that the evaluation only covers selected content types.

Disinformation (§5.5).

Two scenarios from Buchanan et al. (2021):

  • Narrative reiteration: Given a thesis statement (e.g., "COVID-19 is a man-made disease") and 5 supporting headlines, generate additional headlines that support the thesis. Theses were manually written based on clustering headlines from the Misinformation Reaction Frames dataset (Gabriel et al., 2022), covering COVID and climate change.
  • Narrative wedging: 11 prompts encouraging certain voting behavior (Democratic, Republican, or not vote) targeting religious groups (Christians, Muslims, Jews) or encouraging division (e.g., anti-Black racism). Models generate social media posts.

Evaluation is through human annotation (Amazon Mechanical Turk, 3 annotators per generation, $5.50/task). For reiteration, annotators rate quality (5-point scale: strongly support → contradict) and style (5-point scale: strongly agree → strongly disagree that it looks like a real headline). For wedging, annotators answer binary questions about whether the message addresses the intended audience, supports the goal, is divisive, and matches social media style, plus rate hostility (none/covert/overt). Diversity is measured automatically via self-BLEU and entropy. The paper explicitly acknowledges that annotators are not disinformation experts and that "this might overestimate the true utility of the model generations."

Bias (§5.6).

BBQ (Bias Benchmark for Question Answering; Parrish et al., 2022) is selected over prior bias datasets (StereoSet, CrowS-Pairs) because of documented validity concerns with those datasets (Blodgett et al., 2021). BBQ frames bias evaluation as multiple-choice QA with controlled structure: each example is a 2×2 pattern (negative/non-negative question × ambiguous/disambiguated context) with three answer choices (two referencing different social groups, one "Unknown"). Bias is measured separately for ambiguous and disambiguated contexts, tracking whether the model's errors skew toward or against attested social biases. The dataset covers nine demographic categories: age, disability status, gender, nationality, physical appearance, race/ethnicity, religion, socioeconomic status, and sexual orientation. Data is generated from templates with documented biases and vetted by crowdworkers.

Toxicity (§5.7).

Two generative toxicity scenarios complement the core CivilComments toxicity detection:

  • RealToxicityPrompts (Gehman et al., 2020): 100,000 prompts from OpenWebText, stratified into 4 toxicity bins based on PerspectiveAPI scores (25K each from [0, 0.25), [0.25, 0.50), [0.50, 0.75), [0.75, 1.00]). Models continue the prompts, and toxicity is measured on the completions. This tests whether models amplify existing prompt toxicity.
  • BOLD (Dhamala et al., 2021): Prompts are the first 6-9 words of Wikipedia articles mentioning professions, genders, races, religions, or political ideologies — deliberately innocuous. This tests whether models generate toxic content even when the prompt is neutral.

Both use PerspectiveAPI for toxicity detection. The paper notes that RealToxicityPrompts stratifies by prompt toxicity (enabling analysis of how prompt properties affect generation) while BOLD stratifies by social category (enabling analysis of differential toxicity across groups).

Adaptation Procedure

Adaptation transforms a raw language model into a system that makes predictions on test instances using few-shot prompting.

The procedure is standardized across all models and all scenarios (with scenario-specific variations documented in Appendix J):

In-context example selection. Five training examples are included by default (reduced adaptively when examples don't fit in the context window). Examples are selected to ensure class coverage for classification tasks: iterate over possible classes in decreasing frequency order, randomly sampling one representative per class. Crucially, the same training examples are used for all test instances, reflecting true few-shot conditions (Perez et al., 2021) where the model builder genuinely only has a handful of labeled examples. This is in contrast to prior work (e.g., Brown et al., 2020) that selects different examples for each test instance, which can artificially reduce variance. Each evaluation is repeated 3 times with different random seeds for in-context example selection to measure sensitivity.

Prompt construction. The prompt consists of (potentially): instructions (e.g., "The following are multiple choice questions (with answers) about anatomy."), followed by 5 in-context training examples (each with input, reference, and output), followed by the test instance (input and reference, with output prefix to prompt the model). The formatting is specified through parameters: instruction text, input prefix (e.g., "Question:"), reference prefix, output prefix (e.g., "Answer:"), instance prefix, and separators. The paper's philosophy is to treat prompts as user behavior rather than optimizing them: "we should strive for models that perform well on naturalistic prompts" rather than requiring model-specific incantations.

Decoding parameters. Temperature is set to 0 for most scenarios (argmax decoding for short, exact-answer tasks) and to non-zero values for longer-form generation (e.g., 0.3 for summarization, following prior work or empirical tuning). Stop conditions are primarily stop sequences (\n for most scenarios, to separate generated answer from subsequent text) rather than max tokens, to be tokenizer-agnostic. Max tokens is set as a fail-safe based on the maximum reference length. The number of outputs is 1 for most scenarios, 0 for language modeling (since the model "output" is a probability, not a generation).

Multiple choice adaptation. For scenarios with multiple answer choices, three methods are implemented (with a default chosen per scenario based on prior work or naturalness):

  • Joint: All answer choices are presented in a single prompt, and the model predicts the choice label (e.g., "A", "B"). Resembles a standard multiple-choice exam.
  • Separate: Each answer choice is scored independently as a continuation of the question, with the highest-probability choice selected. This is natural when choices complete the input (e.g., HellaSwag where each choice completes a story).
  • Separate-calibrated: Same as separate, but probabilities are calibrated by subtracting the log-probability of the answer choice alone (following Brown et al., 2020, to remove surface-form biases).

The choice of method can dramatically affect accuracy — the paper shows OPT (175B) varies from 30.2% (joint) to 79.1% (separate) on HellaSwag (§8.2, Figure 33) — and the optimal method can differ by model (e.g., on OpenBookQA, Anthropic-LM v4-s3 performs best with joint while other models perform best with separate-calibrated).

Truncation. For test instances longer than a model's context window, inputs are truncated (for non-language-modeling scenarios) rather than chunked (as in language modeling). For language modeling, documents are tokenized with the model's tokenizer and split into non-overlapping chunks of the model's window size, with maximum context from the previous chunk as conditioning tokens.

The priority system. Because the full benchmark is expensive to run (4,939 runs total, 38,001inAPIcosts),eachscenarioisassignedapriority‘38,001 in API costs), each scenario is assigned a priority `p_s \in {1, 2, 3, 4}$` where 1 is highest. All models are evaluated on all priority 1 and 2 scenarios. This enables the benchmark to "dynamically scale" — a user with a limited budget can evaluate only priority 1 scenarios and still get meaningful coverage. The specific priorities are documented in the codebase and reflect subjective judgments about scenario importance and coverage.

The Overall Evaluation Pipeline

Putting it all together, each evaluation run follows this sequence:

  1. Scenario instantiation: A scenario (e.g., BoolQ) provides a list of instances, each with an input string and reference strings (some marked correct). The scenario is divided into training and test splits.
  2. In-context example selection: From the training split, 5 examples are selected following class-balanced sampling (for classification) or random sampling (for generation). These are fixed for all test instances.
  3. Prompt construction: For each test instance, a prompt is built by concatenating instructions (if any), the 5 in-context training examples (formatted with input/output prefixes), and the test instance (with the output prefix prompting the model to generate).
  4. Model query: The prompt is sent to the language model with specified decoding parameters (temperature, stop sequence, max tokens). The model returns a completion string and (for most models) log probabilities of the prompt and completion.
  5. Metric computation: Given the completion and probabilities, all applicable metrics are computed: accuracy (comparing completion to correct references), calibration (if classification and probabilities available), robustness (if perturbations defined for the scenario), fairness (if perturbations or demographic metadata available), bias (if the scenario involves text generation), toxicity (if generation and PerspectiveAPI available), and efficiency (denoised and idealized runtimes). This produces a vector of scores for a single (model, scenario) pair.
  6. Aggregation and visualization: Results are stored and displayed in an interactive web interface that shows quantitative metrics, underlying model predictions, and exact prompts and completions. The complete set of results forms a matrix of scenarios × metrics for each model.

The paper emphasizes that this pipeline is designed to be extensible: new scenarios, metrics, and models can be added by implementing the corresponding abstractions in the open-source codebase. The taxonomy provides the scaffolding for deciding what to add next by making gaps explicit.

4. Key Insights and Innovations

Innovation 1: The Taxonomy-as-Transparency Framework — Making Evaluation Gaps First-Class Citizens

The paper's most distinctive conceptual contribution is not any specific dataset or metric, but the meta-methodology of benchmark construction itself: the two-step process of first taxonomizing the full evaluation design space (scenarios and metrics) and then deliberately selecting a subset, making the gap between aspiration and implementation an explicit, documented feature of the benchmark rather than a hidden limitation. This is a fundamental reframing of what a benchmark is.

Prior assumption: Benchmarks are defined by their contents — the datasets included, the metrics computed. A benchmark like GLUE or BIG-Bench is understood as "the set of tasks it contains." What is excluded is typically invisible, inferred only implicitly by researchers who notice their particular concern is absent. The field's default practice has been to present benchmarks as comprehensive collections without systematic acknowledgment of their blind spots.

What HELM does differently: The paper argues that "holistic evaluation should provide a top-down taxonomy and make explicit all the major scenarios and metrics that are missing." This shifts the benchmark's function from "leaderboard of what we measured" to "map of what we care about, with honest annotations about regions we haven't explored yet." The taxonomy over scenarios — structured as (task, domain, language) triples, with domains further decomposed into what (genre), who (demographic group), and when (time period) — and over metrics — 46 desiderata enumerated from calls for papers across 15 AI venues and classified by what is required to measure each (Table 3) — serves as a structured acknowledgement of incompleteness.

Why this matters beyond HELM: This framework makes the benchmark's values and priorities auditable. The choice to prioritize user-facing tasks over all possible NLP tasks is not presented as objective truth but as a deliberate, debatable judgment. The choice to restrict to English-only models is flagged explicitly in the taxonomy. The classification of metrics by required access makes transparent why certain desiderata (interpretability, privacy, causality) are excluded — because they demand more than black-box access — rather than implying they are unimportant. This enables the community to argue with the benchmark's priorities on explicit terms: "You prioritized X over Y; here's why Y matters more for my context."

The significance of this reframing extends beyond language models. It directly addresses a critique raised by Raji et al. (2021) that "the values encoded in machine learning research" are often implicit in benchmark design, orienting the field's priorities without accountability. By foregrounding the design decisions as an object of scrutiny, HELM converts evaluation from a technical exercise (pick good datasets) into a sociotechnical one (articulate what we value and why, then measure accordingly, then disclose the gaps).

The paper operationalizes this transparency concretely through §10 ("What is missing"), which enumerates specific prioritized gaps across scenarios, metrics, targeted evaluations, models, and adaptation methods — not as an afterthought but as a first-class section that connects the taxonomy (what we want to measure) to the implementation (what we did measure) to the future agenda (what the community should build next). This stands in contrast to the typical "limitations" section that acknowledges weaknesses briefly but does not provide an actionable research agenda.

Evidence anchoring: The taxonomy's practical value is demonstrated by the observation that 21 of HELM's 42 scenarios "have not been previously used in mainstream language model evaluation" (§1.1), and that prior to HELM, models were evaluated on only 17.9% of core scenarios on average (Figure 4). The taxonomy made it possible to identify those gaps systematically rather than anecdotally, and the paper's documentation of missing scenarios (§10.1) provides a concrete roadmap for future work.

Innovation 2: Multi-Metric Decoupling — Measuring Societal Desiderata on the Same Scenarios Where Models Are Deployed

The paper's second major conceptual contribution is the integration of multiple desiderata into core scenario evaluation, breaking the field's established practice of measuring accuracy on task benchmarks and relegating fairness, bias, toxicity, and robustness to their own isolated datasets. This is not an incremental change in what is measured — it is a fundamental shift in where and how societal considerations enter the evaluation process.

Prior assumption: The dominant evaluation paradigm in NLP (and AI broadly) separates metrics by scenario type. Accuracy is measured on GLUE, SuperGLUE, SQuAD, HellaSwag — the benchmarks that define "progress" in the field. Fairness is measured on BBQ or StereoSet. Toxicity is measured on RealToxicityPrompts. Robustness is measured through adversarial attacks or contrast sets. These are separate evaluations, conducted separately, reported in separate papers, and optimized for separately. As the paper puts it bluntly: prior work measures "Accuracy on NaturalQuestions. Robustness on AdversarialQA. Toxicity on RealToxicityPrompts. Bias on BBQ." (Figure 3). The implicit message is that these concerns are orthogonal — you get good at QA, and separately you make sure you're not toxic.

Why this separation is harmful: The paper identifies two consequences. First, it makes trade-offs invisible. If model A is 5% more accurate on summarization than model B but generates 3× more toxic text, you will never discover this if accuracy is measured on CNN/DailyMail and toxicity is measured on RealToxicityPrompts. Second, and more subtly, it relegates metrics beyond accuracy to "second-class citizens" (citing Friedman & Nissenbaum, 1996). When accuracy gets its own prominent benchmarks while fairness is confined to specialist venues (FAccT), the field's attention and resource allocation follows accordingly: researchers optimize for leaderboard accuracy, and fairness is an afterthought paper that gets written (or not) after the model is already built.

What HELM does: For each of the 16 core scenarios (the user-facing tasks like question answering, summarization, and toxicity detection where models are actually deployed), HELM measures 7 metric categories "to the extent possible" — achieving 87.5% coverage of the possible (scenario, metric) pairs (Table 4). This means you can answer questions like: Does the model that's most accurate on CivilComments toxicity detection also exhibit the largest performance disparities across racial groups on the same task? Does the model that generates the best ROUGE-2 scores on CNN/DailyMail also generate the most gender-biased text in those summaries? The paper surfaces specific examples of trade-offs that would be invisible in separate evaluation: for instance, "we see a clear and surprising trade-off: models that tend to have better fairness performance, which depends on task-specific performance, tend to have worse gender bias, which depends on model generations but not task-specific performance" (§8.1, Figure 25).

Why this is methodologically challenging: The paper is candid about why prior work didn't do this. Many metrics require specific data annotations or perturbation methods that are not trivially available for all scenarios. Fairness measurement via demographic metadata requires knowing the demographics of data subjects — available for CivilComments and TwitterAAE but not for most datasets. The paper addresses this through scalable proxies: perturbation-based counterfactual fairness (swap gender pronouns, dialect terms, race-associated names) that can be applied to any scenario's text without requiring demographic annotations. These are acknowledged as imperfect ("should overestimate any performance disparities" because they are deterministic and crude) but enable broad coverage that would otherwise be impossible.

The multi-metric matrix also surfaces where measurement breaks down. The 12.5% of (scenario, metric) pairs that are not measured (Table 4) are mostly cases where the metric is not well-defined (e.g., bias and toxicity for classification scenarios that don't involve text generation, calibration for summarization where there's no natural probability interpretation). Documenting these gaps is itself valuable: it tells the community which combinations of concerns cannot currently be evaluated together and where new measurement methodology is needed.

Evidence anchoring: The paper's 25 top-level findings (§1.2) are only possible because of multi-metric measurement. Finding 4 (strong correlations between accuracy, robustness, and fairness) and Finding 7 (no strong accuracy-efficiency trade-off across model families) both require comparing metrics on the same scenarios. Finding 19 — that the three most accurate models on BBQ are precisely those that exhibit biases aligning with broader social biases in ambiguous contexts (Figure 40) — is a particularly striking example of a phenomenon that only emerges when accuracy and bias are measured together on the same benchmark rather than in separate studies.

Innovation 3: Standardization as a Fairness Intervention — Revealing Model Sensitivity to Evaluation Protocol

The paper's third major contribution is its demonstration that the choice of evaluation protocol is not a neutral implementation detail but a consequential design decision that can dramatically reorder model rankings. While individual prior works have shown that prompt format matters (Zhao et al., 2021) or that in-context example selection matters (Perez et al., 2021), HELM provides the first large-scale, systematic evidence that these sensitivities are pervasive, model-specific, and severe enough to undermine the validity of model comparisons that don't control for them — which is to say, most comparisons in the literature.

Prior assumption: When a paper reports "Model A achieves 80% on HellaSwag" and another reports "Model B achieves 75%," the natural interpretation is that Model A is more capable. The field has largely treated evaluation metric scores as properties of models, comparable across papers and studies. HELM challenges this by showing that the score is a property of the model × evaluation protocol interaction, not the model alone.

The empirical evidence is striking. The paper documents multiple forms of evaluation sensitivity:

Prompt format sensitivity. Simple changes to instruction phrasing, input/output prefixes, and suffixes produce "wild swings in accuracies" (§8.2). For one prompt variant, Anthropic-LM v4-s3 achieves 67.3% on NaturalQuestions (open-book), while BLOOM drops from ~60% to 8.5% on the same variant. This is not a minor tweak — it is a complete reversal of relative model performance depending on prompt formatting alone.

Multiple-choice adaptation sensitivity. The three methods for adapting multiple-choice scenarios (joint, separate, separate-calibrated) produce dramatically different accuracy estimates for the same model on the same scenario. OPT (175B) varies from 30.2% (joint, 5-shot) to 79.1% (separate, 0-shot) on HellaSwag (Figure 33). Even more alarmingly, the optimal method differs across models: on OpenBookQA, five of six models perform best with separate-calibrated, while Anthropic-LM v4-s3 performs best with joint — a 39% accuracy gap between models when compared using the "wrong" method for one of them.

In-context example sensitivity. All 30 models show significant sensitivity to the choice of in-context examples across all scenarios and metrics. On NaturalQuestions (open-book), davinci achieves F1 scores ranging from 0.376 to 0.636 across three seeds of in-context example selection — a 26-percentage-point span dictated entirely by which 5 examples happen to be chosen (Figure 31).

Why this is a fundamental insight, not an engineering inconvenience: The paper argues that these sensitivities expose a deep challenge for the project of standardized evaluation itself. If model A performs best with joint adaptation and model B performs best with separate adaptation, what does it mean to "standardize" — and, critically, is standardization actually fair? The paper acknowledges this tension explicitly: "This poses a fundamental challenge for what it means to standardize language model evaluation in a fair way across models" (Finding 23). You could standardize on joint (favoring Anthropic-LM) or separate-calibrated (favoring OPT), but either choice embeds a preference for one model's interface over another's. The paper's approach is to report all methods and make the sensitivity visible, rather than hiding it behind a single "default" choice.

This insight extends beyond prompting. The paper notes that models differ in their suitability for different adaptation methods more broadly: T0++ was designed for zero-shot prompting, T5 was designed for fine-tuning, and evaluating both under 5-shot prompting is arguably unfair to both in different ways. The paper recommends that "model developers explicitly declare how their models should be evaluated and what the scope is of their generality" — a shift from models as general-purpose artifacts to models as artifacts with intended use conditions that should be respected in evaluation.

Significance: This finding calls into question the validity of every leaderboard that reports single-number scores without documenting sensitivity to evaluation protocol. It implies that the field needs to move from point estimates of model capability to sensitivity ranges, and that fair model comparison requires either matching the evaluation protocol to each model's strengths (which destroys standardization) or reporting enough protocol variants to characterize the interaction (which multiplies evaluation cost). The paper does not resolve this tension but makes it inescapably visible.

Evidence anchoring: The prompting analysis experiments (§8.2, Figures 31, 32, 33) collectively demonstrate that evaluation protocol sensitivity is universal across models and scenarios, order-of-magnitude in its effect sizes, and model-specific in its optimal configuration. Finding 22 states this explicitly: "All models show significant sensitivity to the formatting of prompt, the particular choice of in-context examples, and the number of in-context examples."

Innovation 4: The Instruction-Tuning Signal — Identifying a Training Paradigm Shift Through Comprehensive Benchmarking

While instruction-tuning (Ouyang et al., 2022; Bai et al., 2022; Sanh et al., 2021; Wei et al., 2022a) was known to improve performance before HELM, the paper provides the first broad-coverage, multi-metric, standardized evidence that instruction-tuned models dominate not just on accuracy but across robustness and fairness as well — and that this advantage is decoupled from model scale in ways that challenge the field's assumption that bigger models are necessarily better.

Prior assumption: The scaling laws literature (Kaplan et al., 2020; Hoffmann et al., 2022) established that larger models trained on more data reliably achieve lower perplexity. The natural extrapolation, reinforced by emergent abilities findings (Wei et al., 2022b), was that scale is the primary driver of downstream capability — that the most capable models will be the largest models, all else equal.

What HELM reveals: The head-to-head win rate analysis (Figure 26) shows that text-davinci-002 (instruction-tuned, size undisclosed but likely 175B) wins >90% of accuracy comparisons across all core scenarios. Anthropic-LM v4-s3 (instruction-tuned, 52B) is in the top 3 for accuracy, robustness, and fairness. Meanwhile, TNLG v2 (530B) — the largest model by far at more than 10× the parameters of Anthropic-LM — is second in accuracy but not dominant, and several 100B+ models (BLOOM 176B, J1-Jumbo 178B, YaLM 100B) substantially underperform their scale class in accuracy win rate. The paper notes that "all models that win head-to-head model comparisons for accuracy at a rate well above chance (i.e. > 55%) are at least 50B parameters" (Finding 25), but within that tier, scale is poorly predictive of rank: the smallest models in the top 10 (Anthropic-LM at 52B, Cohere xlarge at 52.4B) are among the most accurate, while the largest model evaluated (TNLG v2 at 530B) is not the most accurate on most scenarios.

Why this is more than a performance result: This finding reframes the scaling conversation from "how big" to "how trained." The paper explicitly interprets the result as evidence that "instruction-tuning and the use of human feedback is an efficient and effective means for improving model accuracy" — that training procedure (instruction-tuning, RLHF) can substitute for model scale (~10× in parameters) in achieving downstream accuracy. This is a diagnostic insight about the relative importance of different dimensions of model development: it suggests that the field should invest at least as much in training methodology as in parameter scaling.

The finding extends beyond accuracy. Anthropic-LM v4-s3 is in the top 3 for robustness and fairness alongside text-davinci-002 and TNLG v2, suggesting that instruction-tuning provides broad benefits across desiderata. The paper does not disentangle whether this is because instruction-tuned models are genuinely more robust/fair or because their higher accuracy mechanically improves robustness and fairness metrics (since the paper finds strong accuracy-robustness and accuracy-fairness correlations across all models, Figure 25). But the practical implication is the same: instruction-tuned models dominate the Pareto frontier.

Limitations acknowledged: The paper notes that it does not evaluate all major models (PaLM, Gopher, Chinchilla are excluded due to access limitations — §10.4) and that scaling both parameters and data optimally (Chinchilla-style) might produce stronger large models than the parameter-only-scaled baselines in the study. The gap between open and non-open models (Figure 28) is presented as a snapshot that "could grow or shrink over time as new models are released," and the paper explicitly frames monitoring this gap as "crucial for tracking the accessibility (or lack thereof) and ultimately the power dynamics associated with language models."

Evidence anchoring: The finding is anchored in the meta-analysis of head-to-head comparisons (Figure 26), the accuracy-vs-access analysis (Figure 28), and the cumulative accuracy over time (Figure 27), which shows clear jumps in state-of-the-art coinciding with the releases of GPT-3 (2020) and Anthropic-LM v4-s3 (late 2021) — both representing paradigm shifts in training rather than pure scale increases. Finding 1 states: "text-davinci-002 performs best on our accuracy, robustness, and fairness metrics, with Anthropic-LM v4-s3 (52B) being in the top 3 for all 3 metrics (despite being more than 10× smaller in model scale compared to TNLG v2 (530B))."

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary benchmark comprises 42 scenarios total — 16 core scenarios (drawn from question answering, information retrieval, summarization, sentiment analysis, toxicity detection, and miscellaneous text classification) and 26 targeted evaluation scenarios (probing language, knowledge, reasoning, memorization/copyright, disinformation, bias, and toxicity). The core scenarios include 9 QA datasets (NaturalQuestions open/closed, NarrativeQA, QuAC, BoolQ, HellaSwag, OpenBookQA, TruthfulQA, MMLU), 2 information retrieval datasets (MS MARCO regular and TREC), 2 summarization datasets (CNN/DailyMail and XSUM), 1 sentiment analysis dataset (IMDB), 1 toxicity detection dataset (CivilComments), and 1 miscellaneous text classification meta-benchmark (RAFT, comprising 11 constituent tasks from real-world applications like banking customer service and legal overruling detection). All scenarios use their standard public train/dev/test splits, though HELM holds out a subset of the training data for evaluation when test labels are private (e.g., RAFT), and for NaturalQuestions uses the dev set exclusively due to unlabeled test data. The number of evaluation instances is capped at 1,000 per scenario per run for feasibility, with the specific instance counts varying by scenario (e.g., NarrativeQA randomly samples 1 question per document; MS MARCO regular evaluates on the dev set of 6,980 queries). For language modeling scenarios (WikiText-103, The Pile, TwitterAAE, ICE), instances are documents chunked into non-overlapping token windows according to each model's context size.

  • Base model(s). The evaluation covers 30 language models from 12 organizations, spanning 16 model families and varying dramatically in scale (410M to 530B parameters), training procedure, and access conditions. Models include: AI21 Labs' J1 family (J1-Large v1 at 7.5B, J1-Grande v1 at 17B, J1-Jumbo v1 at 178B); Anthropic's Anthropic-LM v4-s3 (52B, instruction-tuned with RLHF); BigScience's BLOOM (176B, multilingual) and T0++ (11B, encoder-decoder fine-tuned for zero-shot generalization); Cohere's xlarge v20220609 (52.4B), large v20220720 (13.1B), medium v20220720 (6.1B), and small v20220720 (410M); EleutherAI's GPT-J (6B) and GPT-NeoX (20B); Google's T5 (11B) and UL2 (20B); Meta's OPT (66B and 175B); Microsoft/NVIDIA's TNLG v2 (6.7B and 530B); OpenAI's davinci (175B), curie (6.7B), babbage (1.3B), ada (350M), and instruction-tuned text-davinci-002, text-curie-001, text-babbage-001, text-ada-001 (all undisclosed size), plus code-specialized code-davinci-002 and code-cushman-001 (12B); Tsinghua University's GLM (130B, bilingual English/Chinese); and Yandex's YaLM (100B, English/Russian). These models were chosen to represent the landscape of available language models as of mid-2022, deliberately spanning open (10 models with downloadable weights), limited-access (17 models available through commercial APIs), and closed (3 models provided by their creators specifically for this research) accessibility tiers. The selection was constrained entirely by access — models that could not be accessed (PaLM, Gopher, Chinchilla, LaMDA) were excluded and are flagged as missing in §10.4. The paper explicitly notes that this selection is "a snapshot of the current landscape" and that findings may not generalize to models released subsequently or evaluated under different adaptation procedures.

  • Metrics. The evaluation computes 7 categories of metrics per scenario to the extent possible, achieving 87.5% of the 112 possible (scenario, metric) pairs (Table 4). Accuracy is the scenario-specific standard metric: exact match for classification and multiple-choice QA, F1 for free-form QA, RR@10 for MS MARCO regular, NDCG@10 for MS MARCO TREC, ROUGE-2 for summarization, and bits-per-byte for language modeling. Calibration is measured via Expected Calibration Error with 10 equal-mass bins (ECE-10) and two selective classification metrics (accuracy at 10% coverage and selective coverage-accuracy area under the curve), applicable only to classification tasks where the model provides probabilities. Robustness is the worst-case accuracy across a composite perturbation of each instance: lowercase → contractions → 10% misspelling probability per word → extra spaces (1-3 randomly). For equivariance robustness, contrast sets from BoolQ and IMDB are used to test whether models change predictions appropriately under semantics-altering modifications. Fairness uses two complementary operationalizations: counterfactual fairness via deterministic perturbation (gender pronoun substitution, dialect term substitution for AAE ↔ SAE, race/gender name substitution, gender term substitution — see Appendix D.2 for complete word lists) and performance disparities computed separately per demographic subgroup when metadata is available (CivilComments, TwitterAAE, ICE). Bias (on generated text only) measures demographic representation (total variation distance between observed mention rates of social groups and the uniform distribution, computed over word lists for binary gender and three-class race) and stereotypical associations (same TVD computation but conditioned on co-occurrence with specific profession terms). Toxicity (on generated text only) is the fraction of completions classified as toxic by the PerspectiveAPI at threshold p ≥ 0.5. Efficiency captures training cost (estimated energy in kWh and CO₂ emissions when model creators provide training details, using the formula e = n_GPU × W_GPU × t_train × PUE with PUE=1.1) and inference cost through two metrics: denoised inference runtime (using model providers' own infrastructure with noise from contention factored out) and idealized inference runtime (estimated on uniform A100 GPUs with Megatron for models with public architectures).

  • Baselines. The paper does not compare against prior model performance since its goal is standardized multi-model comparison rather than beating a state-of-the-art. However, implicit baselines exist in several forms. For accuracy, the findings are contextualized against prior reported results on the same scenarios (documented for all model-scenario pairs in the extended Figure 4 matrix in Appendix F), though the paper emphasizes that different evaluation conditions make direct comparison difficult — prior work evaluated models under heterogeneous protocols (fine-tuning vs. prompting, different numbers of in-context examples, different prompt formats). For information retrieval, BM25 retrieval (RR@10 = 19.0% on MS MARCO regular, NDCG@10 = 50.6% on MS MARCO TREC) serves as a classical non-neural baseline. For toxicity detection on CivilComments, chance accuracy (50% for the binary task) provides a floor. For the targeted evaluations, several scenarios have established baselines from their original papers: for GSM8K, the original work reports fine-tuned GPT-3 at ~20% and chain-of-thought prompting at ~55% (though under different decoding parameters); for HumanEval, the original Codex paper reports pass@1 and pass@100; for LSAT, prior work reports ~20-30% for baseline LMs. The paper reports these contextually but does not treat them as primary comparisons since the adaptation procedure is not matched.

  • Generation budget / compute accounting. The evaluation treats each model query as the fundamental unit of cost, reporting total tokens processed (12,169,227,491 across all models), total queries (17,431,479), total API costs ($38,001 for commercial APIs), and total GPU hours (~19,500 for open models). Costs vary dramatically by scenario and model: information retrieval with the MS MARCO boosted setting requires one LM request per query-passage pair (potentially 30+ requests per query), making it the most expensive scenario; summarization with long in-context examples strains context windows; language modeling requires processing every token in a document with no output generation, yielding different cost profiles than generation tasks. The paper introduces a priority system (Appendix H) to manage costs: each scenario is assigned a priority from 1 (highest) to 4 (lowest), and all models are evaluated on all priority 1 and 2 scenarios. This is not a compute-matched comparison (models are not allocated equal FLOPs or tokens) but rather a fixed-workload evaluation where every model performs the same tasks under the same conditions, with cost differences across models (e.g., larger models process tokens more slowly and cost more per API call) documented as part of the efficiency metrics.

  • Cross-validation / statistical protocol. The paper evaluates each (model, scenario) pair with 3 different random seeds for in-context example selection, fixing the same examples across all test instances within a seed to reflect true few-shot conditions (where only a handful of labeled examples are available, as argued by Perez et al., 2021). The variance across seeds is reported to characterize sensitivity to example choice (Figure 31). For the human evaluations of disinformation (reiteration and wedging), each model generation is evaluated by 3 crowdworkers on Amazon Mechanical Turk, and "paired bootstrapped tests with 10,000 samples" are used for significance testing with p < 0.05. Beyond these measures, the paper does not employ formal cross-validation for model selection or hyperparameter tuning — the evaluation protocol is fixed and applied identically to all models, with no per-model optimization of prompts or decoding parameters. The paper acknowledges that this standardization may disadvantage some models (e.g., T0++ was designed for zero-shot rather than 5-shot prompting) and that per-model prompt optimization could yield substantially different results, but argues that interoperability — models working well under the same user-facing prompts — is a desirable property to evaluate.

Main Quantitative Results

The results are organized around four analytical lenses: (1) meta-analysis of inter-metric relationships across all core scenarios, (2) direct head-to-head model comparisons, (3) the relationship between accuracy and other variables (time, access, scale, perplexity), and (4) prompting sensitivity analysis. Each lens exploits the unique structure of the HELM score matrix (30 models × 16 core scenarios × 7 metrics) to answer questions that cannot be addressed by single-metric, single-scenario evaluations.

Meta-Analysis: How Metrics Relate to Each Other Across All Scenarios

Accuracy, robustness, and fairness are strongly and consistently correlated across all scenarios.

Figure 24 shows the relationship between accuracy (x-axis) and each of the other six metrics across all core scenarios, with each scenario represented by a distinct color. The correlation between accuracy and robustness is visually tight — models that are more accurate are almost uniformly more robust to invariance perturbations. The correlation between accuracy and fairness (counterfactual fairness via perturbations) is similarly strong. Figure 25 quantifies these relationships through Pearson correlations per scenario: the accuracy-robustness and accuracy-fairness correlations are concentrated near 1.0 (each small grey dot represents one scenario's correlation), with no scenarios showing a trade-off where higher accuracy comes at the cost of substantially lower robustness or fairness.

This finding is methodologically significant because it challenges the common assumption that accuracy, robustness, and fairness are competing objectives that must be traded off. As the paper notes: "we do not believe that our work should be interpreted as contradicting" prior work finding accuracy-fairness trade-offs (Zhang et al., 2019a; Raghunathan et al., 2020; Dutta et al., 2020), because "we measure fairness differently from these works, and the setting of few-shot prompting of language models is considerably different." The finding is specific to this operationalization (perturbation-based counterfactual fairness under few-shot prompting) and this set of models, and should not be interpreted as a universal claim.

Calibration trends are highly scenario-dependent.

Figure 24 reveals that the relationship between accuracy and calibration error (ECE-10) is not consistent across scenarios — it clusters by scenario rather than following a universal trend. The paper highlights a striking contrast: "for two commonsense-centric QA scenarios, we see accuracy and calibration are highly correlated in OpenBookQA (correlation of greater than 0.8) whereas accuracy and calibration error are highly correlated in HellaSwag (correlation greater than 0.85)." That is, on HellaSwag, more accurate models are less well-calibrated (higher calibration error), while on OpenBookQA, more accurate models are better calibrated. The most accurate models on HellaSwag — text-davinci-002 (81.5% accuracy, ECE-10 = 0.286) and Cohere xlarge v20220609 (81.1% accuracy, ECE-10 = 0.341) — both have calibration errors above 0.28, meaning they substantially overestimate their confidence. This scenario-dependence means calibration behavior cannot be predicted from accuracy alone and must be measured per-task.

Generative harms show surprising relationship to other metrics.

Figure 25 reveals that gender representation bias (lower is better) is positively correlated with fairness (higher is better) — meaning that "models that tend to have better fairness performance, which depends on task-specific performance, tend to have worse gender bias, which depends on model generations but not task-specific performance." This is a clear and surprising trade-off that only emerges from multi-metric measurement on the same scenarios. No scenario shows a negative correlation between fairness and gender bias, and the average Pearson correlation across scenarios is >0.5.

In contrast, toxicity shows near-zero correlation with accuracy and other metrics because "the toxicity rates themselves for toxicity are relatively constant and near zero, with the clear exception of NarrativeQA." Toxicity rates are below 1% for most models on most scenarios, making correlation analysis dominated by noise. Some scenarios show anti-correlation between gender representation bias and toxicity, suggesting that "efforts to reduce one may have side-effects for the other."

Efficiency shows weak, scenario-dependent correlations with other metrics.

Figure 24 shows that there is no strong overarching trade-off between accuracy and inference time across all models — the cloud of points is diffuse. This is because efficiency depends heavily on model architecture and hardware, not just model size. Figure 25 shows that the Pearson correlations between inference time and other metrics are weak and variable across scenarios (the grey dots are widely dispersed). Within model families, efficiency predictably degrades with scale, but across families, the relationship breaks down. The paper notes that this will be explored in more detail in companion work (Narayanan et al., Forthcoming).

Direct Model Comparisons: Who Wins Head-to-Head?

text-davinci-002 dominates accuracy, winning >90% of head-to-head comparisons across all core scenarios.

Figure 26 presents head-to-head win rates: for each model, the fraction of pairwise comparisons (against all other models, across all scenarios) where that model has the higher metric score. A score of 1.0 means the model is the best on every scenario for that metric; 0.5 means comparison is a coin flip. On accuracy, text-davinci-002 is clearly dominant with a win rate above 0.9. The second tier comprises TNLG v2 (530B), Anthropic-LM v4-s3 (52B), and OPT (175B), with win rates clustered around 0.65-0.75. The fact that Anthropic-LM v4-s3 (52B) is competitive with TNLG v2 (530B) despite being >10× smaller is one of the paper's headline findings — it suggests that instruction-tuning with human feedback can substitute for roughly an order of magnitude in parameter count.

A thresholding effect is visible: "all models that win head-to-head model comparisons for accuracy at a rate well above chance (i.e. > 55%) are at least 50B parameters" (Finding 25). Smaller models — regardless of training procedure — cannot consistently outperform larger ones. However, within the ≥50B tier, scale is poorly predictive: "two of the largest models (BLOOM (176B) and J1-Jumbo v1 (178B); both 100B+ parameters) are at the bottom of this tier, whereas Anthropic-LM v4-s3 (52B) and Cohere xlarge v20220609 (52.4B) (the two smallest models in the tier) are in the top half." YaLM (100B) is a notable outlier with a win rate below 25%, which the paper attributes to "significant training on Russian instead of English."

Within model families (e.g., OpenAI's davinci/curie/babbage/ada, Cohere's xlarge/large/medium/small), accuracy win rate is "perfectly monotonically correlated with model scale." This confirms that scaling within a consistent training recipe reliably improves accuracy, but the absolute level is determined primarily by training choices (data, objectives, instruction-tuning) rather than scale alone.

Robustness and fairness rankings mirror accuracy rankings with few exceptions.

The robustness and fairness subfigures in Figure 26 show broadly similar model ordering to accuracy, consistent with the strong correlations found in the meta-analysis. text-davinci-002, Anthropic-LM v4-s3, and TNLG v2 are the top three for robustness; text-davinci-002, Anthropic-LM v4-s3, and OPT (175B) for fairness. BLOOM (176B) is a notable relative gainer in robustness and fairness compared to its accuracy ranking, while GLM (130B) and OPT (175B) roughly interchange positions between accuracy and robustness (GLM is more robust relative to its accuracy; OPT is less).

Toxicity and bias rankings are completely different from accuracy rankings.

The bottom two subfigures of Figure 26 reveal that models that dominate accuracy do not necessarily dominate (or even perform well) on generative harm metrics. T0++ (11B) is the most toxic model in head-to-head comparisons but one of the three least gender-biased. davinci (175B) is one of the four most gender-biased but one of the less toxic models. The rankings for these metrics are essentially uncorrelated with accuracy, emphasizing that measuring only accuracy gives no signal about generative harms and that model selection based on accuracy alone may inadvertently select models with worse bias or toxicity profiles.

Accuracy as a Function of External Variables

The state-of-the-art in accuracy has improved monotonically over time, with clear jumps coinciding with paradigm shifts.

Figure 27 plots cumulative best accuracy over time for each core scenario, tracking the release of progressively more accurate models. The release of GPT-3 (davinci, 2020) clearly establishes a strong baseline across all scenarios, showing a "distinctive improvement" over T5. The next major jump occurs roughly 18 months later with Anthropic-LM v4-s3 (December 2021), the first instruction-tuned model in the study. text-davinci-002 further raises the ceiling. The progression is scenario-dependent: some scenarios (IMDB, BoolQ) saturate near 95% accuracy with multiple models clustering at the ceiling; others (NaturalQuestions closed-book, TruthfulQA) show more gradual improvement with larger gaps between the best and second-best models.

A persistent gap exists between open and non-open models, though it has narrowed with recent open releases.

Figure 28 stratifies per-scenario accuracy by model accessibility. The best limited-access model (text-davinci-002) is universally the best in its category. The best closed model (TNLG v2) is sometimes the best in its category but often surpassed by other closed models (Anthropic-LM v4-s3 on many scenarios). The best open model (OPT 175B generally, but BLOOM 176B and GLM 130B in some cases) trails by varying margins: within 5 percentage points on some scenarios (IMDB, HellaSwag) but with large gaps on knowledge-intensive tasks (MMLU, NaturalQuestions closed-book, TruthfulQA) and information retrieval. The paper frames this optimistically — "some reason for optimism regarding open science" — but notes that "we have yet to see models being open-sourced with significant use of human preferences and reinforcement learning from human feedback, which may further bridge this gap." The evaluation also doesn't include all non-open models (PaLM, Gopher), which would likely widen the gap further.

Model scale within a family predictably improves accuracy; across families, it is a poor predictor.

Figure 29 shows that when model scale alone (in parameters) is plotted against cumulative best accuracy, the relationship is "very chaotic" — there is no clean scaling law correlating parameters to downstream accuracy across model families. The jumps in accuracy at ~12B and ~50B parameters are attributable to specific models (T0++ and Anthropic-LM v4-s3 respectively) rather than being evidence of emergent capabilities at those scales — these models introduced qualitative changes in training procedure (multitask fine-tuning, RLHF) rather than merely being larger. The paper acknowledges that training FLOPs (rather than parameters) might show a cleaner relationship but that the data was "too sparse to attempt a similar analysis."

Perplexity on The Pile is a poor predictor of downstream accuracy across model families.

Figure 30 plots log bits-per-byte (BPB) on The Pile against accuracy across core scenarios. The relationship is "messy" with large scatter. This is partly confounded by the fact that some models were trained on The Pile (GPT-J, GPT-NeoX, OPT, BLOOM) while others were not, making the language modeling evaluation partially contaminated (train-test overlap) for those models and potentially unrepresentative of their true language modeling capability. The paper recommends that "future work to explore new intrinsic/upstream surrogate measures of performance that can be shown to reliably predict downstream results."

Prompting Sensitivity Analysis

All models are sensitive to in-context example choice, but the magnitude varies by scenario and model.

Figure 31 displays the accuracy range (maximum minus minimum) across 3 seeds of in-context example selection. The median range across scenarios is below 0.03 for 24 of 28 models — meaning that for most models, example choice affects accuracy by ≤3 percentage points on average. However, there are stark outliers: YaLM (100B), davinci (175B), curie (6.7B), and ada (350M) show substantially higher variance. Specific scenarios are universally high-variance: NaturalQuestions (open-book) has a median range of 0.173 across all models, with davinci varying from 0.376 to 0.636 F1 depending on which 5 examples are selected — a 26-percentage-point swing. This demonstrates that the paper's choice to use fixed in-context examples (rather than per-instance selection) can lead to high-variance estimates, particularly for knowledge-intensive tasks where example quality matters substantially for establishing the task framing.

The number of in-context examples matters, but zero-shot to one-shot is the most impactful transition.

Figure 32 varies the number of in-context examples (0, 1, 2, 4, 8, 16, capped by context window) for four scenarios and a subset of open models. "All models show clear improvement from n=0 to n=1, sometimes having 0% accuracy in the zero-shot setting." Beyond one example, benefits are inconsistent: the relationship is not monotonically increasing for most model-scenario pairs. The exception is OPT (175B), which shows perfectly monotonically increasing accuracy with more examples for NaturalQuestions (open-book), IMDB, and CivilComments (CNN/DailyMail is the exception where all models perform best at zero-shot, likely because "models may not effectively understand the appropriate length distribution and the poor reference summaries may comparatively mislead the model in the one-shot setting").

Prompt formatting changes dramatically reorder model rankings.

The prompt format ablation varies instruction text, input prefixes, output prefixes, and input suffixes for four scenarios on a subset of open models. The key finding is that "the best prompt formatting is not consistent across models": one variant achieves 67.3% accuracy for Anthropic-LM v4-s3 on NaturalQuestions (open-book) while causing BLOOM to drop from ~60% to 8.5%. Another variant — explicitly mentioning "I am an expert AI assistant who is here to help you with the following" — performs best for GLM across all four tested scenarios, suggesting that different models have learned different "interfaces" during training. This has profound implications for standardization: if prompt formatting that helps one model hurts another, what does it mean to "fairly" compare them?

Multiple-choice adaptation method sensitivity is extreme and model-dependent.

Figure 33 compares three adaptation methods (joint, separate, separate-calibrated) across seven multiple-choice scenarios and six models. The most dramatic finding is on HellaSwag, where all six models show the same ordering (separate > separate-calibrated > joint) — but with enormous effect sizes: OPT (175B) varies from 30.2% (joint, 5-shot) to 79.1% (separate, 0-shot). The ordering is intuitive (HellaSwag is designed as sentence completion, so separate scoring of continuations is natural) but the magnitude is not.

On OpenBookQA, TruthfulQA, and MMLU, five of six models prefer separate-calibrated > separate > joint. The exception is Anthropic-LM v4-s3, which consistently prefers joint — "the adaptation method that elicits the most accurate behavior from Anthropic-LM v4-s3 (52B) elicits the least accurate behavior from the other five models." On OpenBookQA, OPT (175B) achieves 58.6% with separate-calibrated while Anthropic-LM v4-s3 achieves 69.6% with joint — meaning that if both were compared using separate-calibrated, they would be within 3% (58.6% vs. 55.8%), but if compared using joint, they would differ by ~39% (30.2% vs. 69.6%). This is the paper's most striking demonstration that evaluation protocol choice can fundamentally determine which model appears "better."

Ablation Studies and Robustness Checks

The paper's primary ablations are the prompting sensitivity analyses described above (in-context example choice, number of examples, prompt format, and multiple choice method). These are not framed as ablations of a single method but as robustness checks on the evaluation protocol itself. Beyond these:

  • Metric coverage completeness: Table 4 documents which of the 112 possible (core scenario, metric) pairs are measured. The 12.5% that are not measured are mostly cases where the metric is undefined (e.g., calibration for summarization where no natural probability interpretation exists, bias and toxicity for classification-only scenarios with no text generation) or where measurement is judged unreliable (fairness and robustness perturbations for long-form generation in summarization). This is an important transparency measure — it makes explicit where the multi-metric measurement framework breaks down.

  • Oracle vs. predicted difficulty bins (for revision and search): While HELM does not use difficulty-adaptive allocation (that's the prior example paper), it does use a priority system (Appendix H) to manage evaluation cost. Priorities are set subjectively and not empirically validated — a limitation the paper does not explore.

  • Temperature sensitivity: Summarization scenarios use non-zero temperature (0.3) following prior work, while most other scenarios use temperature 0. The paper does not systematically ablate temperature choice, acknowledging this as a limitation.

  • PerspectiveAPI threshold: Toxicity is binarized at p ≥ 0.5 following standard practice. No sensitivity analysis is performed on this threshold, though the paper notes the extensive limitations of PerspectiveAPI and calls for "future work to explore how the qualitative conclusions drawn regarding toxicity depend on the specified toxicity detection mechanism."

  • Contamination documentation (Appendix G, Table 13): The paper documents all known evidence of train-test contamination for each model-scenario pair, distinguishing "strong" contamination (model extensively exposed to evaluation distribution, e.g., The Pile for models trained on The Pile) from "weak" contamination (model possibly exposed through data overlap, e.g., davinci on NaturalQuestions per Brown et al., 2020). The contamination evidence is incomplete — for many models, training data is undisclosed — and the paper explicitly notes that "we have a limited understanding on how contaminated models are, and to what extent this compromises the validity and legitimacy of our evaluation."

Critical Assessment

The experiments genuinely demonstrate the paper's central claim that multi-metric, standardized, broad-coverage evaluation reveals phenomena invisible to traditional accuracy-only, single-scenario benchmarks. The evidence for this is overwhelming: the strong accuracy-fairness correlation (which would be invisible if fairness were measured only on BBQ and accuracy only on QA), the surprising fairness-gender bias trade-off (which requires measuring both on the same scenarios), the scenario-dependence of calibration (which undermines claims about "well-calibrated models" based on any single benchmark), and the extreme sensitivity of model rankings to evaluation protocol choice (which calls into question every leaderboard that reports single-number scores without documenting protocol sensitivity). This is the paper's most important and most robust empirical contribution.

The specific 25 findings vary in the strength of their empirical support:

Finding 1 (instruction-tuning dominance) is well-supported by Figure 26 and the head-to-head analysis, but with two important caveats. First, the comparison is not controlled — text-davinci-002's size is undisclosed, and its training data, architecture, and compute budget are substantially different from other models. The "instruction-tuning" label bundles together RLHF, supervised fine-tuning on human demonstrations, and potentially other undisclosed techniques. The finding demonstrates that text-davinci-002 and Anthropic-LM v4-s3 outperform their peers, not that instruction-tuning per se is the causal factor. Second, PaLM and Gopher — large non-instruction-tuned models that might challenge this finding — were not evaluated due to access limitations (§10.4).

Finding 2 (open vs. non-open gap) is well-documented in Figure 28 but represents a snapshot that is intentionally not causal. The paper acknowledges that the gap "could grow or shrink over time as new models are released" and that the comparison is confounded by the exclusion of some non-open models (PaLM, Gopher) and the fact that open models have not yet incorporated RLHF. The finding is valuable as monitoring infrastructure rather than as a stable empirical claim about the world.

Finding 4 (accuracy-robustness-fairness correlations) is robust across the 16 core scenarios and 30 models as measured. The paper appropriately hedges this finding by noting it is specific to perturbation-based counterfactual fairness under few-shot prompting and should not be interpreted as contradicting prior work that finds accuracy-fairness trade-offs under different operationalizations (e.g., representation learning). A genuine limitation is that the perturbation-based fairness measures are crude — deterministic lexical substitutions cannot capture the full complexity of dialect variation (e.g., African American English differs from Standard American English in syntax, prosody, and pragmatics beyond the lexical substitutions used) or the fact that names are imperfect proxies for race and gender. The strong correlation may partly reflect that both accuracy and fairness are measuring the same underlying phenomenon (does the model understand the task) rather than genuinely decoupled properties.

Finding 6 (generative harms) correctly notes that toxicity rates are near zero for most core scenarios, but this finding is contingent on the specific prompts and scenarios tested. The targeted toxicity evaluations (RealToxicityPrompts, BOLD) demonstrate that toxicity rates rise dramatically when prompts are deliberately toxic or when evaluating different social contexts — undermining any simplistic interpretation of "models are not toxic." The paper appropriately uses the finding to motivate targeted evaluation rather than to claim safety.

Finding 19 (BBQ accuracy-bias relationship) is one of the most striking and concerning findings: the three most accurate models on BBQ (text-davinci-002 at 89.5%, T0++ at 48.4%, TNLG v2 at 44.9%) are also the only three models whose biases in ambiguous contexts "align with broader social biases/discrimination," while all other models show the opposite bias direction (Figure 40). This is a genuine trade-off that only emerges from multi-metric measurement. However, the interpretation is complicated by the enormous gap between text-davinci-002 and all other models — it is unclear whether the finding reflects a general relationship between accuracy and bias, or whether text-davinci-002's training procedure (instruction-tuning, RLHF) produced both high accuracy and concerning bias through mechanisms that don't generalize to other accurate models. Additionally, the BBQ dataset has known limitations (Blodgett et al., 2021 raised concerns about validity in the prior generation of bias benchmarks; Parrish et al. address some but not necessarily all of these for BBQ), and the binary "bias direction" metric collapses a complex construct into a single axis.

Finding 23 (multiple-choice adaptation sensitivity) is one of the paper's most important methodological contributions — the demonstration that model rankings can reverse depending on adaptation method choice, with effect sizes of 39+ percentage points. This finding is robust and was demonstrated across 6 models and 7 scenarios. However, the paper doesn't fully explore why models differ — whether it's due to tokenizer interactions, training data composition, or fundamental differences in how models process multiple-choice formats — leaving the finding more diagnostic (the field has a measurement problem) than explanatory.

The most significant weakness of the experimental design is the single adaptation procedure. All results are conditional on 5-shot prompting with fixed prompts, temperature 0 for most tasks, and no per-model optimization. The paper is explicit about this limitation (§11.1) and demonstrates through the prompting sensitivity analyses that results would change under different protocols. However, the paper does not evaluate any alternative adaptation paradigm — fine-tuning, parameter-efficient methods, chain-of-thought, or retrieval augmentation — all of which are standard in the literature and would likely reorder model rankings. A genuinely holistic evaluation would characterize models under multiple adaptation regimes, not just prompting. The paper's defense — that prompting reflects the "language models as interfaces" paradigm — is reasonable but limiting.

Another significant weakness is the English-only scope. All 42 scenarios are English, with coverage of English varieties (TwitterAAE, ICE) but no non-English languages. This is acknowledged as a major limitation (§10.1) and attributed to the English-only nature of most evaluated models (BLOOM, GLM, and YaLM are partially multilingual but evaluated only for English). The findings about accuracy, fairness, and bias may not generalize to other languages, particularly lower-resourced ones where model performance is known to be substantially worse.

The test set sizes are relatively modest: 1,000 evaluation instances per scenario per run (often fewer, e.g., 355 for NarrativeQA test, 500 for OpenBookQA), with 3 seeds for in-context example selection. For scenarios with inherent variance (the NaturalQuestions open-book case with 26-percentage-point seed variance), these sample sizes may be insufficient to reliably distinguish model rankings. The paper does not report confidence intervals on the accuracy estimates, making it difficult to assess which pairwise model comparisons are statistically significant.

The energy and carbon estimates are acknowledged to be approximate, relying on self-reported or estimated hardware configurations, power draws, and carbon intensities. For models where training details are undisclosed (all commercial APIs, many closed models), no estimates are provided. This means efficiency comparisons are incomplete and may not accurately reflect the true environmental cost of these models.

A missing experiment that would have substantially strengthened the paper is human baseline evaluation on the core scenarios, particularly for toxicity detection (where chance accuracy is 50%) and summarization (where ROUGE scores are acknowledged to "largely fail to discriminate differences we observed in model quality"). Without knowing how humans perform on these tasks under similar conditions (prompted, with the same in-context examples), it is difficult to calibrate whether 66.8% accuracy on CivilComments is impressive or concerning. The paper's human evaluations are restricted to the disinformation scenarios.

6. Limitations and Trade-offs

The Benchmark Is English-Only, Despite Acknowledging That Language Coverage Is Critical

The assumption or constraint. The paper explicitly restricts evaluation to English, stating in §3.1: "we will focus on predominantly evaluating English-only models (with a few exceptions like BLOOM (176B) that are clearly multilingual but we evaluate only for English)." The scenario taxonomy (Figure 8) explicitly includes a Language dimension — "English, Finnish, Chinese, Swahili, ..." — but this dimension is collapsed to a single value in the implemented benchmark. The paper justifies this with a pragmatic argument about the models under evaluation: "we are studying English language models." The targeted evaluations provide some coverage of English varieties (African American English via TwitterAAE, national/regional English varieties via ICE), but no non-English languages are included in any of the 42 scenarios.

The consequence. This scope restriction means that every empirical finding in the paper — the 25 top-level findings, the accuracy-robustness-fairness correlations, the instruction-tuning advantage, the accuracy-bias relationship on BBQ, and all model rankings — is conditioned on English. The paper cannot speak to whether these findings hold for other languages, and there are strong reasons to expect they might not: BLOOM (176B) was explicitly trained as a multilingual model, and its lower accuracy relative to its scale (Figure 26) is partially attributed to "its multilingual training contrasted with our English-specific evaluation." This means the evaluation is systematically biased against models designed for multilingual deployment. More critically, the fairness and bias measurements for African American English (TwitterAAE) reveal consistent and substantial performance disparities — all models perform worse on AAE than on White English (§8.4, TwitterAAE results) — which raises the question of what even larger disparities exist for languages with fewer resources and less representation in training data. The paper's taxonomy makes this gap explicit (the Language dimension remains almost entirely unexplored in the implementation) but cannot quantify what is lost.

What evidence exists in the paper. The TwitterAAE results provide direct evidence of English-internal performance disparities: OPT (175B), the most accurate model on this scenario, degrades from 1.506 BPB on White English to 2.114 BPB on AAE — a substantial gap (Finding 5). The ICE results show further variation across national English varieties, with models performing noticeably worse on Hong Kong and Indian English compared to East Africa and USA. The paper's own analysis of BLOOM's lower-than-expected accuracy for its scale class suggests that multilingual training (as opposed to English-only) penalizes English evaluation metrics. These within-English findings strongly suggest that cross-language evaluation would reveal substantially larger disparities, but the paper provides no data to characterize them.

Mitigation status. The paper acknowledges this as a priority gap in §10.1: "a clear area for improvement is coverage of other languages, as many have called for throughout the history of NLP." It identifies specific desiderata: coverage of "typologically diverse languages" and "cultural sensitivity of language model and language technology evaluation." However, no mitigation is implemented in the current benchmark, and the paper treats this as future work for the community rather than as an immediately actionable extension. The taxonomy + selection methodology is designed to make this gap visible and to provide scaffolding for adding language coverage, but the benchmark as evaluated provides no cross-language evidence. For a practitioner deploying language models in non-English contexts (or in multilingual contexts where users may code-switch or speak non-dominant varieties), the HELM results provide essentially no guidance.

All Results Are Conditional on a Single Adaptation Procedure (5-Shot Prompting), Which the Paper Demonstrates Is Brittle

The assumption or constraint. The paper standardizes all evaluation on 5-shot prompting with fixed prompts, fixed in-context examples, and temperature 0 for most tasks (§7, Appendix J). The authors are explicit about this choice: "we opted to choose relatively simple, generic prompts in order to orient the development of language models towards generic language interfaces that respond robustly to direct natural language, rather than requiring model-specific incantations." They acknowledge that "certainly stronger results could be obtained from more sophisticated prompting (e.g. chain-of-thoughts; Wei et al., 2022c), prompt decomposition (Wu et al., 2022; Press et al., 2022; Arora et al., 2022), and prompt-tuning (Lester et al., 2021; Li & Liang, 2021), potentially leading to qualitatively different findings."

The consequence. The paper's own prompting sensitivity analyses (§8.2) demonstrate that model rankings, effect sizes, and qualitative conclusions are highly sensitive to the exact choice of adaptation protocol. The extreme example: OPT (175B) achieves 79.1% accuracy on HellaSwag with separate (0-shot) adaptation and 30.2% with joint (5-shot) adaptation — a 49-percentage-point swing from two reasonable ways of presenting the same multiple-choice scenario (Figure 33). Similarly, prompt format changes cause BLOOM to drop from ~60% to 8.5% on NaturalQuestions when instructions are altered, while Anthropic-LM v4-s3 improves to 67.3% on the same variant. The in-context example selection causes davinci to vary from 37.6% to 63.6% F1 on NaturalQuestions (open-book) depending on which 5 examples are chosen — a 26-percentage-point swing across three seeds (Figure 31). All of these swings are larger than the differences between top-performing models on most scenarios, meaning that the benchmark's model rankings depend as much on the adaptation protocol as on genuine model capability. The paper's Finding 23 states this explicitly: "model performance is extremely sensitive to how multiple choice scenarios are adapted into prompts," and the optimal adaptation method "can differ (and produce qualitatively different results) across models (Figure 33). This poses a fundamental challenge for what it means to standardize language model evaluation in a fair way across models."

This is a deeper problem than "prompt engineering matters." It implies that the paper's core empirical claims — text-davinci-002 dominates, instruction-tuning helps, accuracy and fairness are correlated — may not hold under different adaptation choices. If Anthropic-LM v4-s3 is optimized for joint multiple-choice adaptation (where it excels) while other models are optimized for separate-calibrated (where they excel), then standardizing on joint biases the comparison toward Anthropic-LM. The paper cannot distinguish between a model that is genuinely more capable and a model whose training procedure made it better at the specific interface HELM chose to test. A model that was explicitly trained for zero-shot evaluation (T0++) or for fine-tuning (T5) is evaluated under 5-shot prompting, potentially misrepresenting its capabilities. The paper's choice to use the same prompt format across all models is a principled commitment to standardization as a value, but the sensitivity analyses demonstrate that this commitment comes at the cost of fairness — standardization inevitably advantages models whose training distribution more closely matches the chosen interface.

What evidence exists in the paper. All four ablation categories (§8.2) provide direct evidence: (1) in-context example sensitivity (Figure 31, showing up to ~26 percentage point variance across 3 seeds), (2) number of in-context examples (Figure 32, showing non-monotonic behavior and scenario-specific optimal shot counts), (3) prompt format sensitivity (showing model-specific optimal prompts with accuracy swings of 30+ percentage points), and (4) multiple-choice adaptation sensitivity (Figure 33, showing extreme sensitivity across methods with model-dependent optimal methods). These are not small effects — they are order-of-magnitude compared to the differences between models in the top tier.

Mitigation status. The paper partially mitigates through transparency — it measures and reports the sensitivity, making it a finding rather than a hidden confound. It also recommends that "model developers explicitly declare how their models should be evaluated and what the scope is of their generality" as a path toward fair comparison. However, the paper does not evaluate any alternative adaptation paradigm (fine-tuning, parameter-efficient methods, retrieval augmentation, chain-of-thought), does not characterize how findings might change under per-model prompt optimization (which would abandon standardization but might better reflect each model's true capabilities), and does not provide guidance on how practitioners should navigate the trade-off between standardized comparison (which yields unreliable rankings) and per-model optimization (which destroys comparability). The limitation is fundamental: the paper demonstrates that its own evaluation protocol is brittle but does not offer a solution, leaving users of the benchmark with rankings that they know are sensitive to choices they might have made differently.

The Difficulty Estimation Cost Is Unaccounted for and Potentially Dominates the Benefits in a Deployment Context

The assumption or constraint. The paper does not use difficulty-adaptive test-time allocation (unlike the reference example), but it does introduce a priority system (§7, Appendix H) to manage evaluation cost by assigning each scenario a priority $p_s \in \{1, 2, 3, 4\}$ and evaluating all models on priority 1-2 scenarios. The priority assignments are subjective and not empirically validated. More fundamentally, the paper's framework requires evaluating every model on every scenario to produce the score matrix, which entails enormous computational cost: 4,939 total runs, 12.2 billion tokens processed, 17.4 million queries, $38,001 in API costs, and ~19,500 GPU hours (§1.2). This cost is documented but not accounted for in the headline findings — the paper does not discuss how the cost of holistic evaluation scales with the number of models, scenarios, or metrics, nor whether the insights gained justify the expense. The cost of difficulty estimation is not at issue here (the paper doesn't do it), but rather the cost of holistic evaluation itself as a prerequisite for the transparency benefits the paper claims.

The consequence. For a practitioner deciding whether to adopt HELM-style holistic evaluation for their own models or deployment decisions, the practical question is: what is the return on investment for comprehensive multi-metric evaluation? The paper demonstrates that multi-metric measurement reveals trade-offs invisible to accuracy-only evaluation (the fairness-gender bias anti-correlation, the BBQ accuracy-bias relationship) and that standardized evaluation exposes protocol sensitivity. But it does not quantify whether these insights could have been obtained more cheaply — e.g., whether a smaller subset of scenarios would have revealed the same patterns, whether the priority system efficiently allocates budget, or whether the insights are stable enough that re-evaluation is unnecessary. The cost of replicating HELM-level evaluation for a new model (or re-evaluating existing models on new scenarios) may be prohibitive for all but the most well-resourced organizations, creating a barrier to entry that contradicts the paper's stated goal of democratizing model understanding.

The priority system partially addresses this by enabling budget-constrained evaluation (evaluate only priority 1 scenarios), but the paper does not validate that priority 1 scenarios are sufficient to recover the key findings. A user running only priority 1 scenarios would miss many of the targeted evaluations and would not be able to compute the full score matrix that enables the paper's meta-analyses.

What evidence exists in the paper. The total costs are reported in §1.2 and Table 5. The priority system is described in Appendix H as a mechanism for dynamic scaling. The cost breakdown by scenario and model is not analyzed — the paper does not discuss which scenarios dominate the budget, whether costs could be reduced by subsampling instances, or whether the 3-seed repetition for in-context example sensitivity is worth its cost relative to the information it provides. The paper's implicit argument is that the cost is justified by the insights (the 25 findings), but this is a post-hoc justification rather than a cost-benefit analysis.

Mitigation status. The paper partially mitigates through the priority system and the open-source codebase, which enable others to run subsets of HELM at lower cost. However, there is no empirical validation of the priority assignments (are priority 1 scenarios sufficient to distinguish model quality?), no analysis of how findings change as the evaluation budget is reduced, and no discussion of practical deployment of HELM as an ongoing evaluation framework (rather than a one-time snapshot). The paper treats the 30-model, 42-scenario evaluation as a fixed artifact rather than as a process that must be repeated as models and scenarios evolve, which limits its utility for practitioners who need to make decisions about their own models on their own tasks under their own constraints.

The Test Set Sizes and Seed Repetition Are Insufficient for Reliable Model Rankings, Given the Demonstrated Variance

The assumption or constraint. The paper evaluates each (model, scenario) pair on a maximum of 1,000 test instances (fewer for many scenarios — NarrativeQA randomly samples 1 question per document, RAFT uses a held-out subset of the public training set) with only 3 repetitions for in-context example selection. The paper does not report confidence intervals on accuracy estimates, does not perform formal significance testing for model comparisons (except for the disinformation human evaluations), and does not discuss the statistical power of its design for distinguishing models in the top tier.

The consequence. Given the demonstrated variance — davinci varying by 26 F1 points across 3 seeds on NaturalQuestions (open-book, Figure 31), multiple models showing 5-10 percentage point standard deviation on individual scenarios — the paper's point estimates of model accuracy are noisy, and model rankings based on those point estimates may not be statistically reliable. When the gap between the top models on a scenario is small (e.g., text-davinci-002 at 81.5% vs. Cohere xlarge at 81.1% on HellaSwag), a 1-2 percentage point variance across seeds could easily reverse the ordering. The paper's head-to-head win rate analysis (Figure 26) aggregates across all scenarios to produce more stable rankings, but win rates do not account for the uncertainty in the per-scenario estimates — a model's win rate could be inflated or deflated by seed noise on a few high-variance scenarios.

The problem is compounded by the fact that the paper uses fixed in-context examples (the same examples for all test instances within a seed), which Perez et al. (2021) showed can produce higher variance than per-instance selection because a single "bad" set of examples affects all predictions. The 3-seed repetition is intended to measure this variance, but with only 3 seeds, the variance estimate itself is noisy — the paper reports the range (max minus min) rather than a standard deviation or confidence interval, which is less informative and more sensitive to outliers.

What evidence exists in the paper. Figure 31 directly shows substantial seed variance, with the median range across scenarios below 0.03 for most models but with clear high-variance scenarios (NaturalQuestions open-book at 0.173 median range). The paper reports that results are "relatively consistent" for most model-scenario pairs but does not quantify what fraction of pairwise model comparisons are statistically significant. The human evaluations (§8.5) use bootstrap significance tests but the main quantitative results do not.

Mitigation status. The paper does not address this limitation. The 3-seed repetition and 1,000-instance cap are treated as practical constraints rather than methodological limitations to be analyzed. The paper acknowledges the general concern about statistical significance in §11.2: "we encourage future work to consider how to better address significance given the scale of this evaluation," but does not provide sensitivity analyses showing how rankings change with additional seeds or instances, nor does it discuss the minimum detectable effect size given its experimental design. For a benchmark that aims to enable direct head-to-head model comparison, the absence of statistical rigor in those comparisons is a significant gap.

The Paper Evaluates Models at a Single Snapshot in Time, Deliberately Excluding Temporal Dynamics That Matter for Deployment

The assumption or constraint. The paper evaluates all 30 models at a single point in time (mid-2022), treating each model as a static artifact. The authors acknowledge this in §6: "for both the private models and commercial APIs, we are evaluating live systems that may be regularly updated... The results we produce are specific to the model versions at the time we queried them." They note that it is "possible that models change over the duration of our evaluation, but we assume they do not change." The adaptation procedure uses only 5-shot prompting with no fine-tuning or continual learning, meaning the evaluation captures zero-shot and few-shot capabilities but not what models could achieve with task-specific adaptation or what they will achieve as they are updated.

The consequence. The benchmark's findings are time-bound in at least three ways. First, commercial API models are continuously updated — the "text-davinci-002" evaluated in mid-2022 may not be the same model available to practitioners in 2024, and the paper's results may not reflect current performance. Second, the landscape of available models evolves rapidly: the paper explicitly excludes models released near the publication date (Galactica, Flan-T5, Tk-Instruct, BLOOMZ) and cannot speak to capabilities that emerged after the evaluation window. Third, and most subtly, the paper cannot distinguish between genuine capability improvements and changes in model behavior that reflect training data or algorithmic updates — if text-davinci-002's toxicity rates change from one version to the next, there is no way to know whether this reflects improved safety training or simply a different snapshot of the training process.

This limitation is particularly consequential for the paper's central function as a transparency intervention. If models change without transparent versioning, the benchmark's results become stale — practitioners relying on HELM to make deployment decisions may be acting on outdated information. The paper's finding about the open-vs-closed model gap (Finding 2, Figure 28) is explicitly framed as a snapshot that "could grow or shrink over time as new models are released," but the paper does not propose mechanisms for keeping the evaluation current.

What evidence exists in the paper. The paper's own exclusion of recently-released models (§10.4: Galactica, Flan-T5, BLOOMZ) demonstrates that the snapshot is already incomplete. The paper documents model versions where available (e.g., Cohere model dates) and timestamps when API queries were made, but does not systematically analyze whether different query times produced different results (which would indicate API drift). The paper's own observation that some models "may be deprecated at some point after our evaluation" (§6) underscores the temporal fragility of the benchmark.

Mitigation status. The paper partially mitigates through the open-source codebase and the living benchmark framing — "HELM [is intended] to be a living benchmark for the community, continuously updated with new scenarios, metrics, and models." However, the paper provides no infrastructure or process for ongoing evaluation, no commitment to re-evaluate models on a schedule, and no mechanism for detecting when previously-evaluated models have changed. The model version documentation is uneven (some model providers disclose versions; others do not), and the paper's recommendation that future work "longitudinally monitor such changes to clarify their nature" (§6) puts the burden on the community rather than on model providers to maintain evaluability. For practitioners who need to make deployment decisions based on current model behavior, HELM provides a historical snapshot rather than a living reference — and the paper does not quantify how quickly that snapshot becomes unreliable.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new model, a new training algorithm, or even a new metric. Its contribution is methodological infrastructure: a framework for designing benchmarks that makes evaluation gaps explicit, measures multiple desiderata on the same scenarios, and standardizes adaptation across models. The effect is to convert the field's implicit evaluation practices into objects of explicit scrutiny, making visible the assumptions, gaps, and protocol sensitivities that prior benchmarks hid behind single-number leaderboards.

The shift is most precisely described as a reconceptualization of what a benchmark is. Prior to HELM, benchmarks were defined extensionally — by the list of datasets they contain (GLUE is its constituent tasks; BIG-Bench is its 200+ tasks). HELM redefines a benchmark as a two-level artifact: an abstract taxonomy (the full space of what could be evaluated, structured as scenarios and metrics) plus a concrete selection (the subset that was actually evaluated, with documented gaps). This reframing makes the benchmark's limitations a first-class feature rather than an embarrassing afterthought. The taxonomy is not window dressing — it is the mechanism that enables §10 ("What is missing") to function as a prioritized research agenda rather than a generic acknowledgment of incompleteness. Future benchmarks that adopt this structure invite the community to argue with their prioritization choices on explicit terms ("you chose to cover X but not Y") rather than on implicit ones ("why didn't you include my favorite dataset?"). This is an incremental but methodologically fundamental shift in benchmark design philosophy, not a paradigm shift in how models are built or evaluated.

The paper resolves a latent tension in NLP evaluation that had been growing for several years. The tension was between two contradictory impulses: (1) the recognition that language models are general-purpose systems requiring broad evaluation across many tasks and many desiderata, and (2) the practical reality that evaluation is expensive, leading to ad hoc coverage where some models were densely evaluated on some tasks and others were barely evaluated at all. The paper quantifies this fragmentation for the first time — models were evaluated on only 17.9% of core scenarios on average, major models shared zero datasets in common in their original publications, and some models had no public evaluation results at all (Figure 4). By demonstrating that this fragmentation is not just aesthetically unpleasing but substantively misleading (the prompting analysis shows that model rankings can reverse depending on protocol choices that prior work varied arbitrarily), the paper makes a case that the fragmentation is a genuine scientific obstacle, not just an organizational inconvenience. The resolution is the standardization + taxonomy framework, which doesn't eliminate protocol sensitivity but makes it measurable and reportable.

The paper also reconciles the separation between capability evaluation and harm evaluation that had become institutionalized in NLP. Prior work typically measured accuracy on task benchmarks (GLUE, SuperGLUE, SQuAD) and relegated fairness, bias, and toxicity to their own specialist datasets (BBQ, RealToxicityPrompts, StereoSet) — usually studied by different communities, published at different venues, and optimized separately. HELM demonstrates that measuring these metrics together on the same scenarios reveals relationships that are invisible in separated evaluation: the strong accuracy-fairness correlation (Finding 4), the surprising fairness-gender bias trade-off (§8.1), and the BBQ accuracy-bias relationship where the most accurate models are precisely those with the most concerning social biases (Finding 19, Figure 40). These findings would not be discoverable if accuracy and bias were measured on different benchmarks, studied by different communities, and reported in different papers. The implication is that harm evaluation should be integrated into capability evaluation as a matter of benchmark design, not treated as a separate specialty. This reframing makes research directions that focus on integrated evaluation more attractive and makes directions that treat fairness or toxicity as isolated properties (measured only on specialist datasets divorced from deployment contexts) less attractive by revealing their blind spots.

The paper also shifts the conversation around model comparison from point estimates to sensitivity ranges. The prompting analysis (§8.2) does not just demonstrate that evaluation protocol matters — it demonstrates that protocol sensitivity is universal (all models show it), large (effect sizes of 30-49 percentage points), and model-specific (the optimal protocol differs across models). This implies that every leaderboard that reports single-number scores without documenting protocol sensitivity is potentially misleading — either because the ranking would change under alternative protocols, or because the protocol choice systematically advantages some models over others (as the finding that Anthropic-LM v4-s3 prefers joint adaptation while five other tested models prefer separate-calibrated demonstrates). The paper does not resolve the tension between standardization (same protocol for all models) and fairness (protocol matched to each model's strengths), but it makes this tension inescapably visible. Future evaluation work that reports point estimates without protocol sensitivity analysis now bears a heavier burden of justification.

Finally, the paper makes pretraining-scale-only narratives less attractive as explanatory frameworks. Finding 25 shows that model scale within a family reliably predicts accuracy, but across families, it is a poor predictor — and the largest model evaluated (TNLG v2 at 530B) is not the most accurate model, while one of the smallest in the top tier (Anthropic-LM v4-s3 at 52B) is in the top 3 for accuracy, robustness, and fairness. This doesn't refute scaling laws (which predict perplexity, not downstream accuracy) but it undermines the simplistic extrapolation that "bigger = better" for deployment-relevant metrics. The paper's demonstration that training procedure (instruction-tuning, RLHF) can substitute for roughly an order of magnitude in parameter count redirects attention toward training methodology as a first-class research direction alongside scale.

Follow-Up Research This Work Enables

Longitudinal re-evaluation of the 30 models to quantify temporal drift in API behavior. The paper's most time-sensitive limitation is that it evaluates models at a single snapshot (§6, §11), while commercial APIs are continuously updated without transparent versioning. A direct follow-up would re-run the full HELM evaluation on the same 30 models (to the extent they remain accessible) at 6-month intervals for 2 years, measuring the change in accuracy, toxicity, bias, and calibration over time for each model. The key question is whether API model behavior drifts in unpredictable ways — e.g., does text-davinci-002's toxicity rate change between versions without disclosure? The paper documents timestamps when API queries were made and records model versions where available, providing a baseline. A follow-up would use the same prompts, same scenarios, and same metrics to isolate temporal change from protocol variance, testing whether the snapshot findings are stable enough to guide deployment decisions or whether they become stale within months. The paper's open-source codebase and fixed prompt templates make this replication straightforward.

Per-model optimal prompt benchmarking to quantify the gap between standardized and adapted evaluation. The paper's prompting analysis (§8.2) demonstrates that the optimal prompt format and multiple-choice adaptation method differ across models, but it does not characterize how much the model rankings would change if each model were evaluated under its own optimal protocol. A strong follow-up would, for a subset of models and scenarios, perform a systematic prompt search (varying instructions, input/output prefixes, number of in-context examples, and adaptation method) to find the per-model optimal configuration for each scenario, then compare the resulting model rankings against the standardized HELM rankings. The central question: does per-model optimization merely shift all models upward by a constant, preserving rankings, or does it substantively reorder them? The paper provides evidence that reordering is likely — the Anthropic-LM vs. OPT divergence on joint vs. separate-calibrated for OpenBookQA (Figure 33) — but hasn't quantified whether this is the exception or the rule across all scenarios and models. This follow-up would directly inform the unresolved tension in §8.2: whether standardization or per-model optimization is the better principle for fair model comparison.

Extending HELM's multi-metric framework to retrieval-augmented and chain-of-thought prompting to stress-test whether the core findings generalize beyond vanilla few-shot prompting. The paper's findings — the accuracy-fairness correlation, the calibration scenario-dependence, the instruction-tuning advantage — are all conditioned on 5-shot prompting with no retrieval and no intermediate reasoning steps. Chain-of-thought (Wei et al., 2022c) and retrieval augmentation (Lewis et al., 2020c; Borgeaud et al., 2022) represent qualitatively different adaptation strategies that are now standard in the literature, and the paper explicitly acknowledges that they were not evaluated (§10.5, §11.1). A follow-up would add 2-3 representative models (e.g., text-davinci-002, Anthropic-LM v4-s3 if accessible, and one open model like OPT-175B) evaluated on a subset of core scenarios (3-4 QA scenarios, 1 summarization, CivilComments) under both chain-of-thought prompting and retrieval-augmented prompting, measuring all 7 metric categories. The key negative result to probe: does chain-of-thought — which substantially improves accuracy on reasoning tasks — also narrow or widen performance disparities across demographic groups on those tasks? The paper's finding that accuracy and fairness are correlated under vanilla prompting does not guarantee that this correlation persists when the model is prompted to reason step-by-step; chain-of-thought could either reduce disparities (by helping all groups equally) or amplify them (if the model's reasoning chains reflect biased associations that manifest in the intermediate steps). The BBQ scenario with its 2×2 structure (ambiguous/disambiguated × negative/non-negative) provides a natural testbed for this question.

Evaluating whether the accuracy-bias relationship on BBQ (Finding 19) replicates across model families and training paradigms. Finding 19 — that the three most accurate models on BBQ are the only three whose ambiguous-context biases align with broader social biases — is one of the paper's most striking and concerning results. But it is based on only 30 models, and the three most accurate models (text-davinci-002, T0++, TNLG v2) represent different training paradigms (instruction-tuning, multitask fine-tuning, standard pretraining), making it impossible to attribute the effect to a specific cause. A follow-up would add models that vary along one axis while holding others constant: add PaLM and Gopher (if access can be obtained) to test whether large non-instruction-tuned models also show the effect; add Flan-T5 and BLOOMZ to test whether instruction-tuning per se causes the alignment; add a model trained with deliberate debiasing to test whether the effect can be decoupled. The BBQ scenario's structured design (bias measured separately for ambiguous and disambiguated contexts, across nine demographic categories) makes it possible to test whether the effect is domain-general or specific to certain bias categories. A negative result — e.g., that PaLM shows the opposite bias direction despite high accuracy — would refine the interpretation by showing that the effect is specific to instruction-tuning or to certain training data mixtures, not a necessary consequence of high capability.

Cost-benefit analysis of the priority system: how much evaluation budget is needed to recover the paper's core findings? The paper introduces a priority system (§7, Appendix H) to dynamically scale the evaluation budget, but does not validate whether priority 1 scenarios are sufficient to recover the main findings — the accuracy-robustness-fairness correlations, the instruction-tuning advantage, the open-vs-closed gap. A practical follow-up would compute the full HELM results at progressively reduced budgets (priority 1 only, then priorities 1-2, then 1-3, then all) and compare the resulting model rankings, inter-metric correlations, and qualitative findings at each budget level against the full benchmark. The key questions are: (1) What is the minimum set of scenarios and metrics needed to reliably identify the top-performing models? (2) Does the priority ordering (which is subjective) correlate with the information gain per dollar? (3) For a practitioner with a fixed compute budget, what is the optimal allocation across scenarios, metrics, and model families? This would convert HELM from a one-size-fits-all benchmark into a budget-aware tool with empirically-grounded recommendations about where to spend limited evaluation resources.

Extending the multi-metric framework to fine-tuned models to test whether the accuracy-fairness correlation persists under task-specific adaptation. The paper's core finding that accuracy, robustness, and fairness are strongly correlated (Finding 4) holds under 5-shot prompting, where models receive effectively no task-specific training. Fine-tuning on in-distribution data might break this correlation: a model that is fine-tuned on a specific dataset might become more accurate at the cost of becoming less robust (overfitting to surface patterns) or less fair (amplifying dataset-specific biases). A follow-up would fine-tune 3-4 open models (OPT-175B, BLOOM, GPT-NeoX) on 3-4 core scenarios (IMDB, CivilComments, BoolQ, NarrativeQA) and re-evaluate all 7 metrics, comparing the fine-tuned results against the few-shot results for the same model-scenario pairs. The paper's framework makes this extension natural — the scenario definitions, metrics, and perturbation-based fairness/robustness measurements are all reusable independent of the adaptation method. A negative result — that fine-tuning substantially weakens the accuracy-fairness correlation — would refine the paper's finding by showing that it is specific to few-shot prompting, not a general property of language models. A positive result — that the correlation holds — would strengthen the paper's claim that multi-metric measurement can be standardized across adaptation methods.

Practical Applications and Downstream Use Cases

Model selection for deployed language technologies. For an organization deploying a language model in production — say, a customer service chatbot, a content moderation system, or a document summarization pipeline — the choice of which model to use has direct consequences for accuracy, fairness, toxicity, and cost. HELM's score matrix enables deployment decisions that account for the specific task and the specific trade-offs that matter in that context. If the deployment is a toxicity detection system (where fairness across demographic groups is critical because misclassification can silence marginalized voices), the CivilComments results in §8.3 show that text-davinci-002 achieves the highest accuracy (66.8%) but drops to 46.3% in the presence of fairness perturbations — while OPT (175B) achieves chance-level accuracy (50.1%) but maintains more consistent (if low) performance across demographic subgroups. A practitioner who values fairness over raw accuracy would make a different model choice than one optimizing headline accuracy alone. The multi-metric matrix makes this trade-off visible; without it, practitioners would select text-davinci-002 as the "best" toxicity detector and potentially deploy a system with substantial fairness vulnerabilities. The paper's finding that generative harms (toxicity, bias) rankings are essentially uncorrelated with accuracy rankings (Figure 26) means that accuracy-only leaderboards provide no signal for these deployment-critical properties.

Budget-constrained evaluation for organizations without large compute resources. The priority system (§7, Appendix H) and the open-source codebase enable an organization with limited compute to run a subset of HELM on their own fine-tuned models or on candidate API models before deployment. A startup building a medical QA system could run only the knowledge-intensive QA scenarios (NaturalQuestions closed-book, MMLU, TruthfulQA, WikiFact) plus the targeted knowledge evaluation, measuring not just accuracy but also calibration (critical in medical contexts where uncertain predictions should trigger human review) and fairness (to ensure the system does not systematically underperform for certain demographic groups). The paper's finding that calibration behavior is highly scenario-dependent (correlating with accuracy on OpenBookQA but anti-correlating on HellaSwag) means that calibration cannot be assumed from accuracy — it must be measured on the specific deployment scenario. The priority system provides the scaffolding for this selective evaluation without requiring the full HELM budget. The paper's documentation of which scenario-metric pairs are computable (Table 4) tells the practitioner what is feasible to measure given their model's access level (e.g., calibration requires probabilities, which some models don't provide).

Monitoring the open-vs-closed model gap for policy and access decisions. The paper's finding that there is a persistent gap between open and non-open models on accuracy (Figure 28) — with the gap largest on knowledge-intensive tasks (MMLU, NaturalQuestions closed-book, TruthfulQA) and smallest on sentiment analysis and commonsense reasoning — provides a concrete baseline for tracking the accessibility of language model capabilities over time. For policymakers or funding agencies deciding whether to invest in open model development or to mandate transparency in model evaluation, the HELM framework provides the measurement infrastructure. Re-running HELM's core scenarios on newly released open models (as they are released) would quantify whether the gap is narrowing (as the paper speculates, noting that "we have yet to see models being open-sourced with significant use of human preferences and reinforcement learning from human feedback") or widening (as non-open models incorporate new training paradigms that remain proprietary). The paper's explicit documentation of which models are open, limited-access, and closed (§6, Table 5) provides the taxonomy for this monitoring. The fact that 21 of HELM's 42 scenarios were "not previously used in mainstream LM evaluation" (§1.1) underscores that prior benchmarks lacked the coverage to serve this monitoring function — the scenarios where the gap is largest (knowledge-intensive, information retrieval) were among the least evaluated in prior work.

Auditing deployed systems for fairness and robustness failures before they cause harm. The paper's perturbation-based fairness and robustness measurements are designed to be applied to any scenario without requiring demographic metadata (§4.5, §4.6). This means an organization deploying a summarization system can measure, before launch, whether the model's summaries systematically associate certain demographic terms with certain topics (stereotypical associations), whether the model generates toxic content at non-trivial rates, and whether accuracy degrades under common user input corruptions (typos, capitalization, contractions). The paper's finding that toxicity rates are near zero for most core scenarios but spike dramatically under targeted toxicity prompts (RealToxicityPrompts toxic split, where some models generate toxic content in >10% of completions) provides a pre-deployment warning: low toxicity on standard evaluation does not guarantee low toxicity in adversarial or edge-case deployment contexts. The contrast between core scenario toxicity (low, Figure 24) and targeted toxicity (high for toxic prompts, §8.4) is precisely the kind of actionable signal that a pre-deployment audit should surface. The perturbation recipes (Appendix D.1 for robustness, D.2 for fairness) are fully specified with word lists, transformation probabilities, and composite perturbation order, making them directly implementable by practitioners without requiring specialized fairness expertise.

When to Prefer This Method

The paper itself does not frame HELM as a "method" to be chosen over alternatives but rather as an evaluation framework that is complementary to existing benchmarks. It does not argue that HELM should replace GLUE, SuperGLUE, BIG-Bench, or the EleutherAI LM Harness — it explicitly builds on all of them (§9, §F) and integrates BIG-Bench scenarios into its codebase. The paper's contribution is the framework (taxonomy + multi-metric measurement + standardization), not a claim that its specific scenario/metric/model selection is universally optimal. The taxonomy is designed to make prioritization choices explicit and debatable, not to assert a single correct prioritization.

However, the paper's structure implies a clear decision rule for when the HELM approach — systematic taxonomy-driven design, multi-metric measurement on deployment-relevant scenarios, and standardized adaptation with documented protocol sensitivity — should be adopted over the traditional approach of assembling a collection of datasets with accuracy as the default metric:

  • Prefer the HELM approach when you need to surface trade-offs across desiderata that are invisible to single-metric evaluation, such as evaluating whether a model that is more accurate on question answering also generates more biased text in its answers, or whether a model that is less accurate overall is substantially more robust to typos and user input variation. The paper's core insight — that the rankings for accuracy, toxicity, and bias are essentially uncorrelated across models (Figure 26) — means that any evaluation that reports only accuracy is missing deployment-critical information about harms and robustness.

  • Prefer the HELM approach when you need to compare models that were developed under different training paradigms and have never been evaluated on the same scenarios under the same conditions. The paper's finding that prior to HELM, models were evaluated on only 17.9% of core scenarios on average (Figure 4) means that for most model pairs, no direct comparison exists. Standardized evaluation on a common set of scenarios is the only way to generate this comparison.

  • Prefer the traditional approach (collection + accuracy) when evaluation budget is severely constrained and the primary goal is a rough capability ranking rather than a fine-grained characterization of trade-offs. The HELM approach requires measuring multiple metrics per scenario and repeating with multiple seeds, multiplying cost compared to a single-metric evaluation. The paper's priority system partially mitigates this but has not been empirically validated for recovering core findings at reduced budgets.

  • Be cautious about interpreting any single-number model ranking from HELM as definitive, given the paper's own demonstration that rankings are sensitive to adaptation protocol choices (Figures 31, 33). The paper's value is in the matrix — enabling inspection of individual (scenario, metric) cells and analysis of trade-off patterns — not in producing a single "best model" designation. If your use case requires a simple top-5 leaderboard, HELM provides the data but the paper warns against simplistic aggregation.