HellaSwag: Can a Machine Really Finish Your Sentence?

ArXiv: 1905.07830

🎯 Pitch

BERT’s near-perfect score on SWAG made headlines for ‘finishing your sentence,’ but HellaSwag reveals these models are just surface learners: they crumble (below 48%) when faced with adversarial examples that are trivially easy for humans. The key is scaling examples into a ‘Goldilocks zone’ where machine-generated wrong answers look on-topic yet are comically absurd.

1. Executive Summary

This paper introduces HellaSwag, a new benchmark for commonsense natural language inference that is trivial for humans (>95% accuracy) yet profoundly difficult for state-of-the-art models (<48% accuracy). The authors construct the dataset via Adversarial Filtering (AF) — a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers — combined with a trifecta of a state-of-the-art generator (OpenAI GPT) generating candidate endings, a state-of-the-art discriminator (BERT-Large) filtering them, and high-quality source text from ActivityNet video captions and WikiHow articles. Scaling up the length and complexity of examples into a critical Goldilocks zone — roughly three sentences of context and two generated sentences — produces endings that are nonsensical to humans yet frequently misclassified by even the strongest transformers, with BERT-Large achieving only 47.3% overall accuracy. The paper establishes that pretrained models like BERT operate more as rapid surface learners that exploit dataset-specific distributional biases rather than performing robust commonsense reasoning — a finding crystallized by the observation that performance drops an additional ~5% on zero-shot categories and that training on SWAG transfers poorly to HellaSwag (34.6% accuracy) — demonstrating that the underlying task of commonsense NLI remains unsolved even when models are evaluated on examples drawn from the same distribution as their training data.

2. Context and Motivation

The Core Problem: When Does a Benchmark Stop Measuring What It Claims to Measure?

The paper addresses a deceptively subtle question: when BERT achieves near-human performance on SWAG (86.7% vs. 88% human), does that mean machines can actually perform commonsense reasoning? The answer, the paper argues, is no — and the gap between what the benchmark measures and what we think it measures is the central problem.

This is not merely a question about whether one more benchmark has been saturated. It gets at a fundamental tension in NLP evaluation: benchmarks are static snapshots, but the models they evaluate evolve rapidly. When a benchmark like SWAG is released, it genuinely tests a capability. But as models become more powerful, they find shortcuts — spurious correlations, stylistic artifacts, annotation biases — that allow them to "solve" the benchmark without acquiring the underlying competence. The problem is that we cannot distinguish between genuine reasoning and sophisticated pattern matching by looking at benchmark scores alone.

The paper frames this as a systemic issue for the field, not a one-off complaint about SWAG. If the community's standard for progress is "beat the previous state-of-the-art on benchmark X," and if benchmarks can be beaten through surface-level statistical learning rather than genuine understanding, then the entire cycle of benchmark-driven progress becomes unreliable. The paper's opening salvo — the New York Times headline "finally, a machine that can finish your sentence" — is presented as exactly the kind of premature conclusion this evaluation failure enables.

Why This Problem Matters: The Distinction Between "Solving a Dataset" and "Solving a Task"

The stakes here are both scientific and practical.

Scientifically, the question of whether models perform genuine commonsense reasoning or surface-level pattern matching speaks to the core capability we are trying to measure. Commonsense NLI requires connecting a narrative to a rich model of how the physical and social world works — understanding that a dog avoiding a bath is dry and will run away again after getting wet, that you don't stop at a red light for "at most two seconds," that you use a knife to sharpen a stone rather than vice versa. If a model can get these questions right without actually modeling world states, then our evaluation is fundamentally not measuring what it purports to measure. The paper phrases this question provocatively in the abstract:

"Does this mean that machines can perform human level commonsense inference?"

The answer the paper provides is a clear no, but the more important contribution is the methodology for reaching that answer and the framework for understanding why superficial success happens.

Practically, the distinction matters because models deployed in real-world settings will encounter novel situations — new activity categories, new how-to domains, new combinations of concepts — that were not represented in their training data. If a model's success on SWAG comes from memorizing dataset-specific stylistic patterns (as the paper shows it largely does), then that model will fail on out-of-distribution examples even when those examples require the exact same underlying reasoning capability. This is exactly what the paper demonstrates with zero-shot category evaluation, where performance drops ~5% on test categories the model never saw during fine-tuning, despite those categories being drawn from the same underlying domain and requiring the same type of reasoning.

The SWAG Backdrop: What Came Before and Why It Appeared Solved

To understand HellaSwag's motivation, we need to understand the trajectory of its predecessor, SWAG (Zellers et al., 2018).

SWAG's design. SWAG presented a four-way multiple choice task: given a context (a video caption), select the most plausible next event (the actual next caption). The wrong answers were generated by a language model and selected via Adversarial Filtering to be challenging for then-state-of-the-art discriminators. At the time of SWAG's release, this was genuinely difficult — the best model (ESIM+ELMo) reached only 59% accuracy, while humans scored 88%.

What happened next. The release of BERT (Devlin et al., 2018) changed everything. BERT-Large, fine-tuned on SWAG, achieved 86.7% accuracy — within 1.3 points of human performance. On its face, this looked like a solved problem. The New York Times headline captured the public perception: machines could now finish your sentences with human-level commonsense.

The paper's diagnosis: BERT did not learn commonsense reasoning. The authors conduct a forensic analysis in Section 3 that reveals the nature of BERT's "success" on SWAG:

1. BERT is a remarkably fast surface learner. Figure 3 shows that BERT-Large reaches 59% accuracy — surpassing the previous state-of-the-art ESIM+ELMo — using only 64 training examples. It needs roughly 16,000 examples to approach human performance, at which point it plateaus. This learning curve suggests BERT is rapidly picking up on dataset-specific patterns rather than slowly acquiring general reasoning capabilities.

2. Context removal has surprisingly little effect. When the context is completely removed and BERT sees only the four ending choices (the "Ending Only" condition in Figure 4), accuracy drops only from 86.7% to 74.8% — a decline of just 11.9 points. This means that nearly 75% of the discrimination between correct and incorrect answers can be done without reading the premise at all. If genuine commonsense reasoning were happening, the context would be essential — you cannot determine whether "she gets the dog wet, then it runs away again" is a plausible next event without knowing that the preceding context involves a woman chasing a dog to give it a bath. The fact that BERT can get most questions right without context implies it is learning something orthogonal to reasoning: distributional differences between human-written and machine-generated text.

3. Word order matters surprisingly little. In the "Shuffled" condition, where the words within each ending are randomly permuted but the context is provided, BERT still achieves 77.0% accuracy. This is a remarkable finding because BERT was never exposed to shuffled text during pretraining — it learned during fine-tuning to ignore word order to a degree that seems incompatible with genuine language understanding. The authors interpret this as evidence that BERT is performing "lexical reasoning" — matching words between context and endings — rather than compositional understanding.

4. Even with neither context nor structure, BERT outperforms ELMo's full performance. When context is removed and words are shuffled ("Shuffled+Ending Only"), BERT still achieves 60.4% — higher than ESIM+ELMo's 59% on the full task. This is the smoking gun: a model operating on scrambled, contextless text fragments outperforms the previous state-of-the-art operating on the full, well-formed task. The capability being measured is not commonsense reasoning; it is the detection of stylistic artifacts.

Where Prior Approaches Fell Short: The Adversarial Filtering Gap

The paper traces BERT's success on SWAG to a specific failure mode in SWAG's construction — one that illuminates a general principle about dataset creation.

The generator-discriminator arms race. SWAG's Adversarial Filtering used a two-layer LSTM as the generator of wrong answers and an ensemble of shallow discriminators (bag-of-words, shallow CNN, perplexity-based MLP) as the adversarial filter. This configuration was strong enough to fool ELMo-based models, but BERT-Large — with its deep bidirectional architecture and massive pretraining — could easily detect the distributional artifacts left by the shallow LSTM generator.

The paper demonstrates this directly in Figure 5 (left). When the authors re-run Adversarial Filtering using BERT-Large as the discriminator, they compare two generators: the original SWAG LSTM and a fine-tuned OpenAI GPT. With the LSTM generator, BERT's accuracy never drops below 75% — the generations are too stylistically distinct from human text for even iterative adversarial selection to close the gap. The AF process "converges" at 75% because no matter how many generations the filter rejects, the remaining ones are still detectably machine-written to BERT. With GPT as the generator, by contrast, BERT's accuracy drops below 30% — GPT's generations are realistic enough that the adversarial filter can successfully select ones that fool even BERT.

This reveals a fundamental principle: Adversarial Filtering is only as strong as its weakest component. If the generator produces text with detectable artifacts, no amount of discriminative filtering can remove those artifacts — they are present in every candidate. The filter can only select among the artifacts, not eliminate them. SWAG's vulnerability to BERT was not a failure of the AF concept but a failure to use a generator strong enough for the discriminator it would eventually face.

The distributional artifact problem in NLI datasets. The paper connects this finding to a broader literature on annotation artifacts in NLI. Gururangan et al. (2018) and Poliak et al. (2018) had shown that standard NLI datasets contain subtle class-conditional biases — for instance, certain words are strongly predictive of entailment or contradiction labels, independent of the premise. Models exploit these biases rather than learning to reason about textual relationships. SWAG's vulnerability to BERT is the same phenomenon in a different guise: the stylistic difference between LSTM-generated text and human-written captions constitutes an annotation artifact that BERT can exploit.

The paper's key insight is that Adversarial Filtering is the right idea but requires generators and discriminators that match the target model's sophistication. If the goal is to build a dataset that resists surface-level statistical exploitation, the dataset construction process must use tools that are at least as powerful as the models that will eventually be evaluated on it. The generators must produce text realistic enough that artifacts cannot be the primary signal, and the discriminators must be sensitive enough to detect and remove whatever subtle artifacts remain.

How This Paper Positions Itself: A New Paradigm for Benchmark Construction

The paper's positioning is ambitious: it argues not just for a new dataset, but for a new paradigm of co-evolving benchmarks that grow in difficulty alongside the models they evaluate.

The static benchmark problem. The traditional approach to NLP benchmarking is to create a dataset, release it, and track progress as models achieve higher scores. This works well when the dataset genuinely measures the target capability and when progress on the benchmark corresponds to progress on the capability. But the paper argues — with SWAG as its case study — that this correspondence breaks down once models become powerful enough to find shortcuts. At that point, the benchmark stops measuring the capability and starts measuring something else (stylistic artifact detection, in SWAG's case), and further "progress" on the benchmark is meaningless.

Adversarial Filtering as a principled solution. AF is not just a method for making one hard dataset — it is a meta-method for making datasets that are hard for whatever models exist at construction time. The key property is that AF creates a dataset that is adversarial for any arbitrary split of D_train and D_test (Section 2, Figure 2). This is crucial because it means the difficulty is not an artifact of domain shift between training and test distributions — the model fails even when trained and tested on data drawn from exactly the same distribution. This is what distinguishes HellaSwag from prior work on adversarial examples (Jia and Liang, 2017; Glockner et al., 2018; Belinkov and Bisk, 2018), which construct out-of-distribution test sets to break models. HellaSwag is adversarial in-distribution — the model sees plenty of training examples from the same distribution, yet still cannot learn to generalize.

The Goldilocks zone as a guiding principle. The paper's investigation of generation length (Figure 5, right) reveals a critical design parameter: there exists a "Goldilocks zone" of text complexity where generations are nonsensical to humans but difficult for models to distinguish from real text. One-sentence generations from WikiHow converge to near-random (35%) after AF, but are also short enough that they may not require deep reasoning. Three-sentence generations converge to 50% — too easy for models. Two-sentence generations sit in the sweet spot: long enough to contain detectable commonsense violations for humans, yet realistic enough that even BERT-Large cannot reliably separate them from human-written text. This is not a universal constant — it depends on the generator and discriminator used — but the concept generalizes: for any given state of model capability, there exists a level of task complexity where humans still excel but machines fail, and finding that level is the key to benchmark construction.

The co-evolution vision. The paper explicitly frames HellaSwag not as a final dataset but as one iteration in an ongoing cycle (Section 6.3):

"What happens when HellaSwag gets solved? We believe the answer is simple: crowdsource another dataset, with the same exact format, and see where models fail."

The vision is that benchmarks should be rebuilt using the best available models as adversaries each time the previous version is saturated. This creates a ratchet: each new dataset version incorporates the models that solved the previous version into its construction, ensuring that the new version is adversarial to those models. The cycle stops only when we reach a point where even the best models cannot be used to generate challenging-but-foolable wrong answers — which would imply that generation itself has reached human-level commonsense, at which point the underlying task might genuinely be considered solved.

The relationship to pretraining. A crucial element of the paper's positioning is that it is not arguing against pretraining. Pretraining on massive corpora is clearly essential — the gap between fastText (30.9%) and BERT-Large (46.7%) in Table 1 is enormous, and even LSTM+BERT-Base (35.9%) substantially outperforms LSTMs with GloVe or ELMo embeddings. The paper's critique is more specific: pretraining plus fine-tuning on a static benchmark is insufficient for measuring whether reasoning has been acquired, because the fine-tuning process can latch onto spurious dataset-specific signals that the pretrained model's representations happen to encode. The solution is not to abandon pretraining but to construct benchmarks that are robust to this kind of exploitation — benchmarks where the only way to succeed is to actually perform the reasoning the task intends to measure.

The Specific Gap HellaSwag Fills

By the end of the background, the paper has established a clear gap:

1. SWAG appeared solved by BERT (86.7% accuracy, near human-level), creating the impression that machines can perform commonsense NLI.

2. Forensic analysis reveals BERT did not learn reasoning — it learned to detect stylistic artifacts in LSTM-generated wrong answers, as shown by its high performance without context (74.8%) and with shuffled words (77.0%).

3. The root cause is a generator-discriminator mismatch in SWAG's AF construction: a shallow LSTM generator produced artifacts that BERT could detect, and the shallow discriminator ensemble couldn't filter them out.

4. No existing dataset tested whether models could perform commonsense reasoning without exploiting such artifacts, because no dataset was constructed using generators and discriminators powerful enough to remove artifacts detectable by the best available models.

HellaSwag fills this gap by using OpenAI GPT (a 12-layer transformer) as the generator and BERT-Large (a 24-layer bidirectional transformer) as the adversarial filter, applied to a new domain (WikiHow) with longer, more complex text that pushes examples into the Goldilocks zone where they are simultaneously easy for humans and hard for machines. The result is a dataset where surface-level statistical exploitation is insufficient, and where model performance — even with substantial in-distribution training data — remains below 48%, compared to >95% for humans.

3. Technical Approach

3.1 Reader Orientation

This paper constructs HellaSwag, a multiple-choice dataset for commonsense natural language inference where each question presents a narrative context and four candidate endings — only one of which is the actual human-written continuation — and the system (or human) must select the most plausible one. The core engineering problem is generating wrong answers that are simultaneously easy for humans to reject yet difficult for even state-of-the-art language models to distinguish from the correct answer, a challenge solved by the Adversarial Filtering pipeline that iteratively refines machine-generated endings using increasingly discriminating classifiers until the remaining wrong answers form an adversarial set against the best available models.

3.2 Big-Picture Architecture (Diagram in Words)

The HellaSwag construction pipeline has five major components feeding into one another:

1. Source Text Corpora — Two domains supply the raw narrative contexts and ground-truth continuations: ActivityNet Captions (video descriptions with temporal event sequences) and WikiHow articles (step-by-step how-to instructions). These provide the scaffolding of real human-written event sequences.

2. Language Generator (OpenAI GPT, fine-tuned per-domain) — Given a context, this component produces candidate wrong endings by sampling from a conditional language model. It generates massively oversampled pools of synthetic continuations (many per context) that are topically related to the context but semantically flawed.

3. Adversarial Filter (BERT-Large, re-initialized per iteration) — A discriminator trained to distinguish real endings from generated ones on a dummy training split. It scores all generated candidates, and those that are too easily classified as machine-written are discarded and replaced with fresh samples that previously fooled the filter.

4. Iterative Refinement Loop — The AF process repeats for multiple rounds, each time retraining the discriminator from scratch on the current adversarial set and replacing the weakest (most detectable) negatives with stronger ones, driving the dataset toward a set of wrong answers that are maximally confusing to the discriminator.

5. Human Validation Layer — Crowd workers perform a final quality-control pass, identifying and removing generated endings that are actually plausible (false negatives that slipped through AF) and ensuring that the remaining wrong answers are genuinely nonsensical to humans, producing the final >95% human accuracy while keeping model accuracy <48%.

Information flows sequentially: source text → generator oversampling → AF iterative selection → human validation → final dataset. At inference time, models see (context, four candidate endings) and must predict which ending is the real one.

3.3 Roadmap for the Deep Dive

• First, the formal structure of the commonsense NLI task as instantiated in HellaSwag — the input/output format, the four-way multiple-choice setup, and the loss function used to train all evaluated models — because every subsequent design decision is shaped by this task formulation.
• Second, the source text corpora (ActivityNet and WikiHow) and why their properties — domain diversity, context length, and event structure — are essential for reaching the Goldilocks zone.
• Third, the generator component: how OpenAI GPT is fine-tuned per-domain, how candidate endings are sampled (including the crucial choice of stochastic decoding over beam search and the use of Nucleus Sampling), and why a 12-layer transformer generator is necessary where an LSTM generator failed for SWAG.
• Fourth, the Adversarial Filtering algorithm itself — the iterative train-and-replace loop, the dummy train/test split strategy, the multi-way training setup, the convergence criteria, and the key property that the resulting dataset is adversarial for any split.
• Fifth, the Goldilocks zone analysis: how the authors systematically varied generation length to find the sweet spot between human easiness and machine difficulty, and why two-sentence WikiHow endings occupy this zone.
• Sixth, the human validation protocol: the multi-round filtering strategy, the probabilistic worker model used to estimate answer quality, and the activity label structure enabling zero-shot evaluation splits.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a dataset construction and empirical analysis paper whose core technical contribution is the scaled-up Adversarial Filtering methodology that produces a benchmark where surface-level statistical exploitation is insufficient and where model performance genuinely reflects (the absence of) commonsense reasoning capability.

The Commonsense NLI Task Formulation

HellaSwag inherits its task structure directly from SWAG: it is a four-way multiple-choice commonsense natural language inference problem. For each question, a model receives a narrative context — typically 1-3 sentences describing a situation — and four candidate endings, exactly one of which is the ground-truth continuation (the actual next event that occurred in the source video or article). The model must select which of the four endings is the correct one.

Input format for BERT-based models. The paper adopts the standard BERT two-sentence input format (described in Appendix C). The context and candidate ending are concatenated with special tokens:

[CLS] context tokens [SEP] candidate ending tokens [SEP]

For example:

[CLS] A woman is outside with a bucket and a dog. The dog is running around trying to avoid a bath. [SEP] She gets the dog wet, then it runs away again [SEP]

This is done independently for each of the four candidate endings, producing four sequences. The [CLS] token's final hidden representation is fed through a linear classification layer to produce a scalar logit for that (context, ending) pair. The four logits are then passed through a softmax to produce a probability distribution over the four choices.

For the ablation conditions in Section 3.2 where context is omitted ("Ending Only"), the input format becomes:

[CLS] candidate ending tokens [SEP]

with no context segment and no separating [SEP] between context and ending because the context is absent.

Training objective. All models are trained with a standard four-way cross-entropy loss. Let $s_i$ be the logit for candidate ending $i \in \{1,2,3,4\}$ for a given question, and let $y \in \{1,2,3,4\}$ be the index of the correct ending. The loss for that question is:

$\mathcal{L} = -\log\left(\frac{e^{s_y}}{\sum_{j=1}^{4} e^{s_j}}\right)$

where $s_y$ is the logit assigned to the correct ending and the denominator sums over all four candidates.

What it computes: the negative log-likelihood of the correct ending under the model's predicted distribution. For each training example, the model produces four raw scores, converts them to probabilities via softmax, and is penalized according to how much probability mass it placed on the correct answer. The loss is minimized when $s_y$ is much larger than the other three logits, meaning the model assigns near-1.0 probability to the correct choice.

Why this form: this is the standard maximum-likelihood objective for multiclass classification, and it is the same objective used by Devlin et al. (2018) for SWAG fine-tuning. It treats the four endings as competing hypotheses and forces the model to discriminate among them, which is exactly the inference problem at test time. An alternative would be binary classification (real vs. fake for each ending independently), but the four-way formulation is harder because the model cannot simply learn a threshold — it must rank the correct ending above three adversarially-chosen distractors, which requires finer-grained discrimination.

During training, the four-way input is constructed by pairing the context with each of the four endings separately, producing four independent forward passes through BERT (or the equivalent for other architectures). At inference time, the same procedure is used, and the ending with the highest softmax probability is selected as the prediction.

Why four choices specifically. The paper maintains the four-choice format from SWAG throughout. This is not arbitrary — it balances two competing concerns. Fewer choices (e.g., binary) would make the task easier for models and harder to calibrate difficulty. More choices (e.g., the six-choice format used during human validation, described in Appendix E) would make the task harder for humans, risking lower human agreement and muddying the interpretation of the human-machine gap. Four choices provides enough distractors to challenge models while keeping the task straightforward for human annotators, as evidenced by the >95% human accuracy achieved.

Source Text Corpora: ActivityNet and WikiHow

The choice and properties of the source text are not incidental — they are a core design parameter that determines whether Adversarial Filtering can find the Goldilocks zone.

ActivityNet Captions (Krishna et al., 2017). This dataset contains temporal video descriptions: each video is annotated with a sequence of captions describing events in order. For example, a video of someone shaving might have captions like "A bearded man is seen speaking to the camera and making several faces" followed by "the man then holds up a razor and begins shaving his face." The temporal structure means that each caption naturally serves as a context, and the next caption is the ground-truth ending. ActivityNet also provides activity labels for each caption (e.g., "shaving," "jumping rope," "sharpening knives"), which the paper exploits to construct zero-shot evaluation splits — a crucial feature discussed below.

The original SWAG dataset used both ActivityNet and LSMDC (movie descriptions; Rohrbach et al., 2017) as source domains. HellaSwag uses ActivityNet exclusively for its video caption domain, dropping LSMDC. The paper does not elaborate extensively on this choice, but it is consistent with the goal of clean evaluation: ActivityNet's activity labels enable the zero-shot generalization test, and the domain is thematically coherent (everyday physical activities) in a way that movie descriptions are not.

WikiHow: a new and challenging testbed. The major expansion in HellaSwag is the addition of WikiHow articles as a second source domain. WikiHow is an online how-to manual containing step-by-step instructions for diverse tasks. The authors scrape approximately 80,000 context and follow-up paragraph pairs from WikiHow, covering topics ranging from "how to make an origami owl" to "how to survive a bank robbery."

The structural properties of WikiHow make it particularly well-suited for the Goldilocks zone:

• Length: Each context contains at most three sentences, as do the follow-up paragraphs. This is substantially longer than ActivityNet captions — Figure 7 shows that WikiHow contexts and endings average around 40-60 WordPiece tokens, versus roughly 10-20 for ActivityNet. Longer text provides more opportunities for a generator to introduce detectable errors (making the task easier for humans) while also providing more surface-level lexical overlap between context and ending (making the task harder for models that rely on word-matching heuristics).

• Diversity: The 80,000 articles span an enormous range of topics, from cooking and crafts to health and finance. This diversity means that models cannot memorize a small set of domain-specific patterns — they must generalize across topics.

• Instructional structure: WikiHow articles are explicitly procedural: they describe sequences of actions to achieve a goal. This aligns naturally with the commonsense NLI format, where the model must predict the next step in a procedure. The structured, goal-directed nature of WikiHow text means that wrong endings can violate commonsense in rich ways — suggesting actions that are physically impossible, logically inconsistent with the goal, or temporally out of order — rather than merely being topically irrelevant.

Domain-specific category labels for zero-shot evaluation. Both ActivityNet and WikiHow provide category labels that group examples by topic. ActivityNet provides activity labels (e.g., "shaving," "doing crunches," "sharpening knives"), and WikiHow provides article categories (e.g., "Youth," "Family Life," "Pets and Animals"). The paper exploits these labels to construct zero-shot evaluation splits: for each dataset (validation and test), the authors create two 5,000-example subsets. The "in-domain" subset contains examples from categories that appear in the training set. The "zero-shot" subset contains examples from randomly chosen held-out categories that the model never saw during fine-tuning. This split structure (described in Section 4.4 and detailed in Appendix H, Figure 13) tests whether models have learned generalizable commonsense reasoning or merely memorized category-specific patterns. The 5% performance drop on zero-shot categories reported in Table 1 (e.g., BERT-Large dropping from 49.7% in-domain to 45.0% zero-shot on the test set) is one of the paper's key pieces of evidence that models have not acquired robust reasoning.

Dataset sizes. After human validation and quality filtering (keeping only the examples with highest human agreement), the final HellaSwag dataset contains approximately 25,000 ActivityNet contexts and 45,000 WikiHow contexts, totaling roughly 70,000 examples. The 10,000-example validation and test sets are split from this pool, with the in-domain and zero-shot subsets each containing 5,000 examples drawn from the corresponding category partitions.

The Generator: Fine-Tuned OpenAI GPT with Stochastic Decoding

The generator's role is to produce candidate wrong endings — continuations that are topically related to the context, grammatically fluent, and superficially plausible, yet semantically flawed in ways that violate commonsense. The quality of the generator directly determines the ceiling of Adversarial Filtering: if the generator produces text with detectable stylistic artifacts (as SWAG's LSTM did), no amount of discriminative filtering can remove those artifacts because they are present in every candidate.

Why GPT over an LSTM. The paper explicitly contrasts two generators in Figure 5 (left). When the original SWAG LSTM (a two-layer recurrent network from Zellers et al., 2018) is used as the generator, BERT-Large's accuracy after Adversarial Filtering converges to approximately 75% — meaning BERT can still distinguish machine-written from human-written endings 75% of the time, even after iterative adversarial selection. The generations contain distributional artifacts (word choice patterns, syntactic regularities, statistical fingerprints of LSTM decoding) that BERT's deep bidirectional representations can detect. When OpenAI GPT (Radford et al., 2018) — a 12-layer transformer decoder pretrained on the BookCorpus (Zhu et al., 2015) — is used instead, BERT's accuracy drops below 30% after AF. GPT's generations are realistic enough that the adversarial filter can successfully select candidates that fool even BERT-Large.

This is the critical technical lesson: the generator must be at least as sophisticated as the discriminator that will eventually be evaluated on the dataset. If the discriminator (BERT-Large, 24 layers, 340M parameters) has vastly more representational capacity than the generator (2-layer LSTM), it will detect the capacity gap as a stylistic signal. Using a transformer generator (GPT, 12 layers, 117M parameters) narrows this gap substantially, making the stylistic signal much weaker and forcing the discriminator to rely on actual semantic plausibility — exactly the capability we want to measure.

Fine-tuning procedure (Appendix B). The pretrained OpenAI GPT model is fine-tuned separately on each domain:

• For WikiHow, the model is fine-tuned for 2 epochs on the scraped article text.
• For ActivityNet, the model is fine-tuned for 5 epochs.
• Both use the default learning rate from Radford et al. (2018).

The different numbers of epochs likely reflect dataset size differences (WikiHow has more data, requiring fewer epochs to converge) and domain complexity, though the paper does not elaborate on this choice.

Stochastic decoding: Nucleus Sampling with $p = 0.98$. The paper makes an explicit and critical choice to use stochastic decoding rather than beam search for generating candidate endings. This is described in Appendix B:

"Importantly, we generate randomly according to the language model distribution, rather than performing beam search – this would bias the generations towards common words."

Beam search, which selects the most likely token sequence under the model, tends to produce generic, high-frequency text that is stylistically distinct from human writing — it lacks the idiosyncratic word choices and structural variability of natural language. For WikiHow endings specifically, the authors use Nucleus Sampling (Holtzman et al., 2019) with $p = 0.98$. Nucleus Sampling works as follows: at each decoding step, the model's output distribution over the vocabulary is sorted by probability, and tokens are sampled only from the smallest set whose cumulative probability mass exceeds $p$. Tokens with cumulative probability in the tail (those with combined mass < 0.02) are zeroed out. This truncates the unreliable tail of the distribution — where low-probability tokens are essentially noise — while preserving diversity in the high-probability region.

The choice of $p = 0.98$ is quite high (the original Holtzman et al. paper used $p = 0.95$ for story generation), meaning only 2% of the probability mass is discarded. This keeps the sampling distribution close to the model's full distribution while eliminating only the most egregious tail tokens. The result is diverse, human-like text that contains occasional semantic errors — exactly the raw material Adversarial Filtering needs.

Why stochastic decoding matters for AF. If the generator produced only high-likelihood beam search outputs, the wrong answers would all be "reasonable" continuations that the language model considers probable. These would be hard for both humans and machines to distinguish from real text, violating the requirement that the dataset be easy for humans. If the generator produced purely random text, the wrong answers would be trivially detectable by simple statistical features. Stochastic decoding with nucleus sampling sits between these extremes: it produces text that is fluent and topically related (because it samples from the high-probability region) but occasionally incoherent or nonsensical (because it doesn't always select the single most likely token). This diversity creates a pool of candidates at varying levels of plausibility, from which AF can select the subset that is hardest for discriminators while still being clearly wrong to humans.

Oversampling. The paper states that candidate endings are "massively oversampled" from the language model — many candidates are generated per context to provide a large pool for AF to select from. For ActivityNet, $k = 9$ assigned indices per example (meaning 9 candidate endings are generated and kept in the pool at any time). For WikiHow, $k = 5$ because "there were fewer good endings produced by the generators after scaling up the sequence length" (Appendix A). The longer WikiHow generations (averaging 41 tokens vs. 13 for ActivityNet) are harder for the language model to produce coherently, resulting in a smaller pool of candidates that are fluent enough to be worth adversarial filtering.

Adversarial Filtering Algorithm: Iterative Train-and-Replace

Adversarial Filtering is the core algorithmic contribution — the mechanism by which a large pool of machine-generated wrong answers is distilled into a small set of adversarial choices that fool even state-of-the-art discriminators. The paper's description in Section 2 and Appendix A provides the full protocol.

The fundamental property: adversarial for any split. The key design goal of AF is stated in Section 2:

"The key idea is to produce a dataset D which is adversarial for any arbitrary split of (D_train, D_test)."

This is the property that distinguishes AF from standard adversarial example construction. In typical adversarial example work (e.g., Jia and Liang, 2017), a model is trained on a clean training set and then evaluated on a manually perturbed test set designed to break it. The model fails because of distribution shift — the test distribution differs from the training distribution. AF, by contrast, creates a dataset where no matter how you split it into train and test, the test set is adversarial. This means that even when the model is trained on examples from the exact same distribution as the test examples, it still struggles. The difficulty is intrinsic to the examples themselves, not an artifact of train-test mismatch.

The iterative loop (Figure 2). The AF procedure operates on a dummy dataset — a pool of contexts, each with one real ending and multiple generated endings — and refines it through repeated rounds. Here is the step-by-step protocol:

Step 0: Initial pool construction. For each context in the source data, the fine-tuned GPT generates a large number of candidate endings (via nucleus sampling or standard stochastic decoding). The real ending is retained as the positive example. The generated endings form the initial pool of negative candidates. For ActivityNet, each context is assigned $k = 9$ generated endings (kept as a fixed-size pool). For WikiHow, $k = 5$.

Step 1: Random train/test split. The dummy dataset (all contexts with their current assigned negatives) is randomly partitioned into $D_{\text{train}}$ (80%) and $D_{\text{test}}$ (20%). This split is performed without regard to video/article IDs — it is a simple random partition. This is important because it means the split does not respect domain boundaries; training and test sets are drawn from the same distribution.

Step 2: Discriminator training. A BERT-Large model is re-initialized from its pretrained checkpoint (not from the previous AF iteration's weights) and fine-tuned on $D_{\text{train}}$ in a multi-way classification setup. The training objective is the same four-way cross-entropy described above: for each training example, the model sees the context paired with the real ending and with exactly three of the $k$ currently-assigned generated negatives (subsampled randomly from the pool of $k$). The model learns to assign high probability to the real ending.

The multi-way setup is described in Appendix A:

"On each training example, the model is given exactly one positive ending and several negative endings, and the model computes a probability distribution over the endings through a softmax."

This matches the final evaluation format: the discriminator is trained in exactly the same four-way multiple-choice setup that downstream models will face, making the adversarial selection directly targeted at the evaluation metric.

Training hyperparameters for AF (Appendix D). The authors randomize some hyperparameters on each AF iteration to prevent the filter from overfitting to a single optimization trajectory:

• Learning rate: sampled log-uniformly between $1 \times 10^{-5}$ and $4 \times 10^{-5}$.
• Cased vs. uncased: with probability 0.5, the cased BERT model is used (where input is not lowercased before tokenization); otherwise, the uncased model is used.
• Number of epochs: 3 epochs during the AF loop (contrasted with 10 epochs for final evaluation on HellaSwag, which the authors found beneficial).
• Batch size: the standard BERT fine-tuning batch size is used (the paper does not specify a different value for AF; the baseline is 16 or 32 from Devlin et al., though Appendix D notes they use 64 for final HellaSwag training to improve stability).

Step 3: Evaluation on D_test and replacement of weak negatives. The trained discriminator is evaluated on $D_{\text{test}}$ — again in the four-way format — to measure how accurately it can identify the real ending among its currently-assigned negatives. Generated endings that the discriminator correctly identifies as wrong (i.e., the discriminator assigns low probability to them relative to the real ending) are considered "too easy." These are replaced with new generated endings — freshly sampled from the GPT generator — that were not in the current pool and that the discriminator finds harder to classify.

The replacement logic is crucial: the goal is not to keep the hardest negatives from the current pool, but to continuously introduce new candidates that might be even harder. This prevents the pool from stagnating at a local optimum where all remaining negatives are still somewhat detectable.

Step 4: Iteration. Steps 1-3 are repeated. On each iteration, the dataset is randomly re-split into new $D_{\text{train}}$ and $D_{\text{test}}$ partitions, a fresh BERT-Large is trained from the pretrained checkpoint, and the replacement process continues. The accuracy of the discriminator on $D_{\text{test}}$ is tracked across iterations (shown in Figure 5).

Step 5: Convergence. The process continues until the discriminator's accuracy on $D_{\text{test}}$ stabilizes — that is, further iterations no longer reduce the accuracy. At this point, the generated negatives in the pool are as hard as the discriminator can make them: any remaining generated ending that is still easily classified would have been replaced in a previous iteration, and newly generated candidates are no more effective at fooling the discriminator than the current ones.

However, there is an important edge case noted in Appendix A:

"Sometimes, BERT never converges (accuracy around 25%), so when this happens, we don't do the reassignment."

If the discriminator's accuracy drops to approximately 25% (random chance for four-way classification), it means the discriminator cannot distinguish real from generated endings at all. In this case, there is no signal to guide replacement — the negatives are already maximally adversarial — so the loop terminates early.

Convergence behavior and the Goldilocks zone (Figure 5). Figure 5 (right) shows the AF convergence trajectories for WikiHow with different ending lengths, using BERT-Large as the discriminator. All three settings start with high discriminator accuracy (70-90%), reflecting that the initial GPT generations are easily distinguishable from human text. As AF iterates, accuracy drops as weak negatives are replaced:

• One-sentence endings converge to approximately 35%, slightly above random (25%). These short generations contain fewer opportunities for semantic error, making them easier for the discriminator to classify even after AF.
• Two-sentence endings converge to approximately 40%. This is the Goldilocks zone: long enough to contain commonsense violations that humans detect but short enough that BERT cannot reliably separate them from real text.
• Three-sentence endings converge to approximately 50%. These longer generations are easier for the discriminator because the additional length provides more surface-level signals (word choice patterns, syntactic regularities across multiple sentences) that reveal their machine origin.

The paper compromises on two-sentence generations for WikiHow, explicitly framing this as the Goldilocks zone:

"Particularly in the two-sentence case, we find ourselves in a Goldilocks zone wherein generations are challenging for deep models, yet as we shall soon see, easy for humans."

Why AF converges at different levels. The convergence accuracy reflects a fundamental tension between generator quality and text length. A perfect generator would produce text indistinguishable from human writing, and AF would converge to 25% regardless of length — the discriminator would have no signal. GPT, while strong, is not perfect. At short lengths (one sentence), there is simply not enough text for the generator's imperfections to manifest as detectable errors — the discriminator converges near random because the signal is too weak. At long lengths (three sentences), the generator's imperfections accumulate across sentences, providing a stronger signal that the discriminator can exploit. At intermediate lengths (two sentences), the signal is present but weak enough that AF can push accuracy down substantially — though not to chance — creating examples where the correct answer and wrong answers look similar to BERT but radically different to humans.

Discriminator Training Details and Multi-Way Setup

The discriminator's role in AF is to provide a signal for which generated endings are "too easy" and need replacement. This requires careful setup to ensure the signal reflects semantic plausibility rather than superficial artifacts.

Multi-way training with subsampled negatives. For each training example in $D_{\text{train}}$, the discriminator is not trained on all $k$ negatives simultaneously. Instead, exactly 3 negatives are subsampled from the pool of $k$ currently assigned to that context, creating a four-way classification problem (1 real + 3 generated). This is done for two reasons:

1. Consistency with evaluation: The final dataset will present 4 choices to models (1 real + 3 generated). Training the discriminator in the exact same format ensures that the adversarial selection optimizes for the same decision problem that downstream models will face.

2. Computational tractability: Training a $(k+1)$-way classifier where $k = 9$ (ActivityNet) would require 10-way softmax, which is more expensive per example. Subsampling to 4-way keeps training efficient while still exposing the discriminator to diverse negatives across epochs.

The accuracy values reported during AF (Figure 5) are computed on $D_{\text{test}}$ using the first 3 assigned negatives for each example — not a random subsample — to maintain consistency.

Re-initialization from pretrained checkpoint. Critically, the discriminator is re-initialized from the original BERT-Large pretrained weights on every AF iteration, not fine-tuned further from the previous iteration's weights. This prevents the discriminator from "remembering" which endings were adversarial in previous rounds and overfitting to the specific pool composition. Each iteration starts fresh, ensuring that the replacement decisions are based on the current pool's properties rather than accumulated bias.

Hyperparameter randomization across iterations. The paper notes in Appendix D that they randomize the learning rate and cased/uncased variant on each AF iteration. This is a form of implicit ensembling: by varying the discriminator's training conditions, different iterations may be sensitive to different types of artifacts, and the aggregate replacement process across iterations is more robust than any single discriminator configuration would be. If the same hyperparameters were used every iteration, the discriminator might consistently miss a particular class of detectable artifacts, leaving them in the pool.

The case where BERT never converges (accuracy ~25%). When the discriminator's accuracy drops to approximately random chance (25%), it means the generated endings are so realistic that even BERT-Large cannot distinguish them from human text. In this case, further replacement is pointless — there is no signal to guide which negatives are "worse" than others. The paper stops AF at this point. This occurred for GPT generations on ActivityNet (Figure 5, left, where accuracy drops below 30% and stabilizes), but not for WikiHow (where accuracy converges to ~40% for two-sentence endings). The difference reflects the inherent difficulty of generating realistic long-form instructional text versus short video captions.

Human Validation Protocol: Ensuring High Human Agreement

After AF converges, the dataset contains contexts each with one real ending and $k$ generated endings. However, AF optimizes for discriminator confusion, not human plausibility. Some generated endings may have slipped through that are actually reasonable continuations — false negatives that would confuse human annotators as much as they confuse BERT. The human validation stage removes these, ensuring the final dataset has high human agreement.

Validation format: six-way choice. Human validation uses a different format from the final dataset. Crowd workers are shown a context and six ending choices: the one real ending and five generated endings from the AF pool. They must select the most plausible continuation. Using six choices rather than four makes the annotator's task harder — they must discriminate among more distractors — which helps identify generated endings that are too realistic (since they will attract votes that should go to the real ending).

Iterative replacement of plausible negatives. Human validation proceeds in rounds. On each round:

1. Workers label a batch of examples.
2. For each example, the paper estimates how likely each generated ending is to be the real one, based on worker votes (using the probabilistic model described below).
3. Generated endings that workers rated as plausible (receiving significant vote share) are flagged as false negatives and replaced with new generated endings from the AF pool that were not previously validated.
4. The process repeats with the new set of negatives.

This iterative human-in-the-loop refinement is analogous to AF itself but with human judgment replacing the discriminator. It pushes the dataset toward a state where the generated endings are clearly wrong to humans — because any ending that humans found plausible was removed and replaced.

Probabilistic worker model for answer quality estimation (Appendix E). The paper describes a Bayesian approach to aggregating worker judgments:

"For each worker, we use the summary of their performance so far to estimate P(answer i is right | worker rates i as best). We can then use this to estimate how confident we are in each answer choice: we want to be confident that workers will not prefer the wrong answers."

In practice, this means:

1. Each worker's historical accuracy is tracked — how often they selected the real ending when they rated an ending as "best."
2. For a new example with worker votes, the probability that each ending is the real one is estimated by combining the worker's reliability with their vote. If a highly reliable worker selects ending A as best, that is stronger evidence that A is the real ending than if an unreliable worker selects it.
3. These probabilities are multiplied across workers (treating workers as independent given the true answer) to produce an aggregate score for each ending.
4. Generated endings with high aggregate scores (meaning multiple workers, especially reliable ones, thought they might be the real ending) are flagged for replacement.

This probabilistic approach is more robust than simple majority voting because it accounts for worker quality variation and produces a continuous confidence score rather than a binary keep/replace decision.

Final selection: keeping the best examples. After multiple rounds of human validation, the paper selects the subset of examples with the highest agreement — those where workers most consistently and confidently identify the real ending. Specifically, they keep the "best 25k examples from ActivityNet and the best 45k from WikiHow" (Appendix E), where "best" means the ones with the highest estimated probability that workers will select the true answer over the three easiest-to-guess negatives. This filtering step removes examples where the ground-truth ending itself is ambiguous or poorly written — a known issue with ActivityNet captions noted in Appendix E:

"Particularly on ActivityNet, we found that there are some contexts where the ground truth answer isn't liked by workers."

By retaining only examples with clear ground-truth endings, the dataset ensures that the human accuracy ceiling genuinely reflects task difficulty rather than annotation noise.

Human performance trajectory (Figure 6). Figure 6 shows how human and BERT accuracy evolve through rounds of validation. On ActivityNet:

• Before human validation (0 annotators): BERT scores 25.5% (near chance), but humans also score only 60% — meaning the generated endings are confusing even to people.
• After 1 round: human accuracy rises to 85%, BERT to 48.4%.
• After 2 rounds: human accuracy reaches 94%, BERT at 57.1%.

The gap widens dramatically through validation because humans are far better than BERT at distinguishing real from generated endings once the most confusing negatives are removed. On WikiHow, the initial gap is already large: humans score 93.5% before any validation, while BERT scores 41.1%. After 2 rounds, humans reach 96.5% and BERT only 46.0%.

Why WikiHow starts easier for humans. The paper attributes this to length (Figure 7). WikiHow's two-sentence generations average 41 WordPiece tokens, versus 13 for ActivityNet. This gives WikiHow generations "three times as many opportunities to make a detectable mistake" (Section 4.3). In a short ActivityNet caption like "she rinses the bucket off with soap and blow dry the dog's head," the semantic error (blow-drying a wet dog with soap?) might be subtle enough that humans need careful reading to catch it. In a longer WikiHow passage, the errors compound and become obvious.

Worker qualification and quality control (Appendix E). The paper implemented several best practices for crowdsourcing:

• Fair pay: up to 37 cents per HIT (Human Intelligence Task) on WikiHow, reflecting the task's difficulty and length.
• Qualification HIT: an auto-graded screening task that filters for workers who perform well. Workers who "tended to prefer the generated endings over the real ones were disqualified from participating," ensuring that the validation pool consists of annotators who can actually do the task.
• Disqualification of unreliable workers: workers who consistently select generated endings as best are removed, preventing their noisy judgments from contaminating the probabilistic model.

Final dataset composition. After all filtering stages — AF convergence, iterative human validation, and best-example selection — the final HellaSwag dataset contains:

• Training set: approximately 40,000 examples? (The paper does not specify the exact training set size; it states the validation and test sets are each 10,000 examples, and the total dataset is roughly 70,000, implying roughly 50,000 training examples, but this is not explicitly stated in the main text.)
• Validation set: 10,000 examples total, comprising 5,000 in-domain (3,200 ActivityNet + 1,800 WikiHow? No — the split sizes in Table 1 show 3,200 ActivityNet validation and 6,800 WikiHow validation in the overall validation set, with the in-domain and zero-shot subsets each containing 5,000 examples drawn proportionally from both domains.)
• Test set: 10,000 examples, structured identically to validation.
• In-domain subsets: 5,000 examples each (validation and test) from categories seen during training.
• Zero-shot subsets: 5,000 examples each from held-out categories.

Zero-Shot Category Splits: Probing Generalization

The paper constructs zero-shot evaluation splits to test whether models have learned generalizable commonsense reasoning or merely memorized category-specific patterns. This is a crucial experimental design choice that provides one of the paper's strongest pieces of evidence that the task is not solved.

Construction procedure (Section 4.4). From the full set of categories in each domain:

1. A random subset of categories is held out entirely — no examples from these categories appear in the training set.
2. The validation and test sets each receive 5,000 examples from in-domain categories (categories represented in training) and 5,000 examples from held-out zero-shot categories.
3. Models are trained on the full training set (which includes only in-domain categories) and evaluated separately on the in-domain and zero-shot subsets.

What this tests. If a model has learned genuine commonsense reasoning — understanding that getting a dog wet leads to it running away, or that you don't stop at a red light for "at most two seconds" — then it should perform equally well on zero-shot categories. The reasoning required is the same regardless of whether the category is "shaving" (seen during training) or "sharpening knives" (zero-shot): in both cases, the model must connect a described situation to physical and social world knowledge.

If, instead, the model has learned category-specific surface patterns — for instance, that "shaving" contexts tend to be followed by endings containing words like "razor" and "face" — then performance should drop on zero-shot categories where those specific lexical associations haven't been memorized.

Result preview. Table 1 shows that BERT-Large drops from 49.7% in-domain to 45.0% zero-shot on the test set — a roughly 5-point decline. While this is not a catastrophic failure (the model still performs well above chance), it is a statistically meaningful gap that suggests a non-trivial fraction of the model's in-domain performance comes from category-specific memorization rather than transferable reasoning. The fact that the gap persists even though zero-shot examples are drawn from the same underlying domains (ActivityNet video captions, WikiHow articles) and require the same type of reasoning makes this finding particularly incriminating: the model is not robust to mild distribution shift within the same task.

The Goldilocks Zone: Length, Complexity, and the Human-Machine Gap

The concept of a Goldilocks zone is the paper's central design insight — not a formal algorithm but a principle that guided the choice of source domain, generation length, and AF parameters.

Definition. The Goldilocks zone is a region of text complexity where:

1. Human accuracy is high (>90%) because the wrong answers contain obvious (to humans) commonsense violations — events that are physically impossible, logically inconsistent, or absurd given the context.
2. Machine accuracy is low (<50%) because the wrong answers are lexically and topically similar enough to the correct answer that surface-level statistical features cannot reliably discriminate them, and deep semantic reasoning (which the models lack) is required.

Finding the zone (Figure 5, right). The paper systematically varies one parameter — generation length — and measures both human and machine performance at each setting:

• One-sentence WikiHow endings: AF converges to ~35% discriminator accuracy, meaning BERT can barely distinguish real from generated. However, at this short length, humans might also struggle because there are fewer words in which commonsense errors can manifest. The paper does not report separate human accuracy for each length setting, but the logic implies that one sentence is below the Goldilocks zone — too short to guarantee human easiness.

• Two-sentence WikiHow endings: AF converges to ~40%. This is the Goldilocks zone: BERT is substantially confused (well above chance but far from human performance), while humans achieve 93.5% even before human validation (Figure 6). The two-sentence length provides enough text for commonsense violations to be obvious to humans while providing enough lexical overlap to fool statistical models.

• Three-sentence WikiHow endings: AF converges to ~50%. This is above the Goldilocks zone — the extra length provides too much signal for the discriminator, making the task easier for machines. The paper does not pursue this setting.

Why length matters. Longer text provides two competing effects:

• For humans: more words = more opportunities for the generator to produce a semantic error. A single implausible clause ("blow dry the dog's head" after "rinses the bucket off with soap") is easier to spot in a 40-word passage than a 10-word passage, because the error stands out against a backdrop of otherwise coherent text. Longer text also provides more context for disambiguation — a statement that seems plausible in isolation becomes clearly wrong when read in full context.

• For machines: more words = more surface-level features (word frequencies, n-gram overlaps, syntactic patterns) that can be used for statistical discrimination. A discriminator can exploit subtle distributional differences between human and machine text that accumulate across sentences. At one sentence, the signal is too weak; at three sentences, it is too strong; at two sentences, it is in the sweet spot where the human advantage in deep semantic processing outweighs the machine advantage in surface-level statistical extraction.

The role of domain choice. WikiHow is essential for reaching the Goldilocks zone because its texts are naturally longer and more complex than ActivityNet captions. ActivityNet captions average ~13 tokens for endings (Figure 7); even at their longest, they are shorter than one-sentence WikiHow endings. This is why ActivityNet starts harder for humans (60% before validation, Figure 6) — the short length means commonsense errors are harder to detect, and even humans need careful reading. WikiHow's procedural, instructional text provides the length and structure needed for the Goldilocks zone to exist.

Summary of Design Choices and Their Justifications

• OpenAI GPT over LSTM as generator: an LSTM produces text with detectable stylistic artifacts that BERT can exploit regardless of AF; a 12-layer transformer narrows the generator-discriminator capacity gap enough that AF can select genuinely confusing negatives.

• Stochastic nucleus sampling ($p = 0.98$) over beam search: beam search produces generic, high-likelihood text that is stylistically homogenous and fails to produce the diverse semantic errors needed for AF; nucleus sampling preserves diversity while truncating only the unreliable probability tail.

• Two-sentence WikiHow endings over one- or three-sentence: empirical investigation shows this length occupies the Goldilocks zone where human accuracy is high and machine accuracy is low; shorter text is too ambiguous for humans, longer text provides too much statistical signal for machines.

• BERT-Large as discriminator over shallower models: using the strongest available model as the adversarial filter ensures that the dataset is adversarial to the best current technology; weaker filters (as in SWAG) leave artifacts that stronger future models will detect.

• Iterative re-initialization of discriminator from pretrained checkpoint: prevents accumulated bias across AF iterations and ensures each round's replacement decisions are based on a fresh model's judgments.

• Hyperparameter randomization (learning rate, cased/uncased) across AF iterations: acts as implicit ensembling, making the aggregate replacement process more robust than any single discriminator configuration.

• Six-way human validation with iterative replacement: a harder task for annotators than the final four-way format, making it easier to identify generated endings that are too plausible; iterative replacement pushes the dataset toward high human agreement.

• Probabilistic worker model with reliability tracking: more robust than simple majority voting because it accounts for worker quality variation and produces continuous confidence scores for replacement decisions.

• Activity labels and WikiHow categories for zero-shot splits: enables clean measurement of generalization vs. memorization by holding out entire categories from training and testing on them.

• Best-example filtering (25k ActivityNet, 45k WikiHow): removes examples where the ground-truth ending is ambiguous or poorly written, ensuring the human accuracy ceiling reflects task difficulty rather than annotation noise.

4. Key Insights and Innovations

Innovation 1: Adversarial Filtering as a Process, Not a One-Time Filter

The most intellectually distinctive move in this paper is the elevation of Adversarial Filtering from a dataset construction technique into a meta-methodology for benchmark evolution. Prior work, including SWAG (Zellers et al., 2018), used AF as a one-time procedure: generate candidates, filter them with a discriminator, validate with humans, and release the dataset. The paper's forensic analysis of SWAG reveals why this static approach breaks: the generator and discriminator used in SWAG's construction were matched to ELMo-era models, and when BERT — a far more powerful model — arrived, it detected distributional artifacts that the original AF pipeline was too weak to filter out.

The paper's reframing of AF as an iterative arms race that must be re-run with the best available models each time a benchmark is saturated is what makes this a conceptual innovation rather than merely an incremental improvement on SWAG. The key insight is that dataset difficulty is relative to model capability, not an absolute property. A dataset that is adversarial for ELMo is not adversarial for BERT unless BERT-level generators and discriminators were involved in its construction. The paper makes this explicit in Section 6.3:

"What happens when HellaSwag gets solved? We believe the answer is simple: crowdsource another dataset, with the same exact format, and see where models fail."

This is not a throwaway line — it articulates a paradigm shift in how the field should think about benchmark construction. Static benchmarks will inevitably be saturated through surface-level statistical exploitation because the models that eventually evaluate on them will be more powerful than the models used to build them. The only sustainable approach is co-evolution: each time a benchmark is solved, rebuild it using the solvers as adversaries in the construction pipeline. The cycle terminates only when even the best models cannot generate wrong answers that are simultaneously foolable by the best discriminators yet trivially distinguishable by humans — which would imply genuine parity between human and machine commonsense.

This reframing matters beyond HellaSwag because it provides a principled answer to the benchmark saturation problem that plagues NLP. The traditional response to saturation — build a harder benchmark in a new domain — is ad hoc and doesn't address the underlying evaluation failure mode. The co-evolutionary approach provides a systematic, algorithmic path forward that is grounded in the same AF machinery that built the benchmark in the first place. It ties benchmark difficulty directly to the state of model capability, creating a ratchet that tightens as models improve.

The innovation is not the AF algorithm itself (which the paper inherits from SWAG) but the recognition that AF must be applied iteratively across model generations, not just across dataset iterations within a single construction. Figure 5 (left) makes this concrete: re-running AF with a stronger discriminator (BERT-Large) on SWAG's original LSTM generations converges at 75%, revealing the old dataset's inadequacy. Re-running it with a stronger generator (GPT) drives accuracy below 30%. This is the empirical signature of the co-evolutionary insight — you need both components upgraded to match the new model generation.

Innovation 2: The Goldilocks Zone as a Diagnostic Principle for Dataset Construction

The concept of a Goldilocks zone — a region of text complexity where human accuracy is high and machine accuracy is low, both sustained — is a genuinely novel diagnostic tool for benchmark design. It is not merely an observation about HellaSwag's WikiHow subset; it is a transferable principle that explains why certain datasets successfully measure reasoning while others become stylistic pattern-matching exercises.

Before this paper, the field lacked a vocabulary for articulating why some adversarial datasets worked (SWAG against ELMo) and others failed (SWAG against BERT). The Goldilocks zone provides that vocabulary. It identifies two failure modes that flank the desired region:

• Below the zone (too short/too simple): Human and machine accuracy are both low, or the gap is small. The text does not provide enough surface area for commonsense violations to be detectable by humans, making the task ambiguous even for people. ActivityNet's one-sentence captions — where humans score only 60% before validation (Figure 6) — sit here.

• Above the zone (too long/too complex): Machine accuracy rises because additional text length provides more surface-level statistical signals (n-gram patterns, word co-occurrences, syntactic regularities) that discriminators can exploit without reasoning. The AF convergence curves in Figure 5 (right) show this directly: one-sentence WikiHow generations converge to ~35% discriminator accuracy (near-random), two-sentence to ~40%, three-sentence to ~50%. The task gets easier for machines as it gets longer, which is the opposite of what you would expect if length forced deeper reasoning.

• Inside the zone: The text is long enough that commonsense errors compound and become obvious to humans (93.5% pre-validation accuracy on WikiHow, Figure 6), yet short enough that surface-level statistical signals are too weak for discriminators to rely on (BERT-Large scores 41.1% on the same data). The human-machine gap is maximized.

The significance of this insight extends beyond HellaSwag because it provides a design criterion for future benchmarks. A dataset builder can now ask: "Is my task in the Goldilocks zone?" by measuring the scaling behavior of human and machine accuracy with respect to example length or complexity. If human accuracy rises with length (more context = clearer commonsense violations) while machine accuracy falls (more context = harder to find shortcuts), you are in the zone. If both rise, your task is too easy — statistical features dominate. If both fall, your task is too hard for humans — annotation noise dominates.

The paper does not formalize this as a mathematical criterion, but the empirical demonstration across two domains (ActivityNet below the zone, WikiHow inside it) and three generation lengths (Figure 5, right) provides the template for others to follow. This is a conceptual advance in benchmark design methodology, not merely a property of one dataset.

Innovation 3: Surface Learning as the Explanation for BERT's "Success" — and a Warning for the Field

The paper's most provocative contribution is its diagnosis of why BERT solved SWAG without acquiring commonsense reasoning. The forensic experiments in Section 3 — context removal, word shuffling, and the combination of both — constitute a dissection of pretrained model behavior that reveals a specific failure mode: rapid surface learning of dataset-specific distributional artifacts during fine-tuning.

This is not merely the observation that "BERT overfits" or "datasets have biases." Those claims were already in the air (Gururangan et al., 2018; Poliak et al., 2018). The paper's distinctive contribution is the specificity of the diagnosis and its implications for how the field evaluates progress:

1. BERT's learning curve (Figure 3) shows it reaches ELMo's full-dataset performance with only 64 examples — a sample efficiency that is incompatible with learning general commonsense reasoning but perfectly consistent with rapidly detecting surface-level stylistic differences between human and LSTM-generated text.

2. Context removal (Figure 4) drops accuracy by only 11.9 points (86.7% → 74.8%) — meaning three-quarters of the discriminative signal comes from the endings alone, not from reasoning about whether the ending plausibly follows from the context.

3. Word shuffling (Figure 4) has an even smaller impact (86.7% → 77.0%) — BERT can achieve near-state-of-the-art performance on text whose word order has been destroyed, something no commonsense reasoning system should be able to do.

4. The combination — no context, shuffled endings — still yields 60.4% (Figure 4), which is higher than ESIM+ELMo's 59% on the full task. A model operating on scrambled, contextless fragments outperforms the previous state-of-the-art operating on well-formed input.

This progression of experiments is a masterclass in diagnostic evaluation. Each condition strips away a layer of information that would be essential for genuine reasoning — first the premise, then the word order, then both — and measures how much performance remains. The fact that substantial performance survives each ablation demonstrates that the original benchmark was measuring something other than what it claimed to measure.

The insight that makes this an innovation rather than a critique is the constructive implication: if surface learning is the mechanism by which models saturate benchmarks, then the only reliable defense is to construct benchmarks where surface features are non-predictive by design. This is exactly what HellaSwag's scaled-up AF achieves — by using a transformer generator whose outputs are stylistically similar to human text (Figure 5, left, where GPT generations drive discriminator accuracy below 30%), the surface-level signal is weakened to the point where models must rely on something deeper, and their performance drops accordingly.

The paper's framing of BERT as a "rapid surface learner" — a model that excels at extracting and exploiting shallow statistical regularities from fine-tuning data — has become influential far beyond this paper. It provides a coherent explanation for the pattern of rapid benchmark saturation followed by brittleness under distribution shift that has characterized the deep learning era in NLP, and it motivates the adversarial benchmark construction methodology that the paper advocates.

Innovation 4: The Generator-Discriminator Gap as the Root Cause of Benchmark Fragility

While not labeled as such in the paper, the analysis in Figure 5 (left) articulates a structural principle that explains why adversarial datasets degrade over time: the gap between the capacity of the generator used to create wrong answers and the capacity of the discriminator eventually used to evaluate on them determines how long the dataset remains challenging.

The principle is this: Adversarial Filtering can only remove artifacts that the filter can detect, but it cannot remove artifacts that the generator never produced in the first place because they are present in every candidate. If the generator (an LSTM) has far less representational capacity than the eventual discriminator (BERT-Large), the generator's outputs will contain systematic distributional fingerprints — statistical regularities that distinguish them from human text — that are present in every single generated ending. No amount of discriminative filtering can eliminate these because filtering selects among the generated endings; it cannot modify them to remove the generator's stylistic signature.

This explains why SWAG degraded: the LSTM generator produced text with artifacts that BERT could detect, and SWAG's shallow discriminator ensemble couldn't identify and filter out those artifacts because the discriminators themselves were too weak. The failure was not in the AF concept but in the mismatch between the generator-discriminator pair used during construction and the model eventually evaluated on the dataset.

The paper's solution — using OpenAI GPT (a 12-layer transformer, 117M parameters) as the generator and BERT-Large (24 layers, 340M parameters) as the discriminator — narrows this gap substantially. GPT's outputs are stylistically realistic enough that BERT's discriminator accuracy drops below 30% after AF on ActivityNet (Figure 5, left). The residual 30% is the "irreducible" signal that even a transformer generator cannot eliminate — the gap between GPT's commonsense reasoning ability and human writing.

This principle is a structural insight about dataset construction, not merely an empirical observation about one dataset. It implies that for any future benchmark, the generator must be at least as capable as the best available model, and the discriminator must be more capable — otherwise, the benchmark will be vulnerable to the next generation of even stronger models that can detect artifacts the construction pipeline couldn't filter. This is the theoretical underpinning of the co-evolutionary vision: each round of benchmark reconstruction must use generators and discriminators at least as strong as the models that solved the previous round.

Innovation 5: Zero-Shot Category Evaluation as a Probe for Reasoning vs. Memorization

The construction of zero-shot category splits (Section 4.4) is a methodological innovation that provides a clean, falsifiable test of whether models have acquired generalizable reasoning or category-specific surface patterns. This is distinct from the more common out-of-distribution evaluation paradigm (e.g., evaluating on a different dataset entirely) because the zero-shot examples are drawn from the same underlying task and domain — they are ActivityNet video captions and WikiHow articles, just from categories held out during training.

The diagnostic power of this split comes from the fact that it holds constant everything about the task except the specific lexical and topical patterns associated with particular categories. A model trained on "shaving" and "doing crunches" captions but tested on "sharpening knives" captions faces the same type of reasoning challenge — identifying physically plausible event sequences — but cannot rely on memorized category-specific word associations. The 5-point performance drop observed for BERT-Large (49.7% in-domain vs. 45.0% zero-shot, Table 1) therefore directly measures the extent to which the model's performance depends on memorization rather than transferable reasoning.

The innovation is the precision of the diagnostic. Prior work on annotation artifacts (Gururangan et al., 2018) demonstrated that models exploit biases, but couldn't quantify how much of a model's performance was attributable to bias vs. reasoning. The zero-shot split provides that quantification: the in-domain accuracy is an upper bound on reasoning (it includes both reasoning and memorization), the zero-shot accuracy is a lower bound on reasoning (excluding memorization of specific categories), and the gap between them measures the memorization component directly.

This design also serves a constructive purpose: it creates a harder evaluation setting without requiring a separate dataset or domain. Future work can report both in-domain and zero-shot accuracy on HellaSwag, and progress on closing the gap between them becomes a metric for whether models are improving at reasoning (which transfers) or merely at surface learning (which doesn't). The paper's vision of co-evolving benchmarks (Innovation 1) implicitly relies on this kind of structured evaluation to track whether genuine capability is being acquired.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use HellaSwag, the dataset constructed in this paper (Section 4). It contains approximately 70,000 total examples drawn from two domains: ActivityNet video captions (~25k examples after filtering) and WikiHow how-to articles (~45k examples after filtering). The validation and test sets each contain 10,000 examples, further partitioned into 5,000 in-domain examples (from categories seen during training) and 5,000 zero-shot examples (from held-out categories). For the transfer experiments in Section 5.1, SWAG (Zellers et al., 2018) is used as a comparison training or evaluation set, with careful handling to avoid overlapping ActivityNet examples across the two datasets.

• Base model(s). The paper evaluates a spectrum of models spanning different architectures, pretraining paradigms, and scales. The primary model of interest is BERT-Large (Devlin et al., 2018): a 24-layer bidirectional transformer with 340M parameters, pretrained on Wikipedia and BookCorpus. For comparison, the authors evaluate BERT-Base (12 layers, 110M parameters, matching the architecture scale of OpenAI GPT), OpenAI GPT (Radford et al., 2018): a 12-layer left-to-right transformer decoder pretrained on BookCorpus, ESIM+ELMo (Chen et al., 2017; Peters et al., 2018): the best-performing ELMo-based NLI model modified with a four-way softmax output layer, a two-layer bi-LSTM sentence encoder with three embedding variants (GloVe, ELMo, and frozen BERT-Base), and fastText (Joulin et al., 2017): an off-the-shelf bag-of-words text classifier. These models were chosen to span the range from shallow statistical baselines through ELMo-era recurrent models to the strongest available pretrained transformers, establishing how performance scales with model capacity and pretraining.

• Metrics. The primary metric throughout is four-way multiple-choice accuracy (%) — the fraction of questions for which the model assigns highest probability to the correct ending among the four candidates. Accuracy is reported overall, per-domain (ActivityNet vs. WikiHow), and per-evaluation-condition (in-domain vs. zero-shot). For human evaluation, predictions from five independent crowd workers are combined via majority vote to produce the reported human accuracy. Models produce logits via a softmax over the four candidates; the predicted answer is simply the candidate with the highest softmax probability. The loss function used during training is four-way cross-entropy (detailed in the Technical Approach section), but the reported metric is always accuracy, not loss.

• Baselines. The baselines span the full range of available NLP technology at the time of writing:

  • fastText (Joulin et al., 2017): A linear classifier over bag-of-words features, trained with binary cross-entropy loss. This establishes a floor for what is achievable with purely lexical information.
  • LSTM + GloVe: A randomly initialized two-layer bi-LSTM with 300-dimensional GloVe word embeddings. The second layer's hidden states are max-pooled across time and fed into an MLP for classification. This establishes performance for recurrent architectures without contextualized pretraining.
  • LSTM + ELMo: Same architecture but using ELMo embeddings (Peters et al., 2018) with learned scalar weights combining each internal layer. ELMo is not updated during fine-tuning. This represents the best pre-BERT recurrent approach.
  • LSTM + BERT-Base: Same architecture but using frozen BERT-Base embeddings, again with learned layer-combination weights. This isolates the contribution of BERT's pretrained representations without the benefit of end-to-end fine-tuning.
  • ESIM + ELMo (Chen et al., 2017; Peters et al., 2018): The best-performing ELMo-based NLI model from prior work, modified with a four-way softmax output layer instead of the original three-way (entailment/neutral/contradiction) setup. This establishes the best pre-BERT result on similar tasks.
  • OpenAI GPT (Radford et al., 2018): A 12-layer left-to-right transformer decoder, fine-tuned end-to-end on HellaSwag. This provides an architectural contrast to BERT (unidirectional vs. bidirectional).
  • BERT-Base (Devlin et al., 2018): A 12-layer bidirectional transformer with 110M parameters, fine-tuned end-to-end. This isolates the effect of model scale compared to BERT-Large.
  • BERT-Large (Devlin et al., 2018): The strongest model evaluated, and the same architecture used as the adversarial filter during dataset construction. This creates the most interesting tension: can the model used to build the dataset also solve it?
  • Human performance: Five independent crowd workers per question, combined via majority vote. This establishes the ceiling for what the task is supposed to measure.

• Generation budget / compute accounting. The paper does not use a "generation budget" concept in the inference-time compute sense because all models produce a single forward pass per (context, ending) pair, and the four candidates are scored independently. The relevant compute metric for model comparison is pretraining compute, not inference compute. For the scaling analysis in Section 6.2 (Figure 12), the authors estimate pretraining compute in GPU-hours using the Roofline model (Williams et al., 2009), converting reported TPU runtimes to equivalent time on an RTX 2080 Ti GPU. The Roofline model focuses on the bottleneck of loading tensors into GPU memory as the primary determinant of runtime, providing a rough hardware-independent measure of computational cost. These estimates are explicitly labeled as approximations and are used only for the extrapolation analysis, not for any FLOPs-matched comparison (unlike the inference-time compute papers that would follow years later).

• Cross-validation / statistical protocol. There is no cross-validation in the model evaluation phase — models are trained once on the training set and evaluated on the fixed validation and test sets. Cross-validation is used only during the Adversarial Filtering construction phase, where the dummy dataset is randomly split into 80% training / 20% testing on each AF iteration to ensure the resulting negatives are adversarial for any arbitrary split. During AF, BERT-Large is re-initialized from pretrained weights on each iteration and the split is re-randomized, preventing overfitting to a particular partition. The final dataset construction uses two-fold cross-validation in the sense that the AF process iteratively creates new train/test splits, but no cross-validation is applied to the downstream model evaluation. Human evaluation uses 200 examples each from WikiHow and ActivityNet for each validation round (0, 1, or 2 previous validators as shown in Figure 6), with domain-weighted averaging to produce overall human accuracy estimates for mixed-domain splits.

Main Quantitative Results

Overall Model Performance on HellaSwag (Table 1)

The headline result is that no model exceeds 48% accuracy on HellaSwag, while humans achieve >95%. Table 1 presents the comprehensive evaluation across all models and evaluation conditions. BERT-Large, despite having been the adversarial filter used during dataset construction, achieves the highest overall model performance at 47.3% on the test set. This is the central tension: the model that was used to select the adversarial wrong answers cannot solve the resulting dataset.

Breaking this down by evaluation condition:

• Overall test accuracy: BERT-Large leads at 47.3%, followed by OpenAI GPT at 41.7%, BERT-Base at 40.5%, and then a substantial drop to LSTM+BERT-Base at 36.2%. The gap between BERT-Large and the next best model is 5.6 percentage points, indicating that additional capacity beyond 12 layers helps, but the absolute ceiling remains far below human performance (95.6%). Random chance is 25.0%.

• In-domain vs. zero-shot (test set): Every model shows a performance decline on zero-shot categories. BERT-Large drops from 49.7% in-domain to 45.0% zero-shot (a 4.7-point gap). OpenAI GPT drops from 44.0% to 39.3% (4.7 points). BERT-Base drops from 42.8% to 38.3% (4.5 points). The consistency of this ~5-point gap across models of different architectures and scales suggests it reflects a fundamental limitation of fine-tuning rather than a capacity constraint — even the largest model has not learned reasoning patterns that transfer across categories. The gap is slightly larger on ActivityNet (BERT-Large: 51.7% in-domain → 45.0% zero-shot on the overall test set, though the table breaks out ActivityNet test at 3,500 examples and WikiHow test at 6,500 examples, with BERT-Large achieving 51.7% on ActivityNet test and 45.0% on WikiHow test — the domain split and category split are partially confounded because WikiHow has more zero-shot categories).

• Domain differences (test set): WikiHow is substantially harder than ActivityNet for all models. BERT-Large achieves 51.7% on ActivityNet test versus 45.0% on WikiHow test — a 6.7-point gap. OpenAI GPT shows the reverse pattern: 43.8% on ActivityNet versus 40.5% on WikiHow, a smaller 3.3-point gap. BERT-Base: 45.7% vs. 37.7% (8.0 points). The larger gap for BERT and the smaller gap for GPT is notable and suggests that bidirectional encoding may provide a relative advantage on ActivityNet's shorter contexts, while GPT's left-to-right generation bias may be better suited to WikiHow's longer procedural text. The paper speculates (Section 5): "One possibility is that the left-to-right structure of GPT is the right inductive bias for WikiHow — perhaps reasoning bidirectionally over long contexts is too much for a 12-layer transformer to learn." This domain-specific architecture interaction is an intriguing but untested hypothesis.

• Validation vs. test consistency: Performance on validation and test sets is nearly identical across all models (e.g., BERT-Large: 46.7% validation vs. 47.3% test; OpenAI GPT: 41.9% vs. 41.7%), indicating that hyperparameters were not overfit to the validation split and that the reported numbers are reliable.

• The pretraining hierarchy: The progression from fastText (30.9%) through LSTM+GloVe (31.9%), LSTM+ELMo (31.7% — interestingly no improvement over GloVe), ESIM+ELMo (33.6%), LSTM+BERT-Base (35.9%), BERT-Base (39.5%), and BERT-Large (46.7%) shows that pretraining matters enormously — but even the strongest pretrained model remains far below human performance. The fact that ELMo does not improve over GloVe embeddings for the LSTM encoder (31.9% vs. 31.7%) is a striking negative result that contrasts with ELMo's strong performance on SWAG (59% for ESIM+ELMo) and reinforces that HellaSwag's difficulty is not addressable by simply adding contextualized word representations — the challenge requires deeper reasoning that static or shallowly-contextualized embeddings cannot capture.

• Freezing BERT-Base hurts substantially: LSTM+BERT-Base achieves 35.9% overall on validation, compared to 39.5% for fine-tuned BERT-Base — a 3.6-point gap. This demonstrates that end-to-end fine-tuning is important for extracting task-relevant knowledge from pretrained representations, consistent with the SWAG finding that ELMo's frozen representations limited ESIM+ELMo's performance.

• fastText as a strong shallow baseline: The bag-of-words model achieves 30.9% overall accuracy — well above the 25% random baseline. This indicates that word-level lexical features alone provide some discriminative signal, likely because machine-generated endings use words differently from human-written ones even after adversarial filtering. However, the signal is weak enough that even fastText's performance is far below ceiling.

SWAG to HellaSwag Transfer (Figure 9)

The transfer experiments in Section 5.1 and Figure 9 directly test whether solving SWAG implies any ability to solve HellaSwag — and vice versa. The results are stark:

• Training on SWAG, evaluating on HellaSwag: BERT-Large achieves only 34.6% overall accuracy on the HellaSwag validation set. This is only 9.6 points above random chance and far below the 46.7% achieved when training on HellaSwag itself. Breaking this down by domain: 48.4% on ActivityNet (which SWAG training includes) but only 28.0% on WikiHow (which SWAG training does not include). This is near-random performance on the new domain — barely 3 points above chance — demonstrating that BERT learned essentially nothing from SWAG that transfers to WikiHow, despite both datasets requiring commonsense reasoning about event sequences. The SWAG-trained model's WikiHow performance is comparable to fastText (27.7%), a bag-of-words baseline that has no pretraining at all.

• Training on HellaSwag, evaluating on SWAG: BERT-Large achieves 69.0% on SWAG's ActivityNet subset and 74.2% on SWAG's LSMDC subset, for an overall weighted accuracy of approximately 71.4%. While this is substantially below the 86.7% achieved when training on SWAG directly, it is far above the reverse transfer direction. The HellaSwag-trained model retains significant SWAG performance, which the paper attributes to two factors: (1) HellaSwag's ActivityNet training examples partially overlap with SWAG's domain (though with different splits), and (2) the more challenging training objective may force the model to learn some transferable features. However, the 15-point drop relative to in-dataset training indicates that significant dataset-specific learning still occurs.

• The asymmetry is the key finding: SWAG → HellaSwag (34.6%) is far worse than HellaSwag → SWAG (71.4%). This asymmetry demonstrates that SWAG is the "easier" dataset — the features learned on SWAG are more dataset-specific and less transferable, while the features learned on HellaSwag are somewhat more general (though still far from fully transferable). This is exactly what you would expect if SWAG performance is driven by stylistic artifact detection (which doesn't transfer to a dataset where those artifacts have been removed) and HellaSwag performance requires grappling with something closer to genuine reasoning (which partially transfers back to an easier variant of the task).

The domain-level breakdown is particularly informative: the SWAG-trained model achieves 48.4% on HellaSwag ActivityNet (the shared domain) but only 28.0% on HellaSwag WikiHow (the new domain). The ActivityNet number, while above the 34.6% overall, is still far below the 53.7% achieved by the HellaSwag-trained model — meaning that even on the same underlying video caption domain, the model trained on SWAG's ActivityNet examples transfers poorly to HellaSwag's ActivityNet examples. This is the strongest evidence that the two datasets, despite sharing a domain, measure fundamentally different things: SWAG measures LSTM-artifact detection, HellaSwag measures something closer to commonsense reasoning, and success at the former does not imply success at the latter even when the domain is identical.

Qualitative Error Analysis (Figure 10, Table 2)

The paper provides selected qualitative examples showing BERT-Large's predictions along with the model's assigned probabilities for each candidate. These are illustrative rather than systematic, but they reveal characteristic error patterns:

• ActivityNet in-domain (Shaving): BERT correctly selects the ground-truth ending ("then holds up a razor and begins shaving his face") with 100.0% probability. The distractor endings are clearly nonsensical (e.g., "shows himself via the washer and dryer rolling down a towel and scrubbing the floor"), and BERT identifies them confidently. This is the easy case where surface-level lexical matching (shaving → razor → face) is sufficient.

• ActivityNet zero-shot (Sharpening knives): BERT selects an incorrect ending ("uses a sharpener to smooth out the stone using the knife") with 100.0% probability, confusing the tool and object roles — you sharpen a knife with a stone, not a stone with a knife. The correct ending ("stops sharpening the knife and takes out some pieces of paper to show how sharp the knife is") receives 0.0%. This is the characteristic commonsense failure: the wrong answer contains all the right words (sharpener, stone, knife) but gets the physical relationship backwards, and BERT cannot detect the semantic reversal.

• WikiHow in-domain (Youth / Blame technology): BERT selects an ending ("It may also be more believable if you are fully aware that you may be flying at high speed on a plane...") with 61.5% probability, while the correct ending ("Most people, your teacher included, have experienced setbacks due to technological problems. This is a great excuse if you had a paper you needed to type and print.") receives only 29.1%. The paper notes that the wrong answer "has more words that match the context of technology (planes, traffic, laptop), but is incoherent." This is lexical false friends — BERT is attracted by surface-level word overlap (technology → laptop, plane, traffic) while failing to detect the narrative incoherence (why would someone suddenly be "flying at high speed on a plane" while discussing homework excuses?).

• WikiHow zero-shot (Family Life / Call them helpers): BERT selects the wrong ending with 99.9% probability, while the correct ending receives only 0.1%. The correct ending provides a concrete, age-appropriate example of calling a child a "helper," while the wrong ending contains plausible-sounding but nonsensical language about helping during lunch breaks. BERT is fooled by the surface plausibility.

These examples illustrate the core failure mode: BERT performs lexical matching between context and ending words but cannot perform the compositional semantic reasoning needed to detect when the matched words are arranged into an incoherent or physically impossible scenario. This is consistent with the shuffled-text experiments in Section 3.2, which showed that BERT's performance is surprisingly robust to word order destruction — exactly what you would expect from a model that relies heavily on lexical overlap rather than compositional understanding.

Pretraining Scaling Analysis (Figure 12)

Section 6.2 and Figure 12 present an extrapolation from current model performance to estimate what computational resources would be needed to reach human-level accuracy on HellaSwag. This is explicitly a rough estimate, not a controlled experiment, but it provides a sobering perspective on the limits of scaling alone.

• Data points: The plot shows accuracy vs. estimated pretraining GPU-hours for four models: ELMo (the weakest, though ELMo performance on HellaSwag specifically is not reported in Table 1 — the plot likely uses the LSTM+ELMo or ESIM+ELMo results), GPT, BERT-Base, and BERT-Large.

• Fit: A log-linear regression line is fit to these points (accuracy vs. log of pretraining hours), and extrapolated to human-level performance (~95%).

• Result: The extrapolation suggests that reaching human-level performance would require approximately 10^9 GPU-hours of pretraining — equivalent to over 100,000 GPU-years on a modern (at the time) RTX 2080 Ti. The paper frames this as evidence that "without algorithmic or computational improvements," scaling alone is not a viable path to commonsense reasoning.

The caveats are significant and the paper acknowledges them: the Roofline model for converting TPU to GPU hours is approximate, the extrapolation from four data points across different architectures assumes a smooth scaling law that may not hold, and the analysis cannot account for future architectural innovations that might change the scaling exponent. Nevertheless, the 100,000-year estimate serves as a rhetorical device to motivate the paper's central argument — that algorithmic advances (in architecture, pretraining objectives, and perhaps reasoning-specific inductive biases) are necessary because brute-force scaling is impractical.

Ablation Studies and Robustness Checks

Context removal ("Ending Only" condition, Figure 4): When context is removed and BERT-Large sees only the four ending choices, accuracy on SWAG drops from 86.7% to 74.8% — a decline of only 11.9 points. On HellaSwag, the corresponding figures are not reported for BERT-Large specifically (Figure 4 shows "Ending Only" only for SWAG), but the SWAG result demonstrates that roughly 75% of the discriminative signal in SWAG comes from the endings alone. This is the key evidence for the claim that BERT's SWAG performance is dominated by surface-level ending characteristics rather than context-dependent reasoning.

Word shuffling ("Shuffled" condition, Figure 4): On SWAG, when the words within each ending are randomly permuted (but context is provided), BERT-Large accuracy drops from 86.7% to 77.0% — a decline of only 9.7 points. The model was never exposed to shuffled text during pretraining, yet it adapts to this setting during fine-tuning and achieves performance only marginally below the full-context unshuffled condition. This demonstrates that word order information is surprisingly dispensable for the discrimination task, further evidence that BERT is performing lexical matching rather than compositional understanding. On HellaSwag, the "Shuffled" condition reduces BERT-Large accuracy to 41.4% (from 46.7%), a smaller relative decline than on SWAG, suggesting that HellaSwag relies somewhat less on lexical matching — but the fact that shuffled-text performance remains at 41.4% (substantially above chance) indicates that lexical-level signals persist even in HellaSwag.

Combined context removal and shuffling ("Shuffled+Ending Only," Figure 4): On SWAG, with neither context nor word order, BERT-Large still achieves 60.4% accuracy — higher than ESIM+ELMo's full-dataset performance of 59%. On HellaSwag, the same condition yields only 31.6% for BERT-Large — far below the full-model 46.7% but still above the 25% random baseline. The contrast between SWAG (60.4%) and HellaSwag (31.6%) under this maximally degraded condition quantifies the extent to which HellaSwag has removed the surface-level artifacts that BERT exploits on SWAG. The residual 6.6 points above chance on HellaSwag under this condition may reflect lingering distributional differences between human and GPT-generated text that even adversarial filtering could not fully eliminate, or it may reflect genuine lexical biases that are correlated with correctness (certain words simply appear more often in correct endings).

Generator quality ablation (Figure 5, left): Comparing Adversarial Filtering convergence when using the original SWAG LSTM generator versus OpenAI GPT, with BERT-Large as the discriminator in both cases. With the LSTM generator, BERT accuracy converges to approximately 75% — the LSTM's generations are so stylistically distinct from human text that even iterative adversarial selection cannot close the gap. With GPT, BERT accuracy drops below 30%. This is the key evidence that SWAG's vulnerability to BERT was caused by insufficient generator quality, not by a fundamental flaw in the AF concept. A sufficiently strong generator (GPT, a 12-layer transformer) combined with a strong discriminator (BERT-Large) can produce a genuinely challenging dataset.

Ending length ablation (Figure 5, right): Varying WikiHow generation length from one to three sentences while keeping the AF discriminator (BERT-Large) constant. One-sentence endings converge to approximately 35% discriminator accuracy, two-sentence to approximately 40%, three-sentence to approximately 50%. This establishes the Goldilocks zone: two-sentence endings strike the balance where the text is long enough for humans to detect commonsense violations (93.5% pre-validation accuracy) but short enough that BERT cannot reliably discriminate real from generated (41.1% accuracy). Three sentences tip the balance too far in the discriminator's favor, likely because the additional length accumulates detectable stylistic signals across multiple sentences.

Adversarial filter strength ablation (Figure 11): Creating alternative versions of HellaSwag's WikiHow subset using different adversarial filters during AF, then evaluating each version with BERT-Large. The filters range from the shallow stylistic ensemble used in SWAG (Zellers et al., 2018) through ELMo+LSTM, GPT, BERT-Base, and BERT-Large. The results show a clear gradient: stronger filters produce harder datasets. The stylistic ensemble (weakest filter) produces a dataset where BERT-Large achieves 83.0% accuracy — nearly saturating the benchmark. An ELMo+LSTM filter produces 71.4% BERT-Large accuracy. GPT as filter produces 64.8%. BERT-Base as filter produces 63.0%. BERT-Large as filter (the actual HellaSwag configuration) produces 41.1% BERT-Large accuracy. The key finding is that using a slightly weaker filter than the final evaluator (BERT-Base vs. BERT-Large — a 3x parameter gap) still leaves substantial room for the stronger model, with BERT-Large achieving 63.0% on the BERT-Base-filtered dataset. The paper interprets this as evidence that the gap between filter and evaluator matters, but the task remains unsolved: "increasing the gap between the filter and the final discriminator is not enough to solve the task" because even with a 3x parameter advantage, accuracy is only 63%. To approach human-level performance would require a far larger gap — or fundamentally different model capabilities.

Human validation rounds (Figure 6): Tracking both human and BERT accuracy through successive rounds of human validation (0, 1, or 2 annotator rounds before final dataset selection). On ActivityNet, initial human accuracy is only 60.0% (before any validation), rising to 85.0% after 1 round and 94.0% after 2 rounds, while BERT rises from 25.5% to 48.4% to 57.1%. On WikiHow, the initial gap is already large: humans at 93.5% and BERT at 41.1% with 0 annotator rounds, rising to 95.5% / 45.4% and 96.5% / 46.0%. The validation process widens the human-machine gap by removing generated endings that are confusing to humans (false negatives that slipped through AF), demonstrating that AF alone produces some endings that are adversarially effective against both humans and machines — human validation is essential to filter these out and produce the >95% human / <48% machine split that characterizes the final dataset.

Transfer experiments (Figure 9): Training BERT-Large on one dataset and evaluating on the other (SWAG ↔ HellaSwag), described in detail above. The key robustness insight is that performance transfers poorly in both directions relative to in-dataset training, but the asymmetry (SWAG→HellaSwag is worse than HellaSwag→SWAG) confirms that HellaSwag is the genuinely harder task that requires more transferable capabilities. The domain-level breakdown within each transfer direction — particularly the 28.0% accuracy on HellaSwag WikiHow when trained on SWAG (which lacks WikiHow) versus 48.4% on HellaSwag ActivityNet (where domain overlap exists) — confirms that domain transfer compounds the difficulty but does not fully explain it.

Zero-shot category generalization (Table 1): The consistent ~5-point drop on zero-shot categories across all models, described above. This is a robustness check showing that the difficulty of HellaSwag is not an artifact of train-test mismatch — even in-domain accuracy (where the model sees the same categories during training and testing) is below 50%. The zero-shot drop provides a lower bound on how much of the model's performance comes from category-specific memorization vs. transferable reasoning.

Critical Assessment

The experiments presented in this paper collectively support a central narrative: HellaSwag is genuinely hard for state-of-the-art models, BERT's success on SWAG was largely attributable to surface-level artifact detection, and the Adversarial Filtering methodology, when scaled to match model capability, produces datasets that resist statistical exploitation. However, the evidence supports these claims with varying degrees of strength, and several important questions are left unaddressed or only partially answered.

On the claim that BERT's SWAG performance reflects surface learning rather than reasoning: The evidence from Section 3 is the strongest in the paper. The progression of ablation experiments — context removal (86.7% → 74.8%), word shuffling (86.7% → 77.0%), and the combination (86.7% → 60.4%) — is methodical and devastating. Each ablation strips away information that would be essential for genuine reasoning, yet substantial performance survives. The learning curve in Figure 3 — 64 examples to surpass ELMo — is incompatible with slow acquisition of general reasoning capabilities. And Figure 5 (left) provides the mechanistic explanation: the LSTM generator left detectable artifacts that BERT could exploit. These experiments collectively make a compelling case.

However, a limitation is that the ablation experiments are performed only on SWAG, not on HellaSwag. The paper reports "Shuffled" (41.4%) and "Shuffled+Ending Only" (31.6%) for BERT-Large on HellaSwag in Figure 4, but does not provide the full suite of ablations (Ending Only, the learning curve with varying training data) for HellaSwag. This leaves open the question of whether BERT's remaining 46.7% on HellaSwag is itself partially attributable to some new form of surface learning or artifact detection. The fact that shuffled, contextless performance is only 31.6% on HellaSwag (vs. 60.4% on SWAG) suggests that surface-level signals are much weaker, but not entirely absent. A full set of HellaSwag ablations would have strengthened the paper's claim that HellaSwag genuinely measures reasoning rather than a different set of surface features.

On the claim that HellaSwag is "trivial for humans (>95% accuracy) yet challenging for machines (<48%)": This claim is well-supported by the reported numbers in Table 1 and Figure 6. Human performance (95.6% test) is measured with five independent workers per question, combined via majority vote, on a sample of 200 examples per domain per validation round — a reasonable protocol, though 200 is a relatively small sample. The 95.6% figure represents the achieved human ceiling, but it is worth noting that this is after multiple rounds of human validation that removed examples where the ground truth itself was ambiguous (Appendix E mentions that "on ActivityNet, we found that there are some contexts where the ground truth answer isn't liked by workers"). This filtering means that the final dataset is selected for high human agreement — it is not a random sample of the original ActivityNet or WikiHow distributions. The human accuracy number on the original, unfiltered distribution would almost certainly be lower (the paper reports 60% for ActivityNet before validation, Figure 6). This doesn't invalidate the claim — the final dataset is what models are evaluated on — but it does mean that "trivial for humans" is partly an artifact of aggressive quality filtering, not an intrinsic property of the domain. Models are being evaluated on a curated subset of examples where humans happen to agree strongly, which may also be a subset where the reasoning required is more straightforward or less ambiguous.

On the claim that Adversarial Filtering produces a dataset that is challenging "even when models are evaluated on items from the training distribution": This is the most important claim in the paper, and the evidence is mixed. The overall test accuracy of 47.3% for BERT-Large (Table 1) clearly shows that the dataset is not solved. The zero-shot vs. in-domain comparison shows that performance drops on held-out categories, but even in-domain performance is only 49.7% — far from ceiling. However, the paper does not fully address the question of why in-domain performance is so low. One possibility (the paper's preferred interpretation) is that the task genuinely requires commonsense reasoning that models lack. An alternative possibility is that 40,000-50,000 training examples is simply insufficient for BERT to memorize the necessary patterns, and that a larger training set would allow surface-level strategies to succeed on HellaSwag just as they did on SWAG. The paper shows a learning curve for SWAG (Figure 3) demonstrating that BERT plateaus around 16,000 examples, and the HellaSwag training set is presumably larger (roughly 50,000 examples, though the exact size is not stated), but no learning curve is shown for HellaSwag. Without this, we cannot rule out the possibility that HellaSwag is hard primarily because of data quantity rather than task nature — perhaps with 200,000 examples, BERT would find new surface-level shortcuts. This is a significant omission.

On the scaling extrapolation (Figure 12) and the claim that reaching human performance would require 100,000 GPU-years: This is the weakest part of the paper's empirical work, and the authors explicitly label it as a "rough estimate." The extrapolation from four data points spanning different architectures (ELMo, GPT, BERT-Base, BERT-Large) assumes a smooth log-linear relationship between pretraining compute and HellaSwag accuracy, but there is no theoretical reason to expect this relationship to hold across different model families with different pretraining objectives, architectures, and training data. The Roofline model for TPU-to-GPU conversion is approximate. More fundamentally, the training data is not held constant — GPT was pretrained on BookCorpus, BERT on Wikipedia+BookCorpus, ELMo on a different corpus — so the "pretraining hours" axis conflates model scale, architecture, and data quantity/quality. The 100,000-year figure is best understood as a rhetorical device to motivate the need for algorithmic innovation, not as a serious computational forecast. This is not necessarily a flaw — the paper is transparent about the limitations — but readers should not treat this number as a meaningful empirical finding.

On the generator quality ablation (Figure 5, left) supporting the claim that SWAG's failure was due to the LSTM generator: This experiment is clean and convincing. Running AF with the same discriminator (BERT-Large) but different generators (LSTM vs. GPT) isolates the generator quality variable, and the 75% vs. <30% convergence accuracy difference is stark. However, the experiment is not perfectly controlled: the LSTM was trained by Zellers et al. (2018) on ActivityNet/LSMDC captions, while GPT was fine-tuned by the current authors on the same or similar data. Differences in training data, hyperparameters, and tokenization could contribute to the performance gap beyond the architectural difference. A controlled comparison with an LSTM and GPT trained on identical data with identical compute budgets would be more definitive, though the paper's claim does not require this level of control — the point is simply that stronger generators produce harder datasets, which the existing comparison amply demonstrates.

On the adversarial filter strength ablation (Figure 11): This experiment is well-designed and addresses a natural question: could a future model with, say, 10x the parameters of BERT-Large simply overawe HellaSwag? The finding that BERT-Large achieves 63.0% on the BERT-Base-filtered version (where it has a 3x parameter advantage over the filter) suggests that parameter scaling alone is insufficient to close the gap to human performance. However, the experiment does not test what happens when the evaluator has a much larger advantage — e.g., a hypothetical BERT-Huge with 10x BERT-Large's parameters evaluated on the standard HellaSwag (filtered by BERT-Large). Extrapolating from the 63.0% result, one might guess that a 10x parameter advantage would push accuracy well above 63% — perhaps into the 70-80% range, still far from human but substantially higher than 47.3%. The paper's conclusion that "increasing the gap between the filter and the final discriminator is not enough to solve the task" is therefore supported for modest gaps (3x parameters) but not for larger ones. This is acknowledged implicitly in the paper's vision of co-evolving benchmarks — HellaSwag is expected to eventually be solved, at which point a new version must be built — but the quantitative relationship between filter-evaluator gap and final accuracy is not explored beyond the single 3x data point.

Missing experiments that would have strengthened the paper:

1. Learning curve for HellaSwag: The paper shows (Figure 3) that BERT's SWAG accuracy varies dramatically with training set size, and uses this to diagnose surface learning. A comparable curve for HellaSwag would reveal whether more training data enables new surface-level shortcuts or whether performance genuinely plateaus, supporting or undermining the claim that HellaSwag requires reasoning.

2. Ensemble or multi-model evaluation: The AF construction uses BERT-Large as the sole discriminator. What would happen if an ensemble of different architectures (GPT, BERT, ELMo-based models) were used as the adversarial filter? Would the resulting dataset be uniformly harder for all architectures, or would it be specifically adversarial to BERT-like models while leaving other architectures less affected? Table 1 shows that GPT outperforms BERT on WikiHow — does this reflect a genuine inductive bias advantage, or is it an artifact of BERT-Large being the filter (making the dataset maximally confusing for BERT-like bidirectional models while leaving unidirectional models with an advantage)? An ablation using GPT as the AF discriminator would address this.

3. Fine-tuning data size vs. pretraining data: The paper argues that BERT's SWAG performance relies on fine-tuning to detect dataset-specific artifacts. One way to test this would be to evaluate BERT on HellaSwag without any fine-tuning, using only the pretrained model's representations (e.g., via a linear probe or zero-shot prompting, though zero-shot prompting was not a standard paradigm at the time). If performance remains near chance without fine-tuning, that would further support the surface learning diagnosis. If performance is surprisingly high, it would suggest that pretraining alone encodes substantial commonsense knowledge that fine-tuning fails to fully extract.

4. Effect of human validation on model performance: Figure 6 shows that human validation increases BERT accuracy (e.g., on ActivityNet, from 25.5% to 57.1% through 2 validation rounds). This is because human validation removes generated endings that are too confusing — false negatives that even humans find plausible. The resulting dataset is one where the wrong answers are clearly wrong to humans but confusing to BERT. An interesting ablation would be to keep the false negatives in the dataset and measure how human and model accuracy co-vary as a function of the false negative rate. This would quantify the tradeoff between human and machine difficulty and provide a more continuous picture of the Goldilocks zone.

5. Calibration and confidence analysis: The qualitative examples in Figure 10 show BERT assigning >99% probability to demonstrably wrong answers (e.g., "uses a sharpener to smooth out the stone using the knife" at 100.0%). A systematic analysis of model calibration — how often high-confidence predictions are wrong — would strengthen the argument that models are not merely uncertain but confidently wrong, which is a stronger indictment of their reasoning capabilities than low accuracy alone.

Conditional boundaries on the paper's claims:

• The claim that "commonsense NLI remains unsolved" holds for the specific task formulation and domains represented in HellaSwag — four-way multiple choice over ActivityNet and WikiHow continuations. The paper does not claim that models cannot perform any commonsense reasoning, only that their success on SWAG did not constitute it. Whether models possess commonsense capabilities that are not elicited by this particular evaluation format is an open question that the paper does not address.

• The claim that "models operate as rapid surface learners" is demonstrated most clearly for BERT on SWAG. The evidence on HellaSwag is consistent with this picture but does not prove it — the 5-point zero-shot drop and the 31.6% shuffled baseline suggest that some surface learning persists, but the lower absolute numbers mean the residual reasoning component (if any) is harder to isolate.

• The Goldilocks zone claim — that two-sentence WikiHow endings occupy the optimal difficulty region — is demonstrated empirically for BERT-Large as the discriminator and GPT as the generator. The exact boundaries of the zone would shift with different model combinations, and the paper does not provide a general method for finding it beyond the empirical sweep shown in Figure 5 (right).

• The scaling extrapolation to 100,000 GPU-years is explicitly labeled as approximate and depends on the assumption that log-linear scaling across different architectures holds — an assumption the paper does not validate and likely cannot validate given the available data.

Overall, the experimental section provides strong evidence for the paper's core narrative — HellaSwag is genuinely hard, SWAG was solved through surface learning, and Adversarial Filtering with strong generators and discriminators produces robust benchmarks — while leaving open important questions about how much of HellaSwag's difficulty is due to data quantity, how the results would change with different adversarial filter architectures, and what the exact scaling relationship is between model capacity and benchmark performance. These limitations do not undermine the paper's contributions, but they define the boundaries within which its claims should be understood.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Unaccounted For

The paper presents HellaSwag as a dataset where strong models fail despite being trained on in-distribution examples, but the process of making the dataset that hard — via iterative Adversarial Filtering with BERT-Large — involves substantial computational overhead that is invisible in the final benchmark scores. Constructing HellaSwag required repeatedly training BERT-Large from scratch on the AF dummy training set, generating massive pools of candidate endings from a fine-tuned GPT, and running multiple rounds of human validation. These costs are not amortized or accounted for in any of the paper's metrics; they are simply the price of dataset creation. This matters for the co-evolutionary vision the paper advocates in Section 6.3, where each time a benchmark is solved, it should be rebuilt with the solvers as adversaries. If each rebuild requires training dozens of copies of the best available model (BERT-Large for AF iterations) and generating millions of candidate endings from a fine-tuned GPT, the computational cost of dataset construction could rapidly exceed the cost of training the models being evaluated. The paper does not estimate the total FLOPs or GPU-hours consumed by AF, nor does it discuss how this cost scales with model size. For the co-evolutionary paradigm to be practical — rather than merely aspirational — the cost of adversarial dataset construction must be substantially lower than the cost of training the models that the dataset is meant to challenge, and the paper provides no evidence that this is the case. The authors do not acknowledge this as a limitation, and no mitigation is proposed beyond the implicit assumption that dataset creation is a one-time cost amortized over many downstream evaluations — an assumption that weakens as the co-evolution cycle accelerates.

Evidence: The AF procedure (Appendix A) requires re-initializing and fine-tuning BERT-Large on every iteration, with hyperparameter randomization, across multiple iterations until convergence (Figure 5 shows roughly 30–40 iterations for ActivityNet and WikiHow). The generator (GPT) must be fine-tuned per-domain (Appendix B) and used to oversample massive pools of candidates. No cost analysis is provided.

Mitigation status: Not addressed. The paper frames the dataset creation process as completed work and does not discuss the computational economics of repeating it.

The Dataset Construction Relies on Models That Were State-of-the-Art Only at the Time of Writing

HellaSwag's adversarial difficulty is fundamentally relative to the models used in its construction — OpenAI GPT as the generator and BERT-Large as the adversarial filter. The paper explicitly acknowledges (Section 6.3) that the dataset will eventually be solved and should then be rebuilt. However, what the paper does not fully explore is whether HellaSwag's difficulty is specifically tuned to BERT-like bidirectional transformers (since BERT-Large was the discriminator during AF) and whether models with radically different architectures, pretraining objectives, or scale would experience the same difficulty. Figure 11 partially addresses this by showing that using weaker filters (stylistic ensemble, ELMo+LSTM, BERT-Base) produces versions of the dataset where BERT-Large achieves higher accuracy (up to 83.0%), confirming that filter strength matters. But the reverse experiment — using BERT-Large as the filter and evaluating a hypothetical model with 10× BERT-Large's capacity — is not performed and cannot be performed at the time of writing. The paper's headline claim that "state-of-the-art models struggle (<48%)" is therefore strictly time-bound: it holds for models available in early 2019, and the paper provides no quantitative bound on how much larger or architecturally different a model would need to be to reach, say, 80% or 90% accuracy. The scaling extrapolation in Figure 12 attempts to address this by projecting from ELMo through BERT-Large, but the extrapolation spans different architectures with different pretraining data and objectives, making it unreliable as a forecast for any specific future model. This limitation does not invalidate HellaSwag's value as a benchmark at its release time, but it means the paper's methodological contribution — adversarial dataset construction — is not accompanied by a method for estimating how long the resulting dataset will remain challenging, which limits its practical utility for benchmark designers planning the co-evolutionary cycle.

Evidence: Figure 11 shows that the gap between filter and evaluator matters — BERT-Large achieves 63.0% when BERT-Base is the filter (a ~3× parameter gap). But the experiment does not test gaps larger than 3×, so the paper cannot predict what happens at 10× or 100× scale differences. Figure 12's extrapolation is explicitly labeled as a "rough estimate" and relies on cross-architecture comparison.

Mitigation status: The paper acknowledges that HellaSwag will eventually be solved (Section 6.3) and advocates rebuilding it, but does not provide a method for estimating when that will happen or how much larger models would need to be. The co-evolutionary vision is proposed without a cost model for the evolution cycle.

Single Task Formulation: Four-Way Multiple Choice

All of the paper's claims about machine commonsense reasoning are based on a single task format: four-way multiple choice over (context, ending) pairs. This is a narrow operationalization of "commonsense natural language inference." A model that performs poorly on HellaSwag might still possess substantial commonsense reasoning capabilities that are not measured by this specific format — for instance, the ability to generate plausible continuations (rather than merely discriminate among provided options), to answer open-ended questions about physical scenarios, or to detect commonsense violations in longer narratives. Conversely, a model that eventually achieves high accuracy on HellaSwag might have learned format-specific heuristics (e.g., "in a four-way choice among machine-generated distractors, the most lexically conservative option tends to be correct") rather than genuine reasoning. The paper's forensic analysis of SWAG in Section 3 demonstrates exactly this kind of format-specific exploitation — BERT learned to detect LSTM-generated stylistic artifacts rather than reason about event plausibility — but the same analysis is not applied to HellaSwag itself. The paper cannot rule out the possibility that the remaining 46.7% BERT-Large accuracy on HellaSwag reflects a new set of format-specific patterns rather than partial commonsense ability. The four-way choice format, while convenient for evaluation and consistent with SWAG, inherently confounds reasoning ability with test-taking strategy in a way that open-ended generation or free-form plausibility judgments would not. The paper does not validate its findings against alternative task formulations, nor does it discuss whether the specific difficulty of HellaSwag transfers to other ways of measuring commonsense reasoning.

Evidence: The task format is described in Section 4 and Appendix C. All models are trained and evaluated exclusively in the four-way multiple-choice setup. The qualitative examples (Figure 10, Table 2) illustrate failure modes but do not test whether models would fail differently under alternative evaluation formats. No experiment varies the number of choices (e.g., binary discrimination) or the response format (e.g., generation, ranking, plausibility scoring).

Mitigation status: Not addressed. The paper treats four-way multiple choice as the natural operationalization of commonsense NLI, inherited from SWAG, and does not discuss format-specific confounds.

The Dataset Is Constructed from Only Two Domains, Both Procedural

HellaSwag draws its examples from exactly two source domains: ActivityNet video captions (describing physical activities) and WikiHow articles (describing procedural how-to instructions). Both domains are fundamentally procedural — they describe sequences of actions or events with temporal order — and both are grounded in physical or practical scenarios. This domain coverage, while much broader than SWAG (which used only video captions), is still narrow relative to the full scope of "commonsense reasoning" as humans understand it. Commonsense encompasses social reasoning (why a person might be upset), psychological reasoning (what someone believes or intends), counterfactual reasoning (what would have happened if), taxonomic reasoning (that a dog is an animal), and quantitative reasoning (that a cup cannot hold a gallon), among many other types. HellaSwag's examples, by virtue of their source domains, heavily emphasize physical and procedural commonsense — what happens next in a sequence of actions, what step follows in a how-to guide. The paper does not claim to cover all types of commonsense, but it also does not discuss the extent to which its findings might be specific to procedural, physically-grounded domains. A model that fails on HellaSwag might succeed on a dataset testing social commonsense, and a model that solves HellaSwag might still lack social or psychological reasoning capabilities. The paper's broad framing — "commonsense natural language inference" in the abstract and introduction — implies a more general measurement of commonsense than the domain coverage supports.

Evidence: Section 4.1 and 4.2 describe the two source domains. Figure 8 and Figure 13 show the diversity of categories within each domain, but all categories fall under physical activities (ActivityNet) or practical how-to tasks (WikiHow). The paper does not analyze which types of commonsense reasoning HellaSwag requires versus which it omits.

Mitigation status: Partially acknowledged. The paper describes the domains in detail and does not claim that HellaSwag covers all commonsense phenomena. However, it does not explicitly discuss the domain coverage as a limitation or suggest that future work should extend the AF methodology to social, psychological, or abstract reasoning domains.

Human Validation Quality Control Is Necessary for the Goldilocks Zone and May Not Scale

The human validation stage (Section 4.3, Appendix E) is essential for achieving HellaSwag's defining property: >95% human accuracy with <48% machine accuracy. Without human validation, the AF-generated dataset contains many false negatives — machine-generated endings that are actually plausible to humans — as shown in Figure 6, where ActivityNet human accuracy starts at only 60% before validation. The multi-round human filtering process, with its probabilistic worker model, iterative replacement of plausible-sounding negatives, and selection of the highest-agreement examples, is what creates the sharp human-machine gap. This reliance on extensive, high-quality human annotation has several consequences the paper does not fully address. First, it makes the AF + human validation pipeline expensive and slow — each round requires recruiting qualified crowd workers, monitoring their performance, and iteratively replacing examples. Second, it introduces a dependency on worker quality that may be difficult to replicate: the paper used a qualification HIT to filter workers, paid up to 37 cents per HIT, and disqualified workers who preferred generated endings over real ones (Appendix E). These quality-control measures are described but their necessity is not ablated — the paper does not show what human accuracy would be with cheaper, less curated annotation or with fewer validation rounds. Third, the requirement that human accuracy be >90% constrains the types of domains and generation lengths that can be used: Figure 5 (right) shows that one-sentence ActivityNet endings converge to near-random for BERT (making them attractive adversarially), but human accuracy on those short endings is only 60% before validation (Figure 6). The Goldilocks zone requires domains and lengths where human accuracy is naturally high, which may not exist for all types of reasoning tasks. For abstract, ambiguous, or subjective domains where even humans disagree, the AF + human validation pipeline may not produce a usable benchmark because the human ceiling is too low to create a meaningful gap.

Evidence: Figure 6 shows human accuracy rising dramatically through validation rounds (ActivityNet: 60% → 85% → 94%). Appendix E describes the worker qualification process and the probabilistic model for aggregating judgments. The paper keeps only the "best" examples (25k ActivityNet, 45k WikiHow) based on human agreement scores.

Mitigation status: The paper describes the human validation protocol in detail (Appendix E) but does not frame the reliance on high-cost, high-quality annotation as a limitation. The co-evolutionary vision in Section 6.3 proposes rebuilding datasets as models improve, but the human validation cost would recur with each rebuild. No method for automating or reducing this cost is proposed.

The Scaling Extrapolation to 100,000 GPU-Years Is Not Empirically Grounded

The paper's most provocative quantitative claim — that reaching human-level performance on HellaSwag would require on the order of 10^9 GPU-hours, or over 100,000 GPU-years (Section 6.2, Figure 12) — is presented as evidence that algorithmic innovation is necessary because scaling alone is impractical. However, the extrapolation rests on assumptions that the paper does not validate and that are unlikely to hold. The four data points used in the fit (ELMo, GPT, BERT-Base, BERT-Large) come from models with different architectures (bi-LSTM with separate language model, left-to-right transformer decoder, bidirectional transformer encoder), different pretraining objectives (language modeling vs. masked language modeling + next-sentence prediction), different pretraining corpora (BookCorpus vs. Wikipedia+BookCorpus), and different tokenization schemes. The "pretraining hours" axis is estimated via the Roofline model (Williams et al., 2009) applied to reported TPU runtimes, converting to an RTX 2080 Ti equivalent — a conversion the authors themselves label "a rough estimate." Fitting a log-linear regression through points that differ on multiple dimensions (architecture, objective, data, tokenization) and extrapolating three orders of magnitude beyond the largest data point (BERT-Large, ~10^6 hours) to reach 10^9 hours is not a meaningful prediction; it is a curve drawn through noise and projected far beyond its support. The paper explicitly acknowledges the roughness of the estimate, but the 100,000-year figure is presented prominently in the discussion as a motivation for algorithmic innovation, and it risks being taken as a quantitative finding by readers who do not examine the methodology. A more defensible analysis would require holding the architecture, objective, and data constant while varying only scale (e.g., a BERT-family scaling curve with models at multiple sizes), which the paper does not have. The extrapolation therefore does not provide evidence that scaling cannot work — it only shows that the available data points, which confound scale with architecture and objective, happen to fall on a line that projects to 10^9 hours at human-level accuracy. A model with a different architecture or training objective could have a different scaling exponent and reach human-level performance at substantially lower cost, a possibility the paper's analysis cannot exclude.

Evidence: Section 6.2 and Figure 12. The paper states the extrapolation is "a rough estimate" and acknowledges using the Roofline model for TPU-to-GPU conversion. The four data points span different model families with different training paradigms. The extrapolation covers three orders of magnitude beyond the largest training run.

Mitigation status: The paper explicitly labels the estimate as rough and the analysis as investigating "the limits" of scaling. However, it does not discuss the confounds (architecture, objective, data) that make the extrapolation unreliable, nor does it suggest what a properly controlled scaling analysis would require. The 100,000-year figure is presented as a motivation for the paper's co-evolutionary vision, but the vision does not depend on this particular number and would be equally well-motivated by the qualitative observation that BERT-Large achieves only 47.3% accuracy despite being the model used to build the dataset.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes the relationship between NLP benchmarks and the models evaluated on them. Prior to HellaSwag, the default assumption in the field was that when a state-of-the-art model achieves near-human performance on a benchmark, the underlying capability is approximately solved. SWAG appeared to follow this trajectory: BERT-Large reached 86.7%, within 1.3 points of human performance, and the natural interpretation — captured by the New York Times headline quoted in the paper — was that machines could now perform commonsense natural language inference. The paper's forensic dismantling of this assumption is what makes its contribution a methodological reframing rather than merely a harder dataset release.

The reframing consists of three interlocking insights, each of which changed how subsequent work approached evaluation:

1. Surface learning is the default explanation for benchmark saturation, not genuine capability acquisition. The paper's dissection of BERT on SWAG — context removal leaves 74.8% accuracy, word shuffling leaves 77.0%, and the combination of both leaves 60.4%, higher than the previous state-of-the-art on the full task — provides a concrete, replicable diagnostic for distinguishing reasoning from pattern matching. This diagnostic travels: any benchmark where performance is robust to scrambled inputs or missing premises is measuring something other than what it claims to measure. The paper gives the field a vocabulary ("rapid surface learners," "dataset-specific distributional biases," "annotation artifacts") and a methodology (ablating context, structure, and their combination) for applying this diagnostic to new benchmarks. After HellaSwag, it became standard practice to ask not just "what accuracy does the model achieve?" but "what is the model actually learning, and how do we know?"

2. Adversarial Filtering is not a one-time construction technique but a co-evolutionary process that must be re-applied with each generation of models. This is the paper's most distinctive conceptual move. SWAG's AF was applied once, using an LSTM generator and shallow discriminators matched to ELMo-era models, and the resulting dataset was static. When BERT arrived — with vastly greater capacity for detecting distributional artifacts — the static dataset collapsed. The paper's demonstration that re-running AF with a stronger generator (GPT instead of LSTM) and a stronger discriminator (BERT-Large instead of a stylistic ensemble) produces a genuinely challenging dataset even for the discriminator itself (BERT-Large scores only 47.3% on the dataset it helped construct) is the empirical proof that AF, properly scaled, works. The implication is that benchmarks must be living artifacts that are rebuilt whenever the models they evaluate catch up to the models used in their construction. This is a paradigm shift in how the field thinks about benchmark lifecycle — from "release once, track progress" to "co-evolve with the state-of-the-art."

3. The Goldilocks zone provides a design principle for finding the boundary between human and machine capability. Prior work on adversarial datasets (both in NLP and in vision) focused on making examples hard for machines, often at the cost of making them hard for humans as well. The paper's systematic variation of generation length (Figure 5, right) reveals that difficulty is not monotonic: one-sentence endings are hard for machines but also ambiguous for humans, three-sentence endings become easier for machines as surface-level signals accumulate, and two-sentence endings sit in the sweet spot where human accuracy is naturally high and machine accuracy remains low. The Goldilocks zone is not a universal constant — it depends on the generator, the discriminator, and the domain — but the concept is transferable: for any given state of model capability, there exists a level of task complexity where the human-machine gap is maximized, and finding that level is the central design challenge for benchmark construction. This insight has influenced subsequent work on adversarial dataset creation far beyond commonsense NLI.

The paper also resolves a genuine contradiction in the literature. Prior to HellaSwag, there was an apparent tension: BERT achieved near-human performance on SWAG (suggesting commonsense NLI was solved), yet models were known to fail on out-of-distribution examples and to exploit annotation artifacts in other NLI datasets. The paper's diagnosis — that BERT solved SWAG by detecting LSTM-generated stylistic artifacts, not by reasoning — reconciles these observations. BERT's success was real but misattributed; it demonstrated sophisticated surface learning on a flawed benchmark, not commonsense reasoning. This resolution is more than retrospective cleanup; it provides a falsifiable model for why future benchmarks might also be saturated without genuine progress, motivating the co-evolutionary approach.

Research directions that become more attractive:

• Adversarial dataset construction with the best available models becomes a first-class research activity rather than an afterthought to model development. The paper implicitly argues that the field should allocate substantial resources to benchmark creation, using models at least as strong as those being evaluated.
• Diagnostic evaluation suites that go beyond aggregate accuracy — measuring performance under context removal, word scrambling, domain shift, and category generalization — become essential companions to any new dataset release, following the template established in Section 3.
• Understanding what pretrained models actually learn during fine-tuning becomes urgent. The paper shows that fine-tuning on SWAG primarily teaches BERT to detect LSTM artifacts; what does fine-tuning on HellaSwag teach? What does fine-tuning on any given dataset teach? The surface-learning hypothesis provides a framework for investigating these questions systematically.
• Architecture-specific inductive biases for reasoning gain importance. The paper notes that GPT marginally outperforms BERT on WikiHow (40.5% vs. 37.7% for BERT-Base, Table 1), speculating that left-to-right generation bias may be better suited to procedural text. This hints that architectural choices matter for reasoning in ways that are not captured by pretraining scale alone.

Research directions that become less attractive:

• Incremental improvements on static benchmarks without accompanying diagnostic evaluation become harder to interpret. A model that achieves 52% on HellaSwag (up from BERT-Large's 47.3%) has made progress, but without knowing why — is it better reasoning, or better detection of residual GPT artifacts? — the field cannot distinguish meaningful from superficial advances.
• Scaling alone as a path to commonsense reasoning is challenged, though not refuted. The extrapolation in Figure 12 is explicitly rough, but the qualitative finding — that BERT-Large, trained on 3.3 billion words, achieves less than 50% accuracy on a task humans find trivial — suggests that scale improvements within the existing paradigm are unlikely to close the 48-point gap to human performance. The paper nudges the field toward algorithmic innovation (better architectures, reasoning-specific objectives, structured knowledge integration) rather than purely computational scaling.

Follow-Up Research This Work Enables

Scaling laws for adversarial dataset difficulty. This paper provides a single data point: BERT-Large (340M parameters) as the adversarial filter produces a dataset where BERT-Large achieves 47.3% accuracy. What is the quantitative relationship between filter capacity and resulting dataset difficulty? A follow-up study could construct multiple versions of HellaSwag using BERT-Tiny, BERT-Medium, BERT-Base, and BERT-Large as adversarial filters (keeping the generator and domain constant), then evaluate each version with a held-out BERT-Huge or a model from a different family entirely. The resulting curve — filter parameters vs. evaluator accuracy — would provide the first scaling law for adversarial dataset construction, directly testing whether the 63.0% accuracy BERT-Large achieves on the BERT-Base-filtered dataset (Figure 11, though the number there is 63.0% for BERT-Large evaluating a BERT-Base-filtered version — wait, Figure 11 shows BERT-Large accuracy after AF, and the "before AF" / "after AF" columns require careful reading) generalizes to larger gaps. The paper's co-evolutionary vision requires knowing how much bigger the next filter must be to keep the dataset challenging; this experiment would provide the first empirical answer.

Generator-discriminator symmetry: what happens when the AF discriminator is not BERT? The paper uses BERT-Large as the sole adversarial filter, which raises the question of whether HellaSwag is specifically adversarial to bidirectional transformer encoders. A direct follow-up would construct parallel versions of HellaSwag using different architectures as the AF discriminator — GPT (left-to-right decoder), XLNet (permutation language model), or T5 (encoder-decoder) — and evaluate all versions with all architectures. If BERT performs worst on the BERT-filtered version and GPT performs worst on the GPT-filtered version, this would reveal an architecture-specific overfitting effect in AF that the paper does not test. If, instead, all models perform uniformly poorly on all versions, the dataset's difficulty is architecture-independent, strengthening the claim that it measures reasoning rather than architectural vulnerability. Table 1 already hints at this possibility — GPT is not dramatically better than BERT on HellaSwag despite BERT being the filter — but a controlled ablation is needed.

The role of training set size: does HellaSwag resist surface learning even with massive data? The paper demonstrates that BERT's SWAG performance scales rapidly with training examples (Figure 3), reaching near-human levels with 16,000 examples, and interprets this as evidence of surface learning. A parallel experiment on HellaSwag — fine-tuning BERT-Large on progressively larger subsets of the HellaSwag training data, from 64 examples to the full 50,000-ish, and measuring both overall accuracy and the in-domain/zero-shot gap — would directly test whether HellaSwag resists surface learning or simply requires more data for the same phenomenon to manifest. If the learning curve looks similar to SWAG's (rapid initial gains, plateau near the full-dataset performance of ~47%), this would suggest that HellaSwag's difficulty is partly a data quantity effect and that a larger dataset might be saturated through the same surface-learning mechanisms. If the learning curve is qualitatively different — slower, with no plateau in sight — this would strengthen the claim that HellaSwag requires reasoning that surface learning cannot provide.

Can models learn to detect GPT artifacts on HellaSwag, and would that constitute solving the task? The paper's key claim is that HellaSwag's GPT-generated wrong answers are free of the stylistic artifacts that made SWAG's LSTM-generated answers exploitable. But GPT, while stronger than an LSTM, is still a 12-layer left-to-right decoder with 117M parameters — less than half BERT-Large's capacity. It is possible that a sufficiently powerful model could learn to detect residual GPT artifacts (subtle statistical regularities in word choice, syntax, or discourse structure) that BERT-Large misses, and that a future model achieving 80%+ accuracy on HellaSwag would be doing artifact detection, not reasoning. A diagnostic follow-up would replicate Section 3's ablation suite on the best available model for HellaSwag: context removal, word shuffling, and the combination. If a model achieving, say, 75% accuracy drops to near-chance under these ablations, it would reveal that HellaSwag was solved through a new generation of surface learning, motivating the co-evolutionary rebuild the paper advocates. If accuracy is robust to ablations, it would suggest that genuine reasoning has been acquired. This experiment is the natural "stress test" for the paper's methodological framework and would validate (or challenge) the co-evolutionary argument.

Extending the Goldilocks zone concept to non-procedural domains. HellaSwag's Goldilocks zone depends on procedural, physically-grounded text from ActivityNet and WikiHow — domains where humans naturally have high agreement about what happens next. What does the Goldilocks zone look like for social commonsense (predicting emotional reactions, social outcomes), abstract reasoning (counterfactuals, logical entailment), or domain-specific expert reasoning (legal, medical)? A follow-up study could apply the AF pipeline with GPT-generation and BERT-Large discrimination to a new domain — for instance, social scenarios from the Social IQa dataset or counterfactual narratives — and systematically vary generation length to find whether a Goldilocks zone exists. The finding might be negative: for social reasoning, where even humans disagree on plausible outcomes, the human accuracy ceiling might be too low (e.g., 70%) to create a meaningful gap with machine performance. This would reveal a boundary condition on AF-based dataset construction — it works only when the domain admits high human agreement, which excludes many important types of reasoning. Alternatively, a new domain might produce an even larger human-machine gap than WikiHow, demonstrating the generality of the approach. Either outcome would refine the paper's methodological contribution.

Replacing human validation with automated false-negative detection. The human validation stage is essential for HellaSwag's >95% human accuracy but is expensive, slow, and introduces a dependency on crowd worker quality (Appendix E). For the co-evolutionary cycle to be practical, this bottleneck needs automation. A follow-up could explore whether the AF discriminator itself — trained to distinguish real from generated endings — can identify false negatives (generated endings that are actually plausible) by examining the discriminator's confidence scores. The hypothesis: generated endings that receive very high discriminator scores (meaning the discriminator thinks they might be real) are candidates for being true false negatives, and removing these automatically might approximate the effect of human validation. Comparing a version of HellaSwag where false negatives are filtered by discriminator confidence thresholds against the human-validated version would reveal how much the human-in-the-loop step matters. If automated filtering works, it makes the co-evolutionary vision far more practical; if it fails, it identifies human judgment as a hard requirement for adversarial benchmark construction, which is an important finding in itself.

Practical Applications and Downstream Use Cases

Benchmark quality certification for model releases. When an organization releases a new pretrained model and reports state-of-the-art results, the community currently has limited tools for determining whether those results reflect genuine capability or surface-level exploitation. HellaSwag, along with its diagnostic ablation suite (context removal, word scrambling, zero-shot category evaluation), provides a litmus test for reasoning claims. A model that achieves 85% on SWAG but only 35% on HellaSwag — or one that shows a small gap between full and ablated conditions on SWAG but a large gap on HellaSwag — can be characterized as having learned different things from the two benchmarks, with HellaSwag performance more indicative of transferable capability. Model release papers can report HellaSwag accuracy alongside the SWAG-style ablation suite as a standard part of the evaluation protocol, giving practitioners a more reliable signal of whether the model has acquired commonsense reasoning or merely sophisticated statistical pattern matching. The 95.6% human ceiling provides a stable, interpretable target that is not subject to the annotation artifacts that plagued static benchmarks.

Data filtering and quality assurance for text generation systems. The Adversarial Filtering methodology, while presented as a dataset construction technique, is also a quality filter for machine-generated text. The AF discriminator — a BERT-Large trained to distinguish real from generated endings — can be deployed as a scoring function that estimates how "human-like" a generated continuation is. For applications where factual accuracy or commonsense plausibility matters (automated news summarization, instructional content generation, dialogue systems), an AF-style discriminator could flag outputs likely to contain commonsense violations before they reach end users. The paper's finding that human validation is essential to remove false negatives (plausible-sounding wrong answers that fool the discriminator) suggests that such a system would need a confidence threshold tuned to the application's tolerance for errors, but the core technology — a discriminator trained adversarially against a strong generator — transfers directly.

Training data augmentation for reasoning tasks. HellaSwag's format — context with one correct continuation and several adversarially-selected wrong continuations — provides a template for generating synthetic training data for commonsense reasoning. Given any domain with sequential text (procedural manuals, narrative stories, dialogue transcripts), one can fine-tune a GPT-style model to generate candidate continuations, apply an AF-like discriminator to select the hardest wrong answers, and produce a four-way multiple-choice training set. This could augment small hand-labeled commonsense datasets with large volumes of synthetic examples, potentially improving model robustness. The paper's finding that transfer from SWAG to HellaSwag is poor (34.6%, Figure 9) but that HellaSwag-trained models retain some SWAG performance (71.4%) suggests that training on adversarially-constructed data produces more transferable features, making it a better candidate for data augmentation than training on statically-constructed datasets.

Human-in-the-loop adversarial content creation for AI safety evaluation. The paper's multi-round human validation protocol — where crowd workers identify machine-generated text that is too plausible and trigger replacement — is a human-in-the-loop adversarial testing framework that extends beyond dataset construction. For AI systems deployed in high-stakes domains (medical advice, legal reasoning, emergency response), a similar protocol could be used to continuously probe for failure modes: generate candidate outputs that are wrong but superficially plausible, have human experts identify which wrong outputs are dangerously convincing, and use those to update the system's training or filtering mechanisms. The paper demonstrates that this iterative human-machine adversarial process widens the gap between human and machine performance (Figure 6) — exactly what you would want for a safety evaluation framework, where the goal is to identify cases where machines are confidently wrong. The probabilistic worker model and iterative replacement algorithm described in Appendix E provide a concrete, operationalized protocol that could be adapted to safety-critical settings.

When to Prefer This Method

The paper does not position HellaSwag against a specific named alternative with explicit tradeoff conditions — it is a benchmark and methodology, not a model to be deployed or a training recipe to be chosen over another. As such, there is no "when to prefer this method" matrix in the sense of algorithm selection. The relevant decision, implicitly framed by the paper, is when to use adversarial dataset construction (AF) versus static human-authored dataset creation for evaluating a capability. The paper's argument implies:

• Prefer AF-based benchmark construction when the target capability is one where models have historically exploited annotation artifacts or stylistic biases, when the best available generators and discriminators are strong enough to produce realistic wrong answers, and when human agreement on the task is naturally high (enabling a large human-machine gap). Commonsense NLI fits these conditions, and the paper's success with WikiHow demonstrates the template.

• Prefer static human-authored benchmarks (or accept their limitations) when the domain admits low human agreement (making the Goldilocks zone inaccessible), when the best available generators are too weak to produce realistic wrong answers (making AF converge at high discriminator accuracy, as the LSTM did in Figure 5 left), or when the cost of iterative AF + human validation is prohibitive relative to the evaluation budget. The paper does not explicitly discuss these boundary conditions, but they follow from the empirical findings.