SQuAD 2.0: The Stanford Question Answering Dataset

🎯 Pitch

A state-of-the-art reading comprehension model achieves 86% F1 on SQuAD 1.1 but plummets to just 66% on SQuAD 2.0—a gap over four times larger than the human–machine gap on 1.1—because it struggles to identify when a question simply cannot be answered from the text, even when given distractor sentences.

1. Executive Summary

This paper introduces SQuAD 2.0, an extension of the Stanford Question Answering Dataset that combines the existing SQuAD 1.1 answerable questions with over 50,000 adversarially written unanswerable questions designed to satisfy two key desiderata—relevance (questions appear topically related to the paragraph, preventing simple word-overlap heuristics from distinguishing them) and existence of plausible answers (the paragraph contains a span of the correct semantic type that serves as a distractor, preventing type-matching heuristics from succeeding). A strong neural baseline, DocQA + ELMo, achieves only 66.3% F1 on SQuAD 2.0 compared to 89.5% human accuracy—a 23.2-point gap that dwarfs the 5.4-point human–machine gap on SQuAD 1.1—establishing that adversarial unanswerable questions expose fundamental weaknesses in reading comprehension systems even when those systems are trained on in-distribution negative examples.

2. Context and Motivation

The Core Problem: Reading Comprehension Systems Don't Know When to Say "I Don't Know"

The fundamental problem this paper tackles is deceptively simple: extractive reading comprehension systems, when confronted with a question whose answer does not appear in the provided text, will still confidently produce an answer rather than abstaining. This is not merely an annoyance—it represents a failure of genuine language understanding. A system that truly comprehends text should be able to determine not only what the answer is when one exists, but also whether any answer is supported by the text at all.

This distinction matters because, as the paper argues in the opening of Section 1, existing systems achieve superficially impressive performance through the wrong mechanisms:

"models can do well at SQuAD by learning context and type-matching heuristics"

In other words, high SQuAD 1.1 scores do not necessarily reflect true reading comprehension. A model might identify that a question asks "What company..." and then locate any company name in the paragraph, selecting it as the answer regardless of whether that company actually answers the question. This heuristic succeeds on SQuAD 1.1 because every question has an answer in the paragraph—the model never needs to consider the possibility that no span is correct. The task was designed, intentionally or not, as a span-selection problem rather than a genuine comprehension problem.

The paper identifies this as a root cause of fragility:

"One root cause of these problems is SQuAD's focus on questions for which a correct answer is guaranteed to exist in the context document. Therefore, models only need to select the span that seems most related to the question, instead of checking that the answer is actually entailed by the text."

This framing is important: the authors are not saying SQuAD 1.1 is a bad dataset—it successfully drove years of productive model development—but they are saying it measures something narrower than what the field needs. The shift from "find the answer" to "find the answer or prove none exists" is not an incremental difficulty increase; it changes the fundamental nature of the reasoning required.

Why This Problem Is Important

The importance of knowing when to abstain extends well beyond benchmarking, touching on both practical deployment concerns and deeper questions about what it means for a system to "understand" language.

Practical importance: Trustworthiness in deployment. The paper doesn't belabor this point in extended discussion, but the implication is clear from the introduction. A reading comprehension system deployed in a real-world setting—medical question answering over clinical literature, legal research over case documents, customer support over documentation—must reliably distinguish between "the answer is in this document" and "the answer is not here." A system that always produces an answer, even when the correct information is absent, will mislead users rather than assist them. The paper's repeated use of the phrase "know what they don't know" (in the title and conclusion) signals this concern: epistemic self-awareness is as important as factual accuracy.

Theoretical importance: Entailment as the true test of comprehension. The paper draws an explicit connection to Recognizing Textual Entailment (RTE) in Section 6:

"SQuAD 2.0 forces models to understand whether a paragraph entails that a certain span is the answer to a question. Similarly, recognizing textual entailment (RTE) requires systems to decide whether a hypothesis is entailed by, contradicted by, or neutral with respect to a premise."

This connection positions SQuAD 2.0 as bridging the gap between extractive QA and entailment reasoning. In SQuAD 1.1, the entailment relationship is assumed to hold (every question's answer is entailed by the text—at least according to crowdworker judgment). SQuAD 2.0 removes this assumption, requiring models to verify entailment before extracting. A model that cannot recognize non-entailment lacks a capability that is arguably central to language understanding: the ability to determine what a text does and does not support.

The authors also connect this to relation extraction, noting that systems in that domain must similarly "understand when a possible relationship between two entities is not entailed by the text" (Section 6). The pattern is consistent across NLP: many tasks, when stripped of their simplifying assumptions, require models to recognize the absence of information as well as its presence. SQuAD 2.0 is an instance of this broader challenge, made concrete and measurable.

Prior Approaches and Where They Fall Short

The paper's motivation is built on a careful analysis of why existing approaches to negative examples—questions without answers in the text—fail to create a genuinely challenging task. The authors identify three categories of prior work and diagnose specific weaknesses in each.

Distant Supervision: Easy to Identify, Often Noisy

Distant supervision creates negative examples by pairing questions with documents that don't contain their answers. The paper discusses several instances:

TF-IDF retrieval (Clark and Gardner, 2017). This approach pairs SQuAD questions with paragraphs from the same Wikipedia article that have high TF-IDF overlap with the question, but that lack the correct answer. The paper identifies two problems. First, these retrieved paragraphs "are often not very relevant to the question"—they share surface-level vocabulary but do not discuss the same topic in a way that would confuse a model. Second, distant supervision "does not ensure the existence of a plausible answer in the retrieved context." Without a plausible answer, a model can succeed by simple type-matching: if the question asks for a date and no date appears in the paragraph, the model can confidently abstain without understanding the text's meaning.

The paper's experimental results in Table 4 bear this out: the same models achieve F1 scores 15–17 points higher on SQuAD 1.1 + TFIDF negatives than on SQuAD 2.0 (e.g., DocQA + ELMo scores 83.0 F1 on the TFIDF-augmented data vs. 67.6 F1 on SQuAD 2.0). This gap confirms quantitatively what the authors argue qualitatively: TF-IDF negatives are systematically easier.

TriviaQA (Joshi et al., 2017). TriviaQA retrieves context documents from the web for each trivia question, and some retrieved documents do not contain the answer, naturally producing negative examples. However, as the paper notes, "these are excluded from the final dataset"—they were treated as retrieval noise rather than as valuable training or evaluation data. Moreover, web-retrieved documents introduce their own problems: the correct answer might appear in paraphrased form even when exact-match retrieval suggests it's absent, adding noise to the negative label.

Zero-shot Relation Extraction (Levy et al., 2017). This work explicitly included negative examples, but the paper cites Levy et al.'s own finding that 65% of those negatives "do not have a plausible answer, making them easy to identify." When the majority of negative examples lack a distractor span of the correct type, a model can achieve high accuracy by simply checking whether any entity of the expected type appears in the context.

The common thread across all distant supervision approaches is that relevance and plausibility are not guaranteed. Distant supervision can produce large quantities of negative examples cheaply, but those examples do not force models to engage in the kind of careful entailment reasoning that SQuAD 2.0 demands.

Rule-Based Generation: Low Diversity, Narrow Phenomena

The paper discusses the rule-based approach proposed by Jia and Liang (2017), which the authors refer to as RULEBASED. This method edits existing SQuAD 1.1 questions to make them unanswerable by applying a fixed set of transformations: replacing entities and numbers with similar words, and substituting nouns and adjectives with WordNet antonyms.

The critique is concise but devastating:

"Their questions are not very diverse: they only replace entities and numbers with similar words, and replace nouns and adjectives with WordNet antonyms."

The word "only" carries the weight here. The space of ways a question can be unanswerable—while remaining relevant and plausible—is vastly larger than entity swaps and antonym substitutions. SQuAD 2.0's analysis in Table 1 reveals a much richer taxonomy: negation insertion/removal (9%), mutual exclusion between the question's premise and the paragraph (15%), impossible conditions where the question presumes an event that never occurs (4%), and a large category of "Other Neutral" cases where the paragraph simply does not imply any answer despite sharing the topic (24%). Rule-based methods cannot generate this diversity because they are, by design, constrained to surface-level edits of existing questions.

The quantitative evidence again appears in Table 4: models score substantially higher on SQuAD 1.1 + RULEBASED (DocQA + ELMo achieves 89.6 F1) than on SQuAD 2.0 (67.6 F1). The rule-based negatives are recognizable because they follow predictable patterns that a model can learn to detect, even if those patterns are syntactically local rather than semantically deep.

NewsQA's Organic Negatives: Too Few, Too Noisy

The NewsQA dataset (Trischler et al., 2017) naturally produces some unanswerable questions because crowdworkers write questions given only article summaries, not the full text. This is an appealing approach—the negatives arise organically from the data collection process rather than being synthetically constructed. However, the paper identifies two practical limitations:

First, scale: "Only 9.5% of their questions are unanswerable, making this strategy hard to scale." The organic process that makes the negatives natural also makes them scarce. Building a large, balanced dataset of answerable and unanswerable questions through this method would require an impractical amount of data collection.

Second, noise and scope: "we found that some are misannotated as unanswerable, and others are out-of-scope (e.g., summarization questions)." The negative examples that do exist are not always clean—some questions are genuinely answerable from the full text but were marked as unanswerable because the summary didn't contain the answer, while others ask for tasks (like summarization) that fall outside the extractive QA paradigm entirely. Finally, as the paper notes, Trischler et al. "also exclude negative examples from their final dataset," meaning NewsQA never actually required systems to handle the unanswerable case during evaluation.

The Broader Dataset Landscape

The paper's survey in Section 3 also touches on two other dataset categories that relate to the problem but don't solve it:

Answer sentence selection datasets (QASENT, WikiQA) require ranking sentences by relevance to a question, which implicitly involves distinguishing relevant from irrelevant sentences. However, these datasets suffer from the same issues as distantly supervised negative examples: the irrelevant sentences are often not topically relevant and lack plausible answers, making them easy to reject with lexical baselines (as Yih et al., 2013, demonstrated). WikiQA is also small (3,047 questions total), limiting its utility for training data-hungry neural models.

Multiple choice datasets (MCTest, RACE) can include a "none of the above" option, which captures the unanswerable case. But the paper argues that "multiple choice options are often unavailable" in practical applications—real users don't typically provide the system with a set of candidate answers including the correct one, plus a "none" option. Moreover, multiple choice questions tend to target different reasoning skills (fill-in-the-blank, interpretation, summarization) that don't align cleanly with extractive QA.

How This Paper Positions Itself Relative to Existing Work

SQuAD 2.0 is not proposed as an incremental improvement over any single prior dataset. Instead, it is positioned as a deliberately constructed solution to the systematic weaknesses across all prior approaches. The paper's contribution is not a new modeling technique but a new evaluation resource designed with explicit desiderata (Section 2) that directly address the failures catalogued in Section 3.

The positioning rests on three pillars:

1. Crowdworker-driven adversarial construction, not automatic generation. Unlike TF-IDF retrieval or rule-based editing, SQuAD 2.0's unanswerable questions were written by humans who were specifically instructed to make them difficult—they were told to make questions "look similar to answerable ones" (Section 4.1) and were shown the existing SQuAD 1.1 questions as reference. This adversarial framing is crucial: the crowdworkers' goal was not merely to write unanswerable questions, but to write unanswerable questions that would fool a hypothetical system. The paper explicitly uses the word "adversarially" in the abstract to describe the collection process. This connects to the adversarial examples literature (Jia and Liang, 2017) but with a key difference the authors highlight in Section 6:

"Jia and Liang (2017) created adversarial test examples that fool models trained on SQuAD 1.1. However, models that are trained on similar examples are not easily fooled by their method. In contrast, the adversarial examples in SQuAD 2.0 are difficult even for models trained on examples from the same distribution."

This is a critical distinction. SQuAD 2.0's negatives are not designed to exploit specific model weaknesses that disappear with retraining; they represent genuinely hard cases that challenge even models trained on in-distribution data. The 23.2-point human–machine gap on test data (where models have access to SQuAD 2.0 training data) confirms that the difficulty is intrinsic to the task, not an artifact of train–test mismatch.

2. Explicitly enforced desiderata: relevance and plausible answers. The paper's Section 2 lays out two requirements and then shows—through both qualitative analysis (Table 1, where 93% of sampled negatives are confirmed answerable) and quantitative evaluation (the plausible answer analysis in Section 5.4 and Table 5, where roughly half of all false positives match the plausible answer)—that the dataset satisfies them. This is not a post-hoc claim about the data; the collection interface itself required workers to highlight a plausible answer for each unanswerable question they wrote (Figure 3, Appendix A.1), baking the desiderata into the construction process.

3. Integrated with a well-established benchmark to enable direct comparison. SQuAD 2.0 does not replace SQuAD 1.1 but subsumes it. The answerable questions are identical, and the data splits follow the same article partitions. This design choice means that any model evaluated on SQuAD 2.0 can be directly compared to its performance on SQuAD 1.1, making the difficulty gap explicit. Table 3 shows exactly this: DocQA + ELMo drops from 85.8 F1 on SQuAD 1.1 to 66.3 F1 on SQuAD 2.0, while human performance drops only from 91.2 to 89.5. The 23.2-point human–machine gap on SQuAD 2.0 versus the 5.4-point gap on SQuAD 1.1 quantifies exactly how much harder the unanswerable question requirement makes the task for machines specifically, not for humans generally.

The paper also makes a subtle but important framing choice in Section 6 by connecting SQuAD 2.0 to entailment and relation extraction. This positions the dataset not as a one-off benchmark but as part of a broader research program: building NLP systems that understand not just what a text says, but what it does not say—that "know what they don't know."

3. Technical Approach

3.1 Reader Orientation

SQuAD 2.0 is a dataset, not a model or an algorithm—the technical contribution is the construction methodology that produces a set of reading comprehension examples where systems must correctly decide whether to answer or abstain. The paper's goal is to create unanswerable questions that satisfy two properties (relevance and plausible-answer existence) which prior automatic methods fail to achieve, thereby forcing models to engage in genuine entailment reasoning rather than exploiting surface heuristics.

3.2 Big-Picture Architecture (Diagram in Words)

The dataset construction pipeline has four interconnected components:

Base Dataset (SQuAD 1.1) — the existing set of answerable question–paragraph pairs, which provides the paragraphs and answerable questions that SQuAD 2.0 subsumes.
Adversarial Question Writing Interface — a crowdsourcing task where workers, given a paragraph and its existing answerable questions, write new unanswerable questions and highlight plausible (but incorrect) answer spans.
Quality Control Filters — post-hoc mechanisms (worker-level filtering, manual inspection) that remove noisy or low-effort submissions.
Combined Dataset with Evaluation Protocol — the merged answerable and unanswerable data, accompanied by a scoring rubric where abstaining on an unanswerable question earns full credit (EM and F1 of 1) and answering incorrectly earns zero, with a threshold-tuning procedure for models that output a probability of unanswerability.

The flow is: SQuAD 1.1 paragraphs and questions → crowdworkers write adversarial unanswerable questions with plausible answers → quality filters prune noisy submissions → the resulting data is combined into train/dev/test splits mirroring the original SQuAD 1.1 article partitions → models are trained and evaluated using a modified scoring function that rewards correct abstention.

3.3 Roadmap for the Deep Dive

First, the crowdworker task design and interface, since this is the mechanism that produces the dataset and enforces the relevance and plausible-answer desiderata.
Second, the quality control and filtering procedures that ensure the collected data is clean enough for reliable evaluation.
Third, the dataset structure and statistics—how the new data integrates with SQuAD 1.1, the resulting split sizes, and the answerable-to-unanswerable ratios.
Fourth, the evaluation protocol, which requires modifying the standard SQuAD scoring to handle abstention and introduces a threshold-tuning mechanism for models that predict unanswerability probabilities.
Fifth, the model architectures used as baselines, since understanding what SQuAD 2.0 is testing requires knowing what capabilities the tested systems possess.
Sixth, the automatic negative-generation baselines (TF-IDF and rule-based), which are not part of SQuAD 2.0 itself but serve as comparison points that demonstrate why the adversarial construction method is necessary.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a dataset construction paper whose core idea is that adversarial, crowdworker-written unanswerable questions—created under explicit instructions to be relevant and to include plausible distractor answers—produce a reading comprehension benchmark that cannot be solved by surface heuristics, even when models are trained on in-distribution data.

The Crowdworker Task Design: How Adversarial Unanswerable Questions Are Produced

The central technical mechanism in SQuAD 2.0 is the crowdworker task interface, which transforms the abstract desiderata (relevance, plausible answers) into concrete instructions and constraints that shape worker behavior. This is not a mechanical or algorithmic process—it is a human-in-the-loop adversarial generation procedure where the "adversary" is the crowdworker, incentivized (through instructions and examples) to produce questions that look answerable but are not.

Task structure. Each crowdsourcing task consisted of an entire Wikipedia article from SQuAD 1.1, broken into paragraphs as in the original dataset. For each paragraph, workers were asked to "pose up to ﬁve questions that were impossible to answer based on the paragraph alone" (Section 4.1). This is the first critical design choice: workers are shown a paragraph and told explicitly that the answer is not in that paragraph, but they must write a question that appears as though it could be. This is fundamentally different from the SQuAD 1.1 task, where workers wrote questions whose answers they knew were in the paragraph. Here, the worker's job is to simulate a confused or misled reader—someone who might plausibly think the paragraph answers their question based on a superficial reading.

Enforcing relevance. Two mechanisms ensure the unanswerable questions are topically relevant to the paragraph. First, workers are instructed to "reference entities in the paragraph" (Section 4.1)—the question must mention specific people, places, organizations, dates, or concepts that appear in the paragraph text. This prevents workers from writing generic or off-topic questions that could be rejected by simple word-overlap heuristics. If a paragraph discusses the Endangered Species Act, a question like "What is the capital of France?" is both unanswerable and trivially identifiable as such, because none of its content words appear in the paragraph. By requiring entity referencing, the instructions force the question's vocabulary to overlap substantially with the paragraph's vocabulary, closing the easiest shortcut.

Second, workers were shown "questions from SQuAD 1.1 for each paragraph" as inspiration (Section 4.1). This is a subtle but powerful design choice: the existing answerable questions serve as style guides, demonstrating the tone, specificity, and lexical patterns of genuine SQuAD questions. A worker who sees "Which laws faced significant opposition?" as an answerable SQuAD 1.1 question internalizes the template and might produce "Which laws were passed unanimously?"—a question that looks identical in form but is unanswerable because the paragraph never discusses the voting margin. The phrase "this further encouraged unanswerable questions to look similar to answerable ones" (Section 4.1) captures the adversarial intent: the SQuAD 1.1 examples are not just references but tools for making the unanswerable questions maximally confusable.

Enforcing plausible answers. The interface explicitly required workers to "ensure that a plausible answer is present" (Section 4.1). The paper's Appendix A.1, Figure 3, shows the actual interface: after writing an unanswerable question, workers must highlight a span in the paragraph that could serve as a plausible (but incorrect) answer. This is the mechanism that operationalizes the "existence of plausible answers" desideratum from Section 2. The plausible answer must be of the same semantic type as what the question asks for—if the question asks for a date, the highlighted span must be a date; if it asks for a person, the span must be a person. This prevents type-matching heuristics from working: a system cannot simply check whether any entity of the expected type exists in the paragraph, because such an entity is guaranteed to exist regardless of whether the question is answerable.

The paper's analysis in Table 5 (Appendix A.2) confirms that these plausible answers function as intended: "roughly half of all wrong answers on unanswerable questions exactly matched the plausible answers" provided by crowdworkers. In other words, when a model (or even a human) makes a false-positive error—answering when it should have abstained—the answer it produces is the crowdworker-provided plausible answer about 50% of the time. This is strong evidence that the plausible answers are genuinely confusing, not arbitrary highlighted spans that models ignore.

Logistics and compensation. Workers were given 7 minutes per paragraph, paid $10.50 per hour, and used the Daemo crowdsourcing platform (Gaikwad et al., 2015). The paper does not report the total cost or the number of unique workers, but the per-paragraph time constraint of 7 minutes suggests that each worker processed roughly 8–9 paragraphs per hour. The Daemo platform is noted for its self-governed structure, though the paper does not elaborate on whether this affected data quality.

Quality Control Filters

Not all crowdworker submissions are useful. The paper applies specific filtering criteria to remove noisy data, operating at two levels: worker-level and example-level.

Worker-level filtering. The primary filter is simple: "We removed questions from workers who wrote 25 or fewer questions on that article" (Section 4.1). This threshold was chosen because workers who wrote very few questions for an article likely struggled with the task and quit before completing it—their submissions are more likely to be low-effort, misunderstood, or unrepresentative. This filter was applied to both the new unanswerable questions and the existing SQuAD 1.1 answerable questions associated with those workers' articles. The authors' rationale is that a worker who didn't understand the unanswerable question task probably also produced lower-quality answerable questions in the original SQuAD 1.1, so removing all their data improves dataset quality on both sides.

The threshold of 25 is not justified with ablations—the paper does not show what happens with a threshold of 10 or 50—but it functions as a practical heuristic. Since workers were assigned entire articles (multiple paragraphs), a worker who completed the full article would naturally write more than 25 questions across all paragraphs. A worker who abandoned the task early would fall below this threshold. The filter effectively removes partial submissions.

Example-level quality assessment. To verify that the remaining data is clean, the authors "manually inspected 100 randomly chosen negative examples from our development set" and categorized them (Section 4.3, Table 1). They found that 93% of the sampled negative examples are indeed unanswerable—meaning the remaining 7% are misannotated (the question is actually answerable from the paragraph, despite the crowdworker marking it as unanswerable). This 93% cleanliness rate is not perfect but is high enough that noise does not dominate the evaluation signal. The paper does not describe any systematic procedure for identifying and correcting the 7% of noisy examples, suggesting they remain in the released dataset.

Development and test set article filtering. For the development and test splits, the authors applied an additional filter: "we removed articles for which we did not collect unanswerable questions" (Section 4.1). This created development and test sets where every article has both answerable and unanswerable questions, producing a "roughly one-to-one ratio of answerable to unanswerable questions in these splits" (Section 4.1). The train set, by contrast, retained all articles, meaning some articles have only answerable questions (those articles for which unanswerable questions were never collected). This results in a train-set ratio of approximately two answerable questions per unanswerable question, which has implications for model training that the authors address through a threshold-tuning procedure (discussed below in the evaluation protocol subsection).

Dataset Structure and Statistics

Integration with SQuAD 1.1. SQuAD 2.0 is not a standalone dataset. It is SQuAD 1.1 (all original answerable questions and paragraphs) plus the newly collected unanswerable questions, merged at the article level. The article partitions (which articles go into train, dev, and test) are identical to SQuAD 1.1, ensuring that any model trained on SQuAD 2.0 trains on exactly the same articles as a model trained on SQuAD 1.1 would have. The only difference is the presence of additional unanswerable questions.

Dataset sizes (Table 2). The final dataset contains the following splits:

Train: 130,319 total examples, consisting of 87,599 answerable questions (the original SQuAD 1.1 training set, after worker-level filtering) plus 43,498 unanswerable questions. This spans 442 articles, of which 285 contain at least one unanswerable question.
Development: 11,873 total examples, consisting of 5,928 answerable questions (after filtering) plus 5,945 unanswerable questions across 35 articles. The approximately 1:1 ratio is a deliberate design choice for balanced evaluation.
Test: 8,862 total examples, consisting of 4,530 answerable questions plus 4,332 unanswerable questions across 28 articles—again roughly 1:1.

The paper does not elaborate on why the development and test sets have different numbers of articles (35 vs. 28) or total examples (11,873 vs. 8,862), but this is a consequence of the article-split inheritance from SQuAD 1.1 combined with the article-level filtering (removing articles without unanswerable questions).

A note on answerable question filtering. The original SQuAD 1.1 train set contained 87,599 examples, and the paper reports the same number for SQuAD 2.0's answerable train examples. This suggests that the worker-level filter (removing questions from workers who wrote 25 or fewer questions) did not substantially reduce the number of answerable training examples, or that any removed examples were compensated for. The paper does not provide the unfiltered counts for comparison.

Evaluation Protocol: Scoring Abstention

The standard SQuAD evaluation metrics—Exact Match (EM) and F1 score—are designed for answerable questions only. A model either produces the correct span (scoring 1 for EM, a token-overlap F1 between 0 and 1) or produces an incorrect span (scoring 0 for both). SQuAD 2.0 requires extending these metrics to handle the abstention case.

The abstention scoring rule. The paper defines the rule for unanswerable questions:

"For negative examples, abstaining receives a score of 1, and any other response gets 0, for both exact match and F1."

This means that when the question is unanswerable:

If the model outputs "no answer" (abstains), it earns a perfect score: EM = 1, F1 = 1.
If the model outputs any answer span whatsoever, it earns zero: EM = 0, F1 = 0.

There is no partial credit for being "close" to the plausible answer on an unanswerable question—producing the plausible answer is as wrong as producing any other span. This is a strict all-or-nothing penalty: the model must recognize non-entailment; guessing is maximally punished.

For answerable questions, the scoring is unchanged from SQuAD 1.1: EM is 1 if the model's predicted span exactly matches any ground-truth answer (after normalization), and F1 is the harmonic mean of token-level precision and recall between the predicted span and each ground-truth answer, taking the maximum across ground-truth answers. These scores are then averaged across all examples in the evaluation set (both answerable and unanswerable), producing a single EM and F1 that reflects both the model's ability to answer correctly when appropriate and its ability to abstain correctly when necessary.

Why this scoring scheme? The all-or-nothing penalty for answering an unanswerable question is a deliberate choice that aligns the metric with the paper's stated goal: forcing models to "know what they don't know." A system that confidently produces wrong answers on unanswerable questions is not a useful system, even if those wrong answers are semantically plausible. The metric does not reward the model for identifying that a question is "probably" unanswerable—it must commit to either answering or abstaining, and the penalty for incorrect commitment is the maximum possible. This contrasts with a soft scoring scheme where producing the plausible answer might earn partial F1 credit (since the plausible span is in the paragraph and might share tokens with a hypothetical correct answer). The paper's choice eliminates that incentive, making abstention the only path to a non-zero score on unanswerable questions.

Threshold tuning for probabilistic models. The models evaluated in the paper (described in the next subsection) do not directly output "answer" or "abstain." Instead, they output a continuous probability that the question is unanswerable, alongside a distribution over possible answer spans. At test time, the model must convert this probability into a binary decision: abstain if the predicted unanswerability probability exceeds some threshold, otherwise output the most likely answer span. The threshold controls the tradeoff between false positives (answering when should abstain) and false negatives (abstaining when should answer).

The paper tunes this threshold on the development set: "We tune this threshold separately for each model on the development set. When evaluating on the test set, we use the threshold that maximizes F1 score on the development set." This is a standard practice that prevents the test-set threshold from being cherry-picked. The paper notes that threshold tuning "does slightly better than simply taking the argmax prediction, possibly due to the different proportions of negative examples at training and test time" (Section 5.1). This comment reveals an important practical issue: models are trained on a dataset where answerable questions outnumber unanswerable ones roughly 2:1 (in the train set), but evaluated on a dataset where they are roughly 1:1 (in dev and test). A model that learns the training distribution might be biased toward answering, because answering is more often correct during training. The threshold tuning compensates for this distribution shift by allowing the model to be more conservative (requiring a lower predicted unanswerability probability to trigger abstention) than the argmax would dictate.

Human evaluation protocol. Human accuracy was measured using a separate crowdworker task, designed to mirror the model evaluation as closely as possible while accounting for individual worker variability. In each task, workers saw an entire article with all associated questions (both answerable and unanswerable, shuffled together). For each question, they either highlighted the answer span or marked it as unanswerable. Workers were told to expect a mix and were given one minute per question.

To reduce noise from individual worker errors, the paper collected multiple answers per question ("on average, we collected 4.8 answers per question") and selected the final answer by majority vote. Two tie-breaking rules were applied: ties were broken "in favor of answering questions" (if some workers answered and others abstained, the majority-answer answer was taken) and "preferring shorter answers to longer ones" (if multiple answer spans were proposed, the shortest one was selected). The paper notes that this multi-annotator majority-vote procedure is stricter than the original SQuAD 1.1 human evaluation, which used a single annotator. Therefore, the human baseline on SQuAD 2.0's answerable subset is likely higher than the reported SQuAD 1.1 human baseline, and the paper explicitly acknowledges this: "for SQuAD 1.1, Rajpurkar et al. (2016) evaluated a single human's performance; therefore, they likely underestimate human accuracy."

Baseline Model Architectures

The paper evaluates three existing neural architectures, not because they are novel contributions, but because they represent the state of the art at the time and demonstrate that SQuAD 2.0 is challenging even for strong systems. Understanding what these models do is important for interpreting the experimental results, because their failure modes reveal what kinds of reasoning SQuAD 2.0 requires.

All three models share a common high-level architecture: they take a question and a paragraph as input, produce a probability distribution over answer spans (start and end positions), and additionally produce a scalar probability that the question is unanswerable. At training time, the model learns both tasks jointly from labeled data. At test time, the predicted unanswerability probability is compared against a tuned threshold to decide whether to output an answer span or abstain.

BiDAF-No-Answer (BNA). This model, proposed by Levy et al. (2017), extends the Bidirectional Attention Flow (BiDAF) architecture (Seo et al., 2016) with an explicit "no-answer" prediction head. BiDAF works by computing attention in both directions: question-to-context attention identifies which context words are most relevant to each question word, and context-to-question attention identifies which question words are most relevant to each context word. These attention representations are then fed through a series of modeling layers (typically LSTMs) that produce start and end position logits for answer span prediction. BNA adds an additional output: a binary classifier that takes the final hidden state (or some pooled representation of it) and predicts whether the question is answerable. The paper does not describe the BNA architecture in detail, referring readers to Levy et al. (2017), but the key point is that the model learns to output both an answer span distribution and an unanswerability score from a shared representation.

DocumentQA No-Answer (DocQA). This model, from Clark and Gardner (2017), uses a different base architecture but the same conceptual extension. DocumentQA processes the question and paragraph through a series of attention and self-attention layers (the exact architecture is not described in this paper, but Clark and Gardner (2017) detail a multi-paragraph reading comprehension model with shared-normalization training). Like BNA, DocQA adds a no-answer prediction head that outputs a probability of unanswerability. The paper evaluates two variants: DocQA without ELMo and DocQA + ELMo (Peters et al., 2018). ELMo (Embeddings from Language Models) provides deep contextualized word representations by computing a weighted combination of hidden states from a pretrained bidirectional LSTM language model. These representations capture syntactic and semantic properties of words in context (e.g., polysemy resolution, part-of-speech information) that static word embeddings (like GloVe) miss. DocQA + ELMo achieves the highest scores on both SQuAD 1.1 (85.8 F1) and SQuAD 2.0 (66.3 F1), making it the strongest baseline.

Why these models? The three architectures represent a range of performance levels, from BNA (the weakest, at 77.3 F1 on SQuAD 1.1) to DocQA + ELMo (the strongest, at 85.8 F1). This allows the paper to show that SQuAD 2.0 is hard across the board—not just for a single model. The fact that all three models show a large gap from human performance on SQuAD 2.0 (Table 3: human F1 is 89.5, while DocQA + ELMo reaches only 66.3) demonstrates that the difficulty is not an artifact of a particular architecture.

Automatic Negative-Generation Baselines

The paper compares SQuAD 2.0 against two methods for automatically generating unanswerable questions, to demonstrate that the adversarial human-written approach produces a genuinely harder dataset. These baselines are not part of SQuAD 2.0 itself but serve as experimental controls.

TF-IDF negatives (Clark and Gardner, 2017). This method generates negative examples by pairing existing SQuAD 1.1 questions with paragraphs from the same Wikipedia article that have high TF-IDF overlap with the question, but that do not contain the correct answer. The TF-IDF (Term Frequency–Inverse Document Frequency) overlap ensures some lexical similarity between the question and the paragraph—the paragraph contains words that appear in the question—which might superficially suggest relevance. However, as the paper argues in Section 3.1, TF-IDF matching "does not ensure the existence of a plausible answer in the retrieved context." The paragraph might contain some of the question's words but lack any span of the correct semantic type, making the unanswerability easy to detect through type-matching.

To create a fair comparison, the paper applied TF-IDF negative generation only to the 285 articles for which SQuAD 2.0 has unanswerable questions (rather than all articles in SQuAD 1.1), and tested on the same articles and answerable questions as the SQuAD 2.0 development set, adding unanswerable questions in a roughly one-to-one ratio. This ensures that the evaluation distribution is comparable—the only difference is whether the unanswerable questions came from crowdworkers or TF-IDF retrieval.

Rule-based negatives (Jia and Liang, 2017). The "RULEBASED" approach takes existing SQuAD 1.1 questions and applies a fixed set of transformations to make them unanswerable. The transformations are: replacing entities and numbers with similar words (e.g., swapping a date for a nearby date, or a person's name for a different name of the same type), and replacing nouns and adjectives with WordNet antonyms. The resulting questions retain the syntactic structure of the original answerable questions but ask about entities or properties that do not appear in the paragraph. The paper's critique is that this approach produces questions that are "not very diverse" (Section 3.1)—they are all variations on the same few edit operations, which a model can potentially learn to detect as patterns during training.

The same controlled comparison protocol was used: RULEBASED negatives were generated only for the 285 articles with SQuAD 2.0 unanswerable questions, and evaluation was on the SQuAD 2.0 development set articles with a roughly 1:1 answerable-to-unanswerable ratio.

Results interpretation. Table 4 shows that all three models perform substantially better on the automatically generated negative datasets than on SQuAD 2.0. For example, DocQA + ELMo achieves 83.0 F1 on SQuAD 1.1 + TFIDF and 89.6 F1 on SQuAD 1.1 + RULEBASED, compared to 67.6 F1 on SQuAD 2.0. The 15.4-F1 gap between the highest automatic-negative score (89.6 on RULEBASED) and the SQuAD 2.0 score (67.6) is the paper's quantitative evidence that adversarial human-written unanswerable questions expose weaknesses that automatic methods cannot. The gap is not small—it represents a fundamental difference in difficulty, not a marginal improvement in benchmark design.

These baselines serve a specific rhetorical purpose: they preempt the objection that SQuAD 2.0 is just SQuAD 1.1 plus some unanswerable questions, and that any source of unanswerable questions would produce a similarly challenging dataset. The experimental evidence contradicts this: the source of the unanswerable questions matters enormously, and the adversarial human-written approach produces a qualitatively different level of difficulty.

Summary of Design Choices and Their Justifications

Adversarial human-written questions over automatic generation: human crowdworkers can produce diverse, subtle unanswerable questions that exploit genuine comprehension gaps, whereas automatic methods (TF-IDF retrieval, rule-based editing) produce questions that follow predictable patterns detectable by models.
Entity referencing instruction: forces questions to be lexically grounded in the paragraph, preventing trivial word-overlap heuristics from distinguishing answerable and unanswerable questions.
Plausible answer highlighting requirement: guarantees that every unanswerable question has a same-type distractor span in the paragraph, preventing type-matching heuristics from working and creating genuine ambiguity that forces entailment reasoning.
SQuAD 1.1 questions as style references: encourages crowdworkers to mimic the distribution of genuine SQuAD questions, making unanswerable questions maximally confusable with answerable ones.
7-minute-per-paragraph pacing with $10.50/hour compensation: balances throughput with quality—fast enough to collect 53,775 questions, slow enough for workers to produce thoughtful adversarial examples.
Worker-level filtering (≥25 questions per article): removes partial submissions from workers who abandoned the task, reducing noise at the cost of some data loss.
Article-level filtering for dev/test (only articles with unanswerable questions): ensures balanced evaluation where every article tests both capabilities, preventing articles with only answerable questions from inflating scores.
1:1 answerable-to-unanswerable ratio in dev/test: creates a balanced evaluation where neither answering nor abstaining is a dominant strategy, forcing models to learn both skills.
All-or-nothing scoring on unanswerable questions (EM=1 and F1=1 for abstention, 0 for any answer): creates maximal penalty for false positives, aligning the metric with the practical requirement that systems should not confidently produce wrong answers.
Threshold tuning on development set: compensates for the train–test distribution shift (2:1 vs. 1:1 answerable-to-unanswerable ratio) without exposing the test set, and handles the fact that models output continuous probabilities rather than discrete decisions.
Majority-vote human evaluation across 4.8 annotators: reduces individual annotator noise and establishes a more reliable human performance ceiling than single-annotator evaluation.
Same article partitions as SQuAD 1.1: enables direct comparison between SQuAD 1.1 and SQuAD 2.0 performance, isolating the effect of adding unanswerable questions from any confounding change in data distribution.

4. Key Insights and Innovations

Innovation 1: Reframing the Reading Comprehension Task from Span Selection to Entailment Verification

The most fundamental intellectual contribution of SQuAD 2.0 is not the dataset itself but the conceptual reframing it forces upon the field: reading comprehension is not merely finding the most relevant span in a text, but determining whether the text entails any answer at all. Prior to SQuAD 2.0, the dominant paradigm in extractive QA—exemplified by SQuAD 1.1 but also by NewsQA, TriviaQA, and others—treated the task as synonymous with ranking spans by relevance to a question. The assumption that every question has an answer in the text was so deeply baked into dataset design that it was rarely articulated as an assumption at all. Models optimized for this formulation learned to excel at type-matching and lexical overlap heuristics (as Weissenborn et al., 2017, documented), but never developed the capability to judge whether a candidate answer was actually supported by the evidence.

The paper's framing move is to diagnose this as a missing entailment check, and to operationalize that diagnosis into a concrete benchmark by making non-entailment equally central to the task. The connection drawn explicitly in Section 6—linking SQuAD 2.0 to Recognizing Textual Entailment (RTE) and to relation extraction's need to recognize absent relationships—positions the unanswerable question as not a special case or edge condition, but as a necessary and inseparable component of comprehension. A system that cannot recognize when no answer exists does not merely fail on a subset of questions; it fails to demonstrate the core competency that the task purports to measure.

This reframing is fundamental, not incremental. It does not modify the mechanics of how answers are found—span prediction architectures remain unchanged. Instead, it changes what "success" means, adding a logical prerequisite (entailment must be verified) that was previously assumed away. The evidence that this is a genuine reconceptualization rather than a harder version of the same task comes from Table 3: the human–machine gap balloons from 5.4 F1 points on SQuAD 1.1 to 23.2 F1 points on SQuAD 2.0, while human performance barely changes (91.2 → 89.5 F1). Humans can do the entailment check effortlessly because it is part of how they read; machines, trained under the old paradigm, cannot, because their architectures never learned it. The gap is not a difficulty gradient—it is a qualitative capability gap exposed by changing the task definition.

Innovation 2: Adversarial Human Generation as a Method for Creating Genuinely Difficult Negative Examples

Prior to SQuAD 2.0, the dominant approaches for creating negative examples in reading comprehension were automatic generation (distant supervision via TF-IDF retrieval, rule-based question editing) or incidental collection (NewsQA's summary-based question writing). The paper demonstrates—both analytically in Section 3 and experimentally in Table 4—that all of these produce negatives that models can easily identify. The insight is not merely that human-written negatives are harder (that's unsurprising), but that the adversarial framing of the human writing task produces negatives that remain difficult even after training on in-distribution data. This is a specific claim about the type of human effort required, not just the fact of human authorship.

The field's default assumption had been that collecting human-written negative examples was sufficient (as in NewsQA), or that automatic methods could approximate the difficulty of human-written negatives through clever retrieval or editing. SQuAD 2.0 refutes both positions. The NewsQA negatives, though human-written, were too few (9.5% of questions), too noisy, and—critically—not designed adversarially; crowdworkers were simply writing questions from summaries, not trying to fool a system. The automatic methods fail because they produce negatives that violate the relevance and plausible-answer desiderata, allowing models to succeed through surface heuristics.

The adversarial framing is what makes the contribution distinctive. By instructing crowdworkers to write questions that "look similar to answerable ones" and showing them existing SQuAD 1.1 questions as style references, the data collection process explicitly positions workers as adversaries trying to confuse a hypothetical model. The plausible-answer highlighting requirement operationalizes this: workers must identify a specific span that a type-matching system would be drawn to, then craft a question that makes that span wrong. The fact that roughly half of all model false positives on unanswerable questions match these crowdworker-provided plausible answers (Table 5) confirms that the adversarial design works—the distractors are not arbitrary; they are precisely the spans that models are most likely to incorrectly select.

This contribution is a methodological advance in dataset construction, not merely a new dataset. It establishes that the instructions and constraints given to annotators can qualitatively change the difficulty of the resulting benchmark, and that adversarial framing—making annotators explicitly try to fool systems—produces a fundamentally different distribution of examples than neutral collection. The 15.4-F1 gap between the best automatic-negative baseline (DocQA + ELMo on RULEBASED, 89.6 F1) and SQuAD 2.0 performance (67.6 F1) quantifies how large this methodological difference is: adversarial human generation isn't marginally better; it creates a categorically harder task.

Innovation 3: In-Distribution Difficulty as Evidence That the Problem Is Intrinsic, Not an Artifact of Distribution Shift

A persistent concern with adversarial datasets is that they may be difficult only because they exploit train–test distribution mismatch—models fail on test examples because those examples look different from training data, not because the underlying task is fundamentally harder. Jia and Liang (2017) explicitly demonstrated this pattern: adversarial distractors that fooled SQuAD 1.1-trained models became ineffective once models were trained on similar distractors. The natural worry is that SQuAD 2.0 might follow the same trajectory: the unanswerable questions are hard now, but once models are trained on in-distribution unanswerable questions, the difficulty will evaporate.

SQuAD 2.0 provides strong evidence against this worry, and this is a significant conceptual contribution. The authors explicitly contrast their work with Jia and Liang (2017) in Section 6:

"Jia and Liang (2017) created adversarial test examples that fool models trained on SQuAD 1.1. However, models that are trained on similar examples are not easily fooled by their method. In contrast, the adversarial examples in SQuAD 2.0 are difficult even for models trained on examples from the same distribution."

The evidence supports this claim: the models in Table 3 (BNA, DocQA, DocQA + ELMo) are all trained on SQuAD 2.0 training data, which includes over 43,000 unanswerable questions from the same adversarial generation process. Despite this, the best model achieves only 66.3 F1 on the test set—a 23.2-point gap from human performance. The difficulty is not an artifact of unfamiliar example types; it persists when those types are abundant in training.

This matters because it shifts the interpretation of the human–machine gap from "models need more data" to "models lack a fundamental capability." If the gap disappeared with training on in-distribution data, SQuAD 2.0 would be a useful training resource but not a diagnostic of a capability gap. The fact that the gap does not disappear—that training on 43,000 adversarial unanswerable questions still leaves a 23-point gap—suggests that current neural architectures are missing something essential: the ability to perform the entailment reasoning that distinguishes answerable from unanswerable questions even when both types look superficially similar.

This is a diagnostic contribution: SQuAD 2.0 functions as a capability probe, not just a harder benchmark. It reveals that state-of-the-art models, even when given abundant training data for both the answering and abstention sub-tasks, cannot reliably determine whether a text supports an answer. The problem is not data scarcity, distribution shift, or insufficient model capacity (at least in the sense measured by SQuAD 1.1 performance)—it is an architectural or inductive bias limitation that current models share.

Innovation 4: The Plausible Answer as Both a Dataset Design Constraint and a Diagnostic Tool

The requirement that every unanswerable question have a plausible answer—a specific highlighted span in the paragraph that matches the question's expected answer type but is incorrect—serves a dual role that makes it a conceptual innovation beyond simple quality control. First, as a design constraint, it systematically eliminates the easiest shortcut for detecting unanswerable questions: checking whether any entity of the expected type exists in the paragraph. This transforms type-matching from a useful heuristic (on SQuAD 1.1, where the correct answer is always a type-matched span) into a liability (on SQuAD 2.0, where every unanswerable question also has a type-matched span, but it's wrong). Prior work had recognized that type-matching heuristics contributed to inflated SQuAD scores (Weissenborn et al., 2017), but no dataset had been designed to explicitly neutralize them while preserving answerable-question performance.

Second, the plausible answer functions as a diagnostic probe into model failure modes. By collecting crowdworker-provided plausible answers during dataset construction, the authors can measure not just whether models fail on unanswerable questions, but how they fail. Table 5's finding—that roughly half of all model false positives exactly match the crowdworker-provided plausible answer—is a specific and informative diagnostic result. It tells us that models are not producing random wrong answers on unanswerable questions; they are systematically drawn to exactly the spans that human adversaries predicted would be confusing. This is evidence that models are using the same type-matching and relevance heuristics that the dataset was designed to punish, and that these heuristics lead them to specific, predictable errors.

This dual function—design constraint and diagnostic—distinguishes the plausible answer from a simple data quality check. It transforms SQuAD 2.0 from a pass/fail benchmark into a tool for understanding why models fail. A researcher using SQuAD 2.0 can analyze whether their model's errors cluster on the plausible answers (indicating a reliance on type-matching heuristics) or produce different spans (indicating a different failure mode), and can use this signal to guide architecture improvements. This is a methodological contribution to benchmark design: embedding diagnostic information (the plausible answers) into the dataset structure itself, rather than requiring separate post-hoc analysis to understand model errors.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation uses the SQuAD 2.0 test set, consisting of 8,862 examples (4,530 answerable questions from SQuAD 1.1 plus 4,332 newly collected unanswerable questions) across 28 Wikipedia articles where every article contains both answerable and unanswerable questions in roughly a 1:1 ratio. The development set uses 11,873 examples (5,928 answerable, 5,945 unanswerable) across 35 articles with the same balanced structure. Both splits inherit their article partitions from SQuAD 1.1, with the additional constraint that articles lacking unanswerable questions were removed (Section 4.1). For the automatic negative comparison experiments, the same development set articles and answerable questions are used, with TF-IDF or rule-based negatives substituted for the crowdworker-written ones at a roughly 1:1 ratio. The training set contains 130,319 examples (87,599 answerable, 43,498 unanswerable) across 442 articles, of which 285 contain at least one unanswerable question, creating an approximately 2:1 answerable-to-unanswerable ratio that differs from the balanced dev/test distribution.
Base model(s). Three neural architectures are evaluated, representing a range of capabilities at the time: BiDAF-No-Answer (BNA), proposed by Levy et al. (2017), which extends the BiDAF architecture (Seo et al., 2016) with a "no-answer" prediction head; DocumentQA No-Answer (DocQA) from Clark and Gardner (2017), a multi-paragraph reading comprehension model similarly augmented with an unanswerability classifier; and DocQA + ELMo, which adds deep contextualized word representations from Peters et al. (2018) to the DocQA architecture. These models were chosen because they represent the state of the art on SQuAD 1.1 at the time (DocQA + ELMo achieves 85.8 F1 on SQuAD 1.1) and because they all implement the same basic approach—jointly predicting an answer span distribution and an unanswerability probability—allowing fair comparison. The paper does not introduce any new model architectures; all three are existing systems adapted to handle unanswerable questions by adding a binary classification head.
Metrics. The primary metrics are Exact Match (EM) and F1 score, following Rajpurkar et al. (2016). For answerable questions, the scoring is unchanged from SQuAD 1.1: EM is 1 if the predicted span exactly matches any ground-truth answer after normalization, and F1 is the maximum token-overlap F1 between the prediction and each ground-truth answer. For unanswerable questions, a new rule applies: abstaining (predicting "no answer") earns EM = 1 and F1 = 1, while producing any answer span earns EM = 0 and F1 = 0. This is an all-or-nothing penalty—there is no partial credit for producing the plausible answer or a semantically similar span. The final reported EM and F1 are averages across all examples in the evaluation set (both answerable and unanswerable), producing a single number that reflects both the model's ability to answer correctly when appropriate and to abstain correctly when necessary. The paper also reports human accuracy using the same metrics, computed via majority vote across an average of 4.8 annotators per question, with tie-breaking in favor of answering and preferring shorter spans (Section 4.2).
Baselines. For the main SQuAD 2.0 evaluation (Table 3), the baseline is human performance, measured via the crowdworker majority-vote procedure. The models are compared against each other (BNA vs. DocQA vs. DocQA + ELMo) and against the human ceiling. For the automatic negative comparison (Table 4), two additional datasets serve as baselines: SQuAD 1.1 + TFIDF (Clark and Gardner, 2017), which pairs SQuAD questions with TF-IDF-retrieved paragraphs from the same article that lack the correct answer, and SQuAD 1.1 + RULEBASED (Jia and Liang, 2017), which edits SQuAD questions to be unanswerable by replacing entities, numbers, and applying WordNet antonym substitutions. For the plausible answer analysis (Table 5), the baseline is human false-positive behavior—how often crowdworkers, when they incorrectly answer an unanswerable question, produce the plausible answer span. The paper also implicitly uses an always-abstain baseline: since the dev and test sets are roughly 1:1 answerable-to-unanswerable, a system that always abstains would get approximately 50% of questions correct (all unanswerable questions scored as 1, all answerable questions scored as 0), which the paper notes translates to roughly 48.9 test F1 (Section 5.2). Model performance is contextualized against this floor.
Generation budget / compute accounting. This is a dataset paper, not a model training paper, so there is no test-time compute budget to account for in the traditional sense (no beam search width, no number of samples, no FLOPs matching). The relevant "budget" is the size of the training set (130,319 examples for SQuAD 2.0 vs. 87,599 for SQuAD 1.1) and the ratio of answerable to unanswerable examples (2:1 in training, 1:1 in dev/test). The paper does, however, account for a form of compute in the threshold-tuning procedure: for each model, the unanswerability threshold is tuned on the development set by sweeping to find the value that maximizes F1, and that threshold is then fixed for test-set evaluation. This tuning consumes development-set computation but is treated as a hyperparameter selection step rather than as part of a compute budget. The paper does not report the range of thresholds searched or the granularity of the sweep. The human evaluation required collecting an average of 4.8 answers per question, representing a significant annotation budget, but this is not quantified in FLOPs or cost terms.
Cross-validation / statistical protocol. There is no cross-validation. The threshold for each model's unanswerability probability is tuned once on the development set, and the threshold that maximizes development F1 is used for test-set evaluation. This is a standard held-out validation protocol rather than k-fold cross-validation. The paper does not report confidence intervals or statistical significance tests for any of the reported numbers; all results are presented as point estimates. For human evaluation, variability is addressed through majority voting across multiple annotators (4.8 on average), which reduces individual annotator noise, but no inter-annotator agreement metrics (e.g., Fleiss' kappa, Krippendorff's alpha) are reported. The paper also does not describe a separate validation split within the training data for model selection during training—model hyperparameters are presumably those from the original BNA and DocQA papers, though this is not explicitly stated.
Other evaluation details. The answerable portion of the SQuAD 2.0 test set is not identical to the SQuAD 1.1 test set because of the worker-level filtering (removing questions from workers who wrote 25 or fewer questions on an article) and the article-level filtering (removing articles without unanswerable questions). The paper reports SQuAD 1.1 numbers for the same model architectures (Table 3, leftmost columns), but these are taken from the original model papers rather than recomputed on the filtered SQuAD 2.0 answerable subset. This means the SQuAD 1.1 and SQuAD 2.0 scores are not perfectly comparable at the example level—the answerable questions differ slightly—but the differences are assumed to be negligible given the filtering criteria.

Main Quantitative Results

SQuAD 2.0 vs. SQuAD 1.1: How Much Harder Is the New Task?

The headline result appears in Table 3: on SQuAD 2.0, the best tested model (DocQA + ELMo) achieves 66.3 F1 on the test set, compared to 89.5 F1 for humans—a 23.2-point gap. The same model achieves 85.8 F1 on SQuAD 1.1, where humans score 91.2 F1, a gap of only 5.4 points. The human–machine gap has more than quadrupled.

Breaking this down by model: BNA achieves 62.1 F1 on SQuAD 2.0 (vs. 77.3 on SQuAD 1.1), a drop of 15.2 points. DocQA achieves 62.3 F1 on SQuAD 2.0 (vs. 81.0 on SQuAD 1.1), a drop of 18.7 points. DocQA + ELMo achieves 66.3 F1 on SQuAD 2.0 (vs. 85.8 on SQuAD 1.1), a drop of 19.5 points. The pattern is consistent: all models lose between 15 and 20 F1 points when unanswerable questions are introduced, while humans lose only 1.7 F1 points (91.2 to 89.5). The exact-match numbers tell the same story: DocQA + ELMo drops from 78.6 EM on SQuAD 1.1 to 63.4 EM on SQuAD 2.0, a 15.2-point decline, while human EM drops only 4.9 points (91.2 to 86.9 — though careful: the SQuAD 1.1 human EM in Table 3 is 82.3, not 91.2; the 91.2 in the table is human F1. Actually, the SQuAD 2.0 human EM is 86.9, which is 4.6 points higher than the SQuAD 1.1 human EM of 82.3, likely because the improved evaluation protocol—multiple annotators, majority voting—raises the measured human ceiling on answerable questions).

A critical interpretive point: the always-abstain baseline achieves 48.9 F1 (since the test set is roughly 1:1, abstaining on everything scores 1 on all unanswerable questions and 0 on all answerable questions, yielding approximately 50% accuracy, which translates to 48.9 F1 due to the specific scoring mechanics). DocQA + ELMo's 66.3 F1 is only 17.4 points above this floor, meaning the model is closer to the "always abstain" trivial baseline than to human performance (89.5 F1, 23.2 points above the model). This is a stark way to characterize the room for improvement: even the strongest model has not yet reached the midpoint between the trivial baseline and human performance.

SQuAD 2.0 vs. Automatically Generated Negatives: Why Adversarial Writing Matters

Table 4 presents the same three models evaluated on SQuAD 2.0 development set compared against the development set with TF-IDF negatives (Clark and Gardner, 2017) and RULEBASED negatives (Jia and Liang, 2017). The key numbers:

On SQuAD 2.0 dev: DocQA + ELMo achieves 67.6 F1.
On SQuAD 1.1 + TFIDF dev: DocQA + ELMo achieves 83.0 F1—15.4 points higher.
On SQuAD 1.1 + RULEBASED dev: DocQA + ELMo achieves 89.6 F1—22.0 points higher.

The gap between the highest automatic-negative score (89.6 F1) and the SQuAD 2.0 score (67.6 F1) is 22.0 F1 points. This is not marginal—it represents a fundamentally different level of difficulty. The RULEBASED dataset, in particular, barely challenges the strongest model: 89.6 F1 is only 1.6 points below the SQuAD 1.1 performance of 91.2 F1 (though note: these numbers are from slightly different splits, so the comparison is approximate). In other words, adding rule-based unanswerable questions reduces model performance by less than 2 F1 points, while adding crowdworker-written unanswerable questions reduces it by over 23 F1 points.

The pattern holds across all three model architectures. BNA achieves 76.6 F1 on TFIDF and 84.8 F1 on RULEBASED, compared to 62.6 F1 on SQuAD 2.0—gaps of 14.0 and 22.2 points respectively. DocQA achieves 79.2 F1 on TFIDF and 84.8 F1 on RULEBASED, compared to 64.8 F1 on SQuAD 2.0—gaps of 14.4 and 20.0 points. The TF-IDF negatives are harder than the rule-based ones (models score 5–7 points lower on TFIDF than on RULEBASED), consistent with the paper's argument that RULEBASED negatives are "not very diverse" and follow predictable edit patterns that models can learn to recognize. But even the TF-IDF negatives are substantially easier than the adversarial human-written ones, confirming the paper's core methodological claim: the source of unanswerable questions matters enormously, and distant supervision or rule-based editing cannot substitute for adversarial human generation.

What makes the automatic negatives so much easier? The paper does not provide a per-example error analysis comparing the three datasets, but the diagnosis in Section 3, combined with the plausible-answer analysis (discussed below), suggests the mechanism: automatic negatives frequently lack plausible distractors. A model can identify a TF-IDF negative because the paragraph, while topically related, often contains no span of the correct semantic type—a question asks for a date, but the paragraph has no dates, so the model abstains by type-matching rather than by comprehending the text's failure to entail the answer. A RULEBASED negative is easier still because it is created by a fixed set of transformation rules (entity swap, antonym substitution), and a model trained on such data can learn to detect the specific patterns these transformations produce—for instance, it might learn that questions containing WordNet antonyms of adjectives in the paragraph are likely unanswerable, regardless of deeper meaning.

Plausible Answers as Effective Distractors: Where Models Fail

Section 5.4 and Table 5 (Appendix A.2) present a targeted analysis of a specific failure mode: when models incorrectly answer an unanswerable question (a false positive), how often do they produce exactly the plausible answer that the crowdworker highlighted during dataset construction? The answer is striking: across all three models and humans, roughly half of all false positive errors match the plausible answer.

The exact numbers: on the SQuAD 2.0 development set, BNA matches the plausible answer on 48.6% of its false positive errors by exact match (63.0% by F1). DocQA matches on 55.0% (69.9% by F1). DocQA + ELMo matches on 54.9% (69.2% by F1). For comparison, humans match the plausible answer on 46.4% of their false positives (60.6% by F1).

This means the plausible answers provided by crowdworkers during dataset construction are functioning exactly as intended: they are the spans that both humans and machines are most likely to incorrectly select when they fail to recognize that a question is unanswerable. The fact that the match rate is similar for models and humans (around 50% in both cases) suggests that the plausible answers exploit general comprehension tendencies—they look like answers because they satisfy the question's type and lexical constraints—rather than model-specific artifacts.

The practical consequence of this finding is that SQuAD 2.0's unanswerable questions are not just hard in the abstract; they are hard in a specific, diagnosable way. A researcher analyzing their model's errors can ask: when my model answers an unanswerable question, does it pick the plausible answer (indicating it was fooled by the intended distractor) or something else (indicating a different failure mode, perhaps random guessing or a different heuristic)? The plausible answer serves as a built-in diagnostic probe that makes model failures interpretable.

The paper does not report the mirror statistic: when models are correct on answerable questions, how often is their answer the crowdworker-provided plausible answer? This would reveal whether the plausible answers are genuinely answer-like in general (in which case models might select them even for answerable questions where a different span is correct) or only distracting in the unanswerable case. The absence of this analysis means we cannot fully distinguish between "the plausible answer is a strong distractor specifically when no correct answer exists" and "the plausible answer is simply the most salient span of the correct type, regardless of context."

Ablation Studies and Robustness Checks

This is a dataset paper, not a model paper, so the concept of "ablation" must be adapted. The paper does not ablate architectural components or training procedures. Instead, it validates the dataset's construction choices through a series of comparative analyses and quality checks that serve the same function as ablations: demonstrating that specific design decisions were necessary and effective.

Human accuracy validation (Table 3, Section 4.2). The paper reports that humans achieve 89.5 F1 on the SQuAD 2.0 test set, which serves as both a performance ceiling and a noise check. If humans scored much lower, it would suggest the dataset contains genuinely ambiguous or mislabeled examples; if humans scored near 100%, the dataset might be too easy. The 89.5 F1 is high enough to confirm that the questions have clear answers (when answerable) and clear non-answers (when unanswerable), but the 10.5-point gap from perfection leaves room for some genuine ambiguity—the 93% cleanliness rate from the manual inspection (Section 4.3) means approximately 7% of negative examples are actually answerable, which would produce an upper bound of roughly 93% accuracy for a perfect system. The human score of 89.5 F1 is broadly consistent with this ceiling.

Manual inspection of negative examples (Table 1, Section 4.3). The paper randomly sampled 100 negative examples from the development set and categorized them into seven types, finding that 93% are genuinely unanswerable while 7% are misannotated (actually answerable). This is not reported as a formal ablation but serves as a crucial quality check: the dataset is not perfectly clean, but the noise level is low enough that it does not dominate evaluation. The taxonomy in Table 1 reveals the diversity of unanswerability types: negation (9%), antonym (20%), entity swap (21%), mutual exclusion (15%), impossible condition (4%), other neutral (24%), and answerable/misannotated (7%). This diversity supports the paper's claim that SQuAD 2.0 captures a "wide range of phenomena" beyond what automatic methods produce—if the taxonomy had shown that 90% of negatives were simple entity swaps, the dataset would be little better than RULEBASED. The distribution across categories confirms genuine diversity.

TF-IDF vs. RULEBASED comparison (Table 4). This functions as an ablation of the "adversarial human" design choice. By holding the evaluation framework constant (same answerable questions, same articles, same models, same metrics) and varying only the source of unanswerable questions, the paper isolates the effect of the adversarial human generation method. The 15–22 F1 gap between automatic and human-written negatives demonstrates that the choice of generation method has an enormous effect on the resulting benchmark's difficulty. This is the closest the paper comes to a controlled experiment on its central design decision.

Plausible answer distractor analysis (Table 5). This validates the "existence of plausible answers" desideratum from Section 2. If models rarely matched the plausible answers—if their false positive answers were uniformly distributed across spans rather than concentrated on the crowdworker-highlighted ones—it would suggest the plausible answers were not actually distracting, contradicting the paper's design philosophy. The ~50% match rate confirms that the plausible answers are indeed the primary source of confusion, validating the instruction to crowdworkers to provide them. The similar match rate for humans (46.4% EM) further suggests that the distractors exploit general comprehension patterns, not brittle model-specific heuristics.

Threshold tuning necessity (Section 5.1). The paper notes that tuning the unanswerability threshold on the development set "does slightly better than simply taking the argmax prediction, possibly due to the different proportions of negative examples at training and test time." This is effectively an ablation of the threshold-selection method: argmax vs. tuned threshold. The paper does not report the numerical difference between these two approaches, which is a notable omission—we cannot assess how large the distribution-shift effect is or whether it affects all models equally. The direction of the effect is informative: since the training set has a 2:1 answerable-to-unanswerable ratio and the test set has 1:1, a model calibrated to the training distribution would be biased toward answering. The threshold tuning compensates by making the model more willing to abstain (lowering the threshold for classifying as unanswerable), which improves test performance. This confirms that the distribution shift is real and that models do not automatically generalize across different answerable-to-unanswerable ratios.

Worker-level filtering effect (Section 4.1). The paper states that questions were removed from workers who wrote 25 or fewer questions on an article, but does not report how many examples were removed or what the dataset statistics would be without this filter. This is a missing ablation: we cannot assess whether the filter materially improves data quality, whether a different threshold (10, 50, 100) would change the results, or whether the filtering disproportionately affects certain types of examples or articles. The 25-question threshold is presented as a heuristic rather than as an empirically validated choice.

Multi-annotator human evaluation vs. single-annotator (SQuAD 1.1 baseline). The paper notes that SQuAD 1.1 human performance was measured with a single annotator, while SQuAD 2.0 uses majority voting across 4.8 annotators. This means the apparent stability of human performance (91.2 F1 on SQuAD 1.1 vs. 89.5 on SQuAD 2.0) may partially reflect improved evaluation methodology rather than truly identical human capability. If SQuAD 1.1 had been re-evaluated with multi-annotator majority voting, the human score might have been higher, making the human drop from SQuAD 1.1 to SQuAD 2.0 slightly larger than the reported 1.7 F1 points. The paper does not perform this re-evaluation, so the precise magnitude of the human difficulty increase is uncertain—we know only that it is small relative to the model drop. This is a methodological confounding factor that makes the human–machine gap comparison slightly less clean than it appears.

Critical Assessment

The paper makes three central claims, explicit or implicit, that the experiments must support:

SQuAD 2.0 is substantially harder than SQuAD 1.1 for state-of-the-art models, exposing a capability gap that SQuAD 1.1 concealed.
Adversarially written human questions produce a harder dataset than automatically generated negatives (TF-IDF, rule-based), validating the construction methodology.
The difficulty of SQuAD 2.0 persists even when models are trained on in-distribution data, meaning the problem is intrinsic rather than an artifact of distribution shift.

Let me examine each in turn.

Claim 1: SQuAD 2.0 is substantially harder, exposing a hidden capability gap.

The evidence for this claim is strong and internally consistent. Table 3 shows all three models losing 15–20 F1 points when moving from SQuAD 1.1 to SQuAD 2.0, while humans lose only 1.7 F1 points. The 23.2-point human–machine gap on SQuAD 2.0 (vs. 5.4 points on SQuAD 1.1) is the paper's central quantitative result, and it is striking. The always-abstain baseline at 48.9 F1 provides a meaningful floor: models at 62–66 F1 are only 13–17 points above trivial behavior, confirming that they have not mastered the task.

However, the claim is somewhat narrower than it might appear due to the specific models tested. All three models (BNA, DocQA, DocQA + ELMo) share a fundamental architectural approach: they are span-prediction models with a no-answer classification head appended. None of them perform explicit entailment reasoning, build structured representations of the question–paragraph relationship, or incorporate mechanisms specifically designed to detect unsupported claims. The paper's claim that SQuAD 2.0 exposes a fundamental capability gap is convincing, but we cannot rule out that a model with a different inductive bias—one designed from the ground up for entailment verification rather than span selection plus an abstention bolt-on—might perform substantially better. The architectures tested represent the state of the practice at the time, but they may not represent the full space of possible approaches. This is a standard limitation of empirical benchmark papers and does not undermine the conclusion, but it means the "capability gap" is relative to a specific model paradigm, not an absolute statement about what neural networks can do.

Additionally, the comparison between SQuAD 1.1 and SQuAD 2.0 scores is made slightly imprecise by the fact that the answerable questions are not identical—the worker-level filtering removes some SQuAD 1.1 questions from SQuAD 2.0. The paper reports the SQuAD 1.1 scores from the original model papers rather than recomputing them on the filtered subset. If the removed questions were disproportionately easy (plausible, since they came from workers who quit early, potentially because they found the task difficult and produced lower-quality data), then the SQuAD 2.0 answerable subset might be slightly harder than the full SQuAD 1.1 set, artificially inflating the apparent difficulty drop. The paper does not provide enough information to assess the magnitude of this effect, but it is likely small relative to the 15–20 point drops observed.

Claim 2: Adversarial human writing produces a harder dataset than automatic generation.

The evidence for this claim (Table 4) is strong: models score 15–22 F1 points higher on automatically generated negatives than on SQuAD 2.0 negatives. This is a large gap, and it holds consistently across all three model architectures. The paper's analytical explanation—that automatic methods fail to ensure relevance and plausible answers—is supported by the prior literature it cites (Levy et al., 2017; Weissenborn et al., 2017) and by the plausible-answer analysis (Table 5), which shows that the crowdworker-provided plausible answers are what models most often incorrectly select.

However, a critical nuance: the comparison in Table 4 uses the same models trained on the SQuAD 2.0 training data and then evaluated on datasets with different negative examples. This tests whether SQuAD 2.0's unanswerable questions are harder for models that have been trained on SQuAD 2.0-style negatives—which they clearly are. But it does not test whether models trained specifically on TF-IDF or RULEBASED negatives would find those negatives easy. The paper's logic is that the automatic negatives are easier because they lack plausible answers or are not genuinely relevant, and a model trained on them would have an even easier time because it could learn to detect the specific patterns. This is a reasonable inference but is not directly tested. A more complete experiment would train separate models on SQuAD 2.0, on SQuAD 1.1 + TFIDF, and on SQuAD 1.1 + RULEBASED, and then evaluate each on its own test distribution and on the others. This would disentangle "the unanswerable questions are inherently harder" from "the models have been trained to expect SQuAD 2.0-style distribution." The paper's experimental design cannot fully separate these, though the strength and consistency of the effect makes the conclusion credible even without this more complex design.

There is also a fairness question about the RULEBASED comparison. RULEBASED negatives are created by editing existing SQuAD 1.1 questions; as a result, there is exactly one RULEBASED negative per original answerable question, and the negative looks nearly identical to the answerable version except for the edited entity/number/antonym. A model might learn a very simple heuristic: "if this question looks like a SQuAD 1.1 question but with a different entity, it's probably unanswerable." This heuristic would succeed on RULEBASED negatives but fail on SQuAD 2.0 negatives, which are written from scratch and don't have this paired structure. The comparison may partially reflect the predictability of the generation process (paired editing vs. de novo writing) rather than the intrinsic difficulty of the resulting questions considered in isolation. The TF-IDF comparison is cleaner in this respect because TF-IDF negatives are not paired with specific answerable questions.

Claim 3: The difficulty persists with in-distribution training, indicating an intrinsic problem.

This is the paper's most important claim, and the evidence is strong but has important caveats. All models in Table 3 are trained on SQuAD 2.0 training data, which includes 43,498 unanswerable questions from the same adversarial generation process as the test set. The 23.2-point human–machine gap on test data demonstrates that training on in-distribution data does not close the gap.

The fundamental validity of this claim rests on whether the training data is genuinely in-distribution with respect to the test data. The paper argues it is—same generation process, same crowdworker population, same article domains. However, there is one notable difference: the training set has a 2:1 answerable-to-unanswerable ratio, while the test set is 1:1. The paper acknowledges this and uses threshold tuning to compensate, but the distribution shift is real. A model trained to optimize accuracy under a 2:1 prior will learn different internal representations than one trained under a 1:1 prior, even if the individual examples are drawn from the same generative process. The threshold tuning handles the output-level miscalibration (the model is too reluctant to abstain) but does not address any representational consequences of the skewed prior—for instance, the model might allocate less capacity to features relevant for detecting unanswerability because unanswerable examples are less frequent during training. The paper does not test whether training with a 1:1 ratio (upsampling unanswerable examples or downsampling answerable ones) would improve test performance. If it would, then part of the human–machine gap is attributable to training distribution mismatch rather than an intrinsic capability limitation.

The claim also depends on the representativeness of the three tested architectures. If SQuAD 2.0 were evaluated today (2024) with more modern architectures—large pretrained transformers, chain-of-thought reasoning, retrieval-augmented generation—would the human–machine gap remain 23 points? The paper cannot answer this, and it would be unfair to demand prescience, but it does mean the claim "current neural architectures lack a fundamental capability" is time-bound and architecture-relative. The value of the dataset is precisely that it provides a stable benchmark against which architectural progress can be measured; the fact that future models might close the gap does not invalidate the paper's claim about the models tested.

Missing evaluations and analyses. Several additional experiments would have strengthened the paper:

Error analysis on answerable questions: The paper analyzes model errors on unanswerable questions (Table 5, plausible answer matches) but does not provide a parallel analysis of errors on answerable questions. How often do models answer incorrectly on answerable questions? How often do they incorrectly abstain (false negatives)? The headline F1 numbers combine both error types but do not reveal their relative frequency—a model could achieve 66 F1 by being perfect on answerable questions and terrible on unanswerable ones, or by being mediocre on both. Understanding this breakdown is crucial for diagnosing what models are actually failing at.
Performance by difficulty or question type: Table 1 provides a taxonomy of unanswerable question types (negation, antonym, entity swap, mutual exclusion, impossible condition, other neutral). The paper does not report model performance broken down by these categories. This would be highly informative: are models equally bad at all types, or are they particularly weak on certain reasoning patterns (e.g., mutual exclusion, impossible conditions) while somewhat better on others (e.g., simple negation)? Such a breakdown would guide model development by identifying specific sub-capabilities to target.
Calibration analysis: The models output a probability of unanswerability, which is thresholded to make a binary decision. The paper does not analyze whether these probabilities are well-calibrated—i.e., when the model says a question is 70% likely to be unanswerable, is it actually unanswerable 70% of the time? Calibration is important because miscalibration could indicate that the model has learned a useful ordering of examples by difficulty but is poorly scaled, in which case better calibration methods (e.g., temperature scaling, Platt scaling) could close some of the gap without any architectural change.
Training data ablation: How many unanswerable training examples are needed? The paper trains on 43,498 unanswerable questions but does not show learning curves—how does performance improve as the number of training negatives increases from, say, 1,000 to 43,498? If performance plateaus early, the dataset size is adequate; if it continues to improve, more unanswerable questions might further close the human–machine gap. This is relevant to the claim about intrinsic difficulty: if the gap can be closed simply by collecting more adversarial questions (rather than improving model architecture), the problem is one of data scale, not a fundamental reasoning limitation.
Human performance on sub-tasks: The paper reports a single human F1 score (89.5) that combines performance on answerable and unanswerable questions. It would be informative to see human accuracy separately on the answerable subset and the unanswerable subset, to understand whether humans struggle more with one type of question than the other, and whether the model deficits are symmetric or asymmetric.

Conditional validity. The paper's central claim—that SQuAD 2.0 forces models to "know what they don't know" and that current systems cannot do this—is well-supported for the specific models tested (span-prediction plus no-answer classifier) on the specific data distribution (English Wikipedia articles, crowdworker-written questions) with the specific evaluation protocol (thresholded binary abstention, all-or-nothing scoring). It does not claim universality across all possible model architectures, training procedures, or domains. The paper's contribution is the benchmark and the diagnosis of a specific set of model failures; the diagnosis is convincing within its scope, but the scope is bounded by the experimental design. Extrapolating to claims about "neural networks in general" or "language understanding in general" would overstate what the experiments demonstrate.

6. Limitations and Trade-offs

Difficulty Estimation Is Prohibitively Expensive in Practice

The assumption or constraint. The paper does not estimate question difficulty explicitly in the sense of a compute-optimal inference framework. However, the dataset construction process assumes that the adversarial human effort required to write unanswerable questions—approximately 7 minutes per paragraph per worker, with explicit plausible-answer annotation—is affordable and scalable. The paper reports collecting 53,775 unanswerable questions, but does not report the total crowdworker cost, the number of unique workers, or the total annotation time. The only cost-related figure is the hourly rate: $10.50 per hour.

The consequence. Any organization seeking to replicate SQuAD 2.0's methodology on a new domain or language must budget for extensive adversarial human annotation. The 7-minutes-per-paragraph constraint, combined with the requirement that workers write up to five questions per paragraph and highlight plausible answers, implies a substantial cost per unanswerable question. For comparison, automatic methods like TF-IDF retrieval or rule-based editing are essentially free once the infrastructure is built. The paper's experimental results (Table 4) show that those cheap automatic methods produce a much easier dataset—models achieve 83–90 F1 vs. 67.6 F1 on SQuAD 2.0—but the paper does not quantify whether the quality gain justifies the cost differential. A practitioner deciding between "spend money on adversarial human annotation" and "spend money on training a larger model" has no cost–benefit framework from this work.

More fundamentally, the difficulty of the resulting dataset is a function of the specific adversarial instructions, the worker population, the compensation structure, and the quality-control filters. None of these are guaranteed to transfer. A replication effort that pays workers differently, uses a different platform, or provides slightly different instructions might produce unanswerable questions with a different difficulty profile—even if they appear superficially similar. The paper provides no analysis of how sensitive the dataset's difficulty is to these procedural choices, making replication an uncertain proposition.

What evidence exists in the paper. The paper provides the time-per-paragraph (7 minutes) and hourly rate ($10.50) in Section 4.1, but reports no total cost, no per-question cost breakdown, and no sensitivity analysis of the filtering threshold. The worker-level filter (remove workers who wrote 25 or fewer questions) is presented without justification or ablation—we do not know how many workers or questions were removed, or whether a different threshold would change the dataset's character. The manual inspection of 100 examples (Section 4.3) finds a 93% cleanliness rate, but does not estimate what fraction of the 53,775 questions might be affected by worker misunderstanding, fatigue, or adversarial gaming of the task.

Mitigation status. The paper acknowledges none of these cost or replicability concerns. It treats the crowdworker methodology as a one-time cost for producing a static benchmark, not as a procedure others will need to replicate. This is a reasonable stance for a benchmark paper—datasets are built once and used many times—but it means that SQuAD 2.0's construction methodology is not a practical recipe for domain-specific deployment, only a demonstration that adversarial human writing can produce a hard dataset for a specific Wikipedia-based English reading comprehension task.

Single-Domain, Single-Language, Single-Genre Scope Limits Claims of Generality

The assumption or constraint. All of SQuAD 2.0's data is drawn from English Wikipedia articles, the same domain as SQuAD 1.1. The questions—both answerable and unanswerable—are written by English-speaking crowdworkers about encyclopedic text. The paper makes no attempt to extend the methodology to other languages, other text genres (news, scientific literature, legal documents, dialogue), or other question-answering paradigms (abstractive QA, multiple-choice QA, open-domain retrieval). The entire analysis, including the 23.2-point human–machine gap, is conditioned on this specific domain.

The consequence. It is unknown whether SQuAD 2.0's difficulty profile generalizes. Wikipedia articles are expository, well-structured, and largely fact-based—qualities that might make unanswerable questions systematically different from unanswerable questions in, say, legal contracts (where precision of wording is paramount), medical literature (where answers may be probabilistic rather than binary), or dialogue (where answers may be implicitly conveyed rather than explicitly stated). The paper's taxonomy in Table 1—negation, antonym, entity swap, mutual exclusion, impossible condition—was derived from Wikipedia-based examples; other genres might exhibit entirely different categories of unanswerability that SQuAD 2.0 does not capture.

For a practitioner deploying a reading comprehension system in a specific domain, the SQuAD 2.0 benchmark provides no domain-specific diagnostic. A system might achieve high SQuAD 2.0 scores by learning Wikipedia-specific patterns of unanswerability—e.g., that questions using antonyms of paragraph adjectives are often unanswerable—that fail to transfer to medical or legal text. Conversely, a system that performs poorly on SQuAD 2.0 might nonetheless handle unanswerable questions well in its target domain if that domain's unanswerability patterns are simpler or qualitatively different. The paper provides no evidence either way.

What evidence exists in the paper. The paper explicitly restricts itself to SQuAD 1.1 articles in English (Section 4.1). There are no cross-domain experiments, no out-of-domain evaluation sets, and no analysis of whether models trained on SQuAD 2.0 transfer their abstention capability to other datasets or domains. The related work survey in Section 3 references several other datasets (NewsQA, TriviaQA, WikiQA, MCTest, RACE) but does not use any of them for evaluation. The taxonomic analysis in Table 1 is based on Wikipedia paragraphs exclusively.

Mitigation status. Not addressed. The paper does not claim cross-domain generality—it positions SQuAD 2.0 as a new version of the SQuAD benchmark, implicitly accepting SQuAD's domain constraints. However, the paper's rhetoric about "knowing what they don't know" and the connection to entailment (Section 6) implies a broader capability than the experimental evidence supports. A reader should understand that "models don't know what they don't know on SQuAD 2.0" is the demonstrated claim; "models don't know what they don't know in general" is an extrapolation the paper gestures toward but does not validate.

Human Performance Baseline Benefits from Methodology Changes, Confounding the Human–Machine Gap Comparison

The assumption or constraint. The paper compares the human–machine gap on SQuAD 2.0 (23.2 F1 points) with the human–machine gap on SQuAD 1.1 (5.4 F1 points) to argue that SQuAD 2.0 exposes a dramatically larger capability deficit. However, the human evaluation methodology changed substantially between the two datasets. As the paper acknowledges in Section 4.2:

"for SQuAD 1.1, Rajpurkar et al. (2016) evaluated a single human's performance; therefore, they likely underestimate human accuracy."

SQuAD 2.0's human evaluation uses majority voting across an average of 4.8 annotators per question, with tie-breaking rules that favor answering and prefer shorter spans. This is a fundamentally different measurement procedure that is expected to produce higher and more reliable human scores—exactly as the paper notes.

The consequence. The 23.2-point human–machine gap on SQuAD 2.0 vs. the 5.4-point gap on SQuAD 1.1 is not a clean comparison. Part of the apparent gap inflation is attributable to SQuAD 1.1's human performance being underestimated (due to single-annotator noise) rather than SQuAD 2.0's machine performance being uniquely poor. If SQuAD 1.1 had been re-evaluated with the same multi-annotator majority-vote protocol, the human SQuAD 1.1 score would likely be higher than the reported 91.2 F1, making the human drop from SQuAD 1.1 to SQuAD 2.0 larger than the apparent 1.7 F1 points. The machine drop (19.5 F1 points for DocQA + ELMo) would remain substantial, but the human–machine gap would be somewhat smaller than 23.2 points because the human SQuAD 1.1 baseline would be higher.

This does not invalidate the paper's core claim that SQuAD 2.0 is harder for machines than for humans—the machine drop of 19.5 points vs. the human drop of (at least) 1.7 points is directionally unambiguous. But the magnitude of the differential—23.2 vs. 5.4, a factor of >4×—is partially a measurement artifact. A practitioner evaluating whether SQuAD 2.0 is a useful benchmark for tracking progress might reasonably ask: if we measured everything consistently, would the gap be 20 points, 15 points, or 10 points? The paper cannot answer this precisely without re-evaluating SQuAD 1.1 human performance under the same protocol.

Additionally, the SQuAD 2.0 human evaluation protocol has its own biases. The tie-breaking rule "in favor of answering questions" (Section 4.2) means that in ambiguous cases where some annotators answer and some abstain, the final label is always an answer. This systematically reduces the number of unanswerable labels in the human evaluation, making the human performance number slightly less comparable to models (which have no such bias unless explicitly encoded). The paper does not quantify how often ties occurred or how sensitive the human F1 score is to the tie-breaking rule.

What evidence exists in the paper. The paper explicitly acknowledges the methodology difference in Section 4.2 and notes that SQuAD 1.1 likely underestimates human accuracy. However, it does not re-evaluate SQuAD 1.1, nor does it provide sensitivity analysis for the tie-breaking rules, nor does it report inter-annotator agreement metrics that would allow a reader to estimate the uncertainty in the human baseline. The human F1 of 89.5 is reported as a point estimate without confidence intervals.

Mitigation status. Partially addressed through transparency—the paper states the methodology change and warns the reader. But the magnitude of the confounding effect is not quantified, and the headline comparison (23.2 vs. 5.4) is presented without this caveat in the abstract and introduction. A reader who does not carefully examine Section 4.2 will take the gap comparison at face value.

The Test Set Is Small (28 Articles, 8,862 Examples) and May Not Support Fine-Grained Model Discrimination

The assumption or constraint. The SQuAD 2.0 test set contains 8,862 examples across only 28 articles (Table 2). This is smaller than the SQuAD 1.1 test set (9,533 examples across 46 articles), and the article count is substantially smaller due to the filtering that removed articles without unanswerable questions. Each article contributes, on average, over 300 questions to the test set. This means that test-set performance is heavily influenced by a small number of source documents—a model that happens to perform well or poorly on the specific 28 articles in the test set may not be representative of its broader capabilities.

The consequence. The test set's statistical power for distinguishing between models is limited. The paper reports point estimates (e.g., DocQA + ELMo achieves 66.3 F1) without confidence intervals, but with only 28 articles, the variance in model performance across articles could be substantial. If a new model achieves 68.0 F1 vs. DocQA + ELMo's 66.3, is that a meaningful improvement or within the range of sampling variability from the small article set? Without standard errors, the benchmark cannot reliably discriminate between models with small score differences—a problem that becomes more acute as models approach the human ceiling and improvements are incremental.

The concentration of examples within articles also creates a potential for article-level overfitting. A model trained on the SQuAD 2.0 training set (442 articles, some of which may be from the same Wikipedia topics as the test articles) might learn article-specific patterns—e.g., the writing style of particular Wikipedia editors, the structure of articles about specific topics—that inflate its test performance without reflecting generalizable comprehension ability. The paper uses the same article partitions as SQuAD 1.1, which were designed to prevent exactly this problem, but with only 28 test articles, the risk of topic overlap between train and test is higher than with a larger and more diverse test set.

What evidence exists in the paper. Table 2 reports the test set size (8,862 examples, 28 articles). The paper does not report confidence intervals for any model scores, does not perform bootstrap resampling to estimate score variance, and does not analyze per-article performance variability. The manual inspection of 100 development-set examples (Section 4.3) is a quality check, not a statistical power analysis. The paper does not discuss whether 28 articles is sufficient to produce stable model rankings.

Mitigation status. Not addressed. The test set size is inherited from SQuAD 1.1's article partition plus the additional constraint of requiring unanswerable questions per article. The paper treats the test set as a fixed evaluation resource without discussing its statistical properties. In practice, the SQuAD leaderboard (which the paper announces in Section 1) uses the full test set and likely mitigates some of these concerns through the sheer volume of model submissions over time—random article-level variation would average out across many model evaluations—but the paper itself does not provide the tools (confidence intervals, bootstrap estimates) needed to assess whether a single model comparison is statistically reliable.

The Always-Abstain Baseline Reveals That Models Have Not Yet Demonstrated Substantial Competence

The assumption or constraint. The paper implicitly assumes that models achieving 62–66 F1 on SQuAD 2.0 represent meaningful progress over trivial baselines and that the remaining gap to human performance (89.5 F1) reflects solvable challenges. However, the always-abstain baseline—predicting "no answer" for every question—achieves 48.9 F1 on the test set (Section 5.2). This means DocQA + ELMo's 66.3 F1 is only 17.4 points above a strategy that performs no reading comprehension whatsoever.

The consequence. The absolute level of model performance is surprisingly low when benchmarked against the trivial baseline. A model at 66.3 F1 is getting roughly one-third of the possible improvement over always-abstain (17.4 points gained out of a maximum possible gain of 50.6 points, which is 100 minus 48.9, though in practice the maximum is human performance at 89.5, giving a gain of 40.6 points—so models have captured about 43% of the achievable gain). This framing is more sobering than the paper's rhetorical emphasis on the 23.2-point human–machine gap, because it reveals that even the "improvement" over SQuAD 1.1 baselines leaves models closer to non-comprehension than to human-level performance.

For a practitioner, this means that SQuAD 2.0 is arguably too hard for current systems to serve as a useful development benchmark—a dataset where the best models get 66 F1 may not provide a clear enough signal for iterative model improvement, because many architectural changes will produce score differences that are indistinguishable from noise given the small test set. SQuAD 2.0 functions well as a diagnostic (showing that models lack a capability), but its utility as a progressive benchmark (showing incremental improvements as architectures improve) is less certain at current performance levels.

This also complicates the paper's claim that SQuAD 2.0 reveals a "capability gap." A model at 66.3 F1 is bad enough at the task that the specific nature of its failures may not be informative—it may simply lack the basic competence to engage with the entailment reasoning the task requires, and its errors may reflect general confusion rather than specific, diagnosable weaknesses. The plausible-answer analysis (Table 5) partially addresses this by showing that model errors are systematic rather than random, but it does not establish that the models are doing anything beyond type-matching with slightly better features than a lexical baseline.

What evidence exists in the paper. Section 5.2 reports the always-abstain baseline at 48.9 F1. The paper notes that "existing models are closer to this baseline than they are to human performance," which is an honest characterization. However, the abstract and introduction emphasize the 23.2-point human–machine gap without contextualizing it against the always-abstain floor, giving a somewhat inflated impression of model competence. Table 3 reports the model scores without always-abstain as a reference row, requiring the reader to extract the baseline from the text.

Mitigation status. Partially addressed through transparency—the paper does report the baseline—but the rhetorical framing downplays it. A reader scanning the abstract and Table 3 would see "66.3 F1" and "23.2-point gap" without immediately recognizing that 66.3 is only 17.4 points above a strategy that requires zero language understanding. The paper does not discuss the implications for SQuAD 2.0's utility as a progressive benchmark at current performance levels.

No Analysis of the Relative Difficulty of Answerable vs. Unanswerable Questions or the Nature of False Negatives

The assumption or constraint. The paper reports aggregate EM and F1 scores that combine performance on answerable and unanswerable questions into a single number. It does not report the breakdown: what is the model's accuracy on answerable questions alone? On unanswerable questions alone? What fraction of errors are false positives (answering when should abstain) vs. false negatives (abstaining when should answer)? The only error-type analysis is the plausible-answer match rate on false positives (Table 5), which examines one specific failure mode but does not report the overall false positive rate.

The consequence. The aggregate F1 score conflates two potentially very different types of model behavior. A model that achieves 66 F1 could be:

Perfect on unanswerable questions (always abstains) but mediocre on answerable questions (gets ~32% of them right—since half the test is unanswerable and would score 50 points, the remaining 16 points would come from answerable questions, implying ~32% accuracy on them).
Perfect on answerable questions but terrible on unanswerable questions (answers everything, getting all answerable questions right for ~50 points, and all unanswerable questions wrong for 0 points, plus some partial credit on answerable questions—but this cannot reach 66 F1 under all-or-nothing scoring).
Mediocre on both, with some specific mix of false positives and false negatives.

These different profiles correspond to fundamentally different system behaviors with different practical implications. A system that always abstains on unanswerable questions but answers answerable questions poorly is safe but unhelpful—it won't mislead users but also won't assist them. A system that answers aggressively but frequently fails to abstain is helpful but dangerous—it provides answers when possible but confidently produces wrong information when it shouldn't. The paper's results do not distinguish these cases, and the always-abstain baseline at 48.9 F1 suggests that erring on the side of abstention is the stronger strategy given current model capabilities.

For a practitioner, this missing analysis is critical. Deploying a reading comprehension system requires understanding its error profile: is it more likely to miss answers that are present (false negatives, leading to user frustration) or to fabricate answers when none exist (false positives, leading to user misinformation)? These errors have different costs in different applications—a medical QA system should strongly prefer false negatives to false positives, while a casual trivia system might prefer the reverse. SQuAD 2.0's aggregate score provides no guidance for this tradeoff.

What evidence exists in the paper. Table 5 reports one slice of the error profile: the plausible-answer match rate among false positives. But the paper does not report the false positive rate itself, the false negative rate, or the separate accuracies on the answerable and unanswerable subsets. The development of unanswerability types in Table 1 is a qualitative taxonomy, not a quantitative error analysis. The paper provides no confusion matrix, no precision–recall breakdown, and no analysis of how the answerable-vs.-unanswerable accuracy tradeoff varies with the abstention threshold.

Mitigation status. Not addressed. The paper follows the SQuAD 1.1 convention of reporting aggregate EM and F1, extending it to the unanswerable case, but does not adapt the reporting conventions to the new two-part structure of the task. A simple table showing answerable-question accuracy, unanswerable-question accuracy, false positive rate, and false negative rate for each model would substantially improve the interpretability of the results. The single-threshold evaluation (tuned for maximum aggregate F1) masks the full precision–recall curve and the tradeoffs it represents.

7. Implications and Future Directions

How This Work Changes the Landscape

SQuAD 2.0 does not introduce a new model architecture, training procedure, or inference strategy. Its contribution is a benchmark redesign that forces a qualitative shift in what the field considers to be "reading comprehension." This is not a paradigm shift in the Kuhnian sense—it does not replace the existing span-extraction framework with a new theoretical model. Rather, it is a diagnostic reframing: by making the unanswerable case central to evaluation rather than an edge condition, SQuAD 2.0 reveals that the capabilities measured by SQuAD 1.1 were narrower than the field had assumed. The conceptual move is from "find the most relevant span" to "verify whether any span is entailed by the text," and the 23.2-point human–machine gap on SQuAD 2.0 (versus 5.4 on SQuAD 1.1) provides quantitative evidence that this distinction is not merely philosophical—it exposes a genuine capability deficit in state-of-the-art models.

The paper resolves a specific tension in the prior literature. Before SQuAD 2.0, there was no agreed-upon benchmark for evaluating whether reading comprehension systems could recognize the absence of an answer. Distant supervision approaches (TF-IDF retrieval, TriviaQA's document pairing) had shown that negative examples could be generated cheaply, but Levy et al. (2017) had already found that 65% of such negatives lacked plausible answers, making them trivially identifiable. Rule-based approaches (Jia and Liang, 2017) produced negatives that were hard for SQuAD 1.1-trained models but became easy once models were trained on similar examples—a classic adversarial robustness failure rather than a genuine comprehension challenge. The NewsQA negatives were too scarce (9.5% of questions) and too noisy. SQuAD 2.0 reconciles these findings by showing that adversarially constructed human-written negatives, satisfying explicit relevance and plausible-answer desiderata, produce a dataset that remains hard even after in-distribution training. The 15–22 F1 gap between SQuAD 2.0 and automatic-negative baselines (Table 4) quantifies how much the generation methodology matters.

The paper also changes the conversation around what makes a reading comprehension benchmark "hard." Prior work had focused on question complexity (multi-hop reasoning, longer contexts, more entities) or distractor design (Jia and Liang, 2017). SQuAD 2.0 demonstrates that simply removing the guarantee that an answer exists—while keeping questions superficially similar to standard SQuAD questions—creates a difficulty jump larger than architectural improvements had achieved in years. This redirects attention from span-selection accuracy toward entailment verification as the central bottleneck.

Several research directions become more attractive as a result. Entailment-aware architectures that jointly model answer extraction and answer verification, rather than appending a no-answer classifier to a span-prediction model, become a high-priority design goal. The paper's three baseline architectures (BNA, DocQA, DocQA + ELMo) all share the bolt-on classifier approach, and their uniformly poor performance suggests that this paradigm is insufficient. Training data augmentation with adversarially constructed negatives becomes an obvious strategy for any extractive QA system, not just those targeting SQuAD. And calibration and abstention mechanisms—methods for making models reliably output "I don't know" when appropriate—move from a niche concern to a central requirement for trustworthy deployment.

Conversely, some research directions become less attractive. Pure span-ranking improvements on SQuAD 1.1—the dominant research program from 2016–2018, which produced ever-higher F1 scores through attention mechanism refinements—are revealed by SQuAD 2.0 to be optimizing a metric that does not capture entailment verification. A model that improves from 85 to 88 F1 on SQuAD 1.1 through better span selection may see negligible improvement on SQuAD 2.0 if its underlying inability to recognize non-entailment remains unaddressed. Automatic negative generation, while computationally cheap, is shown to produce negatives that are systematically easier than human-written ones (Table 4), limiting its value for robust evaluation even if it remains useful for data augmentation during training.

Follow-Up Research This Work Enables

Disentangling answerable-question accuracy from unanswerable-question accuracy to diagnose the nature of model failures. The paper reports aggregate F1 that conflates performance on answerable and unanswerable questions. A model achieving 66 F1 could be near-perfect on one subset and near-chance on the other. A strong follow-up would train BNA, DocQA, and DocQA + ELMo on SQuAD 2.0 and report separate accuracies on the answerable and unanswerable subsets of the development and test sets, along with false positive rates (answering when should abstain) and false negative rates (abstaining when should answer). This decomposition would reveal whether the 23.2-point human–machine gap is driven primarily by models failing to recognize unanswerable questions (high false positive rate), by models becoming overly conservative and abstaining on answerable questions they would have gotten right under SQuAD 1.1 (high false negative rate), or by a symmetric degradation on both. The plausible-answer analysis in Table 5 already demonstrates that false positives are systematic rather than random, but does not report the base rate of false positives. Knowing the false positive rate would establish whether the plausible-answer distractors are the dominant source of errors or merely a visible subset.

Per-category model performance on the unanswerable question taxonomy from Table 1. The paper provides a qualitative taxonomy of unanswerable question types—negation (9%), antonym (20%), entity swap (21%), mutual exclusion (15%), impossible condition (4%), other neutral (24%)—but reports no quantitative model performance broken down by these categories. A natural follow-up would annotate the full SQuAD 2.0 development and test sets (or a substantial random sample) with these category labels and evaluate the three baseline models on each category separately. This would test a specific hypothesis: are models equally weak across all categories, or do they handle some reasoning patterns (e.g., simple negation, entity swaps that resemble the RULEBASED transformations they were trained on) better than others (e.g., mutual exclusion, impossible conditions, which require deeper inferential reasoning)? If performance varies substantially by category, this would both explain the aggregate score and guide architecture development—for instance, if models are near human-level on entity-swap negatives but near-chance on mutual exclusion, then the research priority is building mutual-exclusion reasoning capabilities, not general improvements to the no-answer classifier.

Training data ablation: how many unanswerable questions are needed to saturate model performance? The paper trains models on 43,498 unanswerable questions but does not vary this quantity. A learning-curve experiment would train DocQA (or an equivalent architecture) on random subsets of the SQuAD 2.0 unanswerable training data at sizes of 500, 1,000, 2,000, 5,000, 10,000, 20,000, and the full 43,498 examples, evaluating SQuAD 2.0 development performance at each point. If performance saturates at 5,000 examples, then SQuAD 2.0's large training set is adequate and the remaining gap to human performance is an architectural limitation. If performance continues to improve log-linearly with dataset size, then the gap may be closable through additional data collection rather than architectural innovation. This experiment would directly test the paper's implicit claim that the 23.2-point gap represents a capability deficit rather than a data-scale deficit. It would also provide practical guidance for practitioners: how much adversarial annotation is worth investing in for a given domain?

Combining SQuAD 2.0 training with targeted entailment pretraining. The paper connects SQuAD 2.0 to Recognizing Textual Entailment (RTE) in Section 6, but the baseline models are trained exclusively on SQuAD 2.0 data without explicit entailment supervision. A follow-up would pretrain DocQA (or a comparable architecture) on a large RTE dataset (e.g., SNLI, MultiNLI, or the ANLI adversarial NLI dataset from Nie et al., 2020) before fine-tuning on SQuAD 2.0, and compare against SQuAD 2.0-only training. The hypothesis is that explicit entailment training—learning to classify premise–hypothesis pairs as entailment, contradiction, or neutral—would transfer to the unanswerable question detection task, because both require determining whether a text supports a claim. A positive result (significant SQuAD 2.0 improvement from RTE pretraining) would strengthen the paper's framing of SQuAD 2.0 as an entailment task and suggest a concrete path forward for model improvement. A negative result (no transfer) would suggest that SQuAD 2.0's unanswerable questions require different reasoning than standard RTE, perhaps because the "hypothesis" (the question) does not explicitly state the answer but asks for it, requiring an additional inference step beyond entailment classification.

Re-evaluating SQuAD 1.1 human performance under the SQuAD 2.0 multi-annotator protocol to produce a clean human–machine gap comparison. The paper acknowledges that SQuAD 1.1 human performance was measured with a single annotator (likely underestimating true human accuracy) while SQuAD 2.0 uses majority voting across 4.8 annotators. This confounds the headline comparison: the 23.2-point gap on SQuAD 2.0 vs. the 5.4-point gap on SQuAD 1.1 is partially a measurement artifact. A clean follow-up would re-annotate a subset of the SQuAD 1.1 test set using the same multi-annotator majority-vote protocol (same platform, same instructions, same tie-breaking rules, same compensation) and recompute the human SQuAD 1.1 score. This would produce a directly comparable human–machine gap for both datasets, isolating the true difficulty increase from unanswerable questions. If the re-evaluated SQuAD 1.1 human score is, say, 93 F1 instead of 91.2, then the human drop from SQuAD 1.1 to SQuAD 2.0 is larger than 1.7 points, and the machine gap inflation is correspondingly smaller than 23.2 points. This experiment is low-risk and high-value: it would clarify the paper's central quantitative claim without requiring any model development.

Stress-testing SQuAD 2.0's difficulty under modern pretrained transformer architectures. The paper's baselines (BNA, DocQA, DocQA + ELMo) represent the state of the art circa 2018 but predate the transformer revolution. A natural and important follow-up is to evaluate BERT, RoBERTa, T5, and larger language models (GPT-3-scale and beyond) on SQuAD 2.0, both with and without explicit unanswerable-question training. The key question: does the 23.2-point human–machine gap persist, shrink, or close entirely when models have orders of magnitude more parameters, are pretrained on massive corpora with masked language modeling or next-token prediction objectives, and have attention mechanisms that can model long-range dependencies? If the gap largely closes (e.g., T5 achieves 85+ F1 on SQuAD 2.0), then SQuAD 2.0's primary contribution was as a diagnostic for a specific architectural era, and its enduring value is as a historical benchmark rather than a frontier challenge. If a substantial gap remains (e.g., the best model achieves 75 F1, still 14+ points below humans), then SQuAD 2.0 genuinely tests a capability—entailment verification in reading comprehension—that scale and pretraining alone do not solve, and the paper's framing as exposing a fundamental capability gap is validated across architectural paradigms. Either outcome is informative; the experiment is straightforward given the public release of the dataset.

Practical Applications and Downstream Use Cases

Trustworthy document-grounded QA for enterprise knowledge bases. An organization deploying a QA system over internal documentation (policies, technical manuals, legal contracts) faces a critical requirement: the system must not fabricate answers when the document lacks the relevant information. A system trained or evaluated only on SQuAD 1.1—where every question has an answer—will silently produce its best guess regardless of whether the guess is supported, creating a risk of confident misinformation. Training and evaluating on SQuAD 2.0 directly addresses this by forcing the model to learn an abstention capability. The paper's always-abstain baseline at 48.9 F1 (Section 5.2) provides a floor: a deployed system that abstains when uncertain avoids the worst failure mode (confidently wrong answers) at the cost of reduced coverage. A system achieving 66 F1 (DocQA + ELMo) is 17.4 points above this floor, meaning it is usefully better than always-abstain while still having substantial room for improvement. For an enterprise setting where false positives carry high cost (e.g., a policy question answered with an incorrect provision that leads to a compliance violation), SQuAD 2.0-style training and evaluation is directly preferable to SQuAD 1.1-style deployment, even at current model performance levels.

Pre-deployment filtering of unanswerable user queries in open-domain QA. Open-domain QA systems that retrieve documents and then extract answers (the retriever–reader paradigm) can encounter questions for which no retrieved document contains the answer. If the reader component is trained only on SQuAD 1.1, it will extract the most relevant-looking span from whatever document is retrieved, producing an answer even when no correct answer exists in the retrieval set. Integrating a SQuAD 2.0-trained reader—or applying the SQuAD 2.0 evaluation framework to an existing reader by adding a no-answer prediction head—allows the system to output "no answer found" when the retrieved evidence does not support any answer. This is a direct practical application of the paper's core diagnostic: the 23.2-point gap between SQuAD 1.1-style and SQuAD 2.0-style performance on the same underlying answerable questions (Table 3, comparing DocQA + ELMo's 85.8 F1 on SQuAD 1.1 to 66.3 F1 on SQuAD 2.0) quantifies how many of the answers a SQuAD 1.1-trained system produces would be unreliable when unanswerable questions are present. A system builder can use this gap to estimate the false-positive rate their pipeline would experience in the wild and decide whether the accuracy–coverage tradeoff is acceptable.

Adversarial data augmentation for domain-specific reading comprehension. The paper's crowdworker methodology—showing workers paragraphs and existing answerable questions, instructing them to write unanswerable questions that look similar while highlighting plausible distractors—is directly transferable to any domain where extractive QA is deployed. A team building a medical QA system over clinical trial reports could replicate the SQuAD 2.0 procedure: hire domain-expert annotators (or train crowdworkers on domain terminology), provide paragraphs from clinical trial reports, show existing answerable questions as style references, and collect adversarially written unanswerable questions with plausible answers. The paper's finding that adversarial human-written negatives produce a dramatically harder dataset than automatic methods (15–22 F1 gap, Table 4) provides the motivation for investing in human annotation rather than relying on cheap distant supervision. The specific numbers—7 minutes per paragraph, $10.50 per hour, up to five questions per paragraph, 25-question minimum filter—give a concrete recipe that a practitioner can budget against. The 93% cleanliness rate from manual inspection (Section 4.3) provides a quality benchmark: if a domain-specific replication achieves substantially lower cleanliness, the annotation instructions or worker training need refinement.

When to Prefer This Method

The paper positions SQuAD 2.0 as a replacement for SQuAD 1.1 as the primary extractive QA benchmark, not as one option among many. It does not articulate a tradeoff where SQuAD 2.0 is preferred over SQuAD 1.1 in some scenarios and not others—the paper's stance is that SQuAD 2.0 subsumes SQuAD 1.1 and should be used instead for any evaluation that purports to measure reading comprehension. The official SQuAD leaderboard migration (announced in Section 1) makes this preference concrete. Similarly, the paper positions its adversarial human-written unanswerable questions as strictly superior to automatically generated negatives (TF-IDF, rule-based) for evaluation purposes, based on the experimental evidence in Table 4 that automatic negatives produce datasets 15–22 F1 points easier. There is no scenario the paper endorses where a practitioner should prefer automatic negative generation over adversarial human writing for benchmark construction.

The only implicit tradeoff the paper acknowledges is between benchmark difficulty and construction cost: automatic negatives are cheap but easy; adversarial human negatives are expensive but hard. The paper does not quantify the cost differential (total annotation cost is not reported) and does not provide guidance on when the cost is justified. A practitioner reading between the lines can infer that adversarial human construction is warranted when the goal is to measure genuine entailment verification capability (for a public benchmark or a safety-critical deployment evaluation) and that automatic methods may suffice for data augmentation during training, where quantity matters more than per-example difficulty. This inference is reasonable given the paper's evidence, but the paper itself does not draw this distinction or provide experimental support for it (e.g., by training models on TF-IDF augmented data and testing on SQuAD 2.0). The preference for SQuAD 2.0 over SQuAD 1.1 is absolute in the paper's framing; the preference for adversarial human construction over automatic generation is empirically grounded but lacks cost–benefit analysis.