The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs

🎯 Pitch

Even massive language models like GPT-3 barely beat random chance when interpreting implied meaning—like understanding “I have to work” as a polite “no.” Yet instruction-tuning at the example level, not scale or standard fine-tuning, yields models that nearly match humans on such implicatures, revealing that how you teach a model matters far more than how big it is for pragmatic competence.

1. Executive Summary

This paper studies whether large language models can resolve conversational implicatures—utterances that convey meaning beyond their literal words (e.g., responding “I have to work” to a party invitation implies “no”)—by evaluating four categories of state-of-the-art models on a curated dataset of naturally occurring binary implicatures. The core finding is that instruction-tuning at the example level (fine-tuning on tasks where each individual example receives its own natural-language instruction, as in OpenAI’s InstructGPT series) produces substantially better pragmatic understanding than large-scale pretraining alone, dialogue-specific fine-tuning, or benchmark-level instruction-tuning—with the best example-tuned model (GPT-4) reaching 86.5% accuracy with chain-of-thought prompting, effectively matching average human performance at 86.2%. Pretrained base models perform close to random even at massive scale (GPT-3-175B achieves only 68.7% five-shot accuracy), while models fine-tuned on conversational data (BlenderBot) or benchmark-level instructions (T0, Flan-T5) show minimal improvement over chance, establishing that pragmatic competence does not emerge from pretraining scale or standard fine-tuning recipes—only from the fine-grained diversity of example-level instruction data, and even then largely on context-free “generalized” implicatures while context-heavy “particularized” examples remain challenging.

2. Context and Motivation

The Core Problem: Evaluating Communication, Not Just Language Production

The fundamental problem this paper tackles is that current LLM evaluations fail to capture a crucial dimension of human language: pragmatics—the ability to interpret language in context using beliefs, world knowledge, and social conventions. The authors open with a stark illustration: when prompted "Have you seen my phone?", GPT-3 responds "Yes, I have seen your phone" (Section 1). This is semantically coherent and factually plausible, but pragmatically inept. A human would recognize that the questioner isn't asking about visual detection but about location, and would respond with something like "it's in your bag"—bypassing the obvious follow-up question. This is pragmatic language use, which goes beyond producing grammatically correct sentences to actually communicating efficiently and helpfully.

The authors argue this distinction between linguistic production and genuine communication is not a minor nuance—it's fundamental to what it means to understand language. They frame it through the concept of implicature, an utterance that conveys something other than its literal meaning. Consider their example: Esther asks "Can you come to my party on Friday?" and Juan responds "I have to work." Juan never says "no" explicitly, but humans effortlessly resolve his response as a rejection by using the contextual commonsense knowledge that having to work precludes party attendance. As the authors state in Section 1:

"Both these exchanges contain an implicature—utterances that convey something other than their literal meaning. Implicatures illustrate how context contributes to meaning; distinguishing writing and speaking from communicating (Green,, 1996). We cannot fully understand utterances without understanding their implications."

This is not a niche linguistic curiosity. The term "communication" itself presupposes that the speaker's implications are understood by the addressee. Being able to resolve completely novel implicatures constitutes an essential and ubiquitous aspect of everyday language use. If LLMs are being deployed as conversational agents—and they increasingly are—they must be able to navigate this contextual commonsense knowledge, not just produce fluent text.

Why This Matters: Real-World Impact and Theoretical Significance

The paper identifies two converging trends that make this evaluation gap urgent:

First, the deployment trajectory. LLMs are rapidly moving from text generators to conversational agents. The authors cite work on using LLMs for planning (Huang et al., 2022), commonsense reasoning (Kojima et al., 2022), and as foundations for interactive agents (Adolphs et al., 2022; Reed et al., 2022). When fine-tuned with human feedback, these models obtain higher ratings on desiderata like helpfulness (Ouyang et al., 2022; Bai et al., 2022) and are proposed as conversational agents (Thoppilan et al., 2022). Yet the evaluation benchmarks used to measure this progress—and to align models with human values—do not explicitly capture pragmatic communication skills. The authors note in Section 2:

"What is helpful and informative, as Kasirzadeh and Gabriel, (2022) also point out, depends on the context in which a conversation is held. Consequently, any application that requires communicating with humans will rely on pragmatic communication skills—something that is not explicitly captured by the benchmarks used to evaluate the alignment of LLMs."

An LLM that is "helpful" in the abstract but cannot infer that "I have to work" means "no" is not actually helpful in conversation. The gap between benchmark performance and real-world communicative competence is a deployment risk that the paper argues has been systematically overlooked.

Second, the theoretical puzzle. Pragmatic understanding is acquired naturally by humans without explicit instruction—it emerges from the interplay of linguistic experience and general reasoning about the world. If LLMs are trained on vast corpora of human language, should pragmatic competence not emerge in the same way? The fact that it largely doesn't (as the paper's results will show) raises fundamental questions about what aspects of human-like language understanding are captured by next-word prediction alone, and what requires additional training signals or architectural innovations. This connects to broader debates about whether LLMs understand language or merely exhibit sophisticated pattern matching.

Prior Approaches and Where They Fall Short

The paper positions itself against several strands of prior work, each of which it argues is insufficient:

1. Existing pragmatic evaluation is narrow in scope. There is a body of computational work on pragmatics, but it focuses on specific linguistic phenomena rather than the broad, naturalistic implicatures that dominate everyday conversation. Cianflone et al. (2018) study adverbial presupposition triggers (words like "again" that trigger unspoken presuppositions). Schuster et al. (2020) study scalar inferences (e.g., "some" implying "not all"). Kim et al. (2021) look at false presuppositions in question-answering datasets. Jeretic et al. (2020) and Parrish et al. (2021) are early works on pragmatic understanding in pretrained language models, but they only examine scalar implicatures and presuppositions—not the full range of conversational implicatures. As the authors note, these studies are valuable but miss the broader phenomenon: everyday conversation is filled with implicatures that don't fall neatly into these linguistic categories (e.g., "I have to work" in response to a party invitation).

2. The BIG-bench implicature task has critical methodological flaws. The only prior evaluation that specifically targets implicature resolution in LLMs at scale is the BIG-bench implicature task (Srivastava et al., 2022). The paper identifies three serious problems with this evaluation (Section 2 and Appendix I):

Ambiguous data is discarded. The BIG-bench task contributors removed examples they deemed ambiguous. The authors argue this defeats the purpose: "Implicatures are a type of non-literal, ambiguous language the intended meaning of which humans often easily interpret; comparing the way humans and models do this is precisely what we are interested in." By filtering out ambiguity, BIG-bench likely overestimates LLM performance. The paper's dataset retains these challenging examples—which constitute about 30% of the benchmark—and uses human evaluation to handle genuine ambiguity instead.
Low-quality human evaluation. The difference between BIG-bench's average and best human rater is 18%, compared to 6% for this paper's evaluations. This suggests poor annotation quality, but it's impossible to verify because BIG-bench provides no details on how the human evaluation was conducted.
Only base LLMs are evaluated. BIG-bench uses only pretrained models with no fine-tuning. This leaves open the crucial question the paper investigates: do different fine-tuning strategies affect pragmatic competence?

3. Zheng et al. (2021) made important progress but has limitations. This earlier work was the first to evaluate pretrained language models on conversational implicatures. It demonstrated that implicature is challenging for language models. However, the paper identifies two key limitations: the evaluations require task-specific training (not applicable out-of-the-box), and the models evaluated are relatively small by contemporary standards. Critically, Zheng et al. use synthetic data rather than naturally occurring implicatures. The paper argues this is a significant weakness because synthetic examples may not capture the true distribution of implicatures in natural dialogue—the ambiguity, the reliance on variable world knowledge, the dependence on specific conversational contexts.

4. Standard benchmarks don't capture pragmatic ability. The paper makes the broader point that the standard suite of NLP benchmarks—even those that evaluate "commonsense reasoning" or "helpfulness"—do not specifically target pragmatic understanding. Models can score well on these benchmarks while failing to resolve basic conversational implicatures. This is not just a benchmarking oversight; it means that the development trajectory of LLMs (what gets measured gets optimized) may be systematically neglecting pragmatic competence.

How This Paper Positions Itself

The paper positions itself as filling a specific gap: providing a comprehensive, out-of-the-box evaluation of pragmatic understanding across the full landscape of contemporary LLMs, with a focus on identifying what aspects of training induce this ability. This is not a paper proposing a new architecture or training method—it's a systematic investigation of an underexplored capability.

Several design choices reflect this positioning:

Naturally occurring implicatures over synthetic data. The authors use a dataset of conversational implicatures curated by George and Mamidi (2020) from real dialogues, arguing this "to be a better representation of the true distribution of implicatures in natural discourse." They keep all ambiguous examples, dealing with ambiguity through human evaluation rather than filtering—which they argue is more ecologically valid.
Likelihood ranking over generation. The evaluation protocol (described in Section 3) uses likelihood comparison rather than greedy decoding. A model resolves an implicature correctly if it assigns higher likelihood to a coherent utterance than an incoherent one (where "yes" is swapped for "no" or vice versa). This is a more lenient protocol than requiring exact generation, but the authors argue it still captures implicature resolution because the choice between "yes" and "no" is determined solely by the resolution of the implicature. It also avoids the problem that there are multiple coherent continuations beyond just "yes" or "no."
Four deliberately chosen model categories. The paper evaluates base pretrained models (GPT-3, OPT, BLOOM, etc.), dialogue-fine-tuned models (BlenderBot), benchmark-level instruction-tuned models (T0, Flan-T5), and example-level instruction-tuned models (InstructGPT variants, Cohere-command). This taxonomy is not arbitrary—each category represents a distinct hypothesis about where pragmatic competence might emerge: from scale alone, from conversational training, from broad multitask training, or from fine-grained instruction following. By comparing across these categories, the paper can disentangle which factors matter.
Comprehensive prompt control. Recognizing that LLMs are sensitive to prompt wording, the authors use six different prompt templates (three "structured" and three "natural") per evaluation, and test multiple $k$ -shot settings ( $k \in \{0, 1, 5, 10, 15, 30\}$ ). This allows them to distinguish robust effects from prompt-specific artifacts.
Human baseline with quality control. The paper conducts its own human evaluation (20 annotators, 5 per example in the test set) with attention checks and proper compensation, providing a reliable human benchmark (86.2% average, 92% best) against which to compare models. This addresses the quality concerns with the BIG-bench human evaluation.
Connection to the alignment agenda. The paper explicitly frames pragmatic understanding as part of the alignment problem. In Section 2, they note that models fine-tuned with human feedback are evaluated on "informativeness" and "helpfulness," but these are context-dependent properties that require pragmatic reasoning. An LLM that can't infer implicatures will be systematically unhelpful in ways that current benchmarks miss. This connects the paper's investigation to the broader conversation about what it means for AI systems to be "aligned" with human communication norms.

In essence, the paper positions itself as a wake-up call and diagnostic tool: implicature resolution is a critical capability that current evaluations overlook, most models fail at it despite impressive performance elsewhere, and understanding which training strategies help (and which don't) is essential for building genuinely useful conversational AI. The four-category taxonomy of models isn't just descriptive—it's designed to isolate the causal factors behind pragmatic competence, providing actionable guidance for model development rather than just documenting a limitation.

3. Technical Approach

3.1 Reader Orientation

This paper is an empirical evaluation study — not a system-building paper. There is no new model, no new training procedure, and no new architecture. Instead, the authors design a minimal evaluation protocol that probes whether off-the-shelf LLMs can resolve conversational implicatures, and they systematically apply this protocol across four categories of models to identify which training strategies induce pragmatic understanding. The core idea is deceptively simple: take a conversational implicature (an utterance-response pair where the response implies "yes" or "no" without saying it), wrap it in a prompt template, create an "incoherent" version by swapping "yes" to "no" (or vice versa), and check whether the model assigns higher likelihood to the coherent version. If it does, the model has resolved the implicature. The paper solves the problem of diagnosing pragmatic competence — a capability that standard benchmarks miss entirely — by providing a reproducible, prompt-robust, human-calibrated measurement framework that can be applied out-of-the-box to any language model, whether accessed through an API or run locally.

3.2 Big-Picture Architecture (Diagram in Words)

The evaluation system has five major components:

Dataset (George and Mamidi, 2020): A curated collection of 718 binary conversational implicatures extracted from natural English dialogue, each presented as an utterance-response-implicature triple. The dataset provides the ground-truth labels ("yes" or "no") that define what pragmatically coherent resolution looks like.
Prompt Templates: Six manually designed text templates (three "structured", three "natural") that wrap each example into a complete sentence or instruction for the model. These control for prompt wording sensitivity — a well-known confound in LLM evaluation — by measuring whether performance is robust across different phrasings.
Coherence Test Instances: For each original example, an incoherent counterpart is generated by swapping "yes" to "no" (or vice versa) in the template. This creates a minimal pair: two prompts identical except for a single word, where only the pragmatically coherent one makes sense. The model's preference between these two becomes the evaluation signal.
Likelihood Scoring Mechanism: For each prompt template and each test instance (both coherent and incoherent), the model's assigned likelihood (probability under its parameter distribution) is computed. A model resolves the implicature correctly if it assigns higher likelihood to the coherent sentence than the incoherent one. This is equivalent to a forced-choice comprehension test without requiring the model to generate the correct word.
Few-Shot Context Builder: For $k$ -shot evaluation, $k$ examples are randomly sampled from a held-out development set (118 examples) and prepended to the test prompt. The same randomly sampled examples are used across all models for a given $k$ , ensuring fair comparison. This component tests whether in-context learning can improve implicature resolution beyond zero-shot.

Information flows as follows: a test example enters the system → the prompt template wraps the utterance, response, and implicature into a complete text → two versions are created (coherent and incoherent) → for few-shot evaluation, $k$ examples are prepended → the model computes a likelihood for each version → the higher-likelihood choice is compared to the ground truth → accuracy is aggregated across 600 test examples, six prompt templates, and multiple $k$ values.

3.3 Roadmap for the Deep Dive

First, the dataset and its properties — what kind of implicatures it contains, how many, and why this specific dataset was chosen over alternatives (including the BIG-bench version that uses the same underlying data but discards 30% of it).
Second, the prompt template design — the six templates, their categorization into "structured" and "natural" types, and the rationale for using multiple templates to control for prompt sensitivity.
Third, the coherence test construction — how the minimal pairs are created, why likelihood ranking is preferred over greedy decoding, and what the formal correctness criterion is.
Fourth, the few-shot evaluation protocol — how in-context examples are sampled, why random sampling with fixed seeds is used, and what the development set is.
Fifth, the human evaluation design — how human baselines are collected, how annotators are recruited and compensated, and how attention checks and inter-annotator agreement are measured.
Sixth, the model categorization framework — the four groups (base, dialogue FT, benchmark IT, example IT), why this taxonomy matters for causal inference about training strategies, and which specific models fall into each group.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an evaluation methodology paper whose core contribution is a measurement protocol for pragmatic understanding in LLMs. The "system" is not a trained model but an evaluation framework — a carefully designed set of procedures that transforms a dataset of conversational implicatures into a quantitative measure of a model's ability to resolve pragmatic meaning. The protocol is designed to be applied out-of-the-box to any language model, requiring no task-specific fine-tuning, and to produce results that are robust to prompt wording and comparable across model families.

The Dataset: Conversational Implicatures from Natural Dialogue

The evaluation is built on top of a dataset curated by George and Mamidi (2020), which the authors describe as containing "implicatures that are presented in utterance-response-implicature tuples." The dataset consists of 1001 examples extracted from natural English dialogues. Each example is a triple: an utterance (a question or statement by Speaker A), a response (Speaker B's reply), and an implicature (the implied "yes" or "no" meaning that resolves the response's pragmatic intent).

Filtering to binary implicatures. Of the 1001 examples, the authors identify 718 that are binary implicatures — implicatures whose resolution is specifically "yes" or "no." The paper focuses exclusively on these because the evaluation protocol requires a clean binary choice for the coherence test (swapping "yes" and "no" creates a clear minimal pair). Examples that imply more complex propositions (e.g., "Here's the key to my office" implying multiple things: you can use the stapler, it's in the office, and the office is locked) are excluded from the main evaluation, though the authors note this as a limitation and a direction for future work in the Discussion (Section 5):

"the likelihood ranking approach we take limits our study to implicatures with clear alternative. However, implicatures in natural language can entail more complex propositions. For example, imagine Esther now asking 'Can I use your stapler?' and Juan responding 'Here's the key to my office.' Juan is implicating that (1) Esther can use the stapler, (2) the stapler is located in the office, and (3) the office is currently locked. This leaves ample room for the design of benchmarks with implicatures entailing multiple non-binary propositions."

Train/test split. From the 718 binary implicatures, the authors randomly sample 600 examples for the test set and keep the remaining 118 as a development set. The development set serves a specific purpose: it provides the pool of examples from which few-shot in-context demonstrations are drawn. Critically, the development set is never used for model training, hyperparameter tuning, or strategy selection — it exists solely to supply in-context prompts during few-shot evaluation. This means there is no risk of test-set contamination from the few-shot examples.

Why this dataset over alternatives? The paper makes an explicit argument for using George and Mamidi's naturally occurring implicatures rather than synthetic data (as in Zheng et al., 2021) or the filtered version used in BIG-bench. The key claims are:

Natural distribution: The implicatures come from real dialogues, capturing the ambiguity, context-dependence, and reliance on variable world knowledge that characterize pragmatic language use in the wild. Synthetic data risks being too clean — missing the edge cases and ambiguities that make implicature resolution genuinely challenging.
Retaining ambiguity: The BIG-bench task contributors discarded approximately 30% of the data as "ambiguous according to them" (Appendix I). The paper argues this defeats the purpose: implicatures are inherently ambiguous, and comparing how humans and models handle this ambiguity — where they agree, where they diverge — is precisely the scientific question. By keeping the full dataset (including the 189 examples BIG-bench discarded from what would be the test set), the paper provides a more realistic and challenging evaluation. The authors demonstrate the importance of this choice empirically: on the subset of ambiguous data that BIG-bench removed, humans in their study achieve 71.9% accuracy — well above chance but significantly below the 86.2% average, confirming these are harder examples that nevertheless humans can often resolve. A model evaluated only on the "clean" subset would appear more competent than it actually is.
Manual type annotation: To enable fine-grained analysis of what kinds of implicatures models struggle with, the authors manually labeled 217 of the 600 test examples according to a taxonomy (described in Insight 5 of Section 4 and detailed in Appendix K.9). The types are: generalized (47 examples) — requiring little or no context, like "some" implying "not all"; particularized (94 examples) — requiring specific context, like "I have to work" implying "no" to a party invitation; world knowledge (23 examples) — requiring knowledge of physical facts, like "I wore gloves" implying "no" to "Did you leave fingerprints?"; idiom (42 examples) — containing an idiom or metaphor, like "He's all bark and no bite" implying "no" to "Would he fire me?"; rhetorical question (11 examples) — containing questions with obvious answers, like "Can fish swim?" implying "yes"; and other (383 examples) — not clearly falling into any category. This taxonomy enables the paper to move beyond aggregate accuracy and understand where model failures concentrate.

Example of a dataset entry. To make this concrete, consider one example from the paper's introduction, as it would appear in the dataset:

Utterance: "Can you come to my party on Friday?"
Response: "I have to work"
Implicature: no

This triple captures Juan's response as pragmatically meaning "no" — an inference that requires the commonsense knowledge that having to work precludes party attendance. The dataset contains 718 such triples, each with a binary "yes" or "no" implicature.

Prompt Template Design: Controlling for Sensitivity to Wording

A well-established finding in LLM evaluation is that models are sensitive to prompt wording — the same task presented with different phrasing can yield substantially different results (the paper cites Efrat and Levy, 2020; Tan et al., 2021; Reynolds and McDonell, 2021a, 2021b; Webson and Pavlick, 2021). To control for this confound, the authors manually curate six different prompt templates and measure performance across all of them for every evaluation.

The six templates. The templates are listed in full in Appendix F, Table 5. They fall into two groups of three:

Structured prompts (Templates 1, 3, and 4) use a more formal, key-value format:

Template 1: "Does the following response to the question imply yes or no? question: <utterance> response: <response> implicature: <implicature>"
Template 3: "Is the implied meaning of the following response yes or no: question: <utterance> response: <response> meaning: <implicature>"
Template 4: "What is the intent of the following response, yes or no? question: <utterance> response: <response> intent: <implicature>"

Natural prompts (Templates 2, 5, and 6) embed the example in a pseudo-narrative:

Template 2: "Finish the following text: Esther asked "<utterance>" and Juan responded "<response>", which means <implicature>"
Template 5: "Finish the following text: Karen asked "<utterance>" and William responded "<response>", which means <implicature>"
Template 6: "Finish the following text: Bob asked "<utterance>" and Alice responded "<response>", which means <implicature>"

Design rationale. The two groups represent fundamentally different ways of presenting the task: structured templates frame it as an explicit question-answering task with labeled fields, while natural templates embed it in a narrative that the model simply continues. Within each group, the templates differ only slightly in wording (e.g., "implicature" vs. "meaning" vs. "intent", or "Esther and Juan" vs. "Bob and Alice"). This within-group variation isolates the effect of small lexical choices, while the between-group variation captures larger differences in task framing. By reporting the mean and standard deviation over all six templates, the paper provides a measure of both absolute performance and robustness to prompt wording.

Forced-choice vs. instruction-following. For models where likelihoods are accessible (all open-source models and most API models), the templates are presented as incomplete text that the model is asked to "finish" — the model never has to follow an explicit instruction to output "yes" or "no." Instead, its likelihood for the complete prompt (with the implicature filled in) is directly compared to the likelihood for the incoherent version. For two model classes — GPT-3.5-turbo and GPT-4 — where the API does not provide token-level likelihoods, the templates are modified to use greedy decoding with explicit instructions to output only "yes" or "no" (see Appendix F, Table 6 for the modified completion templates). The authors note (Section 3):

"For two model classes (i.e. GPT-3.5-turbo and GPT-4) we do not have access to likelihoods, and for these models we take the greedy decoding approach, guiding the model to output 'yes' or 'no' explicitly in all prompts"

This creates a slight methodological inconsistency — GPT-3.5-turbo and GPT-4 are evaluated in a different mode than other models — but the authors argue this is acceptable because the forced-choice between "yes" and "no" is still determined by implicature resolution, and the greedy decoding approach is actually more stringent (it requires the model to generate the exact correct token, not just prefer it probabilistically). The paper's overall conclusions do not hinge on this difference, as the Example IT models that perform best include both likelihood-evaluated models (text-davinci-001/002/003) and completion-evaluated models (GPT-3.5-turbo, GPT-4), and the pattern of results is consistent across both evaluation modes.

The Coherence Test: Likelihood Ranking as an Implicature Resolution Probe

The core evaluation mechanism is a minimal pair coherence test. For each example and each prompt template, two versions are created:

Coherent version (x): The prompt template with the ground-truth implicature filled in. For example, using Template 2: "Finish the following text: Esther asked 'Can you come to my party on Friday?' and Juan responded 'I have to work', which means no."

Incoherent version (x̂): The identical prompt template but with the opposite implicature. For the same example: "Finish the following text: Esther asked 'Can you come to my party on Friday?' and Juan responded 'I have to work', which means yes."

The two versions differ in exactly one word — "no" versus "yes" — and every other token is identical. This means the model's relative likelihood for the two versions is determined entirely by whether it finds "no" or "yes" to be the more pragmatically coherent continuation.

Formal correctness criterion. The paper defines the correctness condition precisely (Section 3):

"A model outputs a likelihood $p$ parameterized by weights $\theta$ . We say a model correctly resolves an example $x$ when it assigns $p_\theta(x) > p_\theta(\hat{x})$ ."

In plain language: the model is correct if it assigns higher probability to the coherent sentence than to the incoherent one. This is the full correctness criterion — there is no partial credit, no thresholding, and no normalization across examples. For each test example and each prompt template, the model either satisfies $p_\theta(x) > p_\theta(\hat{x})$ or it does not.

Why likelihood ranking rather than greedy decoding? The paper explains two key advantages (Section 3):

Lenience with respect to correct continuations. The word "no" is not the only coherent continuation of the template — the model might reasonably continue with "that he cannot come" or "declining the invitation" or any number of paraphrases. Greedy decoding would penalize all of these as "wrong" even if the model clearly understands the implicature, because it evaluates whether the single most likely token is "no." Likelihood ranking avoids this problem: it asks whether "no" is more likely than "yes," not whether "no" is the single most likely continuation overall. The paper states:

"The more lenient evaluation does capture implicature resolution, because the choice of 'no' versus 'yes' is only determined by the resolution of the implicature."

Marginalization over alternative correct answers. If the model distributes probability mass across multiple correct answers (e.g., "no," "declining," "that he cannot come"), the greedy decoding probability for any single one might be low, but their collective probability versus incorrect "yes" continuations would be high. The likelihood ranking approach captures this without requiring explicit marginalization.

What likelihood ranking does NOT require. It's worth being explicit about what this evaluation does not measure. It does not require the model to generate the implicature, to explain its reasoning, or to produce any text at all. It does not require the model to follow instructions (for the likelihood-based evaluation). It does not require the model to understand the concept of "implicature" or "pragmatics." It is a pure comprehension probe: given the linguistic context, does the model find the pragmatically coherent reading more probable than the incoherent one?

Limitations of this approach. The paper acknowledges a limitation in Section 5: the likelihood ranking approach restricts the evaluation to implicatures with a clear binary alternative ("yes" vs. "no"). This excludes implicatures that imply more complex propositions, as discussed in the dataset section above. However, within the scope of binary implicatures, the approach is well-motivated: it provides a clean, unambiguous, and computationally cheap signal of pragmatic understanding that can be applied uniformly across all models.

Few-Shot Evaluation Protocol: In-Context Learning of the Task

In addition to zero-shot evaluation (where the model sees only the test example wrapped in a template), the paper evaluates few-shot performance for $k \in \{0, 1, 5, 10, 15, 30\}$ . The few-shot protocol adds $k$ completed examples from the development set to the prompt before the test instance.

How few-shot prompts are constructed. For each test example and each value of $k$ , the authors sample $k$ examples from the development set (118 examples) and prepend them to the prompt. The $k$ examples are presented as complete demonstrations: they include the utterance, response, and the correct implicature. The test example then appears at the end, with the implicature position left for the model to complete (in the likelihood ranking, both "yes" and "no" versions are evaluated; in few-shot, the model sees $k$ examples of the task format before the test instance).

Random sampling with fixed seeds. The paper makes an important methodological choice about how the $k$ examples are selected (Section 3):

"We evaluate the models' $k$ -shot capabilities by randomly sampling $k$ examples from the development set for each test example. We opt for a random sampling approach to control for two sources of randomness. Firstly, examples have different levels of informativeness. Secondly, recent work found that the order in which examples are presented matters (Lu et al., 2022)."

The ideal approach would be to evaluate each test example with all permutations of $k$ examples from the development set, which would require $118! / (118-k)!$ evaluations per test example — completely intractable. Instead, the authors sample one random ordering per test example and use that same ordering across all models. This ensures that differences between models are attributable to the models' capabilities, not to variations in which few-shot examples they happened to receive.

The paper further verifies in Appendix K.2 that the variance due to prompt ordering is small. They run the 5-shot evaluation for the best-performing model (text-davinci-002) with three different random seeds (different orderings of the same set of examples) and find that the standard deviation across seeds ranges from 0.08 to 1.32 percentage points per prompt template — well within the range of other sources of variance and not enough to change any conclusions.

Why the development set is not used for tuning. The 118 examples in the development set are used exclusively as a pool for few-shot demonstrations. They are never used for hyperparameter selection, prompt engineering, or model selection. The paper explicitly states (Section 3): "We randomly sample 600 examples for the test set and keep the remaining 118 as a development set to improve implicature resolution after pre-training through in-context prompting or fine-tuning." The development set's only role is to provide in-context examples — there is no "development" in the traditional sense of tuning.

The few-shot baseline does not "teach" pragmatics in-context. An important nuance that emerges from the experiments (Insight 1 and Appendix K.6) is that for the Example IT models, the few-shot examples do not appear to teach pragmatic reasoning in context. Instead, they serve primarily to prime the model toward the task format — specifically, to calibrate the model to output "yes" or "no" as the relevant continuations. The paper demonstrates this with a random-label experiment (Appendix K.6): when the few-shot examples are given with random (incorrect) labels instead of the true labels, the performance of Example IT models does not degrade. This confirms that the content of the examples matters less than their structural role in establishing the yes/no output format. For base models, random labels do degrade performance somewhat, suggesting that these models are using the content of the demonstrations more substantively.

Human Evaluation Design: Establishing a Reliable Baseline

To determine what "good" performance looks like on this task, the paper conducts its own human evaluation. This is essential because the task is subjective in borderline cases: some implicatures admit multiple interpretations, and what counts as "correct" depends on the conversational norms of fluent English speakers.

Annotator recruitment and compensation. The authors recruited 20 native English speakers through Prolific, a research participant platform. The paper explicitly states the compensation and ethical compliance (Appendix C): "We compensated participants with a UK living wage at 15 GBP an hour, which is 6 GBP an hour more than Prolific recommends at 9 GBP per hour." Prolific complies with GDPR and UK ethical standards. No demographic data beyond native English speaker status was collected. The total cost for the human evaluation was 236 GBP (approximately $300 USD at the time of writing), detailed in Appendix H.

Task design. The 600 test examples were divided into four non-overlapping subsets of 150 examples each. Each subset was annotated by five unique annotators, meaning each example received five independent judgments. The examples were presented in a Google Form, wrapped in Template 2 (the "natural" prompt format: "Esther asked ... and Juan responded ..., which means ..."), with annotators asked to "choose the correct continuation, yes or no" (Appendix H, Figure 6). The decision to use only one prompt template for the human evaluation (rather than all six) was a practical one: applying all six templates would have required six times the annotation budget. The authors note this as a limitation but argue it is acceptable because humans are far less sensitive to minor wording variations than language models.

Attention checks. To ensure annotators were paying attention, the authors inserted three "catch" questions at random positions in the form. These questions did not contain actual implicatures — the response directly and obviously gave away the correct answer (e.g., the response explicitly says "yes"). Annotators who failed at least two of these three questions were considered inattentive. In practice, exactly one annotator was excluded by this criterion; they were paid regardless but their data was discarded.

Human performance results. The results are reported in Appendix H, Table 10:

Average human accuracy: 86.2% ± 2.3 (standard deviation across the four subsets)
Best annotator accuracy: 89.8% ± 2.2
Worst annotator accuracy: 83.5% ± 1.5
Inter-annotator agreement (Cohen's kappa): 0.72 averaged across subsets, described as "substantial" according to standard interpretation guidelines (0.61–0.80)

The relatively small gap between average and best human performance (6 percentage points) contrasts sharply with the BIG-bench implicature task, where the gap was 18 percentage points. The paper argues this indicates higher-quality annotation in their study.

Analysis of human errors. The authors conducted a qualitative analysis of examples where most human annotators disagreed with the ground-truth label (Appendix H). They identified two main sources:

Annotation errors in the original dataset (18 of 30 examples): Some ground-truth labels are simply wrong. For example: "Are you busy?", "I'm drowning in work.", implicature: "no" — but "drowning in work" clearly means "yes, I am busy."
Multiple valid interpretations (12 of 30 examples): Some examples are genuinely ambiguous, and the ground truth chose an interpretation that feels unnatural to most humans. For example: "You don't remember them?", "Leave me alone!", implicature: "yes" — but "Leave me alone" could just as naturally imply "no, I don't want to talk about it."

This analysis reveals that the human performance ceiling on this dataset is not 100% — some examples have ambiguous ground truth, and some ground-truth labels are erroneous. The paper uses the human average (86.2%) and best (92%) as reference points rather than expecting models to reach 100%.

Why 86.2% is a meaningful ceiling. The fact that humans achieve 86.2% on a dataset of naturally occurring implicatures — many of which are genuinely ambiguous — establishes that this is a non-trivial task even for fluent speakers. Expecting an LLM to significantly exceed human performance would imply the model is better at resolving conversational implicatures than the humans who produced them, which is implausible. The paper therefore treats the human average as an approximate upper bound: models that reach 86% are performing at human level on this specific task, and further gains likely reflect overfitting to annotation artifacts rather than genuine pragmatic understanding.

Model Categorization: Four Groups as a Causal Probe

The paper evaluates a wide range of models not just to establish a leaderboard, but to disentangle which aspects of LLM training contribute to pragmatic understanding. To do this, they categorize models into four groups based on their training methodology (Section 4, with detailed model listings in Appendix G, Table 9):

Group 1: Base Models (large-scale pretrained). These are models trained only with next-word prediction on large text corpora, with no task-specific fine-tuning or instruction following. The group includes: BERT (110M), RoBERTa (125M, 355M), GPT-2 (354M, 774M, 1.6B), EleutherAI models (125M through 20B), BLOOM (560M through 176B), OPT (125M through 175B), Cohere's base models (409M through 52.4B), and GPT-3 (estimated 350M through 175B). If pragmatic understanding emerges purely from scale and next-word prediction, the largest models in this group should perform well. The paper explicitly tests this hypothesis.

Group 2: Dialogue Fine-Tuned (Dialogue FT). These are models that have been further trained on conversational data. The paper evaluates only BlenderBot (90M, 2.7B, 9.4B), a model specifically fine-tuned for dialogue (Ng et al., 2019). The hypothesis is that conversational fine-tuning — exposure to the structure and pragmatics of dialogue — might induce implicature resolution capabilities. The paper tests whether this is the case.

Group 3: Benchmark-Level Instruction-Tuned (Benchmark IT). These are models fine-tuned on collections of NLP tasks where a single natural language instruction is written for each benchmark (dataset). The instruction is the same for every example in that benchmark. The group includes T0 (3B, 11B; Sanh et al., 2022) and Flan-T5 (780M, 3B, 11B; Chung et al., 2022). The hypothesis is that broad multitask training with instructions might generalize to novel tasks like implicature resolution. If Benchmark IT models perform well, it would suggest that general instruction-following ability transfers to pragmatic understanding.

The paper draws an important distinction between this group and the next (Section 4):

"For Benchmark IT models, annotators write a single instruction for an entire dataset. The models are then fine-tuned on each example from the dataset with the same instruction. We distinguish this from example-level IT; for that type of fine-tuning each example in a dataset gets a new instruction, resulting in a more diverse dataset."

Group 4: Example-Level Instruction-Tuned (Example IT). These are models fine-tuned on tasks where each individual example receives its own natural-language instruction — dramatically increasing the diversity of instructions the model sees. The group includes: OpenAI's text-ada/babbage/curie/davinci-001, text-davinci-002, text-davinci-003, ChatGPT (GPT-3.5-turbo), GPT-4, and Cohere's command models (6B, 52.4B). These are the models behind the "InstructGPT" paradigm (Ouyang et al., 2022), fine-tuned with reinforcement learning from human feedback and/or supervised instruction following.

The causal logic is: if Example IT models significantly outperform all other groups, and if this holds even when controlling for model size, then something about the example-level instruction diversity — rather than scale, dialogue training, or general instruction following — is the key factor in inducing pragmatic understanding. This is precisely the paper's central finding.

Why this categorization is a causal probe, not just taxonomy. The four groups are designed to test specific hypotheses in a stepwise manner:

Base models test whether scale alone suffices.
Dialogue FT tests whether conversational training data suffices.
Benchmark IT tests whether general instruction-following ability suffices.
Example IT tests whether the granularity of instruction diversity (per-example vs. per-benchmark) matters.

The paper's conclusion — that Example IT is the only group that performs substantially above chance — is made more convincing by this structured comparison: each group adds a hypothesized causal factor, and only the last group shows a clear effect. The fact that Cohere-command (52B) significantly outperforms Cohere-base (52B) — "the only difference between those models is instruction-tuning at the example level" (Section 5) — provides a particularly clean within-family comparison that isolates the effect of example-level instruction tuning from architecture, pretraining data, and model scale.

Models behind APIs: the opacity problem. The authors explicitly acknowledge a limitation (Section 5): many of the Example IT models are behind APIs (OpenAI, Cohere), meaning their exact architectures, sizes, and training details are unknown. They argue this is acceptable because:

The group includes ten models across six model classes and two APIs, so the common pattern is unlikely to be an artifact of any single model's implementation details.
The Cohere-base vs. Cohere-command comparison provides a controlled within-family test.
The Example IT models likely vary significantly in other details (architecture, pretraining data), making their shared instruction-tuning approach the most salient commonality.

Nevertheless, the paper cannot definitively prove that example-level instruction-tuning is the causal factor — only that it is the most plausible explanation given the available evidence. This is a limitation the authors flag for future controlled studies.

4. Key Insights and Innovations

Innovation 1: Instruction-Tuning at the Example Level as the Critical Factor for Pragmatic Competence

The paper's most conceptually significant contribution is not the (unsurprising) finding that pragmatic understanding is hard for LLMs, but the identification of a specific, diagnostic divide in training methodology that separates models that can resolve implicatures from those that cannot. Prior to this work, the dominant assumption in the field — reflected in the evaluation practices of BIG-bench, the scaling law literature, and the general enthusiasm around emergent abilities — was that capabilities like pragmatic reasoning would emerge naturally from scale, or at least from broad conversational or instruction-following training. The paper systematically dismantles this assumption through its four-group taxonomy.

What makes this contribution genuinely novel — rather than a restatement of "fine-tuning helps" — is the granularity of the causal probe. The paper distinguishes between two superficially similar training paradigms: benchmark-level instruction-tuning (T0, Flan-T5) and example-level instruction-tuning (InstructGPT variants, Cohere-command). Both involve training on natural language instructions across many tasks. Both improve zero-shot generalization on standard benchmarks. But they produce qualitatively different pragmatic competence: Benchmark IT models perform close to random (T0-11B: 47.0% 5-shot; Flan-T5-11B: 61.7% 5-shot), while Example IT models at comparable scales reach 75-80% (text-davinci-001: 74.5% 5-shot; Cohere-command-52B: 75.4% 5-shot). The 15+ percentage point gap between these groups, despite their shared instruction-following framing, isolates the diversity of instruction phrasing — each example getting its own instruction rather than a single instruction per benchmark — as the operative factor.

This is a diagnostic advance, not a metric gain. The paper is not proposing example-level instruction-tuning as a new method (it already existed in the InstructGPT paradigm). What's new is the recognition that this specific design choice is a causal prerequisite for pragmatic understanding, while scale, dialogue training, and benchmark-level multitask training are not. The within-family comparison provides particularly clean evidence: Cohere-command-52B outperforms Cohere-base-52B by ~15 percentage points, and the only difference between them is example-level instruction-tuning. More strikingly, Cohere-command-52B outperforms base models more than 3× its size (GPT-3-175B: 68.7%; BLOOM-176B: 65.4%) — a reversal of the standard scaling assumption that larger pretrained models are uniformly more capable.

The theoretical implication is significant: pragmatic understanding does not appear to be latent in pretraining representations, waiting to be unlocked by scale. If it were, the 175B-parameter base models — trained on hundreds of billions of tokens of human dialogue and text — would show some nontrivial ability. They don't. Instead, pragmatic competence requires exposure to the specific kind of instruction diversity that example-level fine-tuning provides — thousands of distinct task framings that force the model to attend to subtle contextual cues about what is being asked. This reframes pragmatic understanding from an "emergent ability" to an induced competency that depends on a particular training recipe.

It is worth noting what this innovation is NOT: it is not a proof that example-level instruction-tuning causes pragmatic understanding (the API models' opacity and lack of controlled A/B testing prevent that), and it is not a claim that other training approaches could never work. It is, rather, a strong empirical clue that redirects research attention from scale to training data diversity as the bottleneck for pragmatic competence — a reframing with direct practical implications for how conversational AI systems should be developed.

Innovation 2: Difficulty-Based Implicature Taxonomy as a Diagnostic Lens

The paper's second conceptual contribution is the introduction of a functional taxonomy of implicature types — generalized, particularized, world knowledge, idiom, and rhetorical question — and the demonstration that model performance varies dramatically across these types, revealing not just aggregate accuracy but where and why models fail. This moves beyond the standard practice in NLP evaluation (reporting a single accuracy number) to a capability diagnostic that exposes the structure of model competence.

This is novel because prior work on pragmatic evaluation in LLMs (Jeretic et al., 2020; Parrish et al., 2021; Zheng et al., 2021; the BIG-bench implicature task) reported only aggregate performance. The field knew implicatures were hard, but didn't know which kinds of implicatures were hard, or whether different models struggled with the same things humans struggled with. The taxonomy answers both questions.

The human-model alignment in difficulty patterns is the key empirical finding (Figure 4 and Appendix K.9, Tables 21-34). Humans achieve 92.17% on generalized implicatures (context-free, like "some" implying "not all") but only 83.18% on particularized implicatures (context-heavy, like "I have to work" implying "no"). Models show the same gradient — but amplified. GPT-4 at 30-shot achieves 86.23% on generalized examples but only 71.97% on particularized ones — a ~14 point gap versus the human ~9 point gap. Cohere-command-52B at 30-shot shows a smaller gap (75.00% vs. 71.78%), but both models demonstrate that particularized implicatures are the hard case. This alignment suggests that the difficulty structure is inherent to the task, not an artifact of model architecture — particularized implicatures genuinely require more contextual reasoning.

What makes this a conceptual advance rather than just an error analysis is the implication for evaluation methodology. The paper argues that discarding "ambiguous" examples (as BIG-bench did, removing ~30% of the data) produces a distorted picture of pragmatic competence precisely because it disproportionately removes the particularized examples that constitute the hard case. A model evaluated only on the clean subset would appear competent on generalized implicatures and never be tested on the context-heavy cases that dominate natural conversation. The taxonomy exposes this: generalized examples are easy for both humans and models; particularized examples are hard for both; and an evaluation that filters out ambiguity hides the gap where it matters most.

The taxonomy also reveals limits to the primary finding about example-level instruction-tuning. Figure 4 shows that while GPT-4 substantially outperforms Cohere-command-52B on generalized examples (86.23% vs. 75.00% at 30-shot), their performance on particularized examples is essentially identical (71.97% vs. 71.78%). This means GPT-4's overall advantage over Cohere-command — and its status as the only model approaching human-level performance — is driven primarily by better handling of context-free implicatures, not by superior contextual reasoning. The hard part of the task — resolving novel particularized implicatures — remains equally challenging for the best models, and equally far from human performance. This is a negative result with diagnostic value: it shows that example-level instruction-tuning helps, but the benefit is concentrated on the easier subtypes, and the core challenge of contextual pragmatic inference remains unsolved.

Innovation 3: Prompt-Robust Evaluation with Built-In Sensitivity Measurement

The paper's third innovation is methodological: it establishes a standard for prompt-robust evaluation in the study of LLM capabilities by making prompt sensitivity a measured variable rather than an uncontrolled confound. This is not a new technique per se — reporting mean and standard deviation over multiple prompts has been advocated before (e.g., in the related work the paper cites on prompt sensitivity: Efrat and Levy, 2020; Reynolds and McDonell, 2021b) — but the paper's systematic application of this principle at scale (six templates × up to seven $k$ -shot settings × dozens of models) produces a novel finding about the interaction between prompt sensitivity and model capability.

The key result is that prompt sensitivity is itself a measure of model competence — and it interacts with the training strategy in a revealing way. The standard deviation over prompt templates ranges from 0.3 percentage points for BlenderBot-2.7B (a model that barely exceeds random performance regardless of phrasing) to 7.0 for T0-11B (a Benchmark IT model that is sensitive to how the task is framed) to 1.8 for GPT-4 (an Example IT model whose performance is consistently high across templates). The pattern is not random: better-performing models in the Example IT group show lower prompt sensitivity, while poorly performing models show either trivial sensitivity (because they're near chance regardless) or high sensitivity (because their limited competence is fragile to wording).

This is an important refinement to the "LLMs are sensitive to prompts" narrative. The paper demonstrates that prompt robustness is not a fixed property of models or evaluation design — it is a learned capability that co-varies with competence. Models that genuinely understand the task (as evidenced by high accuracy) also understand it consistently across different phrasings. Models that are merely pattern-matching at the surface level are either consistently bad or inconsistently mediocre. This implies that measuring prompt sensitivity is not just a methodological hygiene practice but a diagnostic signal: a model with high average accuracy but high prompt variance may have learned surface-level heuristics rather than the underlying capability, while a model with high accuracy and low variance is more likely to have acquired the capability robustly.

The paper also demonstrates a practical interaction between few-shot prompting and prompt sensitivity (Appendix K.4, Figure 7). For Cohere-52B and OPT-175B, there is a clear split between structured prompts (dashed lines) and natural prompts (dotted lines) in zero-shot evaluation — the model performs well on one type and poorly on the other. Few-shot prompting largely eliminates this gap, improving performance on the weaker prompt type much more than on the stronger one. This means in-context examples serve a dual role: they improve overall accuracy (the standard finding) AND they reduce prompt sensitivity (a less obvious but equally important benefit). For evaluation methodology, this implies that few-shot results are more robust and interpretable than zero-shot results, especially when comparing models with different pretraining characteristics.

Innovation 4: Chain-of-Thought as a Bridge to Human-Level Pragmatic Reasoning — With Sharp Boundary Conditions

The paper's final conceptual contribution is a precise characterization of when and how chain-of-thought prompting helps pragmatic reasoning, revealing that CoT is not a general-purpose reasoning enhancer but a selective tool that primarily benefits particularized implicatures — the context-heavy subtype that requires explicit reasoning about conversational relevance.

The headline result — GPT-4 with 5-shot CoT reaches 86.5% accuracy, matching human average performance at 86.2% — is striking but not the intellectual contribution. What matters is the decomposition of where the gain comes from. As shown in Appendix K.9, Tables 33-34, GPT-4's CoT improvement from 5-shot to 5-shot CoT is concentrated almost entirely on particularized examples: 71.97% → 81.63% (a ~10 point gain), compared to generalized examples: 86.23% → 88.77% (a ~2 point gain). This is not because generalized examples are near ceiling — GPT-4 is still ~4 points below humans on generalized examples — but because CoT's reasoning structure (explicitly articulating why the response is relevant to the question) is specifically helpful for the context-dependent inference that particularized implicatures require. A generalized implicature like "some" → "not all" is resolved by lexical knowledge; there's not much to reason about. A particularized implicature like "I have to work" → "no" requires connecting the response to the question through world knowledge about work and parties — exactly the kind of multi-step inference that CoT is designed to externalize.

The negative results are equally important: CoT does not help base models (GPT-3-175B drops from 68.7% 5-shot to 60.3% 5-shot CoT; Table 17), and it does not help text-davinci-001 (drops from 74.5% to 67.3%; Table 3). This means CoT is not a "reasoning module" that can be bolted onto any model — it only amplifies pragmatic competence in models that already possess it to some substantial degree. The models that benefit from CoT (text-davinci-002, text-davinci-003, ChatGPT, GPT-4) are all Example IT models that already perform well without CoT. The models that don't benefit (or are harmed) are either base models or an older Example IT model (text-davinci-001). This boundary condition reinforces the paper's central thesis: pragmatic reasoning requires a foundation that is established by example-level instruction-tuning, and CoT can amplify that foundation but cannot create it from nothing.

The innovation here is not the use of CoT per se (which was established by Kojima et al., 2022, and others) but the demonstration that CoT's effectiveness on pragmatic reasoning is subtype-specific and model-specific, revealing a dependency structure: example-level instruction-tuning is necessary for any non-trivial pragmatic competence; CoT can then improve particularized implicature resolution in the models that have that foundation. This has practical implications for building conversational AI: it suggests that simply prompting a pretrained model to "think step by step" about implicatures will not work, and that the development path to pragmatic competence requires both the right training recipe (example-level diversity) and the right inference-time strategy (CoT for context-heavy cases).

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses a dataset of naturally occurring conversational implicatures curated by George and Mamidi (2020), which originally contains 1001 utterance-response-implicature triples extracted from English dialogues. The authors filter to the 718 examples with binary implicatures (implying "yes" or "no"), randomly sample 600 for the test set, and reserve the remaining 118 as a development set for few-shot in-context demonstrations. The test set retains challenging ambiguous examples that the BIG-bench task discarded—examples that humans in the paper's study resolve at 71.9% accuracy, confirming they are harder but not impossible.
Base models. The paper evaluates 48 distinct model configurations across 10 model families, categorized into four groups: Base Models (BERT, RoBERTa, GPT-2, EleutherAI, BLOOM, OPT, Cohere-base, GPT-3; 125M through 176B parameters), Dialogue Fine-Tuned (BlenderBot; 90M through 9.4B), Benchmark-Level Instruction-Tuned (T0, Flan-T5; 780M through 11B), and Example-Level Instruction-Tuned (text-ada/babbage/curie/davinci-001, text-davinci-002, text-davinci-003, ChatGPT, GPT-4, Cohere-command; sizes partially unknown but ranging from small to the largest available). All models are evaluated off-the-shelf with no task-specific training.
Metrics. The primary metric is implicature resolution accuracy—the fraction of the 600 test examples for which the model assigns higher likelihood to the coherent sentence (e.g., "...which means no") than the incoherent counterpart (e.g., "...which means yes"), as described in the technical approach. For GPT-3.5-turbo and GPT-4, where token-level likelihoods are unavailable through the API, greedy decoding accuracy is used instead (the model must generate the correct "yes" or "no" token). Reported accuracies are means over six prompt templates, with standard deviations serving as a measure of prompt sensitivity. Human accuracy (86.2%) is established through 20 annotators (5 per example) and serves as an approximate upper bound.
Baselines. The evaluation does not employ separate "baseline methods" in the traditional sense—instead, the four model categories serve as comparative baselines against each other. The random baseline is 50% (binary choice). Human performance (86.2% average, 92% best) provides the ceiling. Majority voting is not applicable since the evaluation uses a single model judgment per template, not ensemble aggregation.
Generation budget / compute accounting. There is no "generation budget" in the traditional sense because the evaluation does not involve search or iterative refinement. The evaluation protocol uses single-pass likelihood scoring or greedy decoding for each test instance—the model sees each prompt once and produces a single likelihood (or a single completion). The computational cost is therefore proportional to the number of test examples (600) × number of templates (6) × number of few-shot settings (6 values of $k$ : 0, 1, 5, 10, 15, 30) × number of models evaluated. For models evaluated with open-source tools (EleutherAI's evaluation harness, transformers library), compute details (GPU hours, hardware specifications, carbon emissions) are reported in Appendix M. The largest models (BLOOM-176B, OPT-175B) required up to ~33,000 seconds (about 9 hours) on 16 × NVIDIA A100-40GB GPUs for a single 30-shot evaluation run. The paper estimates a maximum total of 966 GPU hours across all experiments, with exact resource counts per model in Appendix M.
Cross-validation / statistical protocol. There is no cross-validation in the traditional machine learning sense because no models are trained. The primary statistical controls are: (1) prompt variance: every evaluation is run across all six prompt templates, with the mean and standard deviation reported to quantify prompt sensitivity; (2) few-shot example variance: for each few-shot evaluation, $k$ examples are randomly sampled from the development set with a fixed seed shared across all models, ensuring fair comparison; the paper verifies in Appendix K.2 that re-running with different random seeds produces minimal variance (standard deviation across seeds for text-davinci-002 at $k=5$ ranges from 0.08 to 1.32 percentage points per template); (3) API stochasticity: for API-based models, the zero-shot evaluation was repeated 10 times to measure API-induced variance, finding standard deviations of 0.12–0.35 for text-davinci-002 (Table 14) and 0.05–0.10 for Cohere-52B (Table 15), confirming that stochasticity is too small to affect conclusions; (4) inter-annotator agreement: Cohen's kappa for human evaluation averages 0.72 across subsets, indicating "substantial" agreement and validating the human baseline quality.

Main Quantitative Results

Zero-Shot and Few-Shot Performance Across Model Categories

The paper's central empirical result, presented in Table 1 and Figure 2, is a sharp stratification of model performance by training category, with Example IT models dramatically outperforming all other groups. Table 1 reports the best-performing model size from each class at $k \in \{0, 1, 5\}$ , with the standard deviation over the six prompt templates:

Example IT models dominate: GPT-4 achieves 81.8% ± 1.8 (0-shot), 82.3% ± 1.4 (1-shot), 82.0% ± 1.7 (5-shot)—within ~4% of human average performance even without chain-of-thought prompting. text-davinci-003 reaches 79.7% ± 0.6 at 5-shot. Cohere-command-52B reaches 75.4% ± 1.8 at 5-shot. These models approach or exceed 80% accuracy in a regime where all other model categories struggle to cross 70%.
Base models perform close to random at all scales: GPT-3-175B—the largest base model evaluated—achieves only 57.7% ± 4.4 (0-shot) and 68.7% ± 1.5 (5-shot). BLOOM-176B reaches 54.2% ± 1.2 (0-shot) and 65.4% ± 3.4 (5-shot). OPT-175B performs similarly (55.3% 0-shot, 63.9% 5-shot, from Table 61). Even with 30 few-shot examples (Tables 27–32), base models plateau well below Example IT models: GPT-3-175B reaches 68.3% at 30-shot, and OPT-175B reaches 62.4%—still 7–13 points below Example IT models at 5-shot. The gap between the best base model and human performance is at least 17.5 percentage points at all values of $k$ .
Dialogue fine-tuning provides no benefit: BlenderBot-2.7B achieves 53.4% ± 0.3 across all few-shot settings—essentially unaffected by in-context examples and only marginally above chance. BlenderBot-9.4B performs similarly poorly across all evaluations (Tables 70–72). This is particularly informative because it demonstrates that exposure to conversational data during training does not, by itself, induce implicature resolution capabilities.
Benchmark-level instruction-tuning also fails: T0-11B performance actually decreases with few-shot examples, from 55.6% ± 7.0 at 0-shot to 47.0% ± 0.2 at 5-shot (Table 74). Flan-T5-11B reaches only 61.7% ± 4.8 at 5-shot (Table 77). These models were specifically designed for zero-shot generalization, yet they perform at levels comparable to base models of similar scale on this task—confirming that broad multitask instruction-following does not transfer to pragmatic understanding.

Figure 2 visualizes the performance as a function of $k$ , revealing several additional patterns:

Base models benefit from few-shot examples but plateaus early, with most gains occurring between $k=0$ and $k=5$ , after which performance flattens (e.g., GPT-3-175B: 57.2% → 68.7% → 67.3% → 68.3% → 68.3% as $k$ increases from 0 to 30, from Tables 69, 27–32).
Benchmark IT models show a striking degradation with few-shot examples, visible in Figure 2 as a downward trend. This is consistent with these models being explicitly trained for zero-shot generalization—adding in-context examples may interfere with their instruction-following behavior.
Example IT models maintain a large gap over all other groups at every value of $k$ , and this gap does not narrow with additional in-context examples. The bars in Figure 2 for Example IT hover around 75–80% while all other groups remain below 70%, with Base models at ~60% and Benchmark IT below 60%.

A critical nuance, established through the random-label experiment in Appendix K.6 (Table 16), is that the few-shot examples do not appear to teach pragmatic reasoning in context for Example IT models. When the in-context labels are randomized (swapping "yes" for "no" or vice versa), text-davinci-001's 5-shot performance changes from 74.5% ± 1.0 to 73.4% ± 1.2, and Cohere-command-52B changes from 75.4% ± 1.8 to 73.5% ± 2.7—both within confidence intervals of the ground-truth label condition. For base models, random labels do cause degradation: GPT-3-175B drops from 68.7% ± 1.5 to 64.7% ± 1.9, and Cohere-52B drops from 65.1% ± 2.9 to 60.5% ± 1.9. This confirms that Example IT models primarily use few-shot examples as a format primer (establishing the yes/no output structure) rather than learning pragmatic inference strategies from the demonstrations.

Scaling Analysis: Does Bigger Mean Better at Pragmatics?

Figure 3 and Table 2 present scaling results—accuracy as a function of model size (non-embedding parameters) for model families where sizes are known. The key findings are:

For zero-shot evaluation (Figure 3, left panel): No model family shows a significant positive correlation between size and accuracy among base models. The lines are essentially flat, with large models performing no better than small ones. GPT-3's scaling from 350M to 175B shows minimal improvement in 0-shot (51.5% → 57.2% across sizes; Table 66, 67, 68, 69), and the 0-shot scaling curves for OPT, BLOOM, and Cohere-base are similarly flat or even negative. This is a crucial negative result: the largest base models—trained on hundreds of billions of tokens and demonstrating emergent abilities on many benchmarks—do not show emergent pragmatic understanding. Scale alone does not work.

For 5-shot evaluation (Figure 3, right panel): A different picture emerges. Two model families show positive correlations with size:

Cohere-command (Example IT): The steepest scaling curve, increasing from ~66% at 6B parameters to ~75% at 52B parameters (Table 78, 79). The authors calculate the slope as 0.15% per billion parameters, and extrapolate that Cohere-command would reach human-level performance (~86%) at approximately 125B parameters—roughly 2.4× larger than the current largest Cohere model.
GPT-3 (Base): A shallower but still positive slope of 0.04% per billion parameters from GPT-3-6.7B to GPT-3-175B (Table 68, 69). Extrapolating linearly, GPT-3 would need approximately 642B parameters to reach human-level performance—more than 5× the size of the largest Cohere model needed.
Other base model families (OPT, BLOOM, EleutherAI) show flat or inconsistent scaling patterns, with some larger models performing worse than smaller ones (e.g., OPT-175B at 63.9% 5-shot is worse than OPT-13B at 67.4%, from Tables 58, 61). The paper does not explain these non-monotonicities, but they underscore that pragmatic competence is not a reliable function of pretraining scale.

For OpenAI's text-<engine>-001 series (Table 2): Although exact parameter counts are unknown, the model ordering by size (Ada < Babbage < Curie < Davinci) is known. The 5-shot accuracy progresses monotonically: 57.6% ± 2.8 → 66.1% ± 0.3 → 71.3% ± 1.3 → 74.5% ± 1.0. This is the cleanest scaling signal in the paper and further supports the finding that Example IT models benefit from scale in a way that base models largely do not.

The paper's central claim from this analysis is that Example IT models not only achieve higher absolute performance at any given scale, but also exhibit steeper performance gains with increasing model size, suggesting that the combination of example-level instruction diversity and larger model capacity is particularly synergistic for pragmatic understanding.

Prompt Sensitivity and Interactions with Few-Shot Prompting

The paper measures prompt sensitivity via the standard deviation over the six prompt templates for each evaluation. The reported standard deviations in Table 1 and Figure 2 reveal a systematic pattern related to model capability:

The best-performing models show the lowest prompt sensitivity: GPT-4's standard deviation ranges from 1.36 to 2.44 across few-shot settings (Table 87); text-davinci-002 ranges from 0.57 to 2.79 (Table 84). These models are not only accurate but robust—their performance is consistent regardless of whether the prompt is structured or natural, uses "Esther and Juan" or "Bob and Alice," or asks about "implicature" versus "meaning" versus "intent."
Models with near-chance performance also show low prompt sensitivity: BlenderBot-2.7B has a standard deviation of 0.1–0.3 across all $k$ (Table 71), and BlenderBot-9.4B ranges from 0.4 to 1.8 (Table 72). These models are consistently bad, suggesting that prompt variation cannot rescue fundamentally absent capabilities.
Models with intermediate performance show the highest prompt sensitivity: T0-11B has a standard deviation of 7.0 at 0-shot (Table 74), and OPT-13B has 5.5 at 0-shot (Table 58). These models have some latent capability but fail to express it consistently—their performance is fragile to prompt wording in a way that both trivial models (which have no capability to express) and genuinely competent models (which have robust capability) are not.

Appendix K.4 (Figure 7) provides a more detailed analysis of how in-context examples interact with prompt type. For Cohere-52B (base) and OPT-175B, there is a clear split in zero-shot performance between the three structured prompt templates (dashed lines, templates 1, 3, 4) and the three natural templates (dotted lines, templates 2, 5, 6). Cohere-52B achieves ~55% on structured prompts but ~62% on natural prompts at 0-shot; OPT-175B shows the reverse pattern. Few-shot prompting largely eliminates this gap: by $k=30$ , the structured and natural curves have converged for both models, primarily because performance improves on the weaker prompt type rather than degrading on the stronger one. This means few-shot prompting not only improves aggregate accuracy but also reduces the model's sensitivity to how the task is framed—an important practical finding for reliable deployment.

Chain-of-Thought Prompting: Selective Gains for the Best Models

Table 3 reports the results of 5-shot chain-of-thought (CoT) prompting for the Example IT models, using manually written 5-shot CoT demonstrations (one of six templates is shown in Table 8, with the remaining five provided in supplementary materials). The results reveal a bifurcation:

Models that benefit from CoT:

text-davinci-003: 79.7% ± 0.6 (5-shot) → 83.6% ± 0.6 (5-shot CoT), a gain of +4.0%
ChatGPT: 73.9% ± 6.3 → 77.2% ± 1.0, a gain of +3.3%
GPT-4: 82.0% ± 1.7 → 86.5% ± 1.0, a gain of +4.5%
text-davinci-002: 79.6% ± 2.0 → 80.1% ± 0.8, a modest gain of +0.5%

Models that do not benefit:

text-davinci-001: 74.5% ± 1.0 → 67.3% ± 2.6, a decrease of -7.2%
Cohere-command-52B: 75.4% ± 1.8 → 75.3% ± 0.5, essentially unchanged (-0.1%)

GPT-4's 86.5% ± 1.0 with CoT is the headline result: it effectively matches average human performance (86.2%) and is approximately 3.3 percentage points below the best human annotator (89.8%). The paper is careful not to claim superhuman performance—the difference between GPT-4 + CoT (86.5%) and average human (86.2%) is within the standard deviation (±2.3 for humans, ±1.0 for GPT-4) and should be interpreted as parity, not superiority.

The fact that text-davinci-001 degrades with CoT while text-davinci-002 and 003 improve is revealing. text-davinci-001 is the earliest InstructGPT model; text-davinci-002 and 003 incorporate additional training improvements (the exact details are proprietary, but the pattern suggests that later models have better reasoning capabilities that CoT can amplify). Similarly, Cohere-command-52B's flat CoT performance suggests that reasoning-amplification through chain-of-thought is not a universal property of instruction-tuned models but depends on the quality of the underlying reasoning capabilities.

The paper also tested CoT on base models (Appendix K.7, Table 17). The results are uniformly negative: GPT-3-175B drops from 68.7% to 60.3% (-8.4%) with CoT, GPT-3-6.7B drops from 62.4% to 61.0%, and Cohere-6B collapses to 29.2% (due to inability to adhere to the required output format of "Answer: yes/no"). This firmly establishes that CoT only helps models that already have substantial pragmatic competence—it amplifies existing capability but cannot create it, consistent with the broader CoT literature that finds CoT helpful primarily for models above certain scale/capability thresholds.

Difficulty-Based Analysis: Where Models Succeed and Fail

The paper manually labels 217 of the 600 test examples according to a taxonomy of implicature types (Tables 4 and 20): generalized (47 examples), particularized (94), world knowledge (23), idiom (42), rhetorical question (11), and other (383). Figure 4 and the detailed per-type results in Appendix K.9 (Tables 21–34) reveal that performance varies dramatically by implicature subtype, and that the pattern of difficulty aligns between humans and models:

Generalized implicatures (context-free): These are the easiest type for both humans and models. Humans achieve 92.17% (Table 22). GPT-4 achieves 86.23% at 30-shot (Table 32), rising to 88.77% with 5-shot CoT (Table 34)—close to human performance. Cohere-command-52B achieves 75.00% at 30-shot. The gap between humans and the best model on generalized examples is ~3–4 percentage points.

Particularized implicatures (context-heavy): These are substantially harder for everyone. Humans achieve 83.18% (Table 22). GPT-4 achieves only 71.97% at 30-shot (Table 32)—a gap of ~11 points from humans and ~14 points below GPT-4's own generalized performance. GPT-4 with CoT improves substantially to 81.63% (Table 34), closing most of the gap but still ~1.5 points below humans. Cohere-command-52B achieves 71.78% at 30-shot—essentially identical to GPT-4's non-CoT performance, meaning GPT-4's overall advantage is driven entirely by generalized examples. On the core challenge of resolving context-dependent implicatures, the two best models are indistinguishable.

World knowledge and idiom examples: These show similar patterns, with models performing well but below human levels. GPT-4 reaches 95.65% on world knowledge (30-shot) and 86.74% on idioms, compared to human performance of 93.04% and 92.73% respectively. The gap on idioms is larger (~~6 points) than on world knowledge (~~+2 points over humans, likely reflecting annotation noise since GPT-4 exceeds the human baseline).

Rhetorical questions: These are the smallest category (11 examples) and show high variance. GPT-4 achieves 90.91% at 30-shot, slightly above the human baseline of 92.73%.

The key insight from Figure 4 is not just the aggregate accuracy gap between models, but the shape of the difficulty gradient. The bottom row of Figure 4 compares Cohere-command-52B and GPT-4 side-by-side on generalized vs. particularized examples across values of $k$ . Cohere-command achieves similar performance on both types (slightly better on generalized), while GPT-4 shows a large and persistent gap—consistently much better on generalized than particularized. This means GPT-4's edge over Cohere-command is not a general pragmatic superiority but a specific strength at context-free implicatures. On the context-heavy particularized examples that constitute the core challenge of conversational pragmatics, both models perform equivalently and both remain well below human performance.

This analysis also validates the paper's decision not to discard "ambiguous" examples (contra BIG-bench). The particularized examples that constitute the hard case for both humans and models would be disproportionately removed by a filtering approach that eliminates ambiguity. Evaluating only on the clean subset would overestimate model competence by hiding the cases where pragmatic reasoning is genuinely difficult—and where the gap with human performance is largest.

Prompt Template Analysis: Structured vs. Natural Formats

The paper's six prompt templates are categorized into three "structured" (templates 1, 3, 4) and three "natural" (templates 2, 5, 6). While aggregate results average across all six, the per-template breakdowns in Appendix K.10 (Tables 35–87) reveal systematic format effects that interact with model type:

Base models show a consistent preference for natural templates. GPT-3-175B at 0-shot achieves 61.3% on structured templates vs. 53.2% on natural (Table 69), a reverse pattern at 5-shot (68.6% structured vs. 68.8% natural). OPT-175B achieves 58.4% structured vs. 52.3% natural at 0-shot (Table 61). The direction and magnitude vary across models and $k$ values, but base models rarely perform identically across the two format types.

Example IT models are less format-sensitive. text-davinci-002 at 0-shot achieves 72.7% structured vs. 68.5% natural (Table 84). GPT-4 at 0-shot: 82.8% vs. 80.7% (Table 87). The gap between structured and natural is smaller for Example IT models than for base models, consistent with the finding that better-performing models are more prompt-robust.

Template 4 ("What is the intent...") is consistently the most difficult structured template, often producing the lowest accuracy among the three structured variants. This suggests that framing the task as identifying "intent" rather than "implicature" or "meaning" shifts the model's behavior in a way that reduces accuracy, possibly because "intent" connotes a different pragmatic framing that interferes with the binary yes/no judgment.

For completion-based evaluation (GPT-3.5-turbo and GPT-4), the completion templates in Table 6 (which include explicit instructions to "Only output 'yes' or 'no'") produce results that are consistent with the likelihood-ranking approach for other models in the Example IT group, validating that the two evaluation modes are broadly comparable despite the methodological difference.

Ablation Studies and Robustness Checks

Contrastive task framing (Appendix K.1): The paper tests whether reframing the task as a contrastive choice ("Which of the following sentences is coherent: A or B?") rather than a likelihood ranking would improve performance, hypothesizing that presenting both options simultaneously might help models distinguish the pragmatically coherent alternative. For text-davinci-002, the contrastive task performs substantially worse: 59.3% ± 1.3 at 0-shot (vs. 71.3% ± 1.75 for the ranking approach), 63.2% at 1-shot, and 64.0% at 5-shot (Table 11). The degradation is robust across different multiple-choice answer formats (A/B labeling, one/two labeling, full-text labeling). This negative result suggests that the ranking approach—where the model evaluates each version independently and the higher-likelihood one is selected—is better suited to probing pragmatic preferences than a direct comparison framing, possibly because the contrastive format introduces additional complexity (the model must parse the multiple-choice structure) that interferes with the core pragmatic judgment.

Alternative zero-shot instruction prompts (Appendix K.3): Building on the narrative that LLMs can be "prompted to simulate almost anything if you set them up correctly" (Kojima et al., 2022; "let's think step by step"), the paper tests three additional prompt templates adapted from Glaese et al. (2022) for a dialogue agent (Table 7). These templates provide extensive preamble explaining what implicatures are and what the assistant is expected to do. For GPT-3-175B, text-davinci-001, and text-davinci-002, these elaborate instructions do not improve performance over the standard templates (Table 13): GPT-3-175B achieves 59.2% ± 4.5, text-davinci-001 achieves 66.1% ± 3.2, and text-davinci-002 achieves 67.7% ± 9.6—all comparable to or below their best standard-template performance. The highest individual template reaches 74.5% for text-davinci-002, matching but not exceeding the standard template performance. This robustness check confirms that the baseline performance reported in the main experiments is not an artifact of using suboptimal prompts—adding more elaborate instructions does not unlock additional pragmatic competence.

Random-label few-shot experiment (Appendix K.6): To test whether few-shot examples teach pragmatic reasoning or merely prime the output format, the paper evaluates 1-shot and 5-shot performance where the in-context examples have randomized labels (the "yes"/"no" implicature is flipped). As discussed above, Example IT models show no significant degradation (text-davinci-001: 74.5% → 73.4%; Cohere-command-52B: 75.4% → 73.5%), while base models do degrade (GPT-3-175B: 68.7% → 64.7% at 5-shot; Cohere-52B: 65.1% → 60.5%). This confirms that for Example IT models, the few-shot examples serve primarily as a format primer establishing the yes/no output structure, whereas base models use the content of the demonstrations more substantively—and thus are affected when that content is misleading.

Spurious correlation test (Appendix K.8): To check whether the benchmark contains exploitable surface-level cues (e.g., lexical patterns in the utterance or response that correlate with the label independent of pragmatic reasoning), the paper runs GPT-3.5-turbo and GPT-4 with only the utterance (no response) and only the response (no utterance) as input (Tables 18–19). With utterance-only input, GPT-4 achieves 48.9% ± 10.5 at 0-shot and 53.7% ± 0.5 at 5-shot—essentially random, confirming that the utterance alone does not leak the label. With response-only input, GPT-4 achieves 62.6% ± 1.7 at 0-shot and 65.5% ± 1.1 at 5-shot—slightly above chance, suggesting that some responses contain cues that weakly predict the label (e.g., idiomatic responses like "do fish swim?" → yes; negative-polarity responses like "I have to work" → no). However, this is well below the full-context performance (~82%), confirming that the primary signal driving model accuracy is the interaction between utterance and response—the pragmatic inference itself—not isolated lexical cues.

Chain-of-thought on base models (Appendix K.7): The paper tests 5-shot CoT on GPT-3 and Cohere-base models (Table 17) and finds that CoT consistently decreases or does not improve performance. GPT-3-175B drops from 68.7% (5-shot standard) to 60.3% (5-shot CoT), a loss of 8.4 percentage points. GPT-3-6.7B drops from 62.4% to 61.0%. GPT-3-350M and GPT-3-1.3B show small losses. Cohere-6B collapses to 29.2% because it fails to adhere to the required CoT output format (outputting "Answer: yes/no" after reasoning). This negative result is important because it establishes a boundary condition: CoT is not a general-purpose reasoning amplifier but only works for models that already possess a sufficient foundation of the target capability.

Variance over prompt ordering (Appendix K.2): For text-davinci-002 at 5-shot, the paper re-runs the evaluation with three different random seeds (different orderings of the same set of 5 few-shot examples) and finds the standard deviation across seeds ranges from 0.08 (template 6) to 1.32 (template 5) percentage points, with a mean standard deviation of approximately 0.7 (Table 12). The overall mean accuracy across the three seeds is 79.56%, 78.64%, and 78.94%—within ~1 percentage point. This confirms that the random ordering of in-context examples is not a major source of variance and does not affect the paper's conclusions.

API stochasticity (Appendix K.5): For text-davinci-002 and Cohere-52B, the zero-shot evaluation is repeated 10 times to measure variance due to non-deterministic API behavior (Tables 14–15). text-davinci-002's per-template standard deviation across API runs ranges from 0.12 (template 1) to 0.35 (template 2), with a mean accuracy range across runs of 70.25% to 70.58%. Cohere-52B shows even lower variance, with per-template standard deviations from 0.05 to 0.10. This confirms that API stochasticity is negligible relative to the effect sizes being studied (10–20 percentage point differences between model categories) and does not threaten reproducibility of the conclusions, though it does mean exact numerical results may not be perfectly replicable if the underlying API models change.

Human evaluation quality (Appendix H): The paper's human evaluation includes explicit quality controls: attention checks at three random positions in the form (questions that obviously map to "yes"), with annotators who fail at least two checks being excluded (one annotator was excluded and their near-random data discarded). Inter-annotator agreement is measured by Cohen's kappa, averaging 0.72 across the four annotation subsets (Table 10), which falls in the "substantial" range (0.61–0.80) according to standard interpretation. The gap between average (86.2%) and best (89.8%) human performance is 3.6 percentage points, substantially smaller than the 18-point gap in the BIG-bench implicature task, indicating higher annotation quality. The paper also analyzes cases where most humans disagree with the ground truth (Appendix H), finding that of the 30 cases where only 0–1 annotators agree with the label, 18 appear to be annotation errors in the original dataset and 12 admit multiple reasonable interpretations.

Critical Assessment

Claim 1: "Instruction-tuning at the example level is important for pragmatic understanding"

This is the paper's central claim, and the experiments provide strong correlational evidence but stop short of causal proof. The difficulty is the opacity of the Example IT models: all of them are behind APIs (OpenAI, Cohere), meaning their exact training procedures, data mixtures, and architectures are unknown. The paper acknowledges this directly (Section 5):

"Due to the secrecy surrounding the exact implementation of these models we cannot be certain [that example-level IT is the main driver], but we can be relatively confident."

What supports the claim:

The performance gap between Example IT and all other groups is large, robust across prompt templates and $k$ values, and consistent across multiple model families within the Example IT group (OpenAI and Cohere). Ten models across two different API providers all show the same pattern, making it unlikely to be an artifact of a specific implementation.
The within-family comparison—Cohere-command-52B vs. Cohere-base-52B, where the paper states "the only difference between those models is instruction-tuning at the example level"—is the cleanest evidence. The ~15 percentage point gap on 5-shot evaluation cannot be explained by scale or pretraining, since those are held constant.
Benchmark IT models (T0, Flan-T5) share the instruction-tuning paradigm but differ specifically in the granularity of instructions (per-benchmark vs. per-example). Their poor performance suggests that instruction-tuning per se is not sufficient—the diversity of example-level instructions appears to be the operative factor.

What weakens the claim:

The Example IT models differ from base models in multiple correlated ways beyond instruction granularity: they are trained with RLHF (which introduces a reward signal for helpful/harmless outputs), they may use different data mixtures, and they are optimized for dialogue and instruction-following in ways that are not fully documented. The paper cannot isolate instruction granularity from these other factors.
The Cohere-command vs. Cohere-base comparison, while clean within the Cohere family, is a single data point. The paper does not have (and cannot obtain) the same within-family comparison for GPT-3 vs. InstructGPT with only the instruction granularity varied and all other factors held constant.
All Example IT models are from commercial providers who may have undocumented advantages (better pretraining data, longer training, proprietary techniques). The paper cannot rule out that these advantages, rather than example-level instruction tuning specifically, account for the performance difference.

The appropriate characterization is: the evidence is strongly suggestive but correlational. The paper has identified a reliable empirical signature—models with example-level instruction-tuning resolve implicatures much better than models without it—and has ruled out several alternative explanations (scale, conversational training, benchmark-level multitask training). But the specific causal role of instruction granularity, as opposed to other correlated properties of the Example IT training recipe, remains unproven. A controlled experiment that takes a single base model and applies benchmark-level vs. example-level instruction-tuning with all other factors held constant would be needed to establish causality definitively, and the paper explicitly calls for this as future work (Section 5): "Investigating the exact effect of this type of instruction-tuning on pragmatic understanding in a controlled setting is an interesting future work direction."

Claim 2: "Pragmatic understanding has not yet arisen from large-scale pre-training on its own"

This claim holds more directly. The evidence is extensive and consistent: across 24 base model configurations spanning 110M to 176B parameters, 7 model families, and multiple pretraining paradigms (BERT, GPT, BLOOM, OPT, Cohere-base, GPT-3), zero-shot accuracy rarely exceeds 61% (Table 1, Figure 3 left), and the best 5-shot performance (GPT-3-175B at 68.7%) still leaves a 17.5-point gap to human average. The scaling analysis in Figure 3 (left) shows no positive correlation between size and zero-shot accuracy among base models—the largest models (BLOOM-176B, OPT-175B, GPT-3-175B) perform no better than models two orders of magnitude smaller.

Caveats:

The paper evaluates only a single task (binary implicature resolution) on a single dataset. It is possible that base models at 175B+ scale have emergent pragmatic capabilities on other types of implicatures or other pragmatic phenomena that this evaluation doesn't capture. The paper does not claim that pretraining produces no pragmatic understanding, only that it does not produce the specific capacity to resolve conversational implicatures at a level approaching human performance.
The evaluation is English-only. Cross-linguistic generalization is untested.
The paper does not evaluate models beyond 176B parameters (PaLM, Chinchilla, Gopher), which might show different scaling behavior. GPT-4 is evaluated but only as an Example IT model, not in its base form (which is not publicly available).

Despite these caveats, the evidence for the claim is strong because of the breadth of model families tested and the consistency of the null result. If pragmatic understanding were an emergent property of scale in the 10B–200B range, the evaluation would likely have detected it in at least some model families. The paper's conclusion that "scaling analysis shows that it might for much larger scale" (Section 1) is appropriately hedged—it notes that the shallow positive slope for GPT-3 at 5-shot (Figure 3 right) leaves open the possibility that base models at 500B+ parameters might perform better, but there is no evidence of this in the evaluated range.

Claim 3: "GPT-4 reaches average human-level performance with chain-of-thought prompting"

The evidence for this claim is specific and well-supported by Table 3: GPT-4 achieves 86.5% ± 1.0 with 5-shot CoT, compared to human average of 86.2% ± 2.3. However, the claim requires careful qualification:

"Average human-level" is not "human-level." The best human annotator achieves 89.8%—3.3 percentage points above GPT-4. The paper does not claim that GPT-4 surpasses or even matches the best human performance. The appropriate interpretation is that GPT-4 + CoT falls within the range of typical human performance on this specific task, not that it has achieved human-equivalent pragmatic understanding.
The performance is subtype-dependent. As analyzed in the difficulty-based results, GPT-4's human-level aggregate performance masks a large gap on particularized implicatures (71.97% without CoT, 81.63% with CoT vs. human 83.18%). GPT-4 matches or exceeds humans on generalized, world knowledge, idiom, and rhetorical question types, but remains below human performance on the specific subtype (particularized) that most closely captures the core challenge of contextual pragmatic inference. This means the headline number overstates GPT-4's competence on the hardest cases.
The task is narrow. Binary implicature resolution on a curated dataset of 600 conversational examples is a far cry from the full range of pragmatic phenomena in natural dialogue. GPT-4's success on this task does not guarantee competence at other types of implicatures (scalar, conventional, multi-proposition), pragmatic presuppositions, indirect speech acts, or the real-time interactive pragmatic reasoning required in conversation.
GPT-4's base capabilities are unknown. The paper evaluates GPT-4 only in its instruction-tuned form, without access to its base model performance. It is unclear how much of GPT-4's pragmatic competence comes from example-level instruction-tuning (the paper's thesis) versus from its massive scale and pretraining. This is not a flaw in the paper's evaluation—access to base GPT-4 is simply unavailable—but it means the claim about GPT-4's performance cannot be cleanly attributed to the paper's central causal factor.

These qualifications do not invalidate the claim, but they circumscribe it. GPT-4 with CoT achieves an impressive result on a challenging pragmatic task, but the gap between this result and full human-like pragmatic understanding—especially for context-dependent particularized implicatures—remains substantial.

Strengths of the Experimental Design

Comprehensive model coverage. The evaluation spans 48 model configurations across 10 families, covering five orders of magnitude in parameter count (110M to 176B+) and four distinct training paradigms. This is substantially more comprehensive than any prior pragmatic evaluation of LLMs.
Rigorous prompt control. Six prompt templates × up to seven few-shot settings × randomized example selection × fixed seeds across models. The paper goes well beyond the standard single-prompt evaluation to establish that results are robust.
High-quality human baseline. The paper conducts its own human evaluation with explicit quality controls (attention checks, inter-annotator agreement measurement, proper compensation) rather than relying on existing annotations or Mechanical Turk workers. The Cohen's kappa of 0.72 and the small gap between average and best human performance indicate reliable annotation quality.
Multiple robustness checks. The paper tests for prompt ordering effects, API stochasticity, spurious correlations, and random-label robustness—anticipating and addressing potential confounds.
Difficulty-based decomposition. The taxonomy of implicature types and the subtype-level analysis in Figure 4 moves beyond aggregate accuracy to reveal where models actually succeed and fail, providing actionable diagnostic information.

Weaknesses and Missing Experiments

No controlled causal experiment. The paper's central causal claim—that example-level instruction granularity is the key factor—is supported by correlation rather than controlled manipulation. The paper acknowledges this and calls for future work, but the absence of a controlled A/B test (same base model, same data, same training budget, varying only instruction granularity) means the causal role of example-level instructions remains inferred rather than demonstrated.
Single dataset, single language, single pragmatic phenomenon. All results are on one dataset of binary conversational implicatures in English. The paper does not evaluate other pragmatic phenomena (scalar implicatures, presuppositions, indirect speech acts, Gricean maxim violations), other languages, or other types of conversational inference. This limits the generalizability of the finding that example-level IT induces "pragmatic understanding"—it may induce only the specific type tested.
The GPT-4 completion-based evaluation is not fully comparable to the likelihood-based evaluation. For GPT-3.5-turbo and GPT-4, the evaluation uses greedy decoding with explicit instructions to output "yes" or "no" (Table 6), while all other models use likelihood ranking. The paper argues these are comparable because "the choice of 'no' versus 'yes' is only determined by the resolution of the implicature," but the two evaluation modes place different demands on the model (generation vs. comprehension) and may have different difficulty profiles. The paper does not validate that likelihood-based and completion-based evaluations produce equivalent results for any model where both are accessible.
No few-shot CoT for non-Example-IT models except GPT-3 and Cohere-base. The paper tests CoT on base models (Appendix K.7) and finds negative results, but does not test CoT on Benchmark IT or Dialogue FT models. Given the paper's thesis that example-level IT is critical, testing whether CoT could partially rescue Benchmark IT models would have been informative—if CoT helps Flan-T5 but not base models, it would suggest that some reasoning capability is latent in Benchmark IT models that CoT can surface. This experiment is absent.
Difficulty estimation is post-hoc and manual. The taxonomy of implicature types is manually labeled on only 217 of the 600 test examples, leaving 383 examples in the "other" category. This means the subtype analysis in Figure 4 and Appendix K.9 covers only ~36% of the test set. The paper does not report whether the subtype-level patterns hold in the "other" category or whether the "other" examples are systematically different.
No test for whether in-context learning transfers to new implicature types. The few-shot experiments draw demonstrations from a development set that may contain the same subtypes as the test set. The paper does not test whether performance on, say, particularized examples improves when only generalized examples are used as demonstrations (or vice versa), which would test whether the model is learning general pragmatic strategies or just matching surface patterns. This is related to the random-label experiment but distinct—the random-label experiment tests whether content matters, while this would test whether type-specific content matters.
Single human prompt template. The human evaluation uses only Template 2 (the natural "Esther and Juan" format), while models are evaluated across all six templates. The paper argues this is acceptable because humans are less prompt-sensitive, but this assumption is untested. If human performance varies with prompt wording (unlikely but possible), the human baseline would not be directly comparable to the model performance averaged across templates.

Despite these limitations, the experimental design is appropriate for the paper's goals. This is an evaluation study, not a causal intervention study, and the paper's claims are appropriately hedged. The breadth and rigor of the evaluation substantially advance over prior work (BIG-bench, Zheng et al., 2021) and establish a reliable empirical picture of where current LLMs stand on pragmatic understanding. The main gap between what was demonstrated and what was claimed is the causal attribution of the Example IT advantage to instruction granularity specifically—a gap the paper itself acknowledges and flags for future work.

6. Limitations and Trade-offs

The Difficulty Estimation Engine Is Not Evaluated

A practitioner reading this paper might reasonably ask: "How do I know which fine-tuning recipe to use before I've already trained the model?" The paper provides no mechanism for predicting whether a specific training procedure will induce pragmatic competence—it only provides a post-hoc evaluation method. This is not a flaw in the evaluation methodology itself (which is rigorous), but it means the paper's central insight is diagnostic rather than prescriptive. You can use the evaluation protocol to measure whether a model you already have resolves implicatures, but the paper does not tell you how to design a training pipeline that will produce this capability, beyond pointing to the Example IT category and hypothesizing that instruction granularity is the operative factor.

The consequence is that a team building a conversational agent from scratch cannot follow a recipe from this paper. They can evaluate their model, and if it fails, they know that pretraining scale, dialogue fine-tuning, and benchmark-level instruction-tuning are unlikely to help—but the paper's evidence that example-level instruction-tuning will help is correlational, not causal. The Cohere-command vs. Cohere-base comparison (Section 5) is the strongest evidence, but it is a single within-family data point from a proprietary training pipeline. Replicating this result in a new model family with a new training setup is not guaranteed.

The paper is transparent about this (Section 5):

"Investigating the exact effect of this type of instruction-tuning on pragmatic understanding in a controlled setting is an interesting future work direction (e.g. by isolating the effect of data diversity from instructions)."

But this future work is exactly what would be needed to turn the diagnostic into a prescription. The paper's claims would be substantially stronger if they had taken a single base model (e.g., GPT-3-6.7B, which is publicly available through the API) and fine-tuned it with benchmark-level vs. example-level instructions on the same underlying tasks, then evaluated both on the implicature task. This controlled experiment—cited as future work—is the missing piece that prevents the paper from moving beyond correlation to causation.

The Proportional Reasoning Gap: Models Do Well Where Humans Do Well, and Fail Where Humans Fail

The paper's headline finding—that GPT-4 reaches 86.5% with CoT, matching average human performance—masks a more unsettling pattern: the difficulty profile of the task is conserved between humans and models, and the hardest subtype (particularized implicatures) remains substantially unsolved. GPT-4 without CoT achieves 71.97% on particularized examples vs. 86.23% on generalized examples at 30-shot (Tables 32, 34)—a 14.3 percentage point gap. Humans show a smaller but still significant gap: 83.18% vs. 92.17%, a difference of 9.0 percentage points (Tables 22). With CoT, GPT-4 improves to 81.63% on particularized examples, narrowing the gap with humans but still 1.55 points below the human average.

This matters because particularized implicatures are the representative case for conversational pragmatics. A generalized implicature like "some" → "not all" is a lexical pattern that could be learned from distributional statistics without genuine pragmatic reasoning. A particularized implicature like "I have to work" → "no" requires integrating the response with the question through world knowledge—exactly the kind of inference that distinguishes communicating from merely producing text. If the best model matches humans on generalized examples but lags on particularized ones, the gap on the task that actually matters remains open.

The paper acknowledges this indirectly through the subtype analysis (Section 4, Insight 5; Figure 4), but the framing of "GPT-4 reaches human-level performance" (the paper's Insight 4) overstates the case. A more accurate framing would be: GPT-4 reaches human-level performance on the easy half of the task, and remains below human performance on the hard half. For a practitioner deciding whether to deploy GPT-4 as a conversational agent, this distinction matters: the model will perform well on routine implicatures but may still fail on the context-dependent inferences that characterize natural conversation.

The mitigation status is partial. Chain-of-thought prompting substantially closes the gap on particularized examples (from 71.97% to 81.63% for GPT-4), but does not eliminate it. The paper does not identify any technique that brings particularized performance to human parity, suggesting this is a capability boundary for current models rather than a prompting failure.

The Single-Dataset, Single-Phenomenon Evaluation Limits Generality

All results in the paper are drawn from one dataset (George and Mamidi, 2020) of one specific pragmatic phenomenon (binary conversational implicatures) in one language (English). This is explicitly acknowledged:

"the likelihood ranking approach we take limits our study to implicatures with clear alternative. However, implicatures in natural language can entail more complex propositions." (Section 5)

The consequence is that the paper's conclusions about "pragmatic understanding" are extrapolated from a narrow operationalization. Conversational pragmatics encompasses scalar implicatures, conventional implicatures, presuppositions, indirect speech acts, politeness strategies, Gricean maxim reasoning, and many other phenomena. A model that excels at binary implicature resolution may or may not excel at these other pragmatic capabilities. The paper cannot distinguish between "example-level IT induces general pragmatic competence" and "example-level IT induces a specific skill at yes/no implicature resolution in English dialogue"—a critical distinction for any practitioner hoping these models will handle the full range of pragmatic phenomena in conversation.

The paper provides no evidence from other pragmatic tasks, other datasets, or other languages. The BIG-bench comparison (Appendix I) uses the same underlying dataset, so it does not constitute an independent evaluation. The extensive model coverage (48 configurations across 10 families) means the within-task results are robust, but the cross-task generalizability of any finding—including the central claim about Example IT—is completely untested.

A practitioner deploying a model for conversational applications needs to know: if the model handles implicatures from this specific dataset at 80% accuracy, does that mean it will handle novel implicatures in live conversation at a similar rate? The paper provides no distribution-shift evaluation (e.g., testing on a held-out corpus of conversational implicatures from a different source) that would answer this question. The fact that the dataset is "naturally occurring" (Section 3) improves ecological validity over synthetic data, but it is still a single curated snapshot from one research group.

Unknown and Unmeasured Deployment Costs

The paper's evaluation protocol uses single-pass likelihood scoring or greedy decoding—the model sees each prompt once and produces a single output. This gives clean accuracy measurements but ignores practical deployment costs that would affect any real-world use of these models for implicature resolution.

Latency vs. accuracy tradeoff: The paper's best results (GPT-4 with 5-shot chain-of-thought at 86.5%) require generating a full reasoning trace before outputting the answer. Each CoT prompt contains five worked examples plus an instruction to "Write your reasoning after 'Implicature:' and then output either 'Answer: yes' or 'Answer: no'" (Table 8). The generated completions shown in Appendix J are 50–100 tokens of reasoning before the final answer. In a conversational setting, this introduces a latency penalty that may be unacceptable for real-time interaction. The paper does not measure or discuss the latency implications of any evaluation configuration.

Per-example cost: The CoT 5-shot prompt in Table 8 contains approximately 350 tokens of context (five demonstrations plus instructions) before the test example. For GPT-4 or similar API-based models, this means each evaluation requires processing ~400–500 input tokens and generating ~50–100 output tokens—multiplied by 6 templates for reliability. The paper reports total GPU hours for open-source models in Appendix M (estimating up to 966 GPU hours for the full evaluation suite), but does not translate these into per-example cost estimates or compare the cost of different evaluation strategies. A practitioner deciding whether to deploy a simpler model with best-of-N sampling vs. a larger model with CoT needs this information.

Difficulty estimation is post-hoc: The paper's subtype analysis (generalized vs. particularized) is performed manually on 217 of 600 test examples after the fact. There is no automatic method for determining whether an incoming implicature is generalized or particularized before deciding how to process it. If particularized examples benefit disproportionately from CoT (as the results show), an ideal system would apply CoT only to those hard cases and use simpler prompting for easy cases—but this requires a real-time difficulty classifier that the paper does not provide.

The In-Context Example Budget Limits Practical Few-Shot Gains

The paper sweeps $k \in \{0, 1, 5, 10, 15, 30\}$ and finds that most Example IT models plateau after $k=5$ (Figure 2). However, the total number of unique few-shot examples in the development set is 118. At $k=30$ , the test set of 600 examples draws 30 × 600 = 18,000 demonstrations from a pool of only 118 examples, meaning each development example is reused approximately 150 times on average. The paper's random sampling approach means that different test examples see different (but overlapping) subsets of the 118 demonstrations, but the total pool is small enough that the model is essentially seeing the same few dozen examples recycled across all test cases.

This has two consequences. First, the 30-shot results may overestimate the model's few-shot learning capacity—the model is not learning from 30 unique demonstrations per test case but from a small, repeatedly-sampled pool. This creates a risk of overfitting to the specific surface patterns in those 118 examples (names, syntactic structures, implicature types). Second, the paper cannot distinguish between "performance improves with more examples" and "performance improves with more diverse examples." A larger development set might reveal that diversity, not count, is the critical variable—or that performance continues to improve beyond $k=30$ when the demonstrations are varied.

The paper partially addresses the first concern through the random-label experiment (Appendix K.6), which shows that Example IT models do not rely on the content of in-context labels. But this does not address whether performance would differ with a larger development set—the experiment keeps the same 118 examples and only randomizes their labels. The variance over prompt ordering experiment (Appendix K.2) tests different arrangements of the same examples, not different examples.

The paper does not acknowledge this as a limitation. The development set size (118 examples) is determined by the dataset split (600 test / 118 dev from 718 total binary implicatures) and was likely chosen to maximize test set size while retaining enough demonstrations. But the small development set means the few-shot results—especially at higher $k$ values—should be interpreted as measuring the model's ability to extract information from a small, repeatedly-presented demonstration set rather than its general few-shot learning capability from a large, diverse demonstration pool.

The ``Example IT'' Category Is Heterogeneous and Poorly Characterized

The paper's central claim—that models in the Example IT group outperform all others—depends on the coherence of this category as a meaningful grouping. The category includes: OpenAI's text-ada/babbage/curie/davinci-001, text-davinci-002, text-davinci-003, ChatGPT (GPT-3.5-turbo), GPT-4, and Cohere's command models (6B, 52.4B). These models share the property of being "fine-tuned on tasks with natural instructions for each example" (Section 4), but they differ along many unobserved dimensions: pretraining data, pretraining scale (unknown for OpenAI models), RLHF training details, dialogue-specific optimization, safety filtering, and deployment-time system prompts.

The consequence is that the "Example IT" label may be a proxy for other correlated factors that the paper cannot disentangle. For instance, all Example IT models except Cohere-command are also heavily optimized for dialogue and helpfulness through RLHF—properties that the Dialogue FT group (BlenderBot) was supposed to test but that BlenderBot does not represent at the same scale or with the same training methodology. If RLHF is the key factor (not instruction granularity), the paper's taxonomy would be misleading: RLHF-trained models happen to fall in the Example IT group, but Benchmark IT or Dialogue FT models with RLHF might perform equally well. The paper cannot test this because all Example IT models are proprietary, and RLHF-trained variants of Benchmark IT or Dialogue FT models at comparable scales are not available.

The within-family comparison (Cohere-command-52B vs. Cohere-base-52B) partially addresses this by showing that the Example IT advantage holds even when the base model is held constant—but this is a single comparison in a single model family, and Cohere's specific implementation of example-level instruction-tuning may differ from OpenAI's in ways that matter. The paper acknowledges this uncertainty (Section 5):

"Unfortunately, due to the secrecy surrounding the exact implementation of these models we cannot be certain, but we can be relatively confident."

The paper notes that the "most salient commonality" among Example IT models is instruction-tuning at the example level, but "most salient" is a judgment, not a measurement. A practitioner should treat the Example IT grouping as an empirical cluster of high-performing models rather than a verified causal category. The practical takeaway—models sold as instruction-tuned conversational agents perform better on implicatures—is reliable. The theoretical takeaway—that example-level instruction granularity specifically causes this improvement—is plausible but unproven.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new model, training method, or architectural innovation. Its contribution is diagnostic and reframing: it provides a rigorous measurement tool that reveals a specific, previously invisible capability gap in large language models. The impact on the field is therefore not a paradigm shift in how models are trained, but a shift in what constitutes a meaningful evaluation of language understanding—and, by extension, what counts as progress toward genuinely communicative AI.

The magnitude of this shift is incremental in its immediate technical consequences but potentially significant in its long-term influence on evaluation culture. The paper does for pragmatic understanding what earlier work (e.g., the GLUE/SuperGLUE benchmarks, BIG-bench) did for syntactic competence and broad task coverage: it identifies a dimension of language ability that is systematically neglected by existing evaluations, demonstrates that current models—including the most advanced—perform poorly on it, and provides a reproducible protocol that makes the capability measurable. The difference is that pragmatics is not just another task category; it is a defining feature of communication, distinct from text production. The paper's central argument—that benchmarks measuring helpfulness, coherence, and commonsense reasoning miss the fact that "what is helpful depends on context" (Section 2)—has implications beyond implicature resolution. It challenges the field to ask: what other aspects of human communicative competence are we failing to measure?

The work also resolves a specific tension in the prior literature with direct practical consequences. Before this paper, the BIG-bench implicature task (Srivastava et al., 2022) reported that PaLM achieved 91.7% two-shot accuracy on implicature resolution—seemingly close to human performance. This paper's comprehensive evaluation demonstrates that this number was an artifact of methodological choices: BIG-bench discarded approximately 30% of the data as "ambiguous," and those discarded examples are precisely the particularized implicatures that constitute the hardest case for both humans and models. On the full dataset, the best model (GPT-4 with CoT at 86.5%) falls ~6 percentage points below the best human performance (92%), and the gap is concentrated on the context-dependent particularized examples that dominate natural conversation. This reconciliation is important because it prevents the field from prematurely concluding that pragmatic understanding is a solved problem—a conclusion that would direct research attention and commercial investment away from pragmatic competence.

The paper redirects research attention along several axes:

Away from scale as a universal solution. The finding that base models at 175B parameters perform essentially at the same level as models at 100M parameters on zero-shot implicature resolution (Figure 3, left) is a concrete counterexample to the narrative that larger pretrained models are uniformly more capable. Scale matters for pragmatic understanding only when combined with the right fine-tuning recipe (Figure 3, right)—a finding that complicates the scaling-laws narrative and suggests that some capabilities require targeted training signals, not just more parameters and data.
Toward training data diversity as a causal lever. The performance gap between Benchmark IT and Example IT models (e.g., Flan-T5-11B at 61.7% vs. text-davinci-003 at 79.7% at 5-shot; Table 1) isolates instruction granularity—the number of distinct task framings the model sees during training—as a candidate causal factor. This makes instruction diversity engineering a newly tractable research direction: if varying the number of distinct instructions per task affects downstream pragmatic competence, then designing instruction sets to maximize pragmatic understanding becomes a concrete optimization target.
Toward subtype-aware evaluation. The paper's taxonomy of implicature types (generalized, particularized, world knowledge, idiom, rhetorical question) and the finding that model performance varies dramatically across subtypes (Figure 4, Appendix K.9) establishes a template for capability diagnosis rather than aggregate benchmarking. Future evaluations of pragmatic understanding—and perhaps of other complex language abilities—should decompose performance by difficulty subtype rather than reporting a single accuracy number. This makes it possible to identify not just that a model fails but where and why it fails, enabling targeted improvement.
Away from prompt engineering as a substitute for capability. The paper's negative results with elaborate zero-shot instructions (Appendix K.3) and the finding that CoT only helps models that already have substantial pragmatic competence (base models degrade with CoT; Appendix K.7) establish a boundary condition on the "prompting can solve anything" narrative. No amount of prompt engineering induced pragmatic understanding in models that lacked it. This reframes prompt design from a capability-unlocking tool to a capability-amplifying tool—useful only when the underlying competence already exists.

Follow-Up Research This Work Enables

Controlled causal experiment isolating instruction granularity. The paper's central finding—that Example IT models outperform all other categories—is correlational. The models in this group differ from others along multiple dimensions: instruction granularity, RLHF training, dialogue optimization, and proprietary data mixtures. A clean causal experiment would take a single publicly available base model (e.g., Llama-2-7B or Mistral-7B), create two fine-tuning datasets on the same set of underlying NLP tasks (e.g., the Flan collection), one with a single instruction per benchmark (replicating Benchmark IT) and one with a new instruction per example (replicating Example IT), train both variants with the same hyperparameters and budget, and evaluate on the paper's implicature resolution protocol. If the example-level variant significantly outperforms the benchmark-level variant, instruction granularity is causally implicated. If not, other factors (RLHF, data mixture) become the primary suspects. The experiment would also provide a public, reproducible model pair for further analysis.

Cross-phenomenon pragmatic evaluation benchmark. The paper evaluates only binary conversational implicatures. A natural extension is to build a benchmark covering the broader range of pragmatic phenomena: scalar implicatures (e.g., "some" → "not all"), conventional implicatures (e.g., "but," "therefore"), presuppositions (e.g., "John stopped smoking" presupposing John once smoked), indirect speech acts (e.g., "Can you pass the salt?" as a request, not a question), and relevance implicatures (e.g., "It's cold in here" as a request to close the window). Each phenomenon would use a minimal-pair likelihood ranking design modeled on this paper's protocol. The key question: does the Example IT advantage generalize across pragmatic phenomena, or is it specific to the yes/no implicature format? The paper's finding that GPT-4's advantage over Cohere-command is concentrated on generalized examples (Figure 4) suggests that subtype-specific evaluation is essential—a cross-phenomenon benchmark would reveal whether Example IT models are genuinely pragmatically competent or merely skilled at a narrow set of inference patterns.

Difficulty prediction without ground-truth labels. The paper's subtype analysis (generalized vs. particularized) is performed manually post-hoc on a subset of examples. A practical system needs automatic difficulty estimation—not just for evaluation but for adaptive deployment: easy implicatures can be handled with simple prompting, while hard ones require chain-of-thought or more expensive processing. A concrete follow-up would train a classifier (based on a smaller, efficient model) to predict whether an incoming utterance-response pair is a generalized or particularized implicature, using features like: the presence of scalar terms ("some," "all"), the length of the response, the semantic distance between utterance and response, and the model's own internal representations. If such a classifier can achieve >80% accuracy at distinguishing the subtypes identified in the paper, it opens the door to compute-adaptive pragmatic inference—applying expensive reasoning only where needed, analogous to the compute-optimal test-time scaling framework from the prior example paper.

Chain-of-thought as a probe for pragmatic reasoning structure. The paper shows that CoT helps GPT-4 on particularized examples (+10 points; Table 34) and provides example CoT completions in Appendix J. A qualitative follow-up would systematically analyze the CoT reasoning traces to identify what kind of reasoning CoT enables. Do the traces consistently identify the Gricean relevance connection (e.g., "Alice's response must be relevant to Bob's question")? Do they articulate the specific world knowledge needed (e.g., "having to work means you cannot attend a party")? Do they fail in systematic ways (e.g., generating plausible-sounding but incorrect justifications)? A detailed annotation of CoT traces—comparing correct and incorrect completions on the same examples—would reveal whether CoT improves performance by structuring genuine pragmatic reasoning or by providing a template that increases the probability of the correct token through surface-level pattern matching. The paper's five-shot CoT prompt (Table 8) explicitly models the relevance reasoning step ("Alice's response must be relevant to Bob's question because Alice is a cooperative conversational partner"), making this analysis tractable.

Cross-linguistic and cross-cultural pragmatic evaluation. The paper evaluates only English implicatures. Pragmatic norms are culturally and linguistically variable—what counts as a cooperative response, whether indirectness is preferred, and how implicatures are conventionally resolved all differ across languages and communities. A direct extension would replicate the evaluation protocol on implicature datasets in other languages (e.g., using translated versions of the George and Mamidi dataset or collecting new datasets in languages with different pragmatic conventions). The key question: does the Example IT advantage hold across languages, or is it specific to English pragmatic conventions? If Example IT models perform well on English implicatures but poorly on, say, Japanese implicatures (where indirectness norms differ substantially), that would reveal that the pragmatic competence induced by example-level instruction-tuning is culturally specific—an important boundary condition for deploying these models as cross-cultural conversational agents.

Testing whether in-context examples teach general strategies or type-specific patterns. The paper's few-shot experiments draw demonstrations from a development set that contains the same mixture of implicature types as the test set. A controlled follow-up would use type-restricted few-shot prompts: for test examples of type generalized, only use generalized demonstrations; for test examples of type particularized, only use particularized demonstrations; and crucially, cross the types (test on particularized with only generalized demonstrations and vice versa). If performance drops significantly in the cross-type condition, the model is matching surface patterns specific to each subtype rather than learning general pragmatic reasoning strategies. If performance is comparable, the model is extracting abstract reasoning patterns that transfer across subtypes. The paper's random-label experiment (Appendix K.6) only tests whether label content matters; this experiment would test whether demonstration type matters, providing insight into the mechanism of in-context pragmatic learning.

Scaling laws for instruction diversity. The paper's finding that Example IT models show steeper scaling curves than base models (Figure 3, right) suggests an interaction between model size and instruction diversity. A systematic study could vary the number of distinct instructions in the fine-tuning data (e.g., 10, 100, 1000, 10,000 unique instruction phrasings for the same set of underlying tasks) and measure implicature resolution performance as a function of both model size and instruction count. This would produce instruction diversity scaling laws analogous to pretraining data scaling laws—characterizing how much instruction diversity is needed to induce pragmatic competence at a given model scale. The practical payoff would be guidance for training data curation: if pragmatic competence saturates at, say, 5,000 distinct instructions for a 7B model but continues improving to 50,000 for a 70B model, training budgets can be allocated accordingly.

Practical Applications and Downstream Use Cases

Conversational agent quality assurance. Organizations deploying LLMs as customer-facing conversational agents (customer support, virtual assistants, healthcare triage) currently evaluate these systems on metrics like factual accuracy, coherence, and helpfulness ratings. This paper demonstrates that these metrics miss a critical failure mode: the model may produce responses that are factually correct and grammatically coherent but pragmatically misaligned with the user's intent. A customer asking "Do you have this in blue?" and receiving "We have it in red, green, and yellow" (implicature: no, but we have other colors) may not receive an explicitly wrong answer—the model correctly listed available colors—but the interaction fails because the implicature is not communicated. Integrating the paper's evaluation protocol into QA pipelines (e.g., running a random subset of conversational logs through the minimal-pair likelihood test) would detect pragmatic misalignment that current metrics miss, at a cost of a single forward pass per test instance. The protocol's prompt robustness (six templates) and human calibration (86.2% ceiling) make it suitable for production monitoring.

Training data filtering for self-improvement pipelines. When LLMs are used to generate training data for subsequent fine-tuning rounds (as in STaR, ReST $^{EM}$ , or constitutional AI), the quality of generated examples matters critically. A model that misresolves implicatures will generate training data containing systematically incorrect pragmatic inferences—errors that may propagate through further training. The paper's evaluation protocol could be used as a filter: before using model-generated conversational data for self-training, screen a sample for pragmatic accuracy by running it through the implicature resolution test. If the model's performance on generalized examples is high (e.g., >85%) but low on particularized examples (e.g., <75%), the generated data can be weighted or filtered to prioritize examples where the model's pragmatic competence is reliable. This is a lightweight check—it requires no additional training, only the evaluation harness—and could prevent self-reinforcement of pragmatic errors in iterative training loops.

Difficulty-aware routing in multi-model deployment systems. Organizations that deploy multiple model tiers (e.g., a cheap, fast model for routine queries and an expensive, slow model for hard queries) currently route based on heuristics like query length, topic classification, or confidence scores. This paper's subtype analysis (Figure 4) suggests a complementary routing signal: estimated implicature difficulty. A lightweight classifier trained to distinguish generalized from particularized implicatures could flag incoming utterances that contain likely particularized implicatures (context-heavy, requiring world knowledge integration) and route them to a more capable model (e.g., GPT-4 with CoT), while generalized implicatures (context-free, lexically resolvable) go to a cheaper model (e.g., a distilled instruction-tuned model). The paper's finding that GPT-4 + CoT achieves 88.77% on generalized examples but only 81.63% on particularized ones (Tables 33–34), while the gap between human performance on these subtypes is smaller (92.17% vs. 83.18%; Table 22), means that the hard subtype is precisely where the best models are most needed—and where they are currently furthest from human performance. Routing hard cases to the best available model is a practical stopgap while the particularized gap remains open.

When to Prefer This Evaluation Protocol

The paper does not propose a method to be "preferred" over alternatives—it is not a system-building paper, and the evaluation protocol is positioned as a new addition to the benchmarking toolkit rather than a replacement for existing evaluations. The paper compares its protocol against the BIG-bench implicature task (Appendix I) and finds BIG-bench's methodology wanting (discarding ambiguous data, low-quality human annotations, evaluating only base models), but does not articulate a decision rule for practitioners choosing between evaluation approaches. The protocol is presented as a complement to, not a substitute for, standard benchmarks—it measures a capability (pragmatic implicature resolution) that existing benchmarks do not measure, and its adoption does not require abandoning other evaluations. A practitioner would use this protocol when they specifically want to diagnose pragmatic competence in conversational models; they would use standard benchmarks for other capabilities. There is no tradeoff to resolve—only a gap to fill.