Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

ArXiv: 2202.12837

🎯 Pitch

Large language models don't actually need correct labels in few-shot prompts—random labels work almost as well. Their in-context 'learning' relies instead on recognizing the label space, input distribution, and sequence format.

1. Executive Summary

This paper empirically analyzes what makes in-context learning effective by systematically ablating the components of few-shot demonstrations—the input-label mapping (whether each example is paired with its correct label), the distribution of the input text (whether the example inputs come from the target task's distribution), the label space (whether the labels shown are from the correct output vocabulary), and the format (whether inputs and labels are presented as paired sequences)—across 26 classification and multi-choice datasets using 12 models including GPT-3. The central, counter-intuitive finding is that the ground truth input-label mapping contributes only marginally: replacing gold labels with random labels barely hurts performance, with drops of only 0–5% absolute across nearly all models and task types. Instead, the paper identifies that performance gains derive primarily from the independent specification of the input distribution and the label space, with the format of pairing inputs to labels acting as an essential scaffold—in many cases retaining up to 95% of in-context learning improvements using only unlabeled inputs paired with random labels or only the label set paired with out-of-distribution inputs, establishing that models recover task semantics largely from pretraining rather than learning new input-label correspondences at inference time.

2. Context and Motivation

The Core Problem: We Don't Understand Why In-Context Learning Works

In-context learning, introduced by Brown et al. (2020) with GPT-3, is one of the most striking capabilities of large language models: condition a model on a handful of input-label examples (demonstrations) at inference time, and it performs a new task reasonably well—without any gradient updates, fine-tuning, or parameter changes. This capability has driven enormous practical adoption because it means practitioners can deploy language models for new tasks simply by writing a well-structured prompt, bypassing the engineering overhead of dataset curation, hyperparameter tuning, and model retraining that fine-tuning requires. It also opens theoretical questions about whether language models can serve as general-purpose inference engines that acquire new behaviors on the fly, rather than merely retrieving information stored during pretraining.

Despite this widespread use and the empirical success of in-context learning across benchmarks (Zhao et al., 2021; Liu et al., 2021), the paper identifies a fundamental gap: we have almost no understanding of how the model actually learns from the demonstrations and which aspects of those demonstrations drive performance. As the authors state in Section 1:

"there has been little understanding of how it works and which aspects of the demonstrations contribute to end task performance."

This is not a minor oversight—it is a conceptual vacuum at the center of a widely used technique. Without understanding what the model extracts from demonstrations, we cannot answer basic questions: Could in-context learning work without correct labels? Does the model figure out which inputs map to which labels, or does it use the demonstrations for something else entirely? What would happen if we gave it misleading demonstrations? These questions matter for both practical reliability (can we trust the model to honor the task definition we provide?) and for scientific understanding of what large language models learn during pretraining.

The Implicit Assumption: Demonstrations Teach Input-Label Correspondences

Prior to this paper, the working assumption in the field—though rarely stated explicitly—was that in-context learning works the same way supervised learning works: the model examines the provided (input, label) pairs, induces the mapping between inputs and correct outputs, and applies that mapping to new test inputs. This is how humans interpret demonstrations. If I show you three reviews labeled "positive," "negative," and "neutral," you assume the task is sentiment classification and you use those pairings to learn what each label means in context. The natural extrapolation is that language models do the same—that the ground truth pairing of each input to its correct label is the essential signal.

This assumption is so intuitive that it shaped research priorities. When prior work asked "what makes good demonstrations?" (Liu et al., 2021; Lu et al., 2021; Rubin et al., 2021), the question was implicitly: which examples best teach the input-label mapping? Methods for selecting demonstrations (e.g., choosing examples similar to the test input, or retrieving examples with high mutual information) were designed to make the mapping easier to infer. The underlying premise—that the model needs to learn the mapping—was uncontested.

The paper directly challenges this premise. The authors hypothesize that demonstrations might work for reasons completely unrelated to learning the input-label correspondence, and they design experiments to isolate whether the mapping actually matters. This is the central gap the paper addresses: is in-context learning actually "learning" the task mapping from the provided examples, or is it something else entirely?

Why This Matters: Practical and Theoretical Stakes

The answer to this question has immediate practical consequences. If models genuinely learn input-label correspondences from demonstrations, then practitioners must provide accurate labels—incorrect or random labels should catastrophically degrade performance. The demonstrations serve as a miniature supervised training set, and their quality should directly determine output quality. This would mean real-world deployments need careful curation of few-shot examples, with attention to label correctness, class balance, and representativeness.

If, however, the model does not rely on the input-label mapping, then demonstrations serve a different function—perhaps indicating the kind of task to perform without actually teaching the task. The practical implication would be profound: random labels might work nearly as well as gold labels, meaning practitioners could construct demonstrations from unlabeled data or even synthetic data, dramatically reducing the annotation burden for few-shot applications. The reliability question also inverts: if the model ignores label correctness, is it actually performing the specified task, or is it pattern-matching against pretraining priors in ways that might fail silently on inputs unlike anything seen during pretraining?

The theoretical stakes are equally significant. In-context learning has been positioned as a form of meta-learning or implicit Bayesian inference (Xie et al., 2022)—the idea that the model uses demonstrations to infer latent task parameters and then applies those parameters to new inputs. If the input-label mapping is irrelevant, this Bayesian inference account needs substantial revision. The model would not be "learning a new task" in any traditional sense. Instead, it would be leveraging the demonstrations to locate the right behavior within its existing repertoire—essentially, using the demonstrations as a retrieval key rather than a training signal. This distinction between learning and locating is central to the paper's contribution and is the conceptual shift it argues for.

Where Prior Work Falls Short

The paper identifies several categories of prior work, each of which either failed to address this question directly or actively reinforced the conventional assumption.

In-context learning methodology work focused on optimizing demonstration selection and formatting. Liu et al. (2021) studied which examples make good demonstrations; Lu et al. (2021) investigated order sensitivity; Zhao et al. (2021) proposed calibration methods; Holtzman et al. (2021) examined surface form competition; Min et al. (2021a) introduced the channel method (flipping input and label positions for classification). All of these papers implicitly assumed the input-label mapping was what mattered and sought to make it more accessible to the model. None questioned whether the mapping was necessary.

Meta-training work (Chen et al., 2021; Min et al., 2021b) trained models explicitly on an in-context learning objective—presenting sequences of few-shot examples followed by a test input—to improve few-shot generalization. This line of work demonstrated that models can be trained to do in-context learning better, but it did not investigate which components of those training demonstrations the model was actually using. This paper includes MetaICL (Min et al., 2021b), a meta-trained model, in its analysis and finds that the meta-trained model is even less sensitive to label correctness than non-meta-trained models—suggesting meta-training amplifies reliance on superficial demonstration components (format, label space) rather than the mapping.

Theoretical accounts attempted to formalize in-context learning but did not make testable predictions about which demonstration components matter. Xie et al. (2022) proposed that in-context learning can be understood as implicit Bayesian inference where the model uses demonstrations to infer a latent concept (e.g., the distribution over labels given inputs for this task). This account would predict that correct input-label pairings are essential, because they provide the evidence needed for the Bayesian update. The paper's finding that random labels barely hurt performance is difficult to reconcile with this theory—if the model were performing Bayesian inference over tasks, incorrect evidence should lead to incorrect posterior beliefs and degraded performance.

Correlational analyses examined relationships between pretraining data statistics and in-context learning performance. Razeghi et al. (2022) showed that few-shot performance correlates strongly with term frequencies in the pretraining data—the model performs better on tasks whose vocabulary and structure are well-represented in what it was trained on. This work hinted that pretraining, not demonstrations, might be the primary driver, but it did not experimentally disentangle the components of the demonstrations themselves. The current paper builds on this insight but goes further: it establishes through direct ablation that the mapping component of demonstrations is dispensable while other components (input distribution, label space, format) are not.

Instruction-following work (Wei et al., 2022a; Sanh et al., 2022; Mishra et al., 2021b) achieved strong zero-shot performance by training models to follow natural language instructions rather than few-shot demonstrations. This line of work suggested that task specification without examples is possible, but it treated instructions as an alternative paradigm; it did not investigate whether the few-shot demonstrations paradigm might already be functioning as a kind of implicit instruction rather than as supervised training data. The current paper helps connect these literatures by showing that demonstrations serve a role similar to instructions—they specify the task format and output space rather than teaching the task through examples.

Reynolds and McDonell (2021) came closest to the paper's thesis with their claim that demonstrations are for "task location" and that "the intrinsic ability to perform the task is obtained at pretraining time." However, they argued from this that demonstrations are unnecessary—that prompting alone should suffice. The current paper takes a more nuanced position: demonstrations are not unnecessary because they do provide essential scaffolding (input distribution, label space, format), just not through the mechanism everyone assumed (the input-label mapping). Additionally, Reynolds and McDonell (2021) provided no systematic empirical analysis of which demonstration components matter; the current paper provides precisely this through controlled ablation experiments across 12 models and 26 datasets.

How This Paper Positions Itself

The paper explicitly frames its contribution as the first systematic empirical analysis that investigates why in-context learning achieves performance gains over zero-shot inference. In Section 2, the authors state:

"To the best of our knowledge, this paper is the first that provides an empirical analysis that investigates why in-context learning achieves performance gains over zero-shot inference."

This framing is precise: the question is not whether in-context learning works (established by Brown et al., 2020 and many subsequent papers), but why it works—what specific components of the demonstrations are necessary, which are sufficient, and which are irrelevant.

The paper's approach is decompositional and ablative rather than methodological or theoretical. Rather than proposing a new in-context learning method or a formal theory, it takes the existing method and systematically removes or randomizes individual components to measure their impact. This is a classic engineering approach to understanding a black-box system: disable parts one at a time and see what breaks. The breakthrough is applying this mindset to something the field had treated as a monolothic "learning from examples" process.

The paper identifies four distinct aspects of demonstrations that could potentially provide signal, as depicted in Figure 7:

1. The input-label mapping (each x_i correctly paired with y_i)
2. The distribution of the input text (the examples come from the target task's domain)
3. The label space (the examples show the correct output vocabulary)
4. The format (inputs and labels are presented as paired sequences in a particular template)

The critical experimental move is to design demonstration variants that independently ablate each aspect while preserving the others. For example: "demonstrations with random labels" disrupts the mapping while preserving input distribution, label space, and format; "OOD demonstrations with random labels" disrupts both the mapping and the input distribution while preserving label space and format; "demonstrations with random English words" disrupts both the mapping and the label space while preserving input distribution and format. By comparing performance across these variants, the paper quantifies the marginal contribution of each component.

This design enables a clean answer to the central question: the input-label mapping (what everyone assumed was doing the work) contributes at most a few percentage points, while the input distribution, label space, and format together account for nearly all of in-context learning's gains. The paper's position is thus not just critical—showing that the prevailing assumption is wrong—but constructive: it identifies what actually matters and provides a new conceptual framework for understanding demonstrations as task specifiers rather than training data.

A final aspect of the positioning is that the paper explicitly connects its findings to the broader capability landscape of language models. In the Discussion (Section 6), the authors address the implication head-on: if the model doesn't learn the input-label mapping from demonstrations, then the task competence must have been acquired during pretraining. The demonstrations serve to surface latent capabilities—what the authors call the model's "zero-shot capacity, even if it is not always evident from the naive zero-shot accuracy." This reframes in-context learning from a mechanism for acquiring new behavior to a mechanism for accessing existing behavior, which has downstream implications for when in-context learning should be expected to work (tasks resembling pretraining data) versus fail (genuinely novel tasks with no pretraining precedent).

3. Technical Approach

3.1 Reader Orientation

This paper constructs a systematic ablation framework—a set of controlled experiments that surgically disable individual components of in-context learning demonstrations—to measure what actually drives performance gains. Rather than building a new method, the researchers take the standard few-shot prompting pipeline and create variants where the input-label mapping (correct pairings), input distribution (whether examples come from the target task), label space (whether shown labels are from the correct output vocabulary), and format (whether inputs and labels appear as pairs) are independently randomized or removed. The core problem is that the field has been treating in-context learning as a black-box "learning from examples" process without knowing which specific signal the model extracts from those examples; the solution is to isolate each signal and measure its marginal contribution through a factorial-style experimental design across 12 models and 26 datasets.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental apparatus has five major components, arranged as a pipeline that transforms raw task data into measured accuracy:

1. Dataset Collection — 26 classification and multi-choice tasks spanning sentiment analysis, paraphrase detection, natural language inference, hate speech detection, question answering, and sentence completion. Each dataset provides training examples (from which demonstrations are sampled), a label space C (the set of possible outputs), and a test set for evaluation. This component supplies the ground-truth (input, label) pairs and defines what "correct" means for each task.

2. Demonstration Constructor — the core experimental manipulation point. Given a dataset, a random seed, and a specification of which components to ablate (e.g., "use gold labels," "use random labels," "use out-of-distribution inputs," "use random English words instead of task labels"), this component samples k = 16 examples and formats them into a demonstration string. The constructor has five distinct modes corresponding to the variants in Section 5: gold labels (all components intact), random labels (mapping ablated), OOD inputs + random labels (mapping + input distribution ablated), random English words (mapping + label space ablated), and format-removal variants (no labels, labels only). Each mode produces a string like "input1 \n label1 \n\n input2 \n label2 \n\n ... \n\n test_input \n" that the model ingests.

3. Language Model — the inference engine. Twelve decoder-only dense language models are tested: GPT-2 Large (774M parameters), MetaICL (774M, meta-trained on in-context learning), GPT-J (6B), fairseq 6.7B, fairseq 13B, and GPT-3 (175B, Davinci base). Each model is used with both the direct method (predict P(y | demonstrations, x)) and the channel method (predict P(x | demonstrations, y) and select the label that makes the input most probable), following Min et al. (2021a). This component takes a demonstration string concatenated with the test input and outputs either a probability distribution over labels (direct) or a scalar score per label (channel).

4. Evaluation Harness — computes accuracy (for multi-choice) or Macro-F1 (for classification) by comparing the model's predicted label against the ground-truth test label. For each (dataset, seed, variant) combination, the harness runs 5 independent trials with different random seeds (3 for larger models due to compute constraints), then averages per-dataset performance across seeds and macro-averages across datasets. This component produces the numbers that appear in Figures 3–10.

5. Cross-Variant Comparator — the analytical layer that isolates each component's contribution. By computing performance differences between specific variant pairs (e.g., gold labels minus random labels measures the mapping contribution; random labels minus OOD+random labels measures the input distribution contribution when the mapping is already disabled), the comparator quantifies the marginal value of each aspect. This is not a separate code module but the logical structure of the experimental design, reflected in the comparison tables at the bottom of Figures 8–10.

Information flows left to right: a dataset and ablation specification enter the constructor → the constructor samples and formats demonstrations → the LM scores possible outputs → the evaluation harness computes accuracy → the cross-variant comparator isolates component contributions.

3.3 Roadmap for the Deep Dive

• First, the demonstration construction logic and the five variant modes (Section 3.4.1) — this is the core experimental manipulation. Understanding exactly how each variant is built is necessary to interpret every subsequent result, because the entire paper's claims rest on what each variant removes relative to the others. I will walk through the formatting rules, the sampling procedure, and the precise definition of each variant.

• Second, the inference methods (direct and channel) and their interaction with the demonstration variants (Section 3.4.2) — the choice of inference method affects which components matter (e.g., the label space matters for direct models but not channel models). I will explain the mathematical formulation of each method and why they produce different sensitivity patterns.

• Third, the model zoo and the role of meta-training (Section 3.4.3) — the 12 models span a 200× range in parameter count and include one meta-trained model (MetaICL). I will detail which models are used for which experiments, the compute constraints that limit larger-model evaluation, and the specific meta-training procedure for MetaICL (since it produces qualitatively different behavior).

• Fourth, the dataset selection and evaluation protocol (Section 3.4.4) — I will cover the 26 datasets, the choice of k = 16, the 5-seed replication strategy, the minimal vs. manual template distinction, and the metrics (Macro-F1 vs. Accuracy). These details matter because they constrain the generality of the findings and because some results (e.g., per-dataset breakdowns in Figure 12) depend on dataset-specific properties.

• Fifth, the factorial comparison logic and the "four aspects" framework (Section 3.4.5) — this is the analytical skeleton of the paper. I will explain how the four aspects (input-label mapping, input distribution, label space, format) were identified, how each variant maps to a specific disabling pattern, and how pairwise comparisons between variants isolate individual contributions. This framework is what allows the paper to move beyond "random labels don't matter much" to precise statements like "the input distribution contributes 3-16% absolute" or "the format is essential."

• Sixth, the controlled ablation variants in detail (Section 3.4.6) — I will provide the precise construction algorithm for each variant (including Algorithm 1 from the appendix for the a% correct labels variant), the external corpora used (CC-News for OOD inputs, the english-words package for random English words), and the constant-label and test-input variants mentioned in Appendix C.3. These details are scattered across the main text and appendices; I will consolidate them.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that in-context learning performance gains can be decomposed into four independent factors, and that the input-label mapping—the one everyone assumed was central—contributes only marginally while the other three (input distribution, label space, format) account for nearly all of the gain.

3.4.1 Demonstration Construction and the Five Variant Modes

The demonstration is a string formed by concatenating k input-label pairs followed by the test input, with separator tokens between examples. The exact formatting follows model-specific conventions established in prior work: for GPT-2, the input and label are separated by a space and each demonstration example is separated by a space; for MetaICL, GPT-J, and GPT-3, the input and label are separated by a newline (\n) and each demonstration example is separated by three newlines; for fairseq models, both the input-label separator and the inter-example separator are single newlines. These conventions are not arbitrary—they match what each model saw during pretraining or meta-training, reducing distribution shift between the demonstration format and the model's training data.

The paper defines five primary variant modes, each corresponding to a specific pattern of which of the four aspects (input-label mapping, input distribution, label space, format) is intact (marked as "yes" in the comparison tables) versus ablated ("no"):

Mode 1: Demonstrations with gold labels. This is standard in-context learning. For each of the k positions, a training example (x_i, y_i) is sampled uniformly at random from the dataset's training split, and the pair is formatted as described above. All four aspects are intact: the mapping is correct by construction, the inputs come from the task distribution, the labels populate the correct label space C, and the pair format is used. This mode serves as the upper-bound reference for what in-context learning can achieve with full information.

Mode 2: Demonstrations with random labels. The inputs x_1...x_k are sampled from the training data as in Mode 1, but each x_i is paired with \tilde{y}_i sampled uniformly at random from C (the set of possible labels), independent of x_i. The sampling is with replacement, so the same random label may appear multiple times. The input distribution, label space, and format are preserved; only the input-label mapping is ablated. This is the central experimental condition—it answers the question "does the model need to see which specific inputs map to which specific labels?"

For class imbalance robustness, Appendix C.2 explores an alternative where \tilde{y}_i is sampled from the true empirical distribution of labels in the training data rather than uniformly. For example, if a dataset has 70% positive, 20% negative, and 10% neutral labels in the training set, the random labels would reflect this skew. This variant produces slightly smaller performance gaps relative to gold labels (e.g., the gap shrinks from 1.9% to 1.3% absolute for Channel MetaICL, and from 5.0% to 3.5% for Channel GPT-J), suggesting that label distribution information—distinct from input-label correspondence—provides a small additional signal.

Mode 3: OOD demonstrations with random labels. Both the input distribution and the input-label mapping are ablated simultaneously. The inputs x_{i,rand} are sampled from an external corpus (CC-News, Nagel, 2016) rather than from the training data, with length-matching to ensure the sampled sentences have similar character/word counts to the test input. Each out-of-distribution input is paired with a random label sampled uniformly from C. The label space and format remain intact. This variant isolates the contribution of the input distribution: by comparing Mode 3 to Mode 2, we measure how much performance drops when the demonstration inputs switch from in-distribution task text to generic news sentences, holding the (already-disabled) mapping and the (still-intact) label space and format constant.

The CC-News corpus was chosen because it is a large, diverse collection of English news articles, providing semantically unrelated text that does not accidentally resemble any of the 26 evaluation tasks. The length-matching step is important because language models are sensitive to sequence length; if the OOD inputs were systematically shorter or longer than the task inputs, differences in perplexity or generation behavior could confound the comparison.

Mode 4: Demonstrations with random English words. Both the label space and the input-label mapping are ablated simultaneously. The task labels C are replaced with C_rand, a random subset of English words drawn from the english-words Python package (61,569 words total), where |C_rand| = |C| (the same cardinality as the original label space). Each training input x_i is paired with a randomly sampled English word \tilde{y}_i \in C_rand. The resulting demonstrations have the structure "input \n some_random_english_word" repeated k times, followed by the test input. The input distribution and format remain intact; only the label space and mapping are ablated. By comparing Mode 4 to Mode 2, we measure the contribution of using the correct label vocabulary (e.g., "positive"/"negative"/"neutral") versus arbitrary English words (e.g., "unanimity"/"wave"/"potato").

The choice of random English words rather than random tokens or non-words is deliberate: English words preserve the surface-level property that the labels are recognizable lexical items the model has seen during pretraining, but they carry no task-specific semantics. If the model performed equally well with random English words as with the correct labels, that would indicate the label space provides no information beyond "these are English words." If performance drops substantially, it indicates the model is sensitive to which specific words appear as labels—even without correct input-label pairings.

Mode 5: Format-removal variants. Two sub-variants test the importance of the input-label pairing structure itself:

• Demonstrations with no labels (inputs only): The LM is conditioned on the concatenation x_1, x_2, ..., x_k (with example separators) followed by the test input. No labels appear anywhere in the demonstration. This removes the label space, the mapping, and the pairing format, preserving only the input distribution. The prediction must be made from the test input alone, but the model has been primed with in-distribution text from the task.

• Demonstrations with labels only: The LM is conditioned on the concatenation y_1, y_2, ..., y_k (with example separators) followed by the test input. No inputs appear in the demonstrations except the test input. This removes the input distribution, the mapping, and the pairing format, preserving only the label space. The model sees only a sequence of valid task labels before making its prediction.

These format-removal variants are the no-format counterparts of Mode 4 (random English words) and Mode 3 (OOD inputs), respectively. By comparing Mode 4 ("random English words" — format intact, label space ablated) to "labels only" (format ablated, label space intact), we can see whether the format or the label space is more important when the other is present. Similarly, comparing Mode 3 ("OOD + random labels" — format intact, input distribution ablated) to "no labels" (format ablated, input distribution intact) isolates the format's contribution when paired with input distribution information.

The a% correct labels variant (Section 4.2, Algorithm 1). For further granularity on the mapping's contribution, the paper constructs demonstrations where exactly a% of the k examples have correct labels and (100 - a)% have incorrect labels, with a swept across {0, 25, 50, 75, 100}. The construction procedure (Algorithm 1) works as follows: given the k training examples and a target accuracy percentage a, compute n = k × a / 100 (the number of correct pairs to include). Sample n indices uniformly from {1, ..., k} to designate as correct positions. For positions in the designated set, use the gold (x_i, y_i) pair. For positions outside the set, use (x_i, Sample(C \ {y_i})) — the correct input paired with a random label from C excluding the gold label, ensuring the pair is genuinely incorrect rather than accidentally correct by chance. The resulting demonstration has n correct pairs and k - n incorrect pairs in random order.

This variant is more nuanced than the all-or-nothing random labels (Mode 2) because it tests whether the model extracts a graded signal from the proportion of correct examples. If performance degrades smoothly as a decreases, that suggests the model does attend to label correctness to some degree; if performance is flat until very low a, that suggests near-total insensitivity. Figure 4 shows results for a ∈ {0%, 25%, 50%, 75%, 100%} plus the no-demonstrations baseline.

The constant label and test-input variants (Appendix C.3). Two additional exploratory variants are mentioned but not emphasized because they performed poorly:

• Demonstrations with constant labels: Every x_i is paired with the literal string "answer" instead of a task label. This removes the label space information while preserving the pairing format and input distribution. However, performance was consistently worse than Mode 4 (random English words). The authors hypothesize that the constant label "answer" effectively becomes part of the separator between examples rather than functioning as a label, altering the format in a way the random English words do not.

• Demonstrations with test input: Every demonstration example uses the test input x_test as its input (i.e., (x_test, \tilde{y}_1), (x_test, \tilde{y}_2), ..., (x_test, \tilde{y}_k), each paired with a random label). This preserves the input distribution trivially (the inputs are identical to the test input) and uses random labels (mapping ablated), but again performed significantly worse than most other variants. The authors attribute this to the format being distorted: when all inputs are identical, the input effectively becomes part of the example separator, similar to the constant-label case.

3.4.2 Direct and Channel Inference Methods

The paper uses two distinct inference methods for every (model, dataset, variant) combination, following the taxonomy from Min et al. (2021a):

Direct method. The standard approach: compute the probability (or logit) that the language model assigns to each candidate label y ∈ C given the demonstration string concatenated with the test input, and select the label with the highest probability:

$y^* = \arg\max_{y \in C} P(y \mid x_1, y_1, ..., x_k, y_k, x_{\text{test}})$

where x_1, y_1, ..., x_k, y_k is the demonstration (with whatever variant labels/inputs are specified by the experimental condition), x_test is the test input, and P(y | ...) is the model's autoregressive probability of generating the token(s) corresponding to label y as the continuation of the sequence.

What it computes: For each candidate label in the task's output space, the model scores how likely that label is to appear immediately after the demonstration + test input. The label with the highest score is selected. The scores are raw probabilities from the model's output distribution—no calibration or normalization is applied beyond what the model's softmax provides.

Why this form: This is the natural generative interpretation of a language model performing classification: the model is asked to "complete" the sequence with the correct label. The label space C is typically small (2–5 options for classification, 4–5 for multi-choice), so the method requires |C| forward passes (one per candidate), or can be done in a single forward pass if all tokens are scored simultaneously.

Channel method. The "noisy channel" approach from Min et al. (2021a): flip the conditioning direction. Instead of computing P(label | input), compute P(input | label) using Bayes' rule, assuming a uniform prior over labels:

$y^* = \arg\max_{y \in C} P(x_{\text{test}} \mid x_1, y_1, ..., x_k, y_k, y)$

where the model now scores how probable the test input is given that the label is y, conditioned on the demonstrations. The prior P(y) is assumed uniform, so P(y | x_test) ∝ P(x_test | y) and the argmax is equivalent (though Min et al. (2021a) note that calibration can improve results when label distributions are skewed).

What it computes: For each candidate label, the model estimates how likely the test input's text is, given that the "answer" is that label. If the input "The movie was wonderful" has high probability when conditioned on "positive" and low probability when conditioned on "negative", the channel method selects "positive". This inverts the usual generative direction.

Why this form: Min et al. (2021a) found that the channel method often outperforms the direct method for classification tasks, especially with imbalanced or ambiguous label sets. The intuition is that language models are better at evaluating the probability of a full input sentence conditioned on a simple label (an easier language modeling task) than at predicting a single label token from a long context (where the label token probability competes with many other plausible continuations). The channel method also naturally handles cases where the labels are multi-token strings without requiring the model to generate them autoregressively.

Interaction with the demonstration variants. The choice of inference method interacts nontrivially with which demonstration components matter. The direct method requires the model to generate into the label space—the model must output one of the tokens from C as its prediction. Consequently, the direct method is sensitive to whether the label space is correctly specified in the demonstrations: seeing "positive" and "negative" in the demonstrations primes the model to consider those specific tokens as candidate outputs. The channel method, by contrast, conditions on the label rather than generating it, so the model never needs to output label tokens—it only needs to score input text. This predicts (and the results confirm) that the label space matters more for direct models than channel models, while the input distribution matters more for channel models than direct models (since the channel method's entire computation is about scoring the input text's probability under each candidate label).

Both methods are used without additional calibration techniques (contextual calibration from Zhao et al., 2021, or domain-conditional calibration from Holtzman et al., 2021, would be natural extensions but are not explored in the ablation context—the paper wants to measure the raw effect of demonstration components without calibration confounds).

3.4.3 Model Zoo and Meta-Training Context

The paper evaluates 12 language model configurations, spanning six distinct model architectures/checkpoints times two inference methods (direct and channel):

GPT-2 Large (Radford et al., 2019): 774M parameters, publicly available weights, no meta-training with an in-context learning objective. This serves as the base non-meta-trained model at moderate scale. GPT-2 Large was pretrained on web text (WebText corpus) with a standard causal language modeling objective. Its in-context learning ability is emergent from pretraining alone—it was never explicitly trained to condition on demonstration examples.

MetaICL (Min et al., 2021b): Also 774M parameters (initialized from GPT-2 Large), but subsequently meta-trained on a large collection of supervised datasets with an explicit in-context learning objective. The meta-training procedure works as follows: for each meta-training dataset, the model is presented with sequences of the form "(x_1, y_1) ... (x_m, y_m) (x_test, y_test)" where the first m examples serve as in-context demonstrations and the model is trained to predict y_test given the preceding context. This is done across many diverse tasks (the paper ensures evaluation datasets do not overlap with meta-training datasets), teaching the model to expect the in-context learning format and to extract task-relevant information from demonstrations. MetaICL is the only meta-trained model in the set and exhibits qualitatively different behavior (near-zero sensitivity to the input-label mapping, strong reliance on format).

GPT-J (Wang and Komatsuzaki, 2021): 6B parameters, publicly available, no meta-training. A larger non-meta-trained model based on the GPT architecture, trained on the Pile dataset (Gao et al., 2021). GPT-J represents the scale at which in-context learning becomes reliably above-random for classification tasks.

fairseq 6.7B and fairseq 13B (Artetxe et al., 2021): Dense decoder-only models trained by Meta AI, publicly released. These represent the largest publicly available dense LMs at the time the experiments were conducted. No meta-training for in-context learning.

GPT-3 (Brown et al., 2020): The 175B Davinci base model (not the Instruct version), accessed via the OpenAI API. The authors "assume it to be 175B, following Gao et al. (2021) and Artetxe et al. (2021)." GPT-3 is the largest model tested and the one for which in-context learning was originally popularized. No meta-training beyond whatever mixture of formats appeared in the pretraining corpus.

Evaluation scope by model size. Due to computational constraints, the largest models are evaluated on a subset of datasets. The first eight configurations (GPT-2 direct/channel, MetaICL direct/channel, GPT-J direct/channel, fairseq 6.7B direct/channel) are evaluated on all 26 datasets (16 classification, 10 multi-choice). The fairseq 13B and GPT-3 configurations are evaluated on 6 datasets: 3 classification (MRPC, RTE, Tweet_eval-hate) and 3 multi-choice (OpenbookQA, CommonsenseQA, COPA). This reduced evaluation set is noted explicitly in Section 3 and means that direct numerical comparisons between the full 26-dataset averages and the 6-dataset subset are not apples-to-apples. Figure 11 in the appendix provides the 6-dataset numbers for all models to enable comparable cross-model comparisons.

Why this model selection matters. The range of model sizes (774M to 175B, a ~200× span) allows the paper to test whether the insensitivity to label correctness is an artifact of small models or holds at scale. The inclusion of MetaICL alongside its identically-sized base model (GPT-2 Large) provides a controlled comparison of meta-training's effect while holding architecture and parameter count constant. The public/private split (five public models, one API-only) constrains reproducibility but the public models cover the full range of sizes except the extreme 175B scale.

3.4.4 Dataset Selection and Evaluation Protocol

The 26 datasets. The evaluation suite spans six task categories, chosen because they are "true low-resource datasets with less than 10K training examples," include well-studied GLUE and SuperGLUE benchmarks, and cover diverse domains. The full list (from Table 2):

• Sentiment analysis (2 datasets): financial_phrasebank (Malo et al., 2014), poem_sentiment (Sheng and Uthus, 2020)
• Paraphrase detection (2 datasets): medical_questions_pairs (McCreery et al., 2020), glue-mrpc (Dolan and Brockett, 2005)
• Natural language inference (5 datasets): glue-wnli (Levesque et al., 2012), climate_fever (Diggelmann et al., 2020), glue-rte (Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), superglue-cb (de Marneffe et al., 2019), sick (Marelli et al., 2014)
• Hate speech detection (6 datasets): hate_speech18 (de Gibert et al., 2018), ethos-national_origin, ethos-race, ethos-religion (all Mollas et al., 2020), tweet_eval-hate, tweet_eval-stance_atheism, tweet_eval-stance_feminist (Barbieri et al., 2020)
• Question answering (5 datasets): quarel (Tafjord et al., 2019a), openbookqa (Mihaylov et al., 2018), qasc (Khot et al., 2020), commonsense_qa (Talmor et al., 2019), ai2_arc (Clark et al., 2018)
• Sentence completion (4 datasets): codah (Chen et al., 2019), superglue-copa (Gordon et al., 2012), dream (Sun et al., 2019), quartz-with_knowledge, quartz-no_knowledge (Tafjord et al., 2019b)

The term "classification" in the paper refers to tasks with a fixed label space where each test example maps to exactly one label (sentiment, NLI, hate speech, paraphrase detection—16 datasets). "Multi-choice" refers to tasks where each test example has a set of candidate answers and the model must select one (QA and sentence completion—10 datasets). The distinction matters because the label spaces behave differently: classification labels are a fixed vocabulary ({positive, negative, neutral}), while multi-choice "labels" are contentful phrases that vary per question ({water, sand, rock, feet}).

Evaluation data splits. The paper uses the development sets for evaluation following Ye et al. (2021), not the test sets. The training sets (sizes in Table 2) are used only for sampling demonstration examples; the models are never fine-tuned. For each dataset, k = 16 demonstration examples are sampled uniformly from the training data without any selection strategy (no similarity-based retrieval, no mutual information maximization—pure random sampling). This is an important design choice: more sophisticated demonstration selection methods (Liu et al., 2021; Rubin et al., 2021) could potentially make the input-label mapping more informative by choosing examples that better illustrate the task boundaries, but the paper deliberately uses random sampling to avoid conflating selection quality with the intrinsic value of the mapping.

Why k = 16: The paper uses 16 examples "by default for all experiments in the paper, unless otherwise specified." The choice is motivated by providing sufficient demonstration material for the model to extract input distribution and label space information without making inference impractically long. Figure 5 shows ablations with k ∈ {0, 4, 8, 16, 32}, confirming that the random-label results are consistent across k values (the gap between gold and random labels stays within 0.8–1.6% absolute for all k ≥ 8). The fact that performance plateaus after k = 8 for both gold and random labels is itself an interesting finding: it suggests the additional demonstrations beyond a small number do not provide incrementally useful information, which is inconsistent with a "learning from supervised examples" model (where more data should monotonically help) but consistent with a "extracting distributional statistics" model (where a handful of examples suffices to characterize the input domain and output vocabulary).

Replication and seed control (5 random seeds, except 3 for larger models). Five different random seeds are used to sample the k demonstration examples, and experiments are run 5 times per (dataset, model, variant) combination. For fairseq 13B and GPT-3, only 3 seeds are used "due to limited resources." The seed governs which specific training examples are selected for the demonstrations—different seeds yield different input sets and (in the random-label variants) different label assignments. The per-dataset metric is averaged over seeds first, then macro-averaged across datasets. This two-level averaging ensures that a single unusually easy or hard demonstration sample does not dominate the results.

Metrics. Classification tasks use Macro-F1 because several datasets have imbalanced classes (e.g., hate speech detection datasets may have 80% non-hate, 20% hate). Macro-F1 computes the F1 score for each class independently and then averages, giving equal weight to minority and majority classes. Multi-choice tasks use Accuracy (fraction of test examples where the selected answer matches the ground truth). Both metrics are computed per-dataset and then macro-averaged across datasets—meaning each dataset contributes equally to the final reported number, regardless of test set size.

Templates: minimal vs. manual. The default format uses minimal templates taken from Ye et al. (2021), Min et al. (2021b), and Logan IV et al. (2021). These templates are as simple as possible: for classification, the input is presented as-is followed by the label word; for NLI, the premise and hypothesis are concatenated with a [SEP] token; for QA, the question is presented followed by the answer choices. Table 3 in the appendix provides examples. The paper also explores manual templates (Section 4.2, Figure 6), which are dataset-specific human-written prompts taken from Holtzman et al. (2021) and Zhao et al. (2021), e.g., "The question is: {premise} True or False? The answer is: {True|False}" instead of the minimal "sentence1: {premise} [SEP] sentence2: {hypothesis} \n {entailment|not_entailment}". The random-label results hold under both template styles (Figure 6), with manual templates sometimes outperforming and sometimes underperforming minimal templates—there is no consistent template quality advantage, which is why the main experiments use the simpler minimal versions.

3.4.5 The Four-Aspect Decomposition Framework

The paper's analytical contribution is a decomposition of the demonstration signal into four independent aspects, as depicted in Figure 7. This section formalizes how each aspect is defined and how its contribution is measured.

Aspect 1: The input-label mapping. This is whether each specific input x_i in the demonstrations is paired with its correct label y_i as defined by the dataset's ground truth. If this were the primary signal, the model would learn a function f: X → Y by observing k labeled examples, analogous to supervised learning. The mapping is ablated by replacing gold labels with random labels from C (Mode 2), which preserves the input distribution (the x_i are still from the task), the label space (the \tilde{y}_i are still members of C), and the format, but destroys the x_i → y_i correspondence.

Measuring the mapping contribution: Compare Mode 1 (gold labels) to Mode 2 (random labels). The difference Acc(Mode 1) - Acc(Mode 2) is the marginal value of correct input-label pairings given that the input distribution, label space, and format are already provided.

Aspect 2: The distribution of the input text. This is whether the x_i in the demonstrations are drawn from the same underlying distribution as the test inputs. If this matters, it would be because the model uses the demonstration inputs to calibrate its expectations about input length, style, vocabulary, and domain—essentially answering the question "what kind of text am I about to see?" The input distribution is ablated by replacing the training inputs with out-of-distribution sentences from CC-News (Mode 3), which preserves the label space, the format, and (in combination with random labels) ablates the mapping.

Measuring the input distribution contribution: Compare Mode 2 (random labels, in-distribution inputs) to Mode 3 (random labels, OOD inputs). The difference Acc(Mode 2) - Acc(Mode 3) is the marginal value of in-distribution inputs given that the label space and format are provided but the mapping is disabled. This comparison is specifically chosen to measure the input distribution's effect in isolation—not confounded by the mapping, which is already disabled in both conditions.

Aspect 3: The label space. This is whether the labels y_i shown in the demonstrations are members of the correct output vocabulary C for the task. If this matters, it would be because the model uses the demonstration labels to learn "which tokens am I supposed to output?" for classification, or more subtly, "what is the semantic category of the answer?" (e.g., seeing {positive, negative, neutral} indicates a sentiment task; seeing {entailment, contradiction, neutral} indicates NLI). The label space is ablated by replacing task labels with random English words (Mode 4), preserving the input distribution, the format, and (in combination with random assignment) ablating the mapping.

Measuring the label space contribution: Compare Mode 2 (random labels from C) to Mode 4 (random English words). The difference Acc(Mode 2) - Acc(Mode 4) is the marginal value of the correct output vocabulary given that the input distribution and format are provided but the mapping is disabled.

Aspect 4: The format. This is the structural convention of presenting inputs and labels as paired sequences with a consistent separator pattern. The format serves as a metalinguistic cue that tells the model "these are input-label pairs; the last incomplete pair is what you should complete." Without this format, the model might interpret the concatenation as a continuous passage rather than a task specification. The format is ablated by presenting either only the inputs without labels, or only the labels without inputs (Mode 5 variants).

Measuring the format contribution: There is no single comparison that isolates the format because the format is what allows the other aspects to be communicated. Instead, the paper demonstrates the format's importance by showing that removing it (comparing Mode 4 to "labels only," or Mode 3 to "no labels") causes performance to collapse to or below the no-demonstrations baseline, even when the other components (label space alone, or input distribution alone) would be sufficient with the format intact.

The comparison tables in Figures 8–10. Each figure's bottom panel contains a truth table showing which aspects are intact (filled circle, ●) versus ablated (empty circle, ○) for each variant. For example, in Figure 8:

• "Gold labels" (■): format ●, label space ●, input distribution ●, mapping ●
• "Random labels" (■): format ●, label space ●, input distribution ●, mapping ○
• "OOD + Random labels" (■): format ●, label space ●, input distribution ○, mapping ○
• "No demonstrations" (■): format ○, label space ○, input distribution ○, mapping ○

The measurement logic is: the gap between the first and second bars isolates the mapping; the gap between the second and third bars isolates the input distribution (when mapping is already disabled); and so on through the variant chain.

Why this factorial decomposition rather than independent ablation of each aspect? A fully crossed design (2×2×2×2 = 16 variants) would independently toggle each of the four aspects and could measure all interactions. The paper instead uses a nested ablation approach where aspects are disabled cumulatively. This keeps the number of experimental conditions manageable (5–7 variants rather than 16) and reflects the practical reality that some combinations are unnatural or impossible: you cannot have the input-label mapping without the label space (the mapping needs labels to exist), and you cannot have the format include pairing without having both inputs and labels present. The chosen variant chain—disable mapping → disable input distribution → disable label space → disable format—provides a monotonic progression where each step removes one additional source of information.

3.4.6 Controlled Ablation Variants in Detail

This section consolidates the precise construction of each variant, including implementation details from the appendices that are essential for reproducibility.

Demonstrations with gold labels (baseline, all aspects intact).

Construction: Sample k = 16 examples uniformly without replacement from the training split of the dataset. For each sampled example (x_i, y_i), format as "{x_i} \n {y_i}" (using the model-appropriate separator; here \n represents the separator convention). Concatenate all k formatted pairs with the inter-example separator (e.g., triple newline for MetaICL/GPT-J/GPT-3, single newline for fairseq, space for GPT-2). Append the test input as "{x_test} \n" (the incomplete pair that the model should complete). The resulting string length varies by dataset but is typically a few hundred to a few thousand tokens.

Why k = 16: Chosen empirically as providing stable performance without excessive inference cost. Figure 5 shows that accuracy is relatively flat from k = 8 to k = 32, so 16 is safely in the plateau region.

Demonstrations with random labels (Mode 2, mapping ablated).

Construction: Same as gold labels, but for each sampled x_i, sample \tilde{y}_i uniformly from C (the set of possible labels for the task) independent of x_i and y_i. The sampling is with replacement across the k positions, so the same random label may appear multiple times. The inputs remain the correct in-distribution examples.

Uniform vs. true distribution sampling (Appendix C.2): The default uses uniform sampling over C, meaning each label is equally likely regardless of its frequency in the training data. An alternative uses the empirical label distribution from the training set: P(\tilde{y} = c) = count(c) / |training|. This ensures the random labels reflect the same class balance as the gold labels, removing a potential confound where the model picks up on label frequency as a cue. The true-distribution variant slightly reduces the performance gap relative to gold labels (1.9% → 1.3% for Channel MetaICL, 5.0% → 3.5% for Channel GPT-J), indicating that label distribution information provides a small additional signal beyond the label vocabulary itself.

Edge case: what if the random label happens to equal the gold label? The paper does not explicitly prevent this in the default random labels condition. For the a% correct labels variant (Algorithm 1), incorrect pairs are explicitly constructed by sampling from C \ {y_i} (the label space excluding the correct label), ensuring genuine incorrectness. The default random labels condition (Mode 2) uses uniform sampling from the full C, meaning there is a 1/|C| chance that the random label accidentally matches the gold label. For a ternary classification task (|C| = 3), roughly 33% of the random labels would be "correct by accident." The paper's reported gap between gold and random labels (0–5%) thus overestimates the value of the mapping: some of the random-label performance is attributable to accidentally correct pairings. The true-distribution variant partially controls for this but does not eliminate it.

OOD demonstrations with random labels (Mode 3, mapping + input distribution ablated).

Construction: For each of the k positions, sample a sentence x_{i,rand} from CC-News (Nagel, 2016) instead of from the training data. The sampling procedure length-matches: it selects sentences whose character/word length is within a window around the test input's length, ensuring that the OOD inputs are not systematically shorter or longer. Each x_{i,rand} is paired with a random label \tilde{y}_i sampled uniformly from C. Formatting follows the same conventions as the in-distribution modes.

Why CC-News? CC-News is a large corpus of English news articles extracted from Common Crawl. It provides generic, semantically diverse sentences that are unlikely to overlap in topic or style with specialized datasets like financial_phrasebank (financial news sentiment) or poem_sentiment (poetry). Using news text as the OOD source is a conservative choice—if the model derives substantial benefit from even unrelated English text, that indicates the input distribution's contribution is about being English text in a similar length range rather than about domain-specific content.

Length matching implementation: The paper states that "the length of the text [is considered] during sampling, so that sampled sentences have similar length to the test input" but does not provide the exact window size or matching algorithm. This is a methodological detail whose absence is a minor limitation—length mismatch could confound the results if OOD inputs are systematically longer (increasing the distance between the demonstration and the test input, which is known to degrade in-context learning performance; Liu et al., 2021).

Demonstrations with random English words (Mode 4, mapping + label space ablated).

Construction: First, construct C_rand, a set of |C| English words sampled uniformly without replacement from the english-words Python package (which contains 61,569 words). This set has the same cardinality as the original label space but contains arbitrary English words with no task-specific semantics. For each training input x_i, sample \tilde{y}_i uniformly from C_rand (with replacement across the k positions). Format as "{x_i} \n {\tilde{y}_i}" as usual.

Why |C_rand| = |C|? Matching the cardinality controls for a potential confound: if C_rand were smaller or larger than C, the model might pick up on the number of distinct labels as a cue. By keeping cardinality equal, the only difference between Mode 2 (random labels from C) and Mode 4 (random English words) is the semantic content of the label tokens—both use the same number of distinct labels, distributed uniformly across the k examples.

Why the english-words package? It provides a comprehensive list of English words spanning common and rare vocabulary. The key property is that the sampled words are recognizable English tokens (which the LM has encountered during pretraining and can process normally), but they carry no information about the task. If performance with random English words is nearly as good as with random labels from C, it means the label space contributes nothing beyond "the labels are valid English words." If performance drops, it means the model benefits from knowing which specific words constitute the output vocabulary.

Demonstrations with no labels (inputs only, format ablated).

Construction: Format k training inputs as "x_1 \n\n x_2 \n\n ... \n\n x_k \n\n x_test \n" (using the model-appropriate separators). No labels appear anywhere in the demonstration. The model must predict the label from the test input alone, though it has been primed with k examples of in-distribution text. This variant measures how much of in-context learning's gain comes purely from exposure to the input domain, without any label or pairing structure.

Demonstrations with labels only (format ablated).

Construction: Format k gold labels as "y_1 \n\n y_2 \n\n ... \n\n y_k \n\n x_test \n" (using the model-appropriate separators). No inputs appear in the demonstrations except the test input. The model sees only a list of valid labels before being asked to make a prediction. This variant measures how much of the gain comes purely from knowing the label vocabulary, without any demonstration inputs or pairing structure.

Demonstrations with a% correct labels (Algorithm 1, graded mapping signal).

Construction: Given a target accuracy a ∈ {0, 25, 50, 75, 100} and k = 16 training examples:

1. Compute n = k × a / 100 — the number of demonstrations that should have correct labels. Since k = 16 and a is a multiple of 25 (except 0 and 100), n is always an integer (0, 4, 8, 12, 16).
2. Sample n distinct indices uniformly from {1, ..., k} — these positions will get correct labels.
3. For i in the sampled set: append (x_i, y_i) to the demonstration — the gold pair.
4. For i not in the sampled set: append (x_i, Sample(C \ {y_i})) to the demonstration — the correct input paired with a guaranteed incorrect random label (sampled from C excluding the true label).
5. The k pairs are then concatenated in their original order (with indices 1 through k), so correct and incorrect pairs are interleaved rather than grouped.

Why guaranteed-incorrect for the wrong pairs? This ensures that the model receives a clean signal: n demonstrations where the label is correct, k - n where it is definitely wrong. If incorrect pairs were sampled uniformly from C (allowing accidental correctness), the effective number of correct pairs would be n + (k-n)/|C| in expectation, blurring the intended manipulation. The explicit exclusion C \ {y_i} avoids this confound.

What this variant tests that Mode 2 doesn't: Mode 2 replaces all labels with random ones, testing whether the mapping matters at all. The a% variant tests whether there's a dose-response relationship—does performance improve monotonically as the proportion of correct labels increases? If yes, the model is extracting graded information from the mapping even if the all-random case performs surprisingly well. If performance is flat across a, the model truly ignores label correctness.

Discussion section variants (Appendix C.3, explored but not emphasized).

Constant label variant: Every x_i is paired with the literal token "answer". The demonstration becomes "x_1 \n answer \n\n x_2 \n answer \n\n ... \n\n x_test \n". This ablates the label space while preserving the pairing format and input distribution, but differs from Mode 4 (random English words) in that all labels are identical. Performance is worse than Mode 4, hypothesized to be because "answer" functions as a separator rather than a label, effectively breaking the pairing format.

Test-input-as-all-inputs variant: Every demonstration example uses x_test paired with a random label. The demonstration becomes "x_test \n \tilde{y}_1 \n\n x_test \n \tilde{y}_2 \n\n ... \n\n x_test \n". This trivially satisfies the "inputs from the same distribution as the test input" condition (they are identical), but performs worse than Mode 2 or Mode 3. The hypothesis is that identical inputs disrupt the format because the model perceives the input as a fixed separator rather than variable examples.

These exploratory variants are methodologically important because they show that the format is fragile—minor changes that technically preserve the "input-label pair" structure can break performance if they cause the model to parse the structure differently.

3.4.7 The Channel Method's Bayesian Reformulation (Expanded)

The channel method introduced by Min et al. (2021a) and used throughout this paper inverts the prediction direction relative to the standard direct method. For completeness, I expand on the formalism:

Direct method prediction:

$y^*_{\text{direct}} = \arg\max_{y \in C} \log P_{\text{LM}}(y \mid \text{demos}, x_{\text{test}})$

where P_LM is the language model's autoregressive probability for the token(s) corresponding to label y. This is computed by feeding the concatenation [demos, x_test, y] to the model and extracting the log-probability of the label tokens.

Channel method prediction:

$y^*_{\text{channel}} = \arg\max_{y \in C} \log P_{\text{LM}}(x_{\text{test}} \mid \text{demos}, y)$

where the model now computes the probability of the entire test input conditioned on the label and demonstrations. This is computed by feeding [demos, y, x_test] to the model and extracting the log-probability of the test-input tokens. The demos in both cases are the k input-label pairs formatted according to the variant being tested.

Why the channel method is relevant to this paper's analysis: The two methods emphasize different components of the demonstration. The direct method conditions on x_test and must generate a label token, making it sensitive to whether the label space C is correctly specified (the model needs to know which tokens are candidate outputs). The channel method conditions on y and must explain x_test, making it sensitive to whether the input distribution is correctly specified (the model needs to know what kind of text is plausible). This asymmetry predicts—and the results confirm—that:

• Label space matters more for direct models (5–16% drop when switching from random labels to random English words in Figure 9, direct panel).
• Input distribution matters more for channel models (3–16% drop when switching from random labels to OOD+random labels in Figure 8, channel panel).
• The input-label mapping matters little for both but slightly less for channel MetaICL than direct GPT-J.

This interaction validates the decomposition framework: each aspect of the demonstration feeds a different computational pathway in the model, and the pathway that gets used depends on the inference method.

Channel method and multi-choice tasks: For multi-choice tasks where the "label" is a contentful phrase (e.g., "water" as an answer to "What creates a valley?"), the channel method computes P(x_test | y) where x_test is the question and y is the candidate answer. The Bayes' rule interpretation with a uniform prior over answers yields P(y | x_test) ∝ P(x_test | y), so the argmax over y is equivalent. This is the same scoring mechanism—the model evaluates how "natural" the question looks when it already knows the answer is y.

3.4.8 Summary of Factorial Logic and Measurement Strategy

The experimental design can be summarized as a chain of progressive ablations, where each step disables one additional component and the performance difference between adjacent steps measures that component's marginal contribution:

Gold labels (all intact)
   |___ Mapping contribution = Acc(Gold) - Acc(Random labels)
Random labels (mapping disabled, input dist + label space + format intact)
   |___ Input distribution contribution = Acc(Random labels) - Acc(OOD + Random labels)
OOD + Random labels (mapping + input dist disabled, label space + format intact)
   |___ Label space contribution = Acc(OOD + Random labels) - Acc(Random English words)*
Random English words (mapping + label space disabled, input dist + format intact)
   |___ Format contribution = Acc(Random English words) - Acc(No labels or Labels only)
No labels / Labels only (format disabled)

*Note: The label space contribution is measured slightly differently in the paper—between Random labels and Random English words—because both preserve the input distribution. The OOD+Random labels variant (input distribution ablated) is used specifically to isolate the input distribution's effect, not the label space's.

The key methodological insight is that comparisons are always made between variants that differ in exactly one aspect, holding the others constant. This is why the paper can attribute specific numerical ranges to each component (e.g., "3–16% absolute" for the input distribution, "5–16% absolute" for the label space in direct models). Without this controlled ablation structure, it would be impossible to distinguish whether performance drops come from losing the mapping, the input domain, or the label vocabulary—they would be confounded.

The format's contribution is harder to quantify as a single number because removing the format (the pairing structure) fundamentally changes how the other components can be communicated. The paper instead demonstrates the format's necessity by showing that no combination of other components without the format achieves performance above the no-demonstrations baseline, while the same components with the format intact achieve substantial gains. This is a qualitative rather than quantitative finding, but it is robust across all models and task types.

4. Key Insights and Innovations

Innovation 1: The Demonstrations Are Not Teaching the Task — They Are Specifying Where to Look

The paper's most disruptive contribution is a fundamental reframing of what in-context learning is. The field's operating assumption — rarely stated explicitly but embedded in all prior work on demonstration selection and optimization — was that the model learns new input-label correspondences from the provided examples, analogous to supervised learning over a miniature training set. This paper demonstrates that this assumption is wrong, and wrong in a precise, measurable way: the input-label mapping, the very thing everyone thought was the active ingredient, contributes at most 0–5% absolute accuracy across 12 models and 26 datasets (Figure 3).

What makes this a conceptual innovation rather than merely a surprising empirical result is the alternative framing it enables. If the model does not learn the mapping from the demonstrations, then the demonstrations must serve a different function entirely. The paper proposes — and validates through systematic ablation — that this function is task location: the demonstrations specify which latent capability the model should deploy, not how to perform the task. The model already possesses the input-label correspondence from pretraining (having been exposed to language where positive sentiment co-occurs with words like "positive," or where textual entailment relationships are implicitly modeled); the demonstrations simply tell it "activate the sentiment analysis behavior" rather than "here is how to classify sentiment."

This is a fundamental reframing, not an incremental refinement. Prior work — including Reynolds and McDonell (2021), who argued that demonstrations are for "task location" — treated this as a speculative hypothesis without systematic evidence. The current paper provides that evidence through the factorial ablation design, showing precisely which components of the demonstration serve the location function (input distribution, label space, format) and which do not (the mapping). The distinction between "learning a new task" and "locating an existing capability" has cascading implications for when in-context learning should work (tasks resembling pretraining data), when it should fail (genuinely novel tasks with no pretraining precedent), and how practitioners should construct prompts (prioritize clear task specification over correct example labels). The paper crystallizes this in the Discussion (Section 6):

"If we take a strict definition of learning: capturing the input-label correspondence given in the training data, then our findings suggest that LMs do not learn new tasks at test time. [...] However, learning a new task can be interpreted more broadly: it may include adapting to specific input and label distributions and the format suggested by the demonstrations, and ultimately getting to make a prediction more accurately."

This dual definition of "learning" — strict (acquiring new correspondences) versus broad (adapting to distributional cues) — is the paper's key conceptual move. It resolves the apparent paradox that in-context learning does improve over zero-shot (the model is learning something) while remaining insensitive to label correctness (that something is not the input-label mapping). For practitioners, this implies that collecting high-quality labeled examples for few-shot prompting may be less important than ensuring the prompt format matches the model's pretraining distribution, and that the reliability of in-context learning is bounded by whether the task's semantics were already captured during pretraining — a finding with direct implications for deployment risk assessment.

Innovation 2: The Four-Aspect Decomposition as a Diagnostic Framework

Prior work treated demonstrations as monolithic: the question was "which demonstrations work best?" (Liu et al., 2021; Lu et al., 2021; Rubin et al., 2021), implicitly treating the entire (input, label) pair as an atomic unit of signal. The paper decomposes this unit into four independent aspects — the input-label mapping, the distribution of the input text, the label space, and the format — and provides a controlled experimental apparatus for measuring each aspect's marginal contribution.

This decomposition is methodologically innovative in a way that transcends the specific results. It provides a diagnostic toolkit that any researcher can apply to their own models, tasks, or prompt designs. The comparison tables at the bottom of Figures 8–10 are a template for future work: for any new in-context learning variant, one can ask which of the four aspects it affects and measure the performance delta accordingly. This transforms the qualitative question "what makes demonstrations work?" into the quantitative question "how much does each aspect contribute?" — a shift from speculation to measurement.

The framework also reveals asymmetric dependencies between inference methods and demonstration components that were previously invisible. The finding that the label space matters substantially for direct models (5–16% drop when using random English words; Figure 9, direct panel) but negligibly for channel models (0–2% drop; Figure 9, channel panel) — while the input distribution shows the opposite pattern — provides a principled explanation for why channel models sometimes outperform direct models (Min et al., 2021a): they draw signal from a different component of the demonstration (the input distribution, which is always available from the training data) while being insensitive to a component that might be noisy (the label space, which could be incomplete or ambiguous in some tasks). This is not an empirical curiosity but a structural insight about how different inference methods interface with demonstration information, enabling informed method selection based on task properties (e.g., use channel when label space is ambiguous, use direct when input distribution is OOD relative to pretraining).

Innovation 3: Meta-Training Amplifies Reliance on Superficial Cues — A Cautionary Finding

The inclusion of MetaICL (Min et al., 2021b) — a model explicitly trained to perform in-context learning across many tasks — yields one of the paper's most striking and counterintuitive results: the meta-trained model is less sensitive to the input-label mapping than non-meta-trained models, not more. The performance gap between gold and random labels is 0.1–0.9% absolute for MetaICL versus 2.6% average for other models (Section 4.1). Even more tellingly, Direct MetaICL shows near-zero sensitivity to both the input-label mapping and the input distribution (Figure 8, top-left panel) — it achieves strong performance almost entirely from the label space and format alone.

This is a negative result with significant implications. The intuitive expectation for meta-training would be that training a model explicitly on the in-context learning objective teaches it to extract more signal from demonstrations — including the input-label mapping. The paper's finding suggests the opposite: meta-training teaches the model to exploit simpler, more readily available cues (the format, the label vocabulary) and to ignore the harder-to-extract input-label correspondence. The authors hypothesize:

"meta-training encourages the model to exclusively exploit simpler aspects of the demonstrations and to ignore others. This is based on our intuition that (1) the input-label mapping is likely harder to exploit, (2) the format is likely easier to exploit, and (3) the space of the text that the model is trained to generate is likely easier to exploit than the space of the text that the model conditions on."

This is a fundamental insight about the nature of meta-learning objectives, not an incremental finding. It suggests that when we train models to "learn from examples," they may optimize a shortcut: use the surface structure of the examples to identify the task, rather than extracting the deeper relational information the examples contain. This connects to broader concerns about shortcut learning in deep learning (Geirhos et al., 2020) and has practical implications: augmenting models with meta-training may improve average-case performance while making them less capable of genuinely adapting to novel input-label relationships, because they have learned to rely on the distributional and formatting cues that correlate with task identity in the meta-training distribution. This finding complicates the narrative that meta-training is an unqualified improvement for few-shot generalization and suggests that future work should measure not just aggregate accuracy but also sensitivity to the input-label mapping as a diagnostic for whether the model is truly "learning from examples" or merely pattern-matching against format cues.

Innovation 4: The Format as the Essential Scaffold — In-Context Learning Without Either Labels or Inputs

Among the four aspects, the format — the structural convention of presenting inputs and labels as paired sequences — emerges from the ablation analysis as qualitatively different from the others. While the input distribution and label space contribute additive performance gains (3–16% and 5–16% respectively), the format is multiplicatively essential: without it, no combination of other components achieves performance above the no-demonstrations baseline. Conversely, with the format intact, it is possible to retain up to 95% of in-context learning's improvement using demonstrations that provide only the format plus one other component (either inputs with random English words, conveying the input distribution but not the label space, or OOD text with valid labels, conveying the label space but not the input distribution). Figure 10's comparison tables show this dramatically: the variants that preserve the format (random English words, OOD+random labels) substantially outperform the no-demonstrations baseline, while the format-ablated variants (no labels, labels only) perform at or below it.

This is a novel empirical finding about the structure of few-shot prompting that had no precedent in the literature. Prior work had observed format sensitivity (Zhao et al., 2021; Lu et al., 2021) but had not isolated the format as an independent factor whose presence enables the other components to function. The paper's contribution is establishing that the format is not merely "helpful" but is the scaffolding that makes all other demonstration signal usable. The specific numerical result — that pairing random English words with in-distribution inputs preserves 75–95% of in-context learning gains for channel models (Figure 10) — is striking because it means the model can determine what kind of task to perform almost entirely from the structural convention of paired presentation, even when the labels are semantically uninformative. For practitioners, this implies that getting the format right (separators, ordering, template consistency) is more critical than curating high-quality examples — a bad format breaks everything, while a good format enables surprisingly effective performance from nearly content-free demonstrations.

The paper's finding also connects in-context learning to the instruction-following paradigm (Wei et al., 2022a; Sanh et al., 2022) in a concrete way: both the format of few-shot demonstrations and natural language instructions serve as task specification mechanisms that tell the model which latent capability to activate, rather than teaching it a new capability. The demonstrations' format — the structural pattern of input-label pairs — functions as an implicit instruction, as the authors note: "the demonstrations and instructions largely have the same role to LMs." This insight, partially corroborated by Webson and Pavlick (2022) who found that irrelevant or misleading instructions do not substantially degrade instruction-following performance, suggests that in-context learning and instruction-following are two surface variants of the same underlying mechanism: the model uses whatever specification channel is provided (demonstration format or natural language instruction) to locate the appropriate pretrained behavior.

Innovation 5: The Zero-Shot Baseline Was Much Higher Than We Thought

A methodological consequence of the paper's findings is the implicit redefinition of what "zero-shot" performance means. If random labels paired with in-distribution inputs perform nearly as well as gold labels, then a system using only unlabeled training data plus knowledge of the label space can achieve what was previously considered "few-shot" performance. As the authors note:

"it is possible to achieve nearly k-shot performance without using any labeled data, by simply pairing each unlabeled input with a random label and using it as the demonstrations. This means our zero-shot baseline level is significantly higher than previously thought."

This is not a theoretical reframing but a practical calibration insight with immediate implications for how the field reports and compares results. The standard zero-shot baseline — no demonstrations at all, just a prompt and a test input — systematically underestimates what the model can achieve without labeled data. The paper demonstrates that simply adding unlabeled examples with arbitrary labels (the "random labels" condition) closes a significant fraction of the gap between zero-shot and full few-shot performance. This means that comparisons between "zero-shot" and "few-shot" methods in prior work conflate two distinct sources of improvement: the provision of demonstration examples (which specify the input distribution, label space, and format) and the provision of correct labels (which specify the mapping). The paper disentangles these, showing that the former accounts for most of the gain. Future work reporting few-shot improvements should compare against this better zero-shot baseline (demonstrations with random labels) to isolate the genuine contribution of correct label information from the contribution of simply showing the model what the task looks like structurally. This raises the bar for claiming that a method "learns from examples" — it must outperform the random-label baseline, not just the no-demonstration baseline.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on 26 datasets spanning six task categories: sentiment analysis (2 datasets), paraphrase detection (2), natural language inference (5), hate speech detection (6), question answering (5), and sentence completion (4). These include well-studied benchmarks from GLUE (e.g., MRPC, RTE, WNLI) and SuperGLUE (e.g., CB, COPA), plus diverse domain-specific datasets like financial_phrasebank, poem_sentiment, tweet_eval-hate, and openbookqa. All are classification or multi-choice tasks with discrete label spaces. The datasets are "true low-resource datasets with less than 10K training examples" (Section 3), chosen to represent realistic few-shot scenarios. Evaluation uses the development splits following Ye et al. (2021), not held-out test sets. The training splits (sizes ranging from 250 to 9,741 examples; Table 2) are used only for sampling demonstration examples—the models are never fine-tuned.

• Base model(s). Twelve language model configurations are tested, spanning six architectures times two inference methods (direct and channel). The models are: GPT-2 Large (774M parameters; Radford et al., 2019), MetaICL (774M, initialized from GPT-2 Large and meta-trained on a collection of supervised datasets with an explicit in-context learning objective; Min et al., 2021b), GPT-J (6B; Wang and Komatsuzaki, 2021), fairseq 6.7B and fairseq 13B (dense decoder-only models; Artetxe et al., 2021), and GPT-3 Davinci base (assumed 175B; Brown et al., 2020). All are decoder-only, dense language models. The range spans ~200× in parameter count (774M to 175B), enabling analysis of whether findings hold across scale. The first eight configurations (all except fairseq 13B and GPT-3) are evaluated on all 26 datasets; fairseq 13B and GPT-3 are evaluated on a 6-dataset subset (3 classification: MRPC, RTE, Tweet_eval-hate; 3 multi-choice: OpenbookQA, CommonsenseQA, COPA) "due to limited resources" (Section 3). MetaICL is the only meta-trained model and is explicitly chosen to test how meta-training with an in-context learning objective affects sensitivity to demonstration components.

• Metrics. Classification tasks use Macro-F1, computing the F1 score per class and averaging, giving equal weight to majority and minority classes—this is important because several datasets have imbalanced label distributions. Multi-choice tasks use Accuracy (exact match between predicted and ground-truth answer). Both metrics are computed per-dataset first, then macro-averaged across datasets so that each dataset contributes equally regardless of test set size. The paper notes that Macro-F1 is "known to be better for imbalanced classes" (Section 3 footnote).

• Baselines. The paper uses two primary baselines: (1) No demonstrations (zero-shot), where predictions are made via argmax_y P(y | x_test) without any demonstration examples—this is the standard zero-shot baseline against which in-context learning gains are measured throughout; (2) Demonstrations with gold labels, which is standard few-shot in-context learning with k = 16 correctly paired (input, label) examples—this serves as the upper-bound reference for what in-context learning can achieve with full information. The paper does not compare against fine-tuned baselines, retrieval-based demonstration selection methods, or calibration techniques, as the goal is to analyze the in-context learning mechanism itself rather than to maximize absolute performance.

• Generation budget / compute accounting. The paper does not use a "generation budget" in the FLOPs-counting sense seen in test-time compute scaling work. Instead, the relevant resource is the number of demonstration examples k, fixed at 16 by default. All methods (direct and channel) require |C| forward passes per test example (one per candidate label), making the inference cost proportional to the label space size. The paper sweeps k ∈ {0, 4, 8, 16, 32} in Section 4.2 (Figure 5) to test sensitivity to demonstration count, but all other experiments use k = 16. There is no accounting for the computational cost of constructing different demonstration variants (e.g., sampling from external corpora for OOD demonstrations), as this cost is negligible relative to model inference.

• Cross-validation / statistical protocol. Five different random seeds are used to sample the k demonstration examples for each (dataset, model, variant) combination, with experiments run 5 times per condition. For fairseq 13B and GPT-3, this is reduced to 3 seeds due to computational constraints. Performance is averaged over seeds within each dataset, then macro-averaged across datasets. This two-level averaging ensures that results are not driven by a single unusually easy or hard demonstration sample. There is no cross-validation over datasets or data splits—the same development sets are used for all evaluations, and no hyperparameter tuning is performed on held-out data. The demonstration examples are sampled "at uniform from the training data" without any selection strategy, deliberately avoiding the confound of demonstration quality that prior work (Liu et al., 2021; Rubin et al., 2021) explored.

Main Quantitative Results

The Ground Truth Input-Label Mapping Contributes Only Marginally (Section 4.1)

The headline result, shown in Figure 3, is that replacing gold labels with random labels in the demonstrations barely hurts performance across nearly all models and task types. Across all 12 model configurations evaluated on 26 datasets (8 configurations) or 6 datasets (4 configurations), the performance drop from using random labels instead of gold labels is in the range of 0–5% absolute, with smaller drops in multi-choice tasks (1.7% average) than classification tasks (2.6% average). At the same time, using demonstrations with gold labels significantly improves over the no-demonstrations baseline, confirming that in-context learning provides genuine gains—just not through the mechanism of correct input-label pairings.

Breaking this down by model (Figure 3 and Figure 11 for comparable 6-dataset numbers):

• GPT-2 Large (774M, no meta-training): Direct GPT-2 shows modest overall gains from demonstrations but is close to random guessing on many classification datasets (the paper notes these models "are not significantly better than random guessing on many datasets" and are "discuss[ed] less significantly for the rest of analysis"). Channel GPT-2 achieves roughly 45–50% Macro-F1 on classification with gold labels, dropping to roughly 43–48% with random labels—a gap of 2–3% absolute.

• MetaICL (774M, meta-trained): The most striking insensitivity. Across both direct and channel methods and both task types, the gap between gold and random labels is 0.1–0.9% absolute. Direct MetaICL achieves roughly 52–55% Macro-F1 on classification with gold labels and 51–54% with random labels. Channel MetaICL achieves roughly 50–53% with gold and 49–52% with random. The meta-trained model essentially ignores label correctness entirely.

• GPT-J (6B): Shows the largest sensitivity to label correctness among the better-performing models. Direct GPT-J drops by roughly 4–6% absolute on classification (from ~28–30% to ~22–26% Macro-F1 with random labels). Channel GPT-J drops by roughly 3–5% on classification and 1–2% on multi-choice. GPT-J still shows substantial random-label performance well above the no-demonstrations baseline.

• fairseq 6.7B and 13B: fairseq 6.7B shows drops of roughly 2–4% absolute across task types and inference methods. fairseq 13B Channel notably has "significantly better no-demonstrations performance compared to demonstrations with gold labels" in classification (Figure 3, bottom-left panel), an anomalous result that the paper notes but does not explain—it may reflect a mismatch between the model's pretraining distribution and the minimal template format.

• GPT-3 (175B): Evaluated on the 6-dataset subset. Channel GPT-3 achieves roughly 55% Macro-F1 on classification and 63% accuracy on multi-choice with gold labels, dropping to roughly 52% and 61% respectively with random labels—gaps of 2–3% absolute. Direct GPT-3 shows drops of 3–4% absolute. The largest model tested does not exhibit qualitatively different behavior; the insensitivity to label correctness is not a small-model artifact.

Figure 5 shows that this insensitivity is consistent across demonstration counts k. For k = 4, 8, 16, 32, the gap between gold and random labels stays within 0.8–1.6% absolute for Channel MetaICL (classification), Direct GPT-J (multi-choice), and the other well-performing model-task pairs. The gap is slightly larger at k = 4 (4.4% in one case) "likely due to a high variance with a very small value of k." The performance plateau after k = 8 for both gold and random labels (Figure 5) is itself informative: if the model were genuinely learning the input-label mapping from more examples, accuracy should increase with k; the flatness suggests the additional demonstrations are not providing incrementally useful mapping information.

Figure 6 confirms the result holds with manual templates (dataset-specific human-written prompts from Holtzman et al., 2021 and Zhao et al., 2021) as well as minimal templates. The random-label gap remains small (0–4%) under manual templates, and manual templates do not consistently outperform minimal templates—the choice of template format matters less than whether the format is present at all.

The Input-Label Mapping's Contribution Is Not Just All-or-Nothing—It Is Graded but Weak (Section 4.2)

Figure 4 reports the a% correct labels ablation, where the proportion of demonstrations with correct labels is swept from 0% to 100% in 25% increments. The key finding: performance is fairly insensitive to the proportion of correct labels, and using always-incorrect labels (0% correct) still substantially outperforms no demonstrations.

For Channel MetaICL (classification): performance at 100% correct = ~54% Macro-F1; at 75% correct = ~53%; at 50% correct = ~53%; at 25% correct = ~52%; at 0% correct = ~52%. The model preserves roughly 92% of the improvement from in-context learning even when every single label is wrong. For Direct GPT-J (multi-choice): 100% correct = ~44% accuracy; 0% correct = ~42% accuracy—preserving 97% of the improvement. For MetaICL (multi-choice): 100% correct = ~42%; 0% correct = ~41%—preserving 100% of the improvement.

GPT-J (classification) is the notable exception: performance drops more substantially with fewer correct labels (from ~32% Macro-F1 at 100% to ~24% at 0%), though 0%-correct still outperforms no demonstrations by roughly 8% absolute. This is the one model-task pair where the mapping shows non-trivial graded contribution.

Random labels from the true label distribution (Appendix C.2) further narrows the gap: for Channel MetaICL, the gap shrinks from 1.9% (uniform random) to 1.3% (distribution-matched random); for Channel GPT-J, from 5.0% to 3.5%. This indicates that label frequency information—distinct from input-label correspondence—provides a small additional signal.

The Distribution of the Input Text Contributes Substantially (Section 5.1)

Figure 8 reports the OOD demonstrations ablation, where in-distribution training inputs are replaced with sentences from CC-News (a generic news corpus). The comparison isolates the input distribution's contribution: Random labels (Figure 8, blue) vs. OOD + Random labels (Figure 8, green).

The drops are substantial for channel models and Direct GPT-J:

• Channel MetaICL (classification): Drops from ~50% Macro-F1 (random labels, in-distribution inputs) to ~34% (OOD + random labels)—a 16% absolute drop. For multi-choice: drops from ~42% accuracy to ~33%—a 9% drop.
• Channel GPT-J (classification): Drops from ~35% to ~25%—a 10% drop. For multi-choice: drops from ~42% to ~36%—a 6% drop.
• Direct GPT-J (multi-choice): Drops from ~42% to ~38%—a 4% drop. In classification, the drop is ~3%.
• Direct MetaICL: The exception. Performance barely changes (~54% to ~53% in classification; ~42% to ~41% in multi-choice)—the meta-trained direct model shows near-zero sensitivity to the input distribution, consistent with the hypothesis that meta-training encourages reliance on format and label space rather than input content.

For the channel models, the input distribution is often the largest single contributor to in-context learning gains, exceeding both the mapping contribution and (in some cases) the label space contribution.

The Label Space Contributes Substantially for Direct Models but Not Channel Models (Section 5.2)

Figure 9 reports the random English words ablation, where task labels are replaced with arbitrary English words of the same cardinality. The comparison isolates the label space's contribution: Random labels (Figure 9, blue) vs. Random English words (Figure 9, green).

The pattern is starkly asymmetric between direct and channel models:

• Direct MetaICL (classification): Drops from ~54% Macro-F1 (random labels) to ~38% (random English words)—a 16% absolute drop. For multi-choice: drops from ~42% to ~37%—a 5% drop.
• Direct GPT-J (classification): Drops from ~30% to ~22%—an 8% drop. For multi-choice: drops from ~42% to ~34%—an 8% drop.
• Channel MetaICL and Channel GPT-J: Both show drops of only 0–2% absolute—sometimes even a slight increase. The channel models are essentially insensitive to whether the labels are the correct task vocabulary or random English words.

The paper hypothesizes this asymmetry is because "the channel models only condition on the labels, and thus are not benefitting from knowing the label space. This is in contrast to direct models which must generate the correct labels" (Section 5.2). This is a structural insight: the inference method determines which demonstration component the model relies on, with direct models needing the correct output vocabulary and channel models needing the correct input domain.

The paper further notes that the label space effect persists even for multi-choice tasks, where there is "no fixed set of labels" across examples (each question has its own candidate set). The authors hypothesize that "multi-choice tasks still do have a particular distribution of the choices (e.g., objects like 'Bolts' or 'Screws' in the OpenBookQA dataset) that the model uses."

The Format Is Essential—In-Context Learning Without It Collapses (Section 5.3)

Figure 10 presents the format-removal ablations: demonstrations with no labels (inputs only) and demonstrations with labels only. These remove the input-label pairing structure while preserving either the input distribution alone or the label space alone.

The central finding: removing the format collapses performance to at or below the no-demonstrations baseline. Across all models and task types:

• No labels (inputs only, Figure 10, orange): Performance is at or below no demonstrations for all configurations. For Channel GPT-J (multi-choice): drops to ~33%—below the no-demonstrations baseline of ~35%. For Direct MetaICL (classification): ~33% vs. no-demos at ~32%—essentially identical.
• Labels only (Figure 10, dark purple): Similarly at or below no demonstrations. For Channel GPT-J (classification): ~22% vs. no-demos at ~22%. For Channel MetaICL (multi-choice): ~35% vs. no-demos at ~35%.

The format's essential role is further demonstrated by the asymmetric retention patterns when the format is kept:

• With the format intact, it is possible to retain 75–95% of in-context learning gains using only the label space (by pairing OOD inputs with valid labels—the green bars in Figure 10). For Direct MetaICL (classification): OOD + random labels achieves ~53% Macro-F1 vs. ~54% with gold labels—retaining 95% of the gain. For Direct MetaICL (multi-choice): retains 82% of the gain.

• With the format intact, it is possible to retain 82–87% of gains using only the input distribution (by pairing in-distribution inputs with random English words—the purple bars in Figure 10). For Channel MetaICL (classification): random English words achieve ~48% vs. ~50% with gold labels—retaining 82%. For Channel GPT-J (classification): random English words achieve ~33% vs. ~35%—retaining 87%.

• Without the format, the same information is useless. Compare the green bars (OOD + random labels, format intact) to the dark purple bars (labels only, format removed—the no-format counterpart): the format-intact version substantially outperforms the format-removed version for direct models. Compare the purple bars (random English words, format intact) to the orange bars (no labels, format removed—the no-format counterpart): the format-intact version substantially outperforms for channel models.

The paper's summary is precise: "keeping the format plays a significant role in retaining a large portion of performance gains by only using the inputs or only using the labels" (Section 5.3). The format acts as a scaffold that enables the other components to be used. Without it, the model cannot interpret a list of labels or a list of inputs as a task specification.

This finding also explains a pattern visible across Figures 8–10: for every model where one component (input distribution or label space) provides substantial gains when the format is intact, removing the complementary component while keeping the format is significantly better than removing the format while keeping that component. For instance, Direct MetaICL benefits hugely from the label space with the format intact (~54% with random labels) but gets essentially nothing from the label space without the format (~33% with labels only)—the format is what makes the label space usable.

Meta-Training Amplifies Reliance on Superficial Cues (Section 5.4)

Comparing MetaICL to identically-sized GPT-2 Large reveals systematic differences attributable to meta-training:

• The input-label mapping matters even less for MetaICL (0.1–0.9% gap vs. gold labels, Section 4.1).
• The input distribution matters less for Direct MetaICL (near-zero sensitivity in Figure 8, top-left panel).
• The label space matters similarly for Direct MetaICL (~16% drop vs. ~8% for Direct GPT-J in Figure 9).
• The format matters even more—MetaICL retains exceptionally high fractions of in-context learning gains from format-plus-one-component demonstrations (82–95% in Figure 10).

The paper hypothesizes that "meta-training encourages the model to exclusively exploit simpler aspects of the demonstrations and to ignore others" (Section 5.4), based on the intuition that the format and label space are easier to exploit than the input-label mapping, and that the space of text the model must generate (labels, for direct models) is easier to exploit than the space of text the model conditions on (inputs, for direct models). This pattern—meta-training making the model less reliant on the informationally richest component of demonstrations—is a cautionary finding about what meta-learning objectives actually optimize.

Ablation Studies and Robustness Checks

Number of demonstration examples k (Section 4.2, Figure 5): The random-label result holds across k ∈ {4, 8, 16, 32}. The gap between gold and random labels is consistently small (0.8–1.6% absolute for k ≥ 8), and performance plateaus after k = 8 for both gold and random labels. This plateau is itself an important finding: it suggests that the value of additional demonstrations saturates quickly, consistent with demonstrations providing distributional/format information rather than supervised training signal (where more examples would monotonically improve the estimated mapping).

Template style (Section 4.2, Figure 6): Manual templates (human-written, dataset-specific prompts from Holtzman et al., 2021 and Zhao et al., 2021) do not alter the central finding. Gold-to-random gaps remain small (0–4%), and manual templates do not consistently outperform minimal templates—sometimes they are better, sometimes worse. This rules out the concern that the minimal templates are somehow masking the mapping's importance by being too impoverished to convey it.

Random label sampling distribution (Appendix C.2): Sampling random labels from the true empirical distribution of labels in the training data (rather than uniformly) further reduces the gap relative to gold labels: from 1.9% to 1.3% for Channel MetaICL, and from 5.0% to 3.5% for Channel GPT-J. This indicates that label distribution information—knowing which labels are common vs. rare, independent of which inputs they pair with—provides a small additional signal, though the dominant source of gains remains the label space and input distribution rather than the mapping.

Per-dataset breakdown (Appendix C.2, Figure 12): While the overall trend shows consistently small gaps, there are outlier datasets where the gap between gold and random labels is non-negligible. For Channel GPT-J with uniform random labels, the largest gaps are on financial_phrasebank (~13–14% absolute), certain hate speech detection datasets (ethos-race, ethos-religion: ~8–10% absolute), and poem_sentiment (~6–7% absolute). For Channel MetaICL with uniform random labels, the gaps on these same datasets are smaller but still visible (~3–5% absolute). This indicates that the input-label mapping does matter more for some tasks than others—the paper acknowledges this as an important direction for future work (Section 6, Limitations). The datasets with larger gaps tend to be those with more nuanced label semantics (financial sentiment, hate speech categories) where the label space alone may be insufficient to disambiguate the task.

Constant label variant (Appendix C.3): Replacing all labels with the constant token "answer" performs consistently worse than random English words. The paper hypothesizes that constant labels "actually change the format of the demonstrations, since they can be viewed as part of a separator between different demonstration examples." This is a negative result that reinforces the format's fragility: even a variant that technically preserves the input-label structure can break performance if the labels are perceived as separators rather than content.

Test-input-as-all-inputs variant (Appendix C.3): Using the test input itself as every demonstration's input (paired with random labels) performs "significantly worse than most other methods with demonstrations," despite trivially satisfying the "in-distribution inputs" condition. The paper attributes this to format disruption: "using the constant input for all demonstration example significantly changes the format of the sequence, since the input can be viewed as part of a separator." This further demonstrates that the format is not just about the presence of input-label pairs but about their variability—identical inputs break the expected structure.

Extrapolation to generation tasks (Section 6, Limitations): The paper explicitly does not test open-ended generation tasks, acknowledging that "extending of our experiments to such tasks is not trivial, because it requires a variation of the output which has incorrect input-output correspondence while keeping the correct output distribution." This is a significant scope limitation: all findings are restricted to tasks with small, discrete label spaces where "random output" is well-defined. For tasks like summarization or translation, it is unclear what a "random but distribution-matching output" would be, making the core experimental paradigm difficult to extend.

Critical Assessment

The experiments provide strong, multi-faceted support for the paper's headline claim: the ground truth input-label mapping in demonstrations contributes much less to in-context learning performance than previously assumed. The claim is tested across 12 models (spanning a 200× parameter range), 26 datasets, two inference methods (direct and channel), multiple demonstration counts (k), two template styles, and several label randomization strategies (uniform, distribution-matched, graded correctness). The consistency of the finding—gold-to-random gaps of 0–5% absolute in nearly all configurations—makes it one of the more robust empirical results in the in-context learning literature.

However, several genuine weaknesses limit the scope and certainty of the conclusions:

1. The 6-dataset evaluation for the largest models. GPT-3 (175B) and fairseq 13B are evaluated on only 6 datasets, compared to 26 for the smaller models. The 6-dataset results (Figure 11) show similar patterns to the full 26-dataset results for the smaller models, but the smaller evaluation set reduces statistical power and may miss dataset-specific phenomena. Given that the per-dataset breakdown (Figure 12) reveals outlier datasets where the gold-random gap is larger (e.g., financial_phrasebank with ~14% absolute gap for GPT-J), it is possible that the 6-dataset subset happens to exclude such datasets, making GPT-3's insensitivity appear more universal than it is.

2. The "random label" condition may overestimate insensitivity due to accidental correctness. For tasks with |C| = 2 (binary classification), uniformly random labels have a 50% chance of being correct by accident. For |C| = 3, the chance is 33%. The paper does not control for this in the default random labels condition (only in the a% correct variant, which explicitly excludes the gold label from incorrect positions). This means the gold-to-random gap underestimates the true value of the mapping, because some fraction of the "random" labels are actually correct. The true-distribution variant partially addresses this but does not eliminate it. A more rigorous control would have used guaranteed-incorrect labels (as in Algorithm 1) for all random-label experiments.

3. No experiments on tasks with genuinely novel input-label mappings. All 26 datasets are drawn from established NLP benchmarks involving natural language inputs and outputs. The model has almost certainly encountered similar input-label relationships during pretraining (as the paper itself argues). The finding that the mapping does not matter may be specific to tasks where the input-label correspondence is already well-represented in the pretraining distribution. The paper acknowledges this limitation: "our findings suggest that in-context learning may not work on a task whose input-label correspondence is not already captured in the LM" (Section 6). Razeghi et al. (2022) and Rong (2021) provide evidence that synthetic tasks with artificial input-label mappings show different behavior, but the current paper does not include any such tasks. An experiment with counterfactual mappings (e.g., classifying sentiment where "positive" means negative and vice versa, with the mapping only defined by the demonstrations) would directly test whether the model can learn genuinely new correspondences from demonstrations. The paper's framework would predict catastrophic failure, but this prediction is never tested.

4. The format ablation conflates format removal with information removal. The "no labels" and "labels only" variants remove both the format and half the demonstration content. It is impossible to present input-label pairs without both inputs and labels. The paper's claim that "the format is essential" is therefore partially confounded with the claim that having both inputs and labels is essential. A more precise claim would be: "the pairing structure, which requires both inputs and labels to be present, is essential." This is a valid finding but is less surprising than "the format is essential" might suggest—of course you cannot specify a task through incomplete demonstrations.

5. No measurement of interaction effects between components. The nested ablation design (disable mapping → disable input distribution → disable label space → disable format) measures marginal contributions along a single path but does not explore interactions. For example, does the input distribution's contribution depend on whether the label space is also present? The comparison tables provide some insight (e.g., for channel models, the input distribution matters regardless of label space, since the label space never matters much), but a fully crossed design would be needed to measure all two-way and three-way interactions. The paper's design choice is pragmatic—16 conditions would be a combinatorial explosion—but it means the reported "marginal contributions" are conditional on the specific ablation order.

6. The OOD corpus choice (CC-News) may not be sufficiently OOD for some datasets. Several evaluation datasets involve news-like text (financial_phrasebank is financial news; tweet_eval involves social media; climate_fever involves factual claims). For these, CC-News sentences may be closer to in-distribution than intended. A stronger OOD test would use text from a completely different modality (e.g., code, scientific abstracts, or non-English text) to ensure maximal distribution shift. The paper's reported input-distribution contributions (3–16%) may therefore underestimate the true importance of input distribution for tasks where the OOD text is genuinely distant from the training domain.

7. No experiments with instruction-tuned models. The paper evaluates base models (GPT-3 Davinci base, not InstructGPT) and MetaICL (meta-trained for in-context learning but not instruction-tuned). Since the paper explicitly connects its findings to instruction-following models (Section 6), and since instruction-tuned models like GPT-3.5/4 or T0 (Sanh et al., 2022) are now the dominant paradigm, the absence of any instruction-tuned model in the evaluation is a significant gap. It is plausible that instruction tuning changes how models use demonstrations—perhaps making them more reliant on the mapping, or alternatively even less reliant because the instruction already specifies the task.

8. No measurement of statistical significance. The paper reports 5-seed (or 3-seed) averages but does not provide confidence intervals, standard deviations, or significance tests for any of the comparisons. The 0–5% gaps between gold and random labels are described as "marginal," but without variance estimates it is impossible to know whether a 5% gap on a particular model-dataset pair is reliably different from zero. Given the small test sets of some datasets (e.g., superglue-cb has only 56 test examples; tweet_eval-stance_atheism has 52), per-dataset variance could be substantial.

9. The paper's central interpretive claim—that models do not "learn new tasks at test time"—depends on what counts as "learning." The experiments demonstrate that the input-label mapping in the demonstrations contributes little, but the demonstrations as a whole contribute substantially (relative to zero-shot). If "learning" includes adapting to input distribution, label space, and format, then the model is learning something from the demonstrations—just not the specific mapping. The paper acknowledges this dual definition (Section 6), but the framing in the abstract and introduction ("counter-intuitively, the model does not rely on the input-label mapping") may lead readers to over-interpret the finding as "demonstrations don't matter," when the paper actually shows they matter enormously through their non-mapping components.

10. Missing experiment: What happens when the label space is correct but the mapping is systematically reversed? Kim et al. (2022) (cited in the paper's limitations added after the initial preprint) showed that using negated labels (e.g., replacing "positive" with "negative" and vice versa) substantially degrades performance. This suggests that while random labels are harmless, actively misleading labels are not. The paper's random-label condition tests a very specific kind of mapping corruption (random, independent per-example). A systematic corruption (e.g., all labels shifted by one position, or reversed) would test whether the model extracts any relational structure from the demonstrations at all. This experiment is not included.

Bottom line on claims: The paper's central empirical claim—"replacing gold labels with random labels barely hurts performance"—is robustly supported across the tested models, datasets, and conditions. The paper's interpretive claim—"the model does not learn the input-label mapping from demonstrations"—is supported under the specific definition of "learning" as extracting per-example correspondences, but the experiments do not rule out that the model learns other aspects of the task from demonstrations (and indeed the paper's own analysis of input distribution, label space, and format shows it does). The paper's broader implication—"in-context learning works by activating latent capabilities from pretraining"—is consistent with the evidence but is an interpretation, not a direct experimental finding; the experiments show what the model doesn't need from demonstrations (correct pairings), not what neural mechanism actually produces the observed behavior. A definitive test of the "activation" hypothesis would require interventions on the model's internal representations, which are outside this paper's purely behavioral scope.

6. Limitations and Trade-offs

The Task Scope Is Restricted to Classification and Multi-Choice with Small, Discrete Label Spaces

The assumption or constraint. All 26 evaluation datasets are classification or multi-choice tasks where the output space is a small, fixed set of discrete labels. The paper explicitly acknowledges this boundary in Section 6 (Limitations):

"our work is limited to classification and multi-choice tasks. Extension to the open-set tasks such as generation is not trivial, since it is unclear how to remove the input-output correspondence while keeping the correct output distribution."

Generative tasks (summarization, translation, dialogue, code generation) have output spaces that are combinatorially large and unstructured. In such settings, there is no well-defined notion of "random labels"—replacing a target summary with a random summary drawn from some corpus may destroy both the input-output correspondence and the output distribution, making the paper's factorial ablation framework inapplicable. The authors are transparent that this is not a minor extension: "it requires a variation of the output which has incorrect input-output correspondence while keeping the correct output distribution."

The consequence. The paper's central finding—that the input-label mapping contributes only marginally—may not generalize to generative tasks. In fact, for tasks like translation or summarization, the output distribution itself is the primary thing the model needs to capture: a French translation of an English sentence has a specific lexical and syntactic structure that cannot be simulated by random French text. The demonstrations in generative settings may need to provide both the output distribution and the input-output correspondence simultaneously, making the two aspects inseparable in a way they are separable for classification. The paper's framework breaks down precisely where it would be most needed: for the open-ended generation tasks that constitute many of the most important applications of in-context learning (coding assistants, writing tools, translation systems).

Subsequent work partially addresses this gap. The paper notes in its Limitations that Madaan and Yazdanbakhsh (2022) conducted a similar analysis for chain-of-thought prompting, finding that "while simply using a random rationale in the demonstrations... significantly degrades the performance, other types of counterfactual rationales (e.g., wrong equations) do not degrade the performance as much as we thought." This suggests the mapping matters more for rationale generation than for classification, consistent with the intuition that generative outputs couple distribution and correspondence more tightly.

What evidence exists in the paper. None—by the authors' own acknowledgment, this is an unaddressed scope limitation. The paper does not include any generative tasks, and the discussion of Madaan and Yazdanbakhsh (2022) appears only in the Limitations section of the final version, not as experimental validation.

Mitigation status. The paper does not attempt to extend the framework to generative tasks. It flags this as future work and acknowledges the nontrivial conceptual challenge of designing a random-output condition that preserves the output distribution. A possible approach—pairing each input with a different ground-truth output from the training set (preserving the output distribution but breaking the correspondence)—is not explored.

The Input-Label Mapping May Matter Substantially for Specific Tasks; the Aggregate Claim Masks Important Heterogeneity

The assumption or constraint. The paper's headline result—"replacing gold labels with random labels barely hurts performance"—is a statement about average behavior across 26 datasets and 12 model configurations. The per-dataset breakdown in Appendix C.2 (Figure 12) reveals that this average conceals substantial variation. The paper acknowledges:

"different datasets may behave differently. Appendix discusses this aspect, including findings that there are larger gaps between using the ground truth labels and using the random labels in some dataset-model pairs (e.g., in the most extreme case, nearly 14% absolute on the financial_phrasebank dataset with GPT-J)."

This is not a minor caveat. A 14% absolute gap is large—comparable to the entire gap between no-demonstrations and full in-context learning for many model-task pairs. The paper's aggregate claim of "0–5% absolute" is true for the central tendency of the distribution but false for the tails.

The consequence. A practitioner deciding whether to invest in carefully labeled demonstrations cannot use the aggregate finding alone. For tasks like financial sentiment analysis or certain types of hate speech detection, the paper's own data show that correct labels do matter substantially. The consequence is a missing diagnostic: the paper does not characterize which tasks are sensitive to label correctness and which are not, leaving practitioners to guess whether their specific use case falls in the 0–5% regime or the 14% regime. This limits the practical applicability of the finding—a practitioner working on a task that happens to be in the long tail of sensitivity could be misled by the aggregate claim into using random labels and losing substantial accuracy.

The per-dataset Figure 12 also shows that the datasets with larger gaps (financial_phrasebank, ethos-race, ethos-religion, poem_sentiment) tend to involve more nuanced or domain-specific label semantics. This suggests a hypothesis the paper does not test: the mapping matters more when the label semantics are less well-represented in the pretraining data. If true, this would mean the paper's findings are most reliable for common NLP task types (sentiment with simple positive/negative labels, NLI with standard entailment/contradiction/neutral) and least reliable for specialized domains with idiosyncratic label definitions.

What evidence exists in the paper. Figure 12 (Appendix C.2) provides per-dataset gap distributions for four (model, label-sampling-strategy) pairs. The gaps range from near zero to ~14% absolute. The paper does not analyze task properties that correlate with gap size, provide confidence intervals, or test whether the outlier datasets share identifiable characteristics. The per-dataset sample sizes (number of test examples per dataset) vary substantially (from ~50 to ~2,000), meaning the gap estimates for small datasets are noisier, but this variance is not discussed.

Mitigation status. The paper acknowledges the heterogeneity but does not explain or mitigate it. The Limitations section states: "We believe it is important to understand in which cases the ground truth labels matter or not, which we leave for future work." No analysis of dataset features (label space size, domain, task type, pretraining frequency) is correlated with gap size. A natural follow-up—training a classifier to predict which tasks will be sensitive to label correctness based on dataset metadata—is not attempted.

All Findings Are Restricted to Natural Language Tasks Where the Input-Label Correspondence Likely Exists in Pretraining Data

The assumption or constraint. The paper argues that in-context learning works because the model has already internalized the input-label correspondence during pretraining:

"the model has learned the (implicit notion of) input-label correspondence from the language modeling objective alone, e.g., associating a positive review with the word 'positive'."

This implies a clear boundary condition: if a task's input-label correspondence was not captured during pretraining, in-context learning should fail—and the paper's findings about random labels would not hold. The paper does not test this boundary. All 26 evaluation datasets involve natural language inputs and outputs that the model has almost certainly encountered in some form during pretraining on web text, books, and Wikipedia. The claim that "the model does not learn new tasks at test time" (Section 6) cannot be validated without testing on tasks that are genuinely novel relative to the pretraining distribution.

The consequence. The paper's conclusions may not extend to tasks that require learning counterfactual or novel input-label mappings. Consider a task where the model must classify movie reviews as "positive" when the review is negative and "negative" when the review is positive—a mapping that contradicts everything the model learned during pretraining. The paper's framework would predict that this task is impossible via in-context learning because the model's pretrained notion of sentiment (which the demonstrations merely "locate") points in the opposite direction. But the paper never tests this prediction.

This is not a hypothetical limitation—it directly constrains when practitioners can safely use in-context learning. For tasks involving standardized taxonomies (sentiment categories, NLI relations, common QA answer types), the paper's findings are likely robust. For tasks involving organization-specific label schemas, technical jargon, or domain-specific classification systems that the model cannot have seen during pretraining, the paper provides no evidence about whether demonstrations with correct labels are necessary. The Discussion (Section 6) explicitly raises this concern:

"in-context learning may not work on a task whose input-label correspondence is not already captured in the LM. This leads to the research question of how to make progress in NLP problems that in-context learning does not solve."

But the paper does not empirically investigate this boundary.

What evidence exists in the paper. Indirect evidence comes from Razeghi et al. (2022), cited in the Related Work, who showed that in-context learning performance correlates with term frequencies in the pretraining data—suggesting pretraining exposure is indeed the enabling factor. Additionally, Rong (2021), cited in the Limitations, observed that "synthetic tasks with more limited inputs may actually use the ground truth labels more." Neither of these is the current paper's own data. The paper provides no experiment where the input-label mapping is systematically altered from the pretraining distribution to test whether the model can learn it from demonstrations.

Mitigation status. The paper does not test counterfactual or novel mapping tasks. The Limitations section acknowledges that synthetic tasks may behave differently and cites Rong (2021) as evidence that the random-label finding may not hold for such tasks. The Discussion section flags this as an open research question—"whether we need a better way of extracting the input-label mappings that are already stored in the LM, a better variant of the LM objective that learns a wider range of task semantics, or explicit supervision through fine-tuning on the labeled data"—but provides no empirical guidance on which answer is correct.

The Format's Central Role Implies Extreme Fragility to Prompt Design Choices That the Paper Does Not Characterize

The assumption or constraint. The paper's most actionable finding for practitioners is that the format—the structural convention of presenting inputs and labels as paired sequences—is "essential" (Section 5.3): without it, no combination of other components achieves above-baseline performance; with it, even nearly content-free demonstrations (random English words paired with in-distribution inputs) retain 75–95% of in-context learning gains. However, the paper's definition of "format" is binary (paired vs. not paired) and its experimental manipulation is coarse (remove all labels, or remove all inputs). This masks what is likely a continuous sensitivity: not all formats are equally effective, and small format changes (separator tokens, ordering, whitespace, template wording) can cause large performance swings—as prior work has extensively documented (Zhao et al., 2021; Lu et al., 2021).

The consequence. The paper's positive message—"you don't need correct labels, just get the format right"—may create a false sense of simplicity. In practice, "getting the format right" is itself a difficult optimization problem. The paper's own Appendix C.3 provides evidence: the constant-label variant ("answer" as every label) and the test-input-as-all-inputs variant both technically preserve the pairing format but perform substantially worse than random English words or OOD inputs, respectively. The authors attribute this to the format being "changed" because the labels or inputs are perceived as separators. But this means the distinction between a valid format and a broken one is not captured by the binary "paired vs. unpaired" categorization—it depends on subtle perceptual factors the paper does not characterize.

The manual template experiment (Figure 6) partially addresses this by showing the random-label result holds across two template styles, but this is only two styles per dataset, and the paper notes that manual templates "do not always outperform" minimal templates—indicating format quality varies in ways the paper does not analyze. A practitioner trying to apply the paper's findings to a new task faces an unresolved question: "I know the format matters enormously, but how do I design a good one?"

The paper also does not test format variations that might increase sensitivity to the input-label mapping. Perhaps a format that more explicitly separates the demonstration examples (e.g., numbered lists, explicit "Example 1:" prefixes) would encourage the model to attend more to individual pairings. The paper's minimal templates may be partially responsible for the finding that the mapping does not matter—by presenting demonstrations as a continuous block, the format may discourage per-example correspondence learning in favor of distributional extraction.

What evidence exists in the paper. Figure 10 shows that removing the format entirely collapses performance. Appendix C.3 shows that two variants that technically preserve pairing but perceptually alter the format (constant labels, repeated test input) perform poorly. Figure 6 shows that two template styles produce similar patterns. These data points are suggestive but undersystematic: the paper does not sweep format parameters (separator type, example ordering, template wording, whitespace conventions) to map out the sensitivity landscape.

Mitigation status. The paper does not provide a characterization of format robustness or guidelines for format design beyond the high-level finding that pairing matters. The finding that manual templates do not consistently outperform minimal templates (Figure 6) partially mitigates the concern—if format quality mattered enormously for the random-label result, manual templates should show a different pattern—but the paper does not test whether certain format choices interact with the mapping contribution (i.e., whether some formats make the model more reliant on correct labels). This leaves the practical implication ambiguous: the format is essential, but what constitutes a good format remains an empirical question per task and model.

The Findings Are Specific to Base (Non-Instruction-Tuned) Models; Modern Instruction-Tuned Models May Behave Differently

The assumption or constraint. All models tested are base language models: GPT-2, MetaICL (meta-trained for in-context learning but not instruction-tuned), GPT-J, fairseq 6.7B/13B, and GPT-3 Davinci base. None is instruction-tuned (e.g., InstructGPT, T0, FLAN, or any RLHF-trained model). The paper's Discussion (Section 6) explicitly draws a connection to instruction-following models, hypothesizing that "the demonstrations and instructions largely have the same role to LMs" and that the paper's findings "hold for instruction-following models." But this hypothesis is never tested.

The consequence. This is a significant gap for practical relevance. At the time the paper was published (2022, with the updated version in October 2022), instruction-tuned models were already becoming the dominant paradigm for few-shot and zero-shot applications (Wei et al., 2022a; Sanh et al., 2022; Ouyang et al., 2022). If instruction-tuned models use demonstrations differently—for instance, if instruction tuning trains the model to attend more to the specific input-label mapping because it learns that the provided examples define the task—then the paper's central finding may not apply to the models practitioners are most likely to deploy. Conversely, if instruction-tuned models are even less sensitive to label correctness (as Webson and Pavlick, 2022, showed for instructions with irrelevant or misleading text), the paper's findings may understate how little the mapping matters.

The MetaICL results provide partial but imperfect evidence. MetaICL is meta-trained with an in-context learning objective—it learns to expect (demonstrations, test_input, label) sequences—but it is not instruction-tuned in the sense of learning to follow natural language instructions. The finding that MetaICL is less sensitive to the mapping than non-meta-trained models (0.1–0.9% gap vs. ~2.6% average) suggests training models to expect demonstrations may further reduce mapping reliance. But MetaICL is also the only meta-trained model and is only 774M parameters; it is unclear whether this pattern scales to larger instruction-tuned models. The paper's own hypothesis (Section 5.4) is that meta-training encourages exploiting "simpler aspects"—if this is the mechanism, instruction tuning (which also trains models to extract task specification from structured prompts) might produce similar effects, but this is speculation.

What evidence exists in the paper. Indirect evidence only: the MetaICL results (Section 5.4) show reduced mapping sensitivity relative to the identically-sized GPT-2 Large base model; Webson and Pavlick (2022) is cited as showing that "instructions that are irrelevant or even misleading lead to performance gains as much as 'good' instructions do," which the paper interprets as partially verifying its hypothesis for instruction-following models. However, the paper itself runs no experiments on instruction-tuned models.

Mitigation status. The paper does not test instruction-tuned models. The Discussion section flags the connection as a hypothesis and suggests future work: "We leave more analysis on instruction-following models for future work." The updated version (with the reviewer response and limitations section) does not add such experiments.

The Paper Does Not Test Whether Systematically Misleading Labels (As Opposed to Random Labels) Degrade Performance

The assumption or constraint. The paper's central manipulation is replacing gold labels with random labels—labels sampled independently and uniformly from the output space, with no systematic relationship to the inputs or to each other. This tests whether the model needs correct pairings, but it does not test whether the model is sensitive to incorrect pairings that carry misleading signal. Random labels are essentially noise—they do not define an alternative task; they fail to define any task at all. A more stringent test would be systematically misleading labels: for instance, pairing all positive reviews with "negative" and all negative reviews with "positive" (label reversal), or shifting all labels by one position in a fixed cycle (cyclic permutation).

The consequence. The finding that random labels "barely hurt" does not imply that the model ignores labels entirely. It is possible—and indeed likely, given Kim et al. (2022), which the paper's updated Limitations section cites—that systematically misleading labels do degrade performance. Kim et al. (2022) showed that using negated labels substantially reduces accuracy in classification. If confirmed, this would mean the model does extract some signal from the labels in the demonstrations—enough to be misled when the signal consistently points in the wrong direction—even though it does not rely on per-example correctness.

This has practical implications. A practitioner who misunderstands the paper's finding might assume that any labels work equally well, including deliberately wrong ones. The paper's abstract states "ground truth demonstrations are in fact not required—randomly replacing labels in the demonstrations barely hurts performance," which could be (mis)read as "labels don't matter." But the distinction between random labels (noise) and misleading labels (systematic error) is crucial. Random labels are harmless because they convey distributional information about the label space without imposing a consistent alternative mapping. Misleading labels impose a consistent (but wrong) mapping that could compete with the model's pretrained notion of the task.

The paper's a% correct labels experiment (Figure 4) partially addresses this by testing graded correctness, but the "incorrect" labels in that experiment are still random per-example—they are not systematically misleading. A label-reversal condition would test whether the model can override its pretrained priors when the demonstrations consistently signal a different mapping.

What evidence exists in the paper. Only indirect and external. The updated Limitations section cites Kim et al. (2022): "Since the first version of our paper, Kim et al. (2022) showed that using negated labels substantially lowers the performance in classification." The footnote adds: "Note that Kim et al. (2022) estimate the random label performance by interpolating with the performance using negated labels, while our paper samples the random labels at uniform." This citation acknowledges the distinction between random and misleading labels but does not incorporate it into the paper's own experimental design. The paper provides no data on label reversal, cyclic permutation, or other systematic corruptions.

The graded-correctness experiment (Section 4.2, Figure 4) shows that performance is relatively flat as the proportion of correct labels decreases from 100% to 0%, but the incorrect labels in this experiment are independently randomized per position—they do not form a coherent alternative task. This leaves open the question: would performance drop if the incorrect labels were consistently wrong in the same way?

Mitigation status. The paper does not include systematically misleading label experiments. The updated Limitations section acknowledges the Kim et al. (2022) finding but treats it as a boundary condition to be explored in future work rather than integrating it into the analysis. The paper states: "We believe it is important to understand to what extent the model needs the ground truth labels to successfully perform in-context learning." A label-reversal experiment would be the most direct test of this question—does in-context learning require correct labels or merely non-misleading ones?—but it is absent.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes what in-context learning is — moving the field's understanding from "the model learns new input-label correspondences from provided examples" to "the model uses demonstrations to locate and activate existing capabilities from pretraining." This is a conceptual reframing, not an incremental refinement: it overturns the working assumption that had implicitly guided all prior work on demonstration selection, prompt optimization, and in-context learning methodology.

The magnitude of this shift is best understood by examining what the paper makes newly explicable. Prior work contained a paradox: in-context learning reliably outperforms zero-shot (Brown et al., 2020; Zhao et al., 2021), yet self-correction and iterative refinement approaches often failed to improve reasoning (Huang et al., 2023), and models showed extreme sensitivity to demonstration ordering and format (Lu et al., 2021; Zhao et al., 2021). The paper's four-aspect decomposition resolves this tension. If demonstrations serve primarily as task specifiers (providing input distribution, label space, and format) rather than training data (providing input-label correspondences), then:

• Performance improves over zero-shot because the demonstrations narrow the model's output distribution to the right task — but the task competence itself comes from pretraining, explaining why in-context learning works at all without gradient updates.
• Sensitivity to format and ordering makes sense because the format is the essential scaffolding that enables the model to interpret the demonstrations as a task specification — break the format, and the model cannot locate the right behavior.
• The failure of self-correction on novel reasoning tasks (Huang et al., 2023) is predictable: if the model has no pretrained notion of the correct reasoning steps, demonstrations cannot create it, only surface it.

This reframing also reconciles conflicting findings that had puzzled the field. Reynolds and McDonell (2021) argued demonstrations are unnecessary — the capability exists from pretraining. But this paper shows demonstrations are not unnecessary: they provide the format, input distribution, and label space without which the model cannot reliably deploy its latent capabilities. The zero-shot baseline dramatically underestimates what the model can do when properly prompted. Conversely, the Bayesian inference account of Xie et al. (2022) — where demonstrations provide evidence for latent task parameters — struggles to explain why random labels (which provide misleading evidence) barely hurt performance. The paper suggests an alternative Bayesian story: the demonstrations provide evidence about which of many pre-learned tasks to execute, not about the parameters of a new task.

The paper reorients research priorities in specific ways:

• Demonstration selection research becomes less central. If the input-label mapping contributes only 0–5% absolute, then sophisticated retrieval methods for choosing the "best" examples (Liu et al., 2021; Rubin et al., 2021) are optimizing a component that matters relatively little. The gains from such methods are real but are likely coming from optimizing the input distribution and label space specification (choosing examples that better represent the task domain and output vocabulary) rather than from providing higher-quality input-label pairings. This paper provides a framework for diagnosing which component a selection method actually improves.

• Prompt format design becomes a first-class research problem. The paper's most actionable finding is that the format — the structural convention of pairing inputs with labels — is indispensable. Without it, performance collapses to or below zero-shot. With it, even content-free demonstrations (random words paired with in-distribution inputs) retain 75–95% of in-context learning gains. This elevates format engineering from a nuisance variable to a central mechanism: getting the format right is more important than curating high-quality examples. Prior work that treated format as a hyperparameter to tune (Zhao et al., 2021; Holtzman et al., 2021) was implicitly optimizing the most important lever.

• Understanding what pretraining captures becomes the binding constraint. If in-context learning works by surfacing latent capabilities, then the ceiling on in-context learning performance is set by what the model internalized during pretraining. The paper makes this explicit: "in-context learning may not work on a task whose input-label correspondence is not already captured in the LM" (Section 6). This shifts the research frontier from "how do we construct better demonstrations?" to "how do we pretrain models so that more tasks are latent within them?" — or alternatively, "how do we verify what tasks a given model already knows?"

• The connection between in-context learning and instruction-following is clarified. The paper hypothesizes that "the demonstrations and instructions largely have the same role to LMs" — both are task specification mechanisms that locate existing capabilities. This explains why instruction-tuned models (Wei et al., 2022a; Sanh et al., 2022) can achieve strong zero-shot performance: the instruction provides the format and task specification directly, without needing demonstration examples to communicate the input distribution and label space. The paper's contribution is not this hypothesis (which existed in nascent form) but the systematic evidence that demonstrations function this way even in standard few-shot prompting — they are implicit instructions communicated through example structure rather than explicit instructions communicated through natural language.

Follow-Up Research This Work Enables

Testing the pretraining-dependence hypothesis with counterfactual label mappings. The paper's central interpretive claim — that models do not learn new input-label correspondences at test time but instead activate pretrained associations — predicts a sharp boundary: in-context learning should fail catastrophically on tasks where the input-label mapping contradicts the pretraining distribution. A direct test would use counterfactual mappings: for instance, on a sentiment classification dataset, train a model where the demonstrations pair all positive reviews with "negative" and all negative reviews with "positive" (label reversal), and measure whether the model (a) follows the demonstration mapping (indicating genuine learning from examples) or (b) follows the pretrained mapping despite the demonstrations (indicating the demonstrations cannot override pretraining priors). The paper's graded-correctness experiment (Figure 4) shows that 0%-correct random labels barely hurt, but random labels impose no systematic alternative mapping — they are noise. Label reversal imposes a coherent but wrong mapping that directly competes with pretraining. If the model achieves reversed-label accuracy substantially above the no-demonstrations baseline but below the gold-label baseline, that would quantify how much the model can learn from demonstrations when the mapping is clean but counterfactual. The paper's own Limitations section cites Kim et al. (2022) as showing that negated labels degrade performance, but a full dose-response curve (varying the systematicity of label corruption from random to fully reversed) would map out the model's willingness to override pretraining. This experiment is now directly motivated by the paper's framework: it isolates the mapping contribution under the strongest possible signal (a consistent but wrong mapping), testing the limits of "learning at test time."

Extending the four-aspect decomposition to instruction-tuned models. The paper evaluates only base models (GPT-3 Davinci base, not InstructGPT) and one meta-trained model (MetaICL). The Discussion hypothesizes that the findings should hold for instruction-tuned models — that instructions and demonstrations serve the same "task location" function. But this prediction is untested and could fail in informative ways. Instruction tuning might teach models to attend more to the specific input-label mapping (because the training data includes diverse instructions paired with correct examples) or less (because the model learns to extract task identity from the instruction alone). A follow-up study would replicate the paper's core ablations — gold labels vs. random labels vs. random English words vs. OOD inputs vs. format removal — on FLAN-T5, T0, or InstructGPT across the same 26 datasets. The key comparison is the gap between gold and random labels in instruction-tuned vs. base models. If instruction-tuned models show a larger gap, it suggests instruction tuning teaches the model to use provided examples as genuine supervision, contradicting the paper's "task location" interpretation. If the gap is equally small or smaller, it validates the interpretation and extends it to the model class that practitioners predominantly use. The MetaICL results (0.1–0.9% gap) already hint that training on demonstrations reduces mapping reliance, but MetaICL is a 774M model trained only for in-context learning, not instruction following — the generalization to larger, instruction-tuned models is not guaranteed.

Characterizing which tasks are sensitive to label correctness and why. The paper's per-dataset breakdown (Figure 12) reveals substantial heterogeneity: financial_phrasebank shows a ~14% gap for GPT-J while many other datasets show gaps near zero. The paper acknowledges this as a limitation but provides no analysis of what properties predict sensitivity. A systematic follow-up would: (1) measure the gold-random gap for all 26 datasets across all available models, (2) characterize each dataset along dimensions the paper's framework suggests should matter — label space size, label semantic ambiguity (are labels like "positive"/"negative" unambiguous or nuanced like financial sentiment categories?), domain specificity (how likely is the input-label correspondence to appear in generic pretraining data?), and presence of strong lexical cues (do certain words strongly predict certain labels, making the mapping easy to recover from pretraining?), (3) correlate these features with gap size. This would transform the paper's qualitative boundary condition ("the mapping matters when the task is not already captured in pretraining") into a quantitative diagnostic. A strong result would show that gap size is predicted by, say, the frequency of co-occurrence between task-relevant terms and label words in a pretraining corpus proxy (e.g., C4 or the Pile), providing an operational test of the "latent capability" hypothesis. This is directly enabled by the paper's experimental framework — the gap measurement already exists; the missing piece is the characterization of when the gap appears.

Decomposing the format into constituent features and measuring their individual contributions. The paper demonstrates that the format is "essential" — removing it collapses performance — but treats "format" as a binary variable (paired vs. unpaired). The Appendix C.3 results (constant labels, test-input-as-all-inputs) show that not all formats are equal: variants that technically preserve pairing but perceptually alter the structure perform poorly. This opens the question: which specific format features matter? A follow-up study would independently vary format components — separator token type (newline vs. " | " vs. numbered prefixes like "Example 1:"), input-label ordering (label first vs. input first), inter-example spacing, presence of an explicit instruction before the demonstrations ("Classify the sentiment of each text:"), and whether the test input is presented as an incomplete pair (appending the separator) or as a separate query after the demonstrations. Each variant would be tested under both gold-label and random-label conditions to see whether certain format choices increase reliance on the mapping (i.e., produce larger gold-random gaps). The paper's finding that manual templates do not consistently outperform minimal templates (Figure 6) suggests format quality is non-monotonic, but a systematic sweep would map out the sensitivity surface and provide actionable guidelines for practitioners. This is newly important because the paper has established that format is the primary lever — optimizing it is more impactful than curating examples.

Designing demonstrations that maximize performance from unlabeled data alone. A direct practical consequence of the paper's findings is a new class of zero-shot methods that use unlabeled training data: pair each unlabeled input with a random label (preserving the label space and input distribution with the format intact) and use this as the demonstration. The paper shows this approach retains most of the few-shot gain on average, but does not optimize it. A follow-up study would explore: (1) whether sampling labels from the empirical training distribution (matching class balance) outperforms uniform sampling, especially on imbalanced datasets; (2) whether using more unlabeled examples (k > 16) improves performance by better characterizing the input distribution, even with all-random labels; (3) whether filtering the unlabeled examples (e.g., by length, by lexical diversity, or by a pretrained classifier's confidence) improves over random sampling; (4) whether ensembling multiple random-label demonstrations (different random seeds) and aggregating predictions outperforms a single demonstration set. The paper provides the baseline (random labels with uniform sampling, k = 16) but does not explore these optimizations. This line of work would establish a new "enhanced zero-shot" paradigm that uses unlabeled data more effectively than traditional zero-shot but without requiring any label annotation — directly motivated by the paper's finding that the input-label mapping is the least important component.

Stress-testing the framework's limits with synthetic tasks of known pretraining exposure. The paper's central argument — that in-context learning works by surfacing latent pretrained capabilities — predicts that in-context learning should fail on tasks where the input-label correspondence is truly novel relative to the pretraining data. The current evaluation uses natural language benchmarks where pretraining exposure is plausible but unmeasured. A targeted stress test would construct synthetic classification tasks with deliberately novel input-label mappings — for example, classifying strings of random characters into categories with nonsense labels like "zorg," "blip," and "quarn," where the mapping is defined only by the demonstrations (e.g., strings containing "a" are "zorg," strings containing "b" are "blip," etc.). The paper's framework predicts: (a) zero-shot should be at chance (the model has no pretrained notion of the mapping), (b) demonstrations with gold labels should enable above-chance performance if the model genuinely learns the mapping from examples, and (c) demonstrations with random labels should perform at chance (no latent capability to activate). If all three predictions hold, it would directly validate the "task location" interpretation by showing the boundary where in-context learning does require the mapping. If (b) fails — the model cannot learn even from gold labels — it would indicate an even stronger version of the paper's thesis: in-context learning cannot ever learn new correspondences, only surface existing ones. If (c) succeeds — random labels work even on synthetic tasks — it would suggest something more complex is happening (perhaps the model induces the mapping from a single correct accident among the random labels, or uses the input distribution alone to cluster examples). This experiment is now well-defined by the paper's framework and would resolve the largest remaining ambiguity: whether the model can learn new mappings when given correct labels, even though it doesn't need to for natural language tasks.

Practical Applications and Downstream Use Cases

Cost-efficient few-shot deployment using unlabeled data instead of labeled examples. For practitioners deploying in-context learning on classification or multi-choice tasks with a known label space, this paper provides direct cost savings: rather than annotating k = 16 gold-labeled examples per task, one can sample 16 unlabeled examples from the target domain's text distribution, pair each with a random label from the known output space, and achieve performance within 0–5% absolute of using gold labels (Figure 3). For an organization deploying sentiment analysis across 50 product categories, this reduces the annotation burden from 800 hand-labeled examples (50 categories × 16 examples) to zero — only unlabeled text from each category plus knowledge of the label set {positive, negative, neutral} is needed. The paper's 75–95% retention figures for format-plus-one-component demonstrations (Figure 10) mean that even if neither labeled examples nor an explicit label vocabulary is available, pairing in-distribution text with arbitrary English words and getting the format right still outperforms naive zero-shot. The practical recipe is: (1) identify the label space (e.g., {positive, negative, neutral}), (2) collect 16 unlabeled texts from the target distribution, (3) pair each text with a randomly chosen label from the space using the model-appropriate format (newlines, consistent separators), (4) append the test input and run inference. The paper's consistent results across k values (Figure 5) mean that even 8 examples largely suffice — the marginal benefit of additional unlabeled examples plateaus quickly.

Rapid domain adaptation without labeled data. For applications where the target domain differs from generic pretraining data (e.g., legal document classification, medical coding, financial sentiment), the paper's finding that the input distribution contributes 3–16% absolute (Figure 8) provides a principled strategy: use in-domain unlabeled text as the demonstration inputs, even with random labels, to calibrate the model to the target domain's vocabulary, style, and length conventions. A legal tech company deploying a model to classify clauses in contracts (label space: {indemnification, termination, confidentiality, ...}) can collect 16 real contract clauses from their document corpus, pair each with a randomly sampled label from the known class set, and achieve performance that captures most of the in-context learning gain — all without a single human-annotated clause-label pair. The domain adaptation effect comes from the model seeing the target input distribution (legal language, contract structure) in the demonstrations, which narrows its output distribution to the right linguistic register. This is significant because domain-specific annotation is typically more expensive than generic annotation (requiring domain experts), making the random-label approach disproportionately valuable for specialized domains.

Safer prompt engineering through format prioritization over example curation. The paper's finding that the format is "essential" — without it, performance collapses to or below zero-shot regardless of other components — provides a clear prioritization for practitioners debugging in-context learning failures. When a few-shot prompt underperforms, the first diagnostic should be: is the format unambiguously communicating the input-label pairing structure? Common failure modes (extra whitespace, inconsistent separators, missing newlines, template wording that changes across examples) may break the format in ways the paper's constant-label and test-input variants (Appendix C.3) demonstrate can be catastrophic. The paper's numbers provide a concrete threshold: if performance with a given prompt is at or below the zero-shot baseline, the format is likely broken — no amount of example curation will fix it. If performance is above zero-shot by a margin comparable to the paper's random-label results, the format is functioning and further gains (the remaining 0–5%) require optimizing the input-label mapping through example selection or label correctness. This diagnostic workflow — check format first, mapping second — is a direct operationalization of the paper's decomposition framework and can reduce the trial-and-error cost of prompt development. The finding that manual templates do not consistently outperform minimal templates (Figure 6) further suggests that format complexity is not inherently better — consistency and unambiguous pairing structure are the essential properties.