Finetuned Language Models Are Zero-Shot Learners

ArXiv: 2109.01652

🎯 Pitch

Fine-tuning a 137B model solely on task instructions—without examples—enables it to outperform zero-shot GPT-3 on 20 of 25 unseen benchmarks and even beat few-shot GPT-3 on several reasoning tasks. This “instruction tuning” transforms a standard language model into a zero-shot powerhouse, but only when the base model exceeds 8B parameters; smaller models collapse under the same process.

1. Executive Summary

This paper introduces a simple method for improving the zero-shot generalization of large language models. Taking a 137B-parameter pretrained decoder-only model (LaMDA-PT) and fine-tuning it on a mixture of over 60 NLP datasets expressed via natural language instruction templates, the resulting model—dubbed FLAN, for Finetuned Language Net—substantially improves zero-shot performance on unseen task types. The core mechanism is instruction tuning (formatting diverse NLP tasks as "respond to this instruction" prompts during multi-task fine-tuning, e.g., “Translate this sentence to French” or “Does the premise entail the hypothesis?” rather than using sentence-completion-style prompts), which teaches the model to follow task descriptions and generalize that ability to tasks it never saw during fine-tuning. FLAN outperforms zero-shot 175B GPT-3 on 20 of 25 evaluated datasets and even surpasses few-shot GPT-3 by large margins on ANLI, RTE, BoolQ, and several other benchmarks, establishing that the benefits of instruction tuning emerge only at sufficient model scale—models below ~8B parameters actually lose zero-shot generalization ability after instruction tuning because all capacity is consumed by learning the training tasks.

2. Context and Motivation

The Core Problem: The Zero-Shot Gap in Large Language Models

By early 2022, large language models had demonstrated a remarkable capability: few-shot learning. Models like GPT-3 (Brown et al., 2020) could perform novel tasks when given just a handful of input-output examples in their prompt. This was transformative because it meant a single model could handle dozens of tasks without task-specific fine-tuning or architecture modifications, simply by conditioning on examples at inference time.

However, these same models were substantially weaker at zero-shot learning — performing tasks when given only a task description, with no examples at all. The paper opens Section 1 by stating this directly:

"GPT-3's zero-shot performance is much worse than few-shot performance on tasks such as reading comprehension, question answering, and natural language inference."

This gap is visible throughout the GPT-3 paper. For instance, on Natural Questions, GPT-3 achieved 14.6% zero-shot accuracy versus 29.9% with 64 few-shot examples — the difference between barely useful and practically deployable. On TriviaQA, the gap was even more dramatic: 64.3% zero-shot versus 71.2% few-shot. The few-shot examples effectively doubled or tripled the model's usable knowledge on many tasks.

Why does this gap exist? The paper offers a concrete hypothesis rooted in the nature of pretraining:

"One potential reason is that, without few-shot exemplars, it is harder for models to perform well on prompts that are not similar to the format of the pretraining data."

A 137B-parameter language model is trained on a massive corpus of web documents, code, books, and dialog data. During pretraining, it learns to continue text — to predict the next token given a prefix. The pretraining data does not naturally contain sequences like "Determine whether the following hypothesis is entailed by the premise" followed by a structured answer. It contains things that look like tests, yes, but not instructions. Few-shot exemplars bridge this gap by showing the model what the format should look like: here's input A giving output B, here's input C giving output D, now you do E. The exemplars effectively warp the prompt distribution toward something the model has seen during pretraining (collections of similar formatted items).

Without exemplars, the model faces a pure instruction, and instructions are rare in most pretraining corpora. The model may know the answer (it has the factual knowledge or reasoning capability buried in its parameters) but it doesn't understand what is being asked in this unfamiliar format. It defaults to continuing the text as if it were a narrative or document, rather than producing the task-specific output.

Why This Problem Matters

Accessibility and democratization. The few-shot paradigm requires users to:

1. Have a small labeled dataset (exemplars) ready at inference time
2. Understand prompt engineering well enough to format exemplars effectively
3. Pay the inference cost of processing those exemplars (which increases with each few-shot example added to the context)

Zero-shot capability eliminates all three requirements. A user could simply ask the model "Is the following review positive or negative?" without needing to provide any examples of positive or negative reviews. This dramatically lowers the barrier to entry for non-experts and reduces inference costs.

Paving the way for generalist models. The paper frames its work within a broader tension identified in Section 6: the tradeoff between specialist models (one model fine-tuned per task) and generalist models (one model for many tasks). Specialist models require task-specific engineering, separate serving infrastructure, and labeled data for each new task. Generalist models promise to handle arbitrary tasks from a single endpoint. Improving zero-shot performance moves us toward the generalist vision, where a model can be deployed once and then used for many purposes by many users without per-task customization.

The broader vision of instruction-following. Section 1 frames the motivation more aspirationally:

"We leverage the intuition that NLP tasks can be described via natural language instructions, such as 'Is the sentiment of this movie review positive or negative?' or 'Translate "how are you" into Chinese.'"

This is about more than benchmark scores. It's about whether language models can become usable tools that respond to human intent expressed in natural language, rather than oracles that require carefully crafted, distribution-matched prompts with exemplar formatting. If a model can follow instructions, it becomes accessible to anyone who can describe what they want in plain language.

Where Prior Approaches Fall Short

The paper positions itself relative to three paradigms of NLP model usage (visualized in Figure 2), each with limitations:

The pretrain–fine-tune paradigm (BERT, T5). This is the dominant approach circa 2019-2020: pretrain a large language model on unsupervised text, then fine-tune it on a specific downstream task using task-specific labeled data. You end up with one specialized model per task. The limitations:

• Requires many task-specific labeled examples for each new task
• One specialized model for each task — you cannot deploy a single BERT checkpoint that handles sentiment analysis, NLI, and question answering; you need three separate fine-tuned checkpoints
• No generalization to unseen tasks — the fine-tuned model only knows the task it was trained on

This approach produces strong results on individual benchmarks but does nothing for zero-shot generalization or model reuse across tasks.

The prompting paradigm (GPT-3). This addresses the "one model per task" problem: a single pretrained language model can handle many tasks through in-context learning, conditioning on prompt text at inference time without any gradient updates. The limitations the paper identifies:

• Few-shot prompting requires exemplars — obtaining or constructing these exemplars may require labeled data, domain expertise, or prompt engineering skill
• Zero-shot prompting performs poorly on many task types, creating a large performance gap compared to few-shot
• Prompt engineering is non-trivial — practitioners often need to craft prompts that mimic pretraining data formats (e.g., formatting MultiRC questions like a test with an answer key, as GPT-3 does), which requires understanding what the model has seen during pretraining

Critically, these prompts are not instructions in the natural language sense. They are continuations. The model sees a prefix and predicts what comes next. GPT-3's prompt for an NLI task might look like:

At my age you will probably have learnt one lesson.
question: It's not certain how many lessons you'll learn by your thirties. true, false, or neither?
answer:

This works because the model has seen QA-formatted text during pretraining, but it's not a natural instruction. It's a format hack. The appendix FAQ section makes this contrast explicit, showing how the FLAN prompt for the same task is framed as an instruction ("Does the premise entail the hypothesis?") rather than a continuation. The paper's key insight is that this instruction format is more natural for humans but less natural for pretrained LMs — hence the need for instruction tuning to bridge the gap.

Instruction tuning (this paper's contribution). The paper introduces this as a third paradigm that sits between pretrain–fine-tune and prompting. From pretrain–fine-tune, it borrows the idea of using supervision (labeled data) to teach the model. From prompting, it borrows the idea of a single model handling many tasks through inference-time text interactions. The synthesis is:

"Instruction tuning is a simple method that combines appealing aspects of both the pretrain–finetune and prompting paradigms by using supervision via fine-tuning to improve language models' responses to inference-time text interactions."

This is a specific design choice with a specific hypothesis: if you fine-tune a model on enough tasks expressed as instructions, it will learn to follow instructions as a general skill, and that skill will transfer to tasks it has never seen.

Prior Work That Falls Short on Zero-Shot Instruction Following

The paper situates itself against several lines of research in Section 5 and Appendix D. Here are the specific gaps in prior work that FLAN addresses:

QA-based task formulation (McCann et al., 2018; Khashabi et al., 2020). The idea of reformulating NLP tasks as question-answering is not new. The Natural Language Decathlon (McCann et al., 2018) cast ten NLP tasks as QA and showed transfer learning benefits. UnifiedQA (Khashabi et al., 2020) extended this to 20 datasets. However:

"Though these methods are very similar to ours, they mostly focus on multi-task learning instead of zero-shot learning, and—as noted by Liu et al. (2021)—they are generally not motivated by using existing knowledge in pretrained LMs."

In other words, these methods treat QA formatting as a way to unify tasks for joint training, improving performance on the training tasks themselves. They don't study whether the QA-formatting skill generalizes to entirely unseen task types. FLAN's contribution is specifically about that generalization.

Multi-task fine-tuning on instructions at smaller scale (Mishra et al., 2021; Ye et al., 2021). Two recent concurrent works had explored instruction-based fine-tuning for generalization:

• Mishra et al. (2021) fine-tuned a 140M-parameter BART model on instructions with few-shot exemplars and found improved few-shot performance on unseen tasks
• Ye et al. (2021) used MAML-style meta-learning on BART for cross-task few-shot generalization

These papers suggested that instruction-based fine-tuning could work at smaller model scales (hundreds of millions of parameters). FLAN's work differs in a crucial way that turns out to be essential:

"Our work differs from these two papers in that we focus on zero-shot learning, for which we observe the crucial importance of model scale (FLAN is 1,000x larger than BART-base)."

The ablation in Section 4.2 (Figure 7) reveals why this matters: instruction tuning at the 8B-parameter scale and below actually hurts zero-shot performance on held-out tasks. The benefit only emerges at the 68B+ parameter scale. This is a foundational empirical finding — instruction tuning is not just "multi-task learning helps," it's specifically an emergent property of scale, where large models can absorb the training tasks without exhausting their capacity for generalization.

Prompt tuning and continuous prompts (Li & Liang, 2021; Lester et al., 2021). These methods optimize continuous vectors prepended to the input rather than using discrete natural language instructions. While effective, they require per-task optimization (gradient descent on a task-specific validation set) and don't enable zero-shot generalization through language-based instructions. FLAN shows that instruction tuning improves prompt tuning when used as a starting checkpoint (Section 4.5, Figure 10), demonstrating complementarity.

Concurrent work (Sanh et al., 2021; Min et al., 2021). Released after FLAN's initial preprint, these papers also study instruction-based multi-task fine-tuning for generalization. The paper acknowledges these in Appendix D.6 but notes that direct comparison is difficult due to differences in model size, architecture (decoder-only vs. encoder-decoder), pretraining data, and task mixtures. The convergent findings across independent groups strengthen the core claim: instruction-based fine-tuning improves zero-shot generalization, and this finding is robust across implementation choices.

How This Paper Positions Itself

The paper makes a specific, testable claim: instruction tuning is a method for improving zero-shot generalization, and its effectiveness depends on model scale and task diversity. This is not an architecture contribution, a new pretraining objective, or a new dataset. It's a procedure — take a pretrained LM, collect a diverse set of NLP datasets with instruction templates, and fine-tune on the mixture. The innovation is in:

1. The careful experimental design (holding out entire task clusters, not just datasets)
2. The empirical demonstration that scale matters critically (the 8B crossover point in Figure 7)
3. The framing of instruction tuning as a distinct paradigm bridging pretrain–fine-tune and prompting (Figure 2)
4. The breadth of evaluation across 62 datasets and 12 task clusters

The paper explicitly positions the work as an empirical study. It does not claim to have invented multi-task learning, instruction-based task formulation, or zero-shot evaluation. What it contributes is the systematic demonstration that at scale, instruction tuning on a diverse mixture produces a model that can follow natural language instructions for tasks it was never trained on, and that this capability is not present in smaller models — an emergent property that the paper documents rather than explains mechanistically. This empirical result became foundational for subsequent work on instruction-following models (InstructGPT, Alpaca, and the broader instruction-tuning paradigm that followed).

3. Technical Approach

3.1 Reader Orientation

This paper presents instruction tuning, a training procedure that converts a standard pretrained language model into one that can follow natural language instructions to perform tasks it has never seen during training. The system takes a 137B-parameter decoder-only language model (LaMDA-PT, which only generates continuations of text) and fine-tunes it on a diverse collection of NLP datasets where each input-output pair has been reformatted as an instruction followed by a response — for example, turning a translation pair into "Translate this sentence to French: 'The dog runs.' → 'Le chien court.'" The core problem it solves is the zero-shot gap: pretrained language models perform much worse when given only task descriptions (zero-shot) compared to when given task descriptions plus examples (few-shot), because instructions are a rare format in pretraining data. The solution's shape is to expose the model to enough instruction-response pairs across enough different task types during fine-tuning that it learns the general skill of "read the instruction, understand what task is being requested, and produce the appropriate output" — a skill that then transfers to instructions for tasks never explicitly trained.

3.2 Big-Picture Architecture (Diagram in Words)

The FLAN system has four major components:

1. A pretrained 137B-parameter decoder-only language model (LaMDA-PT) — this is the starting model, pretrained on 2.49 trillion BPE tokens from web documents, dialog data, and Wikipedia. It knows English and some non-English text (~10% non-English pretraining data), but it only knows how to continue text, not how to follow instructions. This serves as the foundation onto which instruction-following ability will be grafted.

2. A curated collection of 62 NLP datasets organized into 12 task clusters — these are existing public datasets (sentiment analysis, translation, NLI, etc.) that provide the supervision signal. Each dataset is reformatted using 10 manually written natural language instruction templates that describe the task in plain English (e.g., "Is the sentiment of this movie review positive or negative?"). Some templates "turn the task around" (e.g., generating a movie review for a given sentiment) to increase diversity.

3. An evaluation protocol based on held-out task clusters, not held-out datasets — the key design choice for measuring zero-shot generalization. The 12 clusters group datasets by task type (all NLI datasets together, all translation datasets together, etc.). To evaluate, the authors train a separate checkpoint for each cluster, holding that entire cluster out of the training mixture and training on all other clusters. This ensures the model has never seen any task of that type during instruction tuning.

4. A classification helper mechanism (the "options" suffix) — for classification tasks, the instruction is appended with "OPTIONS:" and the list of valid output classes. This constrains the decoder-only model's output space by making it aware of the permissible answers, avoiding the problem of probability mass being distributed across many alternative phrasings of the same answer.

Information flows as follows: raw NLP datasets are loaded → each example is formatted by randomly selecting one of ten instruction templates for its dataset → formatted examples from all datasets are mixed together into a single training stream → the pretrained LaMDA-PT model is fine-tuned on this mixture for 30,000 gradient steps → at inference time, a user provides a natural language instruction describing a task → the model generates a response in free text (no templates, no exemplars). For unseen tasks, the model relies on having learned the general mapping from instruction phrasing to task execution during fine-tuning.

3.3 Roadmap for the Deep Dive

The explanation proceeds in this order:

• First, the instruction tuning training procedure — how the dataset mixture is constructed, how templates are designed, how data balancing works, and what optimizer/hyperparameters are used. This is the core "what the system does during training" explanation.
• Second, the evaluation protocol using task clusters — how "unseen task" is defined, why holding out entire task clusters matters, and how the cross-validation-like rotation of held-out clusters works. This is essential because the entire paper's claims depend on the integrity of this split.
• Third, the classification options mechanism — the specific design for handling classification outputs from a decoder-only model, and why this was needed over standard rank classification. This is a practical detail that substantially affects results.
• Fourth, the model and pretraining details — the architecture, scale, pretraining data composition, tokenization, and training infrastructure. These technical specifications matter for reproducibility and for understanding the scaling results.
• Fifth, ablation decisions and design rationale — why specific design choices (10 templates per dataset, 30k examples cap, examples-proportional mixing, packing) were made and what alternatives they address. This connects implementation details to the paper's hypotheses about why instruction tuning works.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical methods paper whose core contribution is a training procedure — instruction tuning — plus a rigorous evaluation protocol for measuring zero-shot generalization across task types. The technical approach has no novel architecture, loss function, or dataset creation; its innovation lies in the combination of scale, task diversity, instruction formatting, and careful cluster-based evaluation.

3.4.1 The Instruction Tuning Training Procedure

The training procedure takes a pretrained language model and fine-tunes it on a mixture of instruction-formatted examples drawn from 62 NLP datasets spanning 12 task type clusters.

Dataset collection and task clusters. The authors aggregate 62 publicly available text datasets from TensorFlow Datasets (TFDS), covering both language understanding (NLU, blue in Figure 3) and language generation (NLG, teal in Figure 3). Each dataset is assigned to one of 12 task clusters based on the type of NLP task it represents. The full taxonomy, as shown in Figure 3 and detailed in Appendix G, comprises:

• Natural Language Inference (7 datasets: CB, ANLI R1-R3, MNLI, QNLI, RTE, SNLI, WNLI) — determining whether a hypothesis follows from a premise
• Reading Comprehension (5 datasets: DROP, BoolQ, MultiRC, OBQA, SQuAD) — answering a question given a passage containing the answer
• Closed-Book QA (3 datasets: NQ, ARC easy/challenge, TQA) — answering factual questions without access to a reference passage
• Translation (8 datasets: ParaCrawl EN/ES, EN/DE, EN/FR, WMT-16 EN/CS, EN/DE, EN/FI, EN/RO, EN/RU, EN/TR) — translating between language pairs
• Commonsense Reasoning (4 datasets: HellaSwag, CoPA, PiQA, StoryCloze) — physical/scientific reasoning requiring common sense
• Sentiment Analysis (4 datasets: Sent140, IMDB, SST-2, Yelp) — determining whether text expresses positive or negative sentiment
• Struct-to-Text (4 datasets: DART, CommonGen, E2ENLG, WebNLG) — generating natural language descriptions from structured data
• Coreference Resolution (3 datasets: Winogrande, DPR, WSC273) — identifying which expressions refer to the same entity
• Summarization (11 datasets: AG News, AESLC, CNN-DM, Gigaword, Multi-News, Newsroom, Opinion Abstracts (iDebate and Movie), SamSum, Wiki Lingua EN, XSum) — producing abbreviated summaries of longer texts
• Paraphrase Detection (4 datasets: QQP, MRPC, PAWS, STS-B) — determining whether two sentences are semantically equivalent
• Reading Comprehension with Commonsense (2 datasets: CosmosQA, ReCoRD) — combining reading comprehension with commonsense reasoning
• Miscellaneous (7 datasets: QuAC, CoQA, WIC, TREC, CoLA, Math, Fix Punctuation NLG) — conversational QA, word-in-context disambiguation, question classification, linguistic acceptability, math reasoning, and punctuation restoration

This clustering is the foundation of the evaluation protocol, as it defines what it means for a task to be "unseen" — if a dataset is in the same cluster as a training dataset, it counts as "seen" even if that specific dataset never appeared in training.

Instruction template design. For each of the 62 datasets, the authors manually compose exactly 10 unique natural language instruction templates that describe the task. Each template frames the input-output pair as an instruction to be followed. For example, for a natural language inference dataset, one template reads:

"Read the following and determine if the hypothesis can be inferred from the premise: Premise: <premise> Hypothesis: <hypothesis> OPTIONS: - yes - no"

while another might read:

"<premise> Based on the paragraph above, can we conclude that <hypothesis>? OPTIONS: - yes - no"

Figure 4 illustrates multiple such templates for a single NLI task, showing how the same premise-hypothesis pair can be expressed with different phrasing (different sentence structures, different placement of premise and hypothesis, different question framing).

Crucially, the instruction templates serve two purposes. The first is obvious: exposing the model to varied natural language phrasings of each task so it learns that different wordings can refer to the same underlying operation. The second is more subtle and is explicitly stated in Section 2.1:

"While most of the ten templates describe the original task, to increase diversity, for each dataset we also include up to three templates that 'turned the task around,' (e.g., for sentiment classification we include templates asking to generate a movie review)."

This means some templates ask the model to perform the inverse of the original task — generating an input that would produce a given output, or generating a review that expresses a given sentiment, or generating a question that has a given answer. This "task inversion" is a deliberate design choice to increase the diversity of the instruction-tuning signal and prevent the model from overfitting to a single input-output directionality for each task type.

Training mixture construction. The datasets vary dramatically in size — from a few hundred examples (CB has 250 training examples) to nearly 100,000 (NQ has 87,925). To prevent the largest datasets from dominating the training mixture, two mechanisms are applied:

1. Per-dataset example cap: The number of training examples from any single dataset is limited to 30,000. Datasets with more than 30,000 examples are truncated; smaller datasets are used in their entirety. This ensures that massive datasets like TriviaQA (87,622 examples) do not overwhelm smaller but semantically distinct datasets.

2. Examples-proportional mixing with a maximum mixing rate: Following the approach of T5 (Raffel et al., 2020), the sampling probability for each dataset is proportional to the number of examples it contributes, but with a ceiling. Specifically:

"We limit the number of training examples per dataset to 30k and follow the examples-proportional mixing scheme (Raffel et al., 2020) with a mixing rate maximum of 3k."

In this scheme, a mixing rate maximum of 3,000 means that any dataset does not receive additional sampling weight for examples beyond its first 3,000. So a dataset with 5,000 examples gets the same total sampling weight as a dataset with 30,000 examples — both are treated as having "3,000 effective examples" for mixing purposes. A dataset with only 500 examples would have proportional weight of 500/3000. This prevents dataset size from proportionally determining how often the model sees each task type.

Training hyperparameters (quoted from Section 2.4):

• Base model: LaMDA-PT, a dense left-to-right decoder-only transformer of 137B parameters
• Fine-tuning steps: 30,000 gradient steps
• Batch size: 8,192 tokens (total tokens per batch, not number of examples)
• Optimizer: Adafactor (Shazeer & Stern, 2018)
• Learning rate: 3e-5 (constant or with Adafactor's default schedule — the paper does not specify warmup or decay)
• Input sequence length: 1,024 tokens
• Target sequence length: 256 tokens
• Packing: Multiple training examples are combined into a single sequence by concatenating inputs and targets, separated by a special EOS token (following Raffel et al., 2020). This maximizes hardware utilization by avoiding padding waste on variable-length examples.
• Training time: Approximately 60 hours on a TPUv3 with 128 cores
• Checkpoint selection: The final checkpoint at 30,000 steps is used for all evaluations

The training objective. The model is fine-tuned using standard language modeling cross-entropy loss on the target tokens only. For each instruction-formatted example, the model is conditioned on the instruction (input) and trained to predict the response (target). The loss is:

$\\mathcal{L} = -\\sum_{t=1}^{T} \\log p(y_t | y_{<t}, x; \\theta)$

where $x$ is the instruction-formatted input sequence, $y = (y_1, \\ldots, y_T)$ is the target response token sequence, and $\\theta$ represents the model parameters.

What it computes: For each target token position $t$, the model produces a probability distribution over its vocabulary of 32,000 BPE tokens, conditioned on the instruction text and on all previously generated target tokens. The log-probability assigned to the actual target token is accumulated across all positions. This sum (negated) is the cross-entropy loss — lower loss means the model assigns higher probability to the correct responses.

Why this form: This is the standard autoregressive language modeling objective applied only to the target portion of the sequence. Training on target tokens only (not input tokens) prevents the model from wasting capacity on memorizing the instructions, which are arbitrary text not requiring prediction. The instruction provides conditioning context; the model's job is to learn producing the correct response given that context. An alternative would be to train on both input and target tokens equally (standard LM pretraining), but this would dilute the supervision signal by making the model predict instruction text rather than focusing on the instruction-to-response mapping.

3.4.2 The Evaluation Protocol: Task Cluster Holdout

The paper's central claim — that instruction tuning improves zero-shot performance on unseen tasks — requires a precise definition of "unseen." A naïve approach would hold out individual datasets (e.g., train on 61 datasets, test on 1). But this would not guarantee that the model hasn't learned to perform that type of task, because many NLP datasets share the same underlying task format. For instance, if the model was trained on SNLI (an NLI dataset) and tested on MNLI (another NLI dataset), it would have seen NLI during training even though it never saw the specific MNLI examples. Performance on MNLI would reflect transfer within a task type, not generalization to a new task type.

The paper therefore adopts a more conservative evaluation protocol:

"In this work, we only consider dataset D unseen at evaluation time if no datasets from any task clusters that D belongs to were seen during instruction tuning."

The cluster holdout rotation. To evaluate zero-shot performance across all 12 task clusters, the authors train a separate FLAN checkpoint for each cluster being evaluated. For a given evaluation cluster C:

1. All datasets in cluster C are completely excluded from the training mixture.
2. All datasets in all other clusters are included in the training mixture.
3. A model is fine-tuned on this mixture (the same 30,000-step procedure).
4. The resulting checkpoint is evaluated on all datasets in cluster C.

This means training 12 different checkpoints (one per held-out cluster). Each checkpoint sees a slightly different training mixture because a different cluster is omitted each time.

Special overlapping cluster handling. Two clusters require special treatment because they overlap with others conceptually:

• Reading Comprehension with Commonsense: When evaluating this cluster, both the Reading Comprehension cluster and the Commonsense Reasoning cluster are held out from training, because this hybrid cluster contains elements of both.
• Paraphrase Detection and NLI: These are held out from each other's training mixtures because paraphrase detection can be formulated as bidirectional entailment (sentence A entails B and B entails A), creating conceptual overlap.

These special rules are stated in a footnote in Section 2.2:

"When evaluating on the read. comp. with commonsense cluster, both read. comp. and commonsense reasoning were dropped from instruction tuning. Conversely, the read. comp. with commonsense cluster was not used for instruction tuning when evaluating on read. comp. or commonsense reasoning. We also drop the paraphrase cluster from instruction tuning when evaluating on NLI tasks and vice-versa."

Evaluation computation. For each test dataset, performance is reported in two ways:

1. Average template performance: The model is evaluated using each of the up to 10 instruction templates for that dataset (all templates designed for that cluster's datasets), and the mean performance across templates is reported. This proxies the expected performance given a "typical" natural language instruction, averaging over the specific wording choices.

2. Best dev template performance: A holdout development set (a small subset of the training data, typically 50-200 examples) is used to select the single template that performs best. That template's performance on the test set is reported. This proxies the performance achievable with modest prompt engineering (using a dev set to select the best instruction phrasing from a candidate pool of 10).

The distinction matters because it separates the model's robustness to instruction phrasing variation from its peak achievable performance. If average template performance is close to best dev template performance, the model is insensitive to specific wording. If there's a large gap, performance is brittle and depends on finding the right phrasing.

Comparison baselines. For context, the paper reports:

• LaMDA-PT zero-shot and few-shot: The untuned model evaluated using "the same prompts as GPT-3" (Section 3). A key detail from Appendix E (FAQ): LaMDA-PT cannot use the FLAN instruction templates because, without instruction tuning, it simply continues the text rather than following the instruction. As the FAQ explains:

"So because FLAN prompts are formulated as responding to an instruction, they do not work well for pretrained language models without finetuning. Performance was near zero for most generation tasks. For instance, given the input 'The dog runs.' Translate this sentence to French., LaMDA-PT continues with 'The dog runs after the cat' instead of actually translating the sentence."

Therefore, LaMDA-PT baselines use GPT-3-style prompts (designed to look like pretraining continuations), which represents the best available zero-shot performance without instruction tuning but uses a different prompt format than FLAN.

• GPT-3 175B zero-shot and few-shot: Numbers reported in Brown et al. (2020).
• GLaM 64B/64E zero-shot and one-shot: Numbers reported in Du et al. (2021), a mixture-of-experts model.

Few-shot FLAN evaluation (Section 4.4). In addition to the zero-shot evaluation, the paper studies how instruction tuning combines with few-shot exemplars at inference time. The format is:

"instruct(x_1) ⊕ y_1 ⊕ instruct(x_2) ⊕ y_2 ⊕ ... ⊕ instruct(x_k) ⊕ y_k ⊕ instruct(x)"

where $\\oplus$ denotes string concatenation with a delimiter token inserted between each segment, $x_i$ are example inputs, $y_i$ are corresponding outputs, and $x$ is the new input to classify. Exemplars are randomly drawn from the training set of the evaluation dataset. The number of exemplars $k$ is capped at 16 and also constrained such that the total sequence length (all instructions plus outputs plus delimiters) stays below 960 tokens. These exemplars are only used at inference time — the model was not trained with few-shot formatting for the held-out cluster.

A critical note: many of the evaluation datasets' training examples also appeared in instruction tuning for other clusters (when that cluster was not held out). The few-shot exemplars drawn at inference time for an unseen cluster come from that cluster's training set, which the model never saw during instruction tuning because the entire cluster was excluded.

3.4.3 Classification with the OPTIONS Suffix

Decoder-only language models like LaMDA-PT naturally generate free text, which is ideal for generation tasks (translation, summarization, struct-to-text) but creates a problem for classification tasks where the output must be one of a small set of discrete classes (e.g., "positive" or "negative").

The problem with standard rank classification. The established approach at the time (used by GPT-3, Brown et al., 2020) was rank classification: constrain the model to only the two valid answer tokens (e.g., "yes" and "no"), compute the probability assigned to each, and select the higher one. The paper identifies a subtle flaw in this approach:

"Though this procedure is logically sound, it is imperfect in that the probability mass for answers may have an undesired distribution among ways of saying each answer (e.g., a large number of alternative ways of saying 'yes' may lower the probability mass assigned to 'yes')."

The problem is that the model may have learned many ways to express the same answer during pretraining — "yes," "Yes," "YES," "yes indeed," "that's correct," "affirmative," and so on. Rank classification only looks at the probability assigned to the exact token "yes," which may be low even though the model is confident in the affirmative answer, because its probability mass is spread across synonyms and capitalization variants. This is the "surface form competition" problem (later studied explicitly in Holtzman et al., 2021).

The OPTIONS mechanism. FLAN's solution is to make the model aware of which outputs are acceptable before it generates its response. The instruction template ends with the token "OPTIONS:" followed by a list of the valid output classes for that task. For example, a sentiment analysis instruction reads:

"Movie review: This movie is the best RomCom since Pretty Woman. Is this review positive or negative? OPTIONS: - positive - negative"

The model then generates one of the listed options. Because the model is generating free text (not being forced to choose among a fixed token set), it can take advantage of its language modeling abilities to select the appropriate option based on context, while being constrained by the options list to avoid generating invalid classes.

Design rationale. The OPTIONS mechanism leverages the instruction-following behavior that instruction tuning instills — the model has learned to attend to the instruction text and produce responses that match the requested format. By listing the valid choices in the instruction itself, the model knows (1) what the output space is, (2) exactly how to format each valid answer, and (3) that it should not produce answers outside this set. This eliminates the surface form competition problem because the model is trained to output the exact option string as formatted.

For generation tasks (translation, summarization, QA), no OPTIONS suffix is needed since the output space is free-form text. The model simply generates the appropriate output given the instruction.

3.4.4 Model Architecture and Pretraining

The base model, LaMDA-PT, is described in Section 2.4 and draws on the LaMDA paper (Thoppilan et al., 2022). Understanding the pretrained model's properties is essential because instruction tuning is a fine-tuning procedure applied on top of a specific pretrained foundation — the behavior depends on both.

Architecture. LaMDA-PT is:

"a dense left-to-right, decoder-only transformer language model of 137B parameters"

"Dense" means every parameter is active for every input (as opposed to mixture-of-experts models like GLaM, which route inputs to subsets of parameters). "Decoder-only" means the model uses only the decoder stack of the original transformer architecture (Vaswani et al., 2017), with causal (autoregressive) self-attention that prevents each token from attending to future tokens. "Left-to-right" reinforces that it generates text sequentially from start to finish.

Pretraining data composition. The model is pretrained on:

"a collection of web documents (including those with computer code), dialog data, and Wikipedia, tokenized into 2.49T BPE tokens with a 32k vocabulary using the SentencePiece library (Kudo & Richardson, 2018)."

Several properties matter for FLAN's behavior:

• 2.49 trillion tokens — this is a very large pretraining corpus (approximately 5× larger than GPT-3's 500B tokens, as noted in Appendix C's data contamination analysis). This massive scale provides broad world knowledge and linguistic competence.
• ~90% English, ~10% non-English — the pretraining data contains some non-English text, which is why LaMDA-PT has nonzero translation capability even without instruction tuning, and why FLAN can perform translation tasks.
• Computer code included — this likely contributes to the model's ability to follow structured instructions and generate formatted outputs.
• Dialog data included — this may prime the model for the instruction-response conversational format used in instruction tuning.

A critical distinction: LaMDA-PT has only language model pretraining, not the dialog fine-tuning applied to the released LaMDA chatbot. The "PT" suffix stands for "pretraining," indicating this is the base pretrained checkpoint before any dialog-specific adaptation.

Tokenization. The SentencePiece tokenizer with 32k BPE vocabulary means the model operates on subword units. English text is typically split into common word pieces ("translation" might be one token, but rarer words might be split). Non-English text, especially in scripts not well-represented in the pretraining data, may be split into many tokens. This has implications for translation: translating into non-English languages from English may be harder because the target language requires generating more tokens and the model has seen fewer examples of those tokens' sequencing patterns during pretraining.

What instruction tuning adds to this base. The pretrained model has extensive knowledge (facts, linguistic patterns, reasoning heuristics) from its 2.49T-token pretraining, but it accesses this knowledge through a "completion" interface — it predicts what text typically follows a given prefix. Instruction tuning teaches an additional skill: mapping natural language instruction text to the appropriate task execution behavior. It does not teach new factual knowledge (30,000 gradient steps on limited task data cannot compete with the pretraining scale), but teaches the model to use its existing knowledge in a new way — to follow instructions rather than continue text.

This is why model scale matters (Section 4.2, Figure 7): the model needs enough capacity to absorb the 62 instruction-tuning tasks (which consume some parameters for task-specific patterns) while retaining sufficient residual capacity to generalize to new task types. Smaller models "fill up" with the training tasks and lose their ability to generalize.

3.4.5 Ablation Design and Key Configuration Choices

Beyond the main procedure, the paper uses ablation studies to investigate which aspects of instruction tuning are essential. Each ablation involves training a variant of FLAN with one component altered or removed, then evaluating on held-out clusters.

Number of instruction tuning clusters (Section 4.1). To study how task diversity affects generalization, the authors fix three held-out evaluation clusters (NLI, closed-book QA, and commonsense reasoning) and vary how many of the remaining seven clusters are included in instruction tuning. Clusters are added in decreasing order of number of datasets per cluster. This means:

• 1 cluster: only summarization (11 datasets)
• 2 clusters: summarization + translation (20 datasets total)
• 3 clusters: + reading comprehension (26)
• 4 clusters: + sentiment (30)
• 5 clusters: + struct to text (34)
• 6 clusters: + coreference (37)
• 7 clusters: + conversational QA (39)

The progressive addition tests whether diversity drives generalization (more clusters → better performance on held-out tasks) or whether there are diminishing returns or even negative transfer. The ordering by dataset count means the earliest additions provide the most training data volume; later additions provide more diversity but less volume.

Model scale ablation (Section 4.2). Using the same cluster split as above, the authors instruction-tune models at five scales: 422M, 2B, 8B, 68B, and 137B parameters. Each model follows the same training recipe (30k steps, same data mixture). The key design choice is that all models use the same pretraining architecture and data (just at different sizes), isolating the effect of scale from other confounds like pretraining data quality or architecture differences.

Role of instructions ablation (Section 4.3). To test whether instructions specifically matter (versus multi-task fine-tuning in general), the authors train two ablations:

• No template: Only inputs and outputs are provided during fine-tuning, with no instruction text. The model sees raw task data (e.g., translation input: "The dog runs." output: "Le chien court.").
• Dataset name: The input is prepended with a dataset identifier (e.g., "[Translation: WMT'14 to French] The dog runs."). This provides task-type information but without natural language instruction phrasing.

Both ablations are evaluated using FLAN-style instruction prompts at inference time (the no-template model would not know what task to perform without some instruction). The key design choice is that the evaluation format is held constant (instructions at inference) while the training format varies, isolating the effect of exposure to instructions during training.

Few-shot exemplar format (Section 4.4). When adding few-shot exemplars to FLAN inference, the format interleaves instructions with exemplars rather than using exemplars without instructions. The specific pattern is:

$\\text{instruct}(x_1) \\oplus y_1 \\oplus \\text{instruct}(x_2) \\oplus y_2 \\oplus \\ldots \\oplus \\text{instruct}(x)$

where $\\oplus$ is string concatenation with a delimiter token inserted between each segment. This differs from GPT-3's few-shot format, which does not include instruction text around each exemplar. The design rationale is that the model has been trained to respond to instructions, so even in a few-shot setting, the instruction framing provides helpful context. The exemplar cap (16 maximum, 960 total tokens) is a practical constraint based on the 1,024-token input length — it prevents the exemplars from crowding out the actual instruction and query.

Prompt tuning analysis (Section 4.5). To test whether instruction tuning makes the model more amenable to other forms of task specification, the authors apply prompt tuning (Lester et al., 2021) — optimizing continuous vectors prepended to the input — to both the instruction-tuned FLAN checkpoint and the base LaMDA-PT model. This tests whether the benefits of instruction tuning transfer to a completely different inference mechanism that doesn't use natural language. The prompt tuning setup uses a prompt length of 10, weight decay of 1e-4, no dropout on attention scores, and is applied to SuperGLUE tasks under the same cluster holdout protocol — when prompt-tuning on task T, no tasks from T's cluster were seen during instruction tuning.

3.4.6 Data Contamination Analysis Design

Because FLAN's pretraining data (2.49T tokens) may contain examples from the evaluation benchmarks, the paper includes a data contamination analysis in Appendix C, closely following the methodology of GPT-3 (Brown et al., 2020).

The detection procedure. For each evaluation dataset, the authors identify "dirty" examples — those where any n-gram (approximately n=13, varying by dataset, using the same n values as GPT-3) from the example's text overlaps with the pretraining corpus. N-grams are computed after splitting on spaces. A "clean" subset is created by removing all dirty examples.

The evaluation logic. Performance is computed on both the full dataset (clean + dirty) and the clean-only subset. If performance on the clean subset is meaningfully lower than on the full dataset, it suggests that data contamination inflated the reported results. If performance is comparable or higher, contamination is not driving the gains.

Key nuances. The paper notes several important caveats, echoing GPT-3's analysis:

• N-gram matching is conservative — it produces false positives (overlapping n-grams that don't actually represent memorization of the answer)
• Clean subsets may not be drawn from exactly the same distribution as the full dataset
• With 2.49T pretraining tokens, more false positives are expected than GPT-3's 500B-token analysis
• Two datasets (DROP and SQuADv2) had "almost total overlap" but manual inspection revealed that nearly all overlaps were in the context passages, not in question-answer pairs — the model gains background information but cannot memorize answers to specific questions

Two datasets (PIQA and ReCoRD) showed meaningfully lower performance on the clean subset and are marked with asterisks in the results table (Table 2), acknowledging that data contamination may have contributed to their reported scores.

4. Key Insights and Innovations

Innovation 1: Instruction Tuning as a Distinct Paradigm Bridging Pretrain–Fine-tune and Prompting

The paper's most conceptually significant contribution is the framing of instruction tuning as a third paradigm for using language models, distinct from both the pretrain–fine-tune approach that dominated 2018-2020 (BERT, T5) and the prompting approach popularized by GPT-3. This is not merely a training recipe — it is a re-conceptualization of what role labeled data can play in the era of large pretrained models.

Before FLAN, the field operated with two seemingly orthogonal success stories. On one side, the pretrain–fine-tune paradigm (Devlin et al., 2019; Raffel et al., 2020) used labeled data to create specialist models: fine-tune a pretrained checkpoint on task-specific examples, producing a model that excels at exactly one task. This worked extremely well on benchmarks but required per-task engineering, per-task labeled data, and per-task model serving. On the other side, the prompting paradigm (Brown et al., 2020) used a single pretrained model for many tasks without any gradient updates, conditioning on inference-time text interactions. This was elegant in its generality but left a large zero-shot performance gap, and the best results required few-shot exemplars and prompt engineering — implicit forms of task-specific effort.

The paper's framing in Figure 2 proposes that these paradigms are not mutually exclusive alternatives but axes that can be combined: instruction tuning uses the supervision mechanism of pretrain–fine-tune (gradient updates on labeled data) in service of the generality goal of prompting (one model for many tasks). The labeled data is not used to create a specialist for any particular task, but to teach a general skill — following instructions — that then transfers across tasks. This reframing was not obvious at the time. The dominant assumption was that multi-task fine-tuning improved performance on the training tasks themselves, with zero-shot transfer being a secondary bonus. FLAN inverts this: the training task performance is incidental; the zero-shot transfer is the objective.

This framing matters because it resolves a tension in how the field thought about labeled data for large LMs. If fine-tuning creates specialists, then scaling up pretraining (bigger models, more data) seemed like the only path to generalist models, and labeled data was valuable only for narrow deployment. Instruction tuning suggests instead that labeled data, when formatted as instructions across diverse tasks, can make pretrained models more general, not less. This insight directly enabled the subsequent wave of instruction-tuned models (InstructGPT, Alpaca, Vicuña, and their descendants) by establishing that supervised fine-tuning on task mixtures is not a retreat from generality but a path toward it.

The innovation is conceptual, not architectural. The paper did not invent multi-task learning, natural language task instructions, or zero-shot evaluation. What it contributed was the intellectual synthesis that these pieces form a coherent third paradigm, and the empirical demonstration that this paradigm is viable at scale. This is a fundamental reframing, not an incremental refinement — it changed how researchers think about what fine-tuning is for.

Innovation 2: Scale-Dependent Emergence of Instruction Generalization (the 8B Crossover)

Perhaps the most striking empirical finding in the paper — and the one with the deepest implications for how we understand language model capabilities — is that instruction tuning only improves zero-shot generalization above a critical model size, and actually hurts it below that threshold. Section 4.2 (Figure 7) shows that for models of 422M, 2B, and 8B parameters, instruction tuning on ~40 tasks reduces zero-shot performance on held-out tasks compared to the untuned base model. The benefit only emerges at 68B and 137B parameters.

This is not a quantitative detail — it is a qualitative phase transition in model behavior. The paper offers an interpretation:

"One potential explanation for this result could be that for small-scale models, learning the ~40 tasks used during instruction tuning fills the entire model capacity, causing these models to perform worse on new tasks. Under this potential explanation, for the larger scale models, instruction tuning fills up some model capacity but also teaches these models how to follow instructions, allowing them to generalize to new tasks with the remaining capacity."

This explanation — that generalization requires spare capacity beyond what is consumed by the training tasks — is a specific, falsifiable hypothesis about why emergent abilities arise in large models. It is not a vague appeal to "scale helps." It posits a concrete mechanism: at small scales, all parameters are consumed by learning task-specific patterns for the 40 training tasks, leaving no residual representational capacity to encode the abstract skill of instruction-following. At large scales, the model has enough parameters to both absorb the training tasks and extract the meta-pattern — the common structure across all instructions — that enables generalization.

Before this finding, the literature on multi-task learning and transfer generally assumed that more training tasks improved generalization monotonically (or at worst, hit diminishing returns). The GPT-3 paper had shown that few-shot capabilities improve with scale, but that is a different phenomenon — it measures in-context learning from exemplars, not generalization after fine-tuning. FLAN's result is distinct: it shows that the same fine-tuning procedure produces opposite effects depending on model size, with a sharp crossover around 8B parameters.

This finding has been enormously influential. It is one of the earliest documented cases of an emergent capability in language models — a behavior that is absent or counterproductive at smaller scales and only appears at larger scales. It parallels later findings about chain-of-thought reasoning (Wei et al., 2022), code generation, and other capabilities that exhibit similar phase transitions. The paper does not claim to explain why the crossover occurs at 8B rather than 1B or 68B, and the capacity-based explanation remains a hypothesis rather than a proven mechanism. But the empirical documentation of the phenomenon itself — with a clean ablation across five model sizes using the same architecture, data, and procedure — is a foundational contribution to the study of scaling laws for model capabilities.

This is a fundamental empirical discovery, not an incremental refinement. It changed the conversation from "multi-task fine-tuning helps" to "multi-task fine-tuning helps only at sufficient scale, and the threshold matters."

Innovation 3: Task Cluster Holdout as a New Rigor Standard for Zero-Shot Generalization

The paper introduces a methodological innovation in evaluation design that was largely absent from prior work on task transfer: holding out entire task clusters, not just individual datasets, when measuring zero-shot generalization. Prior work (and much subsequent work) typically defines an "unseen task" as a dataset that was not included in training. The paper argues — correctly — that this definition is too permissive:

"Whereas some prior work defines unseen tasks by disallowing the same dataset to appear in training, we use a more conservative definition that leverages the task clusters from Figure 3."

The reasoning is clear: if a model has been trained on MNLI (a natural language inference dataset) and is then tested on RTE (another NLI dataset), it has not seen RTE examples during training, but it has learned to perform the NLI task type. Good performance on RTE would reflect within-task-type transfer, which is less interesting than generalization to entirely new task types. The cluster holdout protocol ensures that at test time, the model confronts a genuinely unfamiliar kind of problem, not just unfamiliar examples of a familiar problem type.

This design choice makes FLAN's positive results substantially more meaningful than they would be under per-dataset holdout. If FLAN had merely transferred between NLI datasets, one might attribute its performance to surface-level pattern matching within the NLI format. By holding out all NLI datasets simultaneously, the paper ensures that the model must reconstruct the NLI task from the instruction text alone, using only its general instruction-following skill learned from entirely different task types (translation, summarization, sentiment analysis).

The paper also adds specific rules for conceptually overlapping clusters — holding out paraphrase detection when evaluating NLI, and vice versa — that show careful thinking about what "task type" means. Paraphrase detection (are these two sentences equivalent?) is bidirectional textual entailment, and a model trained on one could implicitly learn the other. By holding both out, the paper eliminates this leakage path.

This is a methodological advance that raised the bar for evaluating task generalization. It is incremental in the sense that it formalizes a best practice that some researchers may have intuited, but it is fundamental in its impact: subsequent work on instruction tuning and task generalization has largely adopted cluster-based evaluation (or at least acknowledges the distinction between within-cluster and cross-cluster transfer). The paper's careful cluster definitions and special overlap rules serve as a template for how to think about task similarity in generalization studies.

Innovation 4: Instruction Formatting as a Teachable Meta-Skill, Not Just a Prompt Engineering Trick

The ablation in Section 4.3 (Figure 8) reveals something non-obvious: it is not enough to train on multiple tasks; the instruction format itself is what enables zero-shot transfer. When the model is fine-tuned without instruction text (raw input-output pairs) or with only dataset name prefixes ("[Translation: WMT'14 to French]"), zero-shot performance on held-out tasks drops dramatically — from 55.2% average accuracy (FLAN with instructions) to 37.3% (no template) and 46.6% (dataset name).

This result separates instruction tuning from generic multi-task learning. Prior work on multi-task learning (Collobert et al., 2011; Luong et al., 2016; McCann et al., 2018) had shown that training on multiple tasks could improve performance on those tasks through shared representations. The implicit assumption was that the benefit came from the model learning shared low-level features across tasks — things like syntactic parsing, entity recognition, or semantic similarity that are useful for many NLP tasks.

FLAN's ablation suggests something different: the model is learning instruction following as a skill, not just benefiting from shared representations. The "dataset name" ablation is particularly informative. It provides task-type information (the model knows it is doing translation, or NLI, or sentiment analysis) and the same multi-task training signal as FLAN, but it lacks natural language instruction phrasing. The 8.6-percentage-point gap between FLAN (55.2%) and dataset-name fine-tuning (46.6%) shows that generic task metadata is not sufficient — the model needs to learn the mapping from natural language descriptions to task execution. The further drop with no template (37.3%) confirms that even task-type identification without instruction phrasing is insufficient.

This finding reframes what instruction tuning accomplishes. It is not that the model learns 62 tasks and happens to generalize — it is that the model learns a meta-skill of reading an instruction, identifying what is being asked, and producing the appropriate output format, and this meta-skill transfers because instructions share common linguistic structure across task types ("Is this X or Y?", "Translate A to B", "Answer the following question:", "Determine whether..."). The model learns to parse these instruction templates as task specifications, independent of the specific task content.

This insight is fundamental rather than incremental because it identifies what the model learns during instruction tuning — not task-specific patterns, but a generalizable instruction-parsing capability. It explains why instruction diversity matters (Section 4.1, Figure 6): more task clusters expose the model to more variation in how instructions are phrased, strengthening the general instruction-following skill. It also explains why few-shot exemplars complement instruction tuning (Section 4.4, Figure 9): exemplars provide task-specific formatting guidance that helps the model apply its general instruction-following skill to unfamiliar output formats.

Innovation 5: Verifier-Free Classification Through the OPTIONS Suffix

The paper introduces a small but practically important technique for classification tasks with decoder-only models: appending the valid output classes to the instruction via an "OPTIONS:" suffix. This is technically minor compared to the other innovations but conceptually distinctive because it addresses a specific failure mode — surface form competition in rank classification — without requiring an external verifier, task-specific output heads, or architectural modifications.

Prior work (GPT-3, Brown et al., 2020) used rank classification: constrain the model's output to a small set of tokens (e.g., "yes," "no"), compute their probabilities under the model, and select the highest. The paper identifies a flaw: the model's probability mass for an answer can be distributed across many synonymous phrasings, so the probability assigned to the exact token "yes" may underestimate the model's confidence in the affirmative. The OPTIONS suffix solves this by making the model generate one of the valid options as free text, informed by the list of choices in its context. The model can use its language modeling capability to select the appropriate option and output it in the exact format specified, avoiding the probability-mass fragmentation problem.

This is an incremental technical innovation that proved influential. It is not a fundamental reconceptualization, but it is a clean, effective solution to a real problem that subsequent instruction-tuned models widely adopted or adapted. It also reveals a deeper design principle: for instruction-tuned decoder-only models, constraining the output space through the instruction text is more natural than post-hoc probability ranking, because the model has been trained to respond to instructions, not to serve as a probability estimator. The OPTIONS mechanism coheres with the instruction-following paradigm in a way that rank classification — a technique from the pretrain–fine-tune era — does not.

The evidence for its effectiveness is indirect but pervasive: FLAN's strong classification results (NLI, sentiment, reading comprehension) all use the OPTIONS suffix, and the paper notes that without it, decoder-only models would need alternative mechanisms for handling discrete output spaces. The technique's influence is visible in later work on instruction-tuned models, where constrained decoding or options-based prompting became standard for classification tasks.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on 25 NLP datasets spanning natural language inference, reading comprehension, closed-book question answering, translation, commonsense reasoning, coreference resolution, sentiment analysis, paraphrase detection, struct-to-text, and reading comprehension with commonsense. All datasets are publicly available through TensorFlow Datasets or established benchmarks (WMT, SuperGLUE, etc.); details for each dataset appear in Appendix G. The test sets come from standard splits — typically the TFDS test set when available, or the TFDS validation set when a test set was not released, with the training portion split into train and dev (usually retaining ~200 examples for dev). The evaluation uses the specific test splits that correspond to those reported in GPT-3 (Brown et al., 2020) for comparability.

• Base model(s). The primary model is LaMDA-PT 137B, a dense, left-to-right, decoder-only transformer language model pretrained on 2.49 trillion BPE tokens from web documents, dialog data, and Wikipedia (~90% English, ~10% non-English, including computer code). For the scaling experiments (Section 4.2, Figure 7), variants at 422M, 2B, 8B, 68B, and 137B parameters from the same model family are used. The 137B scale is chosen because it is representative of the largest dense models available at the time and provides a meaningful comparison point against GPT-3 175B.

• Metrics. Accuracy (exact match) for classification and multiple-choice tasks, F1 for DROP, MultiRC, SQuADv1, and SQuADv2, and BLEU for translation. For each evaluation dataset, the paper reports two quantities: (1) average template performance — the mean score across all instruction templates (up to 10) designed for that dataset, and (2) best dev template performance — the score using the single template that achieves the highest result on the development set. The aggregations across datasets are simple arithmetic means within each task cluster.

• Baselines. Four baseline model configurations are compared against:

  • LaMDA-PT zero-shot — the untuned base model evaluated using GPT-3-style prompts (not FLAN instruction templates, since the FAQ in Appendix E notes that LaMDA-PT fails to follow instructions without tuning, e.g., continuing text rather than translating when given a translation prompt).
  • LaMDA-PT few-shot — the same untuned model with k few-shot exemplars (k ∈ {1, 3, 5, 10}, chosen as the largest k that fits within the 1024-token context length). Exemplars are formatted using GPT-3 prompts.
  • GPT-3 175B zero-shot and few-shot — numbers taken directly from Brown et al. (2020), evaluated with prompts optimized for GPT-3.
  • GLaM 64B/64E zero-shot and one-shot — numbers from Du et al. (2021), a mixture-of-experts model with 64B parameters but 64 experts. Additionally, Table 1 and Table 2 report supervised models (T5-11B or BERT-large fine-tuned on the full training set of each target task) as an upper-bound reference.

• Generation budget / compute accounting. All evaluations use greedy decoding for generative tasks (stated explicitly in Appendix A: "For simplicity, we use greedy search for all generative tasks"), except for qualitative open-ended generation examples where random sampling with temperature 0.9 and top-k 40 is used. The compute for instruction tuning is measured in gradient steps (30,000) and batch size (8,192 tokens), but no unit like FLOPs or generation count is used for inference cost comparison. The inference cost of FLAN is not directly compared against baselines in a FLOPs-matched sense — rather, all models are compared at equivalent evaluation regimes (zero-shot vs. zero-shot, few-shot vs. few-shot).

• Cross-validation / statistical protocol. No cross-validation is used for the main results. Each held-out task cluster evaluation uses a single checkpoint (the one trained with that cluster held out) and a single evaluation run per test set. The "best dev template" selection uses a fixed development split for each dataset (typically 50-200 held-out examples from the training set), chosen once and held constant across all models and ablations. No confidence intervals, error bars (except standard deviations across templates in Figure 9), or statistical significance tests are reported. The paper relies on the breadth of evaluation (25 datasets across diverse task types) to establish robustness rather than statistical rigor on individual datasets.

Main Quantitative Results

Natural Language Inference and Reading Comprehension

Figure 5 summarizes zero-shot results for NLI, reading comprehension, closed-book QA, and translation, with full per-dataset numbers in Table 1 and Table 2. The headline finding is that FLAN zero-shot outperforms GPT-3 175B zero-shot on 20 of 25 evaluated datasets, and surpasses GPT-3 175B few-shot on 10 datasets.

On natural language inference (7 datasets: ANLI R1-R3, CB, RTE, and additional ones in Table 2 including MNLI, QNLI, SNLI, WNLI), the gains are particularly striking. For ANLI R1, FLAN's best dev template achieves 47.9% accuracy versus GPT-3 zero-shot at 34.6% and GPT-3 few-shot [50] at 36.8% — FLAN zero-shot outperforms GPT-3 few-shot by over 11 percentage points. For ANLI R3, FLAN reaches 48.5% versus GPT-3 zero-shot 34.5% and GPT-3 few-shot 40.2%. On RTE, FLAN achieves 84.1% best-dev accuracy, compared to GPT-3 zero-shot at 63.5% and GPT-3 few-shot [8] at 72.9%, and also surpasses LaMDA-PT few-shot at 70.8%. The paper attributes this specifically to instruction phrasing (Section 3):

"we phrase NLI as the more natural question 'Does <premise> mean that <hypothesis>?', achieving much higher performance."

The improvement on NLI is not universal — LaMDA-PT already performs well on CB (46.4% zero-shot, 82.1% few-shot with k=7 using GPT-3 prompts), but FLAN's average template score of 64.1% shows high variance across templates for this small 56-example test set (standard deviation of 14.7 percentage points), with the best dev template matching GPT-3 few-shot at 83.9%.

On reading comprehension, the pattern is more mixed. FLAN outperforms GPT-3 on MultiRC (FLAN best dev: 77.5% F1 vs. GPT-3 zero-shot: 72.9% F1) and OBQA (FLAN best dev: 78.4% accuracy vs. GPT-3 zero-shot: 57.6%, vs. GPT-3 few-shot [70]: 65.4%). On BoolQ, FLAN best-dev achieves 82.9%, slightly exceeding GPT-3 zero-shot at 60.5% and GPT-3 few-shot [20] at 77.5%, but is comparable to LaMDA-PT zero-shot (81.0%) and LaMDA-PT few-shot (80.0%), suggesting the base model already performs well on this dataset without instruction tuning. On DROP and SQuADv2 (marked with † in Table 2 because GPT-3 used in-passage exemplars for these, making direct comparison invalid), FLAN shows lower absolute numbers (DROP: 22.7% F1; SQuADv2: 44.2% F1) but substantially improves over LaMDA-PT zero-shot (3.8% and 11.1% respectively). On SQuADv1, FLAN reaches 80.1% F1, compared to LaMDA-PT zero-shot at 22.7% — a dramatic improvement that validates instruction tuning's effectiveness for tasks where the base model has relevant knowledge but cannot express it without proper task formatting.

Closed-Book QA and Translation

On closed-book QA (4 datasets: ARC-easy, ARC-challenge, NQ, TQA), Figure 5 and Table 2 show that FLAN outperforms GPT-3 on all four. For ARC-c (the challenge set of grade-school science questions), FLAN best-dev achieves 63.1% versus GPT-3 zero-shot at 51.4%, and even surpasses GPT-3 few-shot [25] at 51.5%. For NQ, FLAN best-dev reaches 20.7% exact match versus GPT-3 zero-shot at 14.6%, though it trails GPT-3 few-shot [30] at 29.9%. On TQA (Wikipedia dev set), FLAN best-dev (68.1%) slightly exceeds GPT-3 zero-shot (64.3%) but trails GPT-3 few-shot (71.2%). Compared to GLaM, FLAN outperforms on ARC-e (79.6% vs. 76.6%) and ARC-c (63.1% vs. 50.3%) but is slightly behind on NQ (20.7% vs. 23.9%) and TQA (68.1% vs. 71.5%).

On translation (6 language pairs evaluated in the GPT-3 paper: WMT'14 Fr↔En, WMT'16 De↔En, WMT'16 Ro↔En), Table 1 shows FLAN zero-shot outperforms GPT-3 zero-shot on all six directions. For French to English, FLAN best-dev achieves 35.9 BLEU versus GPT-3 zero-shot at 21.2; for German to English, 38.9 BLEU versus 27.2. However, FLAN underperforms GPT-3 few-shot on most pairs — e.g., French to English: FLAN 35.9 vs. GPT-3 few-shot [64] 39.2; German to English: FLAN 38.9 vs. GPT-3 few-shot 40.6. Translating from English into other languages is consistently weaker (e.g., English to German: FLAN 27.0 BLEU vs. GPT-3 zero-shot 24.6, vs. GPT-3 few-shot 29.7), attributed largely to the English-dominant pretraining data and English-centric SentencePiece tokenizer. The paper also evaluates on additional translation pairs beyond those in the GPT-3 paper (WMT'16 EN↔CS, EN↔FI, EN↔TR, EN↔RU, ParaCrawl EN/ES), with results reported in Appendix G but not summarized in the main body.

Tasks Where Instruction Tuning Does Not Help

The paper explicitly highlights a negative result for tasks formulated as language modeling (sentence completions), where instructions are "largely redundant" with the pretraining objective. For commonsense reasoning and coreference resolution (7 datasets: CoPA, HellaSwag, PiQA, StoryCloze, DPR, Winogrande, WSC273), FLAN outperforms LaMDA-PT on only 3 of 7 tasks (Table 2). On HellaSwag, FLAN average-template accuracy is 56.4% versus LaMDA-PT few-shot [10] at 58.8% — a regression. On Winogrande, FLAN best-dev (71.2%) trails LaMDA-PT few-shot (68.4%) by a small margin but GPT-3 few-shot [35] (77.7%) by a larger one. However, StoryCloze is a notable exception: FLAN best-dev achieves 93.4%, surpassing GPT-3 few-shot [5] at 87.7% and approaching supervised BERT-large at 89.2%. The paper's interpretation is that for tasks where the prompt is simply "finish this sentence/paragraph," the instruction format provides no benefit over the model's native completion behavior.

Additional Task Clusters

Results for sentiment analysis, paraphrase detection, struct-to-text, reading comprehension with commonsense, and miscellaneous tasks appear in Table 1 and Table 2. On sentiment analysis, FLAN achieves strong results: 94.3% on IMDB (vs. LaMDA-PT zero-shot 76.9%), 94.6% on SST-2 (vs. 51.0%), and 98.1% on Yelp. For struct-to-text (CommonGen, DART, E2ENLG, WebNLG), FLAN dramatically improves over LaMDA-PT zero-shot, which performs near zero on these tasks — e.g., DART Rouge-2: FLAN best-dev 30.0 vs. LaMDA-PT zero-shot 1.5. However, no GPT-3 results are available for these tasks, so the comparison is only against the untuned base model. For reading comprehension with commonsense (CosmosQA, ReCoRD), FLAN best-dev reaches 60.6% and 72.5% respectively, compared to LaMDA-PT zero-shot at 34.1% and 87.8% (the LaMDA-PT zero-shot result on ReCoRD is marked with an asterisk due to data contamination concerns — see Appendix C).

Few-Shot FLAN Results (Section 4.4)

When few-shot exemplars are added to FLAN inference (Figure 9), performance improves across all task clusters. The average zero-shot FLAN performance across the eight NLU clusters shown is 54.7%, while few-shot FLAN reaches 59.3% — a 4.6 percentage point gain. Exemplars are particularly helpful for struct-to-text (31.0% → 49.4%, a gain of 18.4 points), translation (53.7% → 57.2%), and closed-book QA (59.6% → 60.0%). The paper also notes that standard deviation among templates is lower for few-shot FLAN, indicating reduced sensitivity to prompt engineering. The exemplar budget is capped at 16 exemplars and a total sequence length of 960 tokens (below the 1024-token input limit).

Ablation: Number of Instruction Tuning Clusters (Section 4.1)

Figure 6 shows the effect of progressively adding task clusters to instruction tuning (using a fixed split where NLI, closed-book QA, and commonsense are held out for evaluation). With only 1 cluster (summarization, 11 datasets), average held-out performance is 49.9%. Adding clusters monotonically improves performance: 2 clusters = 55.0%, 3 = 59.3%, 4 = 59.2% (a slight dip with sentiment), 5 = 60.8%, 6 = 61.9%, 7 = 63.5%. The 13.6-percentage-point gain from 1 to 7 clusters confirms that task diversity drives generalization. Notably, the performance does not appear to saturate, suggesting further gains with even more clusters. The sentiment analysis cluster (added from 3 to 4 clusters) provides minimal improvement (59.3% → 59.2%), which the paper notes but does not explain in detail.

Ablation: Model Scale (Section 4.2)

Figure 7 shows the most important ablation result. For models of 422M, 2B, and 8B parameters, instruction tuning reduces zero-shot performance on held-out tasks compared to the untuned base model. The untuned baseline (averaged across 13 held-out tasks) starts at approximately 31% for 422M and rises to roughly 33% for 137B — a flat scaling curve. The instruction-tuned models show a sharply different pattern: performance drops from approximately 31% at 422M to roughly 27% at 8B, then jumps to approximately 43% at 68B and 47% at 137B. The crossover point where instruction tuning becomes beneficial is between 8B and 68B parameters. This is described in the paper as:

"instruction tuning actually hurts performance on held-out tasks. One potential explanation for this result could be that for small-scale models, learning the ~40 tasks used during instruction tuning fills the entire model capacity, causing these models to perform worse on new tasks."

Ablation: Role of Instructions During Fine-Tuning (Section 4.3)

Figure 8 compares three training configurations, all evaluated with natural language instructions at inference time, across four held-out clusters (NLI, reading comprehension, closed-book QA, translation). FLAN with instruction templates achieves an average of 55.2%. When trained without any instruction text ("no template") — only raw inputs and outputs — performance drops to 37.3%. When trained with dataset name prefixes ("[Translation: WMT'14 to French] The dog runs."), performance reaches 46.6%. A fourth configuration, training with dataset name prefixes and evaluating with dataset name prefixes (not natural instructions), achieves 47.0% — slightly higher than dataset-name training with instruction evaluation, but still 8.2 points below FLAN. These gaps demonstrate that natural language instructions during training are essential for the zero-shot instruction-following behavior.

The per-cluster breakdown in Table 3 reveals important heterogeneity: the "no template" ablation performs worst on closed-book QA (25.5% vs. FLAN's 56.6%) and translation (15.0% vs. 30.7%), suggesting these task types particularly depend on instruction-based task disambiguation. The "task/dataset name" ablation performs better on NLI (52.8% vs. FLAN's 56.2%) but much worse on reading comprehension (63.0% vs. 77.4%).

Instruction Tuning Facilitates Prompt Tuning (Section 4.5)

Figure 10 and Table 4 show that prompt tuning (optimizing continuous soft prompts) works substantially better starting from the FLAN checkpoint than from LaMDA-PT. On the SuperGLUE dev set, with only 32 training examples per task, FLAN achieves 63.8% average performance versus LaMDA-PT's 50.0%. With the full training set, FLAN reaches 87.4% versus LaMDA-PT's 79.1%. The largest gains in the low-resource setting are on RTE (83.0% vs. 52.4%) and BoolQ (77.5% vs. 55.5%). The prompt tuning experiments follow the same cluster holdout protocol — when tuning on task T, no tasks from T's cluster were seen during FLAN's instruction tuning.

Data Contamination Analysis (Appendix C)

Figure 12 and Table 5 report the overlap analysis. Across the 25 evaluated datasets, the percentage of "clean" (uncontaminated) examples ranges from 0.6% (DROP) to 99.8% (Winogrande). The paper finds no consistent pattern where performance is higher on dirtier datasets. Two datasets show meaningfully lower clean-subset performance: PIQA (clean accuracy 23.3% vs. full accuracy 23.7%, a -1.7% difference) and ReCoRD (clean accuracy 4.5% vs. full accuracy 4.6%, a -2.7% difference). These are flagged with asterisks in Table 2. DROP and SQuADv2 have near-total overlap (0.6% and 0.9% clean respectively), but manual inspection reveals that the overlaps are almost entirely in context passages (99.6% for DROP, 97.2% for SQuADv2), not in question-answer pairs — meaning the model gains background information but cannot have memorized specific answers, aside from 5 cases in SQuADv2.

Ablation Studies and Robustness Checks

• Datasets per task cluster vs. templates per dataset (Appendix B.1, Figure 11): When the number of instruction tuning clusters is held constant, adding more datasets per cluster improves performance substantially (approximately 10 percentage points on average across three held-out clusters). In contrast, varying the number of instruction templates per dataset (1, 4, or 10) has a negligible effect, especially when there are sufficient datasets per cluster. At one dataset per cluster, 10 templates provides a small boost over 1 template (~2 points), but at four datasets per cluster, there is virtually no difference. This is non-obvious: the paper's initial motivation assumed 10 templates would prevent overfitting to any particular phrasing, but the results suggest the model at 137B scale does not easily overfit to individual instruction wordings when given diverse tasks.

• Best dev template vs. average template: Across almost all datasets, the best dev template exceeds the average template performance, but the gap varies substantially. For NLI datasets, the gap is moderate: ANLI R1 averages 47.7% across templates vs. 46.4% best-dev (actually slightly lower — indicating the best template on dev was not the best on test), while CB shows a 19.8-point gap (64.1% average vs. 83.9% best-dev) driven by high template variance on the tiny 56-example CB test set. For translation, the gap is smaller (WMT'14 En→Fr: 32.9% average vs. 33.9% best-dev), suggesting translation is less sensitive to instruction phrasing than NLI. The large standard deviations in Table 2 (e.g., ANLI R1: ±1.4% for average template) indicate that template choice meaningfully affects results, motivating the best-dev-template metric as a practical upper bound.

• Few-shot exemplar ordering and count: The few-shot FLAN experiments (Section 4.4, Figure 9) use randomly drawn exemplars from the training set, capped at 16 and bounded by a 960-token total sequence length. No ablation on the number of exemplars or their ordering is performed. The paper does not explore whether a different few-shot format (e.g., without interleaved instructions) would work better or worse, though the format was chosen based on the intuition that instruction-tuned models benefit from instruction framing around each exemplar.

• Learning rate and optimizer: Only a single optimizer configuration is tested: Adafactor with a learning rate of 3e-5. No learning rate sweep or optimizer comparison is reported. The 30,000-step checkpoint is used for all evaluations without early stopping or checkpoint selection based on validation performance on the instruction tuning tasks.

• Effect of packing: The paper uses sequence packing (combining multiple examples into one sequence separated by EOS tokens) following T5 (Raffel et al., 2020). No ablation on packing vs. padding is performed, and the paper does not report whether packing affects instruction-tuning performance.

Critical Assessment

The paper's central claims, as established in prior sections, are: (1) instruction tuning substantially improves zero-shot performance on unseen task types, (2) FLAN surpasses GPT-3 zero-shot on most evaluated tasks and GPT-3 few-shot on several, (3) the benefits of instruction tuning emerge only at sufficient model scale (~68B+ parameters), (4) natural language instructions during fine-tuning are essential, and (5) increasing task diversity improves generalization.

Claim 1 (instruction tuning improves zero-shot performance): Supported with a major caveat around task type. The paper demonstrates substantial improvements over LaMDA-PT zero-shot on most NLP tasks (NLI, reading comprehension, closed-book QA, translation, struct-to-text). However, for tasks formulated as sentence completions (commonsense reasoning and coreference resolution), instruction tuning provides minimal or negative benefit — FLAN outperforms LaMDA-PT on only 3 of 7 such tasks. The paper is transparent about this limitation (Section 3: "this negative result indicates that when the downstream task is the same as the original language modeling pre-training objective... instruction tuning is not useful"), but it means the "improves zero-shot performance" claim is conditional on task format. The method works when the task can be naturally expressed as an instruction that differs from the pretraining continuation format; it fails when the task is already a natural continuation. The paper does not operationalize this distinction prospectively — it is discovered post-hoc — and does not quantify what fraction of real-world NLP tasks fall into each category.

Claim 2 (FLAN surpasses GPT-3 zero-shot on 20/25 datasets and GPT-3 few-shot on 10): The comparison is numerically accurate per Table 1 and Table 2, but two important qualifications apply. First, LaMDA-PT and GPT-3 are different models with different pretraining data, architectures, and tokenizers. Differences in performance could reflect pretraining quality rather than instruction tuning. The paper acknowledges this implicitly by reporting LaMDA-PT baselines, which isolate the effect of instruction tuning on the same base model. The LaMDA-PT zero-shot vs. FLAN zero-shot comparison is the cleanest test of instruction tuning's effect, and that comparison consistently favors FLAN on instruction-friendly task types.

Second, the "few-shot" comparison is not strictly matched. GPT-3 chooses the optimal number of exemplars k by dev set performance and can use up to 70 exemplars (as with OBQA, where GPT-3 few-shot uses k=[100]), while FLAN few-shot uses a maximum of 16 exemplars capped by the 960-token limit. FLAN outperforming GPT-3 few-shot despite using far fewer exemplars (e.g., on ANLI: FLAN zero-shot vs. GPT-3 few-shot with 50 exemplars) is genuinely impressive and strengthens the paper's claim. However, the exemplar budgets are not matched, making it not an apples-to-apples comparison. A fairer comparison would give GPT-3 the same exemplar budget as FLAN, but this is not done.

Claim 3 (benefits emerge only at scale, ~68B): This is the paper's most scientifically significant finding and is cleanly demonstrated in Figure 7. However, several limitations exist. The experiment uses only one model family (LaMDA-PT) with one pretraining data mix. It is unknown whether the crossover point would shift with different architectures (encoder-decoder vs. decoder-only), different pretraining data quality, or different instruction tuning task mixtures. The paper tests 422M, 2B, 8B, 68B, and 137B — a large gap between 8B and 68B leaves uncertainty about where exactly the crossover occurs (anywhere between 8B and 68B). A model at ~20-30B parameters would have helped locate the threshold more precisely. The paper's capacity-based explanation (small models "fill up" with training tasks) is speculative and not directly tested — no probing experiments measure whether small instruction-tuned models lose pretraining knowledge or simply fail to acquire the instruction-following meta-skill.

Claim 4 (natural language instructions are essential): Figure 8 provides clean support: removing instructions from fine-tuning drops performance from 55.2% to 37.3% (no template) or 46.6% (dataset name). However, the evaluation always uses natural language instructions for the two ablations. The "no template" model, having never seen an instruction during training, is being asked to perform a task in a format it was not trained for — its poor performance partially reflects this train-test format mismatch. The "dataset name" ablation is more informative because it harmonizes train and test: when trained and evaluated with dataset name prefixes, performance is 47.0%, still well below FLAN's 55.2%. This suggests instructions provide genuine value beyond just task identification, but the absolute gap (8.2 points) is modest relative to the massive gain over zero-shot LaMDA-PT (~20+ points). A missing ablation would be: training with instructions but evaluating with dataset names (to test whether the model has learned a general task-mapping ability vs. instruction-specific formatting).

Claim 5 (task diversity improves generalization): Figure 6 clearly shows monotonic improvement with more clusters (49.9% → 63.5%). However, the cluster ordering is fixed (added by decreasing dataset count), confounding diversity with total training data volume. Adding translation (9 datasets, going from 1 to 2 clusters) provides a large boost, but it is impossible to disentangle whether the gain comes from adding translation specifically or from simply having more training data. A controlled ablation that adds clusters while holding total training examples constant would distinguish these effects but is not performed. The lack of saturation at 7 clusters is suggestive but not conclusive — the range tested (1-7 clusters) may simply be below the saturation point for the 137B model.

Missing baselines and experiments that would strengthen the paper:

• No matched compute or data budget comparison: The paper does not compare instruction tuning to simply continuing pretraining with the same number of gradient steps (30,000) or the same number of tokens. The LaMDA-PT zero-shot baseline has no fine-tuning at all, making it a weak comparison — instruction tuning adds 30,000 steps of additional training, and some fraction of the gain could be from the additional steps rather than the instruction formatting specifically. A baseline of "continue LM pretraining for 30,000 more steps, then evaluate zero-shot" would address this.

• No per-template variance analysis: The paper reports average-template performance and standard deviations but does not analyze which types of templates work best or why. Understanding which instruction phrasings succeed or fail would provide insight into what the model learns and inform template design.

• Single evaluation language (English): All instruction templates and evaluations are in English, despite the model having ~10% non-English pretraining data. Cross-lingual instruction following (e.g., giving instructions in French to perform tasks in French) is not tested, though the translation results suggest the model has some multilingual capability.

• No human evaluation of instruction quality: The paper manually composes 10 templates per dataset but does not report inter-annotator agreement, template quality metrics, or whether templates were iteratively refined based on model behavior. This injects unreported experimenter degrees of freedom that could inflate best-dev-template results.

• Small test sets for some tasks: CB has only 56 test examples, WSC273 has no training data and a tiny test set, and COPA has 100 test examples. Results on these datasets have high variance (e.g., CB average template standard deviation of 14.7 points, Figure 9 error bars visible on NLI). The 20-of-25 and 10-of-25 counts include these small datasets and could change with different test splits.

Conditional nature of the claims: The paper's claims hold under specific conditions: (1) the task type can be expressed as natural language instructions that differ from pretraining continuations, (2) the model is sufficiently large (above some threshold between 8B and 68B), (3) enough diverse task clusters are available for instruction tuning (at least 5-7 in this study, with no evidence of saturation), and (4) a small development set is available for template selection if peak performance is desired. Outside these conditions — small models, sentence-completion tasks, low-resource settings with few training task clusters — instruction tuning provides minimal or negative benefit.

6. Limitations and Trade-offs

Model Scale Requirement: Instruction Tuning Is Counterproductive Below ~68B Parameters

The assumption or constraint. The paper's headline finding is that instruction tuning substantially improves zero-shot generalization — but this is true only for models of sufficient scale. Section 4.2 (Figure 7) demonstrates a sharp crossover: for models of 422M, 2B, and 8B parameters, instruction tuning on ~40 tasks actually reduces zero-shot performance on held-out tasks compared to the untuned base model. The benefit only emerges at 68B and 137B parameters. The paper acknowledges this explicitly:

"The behavior on held-out tasks for the 8B and smaller models, however, is thought-provoking—instruction tuning actually hurts performance on held-out tasks."

And offers only a hypothesized explanation:

"One potential explanation for this result could be that for small-scale models, learning the ~40 tasks used during instruction tuning fills the entire model capacity."

The consequence. This means instruction tuning is not a general-purpose technique that improves any language model — it is specifically a technique for large language models, and deploying it on a smaller model (e.g., a 7B-parameter model that might be practical for on-device deployment or low-latency serving) actively degrades generalization. A practitioner with a sub-68B model who applies instruction tuning hoping for better zero-shot performance will get worse zero-shot performance than if they had simply left the model as-is and used GPT-3-style prompting. This fundamentally limits the accessibility of the method: instruction tuning is only available to teams that can pretrain (or afford to fine-tune) models at the ~70B+ scale, which in 2022 meant a small number of industrial labs.

Furthermore, the crossover point is only coarsely located. The experiment tests 422M, 2B, 8B, 68B, and 137B parameters — a gap of 60B parameters between the largest model that fails (8B) and the smallest model that succeeds (68B). A practitioner using a 13B, 30B, or 65B model cannot determine from this study whether instruction tuning will help or hurt, because the crossover could occur anywhere in that range. The paper's capacity-based hypothesis — that small models "fill up" with the training tasks — is not directly tested through any probing or capacity-measurement experiment, leaving the mechanism unexplained.

What evidence exists in the paper. Figure 7 is the sole source of evidence: average zero-shot accuracy on 13 held-out tasks across five model sizes. The untuned baseline is roughly flat with scale (~31-33%), while the instruction-tuned line drops from ~31% at 422M to ~27% at 8B, then jumps to ~43% at 68B and ~47% at 137B. No intermediate model sizes between 8B and 68B are tested. Table 2 provides full per-dataset results at 137B only — there is no per-dataset breakdown at smaller scales to show which held-out tasks suffer most from the regression.

Mitigation status. The paper does not attempt to mitigate this limitation — no architectural modifications, training curriculum changes, or regularization techniques are explored to make instruction tuning effective at smaller scales. The capacity hypothesis is presented as a potential explanation, not as a diagnosis with a proposed remedy. Section 6 lists "model scale of FLAN 137B makes it costly to serve" as a limitation but does not connect this back to the scaling ablation or propose methods to lower the scale threshold. The finding is treated as an empirical observation rather than a problem to solve.

Difficulty Estimation and Template Sensitivity Are Unaccounted Costs

The assumption or constraint. The paper reports two different performance metrics: average template performance (the mean across up to 10 instruction templates per dataset) and best dev template performance (the single template that performs best on a held-out development set). The gap between these metrics is substantial for many tasks — e.g., CB shows 64.1% average vs. 83.9% best dev (a 19.8 percentage point gap), and RTE shows 78.3% average vs. 84.1% best dev (Table 2). The best dev template metric assumes access to a development set with ground-truth labels to select the optimal instruction phrasing for each task.

The consequence. The best-dev-template numbers — which are the figures used in the headline comparisons against GPT-3 ("FLAN outperforms zero-shot GPT-3 on 20 of 25 datasets") — represent an upper bound achievable only with task-specific labeled data and template engineering. In a genuine zero-shot deployment (no labeled data, no dev set, no per-task tuning), the user writes a single instruction and gets one shot. The paper's average-template metric estimates this scenario, but even this assumes the user's instruction is drawn from the same distribution as the 10 manually composed templates — which requires that the user has the template-design expertise to write natural instructions that the model responds to. A naive user who writes an awkwardly phrased instruction may get performance well below the average-template numbers.

This means the reported gains over GPT-3 are partly attributable to template engineering amortized across evaluation — the authors wrote 10 templates, tested them all, and report both the average and the best. GPT-3's reported zero-shot numbers come from prompts that Brown et al. (2020) optimized through prompt engineering, but the prompt engineering effort is not directly comparable because the prompt formats differ (completion-style vs. instruction-style). A fairer comparison would be: one instruction written by a user with no access to a dev set (FLAN) vs. one prompt written by a user with no access to a dev set (GPT-3). The paper does not perform this comparison, making the zero-shot performance claims somewhat optimistic.

What evidence exists in the paper. The standard deviations in Table 2 reveal the variance: for instance, CB has a standard deviation of ±14.7 points across templates, MNLI-m has ±6.2, WNLI has ±10.6, QQP has ±6.8, and PAWS Wiki has ±6.5. Figure 9 explicitly shows these standard deviations as error bars (orange), confirming that zero-shot FLAN has high template sensitivity. The few-shot FLAN bars show consistently lower standard deviation — a finding the paper notes — but few-shot requires exemplars, which undercuts the zero-shot framing.

Mitigation status. The paper does not fully mitigate this. Reporting both average and best-dev metrics is transparent, but the paper does not analyze which template characteristics drive the variance or provide guidelines for writing effective instructions without a dev set. Section 4.4 (Figure 9) shows that few-shot exemplars reduce template sensitivity (lower standard deviations), but this is not a zero-shot solution. Appendix B.1 (Figure 11) shows that using 10 templates vs. 1 template during training has negligible effect on held-out performance when there are sufficient datasets per cluster, which is informative but does not address inference-time template sensitivity. The paper effectively acknowledges that the best-dev numbers require labeled data without characterizing the cost of obtaining it.

Single Model Family and Single Prompt Paradigm Leave Generality Unproven

The assumption or constraint. All experiments use a single model family: LaMDA-PT, a dense, decoder-only, left-to-right transformer of 137B parameters pretrained on a specific mixture of web documents, dialog data, and Wikipedia with ~90% English and ~10% non-English text. The paper does not test instruction tuning on encoder-decoder architectures (T5), mixture-of-experts models (GLaM, Switch Transformer), or models with different pretraining objectives (e.g., denoising autoencoding as in BART). The paper argues this model is "representative" (Section 4), but provides no cross-architecture evidence.

The consequence. It is unknown whether instruction tuning's benefits are specific to decoder-only autoregressive models, which have a natural "completion" interface that might interact with instruction formatting in particular ways. An encoder-decoder model like T5, which is pretrained with a different objective (span corruption) and has a different generation mechanism (bidirectional encoder, autoregressive decoder), might respond differently to instruction tuning. For instance, T5's pretraining already includes task-specific prefixes ("translate English to German:", "summarize:"), which might partially replicate the instruction-tuning effect without additional fine-tuning. Conversely, T5 might benefit more from natural language instructions because its pretraining prefixes are not natural language. The paper cannot distinguish these possibilities.

More practically, a practitioner using a non-LaMDA model cannot assume the 137B results will transfer. If instruction tuning behaves differently on encoder-decoder models — perhaps the crossover point in Figure 7 shifts, or the optimal number of training clusters changes — then the paper's scaling and diversity findings may not apply. The paper's entire evidence base for "instruction tuning works" is contingent on the LaMDA-PT architecture and pretraining data.

What evidence exists in the paper. None. No cross-architecture experiments are performed. The related work (Appendix D.6) cites Sanh et al. (2021) as having applied instruction tuning to T5-11B (an encoder-decoder model) with positive results, and Mishra et al. (2021) as having applied it to BART (also encoder-decoder), but these are cited as convergent findings without analysis of whether the results differ systematically across architectures. The paper's own experiments are exclusively on LaMDA-PT variants.

Mitigation status. The paper does not mitigate this limitation. It briefly acknowledges in Section 6 that "future work on instruction tuning could include... cross-lingual experiments" but does not mention cross-architecture experiments. The concurrent work citations (Sanh et al., 2021) provide some external validation that instruction tuning can work on other architectures, but since the training data, task mixtures, instruction formats, and model scales differ, one cannot conclude that the specific empirical regularities reported in this paper (the 8B crossover, the effect of adding clusters, the per-task-type results) would replicate.

The Hardest Tasks Show No Benefit — Instruction Tuning Cannot Create New Capability

The assumption or constraint. The paper's method relies on the base model possessing some latent ability to perform the target task — the instruction-tuning step reformats the input to make that latent ability accessible, but it does not teach fundamentally new knowledge or skills that the pretrained model lacks. This is visible in the results pattern: where the base model has near-zero performance, instruction tuning provides near-zero benefit.

The consequence. The clearest example is commonsense reasoning and coreference resolution tasks formulated as sentence completions. Table 2 shows that on 4 of 7 such tasks (HellaSwag, PIQA, Winogrande, DPR), FLAN performs comparably to or worse than the untuned LaMDA-PT, and the paper explicitly acknowledges:

"This negative result indicates that when the downstream task is the same as the original language modeling pre-training objective (i.e., in cases where instructions are largely redundant), instruction tuning is not useful."

But this is not just about sentence completions — it reveals a fundamental capability bound. Instruction tuning amplifies existing capability by making it accessible through a new interface (instructions), but it does not create capability that the pretrained model lacks. This is analogous to the finding in the compute-optimal test-time scaling paper (Snell et al.) that on the hardest problem quintile (bin 5), no amount of test-time compute helps because the base model's pass@1 is effectively zero — there are no correct solutions in the proposal distribution to find.

For a practitioner, this means instruction tuning cannot be used to extend a model to genuinely novel task types where the underlying reasoning or knowledge is absent from pretraining. If the base model cannot perform arithmetic, instruction tuning on addition problems (phrased as instructions) will not teach it to add — the model needs that capacity from pretraining. The paper provides no diagnostic for determining which capabilities are latent ("the model knows but doesn't know how to express it") versus absent ("the model genuinely cannot do this"), making it hard to predict whether instruction tuning will help for a new task type.

What evidence exists in the paper. The sentence-completion results in Table 2 are the primary evidence: on HellaSwag, FLAN average-template accuracy is 56.4% vs. GPT-3 zero-shot at 78.9% and GPT-3 few-shot at 79.3% — the instruction format actively hurts relative to completion-style prompting. On DPR, FLAN is at 60.3% vs. LaMDA-PT few-shot at 57.3% — a small gain that doesn't approach GPT-3's numbers. On Winogrande, FLAN best-dev (71.2%) trails GPT-3 few-shot (77.7%). The translation results provide corroborating evidence: FLAN translates well into English (benefiting from English-dominant pretraining) but poorly from English into other languages, where the base model has weaker generation capability in the target language.

Mitigation status. The paper is transparent about this limitation, stating it explicitly in Section 3 and providing the negative results in Table 2. However, it does not propose any method to overcome it — for instance, augmenting instruction tuning with a small amount of target-task training data to teach novel capabilities, or using the instruction format to elicit chain-of-thought reasoning that might help the model access knowledge it possesses but cannot express directly. The limitation is documented but not addressed.

Difficulty Estimation Cost and Template Design Effort Are Not Accounted For

The assumption or constraint. The paper's evaluation protocol requires:

1. Template design: For each of the 62 training datasets plus each of the ~25 evaluation datasets, 10 manually composed natural language instruction templates must be written. The paper states this was done by the authors (Section 2.1: "we manually compose ten unique templates") and does not report the time, expertise, or iteration required.
2. Best-dev template selection: To achieve the reported "best dev template" numbers, a labeled development set must be available for each evaluation task to select the optimal instruction phrasing. The paper uses 50-200 held-out examples per dataset for this purpose.
3. Cluster holdout checkpoint training: To evaluate across all 12 task clusters, 12 separate fine-tuning runs (each for 30,000 gradient steps) are required, each with a different cluster held out.

The consequence. These costs are not part of instruction tuning's deployment story — they are evaluation infrastructure costs that the paper incurs to measure zero-shot generalization, but they are not amortized or accounted for in any efficiency metric. A practitioner deploying FLAN in a genuine zero-shot setting would:

• Not have the luxury of writing 10 templates and picking the best one on a dev set — they write one instruction and hope it works. The paper provides no guidance on how to write good instructions without a dev set, and the template variance numbers (Table 2 standard deviations) suggest performance is sensitive to phrasing.
• Not need to train 12 cluster-held-out checkpoints — they would train one model on all available task clusters and deploy it. But this means the evaluation results in the paper do not directly correspond to any single checkpoint's performance across all tasks. A practitioner training on all 12 clusters would get a model that can handle (say) NLI zero-shot, but the paper never evaluates that model on NLI — the NLI results come from a checkpoint that held NLI out of training. The paper does not report how including NLI in instruction tuning affects zero-shot NLI performance (as opposed to the held-out setting), because the goal was to study generalization to unseen task types, not post-tuning performance on seen types.

This means there is a gap between the paper's zero-shot evaluation protocol and real-world zero-shot deployment. The paper evaluates each task cluster on a model that never saw that cluster during training — a clean test of generalization. But a user deploying one FLAN model for all tasks would train on all available clusters (including the target task type, if it is among the 12) and would care about performance on both seen and unseen task types. The paper provides no characterization of this mixed regime.

What evidence exists in the paper. The paper does not directly measure these costs. The human effort of template design is unquantified. The computational cost of training 12 checkpoints is mentioned only in passing (60 hours per checkpoint on 128 TPUv3 cores, Section 2.4) but not compared against a single-checkpoint deployment cost. The appendix FAQ (Appendix E) contains a brief discussion of template sensitivity ("The small effect of templates is striking given our original motivation") but does not provide template-writing guidelines.

Mitigation status. The paper partially mitigates the template-design burden by showing in Appendix B.1 (Figure 11) that increasing the number of templates per dataset from 1 to 10 has negligible effect on held-out task performance when there are sufficient datasets per cluster — meaning a practitioner could potentially use fewer templates during training without sacrificing generalization. However, this finding applies to training-time template diversity, not inference-time template selection. The inference-time sensitivity to instruction phrasing (visible in the standard deviations in Table 2) is not mitigated or reduced through any technique. The paper's few-shot FLAN results (Section 4.4) show that adding exemplars reduces template sensitivity, but again this undercuts the zero-shot framing.

Prompt Engineering Confound: LaMDA-PT Baselines Use Different Prompts Than FLAN

The assumption or constraint. FLAN is evaluated using instruction-style prompts ("Translate this sentence to French: 'The dog runs.'") while the LaMDA-PT baselines (both zero-shot and few-shot) are evaluated using GPT-3-style prompts (formatted as sentence completions that mimic pretraining data). The FAQ in Appendix E acknowledges this discrepancy explicitly:

"FLAN prompts are formulated as responding to an instruction, they do not work well for pretrained language models without finetuning. Performance was near zero for most generation tasks. For instance, given the input 'The dog runs.' Translate this sentence to French., LaMDA-PT continues with 'The dog runs after the cat' instead of actually translating the sentence. Hence, we used the established GPT-3 prompts for our LaMDA-PT baselines."

The consequence. This creates a confound in the comparison: the performance improvement from LaMDA-PT zero-shot to FLAN zero-shot reflects both the effect of instruction tuning and the effect of switching from GPT-3-style prompts to instruction-style prompts. The paper cannot fully separate these effects. It is possible that some fraction of FLAN's gains come not from the multi-task fine-tuning, but simply from the fact that instruction-style prompts are better prompts for this model architecture when the model has any exposure to them — even minimal exposure.

The "role of instructions" ablation (Section 4.3, Figure 8) partially addresses this by comparing models trained with different input formats (no template, dataset name, instructions) but evaluated with the same instruction-style prompts. This isolates the effect of training format while holding evaluation format constant. However, the key missing comparison is: LaMDA-PT zero-shot evaluated with instruction-style prompts (which the FAQ says gives near-zero performance) vs. LaMDA-PT zero-shot evaluated with GPT-3-style prompts (the reported baseline). If LaMDA-PT zero-shot with instruction prompts gets ~5% and GPT-3-prompt LaMDA-PT gets ~40%, then the choice of evaluation prompt format alone explains a large fraction of the LaMDA-PT vs. FLAN gap. The paper never reports the near-zero numbers explicitly (only saying "Performance was near zero for most generation tasks"), making it impossible to quantify this confound.

What evidence exists in the paper. Only the FAQ passage quoted above. The paper performs no systematic experiment comparing LaMDA-PT's performance across instruction-style prompts vs. GPT-3-style prompts on all 25 evaluation datasets. The ablation in Figure 8 compares training formats but always uses instruction-style evaluation — it never tests GPT-3-style evaluation on the instruction-tuned model (which might underperform, since the model was trained to follow instructions, not complete sentences). The lack of this cross-format evaluation matrix (instruction-trained vs. GPT-3-trained models, evaluated with instruction-style vs. GPT-3-style prompts) means the paper cannot fully attribute FLAN's gains to instruction tuning per se rather than to better prompt format alignment between training and evaluation.

Mitigation status. The paper is transparent about the reason for the discrepancy (LaMDA-PT cannot follow instructions) and provides the baseline that is practically feasible (GPT-3 prompts for LaMDA-PT). However, it does not provide the missing cross-format evaluation or quantify the prompt-format confound. The "no template" and "dataset name" ablations in Section 4.3 use instruction-style evaluation, which partially isolates the training format effect, but these models are fine-tuned (they have 30,000 additional gradient steps) and are not directly comparable to the untrained LaMDA-PT base model. The confound remains unquantified.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes what supervised fine-tuning is for in the era of large language models. Before FLAN, the dominant mental model was that fine-tuning on labeled data produces specialists — models optimized for a single task at the expense of generality. The pretrain–fine-tune paradigm (BERT, T5) and the prompting paradigm (GPT-3) existed as separate toolkits for separate goals: if you wanted peak performance on one benchmark, you fine-tuned; if you wanted generality, you scaled pretraining and used few-shot prompting. Instruction tuning breaks this dichotomy. It establishes that supervised fine-tuning on diverse tasks, when formatted as instructions, makes models more general, not less — that labeled data can teach cross-task skills rather than just task-specific patterns.

This is a genuine paradigm shift in how the field thinks about the role of supervision for large LMs, not merely an incremental training recipe. The evidence is in the field's subsequent behavior: within two years of FLAN's release, instruction tuning became the standard final stage of LLM training, from InstructGPT to Llama-2-Chat to open-source models like Alpaca and Vicuña. The paper's Figure 2 — instruction tuning as a third paradigm between pretrain–fine-tune and prompting — provided the conceptual scaffolding that enabled this transition. It answered a question the field was primed to ask: "If GPT-3 can do so much with just a few examples, why can't we train models to be better at this?" The answer — instruction tune on diverse tasks, and crucially, only at sufficient scale — was the key insight that unlocked the instruction-following model family.

The paper also resolves a latent tension in prior work around multi-task learning and zero-shot transfer. Prior studies on multi-task QA (McCann et al., 2018; Khashabi et al., 2020) showed that unified formatting helped within the trained task set, but evidence for zero-shot generalization to genuinely new task types was thin and confounded with task similarity. FLAN's cluster-holdout protocol — specifically holding out entire task types, not just individual datasets — provides the first large-scale, rigorous demonstration that a model can generalize to a task type it has never seen, purely from exposure to other task types formatted as instructions. This raised the methodological bar for evaluating generalization: subsequent work could no longer claim zero-shot transfer from holding out individual datasets that shared a format with training datasets. The paper effectively made cluster-based evaluation a best practice.

The scale-dependent emergence finding (Figure 7, the 8B crossover) is arguably the paper's most scientifically significant contribution. It is one of the earliest documented cases of an emergent capability — a behavior that is absent or actively harmful at smaller scales and only appears at larger scales — and it provides a concrete, testable hypothesis for why: at small scales, the model's capacity is consumed by learning the training tasks, leaving no residual capacity to extract the meta-skill of instruction-following. This finding changed the conversation from "large models are better at everything" to "some capabilities only exist above a threshold, and understanding that threshold tells us something about the nature of the capability." It directly presaged later work on emergent abilities (Wei et al., 2022, Chain-of-Thought) and established that scale is not just a quantitative amplifier but a qualitative enabler for certain meta-cognitive skills.

However, the paper also redirects attention away from certain directions. Its negative results — that instruction tuning does not help (and can hurt) on sentence-completion tasks where the prompt already matches the pretraining objective, and that the benefits vanish below ~8B parameters — imply that instruction tuning is not a universal recipe. It works when (a) the target task format differs from pretraining continuations enough to benefit from instruction-based reformatting, and (b) the model is large enough to have spare capacity. This means the field's focus on ever-larger models for instruction following is not just a preference — it's a requirement. Research on making instruction tuning work at smaller scales (through better training curricula, architectural innovations, or distillation from larger instruction-tuned models) becomes a clear priority, because the current method simply fails below the 8-68B threshold.

Finally, the paper's demonstration that instruction tuning improves prompt tuning (Section 4.5, Figure 10) — achieving 87.4% on SuperGLUE versus 79.1% for the untuned model with full training data — suggests that instruction tuning produces model representations that are more amenable to any form of task specification, not just natural language. This is a subtle but important reframing: instruction tuning might not just be about "making models follow instructions," but about making models more controllable in general, with soft prompt optimization as one manifestation. This opens the door to hybrid systems that use natural language for task description and continuous optimization for task refinement, a direction later pursued extensively.

Follow-Up Research This Work Enables

Locate the exact scale threshold for instruction tuning emergence. The paper tests five model sizes (422M, 2B, 8B, 68B, 137B) and finds the crossover between 8B and 68B. A follow-up study should train LaMDA-PT variants at 13B, 30B, and 50B parameters using the identical instruction-tuning mixture and cluster-holdout protocol, measuring the same 13 held-out tasks from Figure 7. The goal is not just to pinpoint the threshold but to characterize the shape of the transition — is it a sharp phase change (near-zero benefit at 50B, large benefit at 51B) or a smooth sigmoid? If the transition is sharp, it supports the capacity-saturation hypothesis (the model "clicks" when it has enough spare parameters to encode instruction-following as a distinct skill). If it is smooth, the hypothesis needs revision — perhaps instruction following is partially learnable at intermediate scales but requires scale to become reliable. This experiment is now feasible because open-source model families (e.g., Pythia, OLMo) provide consistent pretrained checkpoints at fine-grained size intervals, allowing a higher-resolution replication of Figure 7.

Cross-architecture instruction tuning: does the 8B threshold hold for encoder-decoder models? The paper's results are exclusively on decoder-only LaMDA-PT. A critical follow-up would replicate the scaling ablation with an encoder-decoder architecture (T5 or a modern variant) pretrained on comparable data. The hypothesis from the paper — that the crossover point depends on model capacity — predicts that the threshold might shift (encoder-decoder models have different parameter efficiency and different pretraining objectives). Concretely, train T5 variants from 1B to 30B parameters on the same 62-dataset instruction mixture using the same cluster-holdout protocol, and measure zero-shot generalization. If the crossover occurs at a much smaller scale for T5 (say, 3B), it would suggest that bidirectional encoding or the span-corruption pretraining objective provides a better inductive bias for instruction following, and would make instruction tuning more accessible to researchers without 70B-scale compute budgets. If the crossover occurs at a similar or larger scale, it strengthens the claim that raw parameter count — not architecture — is the primary determinant.

Cheap difficulty estimation for instruction sensitivity. The paper's Table 2 shows per-template standard deviations ranging from ~1% to ~15%, meaning a user who writes a single instruction without a dev set faces substantial uncertainty about whether they have chosen an effective phrasing. A follow-up could train a lightweight classifier (perhaps a small BERT variant fine-tuned on FLAN's instruction-to-performance mapping) to predict, given an instruction text alone, whether FLAN will succeed on it — without running the model. The training data already exists implicitly: for each of the ~600+ templates (10 per dataset × 62 datasets), we know FLAN's performance on that template's test set. Train the classifier to predict whether a template's accuracy exceeds a threshold (e.g., within 5% of the best template for that dataset). If the classifier works, it provides a practical tool for instruction-writing: the user drafts an instruction, the classifier scores it, and the user iterates before incurring the inference cost of running FLAN. This would directly address the template-sensitivity limitation the paper documents but does not solve.

Combining instruction tuning with chain-of-thought prompting. The paper evaluates FLAN primarily on tasks with short, direct answers (classification labels, short phrases, single sentences). It does not explore whether instruction tuning improves the model's ability to perform multi-step reasoning tasks. Given that chain-of-thought prompting (Wei et al., 2022, from the same group) was developed contemporaneously, a natural extension is: does instruction tuning on diverse tasks improve zero-shot chain-of-thought reasoning on unseen task types? Concretely, instruction-tune on the same 62 datasets but include some that require step-by-step reasoning (e.g., math word problems from the Math dataset in the miscellaneous cluster, structured reasoning from DROP), and then evaluate zero-shot chain-of-thought on held-out reasoning benchmarks (e.g., GSM8K, StrategyQA, date understanding). If instruction tuning improves chain-of-thought performance without chain-of-thought exemplars being present during instruction tuning, it would demonstrate that the instruction-following meta-skill transfers to how the model reasons, not just what task it performs — a qualitatively deeper form of generalization than the paper currently demonstrates.

Stress-test: instruction tuning on deliberately adversarial or nonsensical instructions. The paper shows that natural language instructions during training are essential (Figure 8, the 55.2% → 37.3% drop when removing instructions). But it does not test which properties of instructions matter. A stress-test could instruction-tune FLAN on a mixture where 50% of the templates are deliberately misleading (e.g., for a sentiment analysis task, the instruction says "Translate this to French" but the target is still the sentiment label). If the model performs well on the normal 50% of instructions but poorly on the adversarial ones, it suggests the model learns to follow instructions as stated, not just map instruction templates to task types — a critical distinction for safety and robustness. If performance on normal instructions degrades when adversarial instructions are in the training mixture, it reveals negative transfer from confusing instruction signals, which would inform deployment practices (e.g., filter instruction-tuning data for internal consistency). This experiment would also probe whether the model's instruction-following is genuinely semantic (understanding what is asked) or largely pattern-based (matching surface-level template features to previously seen task formats).

Does instruction tuning transfer across languages? The paper's instruction tuning and evaluation are entirely in English, despite the model having ~10% non-English pretraining data and the training mixture including translation tasks. A cross-lingual follow-up would test: if instruction tuning is performed in English only (all 62 datasets with English instructions), does the model learn to follow instructions written in other languages at inference time? Concretely, take a FLAN checkpoint trained on English-only instructions and evaluate it on the same tasks (sentiment, NLI, QA) but with instructions manually translated into French, German, and Spanish. The translation cluster in instruction tuning provides some cross-lingual signal (the model learns that instruction text can appear in multiple languages), but it's unknown whether this generalizes to following instructions in non-English languages for non-translation tasks. If the model succeeds, it would demonstrate that instruction following is a language-agnostic meta-skill — the model has learned "respond to the intent behind the words" rather than "respond to English instruction templates." If it fails, it reveals that the meta-skill is tied to the language of instruction tuning, which would motivate multilingual instruction tuning as a necessary step for global deployment.

Practical Applications and Downstream Use Cases

Zero-shot text classification for low-resource deployment. The paper demonstrates that FLAN achieves 94.6% on SST-2 and 98.1% on Yelp sentiment analysis without any task-specific training data, using only a natural language instruction. For a company that needs to classify customer feedback, support tickets, or product reviews into custom categories that change frequently, this eliminates the standard workflow of (a) labeling hundreds of examples per category, (b) fine-tuning a BERT-based classifier, (c) redeploying when categories change. Instead, the user writes an instruction: "Is this customer feedback about billing, technical support, or product quality?" and FLAN classifies directly. The cost savings come from eliminating the labeling and retraining cycle; the performance ceiling is FLAN's accuracy on similar instruction-following classification tasks (~90-98% for binary sentiment, somewhat lower for fine-grained classification as suggested by the 78.3% average-template RTE result). The main risk is template sensitivity — if a naive user writes a poorly phrased instruction, performance may fall below the average-template numbers — which motivates building the instruction-quality classifier described in the follow-up research section above.

Data generation for training smaller task-specific models. FLAN's strong zero-shot performance on generation tasks (e.g., struct-to-text: DART Rouge-2 of 30.0 vs. LaMDA-PT's 1.5) makes it viable as a synthetic data generator for tasks where labeled data is scarce. A practitioner could write an instruction like "Given the following structured data, generate a natural language description" and use FLAN to generate thousands of (input, output) pairs, then train a much smaller, cheaper model (e.g., T5-Base) on the synthetic data. The generated data quality would be bottlenecked by FLAN's zero-shot accuracy, but even imperfect synthetic data can bootstrap a trainable system when no human-labeled data exists. This use case leverages FLAN's generality — the same checkpoint generates data for translation, summarization, QA, and struct-to-text — without requiring per-task fine-tuning. The 60-hour instruction tuning cost on 128 TPUv3 cores is amortized across all downstream data generation tasks, making it cost-effective for organizations that face a diverse portfolio of data-scarce NLP problems.

Instruction-tuned models as a foundation for prompt tuning. Section 4.5 (Figure 10, Table 4) shows that starting from FLAN rather than LaMDA-PT improves prompt tuning results by 13.8 percentage points in low-resource settings (32 training examples, SuperGLUE average: 63.8% vs. 50.0%) and 8.3 points with full training data (87.4% vs. 79.1%). For a practitioner deploying a continuous prompt optimization system (where soft prompts are learned via gradient descent on task-specific data), using FLAN as the base checkpoint rather than a raw pretrained LM provides substantial headroom for performance with the same optimization budget. This is especially valuable for applications where natural language instructions are insufficiently precise — for example, tasks requiring a specific output format or style that is hard to capture in a short instruction — because continuous prompts can learn the precise formatting while benefiting from FLAN's general instruction-following foundation. The paper's specific prompt tuning configuration (prompt length 10, weight decay 1e-4, no attention dropout) provides a reproducible starting point.

Accessibility: enabling non-experts to use language models through natural language. The paper's qualitative examples (Appendix F) demonstrate a use case the paper itself does not emphasize quantitatively but that has enormous practical significance: FLAN responds to instructions that a non-ML-expert might naturally write. "Write a sad story about carrots," "Recommend activities to do on a sunny weekend in Copenhagen," "Rewrite the above sentence in a Shakespearean style" — these are queries that a user might type into a search engine or ask a human assistant, not prompts that require understanding of GPT-3's completion-style formatting. FLAN's zero-shot capability means the user does not need to provide examples, understand few-shot formatting, or know anything about how the model was trained. This dramatically lowers the barrier to entry for language model usage — from machine learning practitioners who understand prompting to anyone who can describe what they want in plain language. The caveat, as the FAQ in Appendix E documents, is that FLAN still fails on some seemingly simple tasks (returning the nth word of a sentence, translating a question instead of answering it in the target language), so the accessible interface can be misleading — the model appears to understand but sometimes fails in non-obvious ways. Deployment in user-facing applications would need to pair this accessibility with appropriate expectation-setting and failure-mode documentation.