The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

🎯 Pitch

Mixing zero-shot, few-shot, and chain-of-thought prompts during instruction tuning—counterintuitively—improves performance in all modes, and simply adding 10% few-shot examples boosts zero-shot results by over 2%. This paper also reveals that task-balancing and data enrichment, not just scale, are the hidden keys to unlocking models that converge faster and higher on downstream tasks.

1. Executive Summary

This paper studies the design decisions behind effective instruction tuning, using the Flan Collection of tasks and methods developed for the Flan 2022 models (Chung et al., 2022) and evaluated primarily on T5-XL (3B) models against benchmarks including MMLU and BIG-Bench Hard. The work identifies and ablates several overlooked but critical techniques—training with mixed prompt settings (mixing zero-shot, few-shot, and chain-of-thought templates within the same training run), task enrichment via input inversion (reversing input-output pairs to generate new tasks, e.g., training a model to produce a question given an answer), and task mixture balancing (weighting different data sources to reflect their contribution quality)—that together enable Flan-T5 to outperform prior public instruction tuning collections by 3–17%+ across evaluation settings. The resulting Flan-T5 model also converges faster and to higher accuracy than raw T5 on single-task finetuning, establishing that instruction-tuned models serve as more computationally efficient starting checkpoints for downstream practitioners—while the paper also surfaces a clear boundary condition: task diversity and mixture balancing, not just raw task count, drive held-out generalization, as identical task counts from different sources produce markedly different scaling behavior.

2. Context and Motivation

The Core Problem: Instruction Tuning Lacks a Systematically Understood Recipe

By early 2023, the field of NLP had converged on a powerful paradigm: take a large pretrained language model and fine-tune it on a diverse collection of NLP tasks formatted with natural language instructions—a process called instruction tuning (Wei et al., 2021; Sanh et al., 2021; Min et al., 2022). This process dramatically improved a model's ability to generalize to unseen tasks presented as instructions at inference time, making models more useful for interactive and zero-shot settings. However, the field suffered from a significant gap: there was no systematic understanding of which design decisions in instruction tuning actually matter. Each public collection—Flan 2021, P3 (the Public Pool of Prompts), Super-Natural Instructions, MetaICL—used different task compositions, different templating strategies, different prompt formats (zero-shot vs. few-shot), and different training procedures, all evaluated on different model sizes and architectures. When one collection outperformed another, it was impossible to attribute the gain to a specific technique. The instruction tuning literature had accumulated methods without measurement.

This paper tackles exactly that gap. Rather than proposing a fundamentally new technique, it systematically ablates the methodological choices in the "Flan 2022 Collection"—the data and methods first used in Chung et al. (2022) to produce Flan-PaLM—and isolates why this combination works so well. The goal is not just to report that Flan-T5 outperforms prior collections (which it does, by 3–17%+), but to tease apart the contributions of individual design decisions: mixed prompt training, input inversion, task scaling, and mixture balancing. In doing so, the paper provides the first detailed ablation study across the major instruction tuning collections, establishing a set of actionable findings for practitioners building instruction-tuned models.

Why This Problem Matters: Practical and Scientific Stakes

The importance of understanding instruction tuning's design decisions extends well beyond academic benchmarking. Several practical and scientific stakes motivate this work:

Instruction tuning is a one-time compute investment with amortized benefits. Unlike single-task fine-tuning, which incurs a per-task training cost, instruction tuning produces a single general-purpose checkpoint that can be reused across many downstream applications. As Section 4 demonstrates, this checkpoint—Flan-T5—serves as a faster-converging and higher-peaking starting point for single-task fine-tuning, reducing the aggregate compute required across many downstream tasks. In an era where the environmental costs of training large models face increasing scrutiny (Wu et al., 2022; Bommasani et al., 2021), establishing instruction-tuned models as standard starting checkpoints has genuine green-AI implications. But for this vision to be realized, the community needs a clear, evidence-backed recipe for how to build good instruction-tuned models—not just a collection of ad-hoc methods.

The public research community was losing ground on instruction tuning to private efforts. By late 2022 and early 2023, high-profile advances in model alignment—particularly InstructGPT (Ouyang et al., 2022) and ChatGPT—were driven by non-public data and human feedback datasets inaccessible to the broader research community. As the paper notes in Section 2:

"The inaccessibility of these resources inhibits the research community's ability to analyze and improve these methods in the public domain."

The paper positions itself as an open-source counterweight, deliberately restricting its purview to publicly available data collections and releasing the entire Flan 2022 Collection—tasks, templates, methods—for public use. This democratization goal is central to the work's motivation: if instruction tuning is to be a foundation for future research (including combining it with human feedback, parameter-efficient tuning, and multimodal extensions), the community needs open, strong starting points with well-characterized design decisions.

Conflicting implicit assumptions about prompt format specialization needed resolution. Prior to this work, a widespread implicit assumption was that instruction-tuned models specialized to particular prompt formats: a model trained on zero-shot prompts would perform best on zero-shot evaluation, while a model trained on few-shot exemplars would perform best on few-shot evaluation. This assumption was reinforced by the fact that most public collections were tuned specifically for one prompt setting—Flan 2021 and P3 for zero-shot, MetaICL for few-shot, Super-Natural Instructions for few-shot with specific formatting. The natural expectation was that mixing formats at training time would create a trade-off, helping one setting at the expense of another.

The paper's finding that adding just 5–10% of few-shot templates dramatically improves zero-shot performance—and vice versa (Section 3.2, Figure 3)—is surprising and runs counter to this assumption. For small models (3B parameters), where capacity constraints might most naturally predict specialization, mixing still helps. This finding has immediate practical implications for anyone building instruction-tuned models: you should not pick a prompt format at training time; you should mix them. The fact that no prior public collection had systematically studied this choice—despite the private InstructGPT work having used mixed prompts—makes this a significant methodological contribution.

The alignment tax problem had no clear remedy. Ouyang et al. (2022) documented an "alignment tax": training with human feedback on open-ended tasks often came at the expense of performance on traditional NLP benchmarks. The paper explicitly engages with this tension, noting in Section 2 that instruction tuning without human feedback "shows great promise in enhancing human preferred responses on open-ended tasks, as well as improving traditional NLP metrics." The extent to which instruction tuning alone—without expensive, proprietary human feedback data—could recover the benefits of alignment while preserving broad NLP capability was an open question. This work's demonstration that Flan-T5 achieves strong performance on both conversational tasks (dialog) and traditional NLP benchmarks (MMLU, BBH, Held-In QA and NLI) provides evidence that instruction tuning can partially address the alignment tax without the downsides.

Where Prior Approaches Fell Short

To understand what the paper contributes, we need to trace the evolution of instruction tuning collections and identify the limitations of each prior wave. The paper's Figure 2 provides a timeline, and Section 2 narrates this progression.

The First Wave: Task Collection Without Systematic Understanding of What Matters

The first wave of instruction tuning (roughly 2020–2021) established the basic paradigm and provided initial evidence of scaling benefits:

Natural Instructions (Mishra et al., 2021) aggregated NLP tasks with crowdsourced instructions and demonstrated that models could generalize to unseen instructions. The contribution was the format and the task diversity hypothesis, but the collection was relatively small compared to what followed.

Flan 2021 (Wei et al., 2021) aggregated 62 NLP tasks with templated instructions and showed that fine-tuning on this mixture improved zero-shot generalization. The key finding was that scaling the number of tasks and model size both contributed to performance. However, Flan 2021 trained exclusively with zero-shot prompt templates—it was designed for the setting where the user provides only an instruction, no exemplars. This was a deliberate choice but left unexplored whether training with few-shot or chain-of-thought prompts would help.

P3 (the Public Pool of Prompts, Sanh et al., 2021; Bach et al., 2022) provided a large repository of prompt templates contributed by the community via the PromptSource tool. T0, the model trained on P3, also focused on zero-shot prompting. A key contribution of P3 was the recognition that having multiple diverse templates per task improved generalization—the same task could be phrased in dozens of different ways, and exposure to this variety made the model more robust to instruction rephrasing at test time. P3 also introduced input inversion ("prompts not intended for the original task") as a task enrichment technique, though without systematic ablation of its contribution.

MetaICL (Min et al., 2022) explored the complementary direction: training models to learn tasks from in-context exemplars (few-shot prompting) rather than from explicit instructions. MetaICL introduced the "noisy channel" approach—inverting input-output pairs to create new training tasks—but focused exclusively on the few-shot setting without instructions.

What was missing across this first wave: While each collection advanced the state of the art, no work had systematically compared these collections under controlled conditions (same model, same evaluation, same size), nor had any work ablated the individual design decisions within a single instruction tuning recipe. The field knew that task diversity, template diversity, and model scaling all helped, but it didn't know which techniques were most responsible for performance gains, whether they were complementary or redundant, or how they interacted with model size and task count.

The Second Wave: More Tasks, But Still No Controlled Ablations

A second wave of work (2022–early 2023) expanded the scale and scope of instruction tuning:

Super-Natural Instructions (Wang et al., 2022c) scaled to 1,600+ tasks with crowdsourced instructions, representing a substantial expansion in task diversity. However, the instructions followed a specific format that emphasized few-shot exemplars with a particular structure. When this paper's authors fine-tune T5-XL on Super-Natural Instructions (Table 1), the resulting model substantially underperforms Flan 2022 (e.g., 50.3% vs. 73.8% on Held-In zero-shot tasks, and 35.6% vs. 50.3% on MMLU zero-shot). The authors and Iyer et al. (2022) both note that Super-Natural Instructions' unique input format and instruction design may limit its benefits for general instruction following.

OPT-IML (Iyer et al., 2022), a concurrent work, assembled a similarly large collection of tasks (~2,200) with careful attention to data mixing proportions. OPT-IML-Max 175B achieved strong results, but the underlying collection, templates, and processing were not publicly released, making it impossible to reproduce or ablate. Importantly, Iyer et al. (2022) did identify that the mixtures of Flan 2021, T0-SF, and T5-style tasks were most beneficial—a finding that this paper corroborates in its own mixture balancing experiments (Section 3.5). But the inaccessibility of OPT-IML's full pipeline meant the community couldn't build on it.

Flan 2022 (Chung et al., 2022) introduced the concept of training with chain-of-thought (CoT) prompts in addition to standard instruction-following data, showing that this enhanced reasoning capabilities at PaLM-scale. However, Chung et al. (2022) was primarily a scaling paper—it focused on emergent abilities when combining Flan 2022 with PaLM 540B, and spent little time analyzing the individual methodological contributions. The paper explicitly acknowledges this division of labor in its introduction:

"Where Chung et al. (2022) focuses on the emergent and state-of-the-art results of combining Flan 2022 with PaLM 540B, this work focuses in on the details of the instruction tuning methods themselves, ablating individual factors, and comparing them directly to prior work by keeping the pretrained model size and checkpoint consistent."

This is a crucial distinction: Chung et al. (2022) shows that Flan 2022 works at scale; this paper shows why it works by decomposing the method.

The specific gap this paper addresses: Across both waves, no work had done a controlled, apples-to-apples comparison of all major public instruction tuning collections on equivalent model sizes, nor had any work systematically ablated each component of the best-performing recipe to quantify its individual contribution. Practitioners choosing an instruction tuning approach had no evidence-based guidance on whether to invest in mixed prompt formats, input inversion, task mixture balancing, or simply scaling the number of tasks. The paper fills exactly this gap.

How This Paper Positions Itself

The paper's positioning can be understood along several dimensions:

Systematic Method Decomposition, Not Method Proposal

The paper's primary contribution is not a new technique—it is the decomposition and measurement of existing techniques that had been used in combination without individual assessment. This is clearest in Table 1, which shows the marginal contribution of each method by removing it from the full Flan 2022 recipe:

Removing few-shot templates: Held-In zero-shot drops 1.3 points (73.8 → 72.5), but Held-In few-shot drops 12.6 points (74.8 → 62.2)—showing that mixing formats is essential for few-shot performance.
Removing input inversion: MMLU zero-shot drops 8.6 points (50.3 → 41.7)—showing input inversion is critical for held-out generalization.
Removing mixture balancing: BBH zero-shot drops 11.1 points (26.2 → 15.1)—showing that how you weight data sources matters as much as what sources you include.
Removing CoT data: BBH-CoT few-shot drops 23.2 points (35.2 → 12.0)—showing that chain-of-thought training is essential for chain-of-thought evaluation.

Each ablation is a subtraction experiment—start from the full recipe and remove one component—which cleanly isolates the marginal contribution. This is a far stronger experimental design than adding components incrementally, because it controls for interactions: removing a component from the full recipe captures its contribution given that everything else is present.

A Deliberate Focus on Public, Reproducible Research

The paper repeatedly emphasizes its commitment to open-source accessibility. The authors explicitly narrow their purview to "open source and accessible data collections, motivated by the goal of democratizing accessibility to research" (Section 2). They note that while instruction tuning with human feedback (RLHF) shows strong results on open-ended tasks, human feedback datasets are "far less publicly available than instruction tuning datasets (and may be model-specific)" (Section 2). This creates a clear boundary: this work studies what can be achieved with instruction tuning without human feedback, leaving the combination with RLHF as additive future work.

This positioning is strategic. By demonstrating that Flan-T5 matches or exceeds OPT-IML-Max 175B (a ~58× larger model) on MMLU and BBH without using human feedback data, the paper makes the case that public instruction tuning methods, carefully applied, can narrow the gap with private efforts that rely on proprietary human feedback. The paper is not anti-RLHF—it explicitly views RLHF as "likely additive to a foundation of instruction tuning methods"—but it argues that characterizing what the public foundation can achieve is valuable in its own right.

Controlled Comparison Through Fixed Model Size

A key methodological choice in the paper's positioning is the decision to run all ablations and comparisons on T5-XL (3B parameters) as a consistent base model. This is motivated explicitly:

"We felt XL was appropriately sized to run large-scale systematic ablations, while being sufficiently large to draw general conclusions."

This is a deliberate departure from Chung et al. (2022), which operated at PaLM 540B scale. The 3B size is large enough to exhibit meaningful instruction tuning behavior (pass@1 rates that respond to methodological choices) but small enough to train dozens of variants for ablation studies without prohibitive compute cost. The risk, which the paper acknowledges implicitly, is that some findings might not transfer to much larger scales—a standard scaling caveat. However, the fact that the same Flan 2022 methods improved PaLM 540B in Chung et al. (2022) provides some evidence of transferability.

Reconciling Conflicting Practices Through Evidence

The paper positions its findings as resolving implicit debates in the instruction tuning community:

Task count vs. task composition: The scaling analysis in Figure 4 shows that Held-Out performance improves log-linearly with task count even out to 1,836 tasks, but Section 3.5 provides the crucial caveat: not all task sources are equally beneficial. Super-Natural Instructions, despite contributing many tasks, has limited scaling benefits. This resolves the tension between "more tasks always help" and "task quality matters"—both are true, but task quality dictates the rate of improvement per task.
Zero-shot vs. few-shot specialization: The finding that mixing prompts improves both settings (Figure 3) resolves an implicit assumption that models must specialize. The mechanism is likely that exposure to few-shot exemplars during training teaches the model something about the task structure that transfers to zero-shot—and conversely, zero-shot training prevents the model from overfitting to the presence of exemplars, improving few-shot generalization.
The contribution of input inversion when tasks are already abundant: Prior work (Sanh et al., 2021; Min et al., 2022) introduced input inversion but studied it in the context of collections with dozens to hundreds of tasks. It was unclear whether it would remain helpful when thousands of tasks were already available. The paper finds it does still help (Table 1, "− Input Inversion"), particularly for Held-Out tasks, suggesting that the structural diversity from inversion provides a qualitatively different form of enrichment than simply adding more forward-direction tasks.

The Computational Efficiency Argument as Motivation

Section 4 proposes a pragmatic reframing of instruction tuning's value: it is not just about zero-shot generalization; it is also about providing a better starting point for single-task fine-tuning. Figure 5 shows that Flan-T5, even without further fine-tuning, often matches or exceeds T5 with task-specific fine-tuning. Figure 6 shows that when you do fine-tune Flan-T5 on a new task, it converges faster (in fewer steps) and to a higher accuracy than fine-tuning T5 from scratch.

This efficiency argument is positioned as a counterpoint to the "instruction tuning is expensive" concern. Yes, producing Flan-T5 requires a large one-time investment in multi-task instruction tuning. But if the resulting checkpoint reduces fine-tuning steps by a substantial factor across thousands of downstream applications, the aggregate compute savings can be positive. The paper frames this as a "green-AI incentive" (Section 4) for the community to adopt instruction-tuned models as standard starting points—a pragmatic argument that extends the paper's relevance beyond the instruction tuning research community to NLP practitioners writ large.

Acknowledged Limitations in Positioning

The paper is careful about what it does not claim:

It does not claim that the specific mixture weights used in Flan 2022 are optimal. The mixture balancing section (3.5) is explicitly described as using "practitioner's intuition" informed by the leave-one-out experiments in Table 2, and the paper notes this "leaves ample room for more sophisticated future work."
It does not claim that task scaling beyond 1,836 tasks continues to help. The scaling curves in Figure 4 show Held-Out performance still improving at the maximum task count, but "we would caution conclusions that task scaling beyond 1800 would translate to increased returns without also paying attention to task diversity and quality."
It does not claim that instruction tuning without human feedback is a complete solution to alignment. The paper explicitly positions human feedback methods as "likely additive to a foundation of instruction tuning methods" and frames its contribution as characterizing what is possible in the public domain without proprietary human feedback data.
It acknowledges that model architecture and pretraining play significant roles (citing Wang et al., 2022a) and that the specific findings are demonstrated on T5-style encoder-decoder models pretrained on the C4 corpus—a deliberate scope limitation that invites replication on other architectures.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily an empirical analysis and ablation study of instruction tuning design decisions—not a proposal for a fundamentally new algorithm. The core idea is that the strong performance of the Flan 2022 models (Chung et al., 2022) stems from a set of identifiable, independently measurable methodological choices—mixing prompt formats during training, enriching tasks through input inversion, balancing data sources, and scaling to thousands of tasks—and that by systematically removing each component, we can quantify its marginal contribution to downstream performance across Held-In, Held-Out, and Chain-of-Thought evaluation settings.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, each contributing a specific mechanism to the final instruction-tuned model:

Task Collection (1,836+ tasks) — an aggregated set of NLP datasets drawn from Flan 2021, P3++, Super-Natural Instructions, and additional reasoning, dialog, and program synthesis sources, each reformatted with instruction templates. This is the raw material.
Template System — a procedure for converting each dataset example into one of three prompt formats: zero-shot (instruction only), few-shot (instruction + 2–5 exemplars), or Chain-of-Thought (instruction + step-by-step reasoning exemplars). Critically, multiple templates exist per task, and templates vary not just in wording but in structural dimensions (instruction placement, spacing, answer option formatting).
Task Enrichment via Input Inversion — a data augmentation step that takes each (input, output) pair in the original dataset and creates new examples where the roles are reversed (e.g., given an answer, generate the question). These inverted examples are mixed into the training data at a rate of 30%.
Mixture Balancer — a weighting scheme over the different data sources (Flan 2021, T0-SF, Super-Natural Instructions, Chain-of-Thought, Dialog, Program Synthesis) that determines what fraction of each training batch comes from each source. The weights are informed by leave-one-out experiments (Table 2) and practitioner judgment.
T5-XL Base Model (3B parameters) — the pretrained encoder-decoder language model that is fine-tuned on the resulting mixture of templated, potentially inverted examples. The output is Flan-T5, a single checkpoint that can be used directly for zero-shot, few-shot, and Chain-of-Thought prompting, or further fine-tuned on a specific downstream task.

Information flows as follows: raw datasets → templatized into zero-shot, few-shot, and CoT formats (with input inversion applied to a 30% subset) → weighted by source into training batches → fine-tune T5-XL on the mixture → produce Flan-T5 checkpoint → evaluate on Held-In tasks (8 QA + NLI validation sets), Chain-of-Thought tasks (5 reasoning datasets), and Held-Out tasks (MMLU's 57 exams, BBH's 23 challenging tasks).

3.3 Roadmap for the Deep Dive

First, the experimental framework (Section A.1 / main paper setup): model choice, training configuration, evaluation taxonomy, and how "Held-In," "Held-Out," and "CoT" evaluations are defined—since all ablations are interpreted relative to these benchmarks.
Second, the mixed prompt training mechanism (Section 3.2): how zero-shot, few-shot, and Chain-of-Thought templates are created, mixed, and what Figure 3 reveals about the interaction between training-time format and evaluation format.
Third, the task scaling analysis (Section 3.3): how training set size is varied from 8 to 1,836 tasks across five model sizes to characterize performance scaling laws—and the crucial caveat that task source, not just count, matters.
Fourth, input inversion (Section 3.4 and Appendix B): the precise mechanism by which (input, output) pairs are reversed, which task types are inverted, the 30% mixing rate, and why this helps Held-Out generalization but not Held-In performance.
Fifth, mixture balancing (Section 3.5): the leave-one-out experiments that rank the contribution of each data source, how the resulting weights were derived, and the connection to OPT-IML's concurrent findings.
Sixth, single-task fine-tuning efficiency (Section 4): the experimental protocol for comparing Flan-T5 vs. T5 as starting checkpoints, including convergence speed and peak accuracy.

3.4 Detailed, Sentence-Based Technical Breakdown

This section decomposes each methodological component of the Flan 2022 instruction tuning recipe, explaining what it does, how it is implemented, and why the specific design choices were made. The explanations draw from the main paper body (Sections 3.1–3.5), Appendix A (Experimental Details), and Appendix B (Input Inversion Details).

Experimental Framework: Model, Training, and Evaluation Setup

All instruction tuning experiments use T5-LM, the prefix-language-model-adapted version of T5 (Lester et al., 2021), with the XL size (3B parameters) as the default, unless otherwise specified in scaling experiments. The authors justify this size choice explicitly:

"We felt XL was appropriately sized to run large-scale systematic ablations, while being sufficiently large to draw general conclusions."

Instruction tuning follows the same protocol described in Chung et al. (2022), implemented using the T5X framework (Roberts et al., 2022). The key training hyperparameters, drawn from Appendix A.1 and the main experimental setup, are: the model is trained on a mixture of templated examples where the few-shot and few-shot Chain-of-Thought prompts are generated with 2, 3, or 5 exemplars per example (sampled randomly during templatization). The mixture weights used to balance data sources were "informed by experiments in Section 3.5, along with the resulting practitioner intuition"—a deliberate acknowledgment that the weights are empirically guided but not algorithmically optimized.

The evaluation taxonomy is structured around three categories:

Held-In tasks: 8 validation sets from tasks that exist within the training collection. These are 4 question answering (QA) datasets—BoolQ, ARC Easy, ARC Challenge, AI2 Middle School Science Exams—and 4 natural language inference (NLI) datasets—ANLI R1, R2, R3, and RTE. The Held-In score is the mean accuracy (or equivalent metric, see Table 3) across these 8 tasks. The purpose of Held-In evaluation is to measure whether the model maintains performance on tasks it has explicitly seen during training, even as the training mixture becomes more diverse.
Chain-of-Thought (CoT) tasks: 5 validation sets prepared with prompts that request step-by-step explanations in their target answers: GSM8K (grade-school math word problems), StrategyQA (implicit reasoning questions), SVAMP (arithmetic word problems with structure variations), Asdiv (diverse math word problems), and CommonsenseQA (commonsense reasoning). The CoT score is the mean accuracy across these 5 datasets. The purpose is to measure whether training with CoT-prompted examples improves the model's ability to produce step-by-step reasoning at inference time.
Held-Out tasks: Two benchmarks that are explicitly excluded from the Flan 2022 training collection. MMLU (Massively Multitask Language Understanding; Hendrycks et al., 2020) consists of 57 exams spanning sciences, social sciences, humanities, business, health, and other subjects. BIG-Bench Hard (BBH; Suzgun et al., 2022) consists of 23 challenging tasks from BIG-Bench where even PaLM underperformed human raters. MMLU tasks were specifically removed from the Super-Natural Instructions portion of the Flan 2022 training data to ensure they are truly Held-Out. The purpose of Held-Out evaluation is to measure genuine generalization—the model's ability to perform tasks it has never seen during training, even from different task families.

For BBH, the paper evaluates in both standard format (zero-shot and few-shot) and with Chain-of-Thought prompting (BBH-CoT), giving a total of 6 distinct evaluation settings represented in Table 1: Held-In zero-shot and few-shot, CoT zero-shot and few-shot, MMLU zero-shot and few-shot, BBH zero-shot, few-shot, and BBH-CoT zero-shot and few-shot.

Metrics are dataset-specific and listed in Table 3. For example, ANLI uses 3-class F1, BoolQ uses AUC-ROC, SQuAD V2 uses F1, and GSM8K uses exact-match accuracy. The paper's results tables report either one metric per dataset (the "Accuracy (%)" column aggregates compatible metrics under a single label for readability) or explicitly separate metrics when aggregation would be misleading.

For single-task fine-tuning experiments (Section 4, detailed in Appendix A.2), the protocol differs: models are fine-tuned for 100,000 steps with a constant learning rate of 0.001, dropout probability of 0.1, and batch size of 128 length-512 sequences. Checkpoints are saved every 20 steps, and test performance is reported on the checkpoint corresponding to the highest validation performance. For tasks without a validation split, 1,024 training examples are held out for validation. For tasks with less than 1K training examples, results are averaged across 3 random seeds.

Mixed Prompt Training: Simultaneous Exposure to Zero-Shot, Few-Shot, and CoT Formats

The central innovation—and the most surprising finding—is that training a model on a mixture of zero-shot, few-shot, and Chain-of-Thought prompt templates within the same training run improves performance in all three evaluation settings, rather than creating a trade-off where each setting improves at the expense of others. The paper calls this "training with mixed prompt settings" (Section 3.2).

What the templates consist of. For each task in the Flan 2022 collection, the authors create three categories of prompt templates:

Zero-shot templates: The input contains an instruction describing the task plus the input example, with no exemplars. For instance, a question answering task might be formatted as "Answer the following question: {question}" with the target being the answer. Multiple zero-shot templates exist per task, varying the wording and structure of the instruction.
Few-shot templates: The input contains an instruction, followed by 2, 3, or 5 input-output exemplars (randomly selected at template creation time), followed by the target input. The exemplars demonstrate the desired behavior before the model must produce the answer. For example, a few-shot template might be: "Classify the sentiment: 'Great movie!' → Positive. 'Terrible food.' → Negative. 'It was okay.' →" with the target being "Neutral".
Chain-of-Thought templates: A specialized variant of few-shot where the exemplars include step-by-step reasoning before the final answer. For math problems, an exemplar would show the intermediate calculations; for reasoning tasks, it would show the logical steps. The target then includes both the reasoning chain and the final answer, formatted so the model learns to produce reasoning before answering.

The crucial design decision is that all three template types are mixed into the same training batches. During training, each example is sampled from one of the three template types according to the mixture ratio. The paper sweeps this ratio systematically in Figure 3, varying the percentage of few-shot templates from 0% (all zero-shot) to 100% (all few-shot), with the remaining fraction being zero-shot templates.

The counterintuitive finding. Figure 3 demonstrates that training on a mix of 0% and 100% few-shot templates is suboptimal for both evaluation settings:

For zero-shot evaluation, performance peaks when approximately 5–90% of training templates are few-shot. Adding as little as 5% few-shot training data improves zero-shot Held-In accuracy compared to training with 0% few-shot (all zero-shot). The improvement is not monotonic—performance reaches a maximum in the intermediate range and then declines slightly if training is 100% few-shot—but any mixture in the 5–90% range outperforms pure zero-shot training.
For few-shot evaluation, adding 10%+ of zero-shot data similarly improves few-shot performance over training with 100% few-shot templates. The Held-In few-shot accuracy curve in Figure 3 (right panel) shows a clear U-shaped or inverted relationship: a mix of formats outperforms either extreme.

"Figure 3 shows (1) adding as little as 5% few-shot training templates can dramatically improve zero-shot performance, and (2) adding 10%+ of zero-shot data improves few-shot performance too."

The mechanism is not explicitly tested, but the paper's framing suggests a transfer effect: exposure to few-shot exemplars during training teaches the model something about task structure (e.g., what constitutes a valid input-output mapping for this task type) that transfers to zero-shot inference. Conversely, zero-shot training prevents the model from overfitting to the presence of exemplars (i.e., learning to rely on the exemplar format rather than the underlying task), which improves few-shot generalization. In pedagogical terms, the model learns the task better when it sees the task presented in multiple formats, rather than memorizing the format-specific cues.

This finding is positioned as "underappreciated" because prior public collections had trained specifically for one prompt setting (Wei et al., 2021; Sanh et al., 2021; Min et al., 2022), assuming a specialization tradeoff. The fact that even 3B-parameter models benefit from mixing—the authors explicitly note they "expected a performance trade-off in finetuning for zero-shot or few-shot prompting performance – particularly for smaller models"—makes the result particularly noteworthy.

Implementation detail for few-shot exemplar counts. The templatization procedure generates few-shot exemplars with 2, 3, or 5 exemplars, randomly selected per example. This means that within a training batch, some few-shot examples will have 2 exemplars, some 3, and some 5. The variation in exemplar count adds another dimension of input diversity beyond just the wording of the instruction. The paper does not ablate this specific choice (fixed vs. variable exemplar count), but it is consistent with the broader theme of maximizing input variability during training.

Chain-of-Thought training as a separate axis. Beyond the zero-shot/few-shot mixing, the paper also trains with Chain-of-Thought templates (which are themselves few-shot with reasoning chains). The CoT ablation in Table 1 shows that removing CoT training data has its largest impact on CoT evaluation settings: BBH-CoT few-shot drops from 35.2% to 12.0% (a 23.2-point decline). This is expected—training with CoT teaches the model to produce reasoning chains, and evaluating without CoT training largely eliminates the ability to do so. But CoT training also provides modest benefits to non-CoT settings (Held-In, MMLU standard), likely because the reasoning chains expose the model to richer linguistic patterns and task structures.

Task Scaling: From 8 to 1,836 Tasks Across Five Model Sizes

Section 3.3 investigates how performance changes as the number of fine-tuning tasks increases, holding other aspects of the Flan 2022 recipe constant. This is the closest the paper comes to a "scaling law" analysis for instruction tuning, analogous to pretraining scaling laws (Hoffmann et al., 2022) but applied to task diversity rather than model or data size.

Experimental design. The authors fine-tune T5-LM-adapted models at five sizes—Small, Base, Large, XL (3B), and XXL (11B)—on randomly selected subsets of the 1,836-task Flan 2022 collection. The subset sizes are: 8, 25, 50, 100, 200, 400, 800, and all 1,836 tasks (after deduplication, the paper reports 1,873 total tasks in some places, reflecting slightly different accounting). Each fine-tuning run is guaranteed to include the Held-In tasks in its subset, which is a crucial design choice: Held-In tasks are always present, so they measure capacity maintenance (can the model still perform a task it knows well when that task has to share model capacity with many other tasks?) rather than generalization. Held-Out tasks (MMLU) are never included, so they measure genuine transfer from training task diversity to unseen task performance.

The key finding: Held-In performance peaks, then degrades; Held-Out improves log-linearly. Figure 4 shows two distinct patterns:

Held-In performance (left panel): For all model sizes, accuracy on the 8 Held-In tasks initially increases as the number of training tasks grows from 8 to roughly 200–400, then decreases as more tasks are added beyond that peak. The peak occurs at different points for different model sizes: larger models peak later (at higher task counts) and degrade less after peaking. T5-Small peaks around 50–100 tasks and degrades noticeably by 1,836 tasks; T5-XXL peaks around 400–800 tasks and shows minimal degradation.
Held-Out performance (right panel, MMLU): Accuracy increases log-linearly (monotonically, with diminishing marginal returns) with the number of training tasks for all model sizes, reaching the highest performance at the maximum task count of 1,836. Only T5-Small appears to possibly exceed its peak before 1,836 tasks; all larger models continue to improve.

The authors interpret this as evidence that "even T5-Base may not have exhausted its capacity with thousands of tasks, and the largest LMs could benefit from thousands more tasks for Held-In and Held-Out task performance."

Interpreting the Held-In drop. The Held-In degradation at high task counts is an important signal about the tension between task diversity and capacity allocation. When the model is fine-tuned on only 8 tasks (the Held-In tasks plus a few others), it can dedicate most of its capacity to mastering those specific tasks. As hundreds or thousands of tasks are added, the model must allocate its representational capacity across a much wider distribution, and performance on any single seen task can drop as a result—a classic multi-task interference phenomenon. Larger models mitigate this because they have more capacity to represent diverse task structures without destructive interference.

The critical caveat about task quality and diversity. The paper explicitly warns against a naïve reading of "more tasks always helps":

"One necessary assumption of this analysis is that all tasks are defined and counted equally. Section 3.5 demonstrates how not all task sources are equally beneficial to training, and the model performance may saturate from too many tasks from one source (e.g., Super-Natural Instructions). We would caution conclusions that task scaling beyond 1800 would translate to increased returns without also paying attention to task diversity and quality."

This connects the scaling analysis directly to the mixture balancing experiments in Section 3.5. The scaling curves in Figure 4 treat all 1,836 tasks as interchangeable units when sampling random subsets, but Section 3.5 shows that adding 100 tasks from Super-Natural Instructions has a different effect than adding 100 tasks from Flan 2021 or T0-SF. The apparent benefit of "more tasks" in Figure 4 is actually a combined effect of more tasks and the fact that sampling more tasks from the full collection inherently samples from more diverse sources.

Task Enrichment via Input Inversion: Reversing Input-Output Pairs

Input inversion is a data augmentation technique introduced in prior work (Sanh et al., 2021; Min et al., 2022) that the paper re-examines to determine whether it remains helpful when thousands of tasks from hundreds of sources are already available. The technique is simple: take an existing supervised example consisting of an input x and a target output y, and create a new training example where the roles are reversed—the model receives y (or a variant of it) as input and is trained to produce x as output.

Why this could help, even with abundant data. The rationale is that input inversion enriches task structure rather than just task quantity. A forward-direction question-answering example teaches the model a specific mapping from questions to answers. An inverted example teaches the complementary mapping from answers to questions, which forces the model to learn a different kind of linguistic relationship (e.g., what makes a well-formed question given an answer) and expands the distribution of input-output patterns the model encounters. Even with hundreds of forward-direction datasets, the structural pattern of inversion (output → input generation) is relatively rare unless explicitly added.

Implementation details (Appendix B). The Flan 2022 collection already implicitly includes some inverted tasks because Flan 2021, P3++, and Super-Natural Instructions contain examples like "generate a question given the answer and context." The paper extends inversion to the remaining datasets in the Flan 2022 collection: Dialog, Program Synthesis, and Chain-of-Thought tasks.

For Dialog tasks, inverted examples ask the model to produce the previous conversational history from the current dialog turn. The instruction template might be something like "Given the current response, what was the preceding conversation?"
For Program Synthesis tasks, inverted examples ask the model to produce the coding problem description given the solution code. The model sees the code as input and is trained to output the natural language problem statement.
For Chain-of-Thought tasks, the inversion is the most sophisticated. The paper considers all permutations of the triple (query, answer, explanation) where at least one of the three appears as the output. Figure 7 in Appendix B illustrates this: given a standard CoT example with Question, Chain-of-Thought reasoning, and Answer, the inversions include (Question + Answer → Chain-of-Thought), (Question + Chain-of-Thought → Answer), and (Chain-of-Thought + Answer → Question), among others. Each permutation requires a corresponding instruction template that tells the model what the input contains and what to predict.

The mixing rate. Inverted examples are mixed in at a rate of 30%—meaning that for every 10 regular examples from a Dialog task, 3 inverted examples are generated. The paper states this rate was chosen "for simplicity, approximately mirroring prior work, and leave the large space of exploration for future work."

The ablation result (Table 1, "− Input Inversion" row). Removing input inversion from the full Flan 2022 recipe produces asymmetric effects:

Held-In performance is essentially unchanged: 73.8% zero-shot with inversion vs. 73.8% without. The model can maintain performance on seen tasks regardless.
CoT performance drops: from 35.8% to 32.2% zero-shot (a 3.6-point decline), and more dramatically from 34.1% to 23.5% few-shot (a 10.6-point decline).
MMLU drops substantially: from 50.3% to 41.7% zero-shot (an 8.6-point decline). This is the largest Held-Out effect.
BBH drops substantially: from 26.2% to 18.4% zero-shot (a 7.8-point decline).

The pattern is clear: input inversion primarily benefits Held-Out generalization and reasoning tasks, not seen-task performance. The mechanism is plausibly that inversion teaches the model a meta-skill—the ability to flexibly remap inputs and outputs in arbitrary ways—that transfers to novel tasks where the model must infer the desired mapping from the instruction alone. On Held-In tasks, the model already has many forward-direction examples to learn from, so the inverted examples add redundant rather than complementary signal.

The paper also notes that these results "invigorate the prospect of data augmentation techniques for LLM finetuning, which had previously been shown to have diminishing returns the longer models are pretrained" (citing Longpre et al., 2020). This is an interesting counterpoint: data augmentation during fine-tuning may be more impactful than augmentation during pretraining precisely because fine-tuning operates on a smaller, more structured dataset where structural diversity is scarcer.

Mixture Balancing: Weighting Data Sources by Contribution Quality

Section 3.5 addresses the question: given a fixed collection of task sources (Flan 2021, T0-SF, Super-Natural Instructions, Chain-of-Thought, Dialog, Program Synthesis), how should they be weighted when constructing training batches? The default approach—equal weighting of all sources—is suboptimal because task sources vary in quality, diversity, format, and downstream transfer value.

The leave-one-out experimental design. To estimate the contribution of each data source, the authors train models on the full mixture minus one source at a time and measure the performance drop on MMLU (a Held-Out benchmark that is sensitive to mixture quality). The ablated sources are: Flan 2021, T0-SF (the subset of P3++ not already in Flan 2021), Super-Natural Instructions, Chain-of-Thought tasks, Program Synthesis, and Dialog. The results are shown in Table 2, which reports accuracy on Held-In, CoT, and MMLU for each ablated mixture.

Ranking of source importance (Table 2). On the MMLU benchmark:

Removing Flan 2021 causes the largest drop: from 47.3% to 45.7%. Flan 2021 contains 62 core NLP tasks with high-quality templates, and the paper notes that Iyer et al. (2022) similarly identify their equivalent of Flan 2021 as the most beneficial mixture.
Removing T0-SF (the non-Flan-2021 subset of P3++) causes the second-largest drop: from 47.3% to 44.7%. Both sources are "among the most beneficial mixtures."
Removing Super-Natural Instructions causes a smaller drop: from 47.3% to 46.8%. This is notable because Super-Natural Instructions contributes a large number of tasks (1,600+), yet the marginal benefit on MMLU is modest. The paper and Iyer et al. (2022) both note this may be due to Super-Natural Instructions' "unique input format and instruction design"—the tasks use a specific few-shot formatting pattern that may not transfer well to the diverse prompt formats used in MMLU evaluation.
Removing Chain-of-Thought causes a moderate drop on MMLU (47.3% to 46.8%) but a dramatic drop on Chain-of-Thought evaluation (29.1% vs. 41.4% with CoT data)—confirming that CoT training is essential specifically for CoT evaluation, with more modest spillover to other settings.
Removing Program Synthesis or Dialog causes minimal or slightly negative drops on MMLU, suggesting these sources are less critical for the Held-Out generalization measured by MMLU.

The "All (Weighted)" row in Table 2. The final row—achieving 48.1% on MMLU vs. 47.3% for equal weighting—represents a mixture where sources are weighted according to their apparent contribution rather than equally. The paper is deliberately vague about the exact weights, stating:

"We used these findings to significantly narrow the mixture weights search space, and used our practitioner's intuition from there. This strategy is simple but effective, as shown in Table 1, but leaves ample room for more sophisticated future work."

This is an honest admission of a limitation: the weights are not algorithmically optimized (e.g., via a hyperparameter sweep or learned weighting). The search space was narrowed by the leave-one-out experiments, and the final weights reflect human judgment informed by those experiments. The resulting Flan-T5 model benefits from this weighting but does not necessarily represent the global optimum.

Why mixture balancing matters for the scaling analysis. The mixture balancing experiments provide crucial context for interpreting Figure 4 (task scaling). If Super-Natural Instructions tasks have limited benefit per task, then adding hundreds of Super-Natural Instructions tasks to the training mixture will produce diminishing returns compared to adding the same number of Flan 2021 or T0-SF tasks. The scaling curves in Figure 4 reflect the specific composition of the Flan 2022 collection, where many tasks come from Super-Natural Instructions. A collection with a different source composition could exhibit different scaling behavior—more tasks might help more if the additional tasks come from high-quality, diverse sources.

The labeled vs. unlabeled distinction in task utility. The paper does not fully unpack this, but the difference between Flan 2021/T0-SF (high-benefit) and Super-Natural Instructions (lower-benefit) likely stems from task curation and format standardization. Flan 2021 and T0-SF consist of well-established NLP benchmarks (question answering, natural language inference, summarization) with templates that closely mirror how those tasks are evaluated at test time. Super-Natural Instructions includes many crowdsourced tasks with more idiosyncratic formats, instructions that may not generalize well, and tasks that may be too niche to transfer. This is consistent with a broader theme in multi-task learning: task relevance to the evaluation distribution matters at least as much as task count.

Reconciling with OPT-IML's mixture weights. Iyer et al. (2022) also found that Flan 2021, T0-SF, and T5-style tasks were most beneficial, and that Super-Natural Instructions had limited scaling benefits. The paper notes the similarity in findings while acknowledging differences in implementation: OPT-IML's collections, templates, processing, and exact mixing ratios are not publicly released, preventing direct comparison. The fact that independent works converged on similar conclusions about which sources matter most strengthens the evidence.

Single-Task Fine-Tuning Protocol: Measuring Instruction Tuning as Pretraining for Downstream Tasks

Section 4 evaluates a pragmatic use case: using the instruction-tuned Flan-T5 checkpoint as a starting point for single-task fine-tuning on a specific downstream task, rather than starting from the raw pretrained T5 checkpoint.

The three-way comparison (Figure 5). For each downstream task, the paper compares:

T5 → FT (blue bars): fine-tune the raw pretrained T5-XL model on the target task. This is the conventional baseline—the standard procedure a practitioner would follow without instruction tuning.
Flan-T5 without FT (beige bars): use the instruction-tuned Flan-T5 checkpoint directly, without any further task-specific fine-tuning. This measures whether instruction tuning alone provides sufficient task knowledge.
Flan-T5 → FT (red bars): take the Flan-T5 checkpoint and further fine-tune it on the target task. This measures the combined benefit of instruction tuning + task-specific adaptation.

The downstream tasks include both Held-In tasks (which Flan-T5 has seen during instruction tuning—ANLI, ARC, BoolQ, CosmosQA, RTE, SQuAD V2, AI2 Science) and Held-Out tasks (which Flan-T5 has never seen—CondaQA, CxC, MedNLI, PubmedQA, WANLI).

Training protocol (Appendix A.2). Single-task fine-tuning uses: 100,000 steps, constant learning rate of 0.001, dropout probability of 0.1, batch size of 128 length-512 sequences. A checkpoint is saved every 20 steps, and test performance is evaluated on the checkpoint with the highest validation performance. For PubmedQA, the unlabeled and artificially generated QA instances are excluded, and only the original human-labeled data is used. For CxC (Crisscrossed Captions), only the text-text portion is used. For tasks with fewer than 1,000 training examples, results are averaged across 3 random seeds.

The convergence speed finding (Figure 6). On five Held-Out tasks—WANLI, MedNLI, CondaQA, PubmedQA, and CxC—the paper plots validation accuracy against the number of fine-tuning steps for both Flan-T5-XL and T5-XL. The curves consistently show that Flan-T5 converges faster (reaches a given accuracy level in fewer steps) and peaks at a higher accuracy than T5. For WANLI, Flan-T5 reaches ~70% accuracy at 50 steps, where T5 requires ~150 steps. For PubmedQA, Flan-T5 peaks around 84% vs. T5's ~82%. This convergence speed advantage is the basis for the paper's "green-AI" argument: if instruction tuning is a one-time cost, but it reduces fine-tuning time across many downstream applications, the aggregate compute savings can be positive.

The pareto improvement claim. The paper states that "For both sets of Held-In and Held-Out tasks examined, finetuning Flan-T5 offers a pareto improvement over finetuning T5 directly." A pareto improvement means Flan-T5 → FT is better than or equal to T5 → FT on every task examined, with strict improvement on at least some. The Figure 5 bar chart visualizes this: the red bars (Flan-T5 → FT) are consistently at or above the blue bars (T5 → FT) across all 12 tasks. The improvement margins vary—some tasks show small gains (e.g., +1.0 on CxC), others show large gains (e.g., +16.7 on ANLI) —but the direction is consistently positive.

The mechanism for improved fine-tuning. The paper does not deeply analyze why Flan-T5 fine-tunes better than raw T5, but the likely mechanisms are: (1) Flan-T5 has already learned generic task-following skills (how to interpret instructions, how to map inputs to outputs in a structured format) that transfer to new tasks, reducing the amount of new learning required; (2) Flan-T5's representations have been shaped by exposure to thousands of diverse tasks, making them more general and adaptable than T5's representations, which are optimized only for language modeling; and (3) the instruction tuning process acts as a form of meta-learning—the model learns how to efficiently adapt to new tasks from limited data, which is exactly what single-task fine-tuning requires.

4. Key Insights and Innovations

Innovation 1: Instruction Tuning Design Is Decomposable—and the Critical Choices Are Not the Obvious Ones

Prior to this work, the instruction tuning literature treated each public collection as a monolithic artifact. Flan 2021, P3, T0, Super-Natural Instructions, and OPT-IML each represented a different combination of task selection, template engineering, prompt format, and training procedure, and when one collection outperformed another, it was unclear which design decision was responsible. The field had accumulated methods without an evidence-backed decomposition of which techniques actually matter, in which contexts, and by how much. This paper introduces the first systematic ablation framework for instruction tuning: start from the full recipe, remove one component at a time, and measure the marginal contribution across Held-In, Held-Out, and Chain-of-Thought evaluation settings. This is a methodological innovation—not a new technique, but a new way of studying techniques—that transforms instruction tuning from an alchemical practice into something approaching experimental science.

What makes this contribution distinctive is not any single ablation result but the structure of the analysis. By running all comparisons on the same base model (T5-XL, 3B), the same evaluation benchmarks, and the same training framework, the paper eliminates the confounding variables—model architecture, pretraining corpus, model size, evaluation protocol—that had made cross-collection comparisons uninterpretable. Table 1 is not merely a leaderboard; it is a diagnostic tool that isolates four independent axes of improvement (mixed prompts, input inversion, mixture balancing, Chain-of-Thought training) and quantifies their contributions. The finding that mixture balancing accounts for an 11-point BBH zero-shot gap, while input inversion accounts for an 8.6-point MMLU gap, tells a practitioner where to invest effort when building their own instruction tuning pipeline. Prior work offered no such guidance: a practitioner choosing between Flan 2021 and P3 was choosing between entire systems, not between individual design decisions with known effect sizes.

This ablation framework also surfaces a non-obvious interaction: the techniques are not merely additive—they contribute to different evaluation settings. Input inversion helps Held-Out generalization but not Held-In maintenance (Table 1: MMLU drops 8.6 points without inversion, Held-In is unchanged). Chain-of-Thought training helps CoT evaluation dramatically (BBH-CoT few-shot drops 23.2 points) but provides only modest spillover to standard evaluation. Mixed prompt training helps both zero-shot and few-shot evaluation simultaneously, defying the expected specialization tradeoff. A practitioner who only tracks aggregate accuracy would miss these differential effects and might discard a technique that provides large benefits in their specific deployment setting. The paper's evaluation taxonomy—separating Held-In, Held-Out, and CoT into distinct reporting categories—is itself a contribution, forcing future work to be precise about which kind of generalization is being improved.

Compared to the pretraining scaling laws literature (Hoffmann et al., 2022; Kaplan et al., 2020), this work is less mathematically precise—it does not propose functional forms for performance as a function of task count or mixture weight—but it establishes the precedent that instruction tuning design decisions can and should be studied with the same rigor as pretraining hyperparameters. This is a conceptual shift: instruction tuning moves from being a downstream application of pretrained models to being a first-class optimization problem with its own design space, its own ablation methodology, and its own interaction effects that are not predictable from pretraining behavior alone.

The key evidence is Table 1, which the reader has already seen: each row represents a subtraction experiment isolating one component's marginal contribution. The 23.2-point CoT drop, the 12.6-point few-shot Held-In drop when removing few-shot templates, and the asymmetric Held-In/Held-Out effects of input inversion collectively demonstrate that the design decisions are both large in magnitude and structurally separable. This is the intellectual infrastructure that the rest of the paper's contributions build on.

Innovation 2: Mixed Prompt Training Violates the Expected Specialization Tradeoff—and the Finding Is Strongest Where It Should Be Weakest

The dominant assumption across the first wave of instruction tuning was that models specialize to their training format: train on zero-shot prompts, and the model performs best on zero-shot evaluation; train on few-shot prompts, and it performs best on few-shot evaluation. This assumption was baked into the design of every major public collection—Flan 2021 and P3/T0 trained exclusively for zero-shot, MetaICL and Super-Natural Instructions trained for few-shot. It was a reasonable assumption: capacity-constrained models should allocate representational resources to the format they will encounter at test time, and mixing formats should dilute that specialization, creating a tradeoff where each setting improves at the expense of the other.

The paper's finding that adding as little as 5% few-shot templates to a mostly zero-shot training mixture improves zero-shot performance (Figure 3, left) is surprising not because it is a large absolute gain—the improvement is on the order of a few percentage points—but because it inverts the expected sign of the effect. Adding few-shot data does not hurt zero-shot performance, even at very low model scales (3B parameters) where capacity constraints are most binding. Conversely, adding 10% zero-shot data to a mostly few-shot mixture improves few-shot performance (Figure 3, right). The optimal mixture lies somewhere in the 10–90% range for both settings, not at the extremes.

Why is this intellectually distinctive? First, it reframes the relationship between training-time format and evaluation-time format from a zero-sum tradeoff to a mutually reinforcing synergy. The mechanism—though the paper does not prove it—is plausibly that exposure to multiple formats teaches the model a more abstract representation of the underlying task that is robust to format variation. Few-shot exemplars during training may teach the model what constitutes a valid input-output mapping for a task type, a signal that transfers to zero-shot settings where the model must infer the mapping from the instruction alone. Zero-shot training may prevent overfitting to the presence of exemplars (i.e., learning "when I see exemplars, I should copy their format" rather than "I should perform the task"), improving few-shot generalization.

Second, the finding generalizes across Held-In and Held-Out settings (both panels of Figure 3), which means the synergy is not specific to tasks the model has seen during training—it extends to genuinely novel tasks. This distinguishes mixed prompt training from simple data augmentation (where seeing more variants of the same task helps on that specific task) and elevates it to a meta-learning strategy: the model learns how to learn from instructions in a format-agnostic way.

Third, the finding is in direct tension with the concurrent practice of training separate models for different prompt settings. OPT-IML, the closest comparison work, also used mixed prompts but did not ablate this choice in isolation (Iyer et al., 2022). InstructGPT (Ouyang et al., 2022) mixed prompts as a design decision but did not report a controlled comparison against single-format training. Without an ablation, it was impossible to know whether mixed training helped or whether these models succeeded despite format mixing. The paper provides that missing evidence, and the evidence says mixing is unequivocally beneficial even for small models.

The qualification worth noting: the improvement from mixing is modest in absolute terms (roughly 2% on Held-In tasks, per the abstract) and the U-shaped curves in Figure 3 show that any mixture in the 10–90% range works about equally well. This is not a hyperparameter-sensitive effect that requires careful tuning—it is a robust finding that pushing the mixture to either extreme (0% or 100%) is uniformly worse. For a practitioner, the actionable takeaway is not "find the exact optimal mixture ratio" but "don't train exclusively for one prompt format"—a simple rule that prior collections had universally violated.

Innovation 3: Task Count and Task Quality Are Separable Axes—and the Scaling Story Is More Nuanced Than "More Tasks Help"

The first wave of instruction tuning established a rough consensus: more tasks → better generalization. Flan 2021 scaled from tens to 62 tasks and observed improvements. Super-Natural Instructions scaled to 1,600+ tasks and reported gains. The natural extrapolation—reinforced by the pretraining scaling laws mindset—was that instruction tuning would follow a similar pattern to language modeling, where more data (in this case, more tasks) monotonically improves performance up to very large scales.

This paper complicates that story in two ways. First, it demonstrates that Held-In and Held-Out performance have opposite scaling behavior (Figure 4): as the number of training tasks increases beyond roughly 200–400, Held-In accuracy on seen tasks decreases (for all model sizes), while Held-Out accuracy on unseen tasks continues to increase log-linearly. This is a capacity allocation tradeoff: the model cannot simultaneously maintain peak performance on every seen task while also generalizing to thousands of unseen ones. The Held-In decline is a real cost of task scaling that prior work had not systematically measured, since most prior evaluations focused exclusively on Held-Out generalization.

Second, and more fundamentally, the paper shows through mixture balancing experiments (Section 3.5, Table 2) that not all task sources are equally beneficial per task, and the apparent benefit of "more tasks" in scaling curves is actually a confound: adding more tasks from the Flan 2022 collection also adds tasks from higher-quality sources (Flan 2021, T0-SF), making it impossible to separate the effect of quantity from the effect of source quality. Super-Natural Instructions contributes many tasks but has limited marginal benefit on MMLU; Flan 2021 contributes fewer tasks but has the largest marginal benefit. If scaling curves were plotted with task count on the x-axis but tasks were only drawn from Super-Natural Instructions, the slope would be shallower. The paper does not produce this counterfactual curve, but the logic is implied by the leave-one-out results.

This is a conceptual contribution rather than a new method: it introduces the idea of task source quality as a first-class variable in instruction tuning design, distinct from task count. Prior work had implicitly assumed tasks were roughly interchangeable—a question-answering task from Super-Natural Instructions counted the same as one from Flan 2021 in scaling analyses. The paper's evidence suggests this assumption is wrong, and that optimal instruction tuning requires curation (selecting high-quality, format-diverse sources) in addition to aggregation (collecting many sources).

The finding also provides a diagnostic tool for future work: when a new instruction tuning collection reports improved performance, researchers should ask whether the gain comes from adding more tasks, adding higher-quality tasks, or both. The paper's leave-one-out methodology (Table 2) offers a template for answering this question: train with and without a specific source, and measure the marginal contribution. If the field adopts this practice, instruction tuning research will move from "my collection is bigger" to "my sources are better chosen," which is a more productive axis of competition.

The connection to the concurrent OPT-IML work (Iyer et al., 2022) strengthens this insight: OPT-IML independently converged on the same source quality ranking (Flan 2021 and T0-SF as most beneficial, Super-Natural Instructions as less so), despite using different models, templates, and training procedures. When independent research teams converge on the same ranking, the signal is likely real and not an artifact of implementation details.

Innovation 4: Instruction-Tuned Models Are Computationally Efficient Starting Points for Single-Task Fine-Tuning—Reframing Instruction Tuning as an Investment, Not a Cost

Section 4 makes a pragmatic argument that shifts the framing of instruction tuning from a capability enabler (it lets models generalize to unseen instructions) to an efficiency multiplier (it produces checkpoints that make downstream task adaptation faster and better). The evidence is that Flan-T5 converges faster (in fewer fine-tuning steps) and to higher peak accuracy than raw T5 on every tested downstream task, both Held-In and Held-Out (Figures 5 and 6). This is not merely a performance gain—it is an efficiency gain that amortizes the one-time cost of instruction tuning across many downstream applications.

What makes this distinctive is not the finding itself (prior work had shown that multi-task fine-tuning could improve downstream transfer, e.g., Aghajanyan et al., 2021; Aribandi et al., 2021; Liu et al., 2022b) but the explicit reframing of instruction tuning as a public good with green-AI implications. The paper argues:

"While instruction tuning is more computationally-expensive than single-task finetuning, it is a one-time cost. On the contrary, pretrained models that require extensive finetuning become more costly when aggregating over many millions of additional training steps."

If the NLP community adopts instruction-tuned checkpoints as standard starting points—rather than each practitioner fine-tuning from the raw pretrained model—the aggregate compute savings across thousands of downstream applications could be substantial. This is not a claim the paper quantifies (it does not estimate aggregate FLOPs savings), but the argument is structurally sound: instruction tuning is a fixed cost that reduces variable costs across many deployments.

The evidence in Figure 6 is particularly compelling because it shows the convergence advantage on Held-Out tasks—tasks the instruction-tuned model has never seen. For WANLI, Flan-T5 reaches ~70% accuracy in roughly 50 steps, where T5 needs approximately 150 steps (a 3× reduction in steps to reach equivalent performance). For MedNLI, the gap is smaller but the direction is consistent. This implies that instruction tuning teaches the model a general-purpose "task adaptation" skill that transfers even to tasks outside the fine-tuning distribution—a form of meta-learning that makes the model a more efficient learner.

The "Pareto improvement" framing—Flan-T5 → FT is never worse than T5 → FT and strictly better on multiple tasks—is rhetorically powerful because it eliminates the "but what if instruction tuning hurts my specific task?" objection. A practitioner can adopt Flan-T5 as a starting checkpoint with the guarantee (at least across the tested tasks) that they will not lose performance compared to starting from T5, and they may gain both speed and accuracy. This removes a barrier to adoption that had likely inhibited instruction-tuned model usage in applied settings.

The connection to the broader trend of model reuse (foundation models as platforms, not just tools) is implicit but important. Just as BERT established pretrained encoders as standard starting points for NLP tasks in 2019, the paper argues that instruction-tuned generative models should become standard starting points for text generation tasks in 2023 and beyond. The efficiency argument provides the economic rationale for this shift: the one-time cost of producing the instruction-tuned checkpoint is borne by a research lab (or a community effort), and the benefits are distributed across all downstream users.

A caveat the paper does not fully explore: the convergence advantage is demonstrated on T5-XL (3B parameters) fine-tuned with full model updating. Whether the same advantage holds for parameter-efficient fine-tuning methods (LoRA, prompt tuning, adapters), which are increasingly used for large models where full fine-tuning is impractical, is an open question. The paper cites work showing that parameter-efficient methods can match full fine-tuning (Lester et al., 2021; Hu et al., 2021), and the convergence advantage likely transfers to these settings as well, but this is not directly tested.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation uses three categories of benchmarks, all distinct from the Flan 2022 training collection for Held-Out purposes. Held-In tasks consist of 8 validation sets: 4 question answering datasets (BoolQ, ARC Easy, ARC Challenge, AI2 Middle School Science Exams) and 4 natural language inference datasets (ANLI R1, R2, R3, and RTE). Chain-of-Thought (CoT) tasks consist of 5 reasoning datasets prepared with step-by-step explanation prompts: GSM8K, StrategyQA, SVAMP, Asdiv, and CommonsenseQA. Held-Out tasks consist of MMLU (57 exams spanning sciences, humanities, and other subjects; Hendrycks et al., 2020) and BIG-Bench Hard (BBH, 23 challenging tasks from BIG-Bench where PaLM underperforms human raters; Suzgun et al., 2022). MMLU tasks were explicitly removed from the Super-Natural Instructions portion of the Flan 2022 training collection to ensure Held-Out status. Dataset-specific metrics are listed in Table 3 (e.g., ANLI uses 3-class F1, BoolQ uses AUC-ROC, GSM8K uses exact-match accuracy); the paper reports these as "Accuracy (%)" for readability when compatible metrics are aggregated.
Base model(s). All instruction tuning experiments use T5-LM, the prefix-language-model-adapted version of T5 (Lester et al., 2021), with the XL size (3B parameters) as the default. The authors justify this choice in Section 3.1: "We felt XL was appropriately sized to run large-scale systematic ablations, while being sufficiently large to draw general conclusions." For the task scaling analysis (Section 3.3), five sizes are used: Small, Base, Large, XL (3B), and XXL (11B). For collection comparisons, all models except OPT-IML-Max 175B are T5-XL (3B) with identical pretraining checkpoints, ensuring controlled comparison. The single-task fine-tuning experiments (Section 4) compare Flan-T5-XL against T5-XL, both starting from the same pretrained base.
Metrics. The primary metric is accuracy (or the dataset-specific equivalent listed in Table 3), computed as the mean across all tasks in each evaluation category. For Held-In, the score is the mean of 8 task metrics. For CoT, it is the mean of 5 task metrics. For MMLU, it is the mean accuracy across all 57 exams. For BBH, it is the mean across 23 tasks. Results are reported separately for zero-shot and few-shot evaluation settings, and for BBH also with Chain-of-Thought prompting (BBH-CoT). In Table 1, each cell reports "zero-shot / few-shot" metrics separated by a slash. For single-task fine-tuning (Figure 5 and Figure 6), validation accuracy is tracked every 20 steps, and test performance is reported on the checkpoint with the highest validation accuracy.
Baselines. The paper compares against several public instruction tuning collections, each fine-tuned on the same T5-XL base model: (1) Flan 2021 (Wei et al., 2021)—62 tasks with zero-shot templated instructions; (2) P3++ (Sanh et al., 2021; Bach et al., 2022)—the Public Pool of Prompts, trained for zero-shot; (3) Super-Natural Instructions (Wang et al., 2022c)—1,600+ tasks with few-shot exemplars; (4) OPT-IML-Max 30B and 175B (Iyer et al., 2022)—concurrent work with ~2,200 tasks, included as reported benchmarks despite different model sizes and architectures; and (5) GLM-130B (Zeng et al., 2022)—another concurrent model, included in Table 1 for MMLU comparison. For single-task fine-tuning baselines (Section 4), the conventional baseline is T5-XL fine-tuned directly on the target task from the pretrained checkpoint.
Generation budget / compute accounting. For instruction tuning, "compute" is measured in training steps and task count rather than inference FLOPs. All T5-XL models in the main comparisons are trained for the same number of steps on instruction tuning data, making the training budget equivalent across methods. For the task scaling experiments (Section 3.3), the "budget" is the number of training tasks (8, 25, 50, 100, 200, 400, 800, or all 1,836), and all models within a size class are trained for the same number of steps. In the single-task fine-tuning section (Section 4), compute is measured in number of fine-tuning steps (up to 100,000), with the convergence speed analysis (Figure 6) comparing how many steps are needed to reach a given accuracy level. The paper does not report total FLOPs or GPU-hours, focusing instead on task count and step count as the relevant scaling variables.
Cross-validation / statistical protocol. The paper does not use k-fold cross-validation for the main instruction tuning experiments—models are trained once on the full Flan 2022 collection using the predefined mixture weights. For single-task fine-tuning with datasets containing fewer than 1,000 training examples, results are averaged across 3 random seeds (Appendix A.2). For tasks without a predefined validation split, 1,024 training examples are held out for validation; for tasks without a test split, the original validation set is used as the test set and 1,024 training examples are held out for validation. The mixture balancing experiments (Section 3.5, Table 2) train each ablated variant once and report the resulting MMLU accuracy; there is no reported variance across random seeds for these experiments. The scaling law experiments (Section 3.3) train each task-subset configuration once per model size without reported variance.

Main Quantitative Results

The paper's empirical findings are organized around four axes of investigation: (1) the effect of each methodological component, measured via subtraction ablations from the full Flan 2022 recipe; (2) the impact of mixed prompt training across different mixing ratios; (3) performance scaling as a function of task count and model size; and (4) single-task fine-tuning efficiency gains from using Flan-T5 as a starting checkpoint. I address each in turn.

Method Ablations: The Marginal Contribution of Each Design Decision

Table 1 (top) presents the core subtraction experiments. Starting from the full Flan-T5 XL recipe (which achieves 73.8% / 74.8% on Held-In zero-shot and few-shot, 35.8% / 34.1% on CoT, 50.3% / 52.4% on MMLU, and 26.2% / 39.3% on BBH), each row removes one component and reports the resulting performance.

Removing Chain-of-Thought training ("− CoT" row). The most dramatic single-component effect is on Chain-of-Thought evaluation: BBH-CoT few-shot drops from 35.2% to 12.0%, a decline of 23.2 percentage points. The BBH-CoT zero-shot performance similarly drops from 33.9% to 18.2% (a 15.7-point decline). The standard CoT evaluation (the 5 reasoning datasets) drops from 35.8% / 34.1% to 28.8% / 24.6%, confirming that CoT training is essential specifically for CoT-format evaluation. However, removing CoT data also affects non-CoT settings: MMLU zero-shot drops from 50.3% to 47.5% (a 2.8-point decline), and BBH zero-shot drops from 26.2% to 18.2% (an 8.0-point decline). This suggests that CoT training provides modest cross-format transfer—exposure to step-by-step reasoning improves the model's general reasoning capability even when CoT prompting is not used at evaluation time.

Removing input inversion ("− Input Inversion" row). The effect is highly asymmetric across evaluation settings. Held-In performance is essentially unchanged: 73.8% / 74.1% vs. the full model's 73.8% / 74.8%. The model can maintain its performance on seen tasks without inverted examples. But Held-Out generalization suffers substantially: MMLU zero-shot drops from 50.3% to 41.7% (an 8.6-point decline), and BBH zero-shot drops from 26.2% to 18.4% (a 7.8-point decline). CoT evaluation also drops: from 35.8% / 34.1% to 32.2% / 23.5%, with the few-shot decline being particularly large (10.6 points). The pattern is that input inversion primarily benefits tasks the model has never seen—the structural diversity from reversed input-output pairs transfers to genuine generalization—while providing no marginal benefit for tasks already well-represented in forward-direction examples.

Removing mixture balancing ("− Mixture Balancing" row). When all task sources are weighted equally rather than using the empirically-informed weights, performance degrades across nearly all settings. BBH zero-shot drops from 26.2% to 15.1% (an 11.1-point decline)—the largest single-setting drop in this ablation. BBH few-shot drops from 39.3% to 24.3% (a 15.0-point decline). MMLU zero-shot drops from 50.3% to 45.4% (a 4.9-point decline). Held-In performance drops from 73.8% / 74.8% to 71.2% / 73.1%. Even BBH-CoT drops from 33.9% / 35.2% to 13.8% / 15.4% (a roughly 20-point decline in both settings). This ablation establishes that how training data is weighted across sources is nearly as consequential as which sources are included—a finding that had been largely overlooked in prior work.

Removing few-shot templates ("− Few Shot Templates" row). When the model is trained only with zero-shot templates (matching the training format of Flan 2021 and P3++), zero-shot performance declines modestly: Held-In zero-shot drops from 73.8% to 72.5% (a 1.3-point decline), and MMLU zero-shot drops from 50.3% to 47.3% (a 3.0-point decline). But few-shot performance collapses: Held-In few-shot drops from 74.8% to 62.2% (a 12.6-point decline), and MMLU few-shot drops from 52.4% to 38.7% (a 13.7-point decline). This asymmetric effect demonstrates that few-shot training templates are essential for few-shot evaluation—the model must see exemplars during training to learn how to use exemplars at test time. The modest zero-shot improvement from keeping few-shot templates (comparing the "− Few Shot Templates" row to the full model on zero-shot metrics) confirms the mixed-prompt synergy: exposure to few-shot templates during training helps zero-shot performance as well, though the effect on zero-shot is smaller than the reverse dependency.

Collection comparisons (Table 1, bottom). When comparing Flan-T5 XL against T5-XL models fine-tuned on alternative instruction tuning collections, the margins are consistently large. Against the next-best T5-XL collection (varying by setting), Flan 2022 achieves improvements of +3.3% on Held-In zero-shot, +12% on Held-In few-shot (the full row header reads "+3.3 / +12"), +10.2% / +8.5% on CoT, +4.2% / +17.6% on MMLU, and −1.9% / +8.5% on BBH (the −1.9% on BBH zero-shot indicates that Flan 2021 achieves slightly higher BBH zero-shot performance; Flan 2021's zero-shot training is apparently sufficient for standard BBH, while Flan 2022's mixed training shows its advantage primarily in few-shot BBH). Most notably, Flan-T5 XL (3B) outperforms OPT-IML-Max 175B (~58× larger) on MMLU zero-shot (50.3% vs. 49.1%) and BBH few-shot (39.3% vs. 35.7%), and on MMLU few-shot (52.4% vs. 47.1%). This establishes that carefully-designed instruction tuning at small scale can match or exceed less-optimized instruction tuning at much larger scale.

Mixed Prompt Training: Sweeping the Ratio of Zero-Shot to Few-Shot Templates

Figure 3 presents the systematic sweep of the percentage of few-shot templates in the training mixture, from 0% (all zero-shot training) to 100% (all few-shot training), with evaluation in both zero-shot and few-shot settings. The results are shown for Held-In tasks (left panel) and Held-Out MMLU (right panel).

Held-In task performance (Figure 3, left). For zero-shot evaluation (presumably the lower curve, though the figure description in the paper does not specify curve colors or styles), performance improves from the 0% few-shot baseline as few-shot data is added, reaching a peak somewhere in the 5–90% few-shot range, and declining slightly if training is 100% few-shot. The improvement from adding just 5% few-shot data is notable—the paper emphasizes this as the minimum viable mixing ratio. For few-shot evaluation (presumably the upper curve), performance at 100% few-shot training is suboptimal; adding 10%+ zero-shot data improves few-shot evaluation accuracy. The curves are relatively flat in the intermediate range (roughly 10–90%), suggesting that the exact mixing ratio is not hyperparameter-sensitive so long as both formats are represented. The stars in the figure indicate peak performance for each evaluation setting.

MMLU Held-Out performance (Figure 3, right). The same pattern holds: zero-shot MMLU evaluation improves when some few-shot data is present in training, and few-shot MMLU evaluation improves when some zero-shot data is present. The optimal mixing range is again broad—any mixture between roughly 10% and 90% few-shot templates outperforms either extreme.

The key quantitative claim from this analysis appears in the abstract and Section 3.2: "adding just 10% few-shot prompts improves zero-shot prompting results by 2%+." The specific 2%+ figure appears to reference the Held-In zero-shot improvement, though the paper does not break out the exact absolute percentage at the 10% mixing point—it is inferred from the curve shape in Figure 3.

A secondary finding embedded in Figure 3 is that the few-shot evaluation curve consistently sits above the zero-shot evaluation curve for both panels, reflecting the well-known advantage of providing exemplars at inference time. The mixing experiment shows that this gap can be narrowed (or the zero-shot baseline raised) through mixed-format training.

Task Scaling: Performance as a Function of Task Count and Model Size

Figure 4 presents scaling curves for Held-In tasks (left panel) and Held-Out MMLU (right panel) as a function of the number of fine-tuning tasks (x-axis, log scale from approximately 8 to 1,836) and model size (separate curves for Small, Base, Large, XL, and XXL).

Held-In scaling (Figure 4, left). For all model sizes, Held-In accuracy initially increases from 8 tasks to a peak somewhere between roughly 100 and 400 tasks, then declines as more tasks are added beyond that peak. The peak location and the severity of the decline are model-size-dependent: T5-Small peaks early (around 50–100 tasks) and degrades noticeably by 1,836 tasks, while T5-XXL peaks later (around 400–800 tasks) and shows minimal degradation at the full 1,836 tasks. The gold stars on the figure indicate each model size's peak performance. The absolute values are not quoted precisely in the text, but the figure's y-axis for Held-In ranges from roughly 40% (Small) to 90% (XXL), with the degradation at maximum task count being more pronounced for smaller models.

Held-Out scaling (Figure 4, right). MMLU accuracy increases monotonically (log-linearly) with the number of training tasks for all model sizes, with the steepest gains in the 8-to-400-task range and continued but diminishing returns out to 1,836 tasks. Only T5-Small appears to saturate or slightly decline before the maximum task count; all larger models continue to improve. The y-axis for MMLU ranges from roughly 20% (Small at 8 tasks) to roughly 50% (XXL at 1,836 tasks). The paper's text interpretation is that "even T5-Base may not have exhausted its capacity with thousands of tasks, and the largest LMs could benefit from thousands more tasks for Held-In and Held-Out task performance."

The interaction with model size. The gap between model sizes on Held-Out tasks widens as the number of tasks increases: at 8 tasks, T5-Small, Base, and Large are relatively close together, but by 1,836 tasks, the curves have clearly separated, with XXL substantially outperforming all smaller sizes. On Held-In tasks, the model-size gap is present at all task counts but widens at higher task counts as smaller models degrade more quickly.

The paper explicitly warns that these scaling curves assume all tasks are interchangeable units, which Section 3.5 shows is not true—the specific composition of the task mixture (which sources are included and in what proportion) matters independently of task count.

Single-Task Fine-Tuning Efficiency: Flan-T5 as a Starting Checkpoint

Figure 5 (bar chart) and Figure 6 (convergence curves) present the results of using Flan-T5 as a starting point for single-task fine-tuning compared to starting from raw T5.

The three-way comparison (Figure 5). Across 12 downstream tasks (7 Held-In and 5 Held-Out), three conditions are compared: T5 fine-tuned directly on the target task (blue bars), Flan-T5 used without further fine-tuning (beige bars), and Flan-T5 fine-tuned on the target task (red bars). The paper reports that Flan-T5 → FT (red) achieves a Pareto improvement over T5 → FT (blue) on every task examined—red bars are always at or above blue bars. The improvement margins vary by task: ANLI shows the largest gain (the figure reports +16.7 for the red-over-blue difference on ANLI, inferred from the bar chart), while tasks like CxC show smaller gains (+1.0). The beige bars (Flan-T5 without fine-tuning) sometimes exceed the blue bars (T5 with fine-tuning), "usually where finetuning data is limited for a task," indicating that instruction tuning alone can match or surpass task-specific training on data-sparse problems.

Convergence speed (Figure 6). On five Held-Out tasks (WANLI, MedNLI, CondaQA, PubmedQA, CxC), curves plot validation accuracy against the number of fine-tuning steps for both Flan-T5-XL and T5-XL. Flan-T5 consistently converges faster: on WANLI, Flan-T5 reaches approximately 70% accuracy at roughly 50 steps, while T5 requires approximately 150 steps to reach the same level—roughly a 3× reduction in required steps. On PubmedQA, the convergence advantage is present but the gap in peak accuracy is smaller (Flan-T5 peaks around 84% vs. T5's ~82%). On CxC, the convergence advantage is visible in the first 50 steps but the curves largely converge by 200 steps. The paper's text states that "Flan-T5 converges much more quickly than T5 during single target finetuning, as well as peaking at higher accuracies."

The computational efficiency argument. The paper frames this finding as having "green-AI incentives": instruction tuning is a one-time cost, but if it reduces fine-tuning steps by a factor of 2–3× across thousands of downstream applications, the aggregate compute savings are positive. This argument is qualitative rather than quantitative—the paper does not estimate total FLOPs for instruction tuning vs. total saved downstream FLOPs—but the convergence curves provide clear evidence that per-task training is more efficient when starting from Flan-T5.

Ablation Studies and Robustness Checks

Mixture source leave-one-out (Table 2). Trains models on the full Flan 2022 collection minus one data source at a time, measuring the impact on Held-In, CoT, and MMLU metrics. The key finding is the differential importance of sources: removing Flan 2021 causes the largest MMLU drop (from 47.3% to 45.7%), removing T0-SF causes the second-largest (to 44.7%), while removing Super-Natural Instructions causes a minimal drop (to 46.8%) despite contributing many tasks. Removing CoT data primarily affects CoT evaluation (dropping from 41.4% to 29.1%), and removing Program Synthesis or Dialog has minimal or slightly positive effects on MMLU. The final row ("All (Weighted)") demonstrates that informed weighting improves MMLU to 48.1% over the equal-weighting baseline of 47.3%. A notable negative finding: removing Program Synthesis actually improves some metrics (Held-In zero-shot rises from 64.9% to 66.9%), suggesting that program synthesis tasks, as included, may introduce interference rather than benefit for certain evaluation settings.

Model size × task count interaction (Figure 4, cross-model comparison). While not framed as an ablation, the scaling curves implicitly test whether the benefits of task scaling depend on model size. The answer is yes: smaller models saturate and degrade on Held-In tasks sooner (T5-Small peaks around 50–100 tasks), while larger models both peak later and degrade less. On Held-Out tasks, the log-linear improvement continues for all sizes except Small, but the slope is steeper for larger models, meaning larger models extract more benefit per additional task. This interaction validates that task scaling and model scaling are complementary rather than independent.

Few-shot exemplar count variation. The training procedure varies the number of few-shot exemplars (2, 3, or 5) within each training batch—this is a design choice rather than an explicitly ablated variable, but it contributes to the input diversity that the paper argues is beneficial. The paper does not ablate fixed vs. variable exemplar counts, leaving open the question of whether the variation itself matters or simply having exemplars of any consistent count would suffice.

Input inversion rate (30% for Dialog, Program Synthesis, CoT tasks). The mixing rate of 30% inverted examples is chosen "for simplicity, approximately mirroring prior work," and the paper explicitly states it leaves "the large space of exploration for future work." No sweep over inversion rates is performed, so the optimality of 30% is untested. It is possible that higher or lower rates would produce different effects, particularly on the CoT tasks where the inversion permutations are most complex (Figure 7 in Appendix B shows multiple permutation types for each CoT example).

Training template structural variations (not a formal ablation). The paper mentions in Section 3.6 that its templatization procedure varies "where the instruction is placed (before or after few-shot prompts), the spacing and separators between few-shot and Chain-of-Thought prompts, and the formatting permutations of answer options (and their targets) for multiple-choice examples, which sometimes includes and sometimes excludes answer options in the inputs or exemplars." These variations were developed through iterative improvement but are not individually ablated. The paper notes: "While we do not have dedicated experiments comparing many iterations of development, we found these procedures dramatically augment input variety and showed repeated performance improvements." This is an honest admission of a methodological gap—the template engineering is sophisticated but its individual components are not isolated.

Flan 2021 vs. P3++ task overlap characterization. The collection uses T0-SF to denote "the subset of P3++ that is not in Flan 2021," deliberately separating the contributions of these two sources in the mixture balancing experiments. This is a necessary design choice for the leave-one-out analysis in Table 2, ensuring that removing "T0-SF" removes only novel tasks not already in Flan 2021. Without this separation, the leave-one-out experiments would confound task overlap with source contribution.

OPT-IML task overlap estimate. The paper estimates that "approximately 94% (2067/2207)" of OPT-IML's tasks are also in the Flan 2022 collection, using OPT-IML's task definition "which does not deduplicate across collections." This high overlap suggests that the performance differences between Flan-T5 and OPT-IML are driven primarily by methodological choices (templating, mixture balancing, mixed prompt training) rather than access to substantially different training tasks. This strengthens the paper's central claim that design decisions, not just task collection scale, determine instruction tuning quality.

Critical Assessment

Claim 1: Flan-T5 outperforms prior public instruction tuning collections by 3–17%+ across evaluation settings.

This claim is supported with an important qualification about the −1.9% on BBH zero-shot. Table 1's "Flan 2022 - Next Best T5-XL" row reports the margin over the best alternative T5-XL collection for each evaluation setting. The zero-shot margins are: +3.3% on Held-In, +10.2% on CoT, +4.2% on MMLU, and −1.9% on BBH. The few-shot margins are larger: +12% on Held-In, +8.5% on CoT, +17.6% on MMLU, and +8.5% on BBH. The −1.9% on BBH zero-shot—where Flan 2021 (which trains exclusively for zero-shot) achieves slightly higher BBH zero-shot accuracy than Flan 2022—is a genuine exception to the claim of universal improvement. Flan 2022's mixed-prompt training appears to trade a small amount of zero-shot BBH performance for large gains in few-shot BBH (+8.5%), and the overall pattern across 10 evaluation settings (if we count BBH-CoT separately) is overwhelmingly positive, but the "4.2%+ and 8.5% improvements on MMLU and BBH" framing in the abstract obscures that the BBH zero-shot comparison is actually slightly negative. The 3–17%+ range is accurate as an overall span but the lower bound should arguably be quoted as −1.9% to +17.6%.

A deeper issue: the "next best T5-XL" collection varies by setting. For Held-In zero-shot, the next best is P3++ (70.5% vs. Flan 2022's 73.8%). For MMLU zero-shot, it is P3++ again (46.1% vs. 50.3%). For BBH zero-shot, it is Flan 2021 (28.1% vs. 26.2%). This means the +4.2% MMLU improvement and the −1.9% BBH improvement are measured against different baselines. The paper does not report how Flan 2022 compares to a single consistently-defined baseline across all metrics (such as Flan 2021, the original Flan collection), which would give a cleaner picture of the marginal improvement from the methodological changes.

The comparison against OPT-IML-Max 175B is impressive—3B outperforming 175B on MMLU and BBH few-shot—but should be interpreted with caution. OPT-IML uses a different base model family (OPT vs. T5) with different pretraining, different architecture (decoder-only vs. encoder-decoder), and the task collections, while overlapping, are not identical. The paper acknowledges these confounds: "these differences may arise from any combination of pre-training, model architecture, and instruction tuning." The 3B-vs-175B comparison is better understood as evidence that Flan-style instruction tuning is highly effective relative to the known alternatives, not as a controlled experiment isolating instruction tuning quality from all other factors.

Claim 2: Training with mixed zero-shot, few-shot, and chain-of-thought prompts improves performance in all settings.

This claim is supported for the zero-shot/few-shot mixing (Figure 3), with the CoT evidence being more nuanced. Figure 3 cleanly demonstrates that any mixture of zero-shot and few-shot templates outperforms either extreme for both zero-shot and few-shot evaluation, on both Held-In and Held-Out tasks. The U-shaped curves are convincing and the result is consistent across evaluation settings. However, the claim about Chain-of-Thought mixing is different: CoT training primarily helps CoT evaluation (Table 1: BBH-CoT few-shot drops 23.2 points when CoT data is removed), with modest spillover to non-CoT settings. The paper frames this as "training with mixed prompt settings... actually yields stronger (2%+) performance in all settings," but the 2%+ figure primarily reflects the zero-shot/few-shot synergy; the CoT contribution to non-CoT settings is smaller and not the primary source of the mixed-prompt benefit.

A missing experiment: the paper does not test whether mixing CoT templates with non-CoT templates at different ratios produces the same U-shaped synergy as mixing zero-shot and few-shot. The CoT data was mixed into the training at a fixed rate (determined by the mixture balancing experiments) and the ablation simply removes it entirely. A sweep over CoT mixing ratios—analogous to Figure 3's sweep over few-shot ratios—would reveal whether too much CoT data degrades non-CoT performance or whether the relationship is monotonically positive.

Claim 3: Task balancing and enrichment techniques (input inversion, mixture balancing) are "overlooked but critical" to effective instruction tuning.

This claim is strongly supported, with both techniques showing large effects in the subtraction ablations. Input inversion accounts for a 8.6-point MMLU zero-shot drop and a 7.8-point BBH zero-shot drop (Table 1). Mixture balancing accounts for a 11.1-point BBH zero-shot drop and a 15.0-point BBH few-shot drop. These are among the largest single-component effects in the ablation table, validating the "critical" characterization. The "overlooked" characterization is supported by the fact that prior public collections either did not use these techniques or did not isolate their contributions: Flan 2021 did not systematically balance mixture weights; P3 and T0 used input inversion but did not ablate it; Super-Natural Instructions used a single dominant source format.

However, "critical" should be interpreted relative to the baseline Flan 2022 recipe, not as an absolute claim that any instruction tuning recipe must include these techniques. If a practitioner had a very different task collection—say, only high-quality, manually curated tasks with no format diversity issues—mixture balancing might matter less. The paper's mixture balancing experiments are specific to the Flan 2022 collection where some sources (Super-Natural Instructions) are substantially less beneficial than others (Flan 2021, T0-SF). In a more homogeneous collection, equal weighting might be near-optimal.

The input inversion ablation has a subtle confound: the paper removed input inversion from the additional datasets (Dialog, Program Synthesis, CoT) but Flan 2021, P3++, and Super-Natural Instructions already contain some inverted tasks. The "− Input Inversion" row therefore removes only a subset of inverted examples, not all inversion from the entire collection. The measured 8.6-point MMLU drop likely understates the total contribution of input inversion if we consider both the explicitly added inversions and the implicit inversions already present in the base collections. A more complete ablation would require removing the implicit inversions from Flan 2021 and P3++ as well—a substantially more invasive experiment that the paper does not perform.

Claim 4: Flan-T5 serves as a more computationally-efficient starting checkpoint for single-task fine-tuning.

This claim is supported by the convergence curves in Figure 6 and the Pareto improvement in Figure 5, but the efficiency quantification is incomplete. Figure 6 convincingly shows that Flan-T5 converges faster than T5 on five Held-Out tasks, with WANLI showing roughly a 3× reduction in steps to reach a given accuracy. Figure 5 shows that Flan-T5 → FT never underperforms T5 → FT across 12 diverse tasks. These are strong results. However, the paper's green-AI argument requires an additional step: demonstrating that the one-time cost of instruction tuning is less than the aggregate savings across many downstream fine-tuning runs. The paper does not make this calculation. It does not report the total FLOPs or GPU-hours for producing Flan-T5 (instruction tuning on 1,836+ tasks for the full training duration), nor does it estimate how many downstream fine-tuning runs would be needed to amortize that cost. The convergence curves show per-task savings, but the total savings depend on how many downstream tasks the community collectively fine-tunes—a number that is inherently unknown.

A related concern: the convergence advantage is demonstrated on T5-XL (3B) with full fine-tuning (all parameters updated). At larger model scales where full fine-tuning is impractical, practitioners increasingly use parameter-efficient methods (LoRA, prompt tuning, adapters). The paper cites work showing these methods can match full fine-tuning but does not test whether the Flan-T5 convergence advantage transfers. A parameter-efficient fine-tuning comparison (e.g., LoRA fine-tuning of Flan-T5 vs. LoRA fine-tuning of T5) would strengthen the efficiency argument for larger-scale use cases.

Claim 5: The paper's findings establish a set of actionable design principles for building instruction-tuned models.

This claim is supported in spirit but the mixture weights remain unspecified. The paper provides clear, actionable principles: (1) mix prompt formats during training, with 10–90% few-shot being a safe range; (2) use input inversion to enrich task diversity, particularly for Held-Out generalization; (3) balance data sources by contribution quality, not just quantity; (4) include CoT training if CoT evaluation is desired; (5) scale to at least hundreds of tasks, with larger models benefiting from thousands. However, the specific mixture weights used to achieve Flan-T5's reported performance are described only as "informed by experiments in Section 3.5, along with the resulting practitioner intuition." A practitioner attempting to replicate Flan-T5 would need to rediscover these weights through their own experimentation, because the weights themselves are not released in numeric form. This is a meaningful gap between "actionable principles" and "reproducible recipe." The open-sourcing of the code and templates partially addresses this (the code presumably contains the default weights), but the paper text itself does not provide the concrete hyperparameters.

Missing experiments that would strengthen the paper.

Several experiments are conspicuous by their absence and would have increased confidence in the findings:

A sweep over input inversion rates. The paper fixes the inversion rate at 30% and does not test whether 10%, 50%, or 100% would be better. Given that input inversion provides large Held-Out benefits (Table 1), understanding the dose-response relationship is practically important—particularly since inverted examples may be lower quality or more synthetic than forward-direction examples, and too much inversion could introduce noise.
A direct Flan 2022 vs. OPT-IML comparison on equivalent base models. Since OPT-IML's collection, templates, and weights are not publicly released, this comparison is impossible for external researchers, but the authors—who presumably have access to OPT-IML's methods—could have included a controlled ablation isolating the effect of Flan's specific templatization and mixing choices from the effect of task collection overlap. The current comparison is across different base models, different architectures, and different training frameworks, weakening causal attribution.
Held-Out evaluation on tasks from different domains than the training collection. Both MMLU and BBH are heavily weighted toward reasoning and knowledge tasks, which overlap with the question-answering and natural language inference tasks in the training collection. Testing on genuinely out-of-domain tasks—e.g., code generation, summarization, translation—would test whether the Held-Out improvements reflect genuine cross-task generalization or cross-domain generalization within related task families.
Variance estimates across random seeds for the main instruction tuning runs. The paper reports single-run results for the instruction tuning ablations (Table 1, Table 2). With a 500-example MMLU test set, a 4.9-point drop from removing mixture balancing (50.3% to 45.4%) is almost certainly statistically significant, but smaller effects (e.g., the 1.3-point Held-In drop from removing few-shot templates) might fall within run-to-run variance. Reporting standard deviations or confidence intervals would clarify which effects are reliably distinguishable from noise.
A pure "no human feedback" vs. "with human feedback" comparison on the same base model. The paper argues that instruction tuning without human feedback can partially close the gap with RLHF-tuned models, but it never directly compares Flan-T5 against an RLHF-tuned variant of the same T5-XL base. The comparison is left implicit (Flan-T5 vs. external RLHF models like InstructGPT, which use different base models). A within-model-family comparison would isolate the contribution of instruction tuning vs. RLHF more cleanly.

Genuine weaknesses that should be acknowledged.

The test set is small relative to the number of reported metrics. The 500 MMLU questions evaluate 57 subjects, meaning some subjects have fewer than 10 questions. A swing of 1–2 questions per subject can produce percentage-point shifts that look meaningful in aggregate. The same concern applies to BBH (23 tasks) and the 8 Held-In tasks. The paper does not report per-task variance or subject-level breakdowns.
The paper uses the same T5-XL base model for all comparisons, which eliminates model family as a confound but limits generalizability. Findings about optimal few-shot ratios, input inversion benefits, and mixture balancing effects might differ on decoder-only architectures (GPT-style), on models pretrained with different corpora (The Pile vs. C4), or on models with different pretraining objectives. The paper's claim that T5-XL is "representative of the capabilities of many contemporary LLMs" is an assertion, not a demonstrated fact. As of 2023, the dominant model architectures for very large LMs are decoder-only (GPT-3, PaLM, LLaMA, Chinchilla), and whether encoder-decoder models respond to instruction tuning in the same way is an open question.
Difficulty estimation is not addressed, in contrast to the example summary paper's focus on prompt difficulty. The Flan Collection applies a single training recipe to all tasks regardless of difficulty. If some tasks are harder for the base model than others (analogous to the MATH difficulty bins in the example paper), a difficulty-conditioned allocation of training resources (more epochs on harder tasks, different mixing ratios for different task types) might further improve performance. The paper does not explore this dimension, treating all tasks as equally deserving of representation in the mixture (modulo source-level balancing). This is a missed opportunity given the example paper's demonstration that difficulty-conditioned allocation is highly effective for inference-time compute.
The 30% input inversion rate is arbitrary and unswept. While the paper explicitly flags this as future work, it means that one of the largest single-component effects (8.6 points on MMLU) is achieved with a hyperparameter chosen "for simplicity" rather than through optimization. The true marginal benefit of input inversion—if optimally tuned—could be substantially larger or smaller than the reported 8.6 points.

Overall, the experimental analysis is thorough for the claims it makes, with the subtraction-ablation framework being the paper's strongest methodological contribution. The results consistently demonstrate that the Flan 2022 design decisions collectively produce large improvements over prior collections, and the individual ablations isolate these contributions with reasonable precision. The primary limitations are the reliance on a single model family (T5 encoder-decoder), the lack of variance estimation for the main results, the arbitrary choice of certain hyperparameters (inversion rate, exact mixture weights), and the absence of difficulty-conditioned or domain-stratified evaluation beyond the Held-In/Held-Out distinction. These limitations are largely acknowledged or implied by the paper's framing, which positions the work as an initial systematic analysis rather than a final optimized recipe.

6. Limitations and Trade-offs

The Cost of Difficulty or Quality Assessment Is Not Accounted For—and the Paper Studies a Uniform Recipe Where It Might Not Apply

The paper applies a single instruction tuning recipe—the same mixture weights, the same input inversion rate (30%), the same prompt mixing ratios—to all tasks regardless of their individual characteristics. Unlike the example paper's explicit difficulty estimation and difficulty-conditioned allocation, this work makes no attempt to stratify tasks by how hard they are for the base model, how much they benefit from specific augmentation techniques, or how close they are to the model's existing capabilities. Every task receives equal treatment within its source category.

The paper's mixture balancing experiments (Section 3.5) operate at the source level (Flan 2021 vs. Super-Natural Instructions vs. Program Synthesis), not at the task level, meaning that all 1,600+ tasks from Super-Natural Instructions receive the same weight regardless of their individual quality or transfer value. This is a coarse allocation that almost certainly misses within-source heterogeneity—some Super-Natural Instructions tasks are presumably more beneficial for Held-Out generalization than others, but the paper's framework provides no mechanism for identifying or upweighting them. The consequence is that the reported Flan-T5 performance likely represents a lower bound on what a task-level difficulty- or quality-aware allocation could achieve. A practitioner who could cheaply identify high-transfer tasks and allocate more training emphasis to them might substantially outperform the uniform-within-source approach.

No experiment in the paper quantifies this lost opportunity. The Held-In/Held-Out evaluation taxonomy captures the distinction between seen and unseen tasks, but there is no ablation that stratifies training tasks by an external difficulty or quality metric and measures whether the benefits of mixture balancing or input inversion are concentrated in specific task subpopulations. The paper does not measure, for instance, whether input inversion helps more on tasks where the base model's zero-shot accuracy is low (analogous to the example paper's finding that beam search helps on medium-difficulty problems but not on easy ones). This is a missed opportunity for finer-grained resource allocation.

The paper partially acknowledges the coarseness of its approach in the mixture balancing section: the final weights were derived from "practitioner's intuition" informed by leave-one-out experiments at the source level. But the fundamental assumption—that uniform treatment within a source is near-optimal—is never tested, and the paper leaves "ample room for more sophisticated future work." A practitioner deploying instruction tuning at scale, with a finite compute budget, would want to know whether investing additional training FLOPs in high-quality tasks is more effective than simply adding more tasks from any source.

Single Model Family and Single Evaluation Paradigm Leave Architecture Transfer Unexplored

All experiments in the paper use T5-LM, an encoder-decoder architecture pretrained on the C4 corpus (Raffel et al., 2020). The paper justifies this choice in Section 3.1: "We felt XL was appropriately sized to run large-scale systematic ablations, while being sufficiently large to draw general conclusions." The claim that the findings generalize is an assertion, not a demonstrated fact. As of the paper's writing in early 2023, the dominant architecture for large language models was decoder-only (GPT-3, PaLM, Chinchilla, LLaMA, OPT), and Wang et al. (2022a)—cited by the paper itself—showed that "model architecture and pretraining objective work best for zero-shot generalization" matter substantially for downstream task performance.

The consequence is that core findings—particularly the magnitude of improvement from mixing prompt formats, the optimal few-shot mixing ratio, and the benefit of input inversion—might not transfer to decoder-only models. Encoder-decoder models process input and output through different pathways (bidirectional encoder for input, autoregressive decoder for output), while decoder-only models process everything autoregressively. This architectural difference could affect how the model represents instructions vs. exemplars vs. target outputs, and thus how mixed-prompt training influences the learned representations. A practitioner using GPT-style models cannot assume that adding 10% few-shot templates will produce the same 2%+ zero-shot improvement observed on T5.

The evaluation benchmarks are also concentrated in specific task families: question answering, natural language inference, mathematical reasoning, and multiple-choice knowledge tests (MMLU). The paper does not evaluate on code generation, summarization, translation, or open-ended generation—domains that differ substantially in output structure and task format from the training collection. The Held-Out claim (generalization to unseen tasks) is tested only on tasks that are structurally similar to the training distribution (reasoning and knowledge QA). A model that excels on MMLU and BBH might not generalize to generating coherent multi-paragraph summaries from instructions it has never seen.

The paper provides no evidence to quantify this risk. No decoder-only model is fine-tuned on the Flan 2022 collection; no cross-architecture comparison is performed; no evaluation beyond the Held-In/CoT/Held-Out taxonomy is conducted. The paper acknowledges the architectural confound when discussing the OPT-IML comparison—"these differences may arise from any combination of pre-training, model architecture, and instruction tuning"—but does not extend this caution to its own claim of generalizability. The mitigation is implicit and deferred: the open-source release of the collection enables future work to test on other architectures, but the paper itself provides no guidance on which findings are architecture-dependent and which are likely universal.

Mixture Weights Are Empirically Informed but Not Algorithmically Optimized—and Not Precisely Specified

A practitioner attempting to replicate Flan-T5 or build a similarly strong instruction-tuned model from their own task collection needs to know the mixture weights—the proportion of training data drawn from each source (Flan 2021, T0-SF, Super-Natural Instructions, CoT, Dialog, Program Synthesis). The paper does not provide these weights in numeric form. Section 3.5 describes the process: leave-one-out experiments ranked sources by importance; these results "significantly narrowed the mixture weights search space"; then "practitioner's intuition" selected the final weights. The "All (Weighted)" row in Table 2 achieves 48.1% MMLU (vs. 47.3% for equal weighting), demonstrating that weighting helps, but the specific weights that produced this result are not reported. The paper's abstract states that the collection, templates, and methods are made publicly available, and the code release presumably contains the default weights, but the paper text itself—the archival record of the research—leaves a critical hyperparameter unspecified.

The consequence is that the paper's central claim—that Flan 2022's design decisions produce a 3–17%+ improvement over prior collections—is reproducible in principle (the code exists) but not fully documented in the paper. A reader cannot determine from the text alone whether the weighting scheme is exotic (e.g., one source gets 50%, another gets 5%) or relatively flat (e.g., everything within a factor of two). The leave-one-out experiments in Table 2 tell us that Flan 2021 and T0-SF are most important, and that Super-Natural Instructions, Dialog, and Program Synthesis are less so, but the continuous weights are absent. This matters because mixture weighting is one of the largest single-component effects in the ablation table (Table 1: removing mixture balancing drops BBH zero-shot by 11.1 points). A practitioner who gets the weights slightly wrong might substantially underperform the reported Flan-T5 results, and without the paper specifying the weights, they cannot know whether their underperformance is due to a flawed replication or a suboptimal weighting choice.

The paper does not attempt to mitigate this limitation beyond noting that the search space was narrowed and that "ample room for more sophisticated future work" exists. No sensitivity analysis is performed—we do not know whether small perturbations to the weights produce large performance changes (in which case exact replication matters) or whether any weighting in a broad basin around the chosen point works about equally well (in which case the exact values are less critical). The table reports only the "All (Equal)" and "All (Weighted)" endpoints, not intermediate weight configurations. This is a methodological gap that reduces the paper's utility as a reproducible recipe while preserving its value as a set of design principles.

The Convergence Efficiency Argument Lacks a Full Cost-Benefit Analysis—It Identifies a Mechanism but Does Not Close the Economic Case

Section 4 demonstrates that Flan-T5 converges faster and to higher accuracy than T5 on single-task fine-tuning, and the paper frames this as having "green-AI incentives" for the community to adopt instruction-tuned checkpoints as standard starting points. The argument is: instruction tuning is a one-time cost; it reduces fine-tuning steps by roughly 2–3× per downstream task; aggregated across many downstream applications, the compute savings exceed the one-time investment.

The paper provides strong evidence for the mechanism (Figures 5 and 6: Flan-T5 converges faster on every tested task) but provides zero evidence for the economic closure of the argument. It does not report: (1) the total compute cost of producing Flan-T5 (instruction tuning on 1,836+ tasks for the full training duration, in FLOPs or GPU-hours); (2) the number of downstream fine-tuning runs at which the cumulative savings exceed that upfront cost; (3) whether the convergence advantage is proportional to the fine-tuning budget or merely shifts the curve leftward by a constant factor. Without item (1), a practitioner cannot assess whether the one-time cost is an investment they can afford; without item (2), they cannot determine whether their expected downstream usage volume justifies the investment; without item (3), they cannot project whether the advantage persists or diminishes at larger fine-tuning budgets (the curves in Figure 6 run to ~200 steps on 3B models—whether the advantage would hold for a 100,000-step fine-tuning run on an XXL model is unknown).

The consequence is that the green-AI argument is suggestive but not actionable. A practitioner convinced by the convergence curves still cannot perform the cost-benefit calculation needed to decide whether to instruction-tune their own model or download a pre-existing Flan-T5 checkpoint. For large organizations that might instruction-tune from scratch, the missing cost data prevents informed decision-making. For smaller practitioners who would download a pre-existing checkpoint, the argument is moot—they incur no instruction tuning cost—but the paper's framing conflates the case for adopting instruction-tuned checkpoints (which the convergence curves support) with the case for producing them (which requires cost data the paper does not provide).

The paper does not acknowledge this gap in its cost-benefit reasoning. The green-AI argument is presented as a conclusion—"there are strong green-AI incentives for the NLP community to adopt instruction-tuned models"—without the quantitative backing that would make it a rigorous claim. A more measured framing would treat the convergence speed result as establishing a mechanism by which instruction tuning could yield aggregate compute savings, conditional on unmeasured parameters (the ratio of one-time training cost to downstream usage volume).

The Test Set Is Modest in Size and the Paper Reports No Variance Estimates—Making Some of the Smaller Claimed Improvements Statistically Uncertain

The primary evaluation benchmarks have limited sizes: MMLU uses 500 questions across 57 subjects, meaning many subjects have fewer than 10 questions; BBH uses 23 tasks; the 8 Held-In tasks are individual validation sets (some with a few hundred examples). The paper reports all main results as single-point estimates without confidence intervals, standard deviations, or statistical significance tests. The subtraction ablations in Table 1 report differences as small as 0.0 points (Held-In zero-shot with vs. without input inversion: 73.8% vs. 73.8%) alongside differences as large as 23.2 points (BBH-CoT few-shot with vs. without CoT training: 35.2% vs. 12.0%). The reader has no way to distinguish which of the smaller differences are reliable and which might reverse under a different random seed or a different train-test split.

The consequence is that some of the paper's specific quantitative claims—particularly those involving small margins—should be treated as directional rather than precise. The claim that input inversion provides zero benefit for Held-In tasks (0.0-point difference) might reflect a genuine null effect, or it might reflect a small positive effect masked by test-set noise. The claim that removing few-shot templates drops Held-In zero-shot by 1.3 points (73.8% to 72.5%) might be a real but small effect, or it might be within run-to-run variance. For the single-task fine-tuning experiments (Section 4, Figure 5), tasks with fewer than 1,000 training examples use averaging across 3 random seeds—this is the only variance mitigation reported anywhere in the paper, and it applies only to a subset of the single-task experiments, not to the core instruction tuning ablations.

The paper's experimental protocol contributes to this limitation: models are trained once per configuration without reported replication. The scaling experiments (Section 3.3, Figure 4) train each (model size × task count) combination once; the mixture balancing experiments (Table 2) train each ablated mixture once. This is understandable for a study with dozens of configurations and 3B-parameter models—full replication would multiply an already substantial compute budget—but it means the paper's conclusions rest on point estimates whose stability is unknown.

The paper does not acknowledge this as a limitation. The standard practice in the instruction tuning literature at the time—evident in the cited works (Wei et al., 2021; Sanh et al., 2021; Chung et al., 2022)—was to report single-run results for large-scale fine-tuning experiments, and the paper follows that convention. However, the detailed ablation framework the paper introduces (Table 1) invites finer-grained interpretation of small differences, and the absence of variance estimates undermines that interpretation. A reader evaluating whether to adopt a specific technique (e.g., "should I spend effort on input inversion for my Held-In tasks?") needs to know whether the 0.0-point difference is zero within measurement error—and the paper does not provide the information needed to make that assessment.

The 30% Input Inversion Rate Is Chosen Arbitrarily, and One of the Largest Single-Component Effects Rests on an Untuned Hyperparameter

Input inversion accounts for an 8.6-point MMLU zero-shot improvement and a 7.8-point BBH zero-shot improvement in the subtraction ablations (Table 1)—making it one of the largest individual contributors to Held-Out generalization. However, the mixing rate of inverted examples (30%—meaning 3 inverted examples are generated for every 10 regular examples) is chosen, in the paper's own words, "for simplicity, approximately mirroring prior work," and the paper explicitly states it leaves "the large space of exploration for future work" (Appendix B). No sweep over inversion rates is performed—there is no experiment testing 10%, 50%, or 100% inversion, and no dose-response curve characterizing how Held-Out performance changes with the inversion ratio.

The consequence is that the reported 8.6-point benefit is achieved with an arbitrarily chosen hyperparameter, not an optimized one. The true benefit of input inversion—if the rate were tuned—could be substantially larger (if 30% is suboptimal) or substantially smaller (if 30% happens to be near-optimal, but the benefit is robust to the exact rate). A practitioner adopting the Flan 2022 recipe gets the benefit of this technique but cannot be confident they are getting the maximum benefit achievable with the same inversion mechanism. Moreover, the fixed 30% rate applies uniformly to Dialog, Program Synthesis, and Chain-of-Thought tasks—three very different task types that might benefit from different inversion rates. CoT tasks, with their complex multi-permutation inversions (Figure 7 in Appendix B), might require a different mixing ratio than Dialog tasks with simpler history-reconstruction inversions.

The paper also notes that Flan 2021, P3++, and Super-Natural Instructions already contain some implicit inverted tasks, meaning the "− Input Inversion" ablation removes only the explicitly added inversions from Dialog, Program Synthesis, and CoT tasks—not all inversion from the entire collection. The 8.6-point drop therefore measures the marginal contribution of supplementary inversion on top of a baseline that already includes some inversions. If the baseline inversions were removed as well (a more invasive experiment), the total contribution of input inversion might be larger, and the optimal supplementary rate might differ. The paper does not disentangle these effects.

The mitigation is limited to an explicit acknowledgment that the 30% rate is arbitrary and that future work should explore the space. This is honest but leaves a significant methodological gap: one of the paper's headline findings—that input inversion is "critical" to effective instruction tuning—is demonstrated with a specific rate, and the reader cannot determine from the paper whether the technique is robust to rate variation (in which case the exact choice matters little) or whether performance is sensitive to the rate (in which case replication requires matching the paper's specific 30% value).

The Paper Does Not Investigate Combining Its Own Techniques—Each Component Is Studied in Isolation, Leaving Interaction Effects Unexplored

The subtraction ablation framework (Table 1) measures each component's marginal contribution by removing it from the full recipe. This design estimates the contribution of each component given that all other components are present. It does not estimate two-way or higher-order interactions: does the benefit of input inversion depend on whether mixture balancing is also used? Does mixed-prompt training help more when input inversion is present? Could certain components be partially redundant, such that the sum of individual contributions overstates the benefit of combining them? The paper's experimental design cannot answer these questions because it never removes two components simultaneously or measures performance on all 2^4 = 16 possible subsets of the four main techniques.

The consequence is that a practitioner with a limited implementation budget—who can only adopt a subset of the paper's recommendations—cannot determine from the ablation results which subset to prioritize. The paper provides independent effect sizes but not conditional effect sizes. If a team can only implement two of the four techniques, should they choose input inversion + mixture balancing (the two largest individual effects on MMLU) or mixed-prompt training + CoT training (which target different evaluation settings)? The paper's evidence does not distinguish between these strategies because the ablation framework was not designed to capture interactions.

A specific interaction the paper should have tested: the relationship between task scaling (Section 3.3) and mixture balancing (Section 3.5). The scaling curves in Figure 4 show that Held-Out performance improves with task count, but these curves use the unweighted mixture. Would the scaling curves look different with optimal mixture weights? Does mixture balancing become more or less important as the number of tasks grows? The paper does not test these combinations, leaving the practitioner uncertain about whether to invest in more tasks, better weighting, or both.

The paper does not acknowledge this limitation. The subtraction-ablation design is presented as sufficient to characterize the contributions of individual components, and the central claims ("input inversion is critical," "mixture balancing is overlooked but critical") are stated in absolute terms that imply independence. A more cautious framing would note that the reported effect sizes are marginal effects in the presence of all other components, and that the benefits of combining a subset of techniques cannot be read directly from Table 1.

The interaction between search/verifier methods and revision methods—analogous to the example paper's explicit acknowledgment that it did not combine PRM search with revisions—is not addressed. Techniques like input inversion and mixed-prompt training both increase input diversity, potentially through overlapping mechanisms. If the mechanisms overlap, the combined benefit might be subadditive, meaning a practitioner implementing both gets less than the sum of individual effects. The paper provides no evidence to bound this subadditivity.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes instruction tuning as a decomposable, evidence-backed engineering discipline rather than a collection of ad-hoc recipes. Before this work, the instruction tuning literature offered practitioners a set of competing collections—each a monolithic artifact with its own task composition, template design, and training protocol—but no principled way to understand why one collection outperformed another or which design decisions were responsible. The paper's systematic subtraction-ablation framework changes this: it demonstrates that the performance of the Flan 2022 collection can be attributed to four independently measurable components (mixed prompt training, input inversion, task scaling, and mixture balancing), each with quantifiable marginal contributions to Held-In, Held-Out, and Chain-of-Thought evaluation settings. This reframes instruction tuning from a "try different collections and see what works" activity into a structured optimization problem where design decisions can be evaluated, compared, and improved incrementally.

The magnitude is best described as a methodological consolidation rather than a paradigm shift. The paper does not introduce a fundamentally new training objective, a new architecture, or a new source of supervision. What it does—and this is genuinely valuable—is demonstrate that the difference between a middling instruction-tuned model (e.g., T5-XL fine-tuned on Super-Natural Instructions, achieving 35.6% on MMLU zero-shot) and a strong one (Flan-T5-XL at 50.3% on MMLU zero-shot) is not primarily about access to more tasks or larger models; it is about a set of relatively simple, overlooked design choices that collectively account for a 14.7-point gap. The paper shows that task mixture balancing alone accounts for an 11.1-point BBH zero-shot improvement (Table 1, "− Mixture Balancing"), and input inversion accounts for an 8.6-point MMLU zero-shot improvement—effects that, prior to this work, were either assumed to be negligible or were never measured in isolation. The paper makes the case that the field had been leaving large performance gains on the table by neglecting these "unsexy" engineering decisions in favor of scaling task count or model size.

The work also resolves a latent tension in the instruction tuning literature between task quantity and task quality. Prior work had established that adding more tasks improved generalization (Wei et al., 2021; Sanh et al., 2021; Wang et al., 2022c), but the scaling curves in those works conflated adding tasks with adding tasks from particular sources. The paper's mixture balancing experiments (Section 3.5, Table 2) disentangle these axes for the first time: Super-Natural Instructions contributes many tasks but has limited marginal benefit on MMLU (dropping from 47.3% to 46.8% when removed), while Flan 2021 contributes fewer tasks but has the largest marginal benefit (dropping to 45.7% when removed). This finding converts a vague intuition—"not all tasks are equally useful"—into a specific, actionable ranking of source contributions, supported by the concurrent and independent convergence of OPT-IML (Iyer et al., 2022) on the same ranking. The implication is that future instruction tuning efforts should prioritize curation of high-quality, format-diverse task sources over indiscriminate aggregation of any available NLP dataset.

The most counterintuitive finding—and therefore the one with the greatest potential to redirect research—is that mixing prompt formats during training improves performance in all formats simultaneously (Figure 3). This violates the natural assumption of format specialization: that a model trained exclusively on zero-shot prompts would perform best on zero-shot evaluation, and that adding few-shot training data would create a tradeoff. The paper shows the opposite: adding just 5% few-shot templates improves zero-shot Held-In performance, and adding 10% zero-shot templates improves few-shot performance. The implication is that training-time exposure to multiple prompt formats teaches the model a more abstract, format-agnostic representation of the underlying tasks—a form of meta-learning that transfers to both zero-shot and few-shot inference. This finding should cause practitioners to stop training separate models for different prompt formats and instead build unified models that see mixed formats during training. It also suggests that the field's prior focus on prompt engineering (crafting the perfect instruction wording) should be complemented by prompt diversity—exposing the model to many structurally different ways of presenting the same task, including formats the model will never see at evaluation time.

The paper also reframes instruction tuning as an economic investment rather than a capability cost. Section 4 demonstrates that Flan-T5 converges faster and to higher accuracy than T5 on single-task fine-tuning across 12 diverse downstream tasks. The convergence curves in Figure 6 show roughly a 2–3× reduction in fine-tuning steps to reach equivalent accuracy on tasks like WANLI, and the Pareto improvement in Figure 5 means a practitioner adopting Flan-T5 as a starting checkpoint never sacrifices performance relative to starting from T5. This finding encourages the NLP community to treat instruction-tuned checkpoints as standard starting points—analogous to how pretrained encoders (BERT, RoBERTa) became standard starting points for classification tasks in 2019—with the economic rationale that the one-time cost of instruction tuning is amortized across many downstream applications. While the paper does not close the full cost-benefit calculation (it does not report the FLOPs for instruction tuning or the break-even downstream usage volume), it establishes the mechanism by which instruction tuning can be a net compute-saver, shifting the burden of proof onto those who would fine-tune from scratch.

The paper makes certain research directions more attractive: controlled ablation studies of instruction tuning design decisions (now that the methodology has been established), task-level quality estimation and dynamic allocation (analogous to the difficulty-conditioned allocation in the example paper but applied to training rather than inference), and the combination of instruction tuning with parameter-efficient fine-tuning methods (to test whether the convergence advantage holds when only a small subset of parameters is updated). Conversely, the paper makes less attractive the direction of simply aggregating more tasks without attention to source quality or format diversity, since the mixture balancing experiments show that additional tasks from low-benefit sources like Super-Natural Instructions provide diminishing returns.

Follow-Up Research This Work Enables

Task-level quality estimation and difficulty-conditioned training allocation. The paper operates at source-level granularity for mixture balancing (Flan 2021 vs. Super-Natural Instructions vs. Program Synthesis), but within each source, individual tasks vary enormously in quality, format coherence, and transfer value. A natural extension is to estimate per-task "instruction tuning value"—perhaps using a held-out validation set of diverse tasks to measure how much adding a given training task improves generalization—and then to allocate training emphasis (sampling probability, number of epochs, or data augmentation budget) proportionally. The example paper demonstrated that difficulty-conditioned allocation of inference compute yields 4× efficiency gains; an analogous difficulty-conditioned allocation of training compute across tasks could similarly improve the efficiency of instruction tuning. A concrete experiment: use the PRM-style scoring approach from the example paper to estimate each training task's difficulty for the base model (pass@1 rate on a small sample), then stratify the training mixture so that medium-difficulty tasks (where the model is close to mastering the task but needs refinement) receive more weight than very easy tasks (already mastered) or very hard tasks (outside the model's capability range). The paper's Held-In degradation at high task counts (Figure 4, left) suggests that the model wastes capacity on tasks it already knows well; targeted allocation could mitigate this.

Architecture transfer: do these findings hold for decoder-only models? All experiments in the paper use T5-LM, an encoder-decoder architecture. A critical stress-test is to replicate the core findings—the U-shaped benefit of mixed prompt training (Figure 3), the Held-Out benefit of input inversion (Table 1), and the source-level mixture balancing (Table 2)—on a decoder-only model family of comparable scale, such as OPT-3B or LLaMA-3B. The mechanism by which mixed-prompt training helps may operate differently in decoder-only architectures, where the model processes instructions, exemplars, and target outputs through a single autoregressive pathway rather than separate encoder and decoder pathways. If the findings transfer cleanly, this strengthens the paper's generalizability claim; if they do not—for instance, if decoder-only models show weaker benefits from input inversion or a different optimal few-shot mixing ratio—this establishes important boundary conditions and motivates architecture-specific instruction tuning recipes. A well-designed study would replicate Table 1's subtraction ablations on both architectures, reporting effect sizes and interaction terms.

The interaction between instruction tuning and parameter-efficient fine-tuning (PEFT). The paper demonstrates that Flan-T5 converges faster than T5 under full fine-tuning (Section 4), but modern practice for large models increasingly uses PEFT methods like LoRA (Hu et al., 2021) or prompt tuning (Lester et al., 2021), where only a small fraction of parameters are updated. It is not obvious that the convergence advantage transfers: the instruction-tuned model's representations may be more general, but PEFT methods have limited capacity to reshape those representations for a new task. A concrete experiment would compare LoRA fine-tuning of Flan-T5 vs. LoRA fine-tuning of T5 on the same 12 downstream tasks used in Figure 5, measuring both convergence speed (steps to reach a given validation accuracy, as in Figure 6) and peak performance. If Flan-T5 retains its advantage under PEFT, this strengthens the economic case for adopting instruction-tuned checkpoints as starting points for all downstream adaptation, not just full fine-tuning. If the advantage diminishes or disappears, this suggests that instruction tuning's benefits are mediated by full-parameter access during adaptation, and the community should invest in PEFT-specific instruction tuning recipes.

Optimal inversion rates and task-type-specific augmentation. The paper fixes the input inversion rate at 30% for Dialog, Program Synthesis, and CoT tasks—a choice made "for simplicity" with no sweep over rates. A natural follow-up would systematically sweep inversion rates (e.g., 0%, 10%, 30%, 50%, 100%) for each task type and measure the dose-response curve on Held-Out generalization. The paper's CoT inversion (Figure 7, Appendix B) uses multiple permutation types (e.g., Question + Answer → Reasoning, Reasoning + Answer → Question), and different permutations may contribute differently to downstream performance. A deeper study could ablate individual permutation types to identify which structural inversions are most valuable, potentially enabling more targeted augmentation with less computational overhead. The 8.6-point MMLU benefit from input inversion (Table 1) is one of the largest single-component effects in the paper, and optimizing the inversion strategy could yield substantially larger gains.

The interaction between instruction tuning and human feedback (RLHF). The paper deliberately restricts its scope to instruction tuning without human feedback, arguing that RLHF is "likely additive to a foundation of instruction tuning methods" (Section 2). This is an untested claim. A critical experiment would compare the downstream performance of Flan-T5 vs. an RLHF-fine-tuned variant of Flan-T5 on both traditional NLP benchmarks (MMLU, BBH, Held-In) and open-ended generation tasks (dialogue coherence, harmlessness, helpfulness). The paper cites Ouyang et al. (2022)'s "alignment tax"—the observation that RLHF can degrade performance on traditional NLP benchmarks—and the hypothesis that instruction tuning without human feedback partially addresses this by improving broad NLP capability before RLHF is applied. Testing this hypothesis requires measuring whether Flan-T5 + RLHF achieves better NLP benchmark performance than RLHF applied directly to T5 (skipping instruction tuning), which would demonstrate that instruction tuning provides a protective foundation against the alignment tax. This experiment is feasible with existing open-source RLHF frameworks and would directly inform the community's decision about where to invest alignment effort.

Synthetic data generation guided by the paper's quality signals. The paper's mixture balancing experiments identify that Super-Natural Instructions tasks have limited marginal benefit on Held-Out generalization, likely due to their unique input format and instruction design. Rather than simply downweighting these tasks, a constructive follow-up would use the paper's performance signals (which sources and task types transfer best) to guide synthetic data generation: prompt a strong LLM to generate new tasks in the style of the high-benefit sources (Flan 2021, T0-SF), with explicit attention to format diversity and structural variety. This leverages the paper's central insight—that source quality and format diversity matter more than raw task count—to generate training data that maximizes per-example transfer value. A concrete evaluation would compare Flan-T5 against a model trained on the original Flan 2022 collection augmented with synthetically generated tasks, measuring whether synthetic tasks in high-benefit formats can push Held-Out generalization beyond the ceiling observed in Figure 4's task scaling curves.

Practical Applications and Downstream Use Cases

Standardized starting checkpoints for the NLP fine-tuning ecosystem. The paper's most directly actionable finding for practitioners is that Flan-T5 provides a Pareto improvement over T5 as a starting checkpoint for single-task fine-tuning (Figure 5): across 12 diverse downstream tasks, fine-tuning from Flan-T5 never underperforms fine-tuning from T5, and on some tasks (ANLI, SQuAD V2) the improvement exceeds 15 percentage points. Figure 6 quantifies the efficiency gain: Flan-T5 converges roughly 2–3× faster than T5 on Held-Out tasks like WANLI and MedNLI. For an applied ML team that fine-tunes models on 10–20 custom tasks per year, adopting a pre-existing Flan-T5 checkpoint (or an equivalent instruction-tuned model for their architecture family) reduces per-task training time by a factor of 2–3× with no loss in accuracy. Since the instruction tuning cost is borne once by the model provider (or downloaded from a public release), this is a pure efficiency win for downstream consumers. The adoption barrier is low: the checkpoint is publicly available, the model uses the same T5 architecture already familiar to practitioners, and the paper demonstrates that even Held-Out tasks (which Flan-T5 has never seen) benefit from the faster convergence.

Cost-efficient multi-task deployment without per-task specialized models. The paper shows that Flan-T5 without any further fine-tuning (the beige bars in Figure 5) often matches or exceeds T5 with task-specific fine-tuning (the blue bars), particularly on tasks with limited training data. For organizations that need to support dozens or hundreds of NLP capabilities (e.g., a customer support platform that must handle question answering, sentiment analysis, entity extraction, and summarization across many domains), deploying a single Flan-T5 checkpoint—rather than maintaining a separate fine-tuned model per task—reduces operational complexity (one model to serve, monitor, and update) while matching or exceeding the accuracy of the per-task baseline. The Held-Out results on MMLU (50.3% zero-shot for Flan-T5-XL vs. 41.4% for Flan 2021, Table 1) demonstrate that this single checkpoint generalizes to entirely new task families without any additional training, making it suitable for environments where new task types emerge frequently and rapid deployment is valued over per-task optimization.

Data augmentation strategy for low-resource task domains. The paper's finding that input inversion contributes 8.6 points to MMLU zero-shot performance (Table 1, "− Input Inversion" row) and 7.8 points to BBH zero-shot provides a concrete, low-cost data augmentation recipe for practitioners working in low-resource domains. If a team has only a few hundred labeled examples for a novel task, they can generate inverted examples (swap inputs and outputs, create complementary task formulations as illustrated in Figure 7 for CoT tasks) and mix these into the training data at roughly 30% to improve generalization. This technique requires no additional data collection, no model-generated synthetic data (which can introduce hallucinations), and no human annotation—it is a purely structural transformation of existing labels. The paper's evidence that the benefit is concentrated in Held-Out generalization (not Held-In maintenance) means the technique is specifically suited for scenarios where the deployment distribution includes task variants not seen during training—exactly the low-resource transfer setting where data augmentation is most valuable.

Instruction tuning as a pre-deployment step before RLHF alignment. The paper's explicit positioning—that human feedback methods are "likely additive to a foundation of instruction tuning methods" (Section 2)—provides a practical deployment strategy for organizations building aligned chatbots or assistants. The sequence would be: (1) instruction-tune on the public Flan 2022 collection (or an equivalent diverse task mixture), producing a model with strong broad NLP capability and instruction-following skills; (2) apply RLHF on top of this checkpoint for the specific alignment objectives (helpfulness, harmlessness, dialogue quality). The paper's results suggest that step (1) mitigates the "alignment tax" documented by Ouyang et al. (2022), because the instruction-tuned model already achieves strong performance on traditional NLP benchmarks (Table 1: 50.3% MMLU, 26.2% BBH zero-shot), and RLHF in step (2) can focus on the open-ended generation quality without needing to preserve capabilities that were never present in the base model. The paper does not test this two-stage pipeline directly, but the logic is supported by the Held-Out generalization results and the paper's framing of instruction tuning as a foundation. For organizations with access to human feedback data but limited RLHF budget, this staged approach may be more resource-efficient than applying RLHF from scratch to a non-instruction-tuned base model.