On Transferability of Prompt Tuning for Natural Language Processing

ArXiv: 2111.06719

🎯 Pitch

Trained soft prompts can be reused across similar NLP tasks in a zero-shot manner, and using them as initialization accelerates prompt tuning by up to 50× while boosting performance. The key to this transferability is not embedding similarity but how prompts activate neurons in the frozen model—a finding that could refocus prompt tuning research on internal model dynamics.

---

1. Executive Summary

This paper empirically studies the transferability of soft prompts across different downstream NLP tasks and PLMs, analyzing whether prompt tuning (PT) — a parameter-efficient method that trains only learnable virtual tokens prepended to the input while freezing the PLM — can be improved through knowledge transfer. The investigation spans 17 tasks of 6 types on RoBERTa and T5 model families, examining two transfer mechanisms: cross-task transfer (directly reusing trained soft prompts from a source task on a target task, or using them as initialization) and cross-model transfer (projecting prompts from one PLM's embedding space to another's via a learned two-layer perceptron projector). The core findings establish that trained soft prompts can effectively transfer to similar tasks in zero-shot settings and that using transferred prompts as initialization — dubbed transferable prompt tuning (TPT) — can accelerate training by up to ~50× in convergence speedup on certain tasks (e.g., SST-2 on T5-XXL achieving 49.7× speedup) while also improving final performance. A novel overlapping rate of activated neurons (ON) metric correlates with transferability better than embedding-distance-based metrics (Spearman's rank correlation of 49.7% vs. 44.7% for the best embedding metric on RoBERTa-LARGE), establishing that how prompts stimulate PLMs' internal neurons is essential to transfer success — though this correlation degrades on larger models due to redundancy, dropping from 52.5% on T5-Small to 36.9% on T5-XXL, with an intersection-based variant (ONI) partially recovering the signal to 46.3%.

2. Context and Motivation

The Core Tension: Prompt Tuning Is Parameter-Efficient but Time-Inefficient

Prompt tuning (PT) occupies an attractive position in the landscape of methods for adapting extremely large pre-trained language models (PLMs) to downstream tasks: it is parameter-efficient — typically training only thousands of learnable parameters (the soft prompts) while freezing the PLM's billions of parameters — yet it can match the performance of full-parameter fine-tuning when the underlying PLM is sufficiently large. Lester et al. (2021) established this parity point, showing that PT's effectiveness scales with model size, making it a promising approach for utilizing massive PLMs without the prohibitive cost of updating all their weights.

However, the paper identifies a critical practical bottleneck: PT requires substantially more training time than fine-tuning to reach convergence. This is not a minor inconvenience — it undermines the very efficiency argument that motivates PT. Figure 2 illustrates the problem concretely: when training RoBERTa-LARGE on MNLI, prompt tuning takes dramatically longer wall-clock time to achieve the same validation accuracy as full fine-tuning. The paper frames this as a speed problem rather than a capability problem — PT eventually works, but it is pathologically slow.

This gap matters because the whole point of parameter-efficient methods is practical deployment. If a method saves parameters but costs significantly more training time, practitioners face an unwelcome tradeoff: save GPU memory at the expense of longer experimentation cycles, or accept the memory cost of fine-tuning for faster convergence. For extremely large PLMs where PT is most promising (since the memory savings are largest and the performance parity holds), the training time penalty becomes even more severe, potentially making PT impractical for iterative development or resource-constrained settings.

Why This Problem Was Underexplored Before This Work

The prompt tuning literature prior to this paper focused predominantly on two questions: (1) Can PT match fine-tuning performance? and (2) How do we design better prompts? The first question motivated scale analyses (Lester et al., 2021) and comparisons across architectures. The second question motivated work on automatic prompt search (Shin et al., 2020; Gao et al., 2021), prompt ensembling, and prompt length optimization. The efficiency question — how to make PT faster while preserving its parameter efficiency — received comparatively little attention.

Several factors contributed to this neglect. Parameter-efficient methods as a category (PT, adapter layers, LoRA, BitFit) were relatively new at the time of this paper's writing, and the community was primarily establishing their effectiveness ceiling rather than their efficiency floor. There was an implicit assumption that since PT trains so few parameters, it should naturally converge quickly — an intuition that proved incorrect in practice. The consistently slower convergence of PT compared to fine-tuning (as shown in Figure 2) was a counterintuitive empirical finding that needed explanation and, more importantly, a solution.

The Motivating Hypothesis: Prompts Encode Transferable Task Knowledge

The paper's central hypothesis is refreshingly intuitive: if soft prompts are the only part of the model being trained, they should concentrate the knowledge required to solve the task. This means trained prompts are not just arbitrary embeddings that happen to optimize a loss — they should be semantically meaningful representations of task-solving strategies conditioned on a specific PLM. If this hypothesis holds, then transferring trained prompts from one setting to another should provide a useful inductive bias, potentially accelerating learning and improving final performance.

This hypothesis connects directly to two well-established paradigms in NLP transfer learning:

Cross-task transfer (task-to-task). The idea of improving target task performance by first training on a related source task has a long history in NLP (Ruder, 2017). In the PLM era, this took the form of intermediate-task training (Phang et al., 2018; Pruksachatkun et al., 2020; Gururangan et al., 2020; Vu et al., 2020), where models are fine-tuned on an intermediate task before fine-tuning on the target task. Vu et al. (2020) empirically characterized which task pairs are "transferable" under this paradigm. However, these explorations focused exclusively on full-parameter fine-tuning — the intermediate task modifies all model weights, making the analysis fundamentally different from prompt-only transfer where the PLM's internal knowledge remains frozen.

The paper hypothesizes that prompts might exhibit stronger cross-task transferability than full fine-tuning for the same reason they are parameter-efficient: they are forced to encode task-relevant signals in a compact, bottlenecked representation that may capture more abstraction and less task-specific superficial statistics. Lester et al. (2021) had already provided a suggestive data point: PT showed "stronger cross-domain transferability than fine-tuning" — but this was a brief observation, not a systematic analysis.

Cross-model transfer (model-to-model). The idea of transferring knowledge between different model architectures or sizes has precedent in Net2Net (Chen et al., 2016), knowledge distillation (Hinton et al., 2015), and knowledge inheritance (Qin et al., 2021). These methods all transfer some form of learned knowledge — weights, logits, or internal representations — from a source model to a target model. The practical motivation is compelling: if you can train prompts on a small, computationally efficient PLM and then project them to a massive, expensive PLM, you get the best of both worlds — cheap training on the small model and powerful inference on the large model.

However, cross-model prompt transfer faces a fundamental obstacle that cross-task transfer does not: different PLMs have inconsistent embedding spaces. A prompt trained for RoBERTa-LARGE (1024-dimensional embeddings) cannot be directly plugged into T5-XXL (4096-dimensional embeddings). The prompts need to be projected across semantic spaces — a mapping problem that has no obvious solution and had not been attempted in the prompt tuning literature.

Where Prior Approaches Fall Short

The paper identifies specific gaps in existing work that motivate its empirical investigation:

1. No systematic study of prompt transferability. While Lester et al. (2021) noted in passing that PT showed strong cross-domain transfer, this was an anecdotal observation on a limited set of domains. No prior work had systematically characterized: Which task pairs transfer well? Does transferability depend on task type? Can prompts from completely different task families (e.g., sentiment analysis to question answering) transfer at all? Without this characterization, there was no principled way to select source tasks for transfer or predict when transfer would help versus hurt.

2. No investigation of cross-model prompt transfer. The prompt tuning literature had focused almost exclusively on single-model setups. The question of whether prompts trained on one PLM could be useful for a different PLM — especially across heterogeneous architectures (encoder-only vs. encoder-decoder) — was entirely unexplored. The technical challenge of projecting prompts across embedding spaces had no established solution.

3. Efficiency, not just effectiveness. Vu et al. (2021) concurrently explored cross-task prompt transfer with initialization ("SPOT"), but their focus was on improving final performance (effectiveness). The paper explicitly distinguishes itself: "Vu et al. (2021) focus on improving the effectiveness of PT but we attempt to improve the efficiency" (emphasis added). The distinction matters: a method that improves final accuracy but takes equally long to train doesn't solve the speed problem that Figure 2 highlights. The paper's goal is to accelerate convergence and improve performance — the speedup numbers in Table 1 (e.g., 49.7× for SST-2 on T5-XXL) reflect this dual objective.

4. No understanding of what drives transferability. Prior work on task transferability (Vu et al., 2020) used metrics like dataset size, task type, and embedding similarity to predict transfer success, but these analyses were for fine-tuning, not prompt tuning. The mechanisms are potentially different: fine-tuning modifies the PLM's internal representations, while PT only modifies the input embeddings, which must then stimulate the frozen PLM to produce desired outputs. The paper hypothesizes that how prompts stimulate the PLM's internal neurons is the key factor — a hypothesis that requires new analysis tools not present in prior transferability studies.

5. Limited scope of existing prompt transfer work. Lester et al. (2021) tested cross-domain transfer (e.g., between different text classification datasets) but did not explore cross-task transfer between fundamentally different task types (e.g., from NLI to sentiment analysis). Vu et al. (2021) tested prompt transfer but did not address cross-model transfer or analyze the neural mechanisms underlying transferability. This paper aims for a more comprehensive analysis along both axes.

How This Paper Positions Itself

The paper explicitly frames itself as an empirical investigation rather than a new method proposal. The abstract states this directly: "we empirically investigate the transferability of soft prompts across different downstream tasks and PLMs." This is important context — the contributions are primarily analytical (characterizing when and why transfer works) and methodological (providing recipes for transfer-based initialization that practitioners can adopt), with the TPT-TASK and TPT-MODEL methods serving as concrete instantiations of the transfer principles discovered.

The paper draws on concepts from two prior lines of work while extending them in novel directions:

• From cross-task transfer learning (Vu et al., 2020): the idea that task similarity determines transfer success, but applied to prompts rather than full model parameters.
• From knowledge transfer across models (Qin et al., 2021; Chen et al., 2016): the idea that learned knowledge can be projected across model boundaries, but adapted to the specific challenge of prompt embedding spaces.

The unifying research question is: Can we make prompt tuning faster by reusing prompts from related settings, and if so, what determines when this reuse succeeds? The paper answers this through a structured empirical program: first characterizing zero-shot transferability (what transfers without any additional training?), then testing transfer-as-initialization (does starting from a transferred prompt accelerate convergence?), and finally analyzing the underlying mechanism (what neural or embedding-level properties predict transfer success?).

A secondary but important motivation is scientific understanding: beyond the practical goal of accelerating PT, the paper aims to understand what trained soft prompts are. Are they simply learned embedding vectors in a high-dimensional space, or do they meaningfully encode task-specific strategies that are visible in the PLM's internal activations? The answer has implications for how we think about PT and parameter-efficient methods more broadly — it speaks to whether prompts should be treated as opaque optimization artifacts or as interpretable representations of task knowledge.

The Practical Stakes

The paper's motivation is not purely academic. The practical scenarios where prompt transfer matters are concrete:

• Multi-task deployment: An organization deploying a large PLM for multiple downstream tasks could maintain a "prompt warehouse" — a library of trained prompts for diverse tasks. When a new task arrives, the most similar existing prompt could be retrieved and used as initialization, dramatically reducing time-to-deployment. The similarity metrics explored in Section 6 (embedding distance, activation overlap) provide the retrieval mechanism for such a warehouse.

• Efficient model upgrading: When a new, larger PLM is released, an organization could project existing trained prompts from the old model to the new model rather than retraining from scratch. The cross-model projectors (Section 5.1) provide the technical pathway, and the TPT-MODEL speedups (e.g., 30× comparable-result speedup for QNLI on T5-XXL, Table 2) quantify the efficiency gain.

• Resource-constrained training: For researchers or practitioners who cannot afford the full training time of PT on a massive PLM, transfer provides a way to amortize the cost: train prompts once on a small model or a data-rich task, then transfer and only fine-tune briefly on the target setting. This makes PT accessible to a wider range of users.

These scenarios ground the empirical analysis in real-world utility, even though the paper does not deploy the methods in production systems.

3. Technical Approach

This is primarily an empirical analysis paper whose core idea is that trained soft prompts concentrate task-solving knowledge conditioned on a specific PLM, and that this knowledge can be transferred across tasks and models to accelerate prompt tuning (PT) — both in the zero-shot setting (where prompts are reused directly) and in the initialization setting (where transferred prompts serve as a warm start for further PT).

3.1 Reader Orientation

The paper builds a transfer framework for soft prompts that enables practitioners to improve the speed and final performance of prompt tuning by reusing prompts from related tasks or from different PLMs, rather than always starting from random initialization. The problem it solves is that PT is parameter-efficient but convergence-slow — Figure 2 shows PT taking substantially more wall-clock time than fine-tuning to reach the same accuracy on RoBERTa-LARGE for MNLI — and the solution is to inject prior knowledge about task-solving strategies through transferred prompts, which can serve either as direct predictions (zero-shot) or as smart initializations that dramatically reduce the number of training steps required.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, arranged in two parallel transfer pipelines plus a shared analysis layer:

1. Base Prompt Tuning (PT) — the standard method: randomly initialized soft prompts (learnable embedding vectors) are prepended to input sequences and trained with task supervision while the PLM's parameters are frozen. This serves as the baseline against which all transfer methods are compared.

2. Cross-Task Transfer Pipeline — takes trained soft prompts from a source task and either (a) reuses them directly on a target task (zero-shot transfer) or (b) uses them as initialization for PT on the target task (TPT-TASK). The only movable pieces are the prompt embeddings themselves; the PLM is shared between source and target.

3. Cross-Model Transfer Pipeline — takes trained soft prompts from a source PLM (e.g., RoBERTa-LARGE), passes them through a learned prompt projector (a two-layer perceptron that maps between embedding spaces), and feeds the projected prompts into a target PLM (e.g., T5-XXL) — again either zero-shot or as initialization (TPT-MODEL). The projector is trained on a small set of tasks with task-specific supervision.

4. Transferability Indicator Analysis — a set of prompt similarity metrics (embedding-distance-based and neuron-activation-based) that aim to predict cross-task prompt transfer success without running transfer experiments. The overlapping rate of activated neurons (ON) metric is the paper's novel contribution here, computed by feeding prompts into the PLM and measuring which feed-forward neurons fire.

5. Evaluation Framework — 17 NLP tasks across 6 types (SA, NLI, EJ, PI, QA, SUM) on two PLM families (RoBERTa and T5), evaluated by comparing zero-shot transfer performance, convergence speedup, and comparable-result speedup against vanilla PT baselines.

Information flows as follows: source task data → PT training on source task or source PLM → trained soft prompt → either (cross-task) direct reuse on target task, or (cross-model) projection through learned projector → target PLM → zero-shot evaluation OR continued PT with transferred initialization → final performance and speedup measurement.

3.3 Roadmap for the Deep Dive

• First, the standard prompt tuning formulation (Equation 1), since every transfer method builds on this foundation and the reader needs to understand what exactly is being transferred.
• Second, the cross-task transfer setting — zero-shot transfer procedure and TPT-TASK (transfer with initialization) — since this is the simpler of the two transfer axes and introduces the concepts of task-type-dependent transferability.
• Third, the cross-model transfer setting — the prompt projector architecture, training objectives (Distance Minimizing and Task Tuning), and TPT-MODEL — since this adds the projector learning problem on top of the cross-task framework.
• Fourth, the prompt similarity metrics and transferability indicators — embedding-based and neuron-activation-based — since these are the analytical tools for understanding why transfer works.
• Fifth, the speedup calculation methodology, since the paper's efficiency claims depend on how convergence and comparable-result speedups are measured.

3.4 Detailed, Sentence-Based Technical Breakdown

Standard Prompt Tuning (PT) Formulation

Prompt tuning, as studied in this paper, is the method from Lester et al. (2021) where only soft prompts are trained while all PLM parameters remain frozen. The formulation is crucial because it defines exactly what is being transferred: a set of learnable embedding vectors that condition the PLM to produce desired outputs.

Given an input sequence of $n$ tokens $X = \{x_1, x_2, \ldots, x_n\}$, the method prepends $l$ randomly initialized soft prompts $P = \{p_1, p_2, \ldots, p_l\}$ before the input, where each $p_i \in \mathbb{R}^d$ is an embedding vector and $d$ is the input dimension of the PLM (e.g., 1024 for RoBERTa-LARGE, 4096 for T5-XXL). The training objective maximizes the likelihood of generating the desired output $y$:

$\mathcal{L} = p(y \mid P, x_1, \ldots, x_n)$

where $y$ is the target output (a label token for classification tasks like sentiment analysis and NLI; a sequence for generation tasks like question answering and summarization), $P$ is the set of $l$ soft prompt embeddings that are the only learnable parameters, and $x_1, \ldots, x_n$ are the frozen input token embeddings. The PLM's internal parameters are completely fixed — no gradients flow into the transformer layers from anything other than the prompt positions.

What this computes: The log-likelihood of the target token sequence given the prompt-prepended input, with the PLM acting as a fixed function that maps the augmented input to an output distribution. Only the prompt embeddings receive gradient updates; the PLM's weights are treated as constants in the computation graph.

Why this form: This is the standard autoregressive or masked language modeling objective that the PLM was pre-trained with, ensuring that the prompts are optimized to elicit the PLM's existing knowledge rather than teaching it new capabilities. The prompts act as a task specification that steers the frozen model toward the desired behavior, analogous to how textual prompts work in few-shot learning but with the advantage of being directly optimized through gradient descent rather than hand-crafted.

For RoBERTa (which uses masked language modeling pre-training), the paper additionally prepends a special [MASK] token before the prompts and trains the prompts so that the PLM fills the target label into the masked position. For T5 (which uses sequence-to-sequence pre-training), the decoder directly generates the output sequence conditioned on the encoder's representation of the prompt-prepended input.

Key hyperparameters (from Appendix A.3): Optimizer is AdamW, learning rate is 0.001, prompt length $l$ is 100 (meaning 100 virtual tokens are prepended to every input). All soft prompts are randomly initialized from scratch for the vanilla PT baseline. For classification tasks, the PLM outputs a distribution over label tokens and the highest-likelihood token is selected as the prediction (ranking-based inference). For generation tasks, the PLM directly autoregressively generates the output sequence.

Cross-Task Transfer: Zero-Shot and TPT-TASK

Cross-task transfer exploits the fact that if a soft prompt is the only parameter capturing task-specific knowledge for a source task, that knowledge may partially apply to a related target task. The paper investigates this in two modes: zero-shot direct reuse (no further training) and initialization-based transfer (TPT-TASK).

Zero-shot transfer procedure. For a source task $t_{\text{src}}$, the paper first conducts standard PT to convergence — training the soft prompts $P^{t_{\text{src}}}$ on the source task's training data using the objective in Equation 1. Then, for a target task $t_{\text{tgt}}$, the trained prompts $P^{t_{\text{src}}}$ are directly prepended to the target task's evaluation examples, and the PLM's predictions are measured without any additional training. The zero-shot transfer performance is compared against two baselines: (a) the original PT performance on the target task (trained from scratch on target data), and (b) randomly initialized prompts evaluated on the target task (the "random prompt" row in Figure 3). The relative zero-shot transfer performance is computed as $\text{zero-shot transfer accuracy} / \text{original PT accuracy} \times 100\%$.

What Figure 3 shows (and why it matters). The grid of relative zero-shot transfer percentages in Figure 3 reveals that transfer success is strongly conditioned on task type. For same-type tasks (e.g., IMDB → SST-2, both sentiment analysis), transfer can achieve 90–100% of the original PT performance and occasionally exceed it (e.g., IMDB → Movie achieves 101% for RoBERTa-LARGE), especially when the source task has more training data — the IMDB dataset contains 25,000 reviews versus Movie Rationales' 1,600, so the source prompt encodes richer sentiment knowledge. For different-type tasks (e.g., sentiment analysis → NLI), transfer performance collapses to near-random baseline (e.g., IMDB → MNLI achieves 37% relative performance for RoBERTa-LARGE, meaning zero-shot transfer gets only 37% of what training from scratch achieves). This task-type dependency is the central empirical finding of the cross-task analysis: prompts are not generic task-solvers but rather encode type-specific strategies.

An important negative result (unified label tokens, Appendix B.2). The paper explicitly tests whether poor cross-type transfer is simply due to different label tokens (e.g., SA tasks use "positive"/"negative" while NLI tasks use "yes"/"no"). By unifying all label tokens to numbers ("1", "2", ...), the paper finds that "the transferability between different-type tasks are generally not improved in this way," concluding that "different-type tasks surely require distinct abilities, which prohibits reusing prompts between them." This rules out a superficial explanation and points toward deeper representational differences.

TPT-TASK: cross-task transferable prompt tuning. For a target task, TPT-TASK selects the source task that achieved the best zero-shot transfer performance in Figure 3 (i.e., the most similar task in the zero-shot sense), retrieves its trained soft prompts $P^{\text{best}}$, and uses them as the initialization for PT on the target task — rather than starting from random initialization. Formally, instead of initializing $P \sim \mathcal{N}(0, \sigma^2)$ randomly, TPT-TASK sets $P^{(0)} = P^{\text{best}}$ and then continues standard PT optimization (Equation 1) on the target task's training data.

Why this order: zero-shot first, then initialization. The paper uses zero-shot transfer performance as a retrospective similarity metric to select the best source task. This is computationally expensive if done naively (you'd need to run PT on every possible source task), but in practice the paper pre-computes the zero-shot transfer matrix (Figure 3) across all task pairs, so selecting the best source task for a given target is a simple lookup. The choice to use zero-shot performance rather than some cheaper similarity metric is pragmatic: it directly measures what matters (how well prompts transfer) rather than relying on a proxy that may not correlate with transfer success.

Performance and speedup (Table 1). TPT-TASK generally achieves better or comparable final performance to vanilla PT (e.g., on T5-XXL, SST-2 improves from 97.4% to 97.8%, laptop improves from 76.6% to 84.2%), while reducing the number of training steps to convergence. The convergence speedup is defined as the ratio of PT convergence steps to TPT-TASK convergence steps. On T5-XXL, speedups are dramatic for some tasks: 49.7× for SST-2, 24.9× for QNLI, 49.9× for SNLI, 29.8× for deontology. However, some tasks show minimal speedup (1.0–1.1×) or even slowdown (0.7× for deontology on RoBERTa-LARGE), indicating that transfer is not universally beneficial — it depends on the quality of the source prompt match. The comparable-result speedup measures the time for TPT-TASK to reach performance equivalent to converged vanilla PT, and is even more dramatic on some tasks: 219.8× for laptop on T5-XXL (meaning TPT-TASK achieves what vanilla PT achieves in roughly 1/220th of the training time), though this metric is not computable (N/A) for tasks where vanilla PT's convergence time is already very short.

Cross-Model Transfer: Projector Architecture and Training

Cross-model transfer aims to use prompts trained on a computationally cheaper source PLM to improve PT on a more expensive target PLM. This is harder than cross-task transfer because different PLMs have incompatible embedding spaces — the source prompt $P^s$ lives in $\mathbb{R}^{l \times d_s}$ while the target PLM expects embeddings in $\mathbb{R}^{l \times d_t}$, where $d_s$ and $d_t$ are the hidden dimensions of the source and target PLMs respectively (e.g., RoBERTa-LARGE has $d_s = 1024$ and T5-XXL has $d_t = 4096$). Direct reuse is geometrically impossible.

Prompt projector architecture. The paper parameterizes the cross-model projection as a two-layer perceptron with a non-linear activation. First, the $l$ virtual tokens of the source prompt $P^s = \{p^s_1, \ldots, p^s_l\}$ are concatenated into a single flat vector $\mathbf{P}^s \in \mathbb{R}^{l d_s}$ by stacking all $l$ embedding vectors end-to-end. The projector then maps this flat vector to the target space:

$\tilde{\mathbf{P}}^s = \text{Proj}(\mathbf{P}^s) = W_2(\sigma(\mathbf{P}^s W_1 + b_1)) + b_2$

where $\mathbf{P}^s \in \mathbb{R}^{l d_s}$ is the flattened source prompt, $W_1 \in \mathbb{R}^{d_h \times l d_s}$ is the first linear transformation (mapping from the flattened source dimension to a hidden dimension $d_h$), $b_1 \in \mathbb{R}^{d_h}$ is the first bias, $\sigma$ is a non-linear activation function (LeakyReLU or tanh — the paper reports they tried both and found "their performance on various PLMs are similar," with reported results using LeakyReLU), $W_2 \in \mathbb{R}^{l d_t \times d_h}$ is the second linear transformation mapping to the flattened target dimension, $b_2 \in \mathbb{R}^{l d_t}$ is the second bias, and $\tilde{\mathbf{P}}^s \in \mathbb{R}^{l d_t}$ is the projected prompt, which is then reshaped into $l$ prompt embeddings of dimension $d_t$ that can be fed into the target PLM.

What this computes: A learned non-linear mapping from the source PLM's prompt embedding manifold to the target PLM's prompt embedding manifold. The two-layer structure with a non-linearity is necessary because the mapping between embedding spaces is unlikely to be a simple linear transformation — different PLMs organize their semantic spaces differently, and the non-linearity allows the projector to learn a warping that aligns the functional meaning of prompts (their effect on model behavior) across architectures.

Why concatenate-and-project rather than per-token projection: The alternative would be to project each of the $l$ prompt tokens independently (sharing the same projection weights across positions), which would be more parameter-efficient but would lose positional information. The concatenation approach treats the entire prompt sequence as a single high-dimensional object, allowing the projection to capture dependencies between different prompt token positions — if the first prompt token tends to control task type and the fifth prompt token tends to control output format in the source PLM, the projector can learn to preserve (or rearrange) this functional decomposition in the target PLM.

Hidden dimension $d_h$ is 768 (from Appendix C.1), meaning the bottleneck representation of the flattened prompt is 768-dimensional regardless of the source and target PLM sizes. This is a design choice that forces the projector to learn a compressed representation of the prompt's task knowledge, which may help generalization to unseen tasks.

Projector training objectives. The paper investigates two approaches to learn the projector parameters $W_1, W_2, b_1, b_2$:

Distance Minimizing trains the projector by minimizing the L2 distance between the projected source prompt and a "parallel prompt" — a prompt independently trained on the same task on the target PLM. Formally, for a specific training task $t$ (e.g., laptop), let $\mathbf{P}^s_t$ be the flattened prompt trained on the source PLM for task $t$, and $\mathbf{P}^t_t$ be the flattened prompt trained directly on the target PLM for the same task $t$. The training objective is:

$\mathcal{L}_{\text{dist}} = \|\text{Proj}(\mathbf{P}^s_t) - \mathbf{P}^t_t\|_2$

where $\|\cdot\|_2$ is the Euclidean (L2) norm.

What it computes: The straight-line distance in the target PLM's prompt embedding space between the projected prompt and the "ground-truth" prompt for that task on the target model.

Why this form: The intuition is that if the projector can map a source prompt to the exact location where the target PLM expects the prompt for that task to be, it has learned the correct cross-model correspondence. The L2 distance is a natural geometry for embedding spaces trained with gradient descent, where nearby points tend to produce similar model behaviors.

Why it fails to generalize (key negative result): As shown in Table 2(a), Distance Minimizing works well for the training task itself (e.g., laptop → laptop achieves 76.6% zero-shot transfer performance — matching the original PT performance exactly because the projector was explicitly trained to reproduce the target prompt), but "falls back to random performance on the other unseen tasks." For example, when the projector is trained on laptop, its zero-shot transfer on MNLI drops to 31.8%, indistinguishable from the random prompt baseline of 31.8%. This is because Distance Minimizing only teaches the projector to reproduce one specific target prompt — it does not learn how PLMs process prompts differently or how to adjust prompts to compensate for architectural differences. The projector essentially memorizes a single mapping without understanding the underlying transformation.

Task Tuning trains the projector by directly optimizing the projected prompts on the target PLM using task-specific supervision — the same objective as standard PT (Equation 1), but with gradients flowing through the projector into the projector parameters rather than into the prompts themselves. Specifically, for a training task $t$, the projected prompt $\text{Proj}(\mathbf{P}^s_t)$ is fed into the target PLM, and the standard task loss (e.g., cross-entropy for classification) is backpropagated to update $W_1, W_2, b_1, b_2$ — the source prompts $\mathbf{P}^s_t$ remain frozen.

$\mathcal{L}_{\text{tune}} = p_{\text{target PLM}}(y \mid \text{Proj}(\mathbf{P}^s_t), x_1, \ldots, x_n)$

where the PLM's parameters are frozen, the source prompts $\mathbf{P}^s_t$ are frozen, and only the projector parameters receive gradients through the chain: $\frac{\partial \mathcal{L}_{\text{tune}}}{\partial \text{Proj}} \cdot \frac{\partial \text{Proj}}{\partial (W_1, W_2, b_1, b_2)}$.

What it computes: The standard task loss on the target PLM, but optimized over projector parameters rather than prompt parameters.

Why this works better than Distance Minimizing: Task Tuning forces the projector to learn not just where to place prompts in embedding space, but how to make prompts that actually work on the target PLM. The projector learns the target PLM's "prompt processing function" — what kinds of prompt embeddings elicit the right task behavior — rather than just a point correspondence. This functional knowledge generalizes to unseen tasks of the same type because tasks of the same type require similar processing (e.g., all NLI tasks require the PLM to compare premise and hypothesis), so a projector that learned to make NLI prompts work on the target PLM can transfer to other NLI tasks.

Generalization results (Table 2a): Task Tuning projectors trained on laptop (an SA task) successfully generalize to other SA tasks — IMDB achieves 82.9%, SST-2 achieves 89.3%, Movie achieves 78.6% — all substantially above the random baseline of approximately 49–51%. However, they fail on different-type tasks (e.g., 32.4% on MNLI, close to random). Similarly, projectors trained on MNLI (an NLI task) generalize to QNLI (82.7%) and SNLI (88.2%) — other NLI tasks — but not to SA tasks. This task-type-specific generalization mirrors the cross-task transfer findings: the prompt projector can learn to compensate for architectural differences within a task family but cannot bridge fundamentally different task types.

Projector training details (Appendix C.1): Optimizer is AdamW, training batch size is 16, learning rate is 0.005, and the inner hidden size $d_h$ is 768. The paper experimented with both tanh and LeakyReLU activations and found similar performance; reported results use LeakyReLU. The projector training tasks chosen are laptop (representing SA type) and MNLI (representing NLI type) — these serve as anchors for generalization within their respective task families.

LayerNorm addition for small PLMs (Appendix C.3). When transferring between small PLMs (e.g., from BERT-BASE to RoBERTa-BASE), the projected prompts sometimes have "quite different L2 norm values" compared to original prompts, making them hard to optimize in TPT-MODEL (Figure 7, Without LayerNorm panels, shows training loss divergence). The paper adds layer normalization to the projector output:

$\tilde{\mathbf{P}}^s = \text{LayerNorm}(\text{Proj}(\mathbf{P}^s))$

This normalizes the projected prompt to have zero mean and unit variance across its elements, which stabilizes optimization. With LayerNorm, the training curves (Figure 7, With LayerNorm panels) show normal convergence. Interestingly, prompts projected with and without LayerNorm achieve "similar zero-shot transfer performance" (Table 8), meaning LayerNorm mainly helps during subsequent fine-tuning (TPT-MODEL) rather than changing the zero-shot quality.

TPT-MODEL: cross-model transferable prompt tuning. For a target PLM and target task, TPT-MODEL first projects the source PLM's trained prompts for that task using the Task Tuning projector, then uses the projected prompts as initialization for PT on the target PLM. Formally: $P^{(0)}_{\text{target}} = \text{Proj}_{\text{Task Tuning}}(\mathbf{P}^s_{\text{target}})$, followed by standard PT optimization. The source PLM prompts $\mathbf{P}^s_{\text{target}}$ are obtained by training on the source PLM for the same target task — this is a "same task, different model" transfer (unlike TPT-TASK which is "different task, same model").

Performance and speedup (Table 2b). For tasks within the same type as the projector-training task, TPT-MODEL achieves substantial speedups: for the projector trained on MNLI, SNLI shows 30× comparable-result speedup and QNLI shows 20× comparable-result speedup on T5-XXL. For the projector trained on laptop, SST-2 shows 19× comparable-result speedup and laptop itself shows 16× convergence speedup. However, tasks of different types from the projector-training task see more modest benefits or even slowdowns (e.g., IMDB with the MNLI projector shows only 1.0× comparable-result speedup, meaning no improvement).

A nuance on the RoBERTa → T5 transfer direction: The paper describes this as the "most difficult setting" because RoBERTa-LARGE and T5-XXL are heterogeneous architectures — RoBERTa is an encoder-only model that reads input bidirectionally, while T5 is an encoder-decoder model where the decoder autoregressively generates output. The prompts must be projected not just across different embedding dimensions (1024 → 4096) but across fundamentally different computation structures. The fact that Task Tuning projectors still work (within task types) suggests that prompts encode task knowledge in a way that is, to some degree, architecture-agnostic — the task-solving strategy can be re-expressed in the target model's "prompt language."

Prompt Similarity Metrics and Transferability Indicators

Section 6 aims to understand why prompts transfer — what underlying properties of trained prompts predict transfer success. The paper evaluates two families of similarity metrics: embedding-distance-based metrics and a novel neuron-activation-based metric. The goal is to find metrics that correlate well with the zero-shot transfer performance observed in Figure 3, so that these metrics can serve as "transferability indicators" for predicting which source tasks will transfer well to a target without running expensive transfer experiments.

Embedding Similarity Metrics. These treat trained soft prompts purely as vectors in the PLM's embedding space, ignoring how the PLM processes them:

Concatenation-based Euclidean similarity ($E_{\text{concat}}$): All $l$ prompt tokens of task $t_1$ are concatenated into a single vector $P^{t_1} \in \mathbb{R}^{l d}$, and similarly for $t_2$. The similarity is:

$E_{\text{concat}}(P^{t_1}, P^{t_2}) = \frac{1}{1 + \|P^{t_1} - P^{t_2}\|}$

where $\|P^{t_1} - P^{t_2}\|$ is the Euclidean (L2) distance between the two concatenated prompt vectors, and the $1 + \, \cdot\,$ in the denominator ensures the similarity ranges from 0 (infinitely far) to 1 (identical), avoiding division by zero.

What it computes: A measure of how far apart the two prompts are in the PLM's embedding space when treated as undifferentiated vectors. The $1/(1+d)$ transformation converts distance to similarity (closer = more similar).

Why this form: L2 distance is the natural geometry for Euclidean space; the reciprocal transformation is a standard way to convert distance to a $[0, 1]$ similarity. However, it assumes that all prompt token positions are aligned (token 1 of $P^{t_1}$ corresponds to token 1 of $P^{t_2}$), which may not be true — prompts are unordered in the sense that tokens at different positions could play similar functional roles.

Concatenation-based cosine similarity ($C_{\text{concat}}$):

$C_{\text{concat}}(P^{t_1}, P^{t_2}) = \frac{P^{t_1} \cdot P^{t_2}}{\|P^{t_1}\| \, \|P^{t_2}\|}$

where $\cdot$ denotes the dot product between the two concatenated vectors.

What it computes: The cosine of the angle between the two concatenated prompt vectors, ranging from -1 (opposite directions) to 1 (identical direction). This is scale-invariant — prompts with different L2 norms can still have high cosine similarity if they point in the same direction.

Why this form: Cosine similarity is the standard metric for comparing embedding vectors in NLP because it focuses on direction rather than magnitude, which often matters more for semantic content. The disadvantage is the same position-alignment assumption as $E_{\text{concat}}$.

Position-invariant variants ($E_{\text{average}}$ and $C_{\text{average}}$). To address the position-alignment assumption, the paper computes pairwise similarities between every token pair across the two prompts and averages them:

$E_{\text{average}}(P^{t_1}, P^{t_2}) = \frac{1}{1 + \frac{1}{l^2} \sum_{i=1}^l \sum_{j=1}^l \|p^{t_1}_i - p^{t_2}_j\|}$

$C_{\text{average}}(P^{t_1}, P^{t_2}) = \frac{1}{l^2} \sum_{i=1}^l \sum_{j=1}^l \frac{p^{t_1}_i \cdot p^{t_2}_j}{\|p^{t_1}_i\| \, \|p^{t_2}_j\|}$

where $p^{t_1}_i$ and $p^{t_2}_j$ are individual prompt token embeddings, and the double sum over $i, j$ computes the average over all $l^2$ token pairs.

What these compute: The average pairwise distance or cosine similarity between all combinations of individual prompt tokens. If the prompts have similar "vocabulary" of token directions, the average will be high even if the tokens are in different orders.

Why this form: Position-invariant metrics recognize that prompt tokens are not inherently ordered — a prompt that places a "task identification" token at position 3 in one training run might place it at position 7 in another, and we shouldn't penalize this. However, averaging over all pairs loses potentially important structural information about token sequences.

Model Stimulation Similarity: Overlapping Rate of Activated Neurons (ON). This is the paper's novel contribution to prompt similarity measurement. Rather than comparing prompts in embedding space, it compares how prompts stimulate the PLM's internal neurons — the idea being that prompts are not just data but "programs" that activate specific computational pathways in the frozen PLM, and two prompts are similar if they activate the same pathways.

The metric builds on the observation from Geva et al. (2021) and Dai et al. (2021) that neurons in the feed-forward layers of transformers correspond to specific model behaviors — individual neurons or small groups of neurons encode interpretable concepts (e.g., temporal relations, syntactic patterns, factual knowledge). If this view is correct, then the set of neurons that a prompt activates is a meaningful "signature" of what the prompt is asking the PLM to do.

Computing activation states. For each Transformer layer's feed-forward network (FFN), given an input embedding $x \in \mathbb{R}^d$:

$\text{FFN}(x) = \max(x W_1^\top + b_1, 0) W_2 + b_2$

where $W_1, W_2 \in \mathbb{R}^{d_m \times d}$ are trainable weight matrices, $d_m$ is the intermediate (hidden) dimension of the FFN (typically 4× the model dimension $d$), and $b_1, b_2$ are bias vectors. The ReLU activation $\max(x W_1^\top + b_1, 0)$ produces a vector of $d_m$ non-negative values — these are the activation values of $d_m$ hidden neurons in that layer.

The paper then binarizes these activations: all positive elements become 1, all non-positive elements become 0, producing a one-hot activation state vector $s \in \{0, 1\}^{d_m}$ for that layer. A neuron is considered "activated" if its pre-ReLU value is strictly positive.

Input construction. To get the activation states for a given prompt $P$, the paper feeds the sequence $\{P, \langle s \rangle\}$ into the PLM, where $\langle s \rangle$ is the special token indicating the start of a sentence (but no actual task input is provided — the goal is to see how the prompt alone stimulates the model, without influence from a specific example). The activation states are collected at the output position — the position used for decoding predictions:

• For T5 (encoder-decoder): the activation states of the decoder module's first position (the position that generates the first output token).
• For RoBERTa (encoder-only, masked LM): the activation states at the [MASK] position.

These output positions are used because they should be "more task-specific" — the prompt's influence on what the model predicts should be concentrated at the locations where predictions are made.

Aggregating across layers. The activation states from $L$ layers (or a subset thereof) are concatenated into an overall activation state vector:

$\text{AS}(P) = [s_1; s_2; \ldots; s_L]$

where $s_i \in \{0, 1\}^{d_m}$ is the binarized activation vector of layer $i$. The paper finds that "the higher layers tend to be more task-specific, which is consistent with the probing results" (Liu et al., 2019a), so they use only the top 3 layers in the main experiments (meaning $\text{AS}(P) \in \{0, 1\}^{3 d_m}$ for RoBERTa's 12-layer architecture or T5's 24-layer decoder).

Computing the ON similarity. The overlapping rate between two prompts $P^{t_1}$ and $P^{t_2}$ is computed as the cosine similarity of their activation state vectors:

$\text{ON}(P^{t_1}, P^{t_2}) = \frac{\text{AS}(P^{t_1}) \cdot \text{AS}(P^{t_2})}{\|\text{AS}(P^{t_1})\| \, \|\text{AS}(P^{t_2})\|}$

where $\cdot$ is the dot product between the two binary activation vectors, and the denominator normalizes by their lengths.

What it computes: The cosine similarity between the binarized neuron activation patterns of the two prompts. Since the vectors are binary (0/1), the dot product counts the number of neurons activated by both prompts (intersection), and the normalization adjusts for the total number of neurons activated by each (so two prompts that activate a small set of identical neurons get a high score, while two prompts that activate many neurons but with different patterns get a lower score).

Why cosine similarity rather than Jaccard or overlap fraction: The paper does not explicitly justify this choice, but cosine similarity on binary vectors provides a natural balance: it penalizes both prompts that activate too few neurons (making the intersection small relative to what could be activated) and prompts that activate too many neurons (making the normalization denominators large). The Jaccard index $|A \cap B| / |A \cup B|$ would be more sensitive to the size of the activation sets, while cosine similarity focuses on the pattern of activated vs. non-activated neurons.

Why binarization: Binarizing the activations (positive → 1, zero/negative → 0) discards information about how strongly neurons fire, keeping only the binary information of whether they fire. This is a deliberate choice: the hypothesis is that which neurons are recruited matters more than fine-grained activation magnitudes, which may be noisy or model-specific. The binarization also makes the metric robust to the scale of activations, which can vary across different prompts or different model instances.

Evaluating similarity metrics as transferability indicators (Table 4, Figure 4). The paper evaluates how well each similarity metric correlates with zero-shot transfer performance using Spearman's rank correlation: for each target task's prompt, source tasks are ranked by their similarity score and separately by their actual zero-shot transfer performance, and the correlation between these two rankings is computed.

On RoBERTa-LARGE, the ON metric achieves a Spearman's correlation of 49.7% — higher than all embedding-based metrics (the best embedding metric, $C_{\text{average}}$, achieves 44.7%, and $E_{\text{average}}$ achieves only 2.8%). This supports the paper's central claim: how prompts stimulate the PLM's internal neurons is more predictive of transfer success than where they sit in embedding space.

However, ON degrades with model scale (Figure 4): The Spearman's correlation drops from 52.5% on T5-Small to 36.9% on T5-XXL. The paper attributes this to model redundancy — larger PLMs have more parameters than needed for any given task (Aghajanyan et al., 2021), meaning "prompts can activate different redundant neurons to do similar jobs." If two prompts for the same task activate different but functionally equivalent sets of neurons (because the PLM has multiple redundant pathways for the same computation), the ON metric will incorrectly judge them as dissimilar. The embedding-based metric $C_{\text{average}}$ also degrades with scale (from 37.1% on T5-Small to 33.5% on T5-XXL), but less dramatically.

ONI: intersection-based variant to combat redundancy (Appendix D.3). To partially recover from redundancy, the paper proposes ONI, which computes activation states as the intersection of activation patterns from three prompts trained on the same task with different random seeds. If a neuron is activated by all three independently trained prompts for the same task, it is likely functionally important rather than a spurious activation; if it varies across seeds, it may be redundant. Using ONI, the Spearman's correlation on T5-XXL rises from 36.9% to 46.3%, partially closing the gap to smaller models. This is a clever but computationally expensive fix (requiring training three prompts per task), and the paper acknowledges it as a "preliminary trial."

Why ON matters for transferability. The ON metric is not just a correlation-mining exercise — it offers a mechanistic hypothesis: prompts transfer well when they activate similar computational pathways in the PLM. If a source task prompt and a target task prompt both activate the same set of feed-forward neurons (the "task-solving circuits"), then the source prompt likely encodes knowledge that is relevant to the target task. If they activate disjoint sets, transfer will fail. This reframes prompt transferability from a geometric problem (distance in embedding space) to a functional problem (overlap in recruited neural circuits).

Cross-model validation of ON (Table 5). The paper further tests whether ON works for cross-model transfer by measuring the similarity between projected prompts (via Task Tuning projector) and original prompts directly trained on the target PLM for the same task. The ON metric correctly identifies that projected prompts are highly similar to original prompts for same-type tasks (52.4% similarity for laptop, 51.0% average for same-type tasks of the laptop projector) but less similar for different-type tasks (46.0%). In contrast, $C_{\text{average}}$ shows uniformly low similarities (3.4–4.1%) regardless of task type, failing to distinguish same-type from different-type transfer. This reinforces that the neuron-activation perspective captures something essential that embedding geometry misses.

Speedup Calculation Methodology

The paper's efficiency claims rely on two speedup metrics (computed in Appendix B.3), and it is important to understand exactly how they are defined to interpret the numbers in Table 1 and Table 2:

Convergence detection. The paper calculates training loss and evaluation score every 100 steps during training. Convergence is defined as the point where "the training loss stops dropping and the evaluation score stops increasing for 300 steps" — meaning three consecutive evaluation periods (300 steps) without improvement.

Convergence speedup:

$\text{Convergence Speedup} = \frac{\text{PT convergence time (steps or wall-clock)}}{\text{TPT convergence time (steps or wall-clock)}}$

What it computes: The ratio of how long vanilla PT takes to converge versus how long the transfer method (TPT-TASK or TPT-MODEL) takes to converge. A value of 49.7× for SST-2 on T5-XXL (Table 1) means TPT-TASK reaches its own convergence point roughly 50 times faster than vanilla PT reaches its convergence point.

Why this definition can be misleading: The converged performance of TPT-TASK might be higher than vanilla PT (as it often is, e.g., SST-2 improves from 97.4% to 97.8%), so the speedup reflects both faster convergence and a higher target. If TPT converges to 98% in 1000 steps while PT converges to 97% in 50000 steps, the 50× speedup conflates "faster training" with "better final result."

Comparable-result speedup:

$\text{Comparable-result Speedup} = \frac{\text{PT convergence time}}{\text{time for TPT to achieve PT's converged performance}}$

What it computes: The ratio of how long vanilla PT takes to reach its own converged performance versus how long TPT takes to reach that same performance level. A value of 219.8× for laptop on T5-XXL (Table 1) means TPT-TASK matches vanilla PT's final accuracy in roughly 1/220th of the training time.

Why this is the more conservative metric: It controls for the fact that TPT often achieves better final performance — it asks "how much faster can I get to the performance I would have gotten without transfer?" rather than "how much faster can I get to a potentially better performance?" Tasks marked "N/A" for comparable-result speedup are those where TPT never reaches PT's converged performance within a reasonable time (or PT converged so quickly that no meaningful speedup was possible).

Why two metrics: They answer different practical questions. Convergence speedup answers "how much faster can I finish training if I use transfer?" Comparable-result speedup answers "how much faster can I achieve acceptable performance if I use transfer?" For a practitioner with a fixed performance target, comparable-result speedup is more relevant; for one who wants to maximize final performance, convergence speedup is more relevant.

4. Key Insights and Innovations

Innovation 1: Prompt Transferability Is Governed by Task Type, Not Surface Similarity

The paper's most consequential finding is not just that prompts can transfer — it's that transfer success is qualitatively determined by whether the source and target tasks are of the same type (sentiment analysis, NLI, summarization, etc.), and that this boundary is sharp and robust across multiple independent lines of evidence. This is a fundamentally different picture from what the field might have expected based on prior transfer learning results.

The prior assumption was that transferability depends on dataset-level properties. In the intermediate-task training literature (Phang et al., 2018; Vu et al., 2020), transfer success between fine-tuning tasks correlates with factors like dataset size, domain overlap, and the number of training examples — continuous variables that suggest a smooth spectrum of transferability where any task might help any other task to varying degrees. The dominant paradigm was to search for the best intermediate task by trying many candidates, treating transferability as an empirical question to be resolved experimentally for each target.

This paper shows the boundary is categorical, not continuous. The zero-shot transfer matrices (Figure 3, both RoBERTa and T5) exhibit a clear block-diagonal structure: same-type tasks form high-transfer clusters (sentiment analysis tasks reliably transfer to other sentiment analysis tasks, achieving 84–101% relative performance on RoBERTa-LARGE), while cross-type transfer collapses to near-random baselines (sentiment → NLI transfer hovers at 37–55%, indistinguishable from the 36–52% random baseline). The cross-model transfer results (Table 2a) replicate this: Task Tuning projectors generalize to same-type unseen tasks but fail catastrophically on different-type tasks.

Why this matters conceptually: It suggests that soft prompts do not learn generic "task-solving knowledge" or superficial dataset statistics — they learn type-specific cognitive strategies that only apply when the underlying reasoning demand is the same. A prompt trained for sentiment analysis learns to extract and classify affective content, and this skill transfers because all sentiment analysis tasks share this core requirement, regardless of domain differences (movie reviews vs. tweets vs. restaurant reviews). But when asked to perform NLI — which requires comparing two sentences for logical entailment — the sentiment prompt is useless because it encodes no knowledge about relational reasoning between premise and hypothesis. The paper explicitly verifies this is not a superficial label-token mismatch (Appendix B.2: unifying label tokens to numbers doesn't restore cross-type transfer), ruling out the most obvious alternative explanation.

This reframes the problem of selecting source tasks for transfer: instead of exhaustively trying candidates, you only need to identify the task type and pick from within that family. The practical implication is substantial — a "prompt warehouse" would be organized by task type, not by dataset name or domain, and retrieval for a new task would start by classifying the task type, not by measuring embedding similarity to existing prompts.

Innovation 2: The Overlapping Rate of Activated Neurons (ON) as a Functional Transferability Metric

The paper introduces a genuinely novel diagnostic concept: prompt similarity should be measured by which neurons they activate in the frozen PLM, not by where they sit in embedding space. This is a conceptual shift from treating prompts as passive data (vectors to be compared geometrically) to treating them as active programs (instructions that recruit specific neural circuits). The ON metric operationalizes this idea and, critically, the paper validates it against actual transfer outcomes rather than merely proposing it as an intuition.

Prior work measured prompt similarity geometrically. Vu et al. (2021) used cosine similarity between prompt embeddings as a transferability indicator — a natural choice if you think of prompts as points in a vector space where distance corresponds to functional similarity. The implicit assumption is that the PLM's embedding space is semantically well-organized: prompts that are close in Euclidean or angular distance should produce similar model behaviors.

This paper demonstrates that geometry is an imperfect proxy. On RoBERTa-LARGE, the best embedding metric (C_average) achieves a Spearman's correlation of 44.7% with zero-shot transfer performance, while ON achieves 49.7% (Table 4). More dramatically, in cross-model transfer (Table 5), C_average assigns uniformly low similarities (3.4–4.1%) regardless of whether the projected prompt is for a same-type or different-type task — it completely fails to distinguish successful transfers from failures — while ON correctly assigns high similarity to same-type projected prompts (51.0% for laptop's same-type generalization) and lower similarity to different-type ones (46.0%). ON sees functional equivalence that embedding geometry misses entirely.

The deeper insight is about what prompt tuning actually does. If two prompts with different embedding vectors activate the same neurons and produce the same behavior, then the embedding representation is not the meaningful level of analysis — the PLM's internal computation is. This aligns with the mechanistic interpretability perspective (Geva et al., 2021; Dai et al., 2021) that feed-forward neurons encode interpretable concepts, and extends it to a practical diagnostic: you can predict whether a prompt will work on a new task by checking whether it stimulates the right conceptual circuits, without running any transfer experiments.

The redundancy problem is itself an insight. The finding that ON's correlation degrades on larger models (from 52.5% on T5-Small to 36.9% on T5-XXL, Figure 4) is not just a limitation — it reveals something about how large PLMs work. The paper attributes this to redundancy (Aghajanyan et al., 2021): larger models have multiple functionally equivalent pathways for the same computation, so two prompts for the same task might activate different but equivalent sets of neurons, reducing ON overlap. The ONI variant (taking the intersection of activations across three seeds, Appendix D.3) partially recovers the correlation (46.3% on T5-XXL), confirming that the drop is indeed due to redundancy rather than ON being fundamentally wrong. This diagnostic — that redundancy masks functional similarity in large models — is a finding with implications beyond prompt tuning, relevant to any method that tries to read out model internals to predict behavior.

Significance beyond this paper. The ON metric is not a better embedding similarity — it's a fundamentally different category of metric based on model responses rather than model inputs. It opens the door to transferability prediction methods that probe the model's internal computation rather than comparing surface representations, which is a richer source of signal that may generalize better as models grow and their embedding spaces become less interpretable.

Innovation 3: Cross-Model Prompt Projection via Task-Conditioned Functional Alignment

The paper demonstrates something that was not obvious a priori: prompts trained on one PLM can be projected to work on a completely different PLM architecture, and this projection can be learned with modest task-specific supervision. This is a fundamental advance because it establishes that prompt knowledge is, to some degree, architecture-agnostic — the task-solving strategy encoded in a prompt can be re-expressed in the target model's "prompt language" once you learn the right translation function.

Prior cross-model knowledge transfer relied on different mechanisms. Knowledge distillation (Hinton et al., 2015) transfers knowledge by training the target model to match the source model's output distributions, requiring access to both models during target training. Net2Net (Chen et al., 2016) and knowledge inheritance (Qin et al., 2021) transfer weights directly, requiring architectural compatibility (same layer types, compatible dimensions). None of these approaches addressed the specific problem of mapping a compact, task-specific representation (a prompt) across incompatible embedding spaces.

The Distance Minimizing failure (Table 2a) is a sharp conceptual lesson. Training a projector to minimize L2 distance to a "ground truth" prompt on the target model works perfectly for the training task (laptop → laptop achieves 76.6%, matching vanilla PT) but collapses to random on any unseen task. This is a case study in why point correspondence is insufficient for functional transfer: the projector learns to reproduce one specific embedding pattern without understanding what makes prompts work on the target PLM. The target PLM's prompt-to-behavior mapping is a function, and Distance Minimizing learns a single point on that function's graph — useless for generalizing to new points.

Task Tuning succeeds because it learns the function, not a point. By backpropagating task supervision through the projector, the method learns what kinds of prompt embeddings elicit correct task behavior from the target PLM — essentially learning the target PLM's "prompt processing function." This functional knowledge is what generalizes within a task type: all NLI tasks require the PLM to compare premise and hypothesis, so a projector that learned to make NLI prompts work has learned something about the intersection of NLI task structure and target PLM computation, which applies to any new NLI task.

The RoBERTa → T5 direction is the strongest test. These are not just different-sized versions of the same architecture — RoBERTa is an encoder-only masked LM, T5 is an encoder-decoder with autoregressive generation. The fact that Task Tuning projectors achieve 89.8% on MNLI and 88.2% on SNLI (Table 2a, MNLI projector) when transferring from RoBERTa-LARGE to T5-XXL demonstrates that the task knowledge in prompts survives a transition across fundamentally different computation graphs. This is not obvious — one might have expected that prompts for encoder-only and encoder-decoder models encode task knowledge in architecturally-specific ways that don't translate.

The practical significance is substantial but bounded. Cross-model transfer makes it possible to train prompts on a small, cheap model and deploy them on a large, expensive model — the TPT-MODEL speedups (e.g., 30× for SNLI on T5-XXL, Table 2b) quantify the efficiency gain. However, the limitation to same-type tasks means you need a family of projectors (one per task type) rather than a single universal projector. The paper does not solve the universal projector problem, but it establishes that type-specific projectors work and characterizes their generalization boundary — a necessary first step toward any universal solution.

Innovation 4: Efficiency Gains from Prompt Transfer Are Task-Dependent and Sometimes Dramatic

While prompt transfer improving PT efficiency is the paper's explicit goal, the structure of when and how much efficiency improves is itself a significant finding. The speedup numbers are not uniformly distributed — they cluster dramatically on certain tasks and are near-zero on others — and this pattern reveals something about why PT is slow in the first place.

Prior assumptions about PT's slowness were vague. The community knew PT was slower than fine-tuning (Figure 2), but there was no systematic understanding of which tasks suffered most or why. The implicit assumption might have been that PT is uniformly slow due to the small number of trainable parameters producing weak gradients, or that the slow convergence was inherent to the optimization landscape of prompting.

The speedup pattern suggests PT slowness is about exploration, not gradient quality. The tasks where TPT-TASK achieves the largest speedups are not random — they are tasks where a same-type source task exists with a similar reasoning structure. On T5-XXL (Table 1), QNLI achieves 24.9× convergence speedup, SNLI achieves 49.9×, and deontology achieves 29.8×. These are all tasks with clear within-type neighbors (NLI tasks for QNLI/SNLI, ethical judgment tasks for deontology) whose prompts encode relevant reasoning strategies. In contrast, tasks with poor zero-shot transfer (e.g., deontology on RoBERTa-LARGE shows 0.7× convergence speedup — a slowdown) see no benefit.

This pattern is consistent with the interpretation that vanilla PT spends most of its training time exploring the prompt embedding space to discover the right "task circuit," and transfer eliminates this exploration by providing a prompt that already activates approximately the right neurons. The TPT-TASK prompt is not perfect (or it would achieve 100% zero-shot transfer), but it's close enough that gradient descent only needs to refine it rather than find it from scratch. The dramatic speedups on some tasks (219.8× comparable-result for laptop on T5-XXL) suggest that random prompts are extremely far from the right region of embedding space for those tasks, and most of PT's training budget is wasted on moving the prompt from a random location to a functionally useful one.

The N/A speedups are equally informative. Tasks where TPT-TASK shows no comparable-result speedup (Table 1, N/A entries) are cases where the transferred prompt is so far from the target solution that it provides no initialization advantage — the optimization path from the transferred prompt is about as long as from a random prompt. This is a natural negative result that bounds the method: transfer helps when the source and target share underlying structure (same task type), and provides no benefit otherwise. There is no continuum of partial benefit for "somewhat similar" tasks — the benefit is binary, determined by the task-type boundary.

Implications for understanding PT optimization. This finding indirectly teaches us something about the loss landscape of prompt tuning: the optimization problem is not just about finding any local minimum (which would be fast regardless of initialization) but about finding a specific region of prompt space that activates the right computational circuits in the frozen PLM. Random initialization places prompts in regions that activate essentially random neural patterns, and gradient descent must cross large, potentially non-convex regions of the loss landscape to reach functionally useful prompts. Transfer effectively "teleports" the prompt to the right basin of attraction, and the remaining optimization is locally well-behaved.

Innovation 5: The Diagnostic Power of Zero-Shot Transfer as a Task-Similarity Probe

The paper's methodological innovation — using zero-shot transfer performance from Figure 3 as a retrospective task-similarity metric to select the best source task for TPT-TASK — is more than an implementation detail. It establishes a general principle: the transferability of learned representations between tasks is itself a high-quality measure of task relatedness, one that can be more informative than surface features like dataset size, domain, or label space.

Prior work on task similarity used indirect proxies. Vu et al. (2020) analyzed transferability between fine-tuning tasks using features like dataset size, task type labels, and embedding similarities of task representations. These are all prospective metrics — they try to predict transferability from properties of the tasks themselves, without running any transfer experiments. The limitation is that these proxies can miss functional similarities that matter for transfer (two apparently different tasks might share underlying reasoning, or two apparently similar tasks might require different skills).

Zero-shot transfer directly measures what matters. By evaluating a trained source prompt on the target task without any adaptation, the paper measures functional similarity — not whether the tasks "look similar" by some embedding or metadata, but whether the source prompt's encoded strategy actually works on the target. When source prompts from IMDB (a large movie review dataset) achieve 101% relative zero-shot performance on Movie Rationales (a small movie review dataset with rationale annotations), this tells us something about the relationship between these datasets that no metadata comparison would capture: large-scale sentiment training produces prompts that encode sentiment detection robustly enough to handle rationale-level annotation noise.

The task-type block structure in Figure 3 validates the metric. If zero-shot transfer were noisy or unreliable, the matrices would show no clear structure — similar tasks would sometimes transfer well and sometimes poorly based on random training variation. Instead, the block-diagonal structure is remarkably clean, with same-type clusters consistently showing high transfer and cross-type pairs consistently failing. This replicable pattern confirms that zero-shot transfer is measuring something stable about task relatedness, not just training noise.

Why this matters beyond prompt tuning. The idea that learned representations (prompts, adapters, fine-tuned weights) serve as task-probing instruments is transferable to other parameter-efficient methods. For adapter-based tuning, one could freeze an adapter trained on a source task and measure its zero-shot performance on a target task to assess task similarity. This provides a richer alternative to embedding-based task similarity metrics (e.g., Task2Vec) because it directly evaluates behavioral transfer rather than geometric similarity in some representation space.

The limitation is computational cost. Computing the full zero-shot transfer matrix (Figure 3) requires training PT on every source task and evaluating on every target task — a quadratic number of experiments. The paper's subsequent development of the ON metric (Innovation 2) can be seen as an attempt to approximate zero-shot transfer without running it, trading accuracy for computational tractability. The fact that ON achieves only ~50% rank correlation with zero-shot transfer means it's a useful but imperfect proxy — the zero-shot transfer measurement remains the gold standard for task similarity in this framework.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses 17 NLP tasks across 6 types: Sentiment Analysis (SA: IMDB, SST-2, laptop, restaurant, Movie, Tweet), Natural Language Inference (NLI: MNLI, QNLI, SNLI), Ethical Judgment (EJ: deontology, justice), Paraphrase Identification (PI: QQP, MRPC), Question Answering (QA: SQuAD, NQ-Open), and Summarization (SUM: Multi-News, SAMSum). Details for each dataset — including sizes, sources, and label spaces — are provided in Appendix A.1, A.2, and A.4. For classification tasks (SA, NLI, EJ, PI), accuracy is the evaluation metric; for generation tasks (QA, SUM), F1 and ROUGE-L are used respectively (Appendix A.2).

• Base model(s). Experiments span two PLM families: RoBERTa (encoder-only, masked language modeling pre-training) and T5 (encoder-decoder, sequence-to-sequence pre-training). The main reported results use the largest variants — RoBERTa-LARGE and T5-XXL — with additional results for smaller sizes (RoBERTa-BASE, T5-Small, T5-Base) in the appendix. The paper argues PT is most promising on extremely large PLMs where it matches fine-tuning performance (Lester et al., 2021), making these large variants the natural testbed. For cross-model transfer, the primary direction studied is RoBERTa-LARGE → T5-XXL, described as "the most difficult setting" due to their heterogeneous architectures (Section 5).

• Metrics. Three categories of metrics are reported: (a) Performance: zero-shot transfer accuracy (absolute and as a percentage of vanilla PT performance), final accuracy after TPT-TASK or TPT-MODEL training; (b) Speedup: convergence speedup (ratio of vanilla PT convergence time to transfer method convergence time) and comparable-result speedup (ratio of vanilla PT convergence time to the time TPT takes to achieve vanilla PT's converged performance), both defined in Appendix B.3; (c) Correlation: Spearman's rank correlation between prompt similarity metrics (embedding-based and ON) and zero-shot transfer performance, used in Section 6 to evaluate transferability indicators.

• Baselines. For all experiments, the primary baseline is vanilla prompt tuning (PT) — randomly initialized soft prompts trained with standard PT as described in Section 3.1. For zero-shot transfer, an additional baseline is randomly generated prompts evaluated without training (the "random prompt" rows in Figure 3 and Table 2), which establishes the performance floor. For cross-model transfer, the Distance Minimizing projector serves as a baseline against the Task Tuning projector (Table 2a). The paper does not compare against full-parameter fine-tuning as a baseline for the transfer methods, though Figure 2 uses fine-tuning to motivate PT's slowness problem.

• Generation budget / compute accounting. The primary "currency" is training time measured in steps (with loss and evaluation computed every 100 steps), not FLOPs or wall-clock time in a standardized hardware environment. Convergence is defined as the point where training loss stops dropping and evaluation score stops increasing for 300 consecutive steps (Appendix B.3). This means the speedup comparisons in Tables 1 and 2 reflect steps-to-convergence ratios under the specific hyperparameter settings used (learning rate 0.001, AdamW optimizer, prompt length l=100), and may not generalize to different optimization configurations. The paper does not account for the computational cost of training source prompts, projectors, or computing the zero-shot transfer matrix — only the target task training time is considered in speedup calculations.

• Cross-validation / statistical protocol. No formal cross-validation is reported for the main results. The zero-shot transfer matrices (Figure 3) are single-run evaluations — each cell represents one source prompt trained to convergence and evaluated once on the target task. For the ONI variant (Appendix D.3), three prompts are trained with different random seeds and their activation intersections are used, but this is for metric computation rather than statistical significance testing. The paper does not report confidence intervals, standard deviations across runs, or statistical tests comparing transfer methods to baselines. The 500-question MATH test set used in the reference example's paper is not present here; this paper uses standard NLP benchmark test sets with their default splits.

Main Quantitative Results

Cross-Task Zero-Shot Transfer Performance (Figure 3, Appendix B.1)

The zero-shot transfer matrices in Figure 3 display the relative performance (zero-shot accuracy / vanilla PT accuracy × 100%) of trained soft prompts from source tasks (rows) when evaluated directly on target tasks (columns) with no additional training.

Same-type transfer consistently succeeds, occasionally exceeding vanilla PT. On RoBERTa-LARGE (Figure 3a), sentiment analysis source prompts transfer to other SA targets at 65–101% relative performance. The strongest case is IMDB → Movie, achieving 101% — meaning the IMDB-trained prompt actually outperforms the prompt trained specifically on Movie Rationales when evaluated zero-shot on Movie. The paper attributes this to IMDB having substantially more training data (25,000 reviews vs. Movie's 1,600), so the source prompt encodes richer sentiment knowledge. Other notable same-type transfers include SST-2 → IMDB at 91%, laptop → restaurant at 93%, and MNLI → QNLI at 79%. On T5-XXL (Figure 3b), the pattern holds: IMDB → Movie achieves 100%, SST-2 → IMDB achieves 84%, MNLI → QNLI achieves 81%, and MNLI → SNLI achieves 98% — the NLI-to-NLI transfer is near-perfect.

Cross-type transfer collapses to near-random. On RoBERTa-LARGE, transferring from sentiment analysis to NLI yields 37–55% relative performance, close to the random prompt baseline of 36–52%. Transferring from NLI to sentiment analysis shows 53–75%, slightly better but still far below same-type transfer. On T5-XXL, the pattern is even starker: IMDB → MNLI achieves 36% relative, MNLI → IMDB achieves 65%, and most cross-type pairs cluster around the random baseline values. The paper explicitly tests whether this is caused by different label tokens (e.g., "positive"/"negative" for SA vs. "yes"/"no" for NLI) by unifying all labels to numbers (Appendix B.2, Figure 6) and finds that "the transferability between different-type tasks are generally not improved," concluding that "different-type tasks surely require distinct abilities."

Edge cases show partial cross-type transfer. Some tasks partially bridge type boundaries. On T5-XXL (Figure 3b), QA and summarization tasks show non-trivial transfer to sentiment analysis: SQuAD → Tweet achieves 73%, Multi-News → laptop achieves 76%. The paper does not deeply analyze these exceptions but notes they exist, suggesting that task-type boundaries are not perfectly impermeable — some tasks may share partial reasoning overlap (e.g., summarization requiring comprehension that aids sentiment detection).

Scale effects are visible. Comparing RoBERTa-BASE and RoBERTa-LARGE zero-shot matrices (Appendix B.1, Figure 5c and 5d), the block-diagonal structure is present at both scales, but the absolute transfer percentages differ. T5-Small and T5-Base (Figures 5a, 5b) show similar patterns to T5-XXL, suggesting the task-type dependency of transfer is robust across model sizes within each family — it is a property of the tasks, not an artifact of model scale.

Cross-Task Transfer with Initialization (TPT-TASK): Performance and Speedup (Table 1)

Table 1 reports three metrics for each of 17 tasks on RoBERTa-LARGE and T5-XXL: vanilla PT performance, TPT-TASK performance, convergence speedup, and comparable-result speedup.

TPT-TASK generally matches or improves final performance. On RoBERTa-LARGE, TPT-TASK achieves better performance than vanilla PT on 9 of 13 tasks, with the largest improvements on laptop (76.4% → 79.1%, +2.7 points), restaurant (83.7% → 85.8%, +2.1 points), and MNLI (87.3% → 87.9%, +0.6 points). Degradations are small: QQP drops from 88.9% to 86.1% (−2.8 points), and MRPC drops from 81.2% to 79.2% (−2.0 points). On T5-XXL, TPT-TASK improves 11 of 17 tasks, with laptop showing the largest gain (76.6% → 84.2%, +7.6 points), followed by deontology (87.0% → 95.3%, +8.3 points) and justice (92.5% → 90.7%, −1.8 points — a small degradation). The paper does not explain why certain tasks degrade; QQP on RoBERTa is the most notable negative case.

Convergence speedups are task-dependent and sometimes dramatic. On RoBERTa-LARGE, speedups range from 0.7× (deontology — a slowdown) to 1.9× (restaurant). Most tasks cluster in the 0.9–1.3× range — modest but real improvements. On T5-XXL, however, the speedups are substantially larger and more variable: SST-2 achieves 49.7× convergence speedup, SNLI achieves 49.9×, deontology achieves 29.8×, QNLI achieves 24.9×, and MNLI achieves 12.5×. The paper does not directly explain why T5-XXL shows such larger speedups than RoBERTa-LARGE, but the pattern is consistent with larger models having more severe PT convergence problems that transfer can address — the random initialization is further from useful prompt regions in T5-XXL's larger embedding space.

Comparable-result speedups are even larger where computable. On T5-XXL, laptop achieves 219.8× comparable-result speedup — meaning TPT-TASK reaches vanilla PT's converged performance in roughly 1/220th the training time. Other large speedups include SST-2 (48.9×), MNLI (12.5×), SNLI (49.9×), and deontology (29.9×). These are substantially larger than convergence speedups for the same tasks, because TPT-TASK often starts at a performance level that vanilla PT takes most of its training budget to reach. The N/A entries for comparable-result speedup indicate tasks where TPT-TASK either never reaches vanilla PT's converged performance within the training budget (e.g., QQP on RoBERTa-LARGE, where TPT-TASK's final accuracy is lower) or where PT's convergence time is so short that no meaningful speedup can be measured (e.g., SST-2 on RoBERTa-LARGE at 1.1× convergence speedup).

The speedup patterns align with the zero-shot transfer patterns. Tasks with the highest zero-shot transfer from some source task also show the largest TPT-TASK speedups. For instance, SST-2 on T5-XXL had 96% relative zero-shot transfer from IMDB (Figure 3b) and achieves 49.7× convergence speedup (Table 1). This alignment validates the core premise: good zero-shot transfer predicts good initialization benefit.

Cross-Model Zero-Shot Transfer Performance (Table 2a, Table 7)

Table 2a reports zero-shot transfer performance for prompts transferred from RoBERTa-LARGE to T5-XXL using projectors trained with two objectives (Distance Minimizing and Task Tuning), evaluated on 13 tasks (SA, NLI, EJ, PI). Table 7 (Appendix C.2) extends this to other model pairs: BERT-BASE → RoBERTa-BASE, RoBERTa-BASE → RoBERTa-LARGE, and T5-Base → T5-XXL.

Distance Minimizing works only on the projector-training task. On the RoBERTa-LARGE → T5-XXL transfer (Table 2a), the Distance Minimizing projector trained on laptop achieves 76.6% on laptop itself — exactly matching vanilla PT on T5-XXL — confirming the projector successfully reproduces the target prompt for its training task. However, on all other tasks, performance collapses to or below the random baseline: IMDB gets 49.6% (random: 49.7%), SST-2 gets 49.0% (random: 49.0%), MNLI gets 31.8% (random: 31.8%). The same pattern holds for the MNLI-trained Distance Minimizing projector: 90.5% on MNLI (matching vanilla PT), but random-level performance everywhere else. This is consistent across all model pairs in Table 7: Distance Minimizing is a dramatic failure at generalization.

Task Tuning projectors generalize to same-type unseen tasks. The laptop-trained Task Tuning projector (Table 2a) achieves 82.9% on IMDB, 89.3% on SST-2, 80.3% on laptop, 85.7% on restaurant, and 78.6% on Movie — all sentiment analysis tasks — substantially above the random baselines (~49–51%). However, it fails on NLI tasks: 32.4% on MNLI, 50.7% on QNLI, 33.6% on SNLI — near random. The MNLI-trained Task Tuning projector shows the reverse: 89.8% on MNLI, 82.7% on QNLI, 88.2% on SNLI (all NLI), but only 49.7% on IMDB and 48.8% on SST-2 (random-level). This task-type-specific generalization is consistent across all model pairs in Table 7.

Cross-architecture (RoBERTa → T5) transfer works within task types. Even though RoBERTa-LARGE and T5-XXL use different architectures (encoder-only vs. encoder-decoder), the Task Tuning projectors successfully transfer prompts between them for same-type tasks. On the MNLI projector, MNLI achieves 89.8%, QNLI achieves 82.7%, and SNLI achieves 88.2% (Table 2a). This is the critical finding: task knowledge encoded in prompts transcends architectural differences, at least within the same task family. The paper does not test the reverse direction (T5 → RoBERTa).

Scale effects in cross-model transfer (Table 7). When transferring between models of the same architecture family (RoBERTa-BASE → RoBERTa-LARGE), Task Tuning projectors show the same pattern: the IMDB/laptop projector achieves 90.4% on IMDB and 76.2% on SST-2 (both SA), but 33.3% on MNLI (near random). The MNLI projector achieves 87.1% on MNLI and 79.4% on QNLI (both NLI), but 67.7% on IMDB (above random but well below same-type transfer). The BERT → RoBERTa transfer (cross-architecture, both encoder-only) shows weaker absolute numbers but the same qualitative pattern.

Cross-Model Transfer with Initialization (TPT-MODEL): Performance and Speedup (Table 2b)

Table 2b reports TPT-MODEL results for RoBERTa-LARGE → T5-XXL using Task Tuning projectors (laptop-trained and MNLI-trained). It shows performance, convergence speedup, and comparable-result speedup for each of 13 tasks.

TPT-MODEL performance is strong for same-type tasks. With the laptop projector, sentiment analysis tasks achieve: IMDB 96.5% (matching vanilla PT at 96.5%), SST-2 97.4% (matching 97.4%), laptop 82.9% (substantially above vanilla PT's 76.6%), restaurant 90.3% (above 88.1%), Movie 97.4% (near 97.9%). For different-type tasks, performance is mixed: MNLI achieves 91.0% (slightly above vanilla PT's 90.5%), but deontology achieves 92.5% (matching vanilla PT) while QQP achieves 90.0% (matching vanilla PT). Some cross-type tasks unexpectedly benefit: the laptop projector helps MNLI slightly (90.5% → 91.0%) and the MNLI projector helps deontology (87.0% → 94.6%) and justice (92.5% → 92.5%). The paper does not analyze why these cross-type improvements occur.

Speedups are substantial within task types. For the laptop projector: SST-2 achieves 1.7× convergence speedup and 19.0× comparable-result speedup; laptop achieves 1.9× and 16.0×; restaurant achieves 1.3× and 6.0×. For the MNLI projector: SNLI achieves 1.4× convergence speedup and 30.0× comparable-result speedup; QNLI achieves 1.1× and 20.0×; MNLI achieves 1.0× and 18.0×; deontology achieves 2.0× and 7.5×. Several tasks show convergence speedup below 1.0× (slowdowns): Movie with laptop projector (0.6×), Movie with MNLI projector (0.4×), and a few others at 0.9×.

Comparison to TPT-TASK on the same model. TPT-MODEL speedups on T5-XXL are generally more modest than TPT-TASK speedups on the same model (Table 1 vs. Table 2b). For example, TPT-TASK achieves 49.7× convergence speedup on SST-2, while TPT-MODEL achieves 1.7×. However, this comparison is somewhat unfair: TPT-TASK uses prompts trained on the same T5-XXL model from a similar task, while TPT-MODEL uses prompts trained on a different, smaller model (RoBERTa-LARGE). The fairer comparison is TPT-MODEL versus training from scratch on T5-XXL, which still shows speedups — just smaller ones than same-task cross-model transfer.

LayerNorm is necessary for small-model transfer (Appendix C.3, Figure 7). When transferring between small models (BERT-BASE → RoBERTa-BASE), the projected prompts without LayerNorm exhibit "quite different L2 norm values" and are "hard to optimize" — Figure 7 shows training loss divergence on MNLI, IMDB, and restaurant. Adding LayerNorm to the projector output stabilizes training and enables normal convergence. Interestingly, Table 8 shows that zero-shot transfer performance is similar with and without LayerNorm, meaning the normalization helps during subsequent fine-tuning (TPT-MODEL phase) rather than improving the initial projection quality.

Prompt Similarity Metrics as Transferability Indicators (Tables 3, 4, 5; Figure 4)

Section 6 evaluates five prompt similarity metrics — E_concat, E_average, C_concat, C_average, and ON — on two criteria: (1) whether they distinguish same-task from different-task prompts (Table 3), and (2) how well they correlate with zero-shot transfer performance (Table 4).

All metrics distinguish same-task from different-task prompts (Table 3). On RoBERTa-LARGE, same-task prompt pairs (trained with different random seeds) achieve: 41.6% E_average, 47.6% C_concat, 39.4% ON (Top 3 layers). Cross-type different-task prompts achieve lower values: 37.6%, 31.7%, and 21.4% respectively. The gap is consistently in the right direction for all metrics. On T5-XXL, the pattern holds: ON (Top 3) achieves 62.0% for same-task vs. 46.1% for different-type tasks. This confirms that trained soft prompts form "distinguishable clusters in the embedding space and also stimulate different abilities within the PLM" (Section 6.2).

ON outperforms embedding-based metrics for correlation with transfer performance (Table 4). On RoBERTa-LARGE, ON achieves a Spearman's rank correlation of 49.7% with zero-shot transfer performance — the highest of all metrics. The best embedding metric is C_average at 44.7%, followed by C_concat at 24.8% and E_concat at 22.6%. E_average performs worst at 2.8% — essentially uncorrelated. On T5-XXL, ON achieves 36.9%, C_average achieves 33.5%, C_concat achieves 31.6%, and E_average achieves −2.5% (a slight negative correlation). The key claim — that model stimulation matters more than embedding geometry — is supported by ON's consistent edge over embedding metrics.

ON degrades with model scale while C_average is more stable (Figure 4). The Spearman's correlation of ON drops from 52.5% on T5-Small to 49.1% on T5-Base to 36.9% on T5-XXL. C_average drops more slowly: 37.1% → 42.4% → 33.5%. The paper attributes ON's degradation to model redundancy: "larger PLMs have higher redundancy... prompts can activate different redundant neurons to do similar jobs" (Appendix D.3), causing functionally equivalent prompts to appear dissimilar to ON. C_average is less affected because redundancy in embedding space is less severe than in activation space.

The ONI variant partially recovers correlation (Appendix D.3, Table 10). ONI (intersection of activation states from three prompts trained with different seeds on the same task) raises T5-XXL's correlation from 36.9% to 46.3% for Top-3 layers, confirming that redundancy is the cause of the degradation. Taking the intersection filters out randomly varying (redundant) neurons and keeps only consistently activated (functionally important) ones, yielding a cleaner signal.

ON also works for cross-model transfer (Table 5). For the laptop Task Tuning projector, ON assigns 52.4% similarity between the projected laptop prompt and the original T5-XXL-trained laptop prompt, and averages 51.0% for same-type tasks — correctly identifying that the projector preserves task-relevant neuron activation patterns for its task family. For different-type tasks, the average drops to 46.0%. In contrast, C_average assigns 3.8% for laptop, 4.1% for same-type tasks, and 3.4% for different-type tasks — uniformly low, failing to distinguish the functional similarity within the task type. This is arguably the strongest evidence for ON's superiority: embedding geometry sees no similarity where functionally the prompts do the same thing on the target model.

Higher layers are more task-specific (Appendix D.4, Figures 8–12). Visualizations of ON across different layer ranges on RoBERTa-BASE show that bottom layers (1–3) have generally high overlap across all task types — these layers encode shared, task-agnostic processing. Top layers (10–12) show sharper distinctions: same-type tasks cluster with high overlap while cross-type tasks show low overlap. This is consistent with probing literature (Liu et al., 2019a) showing that higher layers encode more task-specific representations.

Ablation Studies and Robustness Checks

Label token unification (Appendix B.2, Figure 6): To test whether poor cross-type transfer is caused by different label tokens (e.g., SA uses "positive"/"negative", NLI uses "yes"/"no"), all label tokens are unified to numbers (1, 2, ...) on RoBERTa-BASE. The zero-shot transfer matrix (Figure 6b) shows that "the transferability between different-type tasks are generally not improved" — the block-diagonal structure persists and cross-type transfer remains near random. This rules out the most obvious superficial explanation for the task-type boundary.

Projector training objective (Table 2a, Table 7): The comparison between Distance Minimizing and Task Tuning projectors is replicated across four model pairs (Table 7: BERT-BASE → RoBERTa-BASE, RoBERTa-BASE → RoBERTa-LARGE, T5-Base → T5-XXL, and the main RoBERTa-LARGE → T5-XXL). In all settings, Distance Minimizing generalizes to zero unseen tasks while Task Tuning generalizes within task types. This robustness across architectures confirms that functional (Task Tuning) rather than geometric (Distance Minimizing) projection is the mechanism enabling cross-model transfer.

LayerNorm in projector (Appendix C.3, Figures 7, Table 8): Removing LayerNorm from the projector causes optimization instability in TPT-MODEL on small models (BERT-BASE → RoBERTa-BASE): training loss diverges on MNLI, IMDB, and restaurant (Figure 7, Without LayerNorm panels). Adding LayerNorm stabilizes training (Figure 7, With LayerNorm panels). However, zero-shot transfer performance is nearly identical between the two variants (Table 8), indicating LayerNorm helps during subsequent fine-tuning by regularizing the norm of projected prompts, not by improving the initial projection quality.

Activation layer selection for ON (Appendix D.4, Figures 8–12, Table 9): ON is computed using only the Top 3 layers in the main experiments. Table 9 compares Top 3, Bottom 3, and All 24 layers on RoBERTa-LARGE and T5-XXL. On RoBERTa-LARGE, Top 3 achieves 39.4% same-task similarity vs. 21.4% cross-type, while Bottom 3 achieves 42.8% vs. 39.1% — the distinction is much weaker in bottom layers. All 24 layers (40.0% vs. 29.6%) sits between. On T5-XXL, Top 3 achieves 62.0% vs. 46.1%, while Bottom 3 achieves 80.3% vs. 76.3% — bottom layers show near-uniform activation across tasks. The paper concludes that higher layers are more task-specific, justifying the Top 3 choice. The ON correlation with transfer performance shows a similar pattern: Top 3 achieves the highest Spearman's correlation on both RoBERTa (49.7% vs. 34.3% for Bottom 3, Table 11) and T5-XXL (36.9% vs. 12.4% for Bottom 3, Table 10).

ONI: intersection-based activation (Appendix D.3, Table 10): Taking the intersection of activation states from three prompts trained with different seeds on the same task raises ON's Spearman's correlation on T5-XXL from 36.9% to 46.3% (Top 3 layers). This ablation directly tests the redundancy hypothesis: if functionally equivalent prompts activate different redundant neurons in larger models, intersecting across seeds should filter out redundancy and recover a cleaner signal — which it does, though not to the level of T5-Small's 52.5%. The paper frames this as a "preliminary trial" and encourages future work on overcoming PLM redundancy for better transferable PT.

Non-linear activation choice in projector (Appendix C.1): The paper states they tried both tanh and LeakyReLU as the non-linear activation σ in the projector and found "their performance on various PLMs are similar." Reported results use LeakyReLU. This is a minor robustness check suggesting the projector is not sensitive to activation function choice.

Model scale variations for zero-shot transfer (Appendix B.1, Figure 5): Zero-shot transfer matrices are reported for RoBERTa-BASE, RoBERTa-LARGE, T5-Small, T5-Base, and T5-XXL. The block-diagonal structure and task-type dependency are consistent across all scales, confirming that the finding is robust to model size within each family.

Cross-model transfer across different scales and architectures (Table 7): The Task Tuning projector's within-type generalization is tested across BERT-BASE → RoBERTa-BASE, RoBERTa-BASE → RoBERTa-LARGE, T5-Base → T5-XXL, and RoBERTa-LARGE → T5-XXL. All show the same qualitative pattern, confirming robustness across (a) within-family scale increases, (b) cross-architecture transfers between encoder-only models, and (c) cross-architecture transfers between encoder-only and encoder-decoder models.

Critical Assessment

The paper's central claims — that prompt transfer is feasible and useful, that transferability is governed by task type, that neuron activation overlap predicts transfer success, and that transfer can dramatically accelerate PT — are largely supported by the reported experiments, but with important qualifications and gaps that temper the strength of the conclusions.

Claim: "Trained soft prompts can effectively transfer to similar tasks on the same PLM." The evidence from Figure 3 is consistent and clear: same-type tasks achieve 65–101% relative zero-shot transfer on RoBERTa-LARGE and similar ranges on T5 variants. However, "effectively" is doing substantial work here. A prompt that achieves 65% relative zero-shot transfer on a target task (e.g., laptop → SST-2 on RoBERTa-LARGE at 88% relative, meaning the source prompt gets ~85% accuracy vs. ~96% for vanilla PT) is leaving a non-trivial performance gap. The paper focuses on cases exceeding 90% and on the TPT-TASK speedups, but several same-type transfers are mediocre (e.g., QNLI → MNLI at 79% on RoBERTa-LARGE). "Effectively transfer" should be qualified: same-type transfer works competitively in most cases and excellently in some (especially when the source task has more data), but there are same-type pairs where transfer provides only partial benefit.

Claim: "Trained soft prompts can transfer to other PLMs with a cross-model projector trained on similar tasks." Supported by Table 2a for the within-type cases, but the claim's "on similar tasks" qualifier is doing all the work. The projector does not transfer to different-type tasks — the paper is honest about this limitation. What's missing is an analysis of how many projector-training tasks are needed for practical coverage. With six task types, an organization would need to train six projectors (or at least one per type of interest), and the paper only demonstrates two (laptop for SA, MNLI for NLI). Whether each task type yields a projector that generalizes well within that type is an extrapolation from two data points — the pattern is consistent but the sample is small.

Claim: "When used as initialization, trained soft prompts can significantly accelerate training and also improve performance." The speedup numbers in Table 1 and Table 2b are the headline evidence, but several methodological concerns affect how "significant" these speedups are:

Speedup is measured in steps, not wall-clock time in a controlled environment. The paper defines convergence as the point where loss and evaluation metrics stop improving for 300 steps (with evaluation every 100 steps). This is a reasonable definition but is sensitive to the learning rate, batch size, and other hyperparameters. Faster methods might also be found by tuning PT's learning rate (the paper uses 0.001 uniformly) rather than using transfer — there is no learning rate sweep for vanilla PT to establish that transfer helps beyond what hyperparameter optimization would achieve.

The dramatic speedups on T5-XXL (49.7× for SST-2, 49.9× for SNLI) are not independently replicated. SST-2 is a small dataset with a 97.4% accuracy ceiling; achieving 49.7× convergence speedup might partly reflect that vanilla PT is extraordinarily slow to converge on this task rather than that TPT-TASK is extraordinarily fast in absolute terms. The paper does not report absolute training times in any standardized unit — only ratios. A 49.7× speedup means little if vanilla PT's convergence time is unreasonably long due to suboptimal hyperparameters.

Comparable-result speedup of 219.8× for laptop on T5-XXL is a single data point that strains credulity. Even if TPT-TASK starts from a well-initialized prompt that already achieves near-PT performance, achieving vanilla PT's converged accuracy in 1/220th of the training time implies vanilla PT spends ~219/220 of its training budget doing essentially nothing useful — a claim that deserves more investigation than the paper provides. This could be accurate (if random initialization places prompts in a very flat region of the loss landscape far from any useful basin), but without loss landscapes or optimization trajectory analysis, the reader cannot distinguish genuine initialization benefit from artifact of the convergence detection criterion.

Tasks with slowdowns (N/A or <1.0× speedup) are not analyzed. QQP on RoBERTa-LARGE shows 0.8× convergence speedup — TPT-TASK is slower than vanilla PT. deontology shows 0.7×. The paper reports these numbers without explanation, missing an opportunity to characterize when transfer hurts — a characterization as valuable as when it helps for practitioners deciding whether to use the method.

Claim: "The overlapping rate of activated neurons strongly reflects the transferability." The evidence is that ON achieves the highest Spearman's correlation among tested metrics on RoBERTa-LARGE (49.7%, Table 4) and outperforms embedding metrics. Several qualifications apply:

49.7% correlation means ON explains roughly 25% of the variance (R² ≈ 0.25). This is "better than the alternatives" but far from "strongly reflects" — ON is a weak-to-moderate predictor of transfer success, not a strong one. The paper's language overstates the strength of the correlation.

The correlation degrades on the largest model (36.9% on T5-XXL) — precisely where practical transfer is most needed. If ON works best on smaller models where PT is already reasonably fast, and works worse on larger models where PT's slowness is most problematic, its practical utility as a transferability indicator is limited. ONI partially recovers the signal (46.3%), but requires training three prompts per task — substantial additional computation that partially defeats the purpose of using a cheap metric instead of running actual transfer experiments.

ON is compared only against relatively naive embedding metrics. The paper does not compare ON against more sophisticated task-similarity methods like Task2Vec, CKA (centered kernel alignment), or other representation-similarity metrics that were available at the time. The claim is "ON beats Euclidean and cosine distance on prompt embeddings," not "ON beats the state of the art in task-similarity estimation." The ablation is clean but the comparator set is weak.

ON is computed with an artificial input ({P, <s>} without actual task input). This means ON measures how the prompt stimulates the PLM in the absence of actual task data — it captures the prompt's baseline activation pattern, not how the prompt interacts with real inputs. Transfer performance, however, depends on prompt-input interaction — a prompt that seems similar based on empty-input activation might diverge when processing actual sentences. The paper does not test whether computing ON with real task inputs (or averaged across several inputs) improves correlation.

Missing experiments that would strengthen the paper:

No comparison between TPT-TASK and simply training for longer with a better learning rate schedule. If vanilla PT's slowness is due to suboptimal optimization rather than fundamental difficulty of the loss landscape, better hyperparameter tuning could close the gap without transfer. The paper's fixed learning rate (0.001) and convergence criterion (300 steps without improvement) may prematurely declare convergence for PT while TPT-TASK benefits from the same criterion being applied to a better-initialized optimizer state.

No analysis of why some same-type transfers work better than others. IMDB → SST-2 achieves 96% relative transfer on T5-XXL (Figure 3b), while SST-2 → IMDB achieves 84% — a notable asymmetry. The paper attributes this to IMDB having more data, but doesn't systematically analyze whether dataset size, domain similarity, or other factors predict within-type transfer variation. Such an analysis would make the "prompt warehouse" retrieval idea more actionable.

No test of whether TPT-TASK benefits transfer to genuinely new tasks not in the 17-task suite. All tasks are known benchmark datasets; it's unclear whether a prompt trained on a small set of task types would benefit a truly novel task type.

No FLOPs-matched comparison of cross-model transfer vs. training directly on the target model. TPT-MODEL claims efficiency gains (Table 2b), but the cost of training the source prompt on the source model plus the projector training cost is not factored into the speedup calculation. A fair accounting would compare: (a) train prompt on source model + train projector + TPT-MODEL on target vs. (b) just train PT on target from scratch. The current speedup numbers only measure step (a)'s final phase.

The test sets are standard NLP benchmark sizes. For tasks like SST-2 (1,821 test examples), MNLI (9,815 test examples), and QQP (404,351 test examples — though the paper likely uses a subset), statistical significance of performance differences could be computed but is not reported. A 1–2 point accuracy difference on a 1,821-example test set may or may not be statistically significant, and the paper treats all such differences as real without qualification.

In summary, the experiments genuinely support the qualitative conclusion that prompt transfer is task-type-dependent and that neuron activation overlap is more informative than embedding distance for predicting transfer. The quantitative speedup claims should be treated as upper bounds measured under specific optimization settings, not as guaranteed efficiency gains for practitioners. The paper's strongest empirical contribution is the zero-shot transfer matrix (Figure 3) and its task-type block structure — this is replicated across five model variants (Appendix B.1) and is the foundation upon which all subsequent transfer methods are built. The weakest parts are the large speedup numbers on T5-XXL, which lack sufficient context (absolute training times, hyperparameter sensitivity, cost accounting) to be interpreted as practical deployment guidance.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Accounted For

The assumption or constraint. The entire cross-task transfer framework depends on knowing which source task's prompt will transfer best to a target task before training begins. The paper operationalizes this by pre-computing the full zero-shot transfer matrix (Figure 3) — training PT to convergence on every source task and evaluating zero-shot on every target task — and then selecting the best source task retrospectively for TPT-TASK. The authors acknowledge in Section 4.2 that TPT-TASK initializes "with trained prompts of the source task achieving the best zero-shot transfer performance in Figure 3," but they do not factor the cost of computing Figure 3 into any speedup calculation. For the 13 RoBERTa-LARGE tasks, this requires 13 full PT training runs (one per source task) plus 13 × 13 = 169 zero-shot evaluations. For T5-XXL with 17 tasks, the cost scales to 17 training runs plus 289 evaluations.

The consequence. The headline speedup numbers in Table 1 (e.g., 49.7× convergence speedup for SST-2 on T5-XXL) are computed assuming the best source task is already known — they measure only the target-task training time, completely excluding the cost of discovering which source task to use. In a realistic deployment where no zero-shot transfer matrix pre-exists, a practitioner faces a fundamental chicken-and-egg problem: to get the speedup, they must first spend enormous compute determining which source task to transfer from. The total cost (source task discovery + target task training) could easily exceed the cost of simply running vanilla PT on the target task from scratch, especially for tasks where the speedup is modest (e.g., 1.0–1.2× on many RoBERTa-LARGE tasks). The paper's claim that prompt transfer "can significantly accelerate training" (Section 1) is therefore true only in the amortized setting where the zero-shot transfer matrix has already been built and can be reused across many new target tasks — a setting the paper does not evaluate.

What evidence exists in the paper. The paper provides no cost accounting for computing Figure 3. The speedup calculations in Appendix B.3 define convergence speedup purely as the ratio of vanilla PT convergence time to TPT-TASK convergence time on the target task, with no mention of source task identification cost. Table 1 reports speedups without any caveat about how the best source task was selected. The paper does not report absolute training times in any standardized unit (wall-clock hours, GPU-hours, or FLOPs), making it impossible for a reader to estimate whether the speedup offsets the discovery cost even in an amortized setting.

Mitigation status. The paper partially addresses this gap through Section 6's development of prompt similarity metrics (ON, C_average, etc.) as prospective transferability indicators that could replace expensive zero-shot transfer experiments. If ON achieved near-perfect correlation with zero-shot transfer, one could simply compute ON between the target task's randomly initialized prompts and a warehouse of trained source prompts, selecting the most similar source without running any transfer. However, the correlation is far from perfect — ON achieves only 49.7% Spearman's rank correlation on RoBERTa-LARGE and 36.9% on T5-XXL (Table 4) — meaning similarity-based source selection would often pick suboptimal sources and yield smaller speedups than the oracle selection used in Tables 1 and 2. The paper does not evaluate TPT-TASK using similarity-metric-based source selection; all reported TPT-TASK results use oracle selection from Figure 3. This is a significant gap between the method as evaluated and the method as deployable.

---

The Hardest Transfer Direction (Cross-Type Tasks) Remains Essentially Unsolved

The assumption or constraint. The paper's transfer framework succeeds almost exclusively when source and target tasks are of the same type (e.g., both sentiment analysis, both NLI). Cross-type transfer — from one task family to a fundamentally different one — consistently fails across all settings. In zero-shot cross-task transfer (Figure 3), cross-type pairs achieve near-random performance: IMDB → MNLI gets 37% relative on RoBERTa-LARGE and 36% on T5-XXL, indistinguishable from the random prompt baselines of 36–52%. In cross-model transfer, Task Tuning projectors "still cannot work for different-type tasks" (Section 5.2), with laptop-trained projectors scoring 32.4% on MNLI — again near the random baseline of 31.8%. The authors state explicitly: "different-type tasks surely require distinct abilities, which prohibits reusing prompts between them" (Appendix B.2).

The consequence. This is not a minor edge case — it is a fundamental capability bound that limits the practical scope of prompt transfer. The paper studies 6 task types and shows transfer works within ~2 of them (SA and NLI, with weaker evidence for EJ and PI). For a practitioner with a novel task that does not cleanly fit into any of the existing task type categories — or for whom no source prompt exists within the same type — the paper's methods offer no benefit whatsoever. TPT-TASK would select a source task that provides essentially random initialization, yielding no speedup or even a slowdown (Table 1 shows 0.7–0.9× convergence speedup on several cross-type-adjacent pairs, such as deontology on RoBERTa-LARGE). The paper does not provide a method for making cross-type transfer work; it only characterizes its failure. This means the approach cannot help with genuinely new task types that lack similar predecessors — precisely the setting where accelerating PT would be most valuable, since there is no existing prompt library to draw from.

What evidence exists in the paper. Every experiment that disaggregates by task type shows the same pattern. Figure 3 displays the block-diagonal structure of the zero-shot transfer matrix across five model variants (Appendix B.1), with same-type clusters achieving 65–101% relative transfer and cross-type pairs at 36–55%. Table 2a shows Task Tuning projector generalization collapsing on different-type tasks (32.4–50.7% for sentiment analysis projectors on NLI tasks). The label token unification experiment (Appendix B.2, Figure 6) explicitly rules out the obvious alternative explanation — that different label vocabularies cause the failure — and confirms that "different-type tasks surely require distinct abilities." Table 5 shows ON metric correctly identifying low similarity between projected prompts and original prompts for cross-type tasks (46.0% vs. 51.0% for same-type). The evidence for this limitation is comprehensive and internally consistent across all experimental axes.

Mitigation status. The paper does not attempt to solve cross-type transfer and does not propose a direction for doing so beyond the general statement that "further research shall focus more on prompts' stimulation to PLMs" (Section 1). The failure appears to be fundamental — prompts encode type-specific reasoning strategies that do not compose across task families — and the paper provides no evidence that better projectors, different initialization schemes, or different training objectives could overcome it. This is a hard limitation that a practitioner must accept: prompt transfer only helps when you already have a prompt for a similar-enough task.

---

All Results Are on a Single Model Family (RoBERTa/T5) with 17 Specific NLP Benchmarks

The assumption or constraint. The paper's entire empirical analysis is conducted on two PLM families — RoBERTa (encoder-only) and T5 (encoder-decoder) — using 17 specific NLP benchmark datasets across 6 task types. The authors do not evaluate any decoder-only models (GPT family), any models trained with different objectives (e.g., ELECTRA, BART beyond T5), or any tasks outside the chosen 17. The paper states that the selection "represent[s] two mainstream pre-training types" (Section 3.3), but this is an assertion of representativeness rather than an empirically validated claim of generality.

The consequence. Several of the paper's specific findings may not transfer to other model families or task types, and the paper provides no evidence to bound the generalization. The ON metric's dependence on feed-forward neuron activations assumes the transformer architecture has interpretable FFN neurons — a property demonstrated for BERT/RoBERTa/T5 (Geva et al., 2021; Dai et al., 2021) but not necessarily present in models with different architectures (e.g., models with GLU variants like SwiGLU used in LLaMA, or mixture-of-experts layers where neuron semantics may differ). The finding that PT takes much longer than fine-tuning to converge (Figure 2) is shown only for RoBERTa-LARGE on MNLI — decoder-only models with autoregressive objectives might exhibit different convergence dynamics, as might models with different parameter counts or pre-training data distributions. The specific speedup magnitudes (e.g., 49.7× on SST-2 for T5-XXL) are likely sensitive to model scale, architecture, and the interaction between prompt length (fixed at 100 tokens) and model dimension. A practitioner using a LLaMA-family model, a different T5 variant, or a non-English PLM cannot assume these numbers will replicate.

What evidence exists in the paper. The paper does run some within-family scale variations: RoBERTa-BASE vs. RoBERTa-LARGE (Figure 5c vs. 5d) and T5-Small/Base/XXL (Figures 5a, 5b, 3b). The qualitative patterns (task-type block structure, ON outperforming embedding metrics) are consistent across these scales, which provides some evidence of robustness within each model family. However, the paper reports one cross-family transfer direction (RoBERTa-LARGE → T5-XXL, Section 5) and one cross-architecture within-encoder transfer (BERT-BASE → RoBERTa-BASE, Table 7) — but these are still within the broad class of bidirectional transformer encoders. No decoder-only model is tested, and the reverse direction (T5 → RoBERTa) is not evaluated. The 17-task benchmark suite, while diverse, is still a specific sample of English NLP tasks with well-defined label spaces and evaluation metrics — it does not include tasks like open-ended generation, multi-turn dialogue, code generation, or multilingual tasks where the notion of "task type" may be less crisp.

Mitigation status. The paper does not claim universality and is transparent about its model selection (Section 3.3: "We investigate prompt transferability for two series of PLMs: RoBERTa and T5"). However, it also does not discuss generalization as a limitation or suggest that users of other model families should validate the findings before relying on them. The title's "for Natural Language Processing" implies breadth that the experimental scope does not fully support. A practitioner would need to replicate key experiments (at minimum, the zero-shot transfer matrix for their model and task suite) before adopting the paper's recommendations.

---

Speedup Metrics Are Highly Sensitive to Convergence Definition and Hyperparameters

The assumption or constraint. All speedup numbers in Tables 1 and 2b are computed using a specific convergence criterion: training loss stops dropping and evaluation score stops increasing for 300 consecutive steps, with evaluation every 100 steps (Appendix B.3). The learning rate is fixed at 0.001 for all tasks and all PLMs (Appendix A.3). The paper does not sweep learning rates for vanilla PT to find the fastest-converging configuration, nor does it test alternative convergence criteria.

The consequence. The dramatic speedup numbers — especially the 49.7× for SST-2, 49.9× for SNLI, and 219.8× comparable-result for laptop on T5-XXL (Table 1) — may substantially overstate the practical benefit of transfer if vanilla PT's baseline convergence time is inflated by suboptimal hyperparameters or an overly conservative convergence criterion. If vanilla PT with a better-tuned learning rate (e.g., 0.003, 0.005, or with a schedule) converges 3× faster than the reported baseline, then the 49.7× speedup becomes a ~16× speedup — still meaningful but far less dramatic. The fixed 300-step patience criterion interacts differently with PT and TPT-TASK: if PT's loss decreases very slowly but steadily (taking many steps of marginal improvement before plateauing), it will be declared "converged" much later than if it oscillates or plateaus quickly, even though most of the practical performance gain occurred early. TPT-TASK, starting from a better initialization, might converge sharply and trigger the patience criterion quickly. The speedup ratio then reflects the interaction between the optimization trajectory shape and the patience parameter as much as genuine initialization benefit.

The comparable-result speedup metric compounds this sensitivity. For a task where vanilla PT reaches 90% of its final accuracy in the first 20% of training steps but takes the remaining 80% of steps to squeeze out the last 10%, the comparable-result speedup will be 5× if TPT-TASK reaches that 90% threshold quickly — even though vanilla PT was practically usable much earlier than its "convergence" point. The 219.8× comparable-result speedup for laptop on T5-XXL (Table 1) almost certainly reflects this dynamic: the laptop-trained source prompt likely achieves near-PT-final accuracy immediately, while vanilla PT crawls slowly toward its final value over many steps. The 219.8× number tells us more about vanilla PT's asymptotic convergence tail than about TPT-TASK's absolute speed.

What evidence exists in the paper. The paper provides no learning rate sweep for vanilla PT — the 0.001 rate is used uniformly (Appendix A.3). It does not report training curves (loss vs. steps or accuracy vs. steps) for vanilla PT vs. TPT-TASK, which would allow readers to assess whether convergence is genuinely faster or just declared sooner. The convergence criterion details are buried in Appendix B.3 without sensitivity analysis (e.g., what happens with 500-step patience? 200-step?). The paper does not report wall-clock time in any standardized hardware environment — only step counts and step-count ratios — so it is impossible to separate "faster convergence in optimization steps" from "faster convergence in GPU-hours," which could differ if TPT-TASK's forward passes are more expensive (they are not — the prompt length and model are identical — but this reinforces the opacity of the reported metrics).

Mitigation status. The paper does not address this limitation. The speedup metrics are presented as objective measures of efficiency gain without caveats about hyperparameter sensitivity, convergence criterion interaction, or the distinction between step-count speedup and practical speedup. A practitioner reading Table 1 has no way to calibrate the 49.7× number against their own experience with T5-XXL PT convergence on SST-2, because no absolute baselines are reported. This is the most significant weakness in the paper's quantitative claims — the speedup numbers are the headline practical takeaway, but they are measured in a way that makes them difficult to interpret or reproduce.

---

Cross-Model Transfer Requires Task-Specific Projector Training and Doesn't Solve the Cost Problem

The assumption or constraint. Cross-model transfer via TPT-MODEL requires training a Task Tuning projector on the target PLM using task-specific supervision (Section 5.1, Equation for $\mathcal{L}_{\text{tune}}$). This projector training is itself a form of prompt tuning on the target model — it requires running the target PLM's forward and backward passes on training data from the projector-training task, backpropagating through the projector parameters. The paper uses laptop and MNLI as projector-training tasks (Section 5.1: "we choose laptop and MNLI in experiments"). Once trained, the projector generalizes to same-type tasks (Table 2a) but not different-type tasks.

The consequence. The practical promise of cross-model transfer — "train prompts on a small and computationally efficient PLM and use them on a massive and computationally expensive PLM" (Section 5) — is partially undermined by the projector training cost. To get a projector that works for sentiment analysis tasks on T5-XXL, you must first do something very close to prompt tuning for a sentiment analysis task on T5-XXL anyway during projector training. The projector training cost (running T5-XXL forward/backward passes on laptop's training data, plus the cost of training the source prompt on RoBERTa-LARGE) is not accounted for in TPT-MODEL's speedup numbers (Table 2b). Those speedups measure only the final target-task PT phase after the projector already exists. In the amortized setting — train one projector, then use it for many same-type target tasks — the projector cost might be negligible per target task. But the paper does not evaluate this multi-target scenario, and for a practitioner with a single target task, cross-model transfer likely costs more total compute than simply training PT from scratch on the target model, because you pay for source prompt training + projector training + TPT-MODEL target training, versus just target PT.

Furthermore, the projector's generalization is limited to the task type it was trained on. To cover 6 task types, you would need to train 6 separate projectors (or find that some projectors generalize across types — which the evidence in Table 2a suggests they do not). Each projector requires training on the expensive target PLM with task supervision. The total upfront cost to build a multi-type projector suite on a large PLM could be substantial.

What evidence exists in the paper. The paper reports TPT-MODEL speedups in Table 2b (e.g., 19× comparable-result for SST-2, 30× for SNLI) but does not amortize projector training cost into these numbers. The projector training details (Appendix C.1: optimizer AdamW, learning rate 0.005, batch size 16, hidden dimension 768) describe a non-trivial training procedure whose cost is not quantified anywhere. The paper does not report how many steps or how much wall-clock time projector training takes, making it impossible to compute the total cost of cross-model transfer. The failure of Distance Minimizing (Table 2a) is correctly identified, but the paper does not explore whether cheaper projector training methods (e.g., few-shot, smaller training sets, or freezing part of the projector) could reduce the cost while preserving within-type generalization.

Mitigation status. The paper does not address this cost accounting gap. It acknowledges that a projector is needed and describes how to train one, but treats the projector's existence as a given when computing TPT-MODEL speedups. The abstract states that cross-model transfer can be done "with a cross-model projector trained on similar tasks" without noting the cost of training that projector. A practitioner reading the paper would not learn whether cross-model transfer is net-beneficial for a single target task or only for batched deployment across many same-type tasks.

---

The ON Metric's Correlation with Transferability Degrades on the Largest Models

The assumption or constraint. The overlapping rate of activated neurons (ON) is proposed as a transferability indicator that can predict which source prompts will transfer well to a target task without running actual transfer experiments. The metric measures cosine similarity between binarized feed-forward neuron activation patterns (Section 6.1, Equation 7) induced by different prompts on the same PLM. The paper claims this metric "strongly reflects the transferability" and "is better correlated with prompt transferability than prompt embedding distance-based metrics" (Section 6.2).

The consequence. ON's Spearman's rank correlation with zero-shot transfer performance drops sharply as model size increases within the T5 family: 52.5% on T5-Small, 49.1% on T5-Base, and 36.9% on T5-XXL (Table 10, Figure 4). T5-XXL is precisely the model scale where prompt tuning is most promising (because it's where PT matches fine-tuning performance, per Lester et al., 2021) and where transfer would be most valuable (because PT convergence is slowest on the largest models). The metric that supposedly enables cheap transferability prediction works worst on the model where you most need it. At 36.9% Spearman's correlation, ON explains only ~13.6% of the variance (R²) in transfer performance — it is a weak predictor that would frequently mis-rank source tasks, leading to suboptimal source selection and reduced or eliminated transfer benefits. The paper's own analysis attributes this degradation to model redundancy (larger PLMs having multiple functionally equivalent but neuronally distinct pathways for the same computation), but the ONI variant that partially addresses this (intersection over three seeds, raising T5-XXL correlation to 46.3%, Table 10) requires training three prompts per task — exactly the expensive process that ON was supposed to replace.

What evidence exists in the paper. Figure 4 directly shows the negative correlation between model scale and ON's Spearman's rank correlation for T5 models. Table 10 provides the per-task-type and overall correlations across T5-Small, T5-Base, and T5-XXL, confirming the monotonic degradation. The ONI results in Appendix D.3 (Table 10, bottom rows) demonstrate that redundancy is the likely cause and that intersection-based methods partially recover the signal, but at the cost of tripling the number of prompts that must be trained. The paper does not test ON or ONI on models larger than T5-XXL (11B parameters), so whether the correlation continues to degrade or asymptotes at larger scales is unknown.

Mitigation status. The paper acknowledges this limitation explicitly (Appendix D.3: "We guess that this phenomena may result from PLMs' high redundancy") and proposes ONI as a "preliminary trial" that partially addresses it. However, ONI is not a practical solution — it requires training three prompts per task to compute the activation intersection, which is more expensive than running a few zero-shot transfer experiments for the most promising candidate sources. The paper does not propose a way to overcome redundancy without additional training, and Section 6's conclusion that the ON metric "strongly reflects the transferability" is not appropriately qualified for the large-model regime. For a practitioner working with a 10B+ parameter model, ON is not a reliable transferability indicator, and the paper does not provide an alternative that works well at that scale.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a reframing contribution rather than a paradigm shift: it takes prompt tuning — which the community treated primarily as a model adaptation method competing on parameter efficiency — and reframes it as a knowledge encoding mechanism where trained soft prompts constitute reusable, transferable representations of task-solving strategies. This reframing has several concrete consequences for how researchers think about and work with prompt tuning.

From random initialization to informed initialization. Prior to this work, prompt tuning was almost universally performed from random initialization — the prompts were treated as arbitrary learnable parameters with no structure worth preserving. Lester et al. (2021) had observed in passing that PT showed stronger cross-domain transferability than fine-tuning, but this was a brief observation, not a systematic finding, and the field continued to initialize prompts randomly by default. This paper provides the first systematic evidence that trained prompts have meaningful structure that can be reused across tasks and models, and more importantly, that leveraging this structure through transfer-based initialization can yield order-of-magnitude speedups (49.7× convergence speedup on SST-2 for T5-XXL, Table 1). This shifts the default assumption: a practitioner with access to any trained prompt on a related task should not start from random initialization; they should start from the transferred prompt and fine-tune. The burden of proof is now on random initialization — you would need to show that transfer doesn't help for your specific case to justify ignoring it.

Task type as the organizing principle for transfer. The paper's most robust empirical finding — that prompt transferability is governed primarily by task type (sentiment analysis, NLI, question answering, etc.) rather than by domain, dataset size, or other surface features — provides a simple but powerful organizing principle. The block-diagonal structure of the zero-shot transfer matrices (Figure 3) is replicated across five model variants (Appendix B.1) and across both cross-task and cross-model transfer settings (Tables 2a, 7). This tells practitioners something immediately actionable: when building a "prompt warehouse" for a deployed PLM, organize it by task type, not by dataset name or application domain. When facing a new target task, first identify its task type (SA? NLI? QA?), then retrieve the most data-rich source prompt from within that type — a procedure that requires no expensive similarity metric computation and no zero-shot transfer experiments. The paper's similarity metrics (ON, embedding distances) are attempts to automate this retrieval, but the finding that task type alone is a strong heuristic is arguably more practically useful than the metrics themselves.

Reconciliation of conflicting intuitions about prompt semantics. Prior to this paper, there was a latent tension in how the community implicitly thought about soft prompts. Some researchers treated them as essentially arbitrary embedding vectors — "knobs" that happen to be tuned by gradient descent but have no inherent interpretability. Others, inspired by the success of discrete prompt engineering (where manually crafted textual prompts clearly encode task instructions), suspected that soft prompts might similarly encode task-relevant semantic content in their embedding space. The paper's findings reconcile these views by showing that both perspectives capture part of the truth, but at different levels of analysis. In embedding space, prompts for different tasks are simply points in $\mathbb{R}^{l d}$ — they form "distinguishable clusters" (Table 3) but their Euclidean or cosine distances are only weakly predictive of transfer success (best Spearman's correlation: 44.7% for C_average, Table 4). In activation space, however, prompts are more meaningfully structured — the specific neurons they activate in the frozen PLM's feed-forward layers are substantially more predictive of transfer success (ON: 49.7% on RoBERTa-LARGE). This tells us that prompts are functionally meaningful in terms of the neural circuits they recruit, even if they are geometrically noisy in embedding space. The implication is that future work on prompt interpretability and transfer should focus on model-internal activations rather than embedding-space analysis — a methodological shift that the ON metric operationalizes.

Verifier over-optimization in a different guise. While this paper does not use the term, the failure of certain transfers and the degradation of ON on large models exhibit a phenomenon analogous to what the reference example paper called "verifier over-optimization." When a prompt transfers well zero-shot (e.g., IMDB → Movie achieving 101% on RoBERTa-LARGE, Figure 3a), it means the source prompt has encoded the task-solving strategy so robustly that it dominates within-task variation — the prompt has "overfit" to the task type rather than the specific dataset, which is a feature, not a bug, for transfer. But when ON's correlation degrades from 52.5% on T5-Small to 36.9% on T5-XXL (Figure 4), it means the PLM's internal redundancy has "over-optimized" the representation such that functionally equivalent prompts look neurally dissimilar — the metric that works on small models fails on large ones because large models have too many ways to do the same thing. This parallel to over-optimization in RLHF and test-time compute scaling is a useful conceptual connection: larger models have more degrees of freedom to represent the same function, making similarity measurement harder. The ONI variant (intersection across seeds, recovering to 46.3%) is analogous to ensembling in verifier design — it filters out spurious variation to recover the underlying functional signal.

Which research directions become more attractive. The paper makes the case that understanding how prompts stimulate PLMs is the key to improving prompt transfer. Specifically, it suggests that the neuron-activation perspective (ON, ONI) is a more promising lens than embedding geometry for predicting and explaining transfer. This shifts research attention away from better prompt initializations or better embedding-space similarity metrics (which Vu et al., 2021 explored) and toward methods that measure or manipulate the PLM's internal computation in response to prompts. The paper also implicitly argues that cross-model prompt transfer is feasible but type-bounded, which directs attention toward building projectors that work across task types — perhaps by training on more diverse task families, by using meta-learning to quickly adapt projectors to new types, or by conditioning the projector on task type information.

Which directions become less attractive. The paper's strong negative result on cross-type transfer (zero-shot and via projectors) suggests that building a single universal prompt that works for all tasks — a natural ambition given the success of instruction-tuned models like T0 (Sanh et al., 2022, concurrent work) — is unlikely to succeed through prompt tuning alone. The prompts studied here are task-specific and do not compose across task types. Instruction tuning achieves cross-task generalization by modifying the PLM's weights, not by finding a single prompt that works for everything; this paper's results suggest that soft prompts cannot achieve the same cross-type generalization without modifying the PLM itself. This narrows the scope of prompt tuning as a method: it is powerful for within-type transfer and for accelerating PT on individual tasks, but it is not a path to general-purpose task-agnostic prompting. Researchers interested in general-purpose prompting should look to weight-modifying methods (instruction tuning, adapter fusion) rather than prompt-only transfer.

Follow-Up Research This Work Enables

1. Training a difficulty estimator for prompt transfer: can we predict zero-shot transfer success from the source prompt alone, without running the target task? The ON metric achieves only 49.7% Spearman's correlation on RoBERTa-LARGE, and this degrades on larger models. A natural follow-up would train a dedicated predictor — perhaps a small neural network — that takes the source prompt's embedding or activation pattern as input and predicts its zero-shot transfer performance on a set of held-out target tasks. The training data would come from the zero-shot transfer matrices the paper has already computed (Figure 3): for each of the 13–17 × 13–17 source-target pairs, you have both the prompt's properties (embedding vectors, activation patterns from multiple layers) and the ground-truth transfer performance. A strong follow-up would compare: (a) ON as a zero-shot metric, (b) embedding similarity metrics, (c) a trained predictor using only prompt embeddings, (d) a trained predictor using activation patterns, and (e) a trained predictor using both. The key question is whether a learned predictor can recover the performance of oracle source selection (the best source from Figure 3) without requiring the full zero-shot transfer matrix. This would directly address the paper's unaccounted cost of source task discovery.

2. Testing whether TPT-TASK with similarity-metric-based source selection (instead of oracle selection) preserves the reported speedups. The paper's TPT-TASK results in Table 1 use oracle source selection — for each target task, the source task with the best zero-shot transfer in Figure 3 is chosen. In a realistic deployment where Figure 3 does not pre-exist, a practitioner would need to use a similarity metric (ON, $C_{\text{average}}$, etc.) to select the source task. A critical follow-up experiment would replicate Table 1 but with source tasks selected by ON (or by task type alone, which is effectively a discrete similarity metric) rather than by oracle zero-shot transfer. The hypothesis: for tasks where the best zero-shot source is obvious (within-type, data-rich), similarity-based and oracle selection will agree, and speedups will be preserved. For tasks where the best source is less obvious or where multiple mediocre sources exist, similarity-based selection may choose suboptimally and yield smaller speedups or even slowdowns. Quantifying this gap would tell practitioners whether they need to invest in building the zero-shot transfer matrix or can rely on cheaper heuristics.

3. Does prompt transfer work for decoder-only models (GPT family), and does the ON metric generalize to their architecture? The paper's experiments are exclusively on encoder-only (RoBERTa) and encoder-decoder (T5) models. Decoder-only models like GPT-3, LLaMA, and their instruction-tuned variants were available at the time of this work (GPT-3 was released in 2020) and are now the dominant architecture for large language models. A crucial replication study would test cross-task and cross-model prompt transfer on a decoder-only model family. The questions include: (a) Does the task-type block structure in zero-shot transfer (analogous to Figure 3) hold for autoregressive LMs where "prompts" are prepended to the input and the model generates the output token-by-token? (b) Does ON computed on decoder self-attention layers (rather than encoder-decoder attention) show similar correlation with transfer performance? (c) How does the interaction between soft prompts and the model's existing instruction-following capabilities (if instruction-tuned) affect transferability — does a strong instruction-tuned model make all prompts look more similar (reducing the value of transfer) or more task-specialized (increasing it)? The paper's findings on RoBERTa and T5 provide the experimental template; replicating with LLaMA-7B and LLaMA-13B (or similar) would bound generalization.

4. Building a multi-type prompt projector by training on multiple task types simultaneously. The paper shows that Task Tuning projectors generalize within the task type they were trained on (SA → SA, NLI → NLI) but fail on different types. The natural extension is to train a projector on multiple task types simultaneously — e.g., train one projector on both laptop (SA) and MNLI (NLI) with separate task heads or a shared multi-task objective. The question is whether the projector can learn a type-conditional mapping: when it receives an SA source prompt, it projects to the SA region of the target PLM's prompt space; when it receives an NLI source prompt, it projects to the NLI region. If successful, a single multi-type projector could cover all 6 task types, dramatically reducing the upfront cost of cross-model transfer (from 6 projectors to 1). If unsuccessful — if training on multiple types causes interference and degrades within-type performance — that would tell us something important about the geometry of prompt spaces: that SA and NLI prompts occupy disjoint, non-overlapping regions of the target PLM's effective prompt manifold, and no single linear+nl projection can map both simultaneously. This experiment would use the same projector architecture and training setup as the paper, but with a training set that interleaves batches from two or more task types, and would evaluate generalization on held-out tasks of each type.

5. Measuring whether TPT-TASK speedups persist with tuned learning rates: separating genuine initialization benefit from suboptimal optimization. The paper uses a fixed learning rate of 0.001 for all tasks and all PLMs (Appendix A.3). A rigorous follow-up would perform a learning rate sweep for vanilla PT on the tasks with the largest reported TPT-TASK speedups (e.g., SST-2 and SNLI on T5-XXL) to find the fastest-converging configuration. Then compare: (a) vanilla PT with the best learning rate vs. (b) TPT-TASK with the default learning rate 0.001. If vanilla PT with a tuned learning rate converges 5× faster than vanilla PT at 0.001, then the 49.7× speedup is actually ~10×, which is still meaningful but changes the cost-benefit analysis of transfer. If vanilla PT with any learning rate still converges dramatically slower than TPT-TASK, then the initialization benefit is robust and not an artifact of optimization. This experiment would also provide practical guidance: should a practitioner invest in building a prompt warehouse and transfer pipeline, or simply tune their PT learning rate? The paper's silence on this comparison leaves the speedup numbers open to the critique that they reflect poor baseline optimization rather than genuine transfer benefit.

6. Prompt transfer for sequential revision: can transferred prompts serve as the starting point for iterative self-improvement of model outputs? Connecting this paper to the reference example paper's revision model concept: if a prompt encodes a task-solving strategy, can it be used as the initial "proposal distribution" for iterative revision, where the model conditions on its own previous outputs to improve? The paper's TPT-TASK initializes PT with a trained source prompt and then fine-tunes on the target task — the fine-tuning step is a form of adaptation. A follow-up could test whether the transferred prompt, without any fine-tuning, can serve as a starting point for test-time revision: the PLM generates an answer using the transferred prompt, evaluates its own answer (using the same or a different verifier), and produces a revised answer conditioned on the previous (incorrect) attempt. The hypothesis is that prompts from data-rich source tasks (e.g., IMDB → Movie) provide a strong enough initial strategy that even zero-shot transfer plus a few revision steps matches or exceeds vanilla PT performance. This would connect prompt transfer to the test-time compute scaling literature and position transferred prompts as cheap, high-quality initializations for iterative refinement loops.

Practical Applications and Downstream Use Cases

1. Rapid multi-task deployment of a single large PLM via a prompt warehouse. An organization that has fine-tuned or prompt-tuned a large PLM (e.g., T5-XXL, a LLaMA variant) for several tasks of different types can build a prompt warehouse: a library of trained soft prompts for sentiment analysis, NLI, question answering, summarization, etc., each trained to convergence on a large, high-quality dataset of its type. When a new downstream task arrives — say, a custom sentiment analysis task for internal customer feedback — the practitioner identifies the task type (SA) and uses the warehouse's SA prompt as initialization for prompt tuning on the new data. Based on Table 1's numbers for T5-XXL, this would reduce convergence time by approximately 1–50× depending on the specific source-target pair, with the largest gains when the source dataset is data-rich (e.g., IMDB with 25,000 reviews serving as the source for smaller SA targets). Critically, this does not require computing the full zero-shot transfer matrix — the task-type heuristic alone (pick the largest source prompt of the same type) is sufficient and is supported by the paper's finding that same-type transfer consistently succeeds. The warehouse grows incrementally: each new task adds its converged prompt to the library, potentially serving as a better source for future tasks.

2. Cost-efficient model upgrading when a new, larger PLM is released. When an organization migrates from one PLM to a larger or newer one (e.g., from T5-Base to T5-XXL, or from RoBERTa to a newer encoder), they face the prospect of re-training prompts for all downstream tasks. Cross-model transfer via TPT-MODEL provides a path to amortize this cost: train Task Tuning projectors on a few representative tasks of each type (e.g., one SA task, one NLI task), then use these projectors to map all existing prompts from the old model to the new model. The prompts are used as initialization on the new model and fine-tuned briefly (TPT-MODEL, Table 2b). The efficiency gain depends on how many same-type tasks the projector serves: if an organization has 10 sentiment analysis tasks and trains one SA projector on laptop, the projector training cost is amortized across 10 target tasks, making the per-task cost low. The paper's cross-model speedups (Table 2b: 1.7×–49.9× convergence speedup for same-type tasks on T5-XXL) quantify the per-task benefit. The limitation — that projectors don't generalize across task types — means an organization with diverse task types needs approximately one projector per type, but the paper shows that even two projectors (SA and NLI) provide broad coverage for common NLP applications.

3. Few-shot prompt tuning for low-resource tasks using transfer from high-resource tasks. For tasks where only a small amount of labeled data is available (e.g., a niche classification task with 100 examples), standard prompt tuning may fail to converge or overfit. The paper's zero-shot transfer results (Figure 3) show that a prompt trained on a large same-type dataset can achieve 65–101% of the fully-trained PT performance without any target-task training at all. For a low-resource target task, a practitioner could: (a) retrieve the most data-rich same-type prompt from a warehouse, (b) evaluate zero-shot performance to establish a baseline, and (c) run TPT-TASK with the small target dataset to adapt the transferred prompt. The combination of strong zero-shot baseline + rapid convergence (due to good initialization) means that even 100 examples may suffice to achieve near-PT performance, where training from scratch would require thousands. The paper does not directly test few-shot adaptation (all TPT-TASK experiments use the full target training set), but the zero-shot transfer performance establishes the lower bound, and the convergence speedups (Table 1) suggest that few-shot fine-tuning from a transferred prompt should be effective. A practitioner could set this up using the paper's released code at https://github.com/thunlp/Prompt-Transferability.

4. Debugging and interpreting prompt tuning failures via neuron activation analysis. The ON metric and its layer-wise breakdown (Appendix D.4, Figures 8–12) provide a diagnostic tool for understanding why prompt tuning fails on a particular task or why a transferred prompt doesn't work as expected. If a practitioner trains a prompt for a new task and obtains poor performance, they could: (a) compute the ON similarity between the failing prompt and prompts for known, successful tasks of different types, (b) identify which task type's prompt the failing prompt most resembles in terms of neuron activation patterns, and (c) diagnose whether the model is attempting the wrong task type (e.g., treating a classification task as a generation task) or whether the prompt is simply in a "dead zone" of activation space that doesn't recruit any task-specific circuits. The layer-wise ON visualizations (Figures 8–12) show that top layers are most task-specific — if a failing prompt shows normal bottom-layer activation but abnormal top-layer patterns, the issue is likely in task-specific processing rather than generic language understanding. This diagnostic use of ON does not require the metric to achieve perfect correlation with transfer — it serves as a qualitative probe rather than a quantitative predictor.

When to Prefer This Method

The paper positions prompt transfer (TPT-TASK and TPT-MODEL) not as a wholesale replacement for vanilla prompt tuning, but as an initialization strategy that is beneficial when certain conditions hold. It does not present a structured "prefer A when / prefer B when" decision table, but the experimental results imply clear boundary conditions that a practitioner can use to decide whether to invest in transfer:

• Use TPT-TASK when: (a) you have access to a trained prompt for a task of the same type as your target task (e.g., both are sentiment analysis), (b) the source task's dataset is as large or larger than your target task's dataset (the paper shows IMDB → Movie transfers better than Movie → IMDB because IMDB has more data), and (c) you care about reducing training time — the convergence speedups range from 1.0× to 49.9× depending on the specific pair (Table 1), with the largest benefits on T5-XXL for tasks with strong zero-shot transfer. Do not use TPT-TASK if no same-type source prompt exists — cross-type transfer initialization yields speedups near or below 1.0× and may degrade final performance.

• Use TPT-MODEL when: (a) you have a source prompt trained on a smaller/cheaper PLM and want to deploy on a larger/more expensive PLM, (b) the target task is of the same type as the task used to train the projector (e.g., you trained an SA projector on laptop and now want to transfer prompts for other SA tasks), and (c) you are deploying multiple same-type tasks on the target PLM, so the projector training cost is amortized. Do not use TPT-MODEL for a single target task of a new type — the projector training cost on the expensive target PLM likely outweighs the benefit for a one-off deployment.

• Use zero-shot prompt transfer (no further training) when: (a) the source and target are same-type tasks, (b) the source task has substantially more data than the target task (e.g., IMDB → Movie achieving 101% relative performance on RoBERTa-LARGE, Figure 3a), and (c) your performance requirements are modest — zero-shot transfer achieves 65–101% of fully-trained PT performance for same-type pairs, which may be sufficient for some applications without any target-task training.

• Do not use prompt transfer (just run vanilla PT) when: (a) no same-type source prompt exists for your target task, (b) your target task is of a type not represented in the paper's 6-type taxonomy (the generalization to new types is unknown and likely poor based on the sharp task-type boundaries observed), or (c) you have a very small target training set and zero-shot transfer performance is poor — in this case, the transferred initialization may be far from the target solution and provide no convergence benefit, while vanilla PT from random initialization will converge normally (albeit slowly) given enough target data.

These conditions are inferred from the paper's results rather than stated as explicit recommendations by the authors. The paper's main practical guidance is that transfer "can significantly accelerate training and also improve the performance of PT" (Section 1), with the implicit understanding that the source and target must be task-type-similar — a condition verified by the zero-shot transfer matrix or, in its absence, by the task-type heuristic.