Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging

🎯 Pitch

Model merging lets you add new skills like safety or coding to an existing language model without access to the original training data, and it surprisingly matches full retraining quality while cutting compute costs by up to 95%. For safety, it even beats other methods by reducing harmful refusals and nearly eliminating the over-refusal problem where models incorrectly reject safe requests.

1. Executive Summary

This paper systematically studies how to efficiently add new skills to an already instruction-tuned language model without degrading its original capabilities, evaluating three approaches—continued finetuning (CFT), retraining from scratch (RT), and parallel train then merge (PTM), a family of methods that train on new skills in isolation and later merge the resulting models via weight-space operations (e.g., task arithmetic, linear interpolation, WiSE-FT)—on the Tülu V2 general-purpose instruction mix using Llama 2 7B, with target skills spanning scientific literature understanding, coding, and safety. PTM achieves competitive skill-specific performance to the best retraining baselines while requiring 50–95% fewer training steps, preserves nearly all of the original model’s general skills versus a 10–40% drop for continued finetuning, and proves especially effective for safety—dramatically improving compliance with safe prompts while reducing exaggerated refusals by 30–80% compared to alternatives. The paper also provides a practical heuristic for setting the mixture weight proportional to the ratio of skill-specific to general training steps when held-out data is unavailable, establishing that model merging enables efficient skill addition without access to original training data, though skill interference can degrade performance when multiple new skills are merged simultaneously.

2. Context and Motivation

The Core Problem: How Do You Add a New Skill to an Already-Trained Model?

The fundamental question this paper tackles is one that practitioners face constantly: you have a capable general-purpose instruction-tuned language model, and a new high-quality dataset appears that teaches a skill your model lacks — how do you update your existing model to incorporate that new skill? This matters because the landscape of available instruction data is constantly expanding. New datasets targeting specific capabilities — scientific literature understanding, coding, safety-aligned refusal behavior — are created regularly, and model developers need reliable, cost-effective ways to keep their deployed models current.

The paper frames this as a practical deployment challenge with immediate consequences. From Section 1:

"Adapting general-purpose language models to new skills is currently an expensive process that must be repeated as new instruction datasets targeting new skills are created, or can cause the models to forget older skills."

This is not a theoretical concern. Instruction-tuned models like Llama 3 (AI@Meta, 2024), Mistral 7B (Jiang et al., 2023), and Gemma (Gemma-Team et al., 2024) are widely deployed in production settings. When a new safety benchmark reveals a vulnerability, or when users demand support for a new domain like scientific literature understanding, the developer faces an immediate decision: retrain the entire model at enormous expense, continue finetuning on the new data and risk the model forgetting its general capabilities, or find a third option. This paper explores that third option — model merging — and provides the first systematic comparison of the alternatives for instruction tuning.

Three Specific Pain Points This Problem Creates

The introduction identifies three distinct consequences of this adaptation challenge that motivate the work:

1. Computational cost compounds with each new skill. If a new instruction dataset emerges every few months (as has been the trend), retraining from scratch on the combined data mixture each time becomes prohibitive. For a model the size of Llama 2 7B trained on the Tülu V2 mix (~275,000 examples), a single retraining run requires reprocessing all the general data plus the new skill-specific data. The paper quantifies this explicitly in Section 2.2: for $n$ different data mix variations, retraining requires $n \cdot |G| + \sum_{i=1}^{n} |\mathcal{D}_i|$ training steps, where $|G|$ is the size of the general instruction mix. Since general instruction datasets contain "hundreds of thousands to millions of instances" while skill-specific datasets contain "tens to a hundred thousand instances," the $n \cdot |G|$ term dominates — you pay the cost of retraining on all the old data every time you want to experiment with a new data mixing ratio. This is the regime where we need $50$ - $95\%$ training efficiency improvements to remain practical.

2. Catastrophic forgetting is real and damaging. Continued finetuning (CFT) is the computationally cheaper option — just keep training the instruction-tuned model on the new skill data. But Section 2.1 acknowledges what the continual learning literature has documented for decades going back to McCloskey and Cohen (1989): gradient-based sequential training causes models to overwrite previously learned knowledge. The paper explicitly frames this as a trade-off: CFT "while computationally cheaper, may cause the model to forget the skills from earlier rounds of training." The results in Section 4.2 bear this out starkly — Table 3 shows a 40.1% degradation in general skills when using CFT for safety, an 85.1% degradation in exaggerated refusals compliance, and a 32.5% general skill drop for science. These are not marginal regressions; they are catastrophic failures that render the model unusable for its original purpose.

3. Retraining from scratch is often impossible. A subtle but crucial point raised in Section 2.2: retraining on a combined data mixture requires access to the original training datasets. For many publicly available instruction-tuned models, these datasets are not released:

"retraining is not possible in cases where the pretrained and instruction-tuned models have been released but the general instruction mix has not, such as Llama 3 (AI@Meta, 2024), Mistral 7B (Jiang et al., 2023) and Gemma (Gemma-Team et al., 2024)"

This means that for some of the most widely used open-weight models in the ecosystem, the "just retrain from scratch" advice is literally impossible to follow. A method that can add new skills without requiring the original training data fills a genuine gap in the practitioner's toolkit.

Where Existing Approaches Fall Short

The paper identifies specific limitations in prior attempts to solve this class of problems:

Continued finetuning ignores the forgetting problem or addresses it expensively. Prior work in continual learning has proposed regularization techniques to minimize overfitting during finetuning (Ahn et al., 2019; Lee et al., 2019), recall or replay mechanisms that interleave old and new data during training (Kirkpatrick et al., 2017; Chen et al., 2020), and parameter-efficient methods like LoRA (Hu et al., 2021) that update only a small subset of the model's weights. Section 5 (Related Work) acknowledges these contributions but draws a sharp distinction: "Unlike most of these methods, model merging does not require access to the original training data." This is the key differentiator — replay methods need to store or regenerate old training data, regularization methods reduce but do not eliminate forgetting, and LoRA still requires careful tuning of the adaptation rank and can still cause interference. None of these approaches offer the clean separation that PTM does: train on the new skill entirely in isolation, then combine with the existing model through a single scalar parameter $\omega$ .

Retraining is the gold standard but the expense is rarely justified. The paper positions retraining as the baseline that should achieve the best performance — since the model sees all the data in the right proportions — but acknowledges that the computational and data-access requirements make it impractical in many settings. The paper's contribution is not to argue that PTM beats retraining (it generally doesn't — Table 3 shows RT achieving slightly better or comparable results in most settings), but rather that PTM matches retraining at a fraction of the cost and works when retraining is impossible due to missing data.

Existing model merging methods are known but uncharacterized for instruction tuning. The paper is explicit that it does not invent model merging — Section 2.3 cites task arithmetic (Ilharco et al., 2023), linear interpolation (Rofin et al., 2022), WiSE-FT (Wortsman et al., 2022b), and the broader "branch-train-merge" framework (Li et al., 2022) as prior art. What the paper contributes is the first systematic study of these methods for instruction tuning, comparing them head-to-head against CFT and RT across multiple skill types (science, safety, coding) with rigorous compute accounting. Prior work on task vectors and model patching had primarily been evaluated in vision settings (Ilharco et al., 2023) or on pretraining objectives, not on the instruction-following capabilities that dominate modern LM deployment. This gap matters because instruction tuning produces models with different properties (e.g., multi-task generalization, sensitivity to prompt formatting) that may interact differently with weight-space operations.

No prior work established practical heuristics for merge coefficients. A subtle but practically important gap: existing model merging methods assume access to held-out validation data for selecting the mixture weight $\omega$ , the scalar that controls how much influence the new skill vector has on the final model. As the paper notes in Section 3.2:

"Many instruction datasets do not have validation sets (Ivison et al., 2023; Zheng et al., 2024; Lian et al., 2023; Singh et al., 2024), and thus how to select models is an open question."

Without a validation set, a practitioner merging a new coding skill into their model has no principled way to choose $\omega = 0.2$ versus $\omega = 0.8$ . The paper's heuristic — setting $\omega = |\mathcal{D}| / |G|$ , the ratio of skill-specific to general training steps — fills this gap. It provides a zero-shot rule that the paper shows (Figure 1) consistently picks a point on the trade-off curve that preserves most general performance while substantially improving skill-specific performance.

Safety-specific challenges were unexplored in the merging context. The paper identifies safety as a domain where the trade-offs are particularly acute because safety behaviors (refusing dangerous requests) can directly conflict with general capabilities (complying with harmless requests that superficially resemble dangerous ones). The "exaggerated refusals" problem — where safety-trained models over-generalize and refuse safe prompts — is well-documented in the RLHF literature (e.g., Röttger et al., 2024, cited as the source of the XSTest benchmark). But no prior work had asked whether model merging could decouple safety training from general capability degradation more effectively than standard approaches. The paper's finding that PTM reduces exaggerated refusals by 30–80% compared to CFT and RT (Section 4.2, Table 3) reveals that merging modifies the geometry of the safety-capability trade-off in a way that is not achievable through data mixing alone.

How This Paper Positions Itself

The authors frame their contribution as a comprehensive empirical study rather than a methodological innovation. The abstract explicitly states: "we investigate the effectiveness of adding new skills to preexisting models by training on the new skills in isolation and later merging with the general model." The language — "investigate," "explore," "systematically explore" — signals that the contribution is about understanding when and why existing techniques work for a new application domain (instruction tuning), not about proposing a new merging algorithm.

The paper's positioning relative to prior work can be understood along three axes:

Computational efficiency: PTM requires $|\mathcal{D}|$ training steps total — you train one model on the new skill data and then sweep $\omega$ over different values at merge time, which costs negligible compute. In contrast, CFT requires $\sum_{i=1}^{n} |\mathcal{D}_i|$ for $n$ data amount variations, and RT requires $n \cdot |G| + \sum_{i=1}^{n} |\mathcal{D}_i|$ . The paper's efficiency claims (50–95% improvement) are relative to these baselines, and the exact percentage depends on the ratio of general to skill-specific data. For science, where $|G| = 275{,}464$ and $|\mathcal{D}| = 61{,}349$ , PTM costs 479 steps versus RT's 11,766 steps — a ~96% reduction. For coding, where $|\mathcal{D}| = 156{,}526$ (closer in size to $|G|$ ), the reduction is smaller but still substantial.

Data access: PTM requires only the new skill-specific data $\mathcal{D}$ and the instruction-tuned model $\theta_G$ — crucially, it does not require the original training mix $G$ or the pretrained data. This is the property that makes it applicable to models like Llama 3 and Mistral where $G$ is unknown. The paper positions this as a practical advantage over RT while noting that CFT shares this property (it also only needs $\mathcal{D}$ ) but suffers from forgetting.

Behavioral preservation: This is where PTM is positioned as strictly better than CFT and competitive with RT. The core claim is that "since PTM does not directly change the weights associated with previously learned tasks, it should allow the model to retain more of its original skills compared to other methods" (Section 1). This is a hypothesis grounded in the geometry of the weight-space operations: adding a task vector $\tau_D = \theta_D - \theta_{\text{pre}}$ to $\theta_G$ via $\theta_{\text{final}} = \theta_G + \omega \cdot \tau_D$ means the weights associated with general skills are modified only by the component of $\tau_D$ that overlaps with $\theta_G - \theta_{\text{pre}}$ , while CFT modifies all weights through gradient updates that can overwrite previously learned functions. The results in Figure 3 and Table 3 validate this hypothesis: across all three skill domains, PTM preserves general performance within a few percentage points of the original Tülu model, while CFT degrades it by 10–40%.

The paper is explicitly not claiming that PTM dominates in all dimensions. The authors acknowledge that RT achieves the best overall performance in many settings (Table 3 shows RT matching or slightly exceeding PTM on general skills while often achieving higher skill-specific performance), that PTM can suffer from interference when merging multiple skills (Table 4 shows science performance dropping from 27.8 to 26.6 when all three skills are merged), and that the mixture weight heuristic is a practical workaround rather than a theoretically grounded optimum. The contribution is in characterizing when PTM is the right choice — which is often, given the constraints practitioners actually face — not in arguing that it is universally superior.

3. Technical Approach

3.1 Reader Orientation

This paper is a comparative empirical analysis, not a new method proposal. The core idea is that adding a new skill to an already-trained instruction-tuned language model can be reframed as a weight-space combination problem: train a separate model on only the new skill data, then merge its parameters with the existing generalist model using simple arithmetic operations, avoiding both the catastrophic forgetting of continued finetuning and the computational expense and data-access requirements of retraining from scratch. The paper systematically characterizes when this "parallel train then merge" (PTM) approach works, how it compares to alternatives, and what heuristics make it practical when validation data is unavailable.

3.2 Big-Picture Architecture

The system consists of five logical components, though only three are "active" during any particular skill-addition procedure:

Base Pretrained Model ( $\theta_{\text{pre}}$ ): Llama 2 7B, the foundation model before any instruction tuning. Serves as the common reference point for computing task vectors in two of the three PTM variants.
General Instruction-Tuned Model ( $\theta_G$ ): Created by training $\theta_{\text{pre}}$ on the modified Tülu V2 mix (275,464 examples covering world knowledge, mathematics, reasoning, truthfulness, and open-ended generation). This is the "starting model" that the practitioner wants to extend with new skills.
Skill-Specific Model ( $\theta_{\mathcal{D}}$ ): Created by training either $\theta_{\text{pre}}$ (for task arithmetic and linear interpolation) or $\theta_G$ (for WiSE-FT) on only the new skill dataset $\mathcal{D}$ (SciRIFF for science, internal dataset for safety, CodeFeedback for coding). This model is good at the new skill but has lost general capabilities.
Merge Operation: A weight-space arithmetic procedure that combines $\theta_G$ and $\theta_{\mathcal{D}}$ (and optionally $\theta_{\text{pre}}$ ) into a final model $\theta_{\text{final}}$ using a scalar mixture weight $\omega$ . Three merge formulas are studied (task arithmetic, linear interpolation, WiSE-FT), each using different intermediate task vectors.
Evaluation Suite: Held-out benchmarks for general skills (MMLU, GSM8K, AlpacaEval, Big Bench Hard, TruthfulQA) and skill-specific performance (9 science tasks from SciRIFF, 4 safety metrics including ToxiGen/HarmBench/XSTest, HumanEval+ and MBPP+ for coding). These are used for model selection when validation data exists, and for reporting final performance.

Information flows linearly: choose a skill $\rightarrow$ choose a PTM variant $\rightarrow$ train $\theta_{\mathcal{D}}$ on skill data $\rightarrow$ compute task vector(s) $\rightarrow$ sweep mixture weight $\omega$ $\rightarrow$ select $\omega$ using validation data or heuristic $\rightarrow$ evaluate $\theta_{\text{final}}$ on both general and skill-specific benchmarks.

3.3 Roadmap for the Deep Dive

First, the formal problem definition and training complexity model, because all comparisons between CFT, RT, and PTM are anchored to the number of training steps each method requires, and understanding the cost model is essential for interpreting the paper's efficiency claims.
Second, the three merge formulas that constitute PTM, because task arithmetic, linear interpolation, and WiSE-FT each define a different way to construct and combine task vectors, and their differing dependencies on $\theta_{\text{pre}}$ determine when each is applicable.
Third, the training data and evaluation design, because the paper's findings depend on the specific composition of the general instruction mix, the nature of each skill dataset, and how "general" versus "skill-specific" performance is measured.
Fourth, the model selection procedures, because the paper's central practical contribution is the mixture-weight heuristic $\omega = |\mathcal{D}| / |G|$ , which is only meaningful in the context of how models are selected when held-out data exists versus when it does not.
Fifth, the multi-skill merging procedure, because the paper extends PTM from single-skill addition to combining multiple skill vectors simultaneously, which introduces interference effects not present in the single-skill case.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose technical contribution is not a new algorithm but rather a systematic characterization of existing weight-space merging methods applied to instruction tuning, including cost modeling, comparative evaluation against standard baselines, and practical heuristics for deployment without validation data.

Formal Problem Statement and Training Complexity Model

Section 2 defines the problem precisely. The practitioner has a general-purpose instruction-tuned model $\theta_G$ , trained on a general dataset $G$ , and wants to incorporate a new skill represented by dataset $\mathcal{D}$ to improve performance on a skill-specific evaluation set $E_{\mathcal{D}}$ without degrading performance on a general evaluation set $E_G$ , while minimizing computational cost.

The paper measures computational cost in training steps, defined as the total number of optimization steps required to produce the pool of candidate models from which the final model is selected. Since all experiments use a fixed effective batch size of 128 (Appendix A.2), training steps are proportional to the number of examples processed, making them a hardware-independent cost metric.

For continued finetuning (CFT): The general model $\theta_G$ is further trained on $n$ different subsets $\mathcal{D}_i$ of the skill-specific data, each subsample requiring $|\mathcal{D}_i|$ training steps. The total cost is:

$\text{Cost}_{\text{CFT}} = \sum_{i=1}^{n} |\mathcal{D}_i|$

where $|\mathcal{D}_i|$ is the number of training steps for the $i$ -th subsample of the skill data, and $n$ is the number of different data amounts tried (set to 5 in the paper's experiments).

What it computes: the total number of parameter updates across all CFT variants explored. Since each variant starts from $\theta_G$ and trains on only the new skill data (not reprocessing $G$ ), the cost scales with how many different data amounts are tested.

Why this form: CFT is computationally cheap per-variant because it does not require retraining on the general data, but the linear sum over $n$ variants reflects the fact that each different subsample requires a separate training run from scratch from $\theta_G$ . You cannot reuse intermediate checkpoints to test different data amounts without retraining, because training more steps than necessary (overshooting the optimal amount) causes additional forgetting.

For retraining from scratch (RT): The pretrained model $\theta_{\text{pre}}$ is trained on a mixture of the full general dataset $G$ and each skill-specific subsample $\mathcal{D}_i$ . The total cost is:

$\text{Cost}_{\text{RT}} = n \cdot |G| + \sum_{i=1}^{n} |\mathcal{D}_i|$

where $|G|$ is the number of training steps for the full general instruction mix, and the $n \cdot |G|$ term reflects that the general data must be reprocessed for every mixture variant.

What it computes: the total number of parameter updates across all RT variants, counting both the general data reprocessing (which dominates) and the skill-specific data.

Why this form: the $n \cdot |G|$ term is linear in both the number of mixture ratios tested and the size of the general dataset. This is why RT is expensive: every time you want to test a different ratio of skill-to-general data, you must retrain on all of $G$ from scratch. For the paper's experiments, $|G| = 275{,}464$ examples divided by the batch size of 128 equals approximately 2,151 steps per epoch, and with 2 epochs of training, $|G| \approx 4{,}302$ steps. With $n = 5$ mixture variants, the $n \cdot |G|$ term alone is approximately 21,510 steps before adding any skill-specific data.

For parallel train then merge (PTM): A single model $\theta_{\mathcal{D}}$ is trained on all available skill-specific data (no subsampling), and the influence of the new skill on the final model is controlled entirely by the scalar mixture weight $\omega$ swept after training is complete. The total cost is:

$\text{Cost}_{\text{PTM}} = |\mathcal{D}|$

where $|\mathcal{D}|$ is the number of training steps for the full skill-specific dataset.

What it computes: the cost of a single training run on the skill data, with no multiplier for testing different influence levels because $\omega$ is varied at merge time with negligible computational cost.

Why this form: this is the key efficiency argument. PTM decouples training cost from model selection: you train once, then sweep $\omega$ across arbitrarily many values at merge time (which involves only element-wise addition and scalar multiplication of parameter tensors — operations that cost seconds on a CPU). CFT and RT, by contrast, must retrain for every different level of skill-data influence because the "influence" is baked into the training process (through number of steps or data mixing ratio) rather than controlled by a post-hoc parameter. This architectural difference is what produces the 50–95% training efficiency improvements reported in the abstract.

The paper explicitly states in Section 3.2 that for all experiments, $n = 5$ : "We evaluate five checkpoints for each setting, varying the influence of the skill-specific data on the general model." For CFT and RT, the five levels correspond to training on 5 different amounts of skill-specific data (ranging from roughly 4k to 61k examples for science, following the subsample sizes reported in SciRIFF). For PTM, the five levels correspond to 5 different values of the mixture weight $\omega$ : 0.2, 0.4, 0.6, 0.8, and 1.0.

The Three PTM Merge Formulas

Section 2.3 and Section 4.4 describe three distinct methods for implementing the merge operation. All three produce a final model $\theta_{\text{final}}$ by combining a general-purpose model, a skill-specific model, and optionally the pretrained model through linear operations in weight space. They differ in which models serve as the "base" for task vector computation and how the mixture weight is applied.

Task Arithmetic (Primary Method)

The paper treats task arithmetic as the primary PTM method, using it for all main experiments in Sections 4.1–4.3, and comparing the other two methods against it in Section 4.4.

Step 1: Create the skill-specific task vector. Finetune the pretrained model $\theta_{\text{pre}}$ on all of the skill-specific data $\mathcal{D}$ to obtain $\theta_{\mathcal{D}}$ , then compute the difference:

$\tau_{\mathcal{D}} = \theta_{\mathcal{D}} - \theta_{\text{pre}}$

where $\tau_{\mathcal{D}}$ is the task vector representing the direction and magnitude of weight changes induced by training on skill $\mathcal{D}$ , $\theta_{\mathcal{D}}$ is the model after finetuning on the skill data, and $\theta_{\text{pre}}$ is the frozen pretrained checkpoint.

What it computes: the element-wise difference between the skill-finetuned model's parameters and the pretrained model's parameters. This vector captures what the model had to change in its weights to become good at the new skill, relative to the pretrained starting point.

Why this form: subtracting $\theta_{\text{pre}}$ isolates the skill-specific adaptation. The pretrained model serves as a common reference frame: since both the general model $\theta_G$ and the skill model $\theta_{\mathcal{D}}$ were derived by finetuning $\theta_{\text{pre}}$ (though on different data), expressing the skill model's deviation from $\theta_{\text{pre}}$ as $\tau_{\mathcal{D}}$ allows that deviation to be "added" to $\theta_G$ in a semantically meaningful way. Without this subtraction, you would be adding absolute parameter values rather than changes, which would double-count the pretrained knowledge.

Step 2: Merge the task vector into the general model. Add the scaled task vector to the general instruction-tuned model:

$\theta_{\text{final}} = \theta_G + \omega \cdot \tau_{\mathcal{D}}$

where $\omega \in \mathbb{R}$ is a scalar mixture weight controlling how much influence the new skill has (with $\omega < 1.0$ found experimentally to be better than $\omega = 1.0$ ), and $\theta_G$ is the general instruction-tuned model.

What it computes: each weight in the general model is shifted by a fraction $\omega$ of the corresponding weight change that occurred when the pretrained model was adapted to the new skill. If $\omega = 0$ , the final model is identical to $\theta_G$ (no skill added). If $\omega = 1$ , the full task vector is added. Intermediate values produce a smooth interpolation of influence.

Why this form: this is a simple weighted addition in parameter space, which is computationally trivial (one scalar multiplication and one vector addition per parameter). The scalar $\omega$ provides a single knob that controls the trade-off between retaining general skills ( $\omega \rightarrow 0$ ) and acquiring the new skill ( $\omega \rightarrow 1$ ). The paper's key empirical finding — that $\omega < 1.0$ is better than $\omega = 1.0$ — suggests that adding the full task vector overshoots: the skill model $\theta_{\mathcal{D}}$ has not only learned the new skill but also unlearned general capabilities (since it was trained only on $\mathcal{D}$ without the general mix), so fully adding its deviation from $\theta_{\text{pre}}$ would import that unlearning into $\theta_G$ . Downweighting with $\omega < 1.0$ accepts a partial skill improvement in exchange for retaining more general knowledge.

Linear Interpolation

This method treats both general and skill-specific capabilities as task vectors originating from the pretrained model, and interpolates between them rather than adding one to the other.

Step 1: Create both task vectors. Compute the general task vector and skill task vector by subtracting the pretrained model from each:

$\tau_G = \theta_G - \theta_{\text{pre}}$

$\tau_{\mathcal{D}} = \theta_{\mathcal{D}} - \theta_{\text{pre}}$

where $\tau_G$ represents the weight changes for general instruction-following and $\tau_{\mathcal{D}}$ represents the weight changes for the specific skill.

Step 2: Interpolate between them and add to the pretrained model:

$\theta_{\text{final}} = \theta_{\text{pre}} + \omega \cdot \tau_{\mathcal{D}} + (1 - \omega) \cdot \tau_G$

where $\omega \in [0, 1]$ controls the relative weighting: $\omega = 0$ recovers the general model, $\omega = 1$ recovers the skill-specific model, and intermediate values produce convex combinations.

What it computes: a convex combination of the two task vectors applied to the pretrained base. The constraint that the weights sum to 1 ( $\omega$ and $1 - \omega$ ) ensures that at the extremes, you recover exactly $\theta_G$ or exactly $\theta_{\mathcal{D}}$ .

Why this form: this approach explicitly models both capabilities as deviations from the same pretrained checkpoint and trades them off against each other. This is conceptually cleaner than task arithmetic (which adds a task vector to an already-finetuned model) but has a distinct disadvantage: as $\omega$ increases (more skill influence), the general task vector $\tau_G$ is explicitly downweighted. Task arithmetic, by contrast, adds $\tau_{\mathcal{D}}$ to $\theta_G$ without attenuating $\theta_G$ 's existing weights. The paper's results in Figure 3 (right panels) confirm this disadvantage: linear interpolation achieves strong skill-specific performance but consistently shows much larger degradation in general skills compared to task arithmetic at the same skill-improvement level. The convex combination forces a zero-sum trade-off that task arithmetic avoids.

WiSE-FT (Weight Interpolation for (in)Sample Efficiency — Fine-Tuning)

This method is designed for settings where the pretrained checkpoint $\theta_{\text{pre}}$ is not available. It uses the general model $\theta_G$ as the reference point instead.

Step 1: Continue finetuning on skill data. Train $\theta_G$ further on the skill-specific data $\mathcal{D}$ to produce $\theta_{\text{CFT}}$ . This is a standard CFT step: the model is updated via gradient descent on $\mathcal{D}$ , starting from $\theta_G$ .

Step 2: Compute the CFT task vector:

$\tau_{\text{CFT}} = \theta_{\text{CFT}} - \theta_G$

where $\tau_{\text{CFT}}$ captures what the model changed during continued finetuning — specifically, what it did to become good at the new skill at the expense of general capabilities.

Step 3: Downweight and add back:

$\theta_{\text{final}} = \theta_G + \omega \cdot \tau_{\text{CFT}}$

where $\omega \in [0, 1]$ controls how much of the CFT-induced change is retained.

What it computes: the model after CFT is compared to the general model before CFT, and only a fraction $\omega$ of those changes are kept. This is conceptually similar to task arithmetic but uses $\theta_G$ as the reference instead of $\theta_{\text{pre}}$ .

Why this form: the key advantage is that $\theta_{\text{pre}}$ is not required — only $\theta_G$ and $\theta_{\text{CFT}}$ are needed. This makes WiSE-FT the only PTM variant applicable to models like Llama 3 or Mistral where the pretrained checkpoint may be available but the exact pretraining recipe or intermediate checkpoints are not guaranteed to exist in a form compatible with the instruction-tuned release. However, the paper's results in Figure 3 show that WiSE-FT performs substantially worse than task arithmetic on general skill preservation. The authors diagnose this in Figure 2 and Section 4.4: because the skill-specific data $\mathcal{D}$ is distributionally different from the general instruction mix $G$ , finetuning $\theta_G$ on $\mathcal{D}$ causes large weight changes that are not well-modeled as a small perturbation. Adding back even a fraction of these changes with $\omega \cdot \tau_{\text{CFT}}$ still imports generalization damage. The paper shows that mixing general data into the CFT step (Figure 2, "WiSE-FT + General Data") reduces this problem, confirming the distribution-shift hypothesis.

Training Data and Model Configuration

The base model for all experiments is Llama 2 7B (Touvron et al., 2023). The general-purpose instruction-tuned model $\theta_G$ is created by training the pretrained Llama 2 7B checkpoint on a modified Tülu V2 mix (Section 3.1). The Tülu V2 mix (Ivison et al., 2023) is a curated collection of publicly available instruction-tuning datasets. The paper modifies it by removing three subsets:

Science subset: 7,500 examples removed (to ensure the general model has room to improve on scientific literature understanding when science skills are later added).
CodeAlpaca: 20,000 examples removed (to create headroom for coding skill improvement).
Refusals identified by heuristics: 23,000 examples removed (to create headroom for safety training).

After these removals, the general mix contains 275,464 total instances. This deliberate removal is an important experimental design choice: it simulates a realistic deployment scenario where the general model was trained before the new skill datasets existed, and therefore genuinely lacks those capabilities. If the general model already contained substantial science training data, adding more science via PTM would show smaller gains, making it harder to measure the method's effectiveness.

The three skill-specific datasets are:

SciRIFF (Wadden et al., 2024): 61,349 training examples covering scientific literature understanding tasks including information extraction, question answering, summarization, and claim verification across biomedicine, AI, and other scientific disciplines. The evaluation uses 9 held-out tasks from SciRIFF's validation and test sets (BioASQ, BioRED, DiSCoMaT, Evidence Inference, MultiCite, MUP, QASPER, SciERC, SciFact), measuring exact-match or F1 metrics as appropriate for each task. The validation set is used for model selection in Section 4.1.

Safety dataset: An internally developed dataset of 66,161 examples, each consisting of a potentially dangerous or harmful prompt paired with a refusal response generated by GPT-4. A seed set of prompts was written by humans, and additional prompts were generated by GPT-4 from this seed set. Refusals were collected by prompting GPT-4 and retaining responses classified as refusals. This dataset covers categories including harmful language, malicious uses, misinformation, and others. Evaluation uses four metrics: ToxiGen (toxicity detection, reported as 0–100 with 100 best), HarmBench (automated red teaming, 0–100 with 100 best), XSTest Unsafe (refusal rate on genuinely unsafe prompts, 0–100 with 100 best), and XSTest Safe (compliance rate on safe prompts that superficially resemble unsafe ones — measuring exaggerated refusals, 0–100 with 100 best). For aggregate "safety performance," the paper averages the first three metrics and reports exaggerated refusals (XSTest Safe) separately.

CodeFeedback (Zheng et al., 2024): The single-turn subset consisting of 156,526 instruction/code pairs. Evaluation uses pass@10 on HumanEval+ and MBPP+, sampled with temperature 0.8. These two scores are averaged for the aggregate coding metric.

All models, regardless of the method (CFT, RT, PTM), are trained with identical hyperparameters (Appendix A.2):

Precision: BFloat16
Epochs: 2
Weight decay: 0
Warmup ratio: 0.03
Learning rate: $2 \times 10^{-5}$
Maximum sequence length: 4,096 tokens
Effective batch size: 128
Optimizer: AdamW (implied by the framework; the paper states it follows Ivison et al., 2023)
Hardware: v3-128 TPUs using a fork of EasyLM

The fixed batch size and number of epochs means that the number of training steps for any dataset is directly proportional to its size: $\text{steps} = (2 \times \text{dataset\_size}) / 128$ . For the general mix ( $|G| = 275{,}464$ ), this is $(2 \times 275{,}464) / 128 \approx 4{,}304$ steps. For SciRIFF ( $|\mathcal{D}| = 61{,}349$ ), this is $(2 \times 61{,}349) / 128 \approx 958$ steps. The paper's reported training step counts (e.g., 479 for PTM science, 11,766 for RT science) differ slightly from these calculations because the RT and CFT experiments use varying subsample sizes of $\mathcal{D}$ , not always the full dataset. The cost model's efficiency ratios are based on the actual step counts reported in the tables.

Model Selection Procedures

The paper uses two distinct model selection strategies depending on whether a held-out validation set exists for the new skill (Section 3.2).

When validation data exists (science): The paper uses SciRIFF's validation set to select the best checkpoint from each method group (CFT, RT, PTM). For CFT and RT, five checkpoints are trained corresponding to five different amounts of science data: approximately 4k, 8k, 16k, 32k, and 61k examples (following the subsamples reported in SciRIFF). For PTM, one model is trained on the full 61k examples, and five checkpoints are created by merging with $\omega \in \{0.2, 0.4, 0.6, 0.8, 1.0\}$ . The checkpoint with the highest average validation score across SciRIFF's held-out tasks is selected. The selected model is then evaluated on the test sets for both general skills and science. This is reported in Table 2, where the "Best CFT" model was chosen by science validation performance (achieving 40.6 on science test but only 33.7 on general), the "Best RT" model was similarly chosen (achieving 37.8 science, 50.6 general), and the "Best PTM" model was chosen (achieving 38.2 science, 47.1 general).

When validation data does NOT exist (safety, coding): The paper introduces a heuristic for selecting $\omega$ without any held-out data. The heuristic is:

$\omega = \frac{|\mathcal{D}|}{|G|}$

where $|\mathcal{D}|$ is the number of training steps for the full skill-specific dataset and $|G|$ is the number of training steps for the general instruction mix.

What it computes: the ratio of how much training the model received on the new skill relative to how much training it received on general instruction-following. If the general model was trained for 4,304 steps and the skill model for 958 steps, the heuristic sets $\omega = 958 / 4304 \approx 0.22$ .

Why this form: the intuition is that the mixture weight should reflect the relative amount of evidence the model has seen for each capability. If the skill data is small relative to the general data, the task vector should be downweighted to avoid over-relying on limited skill-specific signal at the expense of the abundant general signal. If the skill data is comparably large, a higher $\omega$ is warranted. This is a zero-shot rule that requires no validation data and no additional computation beyond knowing the sizes of the training datasets (which are known by construction). The paper validates this heuristic in Figure 1, showing that the point on the trade-off curve selected by $\omega = |\mathcal{D}|/|G|$ (highlighted in each panel) consistently achieves strong performance: it preserves most general skills while substantially improving skill-specific performance.

The paper also uses a different selection criterion for the aggregate comparison in Table 3: "We select models in each category based on their average percentage improvement over the baseline model in two dimensions: general skills and performance for science, safety, and coding, and exaggerated refusal compliance rate and safety for the exaggerated refusals rows." This is a multi-objective selection that picks the model that best balances both dimensions, reported as percentage change from the Tülu baseline. This selection criterion is used for the head-to-head comparison but is orthogonal to the practical heuristic; it represents an oracle selection that a practitioner could only approximate.

Evaluation Protocol

The paper evaluates all models on a consistent suite of benchmarks, divided into general and skill-specific categories.

General evaluations (5 benchmarks):

MMLU (Hendrycks et al., 2021): measures world knowledge across 57 subjects, reported as accuracy.
GSM8K (Cobbe et al., 2021): grade-school math word problems, reported as accuracy.
AlpacaEval (Li et al., 2023): open-ended instruction following evaluated by an LLM judge, reported as win rate against a reference model.
Big Bench Hard (BBH) (Suzgun et al., 2022): challenging reasoning tasks, reported as accuracy.
TruthfulQA (Lin et al., 2022): measures truthfulness and resistance to common misconceptions, reported as accuracy.

The "General" score reported in the tables is typically the average across these five metrics (for Tables 2 and 3) or reported individually (Appendix C).

Science evaluations (9 tasks):

BioASQ, BioRED, DiSCoMaT, Evidence Inference, MultiCite, MUP, QASPER (reported as F1 or two separate metrics), SciERC, SciFact (reported with two metrics).
The average across all tasks (on validation or test) is reported as the "Science" score.

Safety evaluations (4 metrics):

ToxiGen: toxicity avoidance, reported as 0–100 with 100 best.
HarmBench: automated red teaming resistance, reported as 0–100 with 100 best.
XSTest Unsafe: refusal rate on unsafe prompts, reported as 0–100 with 100 best.
XSTest Safe: compliance rate on safe but superficially dangerous-looking prompts (exaggerated refusals), reported as 0–100 with 100 best.
The "Safety" score in aggregate tables averages the first three. Exaggerated refusals are reported separately.

Coding evaluations (2 benchmarks):

HumanEval+ (Chen et al., 2021; Liu et al., 2023): function synthesis from docstrings, pass@10 with temperature 0.8.
MBPP+ (Austin et al., 2021; Liu et al., 2023): basic Python programming, pass@10 with temperature 0.8.
The "Coding" score is the average of these two pass@10 values.

The evaluation design is chosen to cover a range of capabilities that might interact differently with the skill-addition methods. The general benchmarks test capabilities that should be preserved; the skill-specific benchmarks test capabilities that should be improved. The safety evaluation is unique in having a directly conflicting metric (exaggerated refusals): improving safety should increase refusal of unsafe prompts but ideally NOT increase refusal of safe prompts. This tension makes safety the most diagnostically interesting domain for evaluating trade-off management.

Multi-Skill Merging

Section 4.3 extends PTM to the case where multiple new skills need to be added to the same model simultaneously. The procedure is straightforward: train separate skill-specific models for each skill (in isolation, using the same PTM variant), compute their individual task vectors, and then merge all of them into the general model at once.

The paper does not provide a separate equation for multi-skill merging, but the implied formula for task arithmetic with $k$ skills is:

$\theta_{\text{final}} = \theta_G + \sum_{j=1}^{k} \omega_j \cdot \tau_{\mathcal{D}_j}$

where $\tau_{\mathcal{D}_j} = \theta_{\mathcal{D}_j} - \theta_{\text{pre}}$ is the task vector for the $j$ -th skill, and $\omega_j$ is the mixture weight for that skill, set using the heuristic $\omega_j = |\mathcal{D}_j| / |G|$ individually for each skill.

What it computes: the general model is shifted by a weighted sum of all skill-specific task vectors, with each skill's influence proportional to its training data size relative to the general data.

Why this form: the additive structure assumes that skill vectors are approximately orthogonal in weight space — adding the science vector and the coding vector simultaneously should improve both skills without them destructively interfering. The paper tests this assumption directly in Table 4 and finds that it partially holds: coding and safety both improve (from 37.6 to 45.3 and 50.3 to 84.0, respectively), but science performance drops (from 27.8 to 26.6). Table 5 diagnoses this by testing pairwise merges: merging science and safety produces reasonable science performance (31.6), merging science and coding produces similarly reasonable science performance (32.1), but merging safety and coding causes science to crash to 18.8 — even though science isn't being directly modified in that pairwise merge. This reveals that the interference is not between the science vector and other vectors directly, but rather that the safety and coding vectors modify some shared parameters that science also depends on. The additive independence assumption breaks down when skill vectors overlap in the directions they modify.

The paper also tests two alternative merging algorithms designed to mitigate interference — TIES (Yadav et al., 2023) and DARE (Yu et al., 2024) — in Appendix B, Table 6. TIES uses a three-step procedure: trim low-magnitude changes, resolve sign conflicts between vectors by taking the majority sign, and then merge. DARE randomly drops a fraction of the task vector components and rescales the remainder. Both produce similar results to standard weighted averaging across all skills, with no mitigation of the science interference problem. This negative result suggests that the interference is not due to sign conflicts (which TIES explicitly addresses) but rather due to the safety and coding vectors both making large-magnitude changes to parameters that science performance depends on — a more fundamental capacity conflict that weight-space arithmetic cannot resolve.

4. Key Insights and Innovations

Innovation 1: Reframing Skill Addition as a Weight-Space Combination Problem, Not a Data Mixture Problem

The dominant assumption in the instruction-tuning community is that adding new skills to a model requires training on the combined data — either by mixing the new data with the old data and retraining, or by sequentially training on the new data with some forgetting-mitigation technique. This paper challenges that assumption at its root by proposing that skill addition can be reframed as a weight-space combination problem: you train a separate model on only the new skill data, then merge its parameters with the existing general model via simple arithmetic operations, completely decoupling the training of new skills from the preservation of old ones.

This is conceptually distinct from prior work on continual learning. Methods like EWC (Kirkpatrick et al., 2017), experience replay (Chen et al., 2020), and LoRA (Hu et al., 2021) all operate within the sequential training paradigm: the model processes new data through gradient updates, and the question is how to constrain those updates to avoid overwriting old knowledge. The implicit assumption is that the model must learn the new skill through the lens of its existing weights, which inevitably creates interference. PTM abandons this assumption entirely — it trains the new skill in a completely isolated weight space, then combines the result with the original model through operations that do not involve gradients. This is not an incremental improvement to sequential training; it is a fundamental architectural shift in how skill acquisition is implemented.

The practical consequence is what the paper calls the decoupling of training cost from model selection. In the sequential paradigm, testing different levels of influence for the new skill — how much should the safety training dominate over general capabilities? — requires retraining the model multiple times with different data mixtures or different numbers of finetuning steps. Each such retraining is expensive. In PTM, this influence is controlled by a single post-hoc scalar $\omega$ swept at merge time, which costs essentially zero compute. This means you can explore the entire trade-off curve between general and skill-specific performance with the cost of one training run plus trivial arithmetic. The paper's finding that PTM requires 50–95% fewer training steps than retraining is a direct consequence of this architectural decoupling, not merely a more efficient training recipe within the sequential paradigm.

The choice to test this idea on instruction tuning rather than pretraining is itself significant. Prior work on model merging — task arithmetic (Ilharco et al., 2023), model soups (Wortsman et al., 2022a), branch-train-merge (Li et al., 2022) — had been demonstrated primarily in vision settings or on pretraining objectives, where the evaluation is typically single-task accuracy. Instruction tuning is a multi-task regime where models must simultaneously perform well on dozens of qualitatively different capabilities (math, reasoning, factual knowledge, open-ended generation). The paper shows that weight-space arithmetic works in this more demanding setting, where interference between capabilities is the central failure mode, not an edge case. This extends the range of problems where merging is known to be effective and opens the door to a modular development workflow where skill-specific models are trained independently and composed at deployment time.

The evidence for this reframing is Tables 2 and 3, which show that PTM achieves skill-specific performance competitive with retraining (e.g., 38.2 vs. 37.8 on science test in Table 2, 88.9 vs. 89.6 on safety in Table 3) while preserving general skills within a few points of the original model (47.1 vs. 49.9 for science PTM, essentially no degradation for safety PTM at −0.13% change). The fact that this works at all — that you can add a task vector derived from finetuning the pretrained model on only the new skill data to an already instruction-tuned model and get coherent behavior — is the paper's most surprising finding and its core conceptual contribution.

Innovation 2: The Mixture Weight Heuristic $\omega = |\mathcal{D}| / |G|$ as a Zero-Shot Model Selection Rule

A persistent practical challenge in model merging is choosing the mixture coefficient $\omega$ — the scalar that controls how much influence the new skill vector has on the final model. Prior work on task vectors (Ilharco et al., 2023) and WiSE-FT (Wortsman et al., 2022b) selected $\omega$ using held-out validation data, measuring performance on a validation set at different $\omega$ values and picking the best one. This is standard practice, but the paper identifies a critical gap: many instruction-tuning datasets do not come with validation splits (Section 3.2 explicitly names Tülu V2, CodeFeedback, OpenOrca, and Aya as examples). Without validation data, the practitioner has no principled way to choose $\omega$ — a scalar that the paper shows dramatically affects the trade-off between skill acquisition and general capability preservation (Figure 1).

The paper's heuristic — $\omega = |\mathcal{D}| / |G|$ , the ratio of skill-specific to general training steps — is deceptively simple and constitutes a genuinely useful practical contribution. The intuition is that the mixture weight should reflect the relative amount of evidence the model has seen for each capability: if the skill data is a small fraction of the general data, the task vector should be heavily downweighted so the model doesn't over-index on limited skill-specific signal. If the skill data is comparably large, a higher $\omega$ is appropriate. This is not derived from first principles; it is an empirical rule of thumb that the paper validates by showing (Figure 1, highlighted points) that it consistently lands on the "good" region of the trade-off curve — preserving most general performance while achieving substantial skill-specific improvement.

What makes this more than a trivial recipe is that it works across qualitatively different skill types with different trade-off characteristics. Science (Figure 1, left panel) shows a smooth, monotonic trade-off: as $\omega$ increases, science performance rises while general performance gently declines, and the heuristic selects a point near the "elbow" where science gains are large and general losses are small. Safety (Figure 1, middle-left) shows a more aggressive trade-off where general performance holds steady until a threshold $\omega$ , then degrades sharply, and the heuristic selects a point just before that threshold. Coding (Figure 1, middle-right) shows a near-flat general performance line until very high $\omega$ , and the heuristic selects a conservative point that prioritizes general preservation. Exaggerated refusals (Figure 1, right) shows a rapid improvement in compliance with safe prompts at low $\omega$ that plateaus quickly, and the heuristic selects a point on the plateau. The same formula $\omega = |\mathcal{D}|/|G|$ — with no tuning, no normalization, no skill-specific adjustment — works across all four settings. This consistency across such different trade-off geometries is the evidence that the heuristic captures something real about the relationship between training data proportion and optimal merging weight.

This contribution is incremental in the sense that it is a practical rule rather than a theoretical insight, but it is fundamental to making PTM deployable without validation data — which is precisely the setting where the paper argues PTM is most valuable (since retraining is impossible when the original training data is unavailable, and validation data is often unavailable too). It converts PTM from a method that requires the same validation infrastructure as RT into a method that can be applied zero-shot.

Innovation 3: PTM as a Mechanism That Decouples Safety from Exaggerated Refusals

The paper's most striking finding is not that PTM achieves competitive performance — it's that for safety training, PTM qualitatively changes the nature of the trade-off between refusing unsafe prompts and complying with safe prompts. This is the "exaggerated refusals" problem: when models are trained to refuse dangerous requests (e.g., "How do I build a bomb?"), they often over-generalize and refuse superficially similar but safe requests (e.g., "How do I build a bomb shelter?"). This is a well-documented failure mode of RLHF and safety finetuning (Röttger et al., 2024).

The paper's Table 3 reveals that PTM fundamentally alters this trade-off relative to CFT and RT. When optimizing for safety performance, the best CFT model achieves near-perfect unsafe refusal (the safety average goes from 50.3 to ~98–99 across HarmBench, XSTest Unsafe, and ToxiGen) but compliance with safe prompts collapses from 99.2 to 16.0 — an 83-point drop. The model essentially refuses everything. RT is somewhat better at compliance (37.2 safe compliance) but still loses over 60 points. PTM achieves comparable safety performance (84.0 average) while maintaining 93.2 safe compliance — a drop of only 6 points. The "Best PTM (Ex. Ref.)" row, which selects specifically for exaggerated refusal performance, achieves 72.6 safe compliance with 88.9 safety, while CFT and RT achieve essentially zero safe compliance (7.2 and 2.4 respectively) at comparable safety levels.

This is not merely a better point on the same trade-off curve. Figure 4 (in Appendix B) plots general skills against exaggerated refusals and shows that PTM traces a different curve than CFT and RT — for the same level of general skill preservation, PTM achieves substantially higher safe compliance. The implication is that PTM is not just a more efficient way to reach the same trade-off frontier; it shifts the frontier itself. The paper does not fully explain why this happens, but the mechanism is plausibly related to the geometry of the merge operation: the safety task vector $\tau_{\text{safety}}$ encodes the direction in weight space that produces refusal behavior, but adding only a fraction $\omega$ of this vector means the model's original compliance behavior (which dominates at $\omega = 0$ ) is attenuated rather than overwritten. In CFT, by contrast, the gradient updates that produce refusal behavior simultaneously erode the representations that support appropriate compliance, because both behaviors depend on some of the same parameters (e.g., those governing whether a prompt is "dangerous-looking"). The decoupling that PTM provides — training the safety behavior in isolation and then adding it at reduced strength — appears to partially separate these entangled capabilities.

This finding is significant beyond the specific safety results because it suggests that PTM may be broadly useful for skills that conflict with general capabilities. Most skill addition is synergistic: adding coding improves coding without hurting math. Safety is adversarial: being more cautious can make you less helpful. The exaggerated refusals result reveals that PTM handles this adversarial case better than data-mixing approaches, which has implications for any domain where a new skill partially competes with existing behavior (e.g., adding formal writing style without losing conversational ability, or adding domain-specific terminology without degrading general fluency).

Innovation 4: Diagnostic Use of Pairwise Merges to Isolate Skill Interference

The multi-skill merging experiment in Section 4.3 produces a puzzling result: when all three skills (science, safety, coding) are merged into a single model, science performance drops from the baseline of 27.8 to 26.6 — a regression even though the science task vector is being added, not removed. Table 4 shows that other skills improve (coding from 37.6 to 45.3, safety from 50.3 to 84.0), so the merge is working directionally — but science gets worse, not better. This is the classic sign of negative interference between task vectors.

What the paper does next is methodologically distinctive: it exploits the near-zero cost of PTM merges to perform a diagnostic ablation that would be prohibitively expensive with CFT or RT. Table 5 reports all pairwise merges — science+safety, science+coding, safety+coding — to isolate where the interference originates. The results are revealing: science+coding produces science performance of 32.1 (an improvement over baseline 27.8), science+safety produces 31.6 (also an improvement), but safety+coding — which does not even involve the science vector — causes science to crash to 18.8. This means the interference is not between the science vector and other vectors directly, but rather that the safety and coding vectors, when combined, modify shared parameters in a way that damages science capabilities that were already present in the general model.

This is a genuinely useful diagnostic technique, not just a result. It demonstrates that because PTM merges have negligible cost (they are just vector additions), you can exhaustively test all $2^k - 1$ combinations of $k$ skill vectors to understand interference patterns — something that would require an exponential number of expensive retraining runs under the RT paradigm. The paper also tests two merging algorithms explicitly designed to reduce interference — TIES (Yadav et al., 2023), which resolves sign conflicts between task vectors, and DARE (Yu et al., 2024), which randomly drops task vector components — and finds that neither helps (Table 6), indicating that the interference is not due to sign conflicts (which TIES addresses) but due to capacity competition: the safety and coding vectors are both trying to modify the same parameters in different directions with large magnitudes, and no linear combination can satisfy all three skills simultaneously.

The significance of this diagnostic approach extends beyond the paper's immediate findings. It suggests a workflow for practitioners: train skill-specific models independently, merge them pairwise to identify interference, and then either accept the interference (if the degraded skill is less important), prune or downweight conflicting vectors, or train additional models specifically to mitigate the interference. The paper does not develop this into a full methodology, but the diagnostic capability is a clear contribution that changes how one would approach multi-skill model development.

5. Experimental Analysis

Evaluation Methodology

Dataset. The experiments use three skill-specific datasets to add new capabilities to a general-purpose instruction-tuned model. The general model is trained on a modified Tülu V2 mix (Ivison et al., 2023) from which the science subset (~7,500 examples), CodeAlpaca (~20,000 examples), and refusal examples (~23,000 examples) are deliberately removed, leaving 275,464 total instances. The three target skill datasets are: (1) SciRIFF (Wadden et al., 2024), 61,349 training examples covering scientific literature understanding tasks including information extraction, question answering, summarization, and claim verification across biomedicine, AI, and other disciplines; (2) an internally developed safety dataset of 66,161 examples, each a potentially harmful prompt paired with a GPT-4-generated refusal, covering categories like harmful language, malicious uses, and misinformation; and (3) the single-turn subset of CodeFeedback (Zheng et al., 2024), 156,526 instruction/code pairs. All evaluations are conducted on held-out test sets, with SciRIFF additionally providing a validation set used for model selection in Section 4.1.
Base model(s). All experiments use Llama 2 7B (Touvron et al., 2023) as the pretrained foundation. The general-purpose instruction-tuned model θ_G is created by fully finetuning Llama 2 7B on the modified Tülu V2 mix. Skill-specific models are created by finetuning either the pretrained Llama 2 7B checkpoint (for task arithmetic and linear interpolation) or θ_G itself (for WiSE-FT) on the respective skill datasets. For the retraining baselines, models are trained from the pretrained Llama 2 7B checkpoint on combined data mixtures. The choice of Llama 2 7B is motivated by its status as a widely-used open-weight model representative of the scale at which practitioners face the skill-addition trade-offs the paper studies.
Metrics. Performance is evaluated along two axes: general skills and skill-specific performance. General skills are assessed using five benchmarks: MMLU (world knowledge accuracy), GSM8K (math accuracy), AlpacaEval (instruction-following win rate), Big Bench Hard (reasoning accuracy), and TruthfulQA (truthfulness accuracy). The aggregate "General" score reported in summary tables is the average across these five metrics. Science performance is measured as the average score across nine held-out SciRIFF tasks (BioASQ, BioRED, DiSCoMaT, Evidence Inference, MultiCite, MUP, QASPER, SciERC, SciFact), using task-appropriate metrics (F1, exact match). Safety is evaluated on four metrics, each normalized to 0–100 with 100 being best: ToxiGen (toxicity avoidance), HarmBench (automated red teaming resistance), XSTest Unsafe (refusal rate on genuinely unsafe prompts), and XSTest Safe (compliance rate on safe prompts that superficially resemble unsafe ones — measuring exaggerated refusals). For aggregate safety reporting, the first three metrics are averaged; exaggerated refusals (XSTest Safe) are reported separately because they represent a conflicting objective. Coding is evaluated using pass@10 on HumanEval+ and MBPP+, sampled with temperature 0.8, with the two scores averaged for the aggregate coding metric.
Baselines. The paper compares three methods for adding new skills: (1) Continued Finetuning (CFT), where the general instruction-tuned model θ_G is further trained on varying amounts of skill-specific data, with five different data subsamples tested per skill; (2) Retraining from scratch (RT), where the pretrained model is trained on mixtures of the full general dataset and varying amounts of skill-specific data, again with five mixture ratios tested; and (3) Parallel Train then Merge (PTM), the primary method under investigation, instantiated through three merging formulas — task arithmetic (Ilharco et al., 2023), linear interpolation (Rofin et al., 2022), and WiSE-FT (Wortsman et al., 2022b) — with task arithmetic serving as the primary PTM variant for most experiments. Additionally, the original unmodified Tülu-trained model (before any skill addition) serves as the zero-shot reference point for measuring degradation and improvement. Two alternative merging algorithms, TIES (Yadav et al., 2023) and DARE (Yu et al., 2024), are tested briefly in Appendix B (Table 6) for the multi-skill merging setting.
Generation budget / compute accounting. Computational cost is measured in training steps, defined as the total number of parameter updates required to produce the pool of candidate models from which the final model is selected. Since all models share identical hyperparameters — effective batch size of 128, two epochs of training, BFloat16 precision — training steps are directly proportional to the number of examples processed and provide a hardware-independent cost metric. For CFT with n different data subsamples D_i, the cost is Σ_{i=1}^{n} |D_i|. For RT with n different mixture ratios, the cost is n·|G| + Σ_{i=1}^{n} |D_i|, where |G| is the number of training steps for the full general mix (approximately 4,304 steps for 275,464 examples at batch size 128 over two epochs). For PTM, the cost is simply |D|, the number of training steps for the full skill-specific dataset, because the mixture weight ω is swept at merge time with negligible computational cost. All experiments use n = 5 variations for fair comparison. This accounting method is what produces the paper's central efficiency claim: PTM requires 50–95% fewer training steps than alternatives because it decouples training cost from model selection.
Cross-validation / statistical protocol. The paper does not use cross-validation in the traditional sense. Instead, model selection follows two distinct procedures depending on data availability. When a held-out validation set exists (science, via SciRIFF's validation split), the best checkpoint from each method group (CFT, RT, PTM) is selected based on the highest average validation score, then evaluated on the held-out test sets. When validation data does not exist (safety, coding), the paper introduces and validates a zero-shot heuristic: set the mixture weight ω = |D| / |G|, the ratio of skill-specific to general training steps. For the aggregate comparison in Table 3, models are selected from each group "based on their average percentage improvement over the baseline model in two dimensions" — effectively an oracle multi-objective selection that picks the model achieving the best balance between general and skill-specific improvement across all five checkpoints. For the PTM checkpoints, ω values are swept at 0.2, 0.4, 0.6, 0.8, and 1.0; for CFT and RT, the five checkpoints correspond to training on different amounts of skill-specific data (ranging from approximately 4k to 61k examples for science). All results are reported as single-point estimates without confidence intervals.

Main Quantitative Results

Single-Skill Science: PTM Matches RT Performance at ~4% of the Compute Cost

Table 2 reports the head-to-head comparison on scientific literature understanding, where model selection uses SciRIFF's validation set. The headline finding: PTM achieves science test performance of 38.2, essentially matching RT's 37.8, while requiring only 479 training steps versus RT's 11,766 — approximately 4% of the compute. The Tülu baseline (no science training) achieves 27.9 on science and 49.9 on general skills.

The detailed trade-offs reveal the characteristic patterns of each method. CFT achieves the highest science score at 40.6, but this comes at catastrophic cost to general skills, which plummet from 49.9 to 33.7 — a 32.5% relative degradation. This is the classic forgetting pattern: continued finetuning on the narrow science distribution overwrites the model's general instruction-following capabilities. RT achieves the best general skill preservation at 50.6 (slightly above the baseline of 49.9, suggesting the added science data may have synergistic effects), and competitive science at 37.8, but at enormous computational cost (11,766 steps — training on the full general mix five times for the five data mixture ratios). PTM occupies a middle ground: science performance of 38.2 (within 0.4 points of RT) and general performance of 47.1 (a modest 2.8-point drop from baseline, versus CFT's 16.2-point drop), at only 479 training steps.

The individual evaluation breakdowns in Appendix C (Tables 7 and 8) reveal where the general degradation concentrates. PTM's general skill losses are uneven: GSM8K drops from 32.5 to 26.5 (a 6-point decline), while TruthfulQA actually improves slightly from 48.3 to 49.6, and MMLU holds nearly steady (49.6 to 48.5). On science, PTM improves substantially on BioRED (33.5 to 58.8), DiSCoMaT (22.8 to 47.2), and SciFact (38.5 to 48.5), but provides essentially no gain on Evidence Inference (15.6 to 11.7) or MUP (19.8 to 19.0), suggesting the science task vector captures some scientific reasoning capabilities but not others.

Cross-Skill Aggregate Comparison: PTM Preserves General Skills While CFT Destroys Them

Table 3 provides the paper's most comprehensive comparison, evaluating all three methods across all three skill domains plus exaggerated refusals, with model selection optimizing the average percentage improvement in both general and skill-specific dimensions. The results establish a consistent rank ordering: RT preserves general skills best (near-zero degradation), PTM preserves them nearly as well (degradation of ~0–1% for safety and coding, ~1.3% for science), and CFT destroys them (degradation of 7.7–40.1%).

For science, PTM shows +1.30% general improvement (unexpected but consistent with the detailed results showing TruthfulQA gains offsetting math losses) and +26.3% science improvement, versus RT's +1.37% general and +39.1% science, and CFT's −32.5% general and +46.0% science. The training cost tells the efficiency story: PTM at 479 steps, CFT at 1,005 steps, RT at 11,766 steps. PTM achieves two-thirds of CFT's science gain with none of the catastrophic forgetting, at less than half the compute cost.

For safety, the pattern is even more favorable to PTM. PTM achieves −0.13% general change (essentially zero degradation), +88.9% safety improvement, and requires 517 training steps. RT achieves +0.66% general, +89.6% safety, at 12,311 steps. CFT achieves +98.9% safety but −40.1% general at 1,551 steps. PTM matches RT's safety performance exactly while preserving general skills equally well, at 4.2% of the compute cost.

For coding, PTM shows +1.43% general improvement, +33.3% coding improvement, and 1,223 steps. RT achieves +0.13% general and +50.7% coding at 14,429 steps. CFT achieves +51.6% coding but −7.73% general at 3,669 steps. Here RT substantially outperforms PTM on skill-specific gains (+50.7% vs. +33.3%), and the efficiency advantage is narrower because the coding dataset (156,526 examples) is closer in size to the general mix. This is consistent with the paper's cost model: as |D| approaches |G|, the relative efficiency advantage of PTM over RT shrinks, though PTM still requires roughly 8.5% of RT's compute.

The exaggerated refusals rows represent a selection criterion that optimizes for XSTest Safe compliance while maintaining safety performance. Here PTM dramatically outperforms both alternatives: PTM achieves −6.45% general change, +72.6% exaggerated refusals improvement, and 517 steps, while CFT achieves −85.1% general and +98.9% exaggerated refusals (but this "improvement" is misleading because the CFT model's general skills are so degraded that it refuses nearly everything, including safe prompts), and RT achieves −39.9% general and +87.2% exaggerated refusals at 12,311 steps. The key number: PTM maintains 93.2 XSTest Safe compliance (Table 4, Tülu baseline is 99.2) while CFT and RT models optimized for exaggerated refusals collapse to 7.2 and 2.4 respectively (Table 12 in Appendix C). This is a 30–80% relative improvement in safe prompt compliance over the alternatives, as claimed in the abstract.

Individual evaluation breakdowns (Tables 9–12 in Appendix C) confirm these patterns. The CFT safety model's general collapse is stark: AlpacaEval drops from 74.3 to 4.7 (near-zero), GSM8K from 32.5 to 24.5, BBH from 44.8 to 34.7. The PTM safety model shows a more nuanced pattern: AlpacaEval drops to 62.9 (moderate), BBH to 41.2 (mild), but TruthfulQA improves to 65.7 (from 48.3 baseline) — an unexpected benefit possibly reflecting that safety training encourages more cautious, truthful outputs. The individual safety metrics (Table 12) show PTM achieving 95.6 on ToxiGen, 89.5 on XSTest Unsafe, and 75.2 on HarmBench, versus CFT's perfect or near-perfect scores across all three but with the exaggerated refusal problem (85.2 on XSTest Safe, meaning it refuses 85% of safe prompts).

The Mixture Weight Heuristic: Consistent Performance Across Skill Types Without Validation Data

Figure 1 plots the trade-off between general and skill-specific performance as ω varies from 0.0 to 1.0 for all four settings (science, safety, coding, exaggerated refusals). Each panel shows a curve traced by 10 checkpoints (evenly spaced ω values plus the heuristic point), with the point corresponding to ω = |D| / |G| highlighted. The key finding: the heuristic consistently selects a point on the curve that achieves most of the available skill-specific improvement while preserving nearly all general performance.

For science (Figure 1, left panel), the trade-off is approximately monotonic: general skills decline gradually from ~50 to ~45 as ω increases from 0 to 1, while science performance rises from ~28 to ~37. The heuristic point (ω ≈ 0.22, since |D_science|/|G| = 61,349/275,464 ≈ 0.22) lands near the "elbow" where science gains are large and general losses are small — achieving roughly 34–35 on science while maintaining ~49 on general.

For safety (Figure 1, middle-left), the curve shape is qualitatively different: general performance holds approximately flat until ω ≈ 0.3–0.4, then degrades sharply. The heuristic (ω ≈ 0.24, since 66,161/275,464 ≈ 0.24) selects a point just before this degradation threshold, achieving substantial safety improvement while avoiding the cliff. This validates the heuristic's ability to adapt to nonlinear trade-off geometries.

For coding (Figure 1, middle-right), general performance is nearly invariant to ω until very high values, while coding performance rises smoothly. The heuristic selects a conservative point that prioritizes general preservation while still capturing meaningful coding improvement.

For exaggerated refusals (Figure 1, right), the curve shows rapid improvement in safe prompt compliance at very low ω (XSTest Safe compliance improves dramatically with just a small safety vector addition), then plateaus. The heuristic lands on this plateau, achieving near-maximum exaggerated refusal benefit with minimal general degradation.

The practical significance is that the same formula — with no tuning, no normalization, no skill-specific adjustment — works across four qualitatively different trade-off geometries. The efficiency of PTM (sweeping ω costs negligible compute) means a practitioner can always check whether the heuristic point is reasonable by generating the full curve in seconds, but the heuristic provides a reliable default when even that minimal validation is impossible.

Multi-Skill Merging: Capability Gains with Interference Effects

Table 4 reports results when all three skill-specific task vectors are merged into a single model, using the heuristic ω_j = |D_j| / |G| for each skill individually. The three-skill PTM model achieves: general performance of 51.1 (slightly above the baseline of 49.9), science of 26.6 (down from baseline 27.8 — a regression), coding of 45.3 (up from 37.6), safety of 84.0 (up from 50.3), and exaggerated refusals of 93.2 (down from 99.2 but dramatically better than CFT's 16.0 and RT's 37.2 for comparable multi-skill models). The training cost for PTM is zero additional steps beyond training the three individual skill models — the merge itself is arithmetic. In contrast, CFT on all three skills costs 2,219 additional steps and RT costs 4,732 steps.

The headline positive result is that PTM successfully adds coding and safety capabilities while preserving general skills and avoiding the catastrophic forgetting and exaggerated refusal problems of CFT and RT. The three-skill PTM model's coding (45.3) is competitive with single-skill PTM coding (46.7 from Table 5), and its safety (84.0) is close to single-skill PTM safety (89.3). The exaggerated refusals advantage persists: 93.2 compliance versus CFT's 16.0 and RT's 37.2.

The notable negative result is the science regression. Despite adding the science task vector, science drops from the baseline 27.8 to 26.6. This is not merely a failure to improve — it's active degradation. The paper hypothesizes interference between skill vectors and diagnoses it through pairwise merges in Table 5.

The Pairwise Diagnostic Analysis (Table 5). By merging skill vectors pairwise (at zero additional training cost), the paper isolates the source of science degradation:

Science + Safety merge: Science performance is 31.6 (improvement over baseline 27.8), general skills 50.8, safety 89.1. No interference.
Science + Coding merge: Science performance is 32.1 (improvement), general skills 51.3, coding 45.5. No interference.
Safety + Coding merge: Science performance crashes to 18.8 (from baseline 27.8), even though the science vector is not involved in this merge. General skills are 52.1, safety 85.0, coding 45.2.
All three skills (from Table 4): Science is 26.6, which is the net effect of the science vector trying to improve science (to ~31–32, as seen in the pairwise merges involving science) and the safety+coding interference dragging it down (to ~18.8, as seen in the safety+coding merge). The result is slightly below baseline — the interference partially but not completely counteracts the science vector's benefit.

This diagnostic reveals that the science drop is not caused by the science vector interfering with other vectors, but by the safety and coding vectors mutually interfering in a way that damages the model's science capabilities. Since the safety and coding vectors are not directly targeting science parameters, this must be an indirect effect: safety and coding both modify parameters that the general model also uses for scientific reasoning, and their combined modifications—when added simultaneously—disrupt those capabilities. This is a fundamental limitation of linear merging: task vectors are not truly orthogonal in weight space, and their interactions can produce degradation in capabilities that none of the individual vectors were targeting.

The paper tests whether alternative merging algorithms designed to minimize interference can help (Table 6). TIES (Yadav et al., 2023) uses a three-step process: trim low-magnitude changes, resolve sign conflicts between vectors by taking the majority sign, and then merge. DARE (Yu et al., 2024) randomly drops a fraction of task vector components and rescales the remainder. Both produce results nearly identical to standard weighted averaging across all five metrics. TIES achieves science 28.0 (versus 26.6 for standard PTM) — a marginal improvement that still represents a regression from baseline. DARE achieves 24.7 — worse than standard PTM. Neither meaningfully mitigates the interference. This negative result is informative: it suggests the interference is not due to sign conflicts (which TIES explicitly resolves) but to magnitude-based competition, where multiple task vectors make large changes to the same parameters that collectively damage some capability. No existing merging algorithm addresses this failure mode.

Section 4.3 also includes a fascinating finding about the single-skill PTM results in Table 5: adding the science vector alone improves general performance (50.3 vs. 49.9 baseline), adding the coding vector alone improves general performance (51.0), and adding the safety vector alone improves general performance (50.3). Even the safety+coding merge (which destroys science) improves general skills to 52.1. This pattern — where skill-specific training transfers positively to general capabilities — suggests that the task vectors encode partially generalizable skills that benefit the model beyond their target domain, a phenomenon that the paper does not explore deeply but that has implications for understanding what task vectors actually represent.

Alternative PTM Methods: Task Arithmetic Dominates for General Skill Preservation

Section 4.4 and Figure 3 compare the three PTM variants: task arithmetic (primary method), linear interpolation, and WiSE-FT. The plots show general vs. skill-specific performance for each method as ω varies, and a consistent pattern emerges: linear interpolation and WiSE-FT can achieve very strong skill-specific performance, but at the cost of dramatic general skill degradation compared to task arithmetic at the same ω.

For science (Figure 3, top-left), task arithmetic traces a favorable curve where science rises from ~28 to ~37 while general declines from ~50 to ~44. Linear interpolation shows a much steeper general decline (from ~50 to below ~35) to reach similar science levels, and WiSE-FT is intermediate but still worse than task arithmetic. For safety (top-right), task arithmetic maintains general performance near 50 until high ω, while linear interpolation degrades general skills rapidly and WiSE-FT shows a sharp drop at moderate ω. For exaggerated refusals (bottom panels), task arithmetic achieves high XSTest Safe compliance (>90) while preserving general skills, while linear interpolation and WiSE-FT sacrifice substantially more general performance for equivalent compliance gains.

The paper diagnoses WiSE-FT's poor performance in Figure 2. The hypothesis is that WiSE-FT's task vector τ_CFT = θ_CFT − θ_G captures weight changes induced by continued finetuning on skill-specific data that is distributionally different from the general instruction data. These weight changes are large and include not only skill acquisition but also catastrophic forgetting of general capabilities. Adding back even a fraction ω of these changes imports this forgetting. The paper tests this by comparing standard WiSE-FT (trained on science data only) against WiSE-FT where the CFT step uses a mixture of science data and a matching amount of general Tülu data. The mixed-data WiSE-FT shows both stronger skill-specific performance and much smaller general degradation, confirming that the problem is distribution shift during the CFT step, not the WiSE-FT formula itself.

The practical implication is clear: when θ_pre is available, task arithmetic is the preferred PTM variant. When θ_pre is not available (the setting WiSE-FT was designed for), the practitioner should consider mixing general data into the CFT step if possible, or accept that general skill preservation will be worse than what task arithmetic would achieve. Linear interpolation is not recommended in any setting because its convex combination formulation forces a zero-sum trade-off between general and skill-specific capabilities.

Ablation Studies and Robustness Checks

PRM aggregation strategy (science validation-based vs. heuristic-based model selection): The paper implicitly ablates model selection strategy by reporting both validation-based results (Table 2, where the best checkpoint is chosen using SciRIFF's held-out validation set) and heuristic-based results (Figure 1 and the aggregate comparisons, where ω = |D|/|G| is used without validation data). The validation-based PTM achieves 38.2 science and 47.1 general (Table 2); the heuristic-based PTM in Figure 1 (left panel, highlighted point) achieves approximately 34–35 science and ~49 general. The heuristic is slightly more conservative (preserving more general skills at the cost of less science improvement), but the difference is modest — roughly 3–4 points on science and 2 points on general. This confirms that the heuristic does not dramatically underperform relative to validation-based selection, supporting the paper's claim that it is a practical alternative when validation data is unavailable.

Mixture weight sweep density (5 vs. 10 checkpoints): Figure 1 uses 10 checkpoints (evenly spaced ω values plus the heuristic), while the main experiments in Tables 2–4 use only 5 (ω ∈ {0.2, 0.4, 0.6, 0.8, 1.0}). The curves in Figure 1 are smooth and well-behaved across all four settings, with no evidence of sharp discontinuities that would be missed by coarser sampling. The 5-checkpoint sweep appears sufficient for capturing the general shape of the trade-off curve, though a practitioner optimizing for a specific operating point (e.g., maximum science improvement subject to ≤2% general degradation) might benefit from finer-grained ω search, which PTM enables at negligible cost.

Skill-specific data quantity (varying subsample sizes for CFT and RT): While not a formal ablation in the PTM context, the CFT and RT results implicitly ablate the effect of training data quantity. For CFT science (Appendix C, Tables 7–8), the "Best CFT" model selected by validation performance achieves 40.6 science but only 33.7 general. The existence of other CFT checkpoints with less science data (and presumably less forgetting) is reflected in the full trade-off curves in Figure 6 (Appendix B), which show CFT sweeping a wide range of general-science trade-offs as data quantity varies. The key observation is that no CFT checkpoint achieves both high general and high science performance — the forgetting is monotonic with data quantity, and the only way to get good science is to accept large general degradation. This validates the paper's claim that CFT cannot simultaneously optimize both objectives, a limitation that PTM avoids by decoupling data quantity (always train on all skill data) from influence strength (controlled by ω).

Alternative merging algorithms for multi-skill (TIES and DARE): Table 6 in Appendix B compares standard weighted-averaging PTM against TIES (Yadav et al., 2023) and DARE (Yu et al., 2024) for the three-skill merge. Both alternatives are designed to reduce interference between task vectors through different mechanisms: TIES trims low-magnitude changes and resolves sign conflicts, while DARE randomly prunes and rescales. Neither improves over standard PTM. TIES achieves 28.0 science (vs. 26.6), 44.5 coding (vs. 45.3), and 82.7 safety (vs. 84.0). DARE achieves 24.7 science (worse), 45.4 coding, 84.7 safety. The general skills are 51.2 (TIES), 49.9 (DARE), vs. 51.1 (standard). The science interference problem persists across all methods, and no method provides a meaningful improvement on any metric. This negative result is informative: it suggests the interference is not due to the specific merge arithmetic but is a more fundamental limitation of linear combination when task vectors share non-orthogonal components that collectively damage certain capabilities. The paper does not explore whether simply downweighting the conflicting vectors (reducing ω for safety and coding) could recover science performance, which would be a natural diagnostic step.

General data mixing in WiSE-FT: Figure 2 ablates the effect of including general data during the CFT step of WiSE-FT. Standard WiSE-FT (trained on science only) shows a trade-off curve with strong science improvement but substantial general degradation. WiSE-FT trained on a mixture of science and general data shows both higher peak science performance and much better general preservation. This confirms the diagnosis that WiSE-FT's poor performance relative to task arithmetic is not inherent to the merge formula but stems from the distribution shift during CFT causing weight changes that encode both skill acquisition and catastrophic forgetting. The practical takeaway: if using WiSE-FT (e.g., because θ_pre is unavailable), mixing in general data during the CFT step substantially improves outcomes. The paper does not explore how much general data is sufficient, leaving this as an open question for future work.

Multi-skill merging as implicit ablation of single-skill interference: The pairwise merge experiment in Table 5 functions as an ablation study revealing that safety and coding vectors, when combined, damage science performance even though neither individually does so. This is a non-obvious interaction that would not be predicted from single-skill results alone. The paper takes advantage of PTM's negligible merge cost to perform this exhaustive pairwise analysis, which would be prohibitively expensive under the RT paradigm. This demonstrates a methodological advantage of PTM beyond its training efficiency: it enables cheap diagnostic experiments for understanding skill interactions.

Critical Assessment

The experimental evidence broadly supports the paper's central claims, but several qualifications are necessary.

Claim 1: PTM achieves competitive performance to retraining with 50–95% training efficiency improvement. This claim is supported with specific numbers and appropriate caveats. Table 2 shows PTM matching RT on science (38.2 vs. 37.8) at ~4% of the compute cost — a ~96% training step reduction. Table 3 shows PTM matching RT on safety (−0.13% vs. +0.66% general change, 88.9% vs. 89.6% safety improvement) at ~4.2% of the compute cost — a ~95.8% reduction. However, the coding results tell a more qualified story: PTM achieves +33.3% coding improvement versus RT's +50.7% at ~8.5% of the compute cost — the efficiency claim holds, but the "competitive performance" claim is weaker. PTM leaves substantial coding performance on the table relative to RT.

The dependence on |D|/|G| is critical and underemphasized in the abstract's "50–95%" framing. As skill datasets grow relative to the general mix, two things happen: (1) the efficiency advantage shrinks because |D| (the PTM cost) approaches n·|G| (the RT cost), and (2) the heuristic ω = |D|/|G| pushes ω higher, which Figure 1 shows can move past the optimal trade-off point. The coding dataset at 156,526 examples is already ~57% the size of the general mix, and the efficiency gain is at the lower end of the claimed range. For a hypothetical skill dataset as large as the general mix, PTM's efficiency advantage would largely evaporate (you'd train on |D| ≈ |G| steps either way), and the heuristic would set ω ≈ 1.0, which the paper shows is consistently suboptimal (Figure 1). The "50–95%" range is therefore accurate for the specific datasets tested but is not a universal property of PTM — it's a function of dataset size ratios.

Claim 2: PTM preserves general skills versus a 10–40% drop for CFT, while achieving similar skill-specific performance. The evidence for the forgetting claim is overwhelming. Table 3 shows CFT general degradation of 32.5% (science), 40.1% (safety), and 7.7% (coding), while PTM shows 1.30% degradation (science, actually an improvement), −0.13% (safety, essentially unchanged), and +1.43% (coding, an improvement). The "similar skill-specific performance" part is more nuanced: on science, the CFT model selected by validation achieves 40.6 versus PTM's 38.2 — a 2.4-point gap. On safety, PTM essentially matches CFT (88.9 vs. 98.9, but CFT's higher number reflects the model refusing everything, not genuinely better safety). On coding, CFT substantially outperforms PTM (51.6 vs. 33.3). So the "similar skill-specific performance" holds for science and safety but not for coding, where the dataset is large enough that CFT's aggressive specialization provides real skill gains that PTM's conservative interpolation cannot match.

The individual evaluation breakdowns (Appendix C, Table 9) reveal that PTM's general skill "preservation" is not uniform. For the safety PTM model, AlpacaEval drops from 74.3 to 62.9 (an 11.4-point decline), and BBH drops from 44.8 to 41.2 (3.6 points). These are individually significant degradations that are partially masked by TruthfulQA improving from 48.3 to 65.7, which pulls the average up. A practitioner who cares specifically about open-ended generation quality might find the AlpacaEval drop unacceptable even if the aggregate "General" score looks fine. The average-based reporting obscures per-task variance that could matter in deployment.

Claim 3: PTM reduces exaggerated refusals by 30–80% compared to CFT and RT. This claim is strongly supported and represents the paper's most robust finding. Table 3 (Exaggerated Refusals rows) shows PTM achieving 72.6% safe compliance improvement and 88.9% safety, versus CFT's 98.9% safety but −85.1% general degradation, and RT's 87.2% safety but −39.9% general degradation. The raw XSTest Safe numbers in Table 12 are even starker: PTM maintains 93.2 compliance (baseline 99.2), while the CFT model selected for safety achieves only 7.2 (it refuses 93% of safe prompts), and the RT model achieves 37.2. The mechanism — PTM adds safety behavior at reduced strength (ω < 1) rather than fully overwriting the model's compliance instincts — produces a qualitatively different outcome than training on safety data, which inevitably entangles refusal of unsafe prompts with refusal of safe ones.

However, there is an important selection effect to note. The "Best CFT (Ex. Ref.)" and "Best RT (Ex. Ref.)" rows in Table 3 select models based on two-dimensional improvement (general + exaggerated refusals). But the raw CFT checkpoints include models trained on less safety data that may refuse less aggressively. The paper does not report the full CFT trade-off curve for exaggerated refusals versus general skills in the same way Figure 1 does for PTM. Figure 4 in Appendix B partially addresses this, showing general skills vs. exaggerated refusals for all three methods, and the PTM curve is clearly shifted toward higher safe compliance at equivalent general performance. This figure confirms the claim even accounting for the full range of CFT and RT checkpoints.

Claim 4: The heuristic ω = |D|/|G| selects good checkpoints without validation data. Figure 1 strongly supports this for all four settings. The heuristic point consistently lands in the "good" region of the curve. But the paper does not perform a systematic sensitivity analysis: how much would performance change if ω were 20% higher or lower than the heuristic? The smooth curves in Figure 1 suggest the answer is "not dramatically" for most settings, but a table quantifying the robustness of the heuristic to misspecification would strengthen this claim. Additionally, the heuristic is validated only for the three specific skill datasets tested; whether it transfers to datasets of different sizes, different domains, or different degrees of distribution shift from the general mix is unknown.

Methodological concerns and missing experiments:

Single model family, single scale. All experiments use Llama 2 7B. The paper argues this model is "representative," but weight-space arithmetic could behave differently at larger scales (where models may be more linear in their finetuning dynamics, as observed in the task vector literature) or with different architectures. A replication on Llama 2 13B or a non-Llama architecture (Mistral, Gemma) would substantially strengthen the generalizability claims.
No confidence intervals. All results are point estimates from single training runs. With a 500-example test set for general evaluations, differences of 1–3 points on aggregate metrics may not be statistically significant. The 2.8-point general skill drop for PTM science (49.9 to 47.1 in Table 2) and the 0.4-point science advantage over RT (38.2 vs. 37.8) are within the range where variance from random seed, data ordering, or evaluation noise could matter. This is a standard weakness of LLM evaluation papers but is worth noting given the claim sizes.
The cost model excludes validation and difficulty estimation. The training-step accounting in Section 2 counts only the steps to train the models, not the cost of evaluating intermediate checkpoints or sweeping ω. For RT and CFT, evaluation cost is proportional to the number of checkpoints tested (5 each, evaluated on the full suite). For PTM, the merge itself is cheap, but evaluating 5 merged models still requires running inference on all benchmarks. This inference cost is identical across methods and relatively small compared to training, but it is not zero. More importantly, the paper's science experiments (Table 2) use SciRIFF's validation set for model selection — this validation set exists because the paper chose a skill dataset that happens to have one. The safety and coding experiments use the heuristic precisely because validation data is unavailable, which is the realistic case. The paper's efficiency claims would be less impressive if every new skill required creating a validation set from scratch.
No test-time compute trade-off exploration. The paper compares methods purely in terms of training efficiency and final model quality, but a critical practical consideration is inference cost. PTM produces a single merged model with no inference overhead. But what if CFT produced a better model that could be run with fewer inference-time samples (e.g., greedy decoding vs. best-of-8)? The paper does not explore whether the performance gaps could be closed by giving the cheaper-to-train model more inference compute — a trade-off that Section 7 of the compute-optimal scaling paper (described in the prompt example) addresses centrally. For the coding results, where PTM underperforms RT by a significant margin (+33.3% vs. +50.7%), this is a particularly relevant question: could PTM coding match RT coding with slightly more test-time sampling?
No exploration of why the science vector degrades when merged with safety+coding. Table 5's pairwise diagnostic elegantly identifies that the safety+coding merge causes science to crash from 27.8 to 18.8, but the paper stops at diagnosis. It does not attempt mitigation strategies beyond testing TIES and DARE (which fail). Natural follow-up experiments would include: reducing ω for safety and coding simultaneously to reduce their interference footprint; computing the overlap between the safety and coding task vectors (e.g., cosine similarity) to quantify their non-orthogonality; or merging only the components of each vector that are orthogonal to the other vectors (a projection-based approach). The paper frames this as an open problem, which is fair, but the absence of even preliminary mitigation attempts limits the practical value of the diagnostic.
The exaggerated refusals-safety trade-off is not fully characterized. The paper shows that PTM achieves a better trade-off than CFT and RT, but it does not establish whether PTM's trade-off is optimal or merely better. Could a more sophisticated merging strategy (e.g., skill-specific ω per layer rather than a global scalar) further improve the safe compliance at equivalent safety? Could training the safety model with a different objective (e.g., including explicit safe-compliance examples in the safety dataset) shift the PTM trade-off curve even further? These are beyond the paper's scope but are the natural next questions given the results.
Limited exploration of why the coding skill gap exists. PTM achieves only +33.3% coding improvement versus RT's +50.7% (Table 3). The paper does not diagnose this gap. One hypothesis: the coding dataset (156,526 examples) is large enough that training on it in isolation (PTM) produces a model that is good at coding but whose coding task vector is too "sharp" — it encodes coding-specific patterns that don't transfer well when added to the general model at the conservative ω suggested by the heuristic. RT, by mixing coding data with general data, learns coding in a way that is more compatible with the general instruction-following distribution. Testing whether a higher ω (closer to 1.0) for coding closes the gap with RT at acceptable general degradation cost would be a straightforward diagnostic.

Summary assessment. The paper's experiments are well-designed for a comparative empirical study and support its main claims with appropriate specificity. The efficiency advantage of PTM over RT is convincingly demonstrated for the tested data scales, though the magnitude depends on the ratio |D|/|G| in ways that practitioners should consider. The forgetting-resistance of PTM relative to CFT is unambiguous and robust across all three skill types. The exaggerated refusals advantage is the paper's strongest result and represents a genuinely useful finding for safety-conscious deployment. The multi-skill interference finding is important as a cautionary result and a demonstration of PTM's diagnostic capabilities, though the lack of mitigation strategies limits its immediate practical value. The absence of confidence intervals, the single-model-family limitation, and the unexplored inference-time trade-offs are standard weaknesses that qualify but do not undermine the core contributions. The paper would be strengthened by a scaling analysis (does the PTM advantage persist at 13B or 70B scale?), a deeper investigation of the coding performance gap, and a systematic robustness analysis of the heuristic to dataset characteristics, but these are natural extensions rather than fatal omissions.

6. Limitations and Trade-offs

The Mixture Weight Heuristic Is Validated Only on Three Datasets with Favorable Ratios

The assumption. Section 3.2 introduces the heuristic ω = |D|/|G| as a zero-shot model selection rule for settings without validation data, and Section 4.2 validates it by showing in Figure 1 that it consistently selects a "good" point on the trade-off curve for science, safety, coding, and exaggerated refusals. The heuristic is grounded in the intuition that "the mixture weight should reflect the relative amount of evidence the model has seen for each capability" — the larger the skill dataset relative to the general mix, the more influence the task vector should have.

The consequence. The heuristic's reliability depends on two factors that the paper does not systematically vary: the ratio |D|/|G| itself, and the degree of distribution shift between D and G. For the three datasets tested, |D|/|G| takes values of approximately 0.22 (science), 0.24 (safety), and 0.57 (coding) — all in the regime where the general data substantially outweighs the skill-specific data, producing ω values well below 1.0. This is precisely the regime where the paper shows ω < 1.0 is beneficial (Section 2.3 notes that "ω < 1.0 is better than naively setting ω = 1"). But what happens when |D| approaches or exceeds |G|? The heuristic would set ω close to or above 1.0 — a value the paper's own experiments show is consistently suboptimal (Figure 1 shows that ω = 1.0 produces the worst general skill preservation in all four settings). A hypothetical new skill with a dataset of 500,000 examples would get ω ≈ 1.8 by the heuristic, which is entirely outside the paper's explored range of ω ∈ [0, 2] for Figure 3 but well beyond the [0, 1] range tested in Section 4.1–4.3. The heuristic thus has an implicit domain of applicability — |D| ≪ |G| — that the paper never formalizes, leaving practitioners with large skill datasets without guidance.

Similarly, the heuristic is validated on skill datasets that are thematically distinct from the general mix (science vs. general chat, safety vs. general chat, coding vs. general chat) but still fall within the broad distribution of instruction-following data. If a practitioner wanted to add a skill from a genuinely different distribution — e.g., legal document generation, medical diagnosis, low-resource language translation — the relationship between data size and optimal ω might change because the skill vector τ_D would encode weight changes that are less compatible with θ_G. The paper's finding that WiSE-FT improves dramatically when general data is mixed into the CFT step (Figure 2, Section 4.4) demonstrates that distribution shift matters critically, but the heuristic includes no term for distribution shift. It assumes all skills are equally compatible with the general model given equal data quantities.

What evidence exists in the paper. Figure 1 provides strong validation for the three specific datasets and their associated |D|/|G| ratios, but these ratios span a narrow range (0.22–0.57). The paper does not test the heuristic on artificially varying dataset sizes (e.g., subsampling the skill data to create different |D|/|G| ratios and checking whether the heuristic tracks the optimal ω), which would directly test whether data quantity ratio is the right sufficient statistic. The multi-skill experiment in Table 4 implicitly provides a weak test: when all three skills are merged, each with its own heuristic ω_j, the combined effect is worse than single-skill PTM for science, suggesting that the heuristic does not gracefully compose across skills. The pairwise diagnostic in Table 5 shows that interference between skill vectors — a phenomenon the heuristic makes no attempt to model — determines multi-skill outcomes more than the individual ω_j values.

Mitigation status. The paper acknowledges this limitation implicitly in Section 6, stating that "we take advantage of PTM's negligible cost to test different mixture weights to plot 10 checkpoints from evenly spaced values of ω as well as the heuristic" (Figure 1 caption), suggesting that practitioners should always sweep ω when possible rather than blindly trusting the heuristic. The paper frames the heuristic as a fallback: "selecting models using heuristics" when "held-out data is not available" (Section 3.2). It does not propose a more sophisticated heuristic incorporating distribution shift or dataset characteristics beyond size, and it does not test the heuristic's robustness to varying dataset sizes within the same skill domain. The paper also acknowledges the broader open question: "Many instruction datasets do not have validation sets... and thus how to select models is an open question" (Section 3.2), positioning the heuristic as a partial answer rather than a complete solution.

Multi-Skill Interference Is a Hard Failure Mode with No Working Mitigation

The assumption. Section 4.3 and the multi-skill merging procedure implicitly assume that task vectors for different skills can be added independently — that training a science model, a safety model, and a coding model in isolation and then summing their task vectors into θ_G will independently improve each skill without destructive interference. This is the additive assumption that underlies task arithmetic as defined in Equation 2 and extended to multiple skills in Section 4.3.

The consequence. The additive assumption breaks down when skill vectors modify overlapping parameters in conflicting ways. Table 4 demonstrates this concretely: the three-skill PTM model experiences a science performance regression (27.8 → 26.6) even though the science task vector is being added, not removed. Other skills improve (coding 37.6 → 45.3, safety 50.3 → 84.0), showing the merge is working directionally, but science gets worse. The pairwise diagnostic in Table 5 reveals the mechanism: the safety and coding vectors, when combined, cause science performance to crash from the baseline 27.8 to 18.8 — even though neither vector individually harms science (science+safety = 31.6, science+coding = 32.1, both above baseline). This is a non-local interference effect: two vectors that do not target science parameters nevertheless damage science performance when added together, because their combined modifications to shared parameters disrupt representations that science reasoning depends on.

The practical consequence is that PTM does not provide a reliable composition mechanism. A practitioner who successfully adds safety to their model and successfully adds coding to their model cannot assume that adding both will work. The interference is not predictable from single-skill results — the science regression only appears when safety and coding are both present, which would not be anticipated by looking at the individual task vectors. For a production pipeline where models are expected to accumulate skills over time, this means each new skill addition requires re-evaluating all previously added skills, because the new vector could interact destructively with existing ones in ways that are invisible before the merge. This undermines the modularity promise that makes PTM attractive in the first place: the vision of independently developed skill vectors that can be composed at deployment time.

The severity of this limitation depends on the specific skills being merged. The paper only tests one three-skill combination (science+safety+coding), so it is unknown whether the interference is specific to this combination (e.g., safety and coding both modify parameters related to formal reasoning and structured output, which science evaluation also depends on) or is a general property of any sufficiently large set of task vectors. The fact that general skills improve in the three-skill merge (from 49.9 to 51.1, Table 4) while science degrades suggests that interference is skill-specific rather than uniformly destructive — some capabilities benefit from the combined task vector additions while others suffer.

What evidence exists in the paper. Table 4 (three-skill merge) and Table 5 (pairwise diagnostics) provide direct evidence of interference. The pairwise analysis in Table 5 is methodologically strong: by exhaustively testing all 2^3 − 1 = 7 combinations of the three skill vectors, the paper precisely localizes the interference to the safety+coding interaction. Table 6 tests two alternative merging algorithms designed to mitigate interference — TIES (Yadav et al., 2023) and DARE (Yu et al., 2024) — and finds that neither helps: TIES achieves science 28.0 (vs. 26.6, a marginal improvement that still represents regression from baseline 27.8), and DARE achieves 24.7 (worse). The paper interprets this negatively: "TIES and DARE also do not mitigate interference relative to the base method on science" (Table 6 caption).

Mitigation status. The paper does not attempt any mitigation beyond testing TIES and DARE, both of which fail. It does not explore obvious follow-up strategies: reducing ω for the conflicting vectors (safety and coding) to shrink their interference footprint while preserving some of their benefit; projecting task vectors onto orthogonal subspaces before merging; or pruning task vector components that have high overlap across skills. The paper frames interference as an open problem: "We leave it to future work to explore" follow-up questions including "When the base mix is not publicly available, is it possible to use data from a different general distribution to preserve general performance?" (Section 4.4), but this question is about general performance preservation, not inter-skill interference specifically. The multi-skill interference finding is presented as a diagnostic contribution — "We take advantage of the efficiency of PTM to attempt to diagnose the degraded science performance" (Table 5 caption) — rather than accompanied by a solution. This is a legitimate research contribution (identifying and localizing a failure mode), but it leaves practitioners without a working multi-skill merging recipe.

All Results Are on a Single Model Family at a Single Scale

The assumption. Section 3.2 states that "We run all of our experiments on top of Llama 2 7B (Touvron et al., 2023)." The paper argues in Section 4 that this model is "representative of the capabilities of many contemporary LLMs," but this claim is not tested. The implicit assumption is that the trade-offs between CFT, RT, and PTM — and the specific properties of task arithmetic that produce the paper's findings — generalize across model scales, architectures, and training regimes.

The consequence. Several of the paper's findings plausibly depend on properties of Llama 2 7B that may not hold for other models. The effectiveness of task arithmetic — adding a task vector derived from finetuning θ_pre on D to the instruction-tuned model θ_G — depends on the linearity of the finetuning trajectory in weight space. Prior work on task vectors (Ilharco et al., 2023) has shown that larger models tend to exhibit more linear finetuning dynamics, meaning that the weight changes induced by finetuning on different tasks are more additive and less mutually interfering at larger scales. If this trend holds, the paper's positive results (competitive skill-specific performance, good general skill preservation) might be even stronger at the 13B or 70B scale. Conversely, the multi-skill interference finding (science regression in the three-skill merge) might be weaker at larger scales where task vectors are more orthogonal. Without testing at multiple scales, the practitioner cannot know whether Llama 2 7B represents a worst case, best case, or typical case for PTM.

Architectural differences also matter. Llama 2 uses a standard transformer decoder architecture; models with mixture-of-experts (MoE) layers, different normalization schemes, or different attention mechanisms might have weight spaces with different geometric properties that affect merge arithmetic. The paper tests only full finetuning (Section 3.2: "We fully finetune all of our models"), which is important because parameter-efficient methods like LoRA (Hu et al., 2021) produce task vectors in a lower-dimensional subspace, potentially changing how interference manifests. A practitioner using QLoRA adapters or prefix tuning cannot directly apply the paper's findings without additional experiments.

The single-model-family limitation also affects the practical advice about WiSE-FT. Section 4.4 diagnoses WiSE-FT's poor performance as resulting from distribution shift between the general instruction data and the skill-specific data during the CFT step, and shows in Figure 2 that mixing general data into the CFT step substantially improves WiSE-FT outcomes. But this diagnosis is based on one model (Llama 2 7B) with one general instruction mix (modified Tülu V2). The degree of distribution shift between general instruction data and a new skill dataset depends on the specific datasets involved, and the amount of general data needed during CFT to preserve performance likely varies. The paper does not provide a recipe for determining this amount, which limits the practical applicability of the WiSE-FT variant.

What evidence exists in the paper. The paper provides no cross-model or cross-scale experiments. All tables and figures are based on Llama 2 7B. The argument that the model is "representative" (Section 4) is an assertion without empirical support. The paper also does not test alternative general instruction mixes (e.g., the full unmodified Tülu V2, OpenOrca, or a different mixture altogether), which would help distinguish whether the results are specific to the particular mix used.

Mitigation status. The paper does not address this limitation explicitly. The authors do not claim to have tested multiple model families or scales, and they do not frame the single-model design as a limitation in the main text or the Limitations section. The choice of a single model is a standard practice in empirical LLM papers where training costs make multi-scale experiments expensive (training Llama 2 13B on all three methods across all three skills would be ~2× the compute), but it means the paper's findings are best understood as existence proof — PTM can work well for instruction tuning at 7B scale — rather than as universal scaling laws. A practitioner using a 70B model, a different architecture, or parameter-efficient finetuning would need to replicate key experiments before relying on the paper's heuristics.

Exaggerated Refusals and Safety Performance Are Measured with Imperfect Proxies

The assumption. Section 3.1 describes the safety evaluations: ToxiGen for toxicity avoidance, HarmBench for automated red teaming, XSTest Unsafe for refusal of genuinely unsafe prompts, and XSTest Safe for compliance with safe but superficially dangerous-looking prompts. The paper treats these four metrics — particularly XSTest Safe — as measuring "exaggerated refusals," the failure mode where safety-trained models over-generalize and refuse harmless prompts. The assumption is that high XSTest Safe scores mean the model appropriately complies with safe requests, and that improvements on these benchmark metrics translate to genuine safety improvements in deployment.

The consequence. Safety benchmarks are known to be imperfect proxies for real-world safety. The XSTest dataset (Röttger et al., 2024) contains 250 safe prompts designed to superficially resemble unsafe ones (e.g., "How do I kill a process?" vs. "How do I kill a person?"). A model that scores well on XSTest Safe has learned to distinguish these specific safe prompts from their unsafe counterparts, but this does not guarantee that it will distinguish novel safe prompts that happen to share surface features with unsafe requests. The paper's finding that PTM dramatically improves XSTest Safe compliance (93.2 vs. 7.2 for CFT in Table 12) could reflect either genuine improvement in the model's ability to discriminate safe from unsafe prompts, or more narrow memorization of the XSTest patterns due to the specific way the safety task vector modifies the model's representations.

Similarly, ToxiGen, HarmBench, and XSTest Unsafe measure refusal of specific dangerous prompts, but a model that refuses 100% of HarmBench prompts (as the CFT safety model achieves in Table 12) may still be vulnerable to adversarial attacks, jailbreaks, or dangerous prompts outside the HarmBench distribution. The paper's "safety average" (averaging ToxiGen, HarmBench, and XSTest Unsafe) collapses multiple distinct safety dimensions into a single number, which can mask qualitatively different safety profiles. For example, the PTM safety model in Table 12 achieves 75.2 on HarmBench, 89.5 on XSTest Unsafe, and 95.6 on ToxiGen — a wide spread suggesting it is much better at avoiding toxicity than at resisting adversarial attacks. The CFT safety model achieves 100.0 on all three, but this uniform perfection is suspicious given that it simultaneously achieves only 7.2 on XSTest Safe — it may be blindly refusing everything rather than exhibiting sophisticated safety discrimination.

The exaggerated refusals finding — the paper's strongest result — is particularly dependent on XSTest Safe. If XSTest Safe overestimates real-world compliance with safe prompts (e.g., because the prompts are constructed adversarially to be easy to distinguish given the right training signal), then the paper's claim that PTM reduces exaggerated refusals by 30–80% may not translate to deployment. The mechanism by which PTM achieves high XSTest Safe while maintaining safety (adding a partially-weighted safety vector rather than fully overwriting the model's behavior) may produce a model that is genuinely better at the safe/unsafe distinction, or it may produce a model that happens to retain compliance on the specific safe prompts in XSTest while still being overly cautious on other safe prompts that were not in the benchmark.

What evidence exists in the paper. The paper reports all four safety metrics individually in Table 12 (Appendix C) and as aggregates in the main text, but does not analyze the safety profile in detail beyond the aggregate scores. The qualitative difference between the CFT model's uniform 100.0 scores and the PTM model's uneven 75.2/89.5/95.6 scores is not discussed. The paper does not test the models on out-of-distribution safety prompts, adversarial jailbreaks, or real-world safety scenarios. The evaluation is entirely benchmark-based, which is standard for the field but limits the strength of safety claims.

Mitigation status. The paper does not address this limitation explicitly. The use of standard safety benchmarks (ToxiGen, HarmBench, XSTest) is defensible as the current best practice in the field, and the paper is transparent about reporting the component metrics separately in Appendix C. The exaggerated refusals result is robust within the benchmark framework — PTM consistently achieves higher XSTest Safe at equivalent safety levels across all checkpoints (Figure 4) — but the paper does not provide evidence that benchmark-measured exaggerated refusals correspond to real-world exaggerated refusals. The safety evaluations cover three distinct threat models (toxicity, adversarial attacks, unsafe prompts), which provides some breadth, but the depth of evaluation within each threat model is limited to the specific prompts in each benchmark. A practitioner deploying a PTM safety model would need to conduct their own red-teaming and real-world safety testing before relying on the paper's safety claims.

The Training Efficiency Gains Exclude Inference Cost and Difficulty Estimation Overhead

The assumption. Section 2 measures computational cost in training steps, defined as "how many training steps are required to create the pool of models we select from." PTM's efficiency advantage — 50–95% fewer training steps than RT — is derived from this metric, which counts only the cost of forward and backward passes during training. The paper does not account for inference cost during evaluation of checkpoints, the cost of the merge operation itself, or any cost associated with estimating dataset characteristics needed for the heuristic.

The consequence. While training dominates total compute for the RT baseline (reprocessing the full general mix multiple times), the efficiency comparison becomes less favorable to PTM when the full lifecycle cost is considered. For CFT and RT, evaluating checkpoints requires running inference on the evaluation suite; for PTM, the same evaluation is needed to sweep ω and select the final model. This evaluation cost is identical across methods and proportional to the number of checkpoints tested (5 in the main experiments, 10 in Figure 1), making it a fixed overhead rather than a differential cost. However, the paper's training-step accounting implicitly assumes this cost is zero, which is a minor overstatement of PTM's advantage.

More importantly, the heuristic ω = |D|/|G| requires knowing |G|, the number of training steps for the general instruction mix. For publicly released models where the general mix is unknown (the very setting where the paper argues PTM is most valuable, since RT is impossible), the practitioner cannot compute |G| without knowing the training recipe. Section 2.2 explicitly states that "retraining is not possible in cases where the pretrained and instruction-tuned models have been released but the general instruction mix has not, such as Llama 3 (AI@Meta, 2024), Mistral 7B (Jiang et al., 2023) and Gemma (Gemma-Team et al., 2024)." But the heuristic requires |G|, which is also unknown for these models. The practitioner would need to estimate |G| from public information about the training recipe (e.g., number of training tokens, batch size, number of epochs) — information that is not always disclosed. For Llama 3, Meta reported the number of pretraining tokens but not the exact composition or size of the instruction-tuning mix. For Mistral 7B, the instruction-tuning data is entirely proprietary. The heuristic is thus not truly "zero-shot" for the models where PTM's data-access advantage is most relevant; it requires metadata that may be unavailable or approximate.

Additionally, the paper's cost model in Section 2.3 claims that PTM's total cost is |D| training steps, and that sweeping ω is free: "in PMT, this is accomplished through the mixture weighting parameter ω. This means that the total training cost for PMT is |D|, dramatically lower than other methods." The merge operation itself — element-wise addition and scalar multiplication on 7B parameters in BFloat16 — is indeed negligible (seconds on a CPU). But evaluating each merged checkpoint on the full evaluation suite (5 general benchmarks + 2–9 skill-specific benchmarks, each requiring inference on hundreds to thousands of examples) is not negligible. For a 7B model, running inference on the Tülu evaluation suite takes hours on a single GPU. Evaluating 5 checkpoints (as in the main experiments) or 10 (as in Figure 1) adds non-trivial inference cost that is not captured in the training-step accounting. This cost is identical across CFT, RT, and PTM (all require evaluating the same number of checkpoints), so it does not change the relative ranking, but it does mean that the absolute efficiency advantage of PTM is smaller than the training-step numbers alone suggest.

What evidence exists in the paper. The paper is transparent about the cost model's scope: Section 2 defines training complexity as "how many training steps are required to create the pool of models we select from," explicitly limiting the metric to training. The paper does not report inference FLOPs, wall-clock time, or total compute including evaluation. The training step counts in Tables 2–4 are precise and well-documented, and the efficiency ratios derived from them are mathematically correct under the stated metric. The paper does not claim to measure total lifecycle cost; it claims only training efficiency.

Mitigation status. The paper partially acknowledges the evaluation cost implicitly by noting that PTM's efficiency advantage comes from decoupling training from model selection: "In PMT, this is accomplished through the mixture weighting parameter ω" (Section 2.3). The paper does not propose a method for reducing evaluation cost (e.g., using a smaller proxy evaluation set, early stopping during ω sweeps, or predicting merge quality without inference). The fact that evaluation cost is identical across methods means the relative training-step advantage is a fair comparison, but the absolute numbers (50–95% reduction) apply only to training compute, not total compute. For a practitioner optimizing total cost, the true savings depend on the ratio of training to evaluation compute, which varies with model size, evaluation suite size, and hardware. The heuristic ω = |D|/|G| partially addresses evaluation cost by providing a default ω that avoids the need for a sweep, but as discussed in the first limitation, the heuristic is not validated for all dataset regimes and the practitioner may still want to verify it with at least a few evaluations.

Only Single-Turn Instruction Following Is Tested; Multi-Turn and Conversational Skills Are Not Evaluated

The assumption. Section 3.1 describes the training and evaluation datasets as consisting of single-turn instruction-response pairs. The general Tülu V2 mix, SciRIFF, the safety dataset, and CodeFeedback all follow this format: a single prompt and a single response, with no dialogue history or multi-turn interaction. The implicit assumption is that the findings about PTM — its ability to add skills while preserving general capabilities — generalize to multi-turn conversational skills, which constitute a large fraction of real-world LM deployments.

The consequence. Multi-turn interaction introduces several complications that do not arise in single-turn settings. The model must maintain conversational context across turns, track long-horizon goals, and adapt its responses based on previous exchanges with the user. These capabilities depend on representations that may be more fragile to weight-space perturbation than single-turn instruction following. Adding a coding skill vector via PTM might, for example, preserve the model's ability to write a Python function when prompted directly, but degrade its ability to iteratively refine code in response to follow-up feedback — a multi-turn coding capability that depends on both coding skill and conversational coherence.

Safety in multi-turn settings is particularly challenging. A model that correctly refuses a dangerous request in the first turn might be vulnerable to multi-turn jailbreaks where an adversary gradually steers the conversation toward dangerous territory across multiple seemingly harmless exchanges. The paper's safety evaluation uses only single-turn prompts from ToxiGen, HarmBench, and XSTest. A PTM safety model that achieves 75.2 on HarmBench (single-turn adversarial prompts) might perform substantially worse against multi-turn adversarial strategies that exploit the model's retained conversational compliance — the very capability that PTM preserves by keeping ω < 1.0. The exaggerated refusals finding (PTM maintains XSTest Safe compliance much better than CFT) could cut both ways in multi-turn settings: the model is less likely to refuse a safe but superficially suspicious prompt, but it may also be more susceptible to multi-turn manipulation that exploits this retained compliance.

More broadly, the single-turn limitation means the paper has not tested whether PTM preserves capabilities that are inherently multi-turn, such as information gathering through clarification questions, negotiation, or interactive teaching. These skills are increasingly important as LMs are deployed as agents, tutors, and assistants, and there is no evidence that single-turn skill vectors transfer benignly to multi-turn settings. A practitioner adding a new skill to a conversational agent would need to evaluate multi-turn performance explicitly, which the paper does not provide a methodology for.

What evidence exists in the paper. All evaluation benchmarks are single-turn. General evaluations include AlpacaEval (single-turn instruction following judged by an LLM) and BBH (single-turn reasoning), but no multi-turn dialogue benchmarks. The science evaluations test information extraction and question answering from scientific text, not interactive scientific reasoning. The coding evaluations (HumanEval+, MBPP+) test function synthesis from a single docstring, not conversational code generation or iterative debugging. The safety evaluations are entirely single-turn refusal checks. The paper provides no multi-turn evaluations and does not discuss the single-turn limitation.

Mitigation status. The paper does not address this limitation. The choice of single-turn datasets and evaluations reflects the current state of the instruction-tuning literature, where most publicly available instruction mixes (Tülu V2, OpenOrca, CodeFeedback) and benchmarks (AlpacaEval, MMLU, HumanEval) are single-turn. Multi-turn instruction tuning is an active research area, and extending PTM to multi-turn skills would require both multi-turn training data and multi-turn evaluation benchmarks, which are less standardized than their single-turn counterparts. The paper's contribution is explicitly scoped to the instruction-tuning paradigm as currently practiced, and the single-turn limitation is a constraint of the problem setting rather than a flaw in the methodology. However, a practitioner should treat the paper's findings as applying specifically to single-turn skill addition and should not assume that multi-turn behavior is equally preserved by PTM without separate evaluation.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reframes skill addition for instruction-tuned language models from a data mixture problem to a weight-space composition problem. The shift is conceptual rather than algorithmic — the paper does not propose a new merging method — but it has practical consequences that change the default approach a practitioner should consider when a new skill dataset appears.

The key insight is that training cost and model selection can be decoupled if skill influence is controlled through a post-hoc scalar ω rather than through data mixing ratios or number of finetuning steps. This decoupling is what produces the 50–95% training efficiency gains over retraining, but its significance goes beyond efficiency. It means that the trade-off between acquiring a new skill and preserving general capabilities can be explored after training is complete, at essentially zero computational cost, by sweeping ω across a range of values and evaluating the merged checkpoints. In the traditional retraining paradigm, each point on the trade-off curve requires a separate expensive training run with a different data mixture. In the continued finetuning paradigm, each point requires training for a different number of steps on the skill data, and the forgetting is monotonic — once general skills are lost to CFT, they cannot be recovered without retraining on the original data. PTM's post-hoc ω sweep transforms model selection from a training-time optimization problem into an inference-time search over a single continuous parameter, which is fundamentally cheaper.

This reframing also resolves a practical contradiction in the instruction-tuning community. On one hand, practitioners want to keep their deployed models current by incorporating new publicly available instruction datasets as they are released. On the other hand, the cost of retraining from scratch on the combined data mixture grows linearly with the number of new datasets — each addition requires reprocessing all previous data. This tension has led to an ad-hoc ecosystem where models are either frozen after initial training (missing out on new capabilities) or continually finetuned (suffering from forgetting). PTM offers a third path: maintain a library of independently trained skill-specific task vectors, and compose them into the base model at deployment time. The paper shows that this composition works for single-skill addition (Tables 2–3) and partially works for multi-skill addition (Table 4, with interference caveats), establishing the viability of a modular development workflow where skill-specific training can happen in parallel across different teams without coordination on data mixtures.

The safety findings — that PTM reduces exaggerated refusals by 30–80% compared to CFT and RT while maintaining comparable safety performance (Table 3, bottom rows; Table 12) — suggest that weight-space merging may be particularly valuable for skills that conflict with general capabilities. In standard finetuning, safety training inevitably entangles refusal of dangerous prompts with refusal of superficially similar safe prompts, because the gradient updates that encode refusal behavior also erode the representations that support appropriate compliance. PTM partially decouples these entangled behaviors: the safety task vector τ_safety encodes the direction in weight space that produces refusal, but adding only a fraction ω of this vector attenuates its impact on compliance-related parameters. This is not a solution to the alignment tax — the PTM safety model still shows some decline in general skills (AlpacaEval drops from 74.3 to 62.9 for the safety PTM, Table 9) — but it shifts the trade-off frontier relative to what data mixing can achieve. This finding redirects attention toward understanding why weight-space composition produces different trade-off geometries than data-space mixing, which is a question about the structure of neural network loss landscapes that the paper opens but does not answer.

The multi-skill interference diagnostic (Tables 4–6) introduces a methodological capability that is unique to the PTM framework: exhaustive pairwise analysis of skill interactions is now feasible because merging has negligible cost. In the RT paradigm, testing all 2^k − 1 combinations of k skills would require an exponential number of expensive retraining runs. With PTM, this is a matter of running merge arithmetic and inference on the combined models. The paper demonstrates this by localizing the science degradation in the three-skill merge to the interaction between safety and coding vectors (Table 5), and by testing alternative merging algorithms (TIES, DARE) in minutes rather than GPU-days (Table 6). This diagnostic capability changes how one would approach multi-skill model development: rather than training a monolithic model on a combined data mixture and hoping skills don't interfere, one can pre-emptively test for interference between skill vectors and adjust the merge strategy accordingly. The paper does not provide mitigation strategies, but the diagnostic itself is a contribution that enables a more systematic approach to skill composition.

The paper's finding that alternative merging algorithms (TIES, DARE) do not mitigate interference (Table 6) redirects research attention away from merge-arithmetic refinements and toward understanding and controlling skill vector geometry. TIES resolves sign conflicts between vectors, and DARE randomly prunes vector components — neither addresses the core problem that safety and coding vectors make large-magnitude changes to overlapping parameters in ways that disrupt science performance. This suggests that the interference is not an artifact of the specific merge formula but a more fundamental property of how task vectors interact in weight space when they are not approximately orthogonal. Future work on multi-skill merging should prioritize measuring and enforcing orthogonality between skill vectors, or developing non-linear composition mechanisms, rather than refining the linear combination formula.

Follow-Up Research This Work Enables

Projection-based multi-skill merging to mitigate interference. The pairwise diagnostic in Table 5 reveals that the safety and coding vectors, when added together, damage science performance even though neither individually interacts with science. This is consistent with the safety and coding vectors sharing non-orthogonal components that, when summed, produce large-magnitude changes to parameters that science reasoning depends on. A natural follow-up would be to project each task vector onto the subspace orthogonal to all other task vectors before merging: compute the singular value decomposition of the stacked task vectors, identify directions with high overlap across skills, and retain only the orthogonal components of each vector. The hypothesis is that the orthogonal components encode skill-specific capabilities while the shared components encode general modifications (e.g., changes to token representations) that cause interference when summed. A successful experiment would show that projected merging recovers the single-skill science performance (27.8 baseline, or ideally the 31–32 achieved in pairwise merges with science) in the three-skill merge setting while maintaining coding and safety gains. This would directly test whether interference is driven by vector overlap in weight space, and would produce a practical multi-skill merging recipe. The paper's existing pairwise diagnostic framework (Table 5) provides the baseline numbers for this comparison at zero additional training cost.

Scaling analysis of PTM trade-offs across model sizes. The paper's results are entirely on Llama 2 7B, but prior work on task vectors (Ilharco et al., 2023) has shown that larger models exhibit more linear finetuning dynamics — weight changes for different tasks become more additive and less mutually interfering. A natural extension would replicate the PTM vs. CFT vs. RT comparison at 13B and 70B scales (using Llama 2 or a comparable model family) on the same science, safety, and coding datasets. The key measurements would be: (1) whether the PTM general skill preservation advantage over CFT grows or shrinks with scale (if larger models are more linear, the additive assumption in Equation 2 becomes more valid, and PTM should improve relative to CFT); (2) whether the multi-skill interference documented in Tables 4–5 diminishes at larger scales (if task vectors become more orthogonal, the science regression in the three-skill merge should shrink or disappear); and (3) whether the coding performance gap between PTM and RT (+33.3% vs. +50.7% in Table 3) closes at larger scales. A finding that interference shrinks with scale would make PTM's modular development vision substantially more attractive for large-scale deployment, while a finding that interference persists would redirect attention toward the projection and orthogonality approaches described above. The experiment requires training general models and skill-specific models at each scale, which is expensive but feasible with sufficient compute.

Skill-specific ω per layer or per module. All experiments in this paper use a single global scalar ω applied uniformly to all parameters of the task vector (Equation 2: θ_final = θ_G + ω · τ_D). This is a coarse control: it assumes that all layers and all parameter types (attention weights, MLP weights, layer norms) should be influenced equally by the new skill. However, prior work on task vectors (Ilharco et al., 2023) and mechanistic interpretability suggests that different layers encode different types of knowledge — lower layers tend to encode syntactic and surface-level features while upper layers encode semantic and task-specific features. A natural follow-up would sweep layer-specific or module-specific ω values to see whether finer-grained control improves the general-skill trade-off. Concretely: train the skill model as before, but at merge time, apply different ω values to the attention parameters, the MLP parameters, and the layer norm parameters of each transformer block, or even a separate ω per block. The experiment would measure whether the Pareto frontier in Figure 1 shifts outward when ω is parameterized more richly — i.e., whether the model can achieve higher skill-specific performance at the same general performance, or higher general performance at the same skill-specific performance, compared to the global ω baseline. This is computationally cheap with PTM (once the skill model is trained, sweeping a per-layer ω grid requires only inference evaluation, no retraining) and could produce a practical recipe for maximizing skill acquisition while minimizing interference. The paper's existing evaluation suite (Tables 7–12) provides the metrics for this comparison.

Training data for skill vectors: how much is enough? The paper's heuristic ω = |D|/|G| ties the mixture weight to training data quantity, implicitly assuming that the optimal ω increases monotonically with dataset size. But the relationship is likely non-monotonic: very small datasets may produce noisy task vectors that should be heavily downweighted regardless of their size relative to G, while very large datasets may produce task vectors that are so different from the general distribution that even small ω values cause large interference (as seen with WiSE-FT in Figure 3, where skill-specific training on a narrow distribution produces weight changes that damage general skills even when downweighted). A systematic study would fix a single skill (e.g., science, using SciRIFF subsamples) and vary |D| from ~1,000 to ~100,000 examples, training a task vector at each size and evaluating the optimal ω (the ω that maximizes the sum of general and science performance) at each data quantity. The result would be a curve mapping dataset size to optimal ω that could refine the heuristic — perhaps ω ∝ sqrt(|D|/|G|) for very small datasets (to downweight noisy vectors) and ω ∝ log(|D|/|G|) for very large datasets (to dampen interference). This experiment would also test whether the quality of the task vector saturates at some dataset size, which would inform practitioners about how much skill-specific data is worth collecting before diminishing returns set in.

Combining PTM with parameter-efficient finetuning. The paper uses full finetuning for all models (Section 3.2), but many practitioners use parameter-efficient methods like LoRA (Hu et al., 2021) for cost reasons. The interaction between PTM and LoRA is non-obvious: LoRA task vectors live in a low-rank subspace of the full weight space, which could either reduce interference (because the vectors have fewer dimensions in which to overlap) or increase it (because the low-rank constraint forces different skills into a shared subspace). A concrete experiment: train LoRA adapters (rank 8, 16, 64) for science, safety, and coding on top of the same pretrained Llama 2 7B base, merge them using task arithmetic in the LoRA weight space (i.e., compute τ_D as the difference between the adapted and base LoRA matrices), and sweep ω as in the paper. Compare against the full-finetuning PTM results in Tables 2–3 on both single-skill and multi-skill performance. The hypothesis would be that low-rank LoRA vectors interfere less (better multi-skill science performance than Table 4's 26.6) but also achieve lower peak skill-specific performance (worse single-skill scores than full finetuning). The result would inform the growing number of practitioners who use LoRA for instruction tuning about whether PTM is a viable skill-addition strategy in the low-rank regime.

Applying PTM to safety benchmarks with adversarial multi-turn evaluations. The paper's safety evaluation is entirely single-turn (ToxiGen, HarmBench, XSTest), but real-world safety failures often involve multi-turn adversarial strategies where an attacker gradually steers a conversation toward dangerous territory. A critical stress-test of PTM safety would evaluate the merged safety models on multi-turn jailbreak benchmarks (e.g., the multi-turn subset of HarmBench, or custom multi-turn adversarial prompts that chain superficially safe queries to elicit dangerous responses). The concern is that PTM's retained conversational compliance — the very property that produces the strong exaggerated refusals results (93.2 XSTest Safe in Table 12) — might make the model more susceptible to multi-turn manipulation than the CFT safety model, which refuses nearly everything (7.2 XSTest Safe) and thus resists multi-turn attacks by shutting down the conversation early. A finding that the PTM safety model is more vulnerable to multi-turn jailbreaks than the CFT model would reveal a previously invisible trade-off: single-turn compliance and multi-turn robustness may conflict, and PTM's advantage on the former may come at the cost of the latter. Conversely, a finding that PTM maintains multi-turn robustness (perhaps because the safety vector encodes refusal behavior that generalizes across turns, even at reduced ω) would substantially strengthen the paper's safety claims. This experiment requires constructing or adopting multi-turn safety benchmarks and running inference on the existing PTM checkpoints (which have already been trained), making it a low-cost but high-information follow-up.

Practical Applications and Downstream Use Cases

Modular model development for organizations with multiple specialized teams. In a large organization, different teams may independently develop instruction-tuning datasets for different capabilities — a science team builds SciRIFF, a safety team builds refusal data, a coding team builds CodeFeedback. In the current paradigm, these datasets must be combined into a single training mixture and a single model trained, requiring coordination between teams on data ratios, formatting conventions, and training schedules. PTM enables a modular alternative: each team trains their skill-specific model independently on top of the shared pretrained checkpoint, produces a task vector, and releases it. The final deployed model is assembled by merging the general instruction-tuned model with the desired set of skill vectors at deployment time, with per-skill ω values controlling each skill's influence. The paper's results show this works for single-skill addition (Tables 2–3) and partially for multi-skill addition (Table 4), with the interference diagnostics in Table 5 providing a framework for identifying which skill combinations are problematic. The 50–95% training efficiency gains mean each team can iterate on their skill dataset without retraining or even loading the general model, and the post-hoc ω sweep means the trade-off between skills can be adjusted after all training is complete — if users report that the model is refusing too many safe requests, the safety ω can be reduced without touching any other skill. The paper's finding that general skills are preserved within a few percentage points of baseline (Table 3, PTM rows) means skill teams do not need to worry about degrading capabilities they are not targeting.

Safety deployment with reduced exaggerated refusals in production chatbots. A persistent problem in deployed chatbots is that safety training causes the model to refuse harmless queries that superficially resemble dangerous ones — asking "How do I kill a process in Linux?" triggers a refusal because the model over-generalizes from "How do I kill a person?" The paper's safety results (Tables 3 and 12) show that PTM with task arithmetic achieves comparable safety performance to CFT and RT (88.9% safety improvement vs. 89.6% for RT) while maintaining 93.2% compliance with safe prompts (XSTest Safe, Table 12), compared to 7.2% for the best CFT safety model and 37.2% for the best RT safety model. In absolute terms, the PTM safety model correctly complies with approximately 13× more safe prompts than the CFT model while refusing genuinely dangerous prompts at comparable rates. For a production chatbot handling millions of queries per day, reducing the false refusal rate from 93% to 7% — while maintaining safety — translates directly to user satisfaction and task completion. The deployment recipe: train a safety-specific model on refusal data using the pretrained checkpoint, compute τ_safety, merge with the production instruction-tuned model at ω ≈ 0.24 (the heuristic for the paper's safety dataset size), evaluate on XSTest to verify safe compliance, and deploy. If exaggerated refusals are still higher than desired, reduce ω further — this costs no retraining and takes seconds.

Efficient skill addition when the original training data is unavailable. Many widely-used open-weight models — Llama 3, Mistral 7B, Gemma — release model weights but not their instruction-tuning datasets. When a new skill dataset is released, the practitioner of such a model cannot retrain from scratch (RT) because they do not have the original general instruction mix. CFT is possible but causes catastrophic forgetting (32–40% general skill degradation in Table 3). PTM via task arithmetic requires only the pretrained checkpoint (which is available for all these models), the instruction-tuned model (available), and the new skill dataset. The paper's results validate this workflow: train a skill model by finetuning the pretrained checkpoint on the new skill data, compute τ_D, and add it to the instruction-tuned model with ω set by the heuristic (which requires estimating |G| from public training recipe metadata — e.g., number of instruction-tuning tokens, batch size, epochs). The paper's finding that PTM preserves general skills within ~1% of baseline (Table 3, PTM rows for safety and coding) while achieving substantial skill-specific improvements means that even without access to the original training data, a practitioner can keep their deployed model current with new capabilities as datasets are released. When the pretrained checkpoint is also unavailable (which is rare for open models), WiSE-FT with general data mixed into the CFT step (Figure 2) provides a fallback, though with worse general skill preservation than task arithmetic.