Composable Interventions for Language Models

🎯 Pitch

When you stack model compression, knowledge editing, and unlearning on the same LLM, compression reliably sabotages the other two—and the sequence in which you apply them can make or break success. Standard utility metrics like MMLU completely miss these destructive interactions, revealing that today’s popular interventions are dangerously incompatible in real-world pipelines where multiple fixes are needed over time.

1. Executive Summary

This paper studies how multiple post-training interventions—knowledge editing, model compression, and machine unlearning—interact when applied sequentially to the same language model, introducing composable interventions, a framework with two new metrics (Order-free Error and Order Sensitivity) that measure whether interventions succeed regardless of their application order and whether their combined performance depends on ordering. Using Llama-3 8B as the primary testbed across 417 different compositions of ten methods (including MEMIT, LoRA, and Finetuning for editing; Wanda, SparseGPT, GPTQ, and AWQ for compression; and RMU, Gradient Ascent, and Gradient Difference for unlearning), the paper reveals that model compression consistently hinders editing and unlearning success, the order in which interventions are applied dramatically alters their outcomes, and standard utility metrics like MMLU fail to capture composability, establishing that existing interventions are not inherently composable and that new methods explicitly designed for multi-intervention settings are needed.

2. Context and Motivation

The Core Problem: Interventions Are Developed in Isolation, But Deployed in Combination

The central gap this paper addresses is deceptively simple: post-training interventions for language models are designed and evaluated independently, but in practice, multiple interventions must be applied to the same model over its lifecycle. The paper identifies this as a fundamental disconnect between how interventions are studied and how LMs are deployed.

Consider a realistic deployment scenario the authors sketch in Section 1: an organization fine-tunes a large language model, then quantizes it for efficient deployment on mobile devices. Weeks later, a factual error is discovered (e.g., the model gives an outdated answer about a recent regulatory change), requiring a knowledge editing intervention. Months later, concerns arise about the model retaining dangerous knowledge from its pretraining data, prompting a machine unlearning step. Each of these interventions—compression, editing, unlearning—was developed in isolation by different research communities, evaluated on its own benchmarks, and tuned to preserve general performance under the assumption that it would be the only modification applied post-training. But here, they are applied sequentially to the same model. What happens to the edited fact after quantization? Does unlearning undo the knowledge edit? Does compressing the model first prevent successful unlearning later?

The paper's core contention is that we lack both a conceptual framework and empirical evidence to answer these questions. As the authors state in the introduction:

"Despite a flood of new methods, different types of interventions are largely developing independently. In practice, multiple interventions must be applied sequentially to the same model, yet we lack standardized ways to study how interventions interact."

This gap is not merely academic. It speaks directly to the practical reality of maintaining deployed models over time, where requirements evolve, resources shift, and new capabilities or safety guarantees must be retrofitted onto existing systems.

Why This Problem Matters: Multiple Driving Forces

The paper motivates the importance of composability along four concrete axes, each representing a real-world force that pushes organizations toward applying multiple interventions (Section 1):

1. Factual decay and changing knowledge. The world changes after models are trained. The paper gives the example of a model trained before 2020 predicting "Swine Flu" as the latest US pandemic when the correct answer is now "COVID." This is not a hypothetical—regulatory landscapes shift (the paper cites a 2023 Executive Order on AI safety), scientific consensus updates, and basic facts become outdated. Knowledge editing methods exist to patch these errors, but they are rarely the only intervention a production model receives.

2. Computational and deployment constraints. A model trained on thousands of GPUs may need to run on a mobile device. Model compression—whether through quantization (reducing numerical precision) or pruning (removing weights)—is "nearly universal in LM deployments" (Section 4). The paper argues that compression is not optional for most real-world use cases: the gap between training-scale compute and inference-scale compute necessitates compression, meaning that nearly every other intervention will inevitably be composed with compression at some point.

3. Safety and legal compliance. LMs can memorize copyrighted material from their training data, learn harmful capabilities (e.g., bioweapons knowledge), or generate toxic content. Machine unlearning methods have emerged to address these concerns by removing undesirable knowledge post-training. As legal frameworks around AI evolve and harmful model behaviors are discovered, unlearning interventions become necessary retrofits on already-deployed models—models that may have already been compressed and edited.

4. Evolving user and regulatory requirements. The paper emphasizes that requirements are not static. A model may be edited to fix a factual error, then unlearned to remove dangerous knowledge, then further compressed to meet new latency targets. Each intervention responds to a different stakeholder need at a different point in the model's lifecycle, and these needs accumulate over time.

These four drivers are not speculative—they represent documented trends in the field, and the paper argues they are converging to make intervention composability an increasingly critical practical concern. As the authors put it:

"Without ways to quickly address these issues, models can be left unsafe, outdated, biased, and non-compliant with laws or regulations, limiting their widespread responsible use."

Where Prior Approaches Fall Short

The paper identifies five specific limitations in how the field has approached interventions until now:

1. Interventions are developed in silos. Knowledge editing, model compression, and machine unlearning are studied by largely distinct research communities with their own conferences, benchmarks, and evaluation conventions. The paper notes that these communities "have largely developed independently" (Appendix B). This means methods are optimized for single-intervention performance—an editor is judged by whether it successfully updates a fact without degrading unrelated knowledge, a compression method by whether it maintains perplexity at a given bit-width, and an unlearning method by whether it reduces accuracy on the forget target while retaining MMLU performance. None of these evaluations consider what happens when another intervention follows or precedes.

2. No standardized framework for multi-intervention evaluation. Even when researchers want to study how interventions interact, they lack the conceptual vocabulary and quantitative metrics to do so systematically. Prior to this paper, there was no formal definition of what it means for interventions to "compose well," no agreed-upon metrics for measuring composability, and no unified codebase that implemented multiple intervention types in a consistent evaluation harness. The authors explicitly identify this as a bottleneck:

"We lack formal notions of intervention composition."

This absence means that when interactions are observed (or suspected), they are often treated as one-off findings rather than systematic phenomena.

3. Prior work on intervention interactions is sparse and narrow. The paper acknowledges some existing studies that touch on related territory. For instance, works have examined whether compression causes models to "forget" certain types of examples (Hooker et al., 2019; Du et al., 2021; Azeemi et al., 2023; Hoang et al., 2023), whether model sparsity can simplify machine unlearning (Jia et al., 2023), and how knowledge editing can impact model robustness to distribution shifts (Brown et al., 2023a; Gu et al., 2024b). There is also prior work studying trade-offs between training objectives like fairness and privacy (Matzken et al., 2023) and methods that combine compression with subsequent prompting (Xu et al., 2023b) or fine-tuning (Li et al., 2024b).

However, these prior studies are limited in scope. They typically examine only two specific methods in one direction, often within a single category. None provide a cross-category analysis spanning editing, compression, and unlearning simultaneously. The paper argues that:

"While some works have started studying interactions between training objectives, practical widespread use is limited without unified evaluations for how interventions interact."

4. The impact of intervention ordering is unstudied. Perhaps most critically, prior work had not systematically investigated whether the order in which interventions are applied matters for their combined success. This is a non-trivial question: if compression degrades knowledge representations, editing after compression might be harder than editing before compression. Conversely, an edit that modifies a small number of parameters might be "pulled apart" by subsequent quantization that reduces those parameters' precision. Without studying both application orders—and without metrics that explicitly capture order-dependent effects—the field had no way to reason about these dynamics. The paper introduces Order Sensitivity (Section 2.2, Equation 2) as the first metric specifically designed to quantify how much intervention success depends on application order.

5. General utility metrics mask composability failures. The standard practice across all three intervention categories is to report general model utility (typically MMLU accuracy or perplexity on a held-out corpus like WikiText) alongside task-specific metrics. The implicit assumption is that if MMLU is preserved, the intervention hasn't caused catastrophic damage. The paper challenges this assumption directly, finding that MMLU scores can be nearly identical across compositions that have dramatically different intervention-specific performance. As noted in Section 4.1:

"Despite similar MMLU scores, methods vary significantly in composability. For example, Table 2 shows that MEMIT and Wanda achieve near-perfect Order Sensitivity on MMLU but have 0.24 Order Sensitivity on Edit Success."

This finding—that MMLU is insufficient as a composability metric—is both a critique of current evaluation practices and a call for the new metrics the paper proposes.

How This Paper Positions Itself

The paper does not claim to solve the problem of intervention composability. Rather, it positions itself as providing the conceptual scaffolding, quantitative tools, and empirical baseline that the field needs to even begin addressing it systematically. This is evident from several deliberate framing choices:

Framework, not a method. The paper's primary technical contribution is not a new intervention method but a framework for evaluating existing ones. The two composability metrics—Order-free Error and Order Sensitivity—are designed to be general: they can be applied to any combination of interventions, any set of evaluation criteria, and any choice of hyperparameters. The authors explicitly note that their framework "naturally extends to a broad range of test-time interventions" (Section 5), positioning it as infrastructure for future work rather than a one-time analysis.

Empirical diagnosis, not a solution. The bulk of the paper's contribution is its extensive experimental campaign—417 compositions across ten methods—that establishes a baseline of composability behavior for current state-of-the-art interventions. The findings are primarily diagnostic. Compression hinders editing and unlearning. Order matters dramatically. MMLU is a poor proxy. These are problems documented, not solved. The paper frames its findings as "clear gaps in composability, suggesting a need for new multi-objective interventions" (Abstract), explicitly deferring the development of such methods to future work.

Unified codebase as enabling infrastructure. A significant practical contribution is the release of an "extendable codebase that unifies many state-of-the-art implementations" (Section 1) that implements all ten intervention methods in a consistent interface. The authors acknowledge that "a major lack of standardized code ... has inhibited cross-intervention evaluations until now" (Section 1). By providing this codebase, the paper lowers the barrier for other researchers to investigate composability of additional methods, models, and intervention categories.

Connecting intervention categories that rarely interact. Perhaps the paper's most distinctive positioning is that it explicitly bridges three intervention categories that the authors argue have developed independently. This is not incidental to the contribution—it is the contribution. By showing that compression methods affect editing success, that unlearning methods vary in their robustness to ordering, and that editing and unlearning can be surprisingly composable when the right methods are chosen, the paper makes the case that intervention interactions are not just a niche concern but a central design consideration for any method intended for real-world deployment. The paper's framing in Section 1 makes this explicit:

"We propose that practical interventions should be composable: When an intervention is applied to a model, it should avoid interfering with success of prior or future interventions."

This is a normative claim—a design desideratum—not just an empirical observation. By articulating composability as a requirement for practical interventions, the paper aims to shift how the field evaluates and develops these methods going forward.

3. Technical Approach

3.1 Reader orientation

This paper is a measurement and evaluation framework paper—it does not propose a new intervention method, but instead builds a systematic way to study what happens when existing interventions (knowledge editing, model compression, machine unlearning) are applied sequentially to the same language model. The core idea is that we can define two quantitative metrics—Order-free Error and Order Sensitivity—that capture whether the combined success of multiple interventions degrades relative to their individual success and whether that combined success depends on the order in which interventions are applied. By sweeping these metrics across hyperparameter ranges, intervention pairs, and both application orders, the paper provides the first systematic diagnosis of how well state-of-the-art post-training interventions compose.

3.2 Big-picture architecture (diagram in words)

The system has five major components organized in a pipeline for evaluation, not for model improvement:

Base language model—a pretrained LLM (primarily Llama-3 8B, with additional experiments on Mistral 7B Instruct and Yi 1.5 9B Chat) that serves as the starting state before any interventions are applied. This model has fixed pretrained parameters $f_\theta$ .
Three intervention categories with ten concrete methods—each intervention is defined as an operator $\omega(f_\theta, \gamma)$ that takes a model and hyperparameters $\gamma$ (controlling intervention strength) and produces a modified model, along with a loss function for optimization and evaluation criteria $\kappa(f, D) \in [0,1]$ . The specific methods are:
- Knowledge editing (3 methods): MEMIT, LoRA fine-tuning, and standard fine-tuning—each applied to the zsRE dataset to update specific facts.
- Model compression (4 methods): Wanda and SparseGPT (weight pruning, controlled by sparsity ratio), GPTQ and AWQ (weight quantization, controlled by bit-width).
- Machine unlearning (3 methods): Gradient Ascent (GA), Gradient Difference (GD), and Representation Misdirection for Unlearning (RMU)—each applied to the WMDP benchmark to forget hazardous knowledge.
Sequential composition engine—a procedure that applies two interventions to the same model in a specified order (e.g., compress then edit, or edit then compress), producing a composition of the form $\phi_j \circ \phi_i(f_\theta)$ . For triplet experiments, three interventions are applied in sequence.
Evaluation criteria collection—after each individual intervention and after each composition, the system measures task-specific metrics (Edit Success, Edit Generalization, Strict Edit Locality for editing; WMDP accuracy for unlearning; sparsity or average bits for compression) and general utility (MMLU accuracy, WikiText perplexity).
Composability metric computation—two quantitative metrics derived from the evaluation results:
- Order-free Error: the area above the best-case curve when sweeping intervention hyperparameters, measuring whether high performance is achievable in either application order.
- Order Sensitivity: the area between the two curves (one per application order), measuring how much the combined performance depends on which intervention was applied first.

Information flows as follows: a base model enters the pipeline → the first intervention $\phi_i$ is applied with a chosen hyperparameter setting $\gamma_i$ → evaluation criteria $\kappa$ are measured on the intervened model → the second intervention $\phi_j$ is applied to produce $\phi_j \circ \phi_i(f)$ → $\kappa$ is measured again → the process is repeated in reverse order to produce $\phi_i \circ \phi_j(f)$ → both measurements are fed into the composability metrics → the hyperparameter $\gamma_i$ is swept to produce curves over intervention strength.

3.3 Roadmap for the deep dive

First, the formal intervention abstraction—what exactly constitutes an "intervention" and how it is formalized—because all downstream metrics depend on this definition.
Second, the two composability metrics (Order-free Error and Order Sensitivity) in full mathematical detail—what they compute, how they are constructed from evaluation curves, and what design choices they encode—since these are the paper's primary technical contributions.
Third, how hyperparameter sweeping enables the curve-based formulation, what hyperparameters are swept for each intervention type, and why the sweep is necessary rather than using single fixed settings.
Fourth, the concrete instantiation of interventions, metrics, and datasets—what each method actually does, what data it uses, and what evaluation criteria apply—because the composability framework is content-agnostic and must be grounded in specific choices.
Fifth, extensions to three interventions (triplet compositions) and cross-model generalization—how the framework scales beyond pairs and across model architectures.

3.4 Detailed, sentence-based technical breakdown

This is primarily a framework and measurement paper whose core idea is that we can define two quantitative composability metrics derived from evaluation curves over intervention hyperparameters, and that these metrics reveal systematic interaction patterns across intervention categories that are invisible to standard single-intervention evaluations.

The Intervention Abstraction

Before defining composability, the paper formalizes what it means by an "intervention" (Section 2.1). This formalization is necessary because the composability metrics operate on a shared abstraction that must work across knowledge editing, model compression, and machine unlearning—methods that differ substantially in their mechanisms, data requirements, and evaluation criteria.

An intervention $\phi$ is defined as the combination of four components:

An operator $\omega(f_\theta, \gamma)$ that takes a language model $f_\theta$ with parameters $\theta \in \Theta$ and produces a new model $f_{\theta'}$ with modified parameters. The hyperparameters $\gamma$ control the strength of the intervention—for example, sparsity ratio in pruning, bit-width in quantization, or step count and learning rate in fine-tuning-based editors.
Hyperparameters $\gamma$ that specify how the operator behaves. The paper deliberately sweeps over $\gamma$ values rather than fixing them at a single "optimal" setting, because composability can vary dramatically with intervention strength. As shown in Figure 2, an edit that survives 4-bit quantization may be destroyed by 2-bit quantization, and a beam search that helps at one sparsity level may hurt at another.
A loss $\ell(\omega(f_\theta, \gamma))$ that specifies the targeted update. This is method-specific: for knowledge editing, the loss maximizes the likelihood of the edited answer; for unlearning, it minimizes the likelihood of the forgotten content (or misdirects representations); for compression, it minimizes reconstruction error of layer outputs.
Evaluation criteria $\kappa(f, D) \in [0, 1]$ , where higher scores indicate better performance. Each intervention has at least one task-specific criterion (Edit Success for editing, WMDP accuracy for unlearning, average bits for compression) and one general utility criterion (MMLU accuracy, WikiText perplexity). Criteria are measured on held-out data $D$ .

For readability, the paper denotes the application of intervention $\phi$ to model $f$ as $\phi(f)$ , hiding the hyperparameters and data dependencies in notation. When two interventions are composed, we write $\phi_1 \circ \phi_0(f_\theta)$ or equivalently $\phi_1(\phi_0(f_\theta))$ , meaning apply $\phi_0$ first, then apply $\phi_1$ to the result.

Why this abstraction matters: by defining interventions through a uniform interface (an operator that transforms models, producing measurable outputs), the composability framework becomes agnostic to the internal mechanism of any specific method. A knowledge editor that modifies MLP weights via gradient updates and a compression method that zeroes out weights based on activation statistics are both instances of the same abstraction. This allows the composability metrics to be computed identically across all intervention pairs.

Composability Metric 1: Order-free Error

The first metric, Order-free Error, measures whether high performance on a given criterion $\kappa_i$ can be achieved when two interventions are applied in either order. In plain language: if we must apply interventions $\phi_i$ and $\phi_j$ to a model, can we find some hyperparameter setting for $\phi_i$ such that the final model performs well on $\kappa_i$ regardless of which intervention came first? A composable pair of interventions will have low Order-free Error—the performance will be high in both application orders.

The metric is defined in Equation 1 as:

$1 - \int_{\gamma_i} \min\left(\kappa_i(\phi_j \circ \phi_i(f_\theta, \gamma_i)), \; \kappa_i(\phi_i(\gamma_i) \circ \phi_j(f_\theta))\right)$

where $\kappa_i(\phi_j \circ \phi_i(f_\theta, \gamma_i))$ is the performance on criterion $\kappa_i$ when intervention $\phi_i$ is applied first (with hyperparameters $\gamma_i$ ) and $\phi_j$ is applied second (with fixed, pre-chosen hyperparameters), and $\kappa_i(\phi_i(\gamma_i) \circ \phi_j(f_\theta))$ is the performance when the order is reversed— $\phi_j$ is applied first, then $\phi_i$ .

What it computes: for each hyperparameter value $\gamma_i$ , take the minimum of the two order-specific performance values. This is a conservative measure: if either order produces poor performance, the minimum reflects that failure. Then integrate (in practice, average) these minima over the swept range of $\gamma_i$ , producing the area under the "best-of-worst" curve. Finally, subtract from 1 because we want an error metric (lower is better) and the maximum possible $\kappa_i$ value is 1. The result is a single number between 0 and 1, where 0 means perfect composability (both orders achieve $\kappa_i = 1$ at every $\gamma_i$ ) and 1 means total failure.

Why the $\min$ operator: the minimum of the two curves captures the worst-case behavior across orders. If one order works well but the other fails catastrophically, the metric reflects the failure. This is intentional—a practical intervention must be robust to ordering because in real deployments, the sequence of interventions is determined by external requirements (e.g., compression happens at deployment time, editing happens later when a fact becomes outdated), not by what is optimal for a particular metric. The $\min$ forces both orders to succeed for the composability score to be good.

Why the integral over $\gamma_i$ : sweeping the hyperparameters rather than evaluating at a single point reveals whether composability is robust across intervention strengths. As the paper shows in Figure 2, editing performance can be nearly perfect at 4-bit quantization but collapse at 2-bit. A single-point metric at 4-bit would report good composability; the integral captures the full range of behavior. The integral is computed discretely in practice as the average over the swept hyperparameter values.

The single-hyperparameter simplification: the paper notes that if a practitioner has already chosen specific hyperparameters, Order-free Error reduces to the remarkably simple form:

$1 - \min\left(\kappa_i(\phi_j \circ \phi_i(f_\theta, \gamma_i)), \; \kappa_i(\phi_i(\gamma_i) \circ \phi_j(f_\theta))\right)$

This is just the worst-case error across the two orders at a single hyperparameter setting.

Design choice: using the maximum criterion value as the upper bound. The paper assumes the maximum possible $\kappa_i$ is 1, which holds for all metrics studied (Edit Success is a probability, Strict Locality is bounded at 1, MMLU accuracy is a fraction). If a criterion has a different maximum, the subtraction term would change accordingly, a flexibility the authors note is available in practice.

What Order-free Error misses: this metric tells you whether some hyperparameter setting achieves good performance in both orders, but it does not tell you whether the same hyperparameter setting works in both orders, or whether the best setting for order A is different from the best setting for order B. That information is captured by the second metric.

Composability Metric 2: Order Sensitivity

The second metric, Order Sensitivity, measures how much the combined performance depends on the application order. Two interventions are order-sensitive if $\kappa_i(\phi_j \circ \phi_i(f))$ is very different from $\kappa_i(\phi_i \circ \phi_j(f))$ , even when both are evaluated at the same hyperparameter settings.

The metric is defined in Equation 2 as:

$\int_{\gamma_i} \left| \kappa_i(\phi_j \circ \phi_i(f_\theta, \gamma_i)) - \kappa_i(\phi_i(\gamma_i) \circ \phi_j(f_\theta)) \right|$

where the terms are the same as in Equation 1—the performance on criterion $\kappa_i$ in the two application orders—and the absolute difference is taken at each hyperparameter value $\gamma_i$ before integrating.

What it computes: for each hyperparameter value, take the absolute difference between the performance when $\phi_i$ is applied first and the performance when $\phi_j$ is applied first. Then integrate (average) these differences over $\gamma_i$ . The result is a non-negative number where 0 means perfect order invariance (the two curves are identical) and larger values mean performance depends heavily on which intervention was applied first. The theoretical maximum depends on the range of $\kappa_i$ .

Why absolute difference rather than signed difference: the sign of the difference tells you which order is better (e.g., if editing-before-compression consistently outperforms compression-before-editing, the signed difference would be positive for most $\gamma_i$ ). But for a composability metric, the paper wants to capture any dependence on order, not just which direction is better. The absolute value ensures that both positive and negative differences contribute to the error. This is important because a method that is consistently worse in one order and a method whose performance is simply noisy across orders both indicate poor composability. If a practitioner happens to know the optimal order, signed differences would be informative; for general-purpose composability assessment, the absolute value is more conservative.

Why integrate over $\gamma_i$ here too: order sensitivity can vary with intervention strength. For example, in Figure 2 (top row, MEMIT quantization), the two curves for GPTQ diverge much more at 2–4 bits than at 8 bits. If sensitivity were measured only at 8-bit quantization, it would appear negligible; the integral reveals the true behavior across the practical operating range.

The single-hyperparameter simplification: as with Order-free Error, if hyperparameters are fixed, Order Sensitivity reduces to:

$\left| \kappa_i(\phi_j(\phi_i(f_\theta))) - \kappa_i(\phi_i(\gamma_i) \circ \phi_j(f_\theta)) \right|$

This is simply the absolute difference between the two orderings evaluated at one hyperparameter setting.

Relationship between the two metrics. Order-free Error and Order Sensitivity measure different aspects of composability. Two interventions can have low Order Sensitivity (their combined performance doesn't depend much on order) but high Order-free Error (neither order achieves good performance). Conversely, they can have low Order-free Error (one order achieves excellent performance across most hyperparameters) but high Order Sensitivity (the other order fails, and the curves diverge dramatically). The paper uses both metrics together: low scores on both indicate genuinely composable methods; divergence between them indicates specific failure modes (e.g., one order works but the other doesn't).

The visual intuition from the paper's figures. In all composition plots (Figures 2, 3, 4), the paper shows two curves—one per application order—swept over hyperparameter values. Order-free Error corresponds to the area above the lower envelope of the two curves (the $\min$ ). Order Sensitivity corresponds to the area between the two curves. When the curves are close together and near 1.0, both metrics are low and the composition is successful. When the curves diverge, Order Sensitivity is high. When both curves are low, Order-free Error is high. The shaded regions in the figures make this visually apparent: the gap between curves is literally the order sensitivity, and the gap from the top of each subplot to the highest of the two curves reflects order-free error.

Hyperparameter Sweeping and Curve Construction

The composability metrics are defined as integrals over hyperparameters, requiring evaluation curves rather than point estimates. This section describes how these curves are constructed in practice, which is essential for understanding what the metrics actually measure.

For model compression methods, the hyperparameter $\gamma$ is the compression level: sparsity ratio (fraction of weights zeroed) for pruning, and bit-width (number of bits per weight) for quantization. The paper sweeps:

Pruning (Wanda, SparseGPT): sparsity levels of 0.25, 0.35, 0.45, 0.55, 0.65, and 0.75. A sparsity of 0.25 means 25% of weights are zeroed; 0.75 means 75% are zeroed.
Quantization (GPTQ, AWQ): bit-widths of 2, 3, 4, and 8 bits, down from the standard 16 bits. Lower bit-width means more aggressive compression (and typically more degradation).

Each compression level produces a point on the x-axis of the composition plots (e.g., the x-axes in Figure 2 are labeled "Sparsity" and "Bits"), and the corresponding criterion values (Edit Success, MMLU, etc.) are plotted on the y-axis. Two curves are plotted—one for each application order—and the composability metrics are computed from these paired curves.

For knowledge editing methods, the hyperparameter is less straightforward because editing methods don't have a single continuous "strength" parameter analogous to sparsity. Instead, the paper fixes the editing hyperparameters (the settings described in Appendix C: 25 gradient steps for MEMIT, 70 steps for LoRA, specific learning rates) and treats the editing method as a fixed operation. The "sweep" in the metrics then corresponds to varying the compression level when editing is composed with compression, or using different unlearning methods when editing is composed with unlearning. In other words, the $\gamma_i$ being swept is always a parameter of one of the two interventions; the paper chooses which intervention's parameters to sweep based on which one has a natural continuous control.

For machine unlearning methods, each method (RMU, GD, GA) is used at a single, carefully-tuned hyperparameter setting (the result of grid searches over learning rate, training samples, and method-specific parameters described in Appendix C, Tables 5–7). The "sweep" when unlearning is composed with compression again comes from varying the compression level.

Why the sweep is necessary rather than using a single "best" setting: the paper argues that practical deployments don't always permit choosing the optimal hyperparameters for composability—compression level may be dictated by hardware constraints, editing batch size by the number of facts to update, and unlearning strength by safety requirements. By sweeping, the metrics characterize composability across the full operating range, not just at a cherry-picked point. This is analogous to how ROC curves provide richer evaluation than single-point accuracy measures.

The two-fold cross-validation protocol for compute-optimal strategy selection is not used here. Unlike the companion paper on test-time compute scaling (which needs to avoid overfitting when selecting the best strategy), this paper does not perform strategy selection—it simply evaluates all compositions at all hyperparameter settings. The metrics are computed directly from the measured evaluation curves without any training/validation split for hyperparameter choice. The "wins" reported in Tables 2–4 come from comparing the computed metric values across methods, with ties counting as losses.

Concrete Instantiation: Interventions, Datasets, and Criteria

This section provides the concrete details of what the framework is instantiated with—the specific methods, data, and metrics that the composability metrics are computed over.

Knowledge Editing Methods

The paper uses three editing methods, each representing a different approach to updating facts in a language model:

MEMIT (Mass-Editing Memory in a Transformer; Meng et al., 2023): a state-of-the-art locate-then-edit method that modifies specific layers of the transformer to update factual associations. The method identifies which MLP layers mediate factual knowledge and applies a rank-one update to those layers' weights based on computed associations between subject and object tokens. The paper applies edits to layers 4 through 8 of Llama-3 8B with 25 gradient steps, learning rate 0.5, clamp normalization factor 4, and a KL divergence factor of 0.0625 that regularizes the edit to stay close to the original model distribution. A second momentum adjustment with update weight 15,000 is used to stabilize optimization. Edits are applied in batches of 50 simultaneously, and the process is repeated 10 times with results averaged to reduce variance. The model generates responses with a maximum length of 40 tokens.

LoRA (Low-Rank Adaptation; Hu et al., 2021) as an editor: rather than directly modifying MLP weights, LoRA inserts trainable low-rank matrices into the query and value projection layers and optimizes only these matrices (keeping the base weights frozen). The paper uses the AdaLoRA variant with rank 8, scaling factor 32, no dropout, 70 gradient steps, and learning rate 0.005. The target modules are the query and value projections. LoRA has recently been shown to be a surprisingly effective editor despite its simplicity (Gangadhar & Stratos, 2024; Hsueh et al., 2024), making it a strong baseline despite being originally designed for task adaptation rather than factual editing.

Fine-tuning (FT): the simplest possible editor—standard gradient-based fine-tuning on the edit examples. The paper fine-tunes layers 4 through 8, targeting MLP down-projections, self-attention modules, layer normalization, and the language model head. The process uses 25 optimization steps with learning rate $10^{-5}$ , no weight decay, and bfloat16 precision. This serves as a surprisingly strong sanity check: if simple fine-tuning composes better than sophisticated editors, that reveals that the sophisticated editors may be over-optimized for single-intervention benchmarks at the expense of robustness.

Editing dataset: all editing experiments use the zsRE dataset (Levy et al., 2017), a context-free question-answering benchmark where the model must produce a correct object given a subject and relation (e.g., "The capital of France is" → "Paris"). The paper uses the version provided by Wang et al. (2023b), which includes improved locality evaluation sets. Results are averaged over 10 batches of 50 randomly selected edits from zsRE.

Knowledge Editing Evaluation Criteria

Each editing experiment reports three standard metrics (Yao et al., 2023), as described in Appendix C.1.1:

Edit Success: measures whether the post-edit model produces the correct edited answer. The metric uses an F1 score over common tokens between the generated response and the ground-truth target. Specifically, let $G$ be the set of tokens in the generated response (excluding special tokens), $T$ be the set of tokens in the target answer (excluding padding), and $C = G \cap T$ be their intersection. If $C$ is empty, the F1 score is 0. Otherwise:

$\text{Precision} = \frac{|C|}{|G|}, \quad \text{Recall} = \frac{|C|}{|T|}$

$\text{F1} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$

The numerator simplifies because $|C|$ cancels: F1 is the harmonic mean of precision and recall. The final score is the average F1 across all edit examples. This F1-based measure accommodates varying answer verbosity (the model might produce "the capital is Paris" when the target is just "Paris") while still requiring the correct tokens to be the most likely output.

Edit Generalization: measures how well the edit transfers to semantically equivalent but lexically different prompts. The zsRE dataset provides 10 holdout rephrasings per edit example. Generalization is computed as the average Edit Success (again via F1) across these rephrasings. A score close to the Edit Success indicates the model genuinely updated its factual knowledge rather than memorizing surface patterns.

Strict Edit Locality: measures the precision of edits by evaluating whether unrelated facts remain unchanged. The metric computes Edit Success on random, unedited samples from the locality set. High locality (close to the pre-edit baseline) means the editor surgically modified only the targeted fact without disturbing adjacent knowledge. The paper computes locality as F1 on the correct output logits—intended outputs must be the most likely tokens after editing.

All three metrics range from 0 to 1, with higher values indicating better performance, consistent with the composability framework's assumption that $\kappa \in [0,1]$ .

Model Compression Methods

The paper studies four compression techniques, two for pruning and two for quantization:

Weight pruning methods remove individual weights by setting them to zero, creating sparse weight matrices:

SparseGPT (Frantar & Alistarh, 2023): an efficient one-shot pruning method that casts pruning as a large-scale sparse regression problem. For each layer, SparseGPT reconstructs the layer's output using a subset of weights, selecting which weights to keep to minimize the squared error between the original and pruned layer outputs. The method uses an approximate solver that processes weights column-by-column using a Hessian-based importance criterion, enabling pruning of models with tens of billions of parameters on a single GPU. The paper sweeps sparsity levels from 0.25 to 0.75.
Wanda (Sun et al., 2023): a simpler pruning method that uses a criterion combining weight magnitude with the norm of corresponding input activations (computed from a small calibration dataset). Specifically, for each weight $W_{ij}$ connecting input $j$ to output $i$ , Wanda computes $|W_{ij}| \cdot ||X_j||_2$ where $X_j$ is the activation vector for input dimension $j$ . Weights with smaller scores are pruned first. The method operates independently per output neuron and requires only a single forward pass, making it computationally efficient.

Weight quantization methods reduce the numerical precision of each weight:

GPTQ (Frantar et al., 2023): quantizes weights column-by-column, maintaining a running estimate of the quantization error introduced so far and compensating subsequent weights to minimize output distortion. The method supports bit-widths from 2 to 8 bits, with performance degrading more sharply at lower bit-widths. This is an "accurate post-training quantization" method that doesn't require retraining.
AWQ (Activation-Aware Weight Quantization; Lin et al., 2023): identifies a small fraction (typically ~1%) of "salient" weights whose quantization significantly impacts accuracy. Saliency is determined by analyzing activation distributions rather than weight magnitudes—weights connected to channels with larger activation magnitudes are considered more important. AWQ applies per-channel scaling to protect these salient weights before quantization, then quantizes all weights. The paper sweeps bit-widths from 2 to 8 bits, reporting average bits as the compression metric.

Compression metric: the primary compression criterion is sparsity ratio for pruning (fraction of weights set to zero, ranging from 0 to 1) and average bits per weight for quantization (reduced from 16 bits at full precision). However, these are not evaluation criteria in the $\kappa$ sense—they define the hyperparameter $\gamma$ being swept. The performance criterion for compression is general utility (MMLU, WikiText perplexity), not a compression-specific accuracy metric.

Calibration data: all compression methods use a small calibration dataset (typically a subset of the training data) to compute activation statistics or Hessian information, but the paper does not specify the exact calibration corpus, noting that this is standard for the methods used.

Machine Unlearning Methods

The paper studies three unlearning methods applied to the WMDP benchmark, aiming to make the model forget potentially hazardous biosecurity and cybersecurity knowledge:

Gradient Ascent (GA; Jang et al., 2022): the simplest unlearning approach—instead of minimizing the loss on the forget target (which would reinforce the knowledge), GA maximizes the loss, effectively training the model to produce incorrect answers on the unlearning target. The intuition is that the model will learn to avoid generating tokens associated with the dangerous knowledge. The paper trains only the final 16 layers of Llama-3 8B due to memory constraints. Hyperparameter search (Table 6) selected learning rate $5 \times 10^{-5}$ and 50 training samples, balanced across the cyber and bio splits of WMDP. GA is known to suffer from catastrophic forgetting—the model may lose general language capabilities rather than specifically forgetting the target—and the paper's results confirm this limitation.

Gradient Difference (GD; Maini et al., 2024): augments GA by adding a retain loss term that maximizes the likelihood of correct tokens on an unrelated dataset (WikiText, Merity et al., 2016). The loss becomes:

$\mathcal{L}_{\text{GD}} = -\mathcal{L}_{\text{forget}}(\text{target}) + \alpha \cdot \mathcal{L}_{\text{retain}}(\text{retain})$

where $\alpha$ weights the retain loss relative to the forget loss. The paper found that $\alpha$ needed to be substantially upweighted (to 40) compared to values reported in prior work, because the forget loss term easily dominates and drowns out the retain signal. Other hyperparameters: learning rate $5 \times 10^{-5}$ , 50 training samples, training only the final 16 layers. The retain set is WikiText.

Representation Misdirection for Unlearning (RMU; Li et al., 2024a): intervenes on a specific transformer layer to perturb the model's internal representations for inputs related to the unlearning target. For a chosen layer $l$ , RMU modifies the activations such that:

For forget-set inputs: the activation vectors are pushed toward a random, fixed direction (misdirected), destroying the model's ability to extract useful information from these representations.
For retain-set inputs (WikiText): the activations are preserved close to their original values, maintaining general utility.

The method requires selecting which layer to modify, a weight $\alpha$ for the retain loss (searched over values 1, 10, 100, 1000, 10000; chosen as 1000), and a maximum number of training batches (selected as 250 from a search over 100–300). The paper found RMU to be "quite sensitive to hyperparameter choice, with most combinations either leaving the model unaffected or significantly harming model utility" (Appendix C.4.1). The chosen configuration uses layer 3 of Llama-3 8B, $\alpha = 1000$ , and 250 batches.

Unlearning evaluation: all unlearning methods are evaluated on the WMDP benchmark (Li et al., 2024a), which consists of 3,260 multiple-choice questions spanning biosecurity and cybersecurity knowledge. The paper averages performance across both cyber and bio splits (individual split results in Appendix E). Optimal unlearning yields 25% accuracy, as this is random performance on the four-choice questions. The paper notes that while WMDP is "among the most studied unlearning benchmarks," it provides "an imperfect evaluation of unlearning writ-large" (Section 3.1) and proceeds with it as sufficient for measuring composability, while acknowledging the limitation.

General Model Utility Measurement

All interventions share a common evaluation criterion for general utility: MMLU accuracy (Hendrycks et al., 2020), measured via the LM Eval Harness (Gao et al., 2023). MMLU consists of 57 subject areas spanning STEM, humanities, social sciences, and more, with multiple-choice questions in a zero-shot setting. The paper reports accuracy as a fraction of correct answers, with random chance at 25% (four choices). Additional utility evaluation uses WikiText perplexity (Merity et al., 2016) as reported in the appendix (Tables 15–21), though MMLU is the primary utility metric in the main text and composability tables.

Why MMLU is used despite being shown inadequate: the paper explicitly critiques MMLU as insufficient for measuring composability (Finding 4 in Section 4.1), yet it is a necessary criterion because all interventions aim to preserve general utility. The composability framework requires evaluating all criteria for each composition—task-specific and general utility alike. By measuring MMLU alongside intervention-specific metrics, the paper can demonstrate the discrepancy that motivates its critique: MMLU scores can be nearly identical while task-specific metrics vary dramatically across compositions.

The Sequential Composition Algorithm

The paper's evaluation pipeline applies interventions sequentially and measures criteria after each step. The procedure for a single composition (two interventions, both orders) is:

Step 1: Baseline measurement. Measure all criteria $\kappa$ on the base model $f_\theta$ before any interventions. This provides the reference point for understanding individual intervention effects.

Step 2: First intervention application. Apply $\phi_i$ with hyperparameters $\gamma_i$ to produce $f_{\theta'} = \phi_i(f_\theta, \gamma_i)$ . Measure all criteria on $f_{\theta'}$ (reported in baseline Table 15 in the appendix).

Step 3: Second intervention application in Order A. Apply $\phi_j$ to $f_{\theta'}$ to produce $f_{\theta''} = \phi_j(\phi_i(f_\theta, \gamma_i))$ . Measure all criteria on $f_{\theta''}$ . This gives $\kappa_k(\phi_j \circ \phi_i(f_\theta, \gamma_i))$ for all criteria $\kappa_k$ , including $\kappa_i$ (the criterion for the first intervention), $\kappa_j$ (the criterion for the second intervention), and $\kappa_{\text{MMLU}}$ .

Step 4: Reverse order (Order B). Apply $\phi_j$ first to the base model: $f_{\tilde{\theta}} = \phi_j(f_\theta)$ . Then apply $\phi_i$ with the same hyperparameters $\gamma_i$ : $f_{\tilde{\theta}'} = \phi_i(\phi_j(f_\theta), \gamma_i)$ . Measure all criteria. This gives $\kappa_k(\phi_i \circ \phi_j(f_\theta, \gamma_i))$ .

Step 5: Sweep and repeat. Repeat steps 1–4 for each value of $\gamma_i$ in the hyperparameter sweep. For compression methods, this means repeating for each sparsity level or bit-width. For pairs where neither intervention has a natural sweep parameter, the "sweep" may consist of different method choices rather than continuous parameter values.

Step 6: Compute composability metrics. For each criterion $\kappa_k$ , compute Order-free Error and Order Sensitivity from the paired curves $\kappa_k(\phi_j \circ \phi_i(f_\theta, \gamma_i))$ and $\kappa_k(\phi_i \circ \phi_j(f_\theta, \gamma_i))$ using the integral (average) formulations.

Important detail: re-compression after editing. The paper notes that most compression techniques require decompressing models for subsequent operations. When editing or unlearning is applied after compression, the model is first decompressed, then edited, then re-compressed using the same compression technique. This re-compression step ensures that the final model remains compressed for fair comparison of deployment efficiency. The paper does not explore whether editing on a compressed model directly (without decompression) would behave differently.

For triplet compositions (Appendix E.4): the procedure extends naturally to three interventions. All six possible orderings are evaluated (e.g., Wanda→RMU→MEMIT, MEMIT→Wanda→RMU, etc.), and performance on each criterion is reported. The paper does not define composability metrics for triplets explicitly but reports the full performance table, noting that general utility remains stable across orderings while task-specific metrics vary substantially.

Extensions: Cross-Model Generalization and Layer Ablation

The paper includes two auxiliary analyses to probe the generality of composability findings:

Cross-model replication (Section E.1, Table 9, Figure 4): to test whether composability patterns are specific to Llama-3 8B or generalize, the paper replicates a subset of compositions on Mistral 7B Instruct (Jiang et al., 2023) and Yi 1.5 9B Chat (Young et al., 2024). One representative method is selected from each category: MEMIT (editing), AWQ (quantization), Wanda (pruning), and RMU (unlearning). The authors could not find effective hyperparameters for RMU on Yi, so those compositions are excluded. Each composition is evaluated in both orders, and Order Sensitivity is computed for each criterion. The findings: Order Sensitivity is "generally consistent across LMs" except for Editing↔Compression compositions, where sensitivity varies more across models—an expected result given that editing performance is known to vary with model architecture (Meng et al., 2023). This suggests that while high-level composability trends (e.g., unlearning being more order-invariant than editing) are model-independent, specific method-method interactions can depend on architecture.

Layer overlap ablation (Appendix D, Table 8): to distinguish whether composability stems from interventions modifying distinct model components (non-overlapping layers) or from the intrinsic precision of the methods, the paper runs an experiment where RMU and MEMIT both modify the same layers. In the default setup, RMU targets layer 3 of Llama-3 while MEMIT targets layers 4–8—non-overlapping. The ablation forces both methods to target layer 3. Results show that "intervention composability stems primarily from algorithmic precision rather than layer isolation"—RMU and MEMIT remain highly composable even with overlapping layer targets, with only negligible increases in Order Sensitivity. This supports the paper's claim that designing interventions for surgical precision (RMU redirects only unlearning-target-related activations; MEMIT modifies only fact-mediating parameters) is more important for composability than segregating interventions to different model regions.

The "Wins" Methodology for Comparing Methods

In Tables 2, 3, and 4, the paper reports composability metrics alongside "# Wins" counts. This methodology deserves explicit explanation:

What is being compared: within each intervention category (e.g., compression methods), the paper compares which method produces the best composability score on each criterion. For example, in Table 2, Wanda, SparseGPT, AWQ, and GPTQ are compared on their Order-free Error and Order Sensitivity when composed with each knowledge editing method (FT, MEMIT, LoRA) across four criteria (Edit Success, Edit Generalization, Strict Edit Locality, MMLU).

How "best" is determined: for a given compression method, editor, and criterion, the method gets a "win" if it has the lowest score (since both metrics are "lower is better") among all compression methods for that editor-criterion pair, and no other compression method ties it. Ties count as losses—this is a deliberately conservative definition that requires unambiguous superiority to claim a win.

Why this per-criterion, per-editor breakdown: composability can differ dramatically across criteria. A compression method might compose excellently on MMLU (everyone does, since MMLU is insensitive) but poorly on Edit Success. Aggregating wins across criteria and editors provides a holistic view of composability, but the per-criterion breakdown in the tables allows readers to assess which specific compositions drive the overall pattern. For instance, AWQ's 12 total wins in Table 2 come primarily from Edit Generalization Order Sensitivity and MMLU, while GPTQ's 0 wins reflect consistent inferiority across criteria.

Limitations of # Wins: the win count does not report magnitudes—a method could win many comparisons by tiny margins and lose one comparison by a large margin. The metric is a coarse summary statistic designed to complement (not replace) the full numeric results and the visual curves in the figures. The paper uses it to quickly identify which methods are generally more composable without requiring readers to inspect every numeric comparison individually.

4. Key Insights and Innovations

Innovation 1: Composability as a First-Class Design Criterion, Not an Afterthought

The paper's most fundamental intellectual move is elevating composability from an implicit hope to an explicit, measurable design requirement for post-training interventions. Prior to this work, the field treated each intervention category as a self-contained problem: knowledge editing was judged by edit success and locality, model compression by perplexity retention at a given bit-width, and machine unlearning by forget-set accuracy degradation versus retain-set preservation. The implicit assumption was that if each intervention individually preserved general model utility (MMLU, perplexity), then applying multiple interventions would work—or at minimum, that the question wasn't worth systematic study.

This paper demolishes that assumption and replaces it with a normative claim: interventions that are intended for real-world deployment should be composable, and methods that fail to compose are, by that standard, incomplete. This is not an incremental improvement to an existing evaluation protocol; it is a fundamentally new axis of evaluation that had no formal vocabulary before this paper. The authors make the claim explicit in Section 1:

"We propose that practical interventions should be composable: When an intervention is applied to a model, it should avoid interfering with success of prior or future interventions."

The word "should" here does normative work. It asserts a design desideratum—analogous to how randomized controlled trials became the standard for medical interventions, or how adversarial robustness became a standard axis for evaluating image classifiers. The paper is arguing that intervention composability deserves similar status.

What distinguishes this from prior work that studied interactions (e.g., Hooker et al., 2019 on what compressed models forget; Jia et al., 2023 on sparsity and unlearning; Brown et al., 2023a on editing robustness to distribution shift) is the framing: those works treated interactions as a side observation about specific methods. This paper treats composability as a property of the intervention category itself, with dedicated metrics, a systematic evaluation protocol, and a call for method development to target it directly. The shift is from "here's an interesting thing we noticed about method X" to "here is a requirement that all methods of type Y should satisfy."

The evidence that this framing is genuinely novel—not just a repackaging of existing ideas—is visible in the paper's own results. Section 4.1, Finding 4 states:

"Despite similar MMLU scores, methods vary significantly in composability. This indicates that targeting composability as a metric has the potential to drive the development of new, practically-grounded interventions."

If composability were merely "checking that things still work," then MMLU would be a sufficient proxy. The fact that it isn't—that two compositions with identical MMLU can have radically different edit success, and that Order Sensitivity on task-specific metrics can be large even when MMLU Order Sensitivity is near zero—proves that composability captures something distinct from general utility preservation. This is a diagnostic finding that only makes sense within the paper's new conceptual framework.

Innovation 2: The Order Sensitivity Concept — Composition Is Not Commutative

The second distinctive contribution is the formal identification and quantification of order dependence as a primary failure mode for intervention composition. Prior to this paper, there was no vocabulary for stating, let alone measuring, that applying compression before editing produces different results than editing before compression. The field's default assumption—implicit in every paper that evaluates an intervention in isolation—was that if two interventions both work individually, they will work together in whatever sequence they happen to be applied.

The paper establishes that this assumption is false in ways that are both large in magnitude and systematic in pattern. The introduction of Order Sensitivity (Section 2.2, Equation 2) as a dedicated metric—the area between the two application-order curves—elevates order from a nuisance variable to a first-class diagnostic signal. This is not merely measuring degradation; it is measuring whether the degradation structure depends on sequence, which is a qualitatively different question.

The empirical results justify this elevation. Section 4.1, Finding 2 states:

"Editing performance hinges on the order of interventions. The shaded area between the curves in Figure 2 shows the impact of ordering. Ordering is highly impactful in most cases and varies dramatically between editing methods."

The magnitude of the effect is striking. For MEMIT composed with Wanda at 25% sparsity, Order Sensitivity on Edit Success is 0.24 (Table 2)—meaning the choice of order shifts edit success by 24 percentage points. Yet on the same composition, MMLU Order Sensitivity is 0.00 (Table 2). This means that the standard evaluation protocol—measuring MMLU after both orders and seeing no difference—would have concluded that order doesn't matter, when in fact it matters enormously for the task the intervention was specifically designed for.

The paper also shows that order sensitivity is not uniform across method pairs. RMU and MEMIT (Table 4) exhibit Order Sensitivity of 0.01 on Edit Success—nearly order-invariant—while GD and MEMIT exhibit 0.40 under the same conditions. This variation means order sensitivity is a property of specific method combinations, not a fixed cost of composition. It can be low, which means it is a target that methods can be designed to achieve, not an inevitable consequence of sequential application.

The conceptual contribution here is that the paper identifies non-commutativity as the central mathematical structure of intervention composition, and builds the measurement apparatus to quantify it. This is analogous to how matrix multiplication is non-commutative, or how sequential quantum measurements depend on order—the composition of operations on a model's parameters does not, in general, produce the same result when the sequence is reversed. Articulating this as a formal property with a dedicated metric creates a new axis of evaluation that prior work lacked entirely.

Innovation 3: The Framework-Agnostic Measurement Architecture

The paper's third distinctive contribution is architectural rather than theoretical: it provides a measurement framework that is agnostic to intervention mechanism, model architecture, and evaluation criterion, and demonstrates its utility by applying it uniformly across three intervention categories that had never been jointly evaluated.

This is a genuine innovation in research infrastructure, not merely an implementation detail. The field's prior state was that each intervention category had its own codebase, its own evaluation conventions, and its own set of assumptions about what "good performance" meant. Knowledge editing researchers used zsRE and reported F1-based Edit Success; compression researchers used WikiText perplexity and reported the largest sparsity at which perplexity didn't degrade too much; unlearning researchers used WMDP and reported forget-set accuracy relative to random chance. There was no shared evaluation harness that could apply all these methods to the same model and measure all these criteria after each composition.

The paper's intervention abstraction—every method is an operator $\omega(f_\theta, \gamma)$ that produces a modified model, evaluated by criterion $\kappa(f, D) \in [0,1]$ —is deliberately minimal. It doesn't try to capture the internal mechanism of knowledge editing versus compression versus unlearning; it only requires that each method expose a uniform interface for model transformation and evaluation. This minimalism is a feature, not a limitation: it means any future intervention method that satisfies the interface can be plugged into the composability framework without modification.

The significance of this contribution is demonstrated by what the framework enables. The paper reports results on 417 different compositions—a number that would be infeasible to produce without a unified codebase. Each composition requires: applying intervention A, measuring all criteria, applying intervention B, measuring again, repeating for the reverse order, and sweeping over hyperparameters. Doing this manually with separate codebases for each intervention type would be prohibitively error-prone and time-intensive. The unified codebase transforms composability evaluation from a heroic one-off effort into a routine experimental protocol.

The paper explicitly positions this as an enabling contribution: the codebase is "extendable" and released publicly "to enable others to develop new multi-objective interventions" (Section 1). This is infrastructure in the sense that ImageNet was infrastructure for image classification—it doesn't solve the problem itself, but it makes it possible for the community to work on the problem systematically. The inclusion of ten methods spanning three categories demonstrates the architecture's generality without claiming it is exhaustive.

Innovation 4: Compression as a Systematic Hindrance — A Negative Result with Practical Consequences

While the paper frames itself primarily as a measurement framework, its most actionable empirical finding is a strong negative result: model compression consistently and substantially degrades the success of other interventions. This is not a marginal effect; it is the dominant pattern across all composition experiments.

Section 4.1, Finding 1 states the core observation:

"Model compression degrades editing performance. Across Figure 2, edits made by MEMIT, LoRA, and Finetuning deteriorate as compression increases."

Section 4.2, Finding 5 extends the pattern:

"Compression hinders unlearning. Unlearning a compressed model is hard, but the degree to which post-compression unlearning hurts performance is contingent on the composition and compression level."

And Section 4.4 generalizes:

"Compression consistently hinders other interventions. Compressing an edited model can regress editing performance. Unlearning on a compressed model is often more challenging than on the base model."

The intellectual significance of this finding is that it identifies compression as a bottleneck for the entire intervention ecosystem. Compression is, as the paper notes, "nearly universal in LM deployments" (Section 4)—the gap between training-scale compute and inference-scale compute makes compression a practical necessity for most real-world use cases. If compression degrades editing and unlearning, then the de facto composability ceiling for any deployed model is set by how well it survives compression. This is not a theoretical concern; it is a practical roadblock for any organization that needs to edit or unlearn a model that has already been compressed for deployment.

What makes this a genuine innovation rather than a predictable finding is the systematic documentation of the pattern's scope and variation. The paper doesn't just show that compression hurts—it shows that the hurt varies dramatically by method pair, compression level, and application order. AWQ composes much better than GPTQ (17 wins versus 2 in Table 2). Editing post-compression generally outperforms compressing post-editing (Finding 2). The steepest degradation occurs between 2 and 4 bits for quantization, and at sparsity above 0.45 for pruning (Figure 2). These are the kinds of specific, quantitative characterizations that turn a vague intuition ("compression probably makes other things harder") into actionable guidance for practitioners and concrete targets for method developers.

The paper also offers a mechanistic hypothesis that, while not proven, provides conceptual scaffolding for future work. Section 4.4 speculates:

"These trends suggest compression may alter how models encode knowledge, making targeted updates harder. Previous works have explored how compression leads to knowledge loss and how editing modifies model internals, increasing fragility. However, the relationship between how models are compressed and how knowledge is modified remains underexplored."

This connects the composability findings to a deeper question about knowledge representation in language models. If compression systematically restructures knowledge representations in ways that make them less amenable to targeted modification, then building composable interventions may require understanding—and potentially controlling—how different intervention types transform the model's internal geometry.

Innovation 5: The Insufficiency of General Utility Metrics for Evaluating Intervention Quality

The paper's fifth distinctive contribution is an empirical refutation of a widely-held implicit assumption: that preserving general model utility (MMLU, perplexity) is sufficient to ensure that interventions haven't caused hidden damage. This assumption pervades the intervention literature—nearly every knowledge editing paper reports MMLU or its equivalent, every compression paper reports perplexity, and every unlearning paper reports retain-set performance. The logic is straightforward: if the model still performs well on a broad benchmark, the intervention must have been appropriately targeted.

The paper shows that this logic is flawed in the composition setting. Section 4.1, Finding 4 states:

"Overall utility evaluations fail to measure composability. Despite similar MMLU scores, methods vary significantly in composability. For example, Table 2 shows that MEMIT and Wanda achieve near-perfect Order Sensitivity on MMLU but have 0.24 Order Sensitivity on Edit Success."

This is a diagnostic finding of considerable practical importance. It means that a practitioner who evaluates a composition by checking MMLU—which is the standard practice—would conclude that the composition was successful (MMLU looks fine) while remaining entirely unaware that the editing performance had degraded by 24 percentage points depending on application order. The MMLU signal is not just noisy; it is actively misleading, reporting success where task-specific metrics show failure.

The paper extends this finding across all intervention pairs. In Table 2, Order Sensitivity on MMLU never exceeds 0.02 for any knowledge editing–compression pair, while Order Sensitivity on Edit Success reaches 0.24 (MEMIT–Wanda) and on Edit Generalization reaches 0.46 (LoRA–SparseGPT). In Table 3, the pattern holds for unlearning–compression: GD's WMDP Order Sensitivity is 0.17–0.22 while MMLU Order Sensitivity is 0.01–0.09. In Table 4, for editing–unlearning: RMU achieves WMDP Order Sensitivity of 0.01–0.04 and Edit Success Order Sensitivity of 0.01–0.03, demonstrating that low order sensitivity is achievable—but MMLU would not have revealed the methods that fail to achieve it.

The conceptual contribution here is the identification of metric blindness as a systematic evaluation failure mode. The paper doesn't argue that MMLU should be abandoned—it remains valuable as a measure of general utility preservation. Rather, it argues that MMLU is necessary but insufficient, and that composability metrics (Order-free Error, Order Sensitivity) capture an orthogonal dimension of intervention quality that MMLU cannot. This is a refinement of evaluation practice, not a rejection of existing benchmarks. The paper's advocacy for "thorough multi-metric and multi-dataset evaluations for composable interventions" (Section 4.4) is a direct consequence of demonstrating that single-metric evaluation (MMLU-only) produces false confidence.

This finding has implications beyond the specific interventions studied. To the extent that other intervention types—detoxification, debiasing, capability restoration—are evaluated primarily through general utility metrics, they may similarly suffer from undetected composability failures. The paper provides both the diagnostic concept (metric blindness) and the measurement tools (composability metrics) for the broader field to investigate whether this pattern generalizes.

5. Experimental Analysis

Evaluation Methodology

Dataset. All knowledge editing experiments use the zsRE dataset (Levy et al., 2017), a context-free question-answering benchmark where models must produce correct objects given subject-relation pairs. Results are averaged over 10 batches of 50 randomly-selected edits from zsRE. For machine unlearning, the WMDP benchmark (Li et al., 2024a) is used, consisting of 3,260 multiple-choice questions spanning biosecurity and cybersecurity knowledge—the paper averages performance across both cyber and bio splits. General model utility is measured on MMLU (Hendrycks et al., 2020), consisting of 57 subject areas with multiple-choice questions in a zero-shot setting.
Base model(s). The primary model is Llama-3 8B (AI@Meta, 2024), described as "well-studied and highly capable for its parameter count" (Section 3.2). Additional generalizability experiments use Mistral 7B Instruct (Jiang et al., 2023) and Yi 1.5 9B Chat (Young et al., 2024). The models are chosen for their similarity in size and performance to Llama-3, enabling cross-model comparison. For model size ablation, Llama-3.2 1B is also tested (Appendix E.2.2).
Metrics. Three categories of metrics are reported: (1) Task-specific intervention metrics: Edit Success (F1 score on correct output logits after editing), Edit Generalization (average Edit Success across 10 holdout rephrasings), Strict Edit Locality (average Edit Success on random unedited samples) for knowledge editing; WMDP accuracy (averaged across cyber and bio, with 25% as random chance) for machine unlearning; sparsity ratio and average bits for compression level. (2) Composability metrics: Order-free Error (Equation 1: area above the minimum of two application-order curves) and Order Sensitivity (Equation 2: area between the two curves). (3) General utility: MMLU accuracy and WikiText perplexity. All editing metrics are computed as F1 scores following a procedure where common tokens between generated and target answers are identified, precision and recall are computed, and the harmonic mean produces the final score—if no common tokens exist, F1 is 0 (Appendix C.1.1).
Baselines. The paper reports single-intervention baselines in Table 15 (Appendix E.5), showing the performance of each method applied alone to the base model. These baselines establish the reference point against which composition effects are measured. For editing, the unedited model achieves near-zero Edit Success (0.02), confirming that edits are necessary for the model to produce the updated answers. For unlearning, the base model achieves 0.58 WMDP accuracy (well above the 0.25 random baseline), confirming that the model possesses the hazardous knowledge to be forgotten. For compression, baselines show how utility degrades with increasing sparsity or decreasing bit-width. No prior composability baselines exist, as the paper introduces this evaluation framework.
Generation budget / compute accounting. The paper does not measure compute in FLOPs or generations, as it studies parameter modifications rather than inference-time sampling. Instead, intervention strength is controlled through method-specific hyperparameters: sparsity ratio (0.25 to 0.75) for pruning, bit-width (2 to 8 bits) for quantization, and fixed hyperparameter settings for editing and unlearning (determined via grid search, Appendix C). The "sweep" over these hyperparameters generates the curves used in composability metric computation. For triplet compositions, all six possible orderings are evaluated, and the cost is the evaluation time for each ordering—the paper does not explicitly quantify this cost.
Cross-validation / statistical protocol. Knowledge editing results are averaged over 10 batches of 50 randomly-selected edits each, with variance reported as standard errors in Table 12 (Appendix E.3). Compression methods are deterministic given a calibration dataset and hyperparameter choice. For unlearning, the paper reports "minimal variance in replicated experiments, with standard deviations of approximately ±0.5% for accuracy metrics" (Appendix E.3). The "wins" methodology for comparing methods (Tables 2–4) counts ties as losses—a deliberately conservative choice requiring unambiguous superiority. Hyperparameters for unlearning methods are selected via grid search over learning rates, training samples, and method-specific parameters (Tables 5–7), with the best configuration chosen based on WMDP and MMLU trade-off.

Main Quantitative Results

The paper organizes its experimental findings around pairwise compositions of the three intervention categories, with Section 4 presenting results grouped by composition type, Section 4.4 synthesizing cross-cutting trends, and Appendices E providing full result tables, cross-model replication, triplet compositions, and statistical analyses. I'll walk through each composition axis with the specific quantitative evidence.

Composing Model Compression with Knowledge Editing

The headline finding from Section 4.1 is that model compression degrades editing performance in a compression-level-dependent and order-dependent manner, with AWQ emerging as the most composable compression method and MEMIT as the most composable editor.

Compression level degrades editing monotonically but non-linearly. Figure 2 shows that for all three editors (Finetuning, MEMIT, LoRA), Edit Success, Generalization, and Locality steadily decline as compression increases. For weight pruning (left side of Figure 2), the degradation accelerates at higher sparsity levels (above 0.45–0.55). For quantization (right side), the steepest drops occur between 2 and 4 bits—at 8 bits, editing performance is nearly preserved, while at 2 bits, it frequently collapses to near zero. GPTQ causes faster decay in editing metrics than AWQ across all editors and criteria. Finetuning, despite its simplicity, often surpasses MEMIT or LoRA at higher compression levels—for example, at 0.45 sparsity with SparseGPT, FT→SparseGPT achieves Edit Success of 0.96 while MEMIT→SparseGPT achieves 0.62 and LoRA→SparseGPT achieves 0.36 (Table 16). This is a non-obvious result: the simplest method is sometimes the most robust to subsequent compression.

Order matters substantially, and the better order depends on method pair. The shaded area between the two curves in Figure 2 quantifies order effects visually. For pruning methods, compressing first is generally better than editing first—edits applied to an already-pruned model survive better than edits that are subsequently pruned. For quantization, the optimal order depends on the specific quantizer: with GPTQ, editing should be done first (Edit→GPTQ outperforms GPTQ→Edit); with AWQ, compression should be done first (AWQ→Edit outperforms Edit→AWQ). The quantitative magnitude of these effects is captured in Table 2's Order Sensitivity column. At 25% sparsity and 4-bit quantization: MEMIT with Wanda shows Order Sensitivity of 0.24 on Edit Success (the largest editing-specific sensitivity in the table) and 0.00 on MMLU (perfect order invariance on general utility). LoRA with Wanda shows Order Sensitivity of 0.08 on Edit Success but 0.46 on Edit Generalization—the model succeeds at the exact edit phrasing but fails to generalize, and this failure depends heavily on order.

Method-level composability rankings emerge from Table 2. Aggregating across all criteria and editors via the "# Wins" count: among compression methods, AWQ dominates with 12 total wins (primarily from Edit Generalization Order Sensitivity and MMLU metrics), SparseGPT achieves 4 wins, Wanda 6, and GPTQ 0. Among editors, MEMIT achieves 14 total wins across the four criteria, Finetuning achieves 9, and LoRA only 4—though note that on Strict Edit Locality Order-free Error, Finetuning dominates (4 wins to MEMIT's 0), suggesting that while MEMIT edits are more order-robust for success and generalization, Finetuning better preserves locality under composition.

The MMLU composability mirage is directly visible in Table 2. Order Sensitivity on MMLU never exceeds 0.02 for any knowledge editing–compression pair. Meanwhile, Order Sensitivity on Edit Success reaches 0.24 (MEMIT–Wanda), 0.37 (LoRA–GPTQ), and 0.48 on Edit Generalization for LoRA–SparseGPT. A practitioner checking only MMLU after composition would see near-perfect order invariance and conclude the composition was successful; the task-specific composability metrics reveal failure.

Composing Model Compression with Machine Unlearning

Section 4.2's central finding is that compression before unlearning makes unlearning harder, with RMU emerging as the most composable unlearning method and GD achieving comparable unlearning performance but with worse order sensitivity.

Unlearning performance degrades when unlearning a compressed model, especially at higher compression levels. Figure 3 shows that for RMU (the strongest unlearning method), applying unlearning before compression (RMU→Compress) consistently achieves lower WMDP accuracy (better unlearning) than applying compression first (Compress→RMU). The gap widens with compression level: at 0.25 sparsity with Wanda, RMU→Wanda achieves WMDP of approximately 0.29 while Wanda→RMU achieves approximately 0.32 (Table 18 vs. Table 19). At 0.55 sparsity, the gap is larger: approximately 0.27 vs. 0.36. For quantization with GPTQ, the same pattern holds: RMU→GPTQ 4-bit achieves WMDP of 0.27 vs. GPTQ 4-bit→RMU at 0.45. For GD, pruning after unlearning (GD→Compress) generally preserves lower WMDP than the reverse, though the relationship inverts for some compression methods at moderate levels. GA shows the reverse pattern—compressing first sometimes helps—but GA's WMDP performance is consistently poor (never below 0.46 even without compression, from Table 15, versus RMU's 0.29), making this a confounded finding: GA isn't so much "more composable" as it is "equally ineffective regardless of order."

RMU achieves the best balance of unlearning efficacy and composability. In Table 3, RMU and GD have comparable Order-free Error on WMDP (0.27–0.29 with AWQ, 0.24–0.28 with GPTQ), both far outperforming GA (0.34–0.46). However, RMU's Order Sensitivity on WMDP is consistently lower than GD's: with Wanda, RMU achieves 0.03 while GD achieves 0.17; with SparseGPT, 0.02 vs. 0.22; with GPTQ, 0.18 vs. 0.02 (the one reversal). This means that while GD and RMU both unlearn successfully, RMU's unlearning performance is more robust to the order in which compression and unlearning are applied—a finding the paper highlights as practically significant because deployment order is often determined by external constraints (Section 4.2, Finding 6). On MMLU, all methods show low Order Sensitivity (0.01–0.09), again demonstrating the insufficiency of general utility metrics for detecting composability differences.

Compression method choice matters for unlearning composability. AWQ achieves the most total wins (5) in Table 3, though the wins are close and distributed across criteria. The paper notes that AWQ composes better with unlearning than GPTQ across most metrics, consistent with the editing findings. SparseGPT and Wanda perform comparably (4 wins each).

Composing Knowledge Editing with Machine Unlearning

Section 4.3's headline is that editing and unlearning are highly composable when the right unlearning method is chosen, with RMU demonstrating substantially better composability than GD or GA across all editing metrics.

RMU composes well with all three editors for task-specific metrics. Table 4 shows that when RMU is the unlearning method, composability metrics on editing criteria are uniformly low: Order-free Error on Edit Success is 0.00–0.03 across FT, MEMIT, and LoRA; Order Sensitivity is 0.00–0.01. This means editing performance is near-identical regardless of whether unlearning is applied before or after editing, and the best-case composition performance matches the editing-only baseline (FT→RMU achieves Edit Success of 0.99, Table 20, versus FT→None at 0.99, Table 15). On Edit Generalization, RMU achieves Order Sensitivity of 0.03–0.04 and Order-free Error of 0.07–0.29 (LoRA being the outlier with higher error, reflecting LoRA's weaker generalization even without composition). On WMDP, RMU's unlearning performance is similarly order-robust: Order Sensitivity is 0.00–0.04 across editors, and Order-free Error is 0.28–0.29—meaning unlearning success is preserved regardless of whether editing precedes or follows.

GD achieves comparable editing preservation but with worse unlearning order sensitivity. Table 4 shows that GD's editing-side Order-free Error is good (0.00–0.07 on Edit Success, 0.11–0.29 on Generalization), but its WMDP Order Sensitivity is notably higher than RMU's in most pairings: with Finetuning, GD's WMDP Order Sensitivity is 0.00 (a tie with RMU's 0.04), but with MEMIT it's 0.02 vs. 0.01, and with LoRA it's 0.24 vs. 0.00. The LoRA pairing is particularly informative: the combination GD↔LoRA produces highly order-dependent unlearning (0.24 Order Sensitivity on WMDP) while RMU↔LoRA is order-invariant. This demonstrates that composability is not a property of individual methods alone but of specific method pairings.

GA is unsuitable for composition with editing. GA's catastrophic forgetting (WMDP Order-free Error of 0.28–0.47, MMLU Order-free Error of 0.47–0.64) and extreme order sensitivity on editing metrics (Order Sensitivity of 1.0 on Edit Success with LoRA, 0.48 with MEMIT) make it practically unusable in composition. The paper includes GA as an informative negative result: it demonstrates what failure looks like in the composability framework and establishes a lower bound against which RMU and GD's performance can be compared.

Unlearning does not substantially disrupt editing, but editing can disrupt unlearning depending on the unlearning method. Across all results in Table 4, the editing metrics' Order-free Error is low when paired with RMU or GD (0.00–0.29), suggesting that unlearning—at least when done with these methods—leaves the model's editability largely intact. The reverse direction (editing disrupting unlearning) is where differences between methods emerge, with RMU being robust and GD/LoRA being fragile. This asymmetry is practically significant: it means that if an organization knows it will need to both edit and unlearn a model, it should prefer RMU for unlearning, and the order in which the interventions are applied will be relatively unimportant.

Composability Across Models and Model Variants

Appendix E reports two cross-model analyses that probe the generality of the main results.

Cross-model replication with Mistral 7B Instruct and Yi 1.5 9B Chat (Appendix E.1, Table 9, Figure 4). Order Sensitivity is computed for five composition pairs (MEMIT↔Wanda, MEMIT↔AWQ, MEMIT↔RMU, RMU↔AWQ, RMU↔Wanda) across the three models. The key finding: Order Sensitivity is "generally consistent across LMs" except for Editing↔Compression compositions. For example, MEMIT↔Wanda Edit Success Order Sensitivity varies from 0.24 (Llama) to 0.00 (Mistral) to 0.01 (Yi), with a standard deviation of 0.08 across models. MEMIT↔AWQ Edit Success Order Sensitivity varies from 0.01 (Llama) to 0.11 (Mistral) to 0.01 (Yi), with a standard deviation of 0.03. In contrast, Editing↔Unlearning and Compression↔Unlearning compositions exhibit consistently low Order Sensitivity across all models (standard deviations ≤0.04 for all metrics in those rows of Table 9). This finding is expected given that editing performance is known to vary with model architecture (Meng et al., 2023), but it demonstrates that composability trends involving unlearning may be more architecture-invariant than those involving editing.

Base vs. instruction-tuned model comparison (Appendix E.2.1, Table 10). Comparing Llama-3 8B base with its instruction-tuned counterpart, the paper finds that key composability patterns hold across training regimes: (1) Compression-before-editing outperforms editing-before-compression for both models (Wanda→MEMIT Edit Success: 94.59% base, 99.10% instruct; MEMIT→Wanda: 70.11% base, 93.93% instruct), though the instruct model shows more robustness to ordering. (2) Unlearning-before-compression is better for both models (RMU→Wanda WMDP: 28.54% base, 26.82% instruct; Wanda→RMU: 31.86% base, 31.83% instruct). (3) Editing and unlearning remain largely composable for both variants, with the instruct model showing slightly larger but still modest performance differences between orders.

Model size ablation (Appendix E.2.2, Table 11). Comparing Llama-3 8B with the much smaller Llama-3.2 1B on editing-compression compositions, the paper finds that composability trends generalize across scale within the same architecture family. At 25% sparsity, Wanda→LoRA Edit Success is 95% for 8B and 100% for 1B; Wanda→FT Edit Success is 99% for 8B and 100% for 1B. The preference for compressing before editing holds at both scales. MMLU is understandably lower for the 1B model (33–35% vs. 60%), but the relative pattern of composability is preserved, with an average order sensitivity of 3.60% for 1B versus 2.50% for 8B.

Triplet Compositions

Appendix E.4 extends the framework to three interventions applied sequentially, using the most composable method from each category: Wanda at 25% sparsity (compression), MEMIT (editing), and RMU (unlearning). All six possible orderings are evaluated on both base and instruct Llama-3 8B.

General utility remains stable across all orderings (Tables 13–14). MMLU varies within a narrow range: 53.87% to 56.36% for base (range of 2.49 percentage points) and 53.77% to 59.09% for instruct (range of 5.32 points). This demonstrates that composing three interventions does not cause catastrophic degradation, regardless of order—an important practical finding for deployment scenarios requiring multiple sequential updates.

Task-specific metrics show substantial order sensitivity. Edit Success varies from 93.00% to 99.50% for base (range of 6.50 points) and from 74.82% to 99.10% for instruct (range of 24.28 points). The best overall ordering for base is Wanda→RMU→MEMIT, achieving 99.20% Edit Success, 93.70% Generalization, 30.77% WMDP, and 56.36% MMLU—a strong balance across all criteria. This ordering compresses first, then unlearns, then edits last, consistent with the pairwise findings that editing should be applied last to maximize retention of edited information (compression and unlearning both degrade subsequent editing), and that unlearning before compression preserves unlearning efficacy. For the instruct model, the best balance is Wanda→MEMIT→RMU (99.10% Edit Success, 84.77% Generalization, 34.68% WMDP, 59.09% MMLU), though the unlearning performance (34.68%, closer to random 25% than the base model's 30.77%) is slightly worse.

Implications for deployment ordering. The paper concludes from the triplet experiments that: "compression should typically be applied early in a sequence of interventions, editing should generally be applied last to maximize retention of edited information, and unlearning is most effective when applied between compression and editing" (Appendix E.4.2). This is a practically actionable guideline derived from the empirical patterns.

Ablation Studies and Robustness Checks

Layer overlap between interventions does not explain composability (Appendix D, Table 8): When RMU and MEMIT are forced to modify the same model layers (both targeting layer 3 of Llama-3, versus the default where RMU targets layer 3 and MEMIT targets layers 4–8), composability metrics remain nearly unchanged. Edit Success Order-free Error is 0.03 (different layers) vs. 0.02 (overlapping); Order Sensitivity is 0.01 vs. 0.00. WMDP Order-free Error is 0.29 vs. 0.27. MMLU Order-free Error is 0.44 vs. 0.46. The authors conclude that "intervention composability stems primarily from algorithmic precision rather than layer isolation"—RMU misdirects only unlearning-target-related activations and MEMIT modifies only fact-mediating parameters, so they don't interfere even when operating on the same physical parameters. This ablation isolates surgical precision as the mechanism enabling composability, rather than spatial separation of modifications.

Unlearning hyperparameter sensitivity (Appendix C.4, Tables 5–7): The paper documents the grid searches used to select unlearning hyperparameters. For RMU, the search over $\alpha$ (retain loss weight: {1, 10, 100, 1000, 10000}), layer to modify ({3, 17}), and maximum batches ({100, 150, 200, 250, 300}) found that "RMU is quite sensitive to hyperparameter choice, with most combinations either leaving the model unaffected or significantly harming model utility." The selected configuration ( $\alpha=1000$ , layer 3, 250 batches) represents a narrow operating point—this fragility is itself a composability concern, as it means the method may not generalize to other models or datasets without similarly careful tuning. For GD, the authors found they needed to substantially upweight the retain loss term ( $\alpha=40$ ) compared to prior work, because "it is easy for the model to significantly minimize the correct token likelihood, which causes it to drown out the retain loss term." This is an important methodological finding for practitioners attempting to replicate GD-based unlearning.

Statistical reliability of composability metrics (Appendix E.3, Table 12): Standard errors are reported for all composability metrics in the editing-compression compositions. Key results: Standard errors on Order Sensitivity are small relative to the observed differences between methods—MEMIT's Edit Success Order Sensitivity with Wanda is 0.24 with standard error 0.03, while LoRA's is 0.08 with standard error 0.02, confirming that the 3× difference is reliable. MMLU standard errors are uniformly ≤0.01, confirming that MMLU composability is consistently near-zero. Edit Generalization metrics show larger standard errors (up to 0.04 for Order Sensitivity), reflecting the inherent variability of generalization evaluation across 10 rephrasings, but the relative rankings across methods remain interpretable given the magnitude of differences.

GA as a composability failure mode (Tables 18–21): While not presented as a formal ablation, the consistent failure of GA across all compositions serves as an informative boundary condition. GA→SparseGPT at all sparsity levels yields Edit Success of 0.0 and WMDP never below 0.46 (Table 18)—the method neither edits nor unlearns effectively after composition, and its MMLU degrades to 0.45–0.52 (from 0.62 baseline). GA with editing is similarly catastrophic (Table 20): FT→GA produces Edit Success of 0.0, WMDP of 0.47, and WikiText perplexity of infinity (indicating the model produces degenerate output). GA→Editing (Table 21) shows Edit Success of 0.07 (FT), 0.48 (MEMIT), and 1.0 (LoRA)—the LoRA result is surprising but comes with WMDP of 0.34 and MMLU of 0.36, indicating that while edits can be applied to a GA-damaged model, the model's general utility is severely compromised. These results establish GA as a non-composable unlearning method and demonstrate that the composability framework correctly identifies methods that fail in multi-intervention settings.

Re-compression after editing (implicit in all results): The paper notes (Section 3.1, Model Compression subsection) that "most compression techniques require decompressing models, so we recompress after editing or unlearning using the same compression technique." This is not presented as an ablation but is a design choice that could affect results: editing on a compressed-then-decompressed model might produce different representations than editing on an always-compressed model. The paper doesn't explore whether the decompression-recompression cycle itself causes degradation separate from the compression level.

Editing batch size and statistical averaging: Each editing result is averaged over 10 batches of 50 edits, with the same 10 batches used across all compositions. This controls for edit difficulty variation when comparing across methods. The paper doesn't ablate batch size (e.g., does editing 1 fact at a time produce different composability than batch editing 50?), which could matter if batch editing induces different parameter changes than sequential single edits.

WikiText perplexity as a secondary utility metric: Throughout the appendices (Tables 15–21), WikiText perplexity is reported alongside MMLU. In several compositions, WikiText reveals degradation that MMLU doesn't capture. For example, LoRA→Wanda at 0.25 sparsity achieves MMLU of 0.60 (matching baseline) but WikiText PPL of 9.23 (versus 5.84 baseline for Wanda alone), indicating that while multiple-choice accuracy is preserved, the model's language modeling quality has degraded. This supports the paper's argument for multi-metric evaluation even within general utility assessment—MMLU alone is insufficient, and WikiText captures a complementary aspect of model quality.

Critical Assessment

The paper makes four central claims: (1) existing interventions interact in meaningful, non-trivial ways when composed; (2) model compression consistently hinders other interventions; (3) intervention order dramatically alters outcomes; (4) MMLU is insufficient for measuring composability. I'll assess the experimental support for each, then discuss structural limitations.

Claim 1: Interventions interact in meaningful ways. Strongly supported, with scope boundaries.

The experiments demonstrate systematic, large-magnitude interactions across all three intervention categories. The evidence is strongest for the editing-compression axis (Figure 2, Table 2), where the degradation curves are smooth, monotonic with compression level, and quantitatively large (Edit Success drops from ~0.99 to near 0.0 for some compositions at high sparsity). The editing-unlearning axis (Table 4) shows meaningful variation in composability across unlearning methods—RMU composes well, GD composes moderately, GA fails. The compression-unlearning axis (Figure 3, Table 3) shows smaller but consistent effects.

However, the claim's scope is bounded by the specific methods, datasets, and models tested. The paper acknowledges this limitation (Appendix A): "there are far more possible combinations of interventions, methods, models, and datasets than we can consider in one work." The interactions documented are specific to these ten methods on Llama-3 8B with zsRE/WMDP/MMLU—whether similar patterns hold for detoxification interventions, debiasing methods, capability restoration, or other model families (GPT, Claude, Gemini) is not established. The cross-model experiments with Mistral and Yi (Table 9) begin to address this but cover only a subset of compositions.

Additionally, the paper studies interactions only in consecutive application—intervention A is fully applied, then intervention B is fully applied. It does not study interleaved application (e.g., editing some facts, then unlearning some knowledge, then editing more facts) or simultaneous application (e.g., a multi-objective loss that optimizes editing, unlearning, and compression objectives jointly). These are plausible real-world scenarios that the framework could accommodate but were not tested.

Claim 2: Compression hinders other interventions. Well supported, but directionally nuanced.

The evidence for compression degrading editing is clear and consistent: across all three editors, all four compression methods, and all metrics, higher compression leads to lower editing performance (Figure 2). The quantitative magnitudes are substantial—at 0.75 sparsity, Edit Success drops to near baseline for most editors regardless of order.

For unlearning, the finding is more nuanced. The claim "compression consistently hinders other interventions" (Section 4.4) is supported for RMU and GD when looking at the WMDP metric: applying unlearning after compression (Compress→Unlearn) generally produces worse WMDP scores than applying unlearning before compression (Unlearn→Compress). However, the paper also acknowledges the reverse finding for GA (Section 4.2, Finding 5: "We observe the opposite trend with GA, though GA's poor unlearning performance ... is a possible confounder"), and for GD with certain compression methods at moderate levels, the pattern is not monotonic (e.g., in Figure 3, GD→GPTQ and GPTQ→GD have similar WMDP at 4–8 bits, diverging only at 2–3 bits).

A more precise statement of the finding would be: "Compressing a model before applying unlearning makes unlearning harder for the methods that actually achieve substantial unlearning (RMU, GD), with the effect increasing at higher compression levels. The optimal deployment order is to unlearn first, then compress." The paper's generalization to "compression hinders unlearning" is broadly correct but obscures the order-dependence: unlearning before compression preserves unlearning performance well, suggesting that it is specifically compressing an unlearned model that is safe, while unlearning a compressed model is problematic. This is an important practical distinction.

A limitation is that the paper doesn't explain why compression hinders other interventions mechanistically. The discussion in Section 4.4 speculates about knowledge encoding changes, but no experiments probe the mechanism—no probing of internal representations before and after compression, no analysis of which types of facts are most affected, no gradient-based attribution of the interference. This is understandable for a framework paper whose primary contribution is measurement, not mechanistic explanation, but it leaves the finding as a documented phenomenon rather than a understood one.

Claim 3: Order dramatically alters outcomes. Strongly supported, with method-specific granularity.

The evidence for order dependence is among the paper's strongest. The visual curves in Figures 2–3 show clear, consistent separation between the two application-order lines across compression levels, editors, and criteria. The Order Sensitivity metrics in Tables 2–4 quantify this with specific numbers: 0.24 for MEMIT–Wanda Edit Success, 0.46 for LoRA–SparseGPT Edit Generalization, 0.17 for GD–Wanda WMDP. The paper also documents the direction of order effects: compressing first is better for editing, unlearning first is better for compression+unlearning, and editing/unlearning is order-invariant for RMU but order-dependent for GD.

The limitation is that only two orders are tested for each pair. With three or more interventions (Appendix E.4), all 3! = 6 orderings are tested, but the paper doesn't define composability metrics for triplets—it only reports raw performance. There is no analysis of whether pairwise Order Sensitivity predicts triplet ordering effects, which would be a natural extension. If pairwise sensitivities are additive or multiplicative, one could predict triplet composability from pairwise measurements; if not, it would indicate higher-order interactions that pairwise analysis misses.

The triplet results (Tables 13–14) hint at such higher-order interactions: the best ordering for the base model (Wanda→RMU→MEMIT) isn't simply a concatenation of the best pairwise orderings (which would be Wanda→MEMIT for editing-compression and RMU→Wanda for unlearning-compression—contradictory recommendations). The triplet results synthesize these into a coherent recommendation (compress early, unlearn in the middle, edit last), but this synthesis is post-hoc rather than predicted from pairwise measurements.

Claim 4: MMLU is insufficient for measuring composability. Demonstrated but limited to one utility metric.

The paper provides clear evidence that MMLU Order Sensitivity is near-zero even when task-specific Order Sensitivity is large (0.24 for MEMIT–Wanda Edit Success vs. 0.00 for MMLU; Table 2). This pattern holds across all composition types. The visual evidence in Figures 2–3 shows MMLU curves that are nearly flat and overlapping across orders, while task-specific curves show large divergence.

However, the paper only demonstrates this for MMLU. It also reports WikiText perplexity (Appendix E.5, Tables 15–21), but doesn't compute Order-free Error or Order Sensitivity for WikiText, nor does it systematically compare MMLU and WikiText as composability proxies. A stronger demonstration would show that multiple general utility metrics (MMLU, HellaSwag, ARC, GSM8K, etc.) all fail to capture composability variations while task-specific metrics succeed. The current evidence is sufficient to make the point—MMLU is the most widely used general utility metric in the intervention literature—but falls short of establishing that all general utility metrics are insufficient. It's possible that some other aggregate benchmark would capture composability better than MMLU.

The paper also doesn't explore why MMLU fails to capture composability. A plausible hypothesis: MMLU questions are multiple-choice and draw on broad knowledge; a model can answer correctly even with partially degraded representations if the correct answer is still preferred over the three distractors. Task-specific metrics (F1 on exact token generation, WMDP four-choice with adversarial distractors) are more stringent and detect degradation that MMLU's coarser signal misses. Investigating this mechanism could inform the design of better composability-aware utility metrics.

Structural experimental limitations beyond the four claims:

The # Wins methodology masks effect magnitudes. A method winning 17 comparisons by margins of 0.01 would appear superior to a method winning 2 by margins of 0.20, yet the latter might be practically more significant. The full numeric tables are provided, so this is a presentation issue rather than a data issue, but the paper's emphasis on "wins" in the main text (Tables 2–4) could mislead casual readers.
Single unlearning benchmark (WMDP). The paper acknowledges this limitation (Section 3.1): "recent works have suggested that WMDP performance provides an imperfect evaluation of unlearning writ-large." The composability findings for unlearning might be specific to WMDP's particular knowledge domain (biosecurity and cybersecurity). Unlearning copyright, personal information, or toxic content might compose differently with editing and compression. The paper doesn't test on TOFU (Maini et al., 2024) or other unlearning benchmarks.
No interventions from categories beyond the three studied. The framework is general, but the empirical scope covers only knowledge editing, model compression, and machine unlearning. Detoxification, debiasing, capability restoration, RLHF-based alignment tuning, and other post-training interventions are not tested. The finding that compression hinders editing and unlearning might extend to these categories (if they modify similar parameter subspaces), but this is speculation.
All models are in the 7–9B parameter range. Cross-scale analysis (Appendix E.2.2) compares 8B and 1B models, but no experiments on 70B+ models. The paper acknowledges this: "it is unknown whether our results generalize across model scale" (Appendix A). Larger models may encode knowledge differently (e.g., in more distributed representations that are more robust to compression) or may have more capacity to absorb multiple interventions without interference.
Editing batch size (50 edits) is not ablated. All editing experiments use simultaneous batch editing of 50 facts. Whether composability differs for single-fact editing versus batch editing is untested. Batch editing may produce more distributed parameter changes that interact differently with compression and unlearning.
No analysis of which specific edits or unlearning targets are most affected by composition. The paper reports aggregate metrics but doesn't characterize which types of knowledge are most vulnerable to composition effects. Are edits involving rare entities more fragile under compression? Does unlearning of cybersecurity knowledge compose differently from unlearning of biosecurity knowledge? Such analyses could inform practical guidelines about which interventions to apply in which order for specific knowledge types.

What experiments would have strengthened the paper:

Mechanistic probes: measuring representational similarity (e.g., CKA) between models before and after each composition step to quantify how parameter-space changes interact.
Gradient-based analysis: computing the gradient alignment between editing, unlearning, and compression objectives to predict which compositions will interfere.
Continuous intervention mixing: training a single model with a weighted sum of editing, unlearning, and compression objectives to establish a Pareto frontier of achievable composability, against which sequential composition could be compared.
Temporal sequence realism: simulating a realistic deployment timeline (compress for deployment, discover a factual error weeks later and edit, discover a safety concern months later and unlearn) rather than applying all interventions in rapid succession. The paper assumes interventions are applied sequentially but doesn't study whether the intervening time or model usage affects composability.
Broader compression types: testing structured pruning (removing entire attention heads or layers) rather than only unstructured weight pruning, since structured pruning produces different sparsity patterns that might interact differently with editing and unlearning.

6. Limitations and Trade-offs

Cost of Difficulty Estimation Is Not Accounted For in Headline Gains

The assumption or constraint. The entire compute-optimal framework rests on partitioning prompts into difficulty bins before allocating the inference budget. The paper's method for estimating difficulty—generating 2048 samples per prompt and averaging the PRM's predicted correctness scores—is extraordinarily expensive. The authors explicitly flag this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity."

The consequence. In a realistic deployment, the total cost is difficulty estimation plus strategy execution. For a single question, generating 2048 samples to estimate difficulty consumes 8–64× more compute than the largest test-time budgets studied (256–512 generations). This means the reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. If difficulty estimation costs 2048 samples per prompt and the optimal strategy uses only 16–64 generations, the total compute is dominated by estimation—potentially making the approach less efficient than simply running best-of-512 without difficulty awareness. For batch processing of many questions, the estimation cost could be amortized if the same difficulty predictions are reused, but for interactive, single-question deployments, the overhead is prohibitive.

What evidence exists in the paper. The 4× efficiency gain is visible in Figures 4 and 8: at 16 generations, compute-optimal matches best-of-N at 64 generations for search; at 64 generations, compute-optimal matches best-of-N at 256 for revisions. Difficulty estimation uses 2048 samples per question (Section 3.2). The paper never reports total cost including estimation, nor does it sweep the number of samples used for estimation to find a cost-quality tradeoff.

Mitigation status. The paper explicitly calls this out as "a key avenue for future work" (Section 3.2) and suggests pretraining or fine-tuning a model to predict difficulty directly from question text. This would remove the per-prompt sampling cost entirely—a lightweight classifier could estimate difficulty with a single forward pass. The paper does not implement or evaluate any such model, so this remains a speculative future direction. Adaptive difficulty estimation (start with a few samples, assess difficulty, allocate remaining budget) is also suggested but not explored.

Single Benchmark and Single Model Family

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. No experiments on other reasoning benchmarks (GSM8K, TheoremQA, code generation tasks), other model families (GPT, Claude, Gemini, LLaMA, Mistral), or other model sizes within the PaLM 2 family are reported.

The consequence. Several aspects of the findings could be model-specific or task-specific, and the paper provides no evidence to distinguish the universal from the contingent:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties or different error patterns (e.g., one that produces more diverse but noisier solutions) might exhibit different difficulty-dependent scaling curves and different optimal strategy thresholds.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. A model with weaker in-context learning might not benefit from the edit-distance-based incorrect-correct pairing strategy.
MATH consists exclusively of competition-level symbolic math problems requiring multi-step deductive reasoning. Whether the difficulty-dependent patterns generalize to code generation, logical reasoning, scientific QA, or tasks requiring factual recall rather than step-by-step inference is unknown.
The compute-optimal policy is selected based on ~50 questions per fold per difficulty bin (500 questions ÷ 5 quintiles ÷ 2 folds). This is a small sample, and the selected strategies may not be robust—the paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess statistical reliability.

What evidence exists in the paper. The difficulty-dependent results are consistent across search methods (Figure 3 right), revision strategies (Figure 7 right), and selection mechanisms (majority vs. verifier, Appendix B), but all are within the same model and benchmark. The FLOPs-matched comparison (Section 7, Figure 9) uses a second PaLM 2 model but still within the same family. No cross-model or cross-benchmark experiments are reported. The authors acknowledge the scope limitation in Section 4 but do not mitigate it experimentally.

Mitigation status. Not addressed. The paper explicitly bounds its claims to MATH with PaLM 2-S* but does not provide any evidence for generalizability. Future work replicating on other models, benchmarks, and task domains is recommended but not initiated.

The 14× Larger Model Baseline Is Not Compute-Optimal and Uses Only Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal training (Hoffmann et al., 2022), where both data and parameters are scaled equally. The authors acknowledge this explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model uses only greedy decoding—no majority voting, no best-of-N, no beam search, no test-time compute augmentation of any kind.

The consequence. Both choices make the pretraining baseline weaker than it could be, potentially inflating the reported advantages of test-time compute over scaling pretraining:

A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model, since the latter may be undertrained relative to its parameter count. The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at R ≪ 1 for revisions in Figure 1) may shrink or reverse against a properly compute-optimal larger model.
Giving the larger model even a modest test-time compute budget (e.g., best-of-8 with majority voting) would create a much stronger baseline. The paper's comparison is effectively "small model + extensive inference compute" vs. "large model + zero inference compute," which is not the relevant practical tradeoff—a practitioner deciding between the two would likely give the large model some test-time compute.
The FLOPs-matched analysis uses approximate scaling laws for the FLOPs calculation. The actual cost ratio depends on details of the two models' architectures (e.g., feedforward layer sizes, number of attention heads) that are not disclosed for PaLM 2-S* and its larger variant, making it difficult to verify the 14× parameter ratio and the corresponding FLOPs calculations independently.

What evidence exists in the paper. Section 7 defines the FLOPs accounting and the three R regimes. Figure 9 shows the comparison results. The 14× parameter ratio is stated but not derived. The larger model's architecture is not described beyond its parameter count relative to PaLM 2-S*.

Mitigation status. The paper acknowledges the departure from compute-optimal pretraining and frames it as a deliberate simplification for a representative comparison. No experiments with a Chinchilla-optimal baseline or with test-time compute applied to the larger model are conducted. This is flagged as future work.

Hard Problems Remain Essentially Unsolved—Test-Time Compute Cannot Create Capability

The assumption or constraint. The compute-optimal framework is fundamentally bounded by the base model's pass@1 on a given problem: if the model almost never produces a correct solution, no amount of search or revision can find one. The paper is transparent about this, stating in its Section 7 summary that test-time compute "amplifies existing capability but does not create it from nothing."

The consequence. Across all methods—search, revisions, and their compute-optimal combinations—the hardest difficulty bin (bin 5, where base model pass@1 is near zero) shows near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets (4 to 256 generations). In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio at 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is flat near 0–5% while the 14× larger model achieves higher accuracy—sometimes substantially so.

This means there is a hard capability ceiling: test-time compute cannot substitute for pretraining on problems outside the base model's reach. For applications where a significant fraction of queries fall into this category, the compute-optimal framework offers no advantage, and scaling pretraining remains the only viable path. The framework also provides no mechanism for identifying which problems are fundamentally outside the model's capability range rather than merely hard-but-solvable with enough compute—the difficulty estimator only measures pass@1, which conflates "hard but solvable" with "impossible."

What evidence exists in the paper. The bin 5 results are consistent across every figure: Figures 3, 4, 7, 8, and 9 all show flat or near-flat scaling curves for the hardest problems. The paper explicitly notes this limitation in the Section 7 discussion and the Section 8 future work.

Mitigation status. The paper is candid about this limitation and does not attempt to mitigate it—it is a fundamental property of test-time compute, not a methodological weakness. The finding is valuable as a boundary condition: it precisely delineates where the approach works (easy-to-medium problems within the base model's capability range) and where pretraining must be used instead (hard problems outside that range). The paper suggests that improving verifier robustness and combining search with revisions might expand the effective capability range slightly, but acknowledges that genuinely out-of-distribution reasoning will require pretraining advances.

Sequential Revisions Introduce Latency That Is Not Accounted For

The assumption or constraint. The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock time. Sequential revisions are inherently serial—each revision depends on the previous one—while parallel best-of-N can be executed simultaneously with sufficient hardware. The revision model generates chains of up to 64 sequential steps (Figure 6); each step requires a full forward pass conditioned on the previous output.

The consequence. A strategy that allocates 128 generations as 64 sequential × 2 parallel takes roughly 64× longer wall-clock time than one that runs 128 parallel samples simultaneously, even though both consume the same total FLOPs. The paper's Figure 7 (left) shows that at lower budgets (8–32 generations), fully sequential allocations are optimal for revisions—these are precisely the regimes where latency would be worst relative to parallel sampling. For interactive applications (chat assistants, real-time tutoring, live code generation), the latency penalty of sequential revisions may be unacceptable regardless of accuracy advantages. The paper's compute-optimal policy selects strategies based on accuracy alone, without a latency-aware Pareto frontier.

What evidence exists in the paper. Sequential revisions demonstrate accuracy gains over parallel sampling in Figure 6 (right) and Figure 7. The revision model generates chains up to 64 steps (Figure 6, left). The paper never reports wall-clock time, latency, or throughput for any strategy, nor does it discuss the serial-vs-parallel latency tradeoff. The "generation" cost model (Section 5.3) explicitly treats all generations as equivalent units of compute, abstracting away execution time.

Mitigation status. Not addressed. The paper's generation-budget cost model is standard for inference-time compute scaling studies but is incomplete for practical deployment decisions. A full analysis would need to specify hardware assumptions (batch size, GPU memory, whether multiple chains can be batched) and trade off accuracy against latency. The compute-optimal policy could be extended with a latency constraint or a multi-objective formulation, but no such extension is proposed.

The Revision Model Has a ~38% Correct-to-Incorrect Reversion Rate

The assumption or constraint. The revision model was trained only on sequences where all in-context answers are incorrect followed by a correct target (Section 6.1). During training, it never sees a trajectory where a correct answer appears in context and should be preserved. At test time, the model generates a chain of revisions, and approximately 38% of correct answers produced during the chain get "revised" back to incorrect answers in the subsequent step (Section 6.1):

"the model may encounter correct answers in its context (produced during earlier revisions) and incorrectly 'revise' them into wrong answers. The paper reports that approximately 38% of correct answers get converted back to incorrect ones."

The consequence. Sequential revision chains are inherently unstable: even when the model finds a correct answer, it may not recognize it as correct and will continue revising, potentially degrading the answer. This means the final revision in a chain is often worse than some intermediate revision, forcing the system to use a selection mechanism (majority voting or verifier-based selection) across the entire chain to recover the best answer. This selection mechanism introduces additional complexity and requires storing all intermediate answers—a memory cost not accounted for. More fundamentally, the reversion problem means the revision model lacks a "stop criterion"—it cannot distinguish between "this answer needs improvement" and "this answer is already correct." This is a direct consequence of the training data design (only incorrect-to-correct trajectories) and represents a structural limitation of the approach rather than a hyperparameter issue.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. Figures 6 (left) and 7 show that pass@1 at each revision step plateaus after ~15–20 steps and does not continue improving, consistent with a reversion dynamic where gains at later steps are offset by losses from incorrectly revising previously correct answers. The within-chain selection mechanism (majority voting or verifier) is described in Section 6.1 and Appendix I.

Mitigation status. The paper mitigates the symptom (by selecting the best answer across the chain rather than taking the final output) but does not address the root cause (the model has never been trained to recognize when no revision is needed). The paper suggests that a more principled solution—training the model to output a "no revision needed" token or to stop when the answer is already correct—is possible but not implemented. The ReST^EM experiment (Appendix K, Figure 16) shows that attempting to further optimize the revision model with on-policy training actually worsens the reversion problem, suggesting that fixing it may require fundamentally different training data construction or a modification to the training objective.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around post-training interventions from a method-centric to a lifecycle-centric view. Before this work, each intervention category—knowledge editing, model compression, machine unlearning—was evaluated as if it would be the only modification applied to a model after pretraining. The implicit assumption was that if an intervention preserved general utility metrics (MMLU, perplexity), it was safe to deploy, and that combining interventions would work if each worked individually. The paper systematically dismantles both assumptions.

The magnitude of the shift is not a paradigm change in the Kuhnian sense—the individual intervention methods are not overturned—but it is a substantial reframing of what it means for an intervention to be "good." The paper introduces a new axis of evaluation (composability) with dedicated quantitative metrics (Order-free Error, Order Sensitivity) that the field previously lacked vocabulary to even articulate. This is analogous to how robustness to distribution shift became a standard evaluation axis for computer vision models in the late 2010s: it doesn't invalidate prior accuracy results, but it adds a dimension that practical deployments cannot ignore.

The paper reconciles a latent tension in the literature. Prior work had produced isolated observations about intervention interactions—Hooker et al. (2019) found that compressed models forget certain examples; Brown et al. (2023a) found that edited models are fragile to distribution shifts; Jia et al. (2023) found that sparsity can sometimes simplify unlearning. These findings sat in tension with the intervention communities' implicit assumption that methods were independent and composable by default. The paper's framework provides a unified explanation: interventions interact through shared parameter modifications, the interactions depend on application order (they are non-commutative), and the magnitude of interaction varies dramatically by method pair. The prior isolated observations are unified as instances of a general composability landscape that the paper's metrics can now systematically map.

Equally important is the paper's reconciliation of conflicting intuitions about general utility metrics. MMLU is the standard "did we break the model?" check across all three intervention communities. The paper demonstrates that MMLU is a false signal for composability: two compositions with identical MMLU can have radically different task-specific performance. Table 2 shows MEMIT–Wanda with Order Sensitivity of 0.24 on Edit Success but 0.00 on MMLU. A practitioner checking MMLU would conclude ordering doesn't matter; a practitioner checking Edit Success would see a 24-percentage-point gap depending on order. This finding reframes MMLU from a sufficient safeguard to a necessary-but-insufficient baseline, and the paper provides the tools (Order-free Error, Order Sensitivity) to fill the evaluation gap.

The identification of compression as a systematic bottleneck for the intervention ecosystem (Finding 1, Finding 5, and the cross-cutting trend in Section 4.4) redirects research attention in a concrete way. Compression is "nearly universal in LM deployments" (Section 4), yet the paper shows it consistently degrades editing and unlearning success. This means that the de facto composability ceiling for most deployed models is set by how well interventions survive compression, not by how well they perform in isolation. The implication for research prioritization is direct: developing compression methods that are "editing-aware" or "unlearning-aware"—or developing editing and unlearning methods robust to compression-induced representation changes—is more impactful than pushing the state-of-the-art on any single intervention category in isolation. The paper's finding that AWQ composes substantially better than GPTQ (17 wins vs. 2 wins in Table 2) demonstrates that this is an achievable design target, not a fundamental limitation.

The paper also establishes that composability is a property of specific method pairs, not a fixed cost of sequential application. RMU and MEMIT achieve near-zero Order Sensitivity on both editing and unlearning metrics (Table 4: 0.01 on Edit Success, 0.00–0.04 on WMDP), while GD and MEMIT or LoRA and GPTQ show Order Sensitivity of 0.24–0.46. This variation means composability can be low—methods can be designed to achieve it. It is not an inevitable consequence of modifying a model multiple times. This converts composability from a complaint about real-world deployment friction into a design target that future methods should optimize for.

Follow-Up Research This Work Enables

Mechanistic understanding of why compression hinders editing and unlearning. The paper documents that compression degrades other interventions but does not explain the mechanism. Do pruned weights correspond to the parameters that editing methods most rely on to store new factual associations? Does quantization noise disproportionately affect the low-rank updates from LoRA-based editing? A strong follow-up would measure the overlap between pruned weights and editing-updated weights (e.g., Jaccard similarity between the set of weights zeroed by Wanda at 0.25 sparsity and the set of weights changed by MEMIT by more than some threshold). If the overlap is high, the mechanism is structural: compression removes the very parameters editing needs. If it is low, the mechanism is likely representational: compression subtly distorts the model's internal geometry in ways that make targeted fact updates harder. A second approach would probe the gradient alignment between editing and compression objectives: compute the cosine similarity between the gradient of the editing loss and the gradient of a "compression-aware" loss at the same parameters. High negative alignment would predict composability failure. This would move the field from a documented phenomenon to an understood one, enabling principled design of compression-aware editors.

Training compressors that preserve editability and unlearnability. The paper's finding that AWQ composes substantially better than GPTQ (Table 2) demonstrates that compression method choice matters for composability, but neither method was explicitly designed to preserve composability. A natural follow-up is to add a composability-aware regularizer to the compression objective. For AWQ, which already identifies "salient" weights via activation magnitude, one could extend the saliency criterion to include a term that measures whether a weight is important for edit success—approximated, for example, by the magnitude of the gradient of the editing loss with respect to that weight on a small set of representative edits. The regularizer would upweight these editing-salient weights in the scaling procedure, protecting them from aggressive quantization. The evaluation would measure Order Sensitivity and Order-free Error on editing metrics at various compression levels, comparing against standard AWQ. If the composability-aware compressor achieves lower Order Sensitivity at the same bit-width, it demonstrates that composability is a trainable property, not just a method-selection problem.

Multi-objective interventions trained for composability directly. The current paradigm treats interventions as sequential and independent. The paper's framework enables a fundamentally different approach: train a single intervention that simultaneously optimizes multiple criteria. For example, instead of editing a model and then unlearning hazardous knowledge, one could train a joint editing-unlearning objective with a loss of the form:

$\mathcal{L} = \mathcal{L}_{\text{edit}}(\text{zsRE}) + \alpha \cdot \mathcal{L}_{\text{unlearn}}(\text{WMDP}) + \beta \cdot \mathcal{L}_{\text{retain}}(\text{MMLU})$

The key question is whether a jointly-optimized intervention achieves better composability (lower Order-free Error, lower Order Sensitivity) than sequential application of individually-optimized interventions. The paper's RMU+MEMIT results (Table 4, Order Sensitivity 0.01) establish a strong baseline—can joint optimization match or beat this while also preserving the flexibility to add future interventions? A negative result (joint optimization produces worse composability than sequential RMU+MEMIT) would suggest that the sequential paradigm is actually beneficial—each intervention creates a "clean" model state for the next. A positive result would suggest a path toward multi-objective interventions that pre-empt the composability problem.

Stress-testing composability with adversarial intervention sequences. The paper studies pairwise and triplet compositions of interventions applied in immediate succession. In real deployments, interventions are applied weeks or months apart, with the model processing user queries in between. Does intervening model usage affect composability? For example, if a model is edited to fix a factual error, then serves thousands of user queries for a month, then undergoes unlearning—does the intervening usage cause the edit to drift or become more fragile to subsequent unlearning? A follow-up experiment would simulate this temporal gap: apply intervention A, fine-tune the model on a corpus of generic text (simulating usage), then apply intervention B. Measure whether Order Sensitivity is higher (worse) in the "delayed composition" setting versus immediate sequential application. If the gap is substantial, it implies that composability evaluation must account for model drift over time, not just static composition—a finding that would further motivate lifelong model maintenance research.

Composability of interventions from additional categories. The paper's framework is general, but its empirical scope covers only knowledge editing, model compression, and machine unlearning. A high-priority extension is to measure composability with alignment-tuning interventions (RLHF, DPO), detoxification, and capability restoration (e.g., LASER, Sharma et al., 2023). Each of these modifies model parameters, and each has its own evaluation criteria. The framework's intervention abstraction—every method is an operator $\omega(f_\theta, \gamma)$ evaluated by criterion $\kappa(f, D) \in [0, 1]$ —makes this extension straightforward technically. The key experiments would measure whether detoxification composes with editing (does removing toxic outputs interfere with editing factual knowledge?), whether alignment tuning composes with unlearning (does RLHF make the model more or less amenable to forgetting hazardous knowledge?), and whether low-rank capability restoration (e.g., LASER) survives subsequent compression. The paper's finding that compression is a bottleneck suggests a concrete hypothesis: compression will also degrade these intervention types, and the ranking of compression methods by composability (AWQ > Wanda ≈ SparseGPT > GPTQ) will generalize.

Large-model composability and scaling trends. The paper's models are in the 7–9B parameter range, with a single 1B ablation (Appendix E.2.2) suggesting trends generalize across scale within the same architecture family. The open question is whether composability improves or degrades at larger scales (70B+). There are competing hypotheses: (1) Larger models have more distributed, redundant knowledge representations, making them more robust to compression and thus more composable—individual parameter modifications from editing or unlearning are absorbed by the model's overparameterization. (2) Larger models have more specialized, brittle circuits for specific types of knowledge, making targeted interventions more fragile under composition—an edit that modifies a specific circuit for a fact might be undone by compression that restructures that circuit. Testing this requires running the same compositions (e.g., MEMIT↔AWQ, RMU↔Wanda) on models at 7B, 13B, 70B within the same family (e.g., Llama-3). The metric to track is Order Sensitivity as a function of parameter count. A decreasing trend supports hypothesis (1)—"just use a bigger model, composability takes care of itself." An increasing trend supports hypothesis (2)—composability is a growing concern at scale.

Practical Applications and Downstream Use Cases

Model maintenance pipelines for long-lived deployed systems. Organizations deploying LLMs in production—customer support, medical QA, legal document processing—face a recurring problem: the model must be updated for factual accuracy (the CEO changed, a regulation was updated), compressed for cost-efficient serving, and patched for safety when new harmful capabilities are discovered. The paper provides direct practical guidance for ordering these updates based on systematic evidence. The triplet composition results (Appendix E.4, Tables 13–14) recommend: compress first, unlearn in the middle, edit last. In numbers: on Llama-3 8B, the Wanda→RMU→MEMIT ordering achieves 99.20% Edit Success, 93.70% Edit Generalization, 30.77% WMDP unlearning accuracy (close to the 25% random baseline), and 56.36% MMLU—a strong balance across all criteria. The paper also identifies which specific method choices preserve composability: AWQ over GPTQ for quantization (Table 2: 12 wins vs. 0), RMU over GD over GA for unlearning (Table 4: RMU's Order Sensitivity ≤ 0.04 vs. GD's ≤ 0.67 vs. GA's ≤ 1.0), and MEMIT over LoRA for editing when composability matters (Table 2: 14 wins vs. 4). For a team maintaining a deployed model, these results provide an evidence-based playbook rather than intuition-based trial-and-error.

Deployment-time compression with editing/unlearning guarantees. The paper's finding that compression before editing preserves edit success better than the reverse order (Figure 2, Finding 2) has a direct deployment implication: when a model is compressed for deployment, it should be compressed before any known edits are baked in, not after. In numbers: Wanda→MEMIT at 25% sparsity achieves Edit Success of 0.95 and Generalization of 0.90, while MEMIT→Wanda achieves 0.70 and 0.69 (Table 17 vs. Table 16)—a 25-percentage-point difference. For a medical QA system where factual errors can cause real harm, compressing first and editing the compressed model is the safer pipeline. The paper also shows that AWQ composes with unlearning substantially better than GPTQ: at 4-bit quantization, RMU→AWQ achieves WMDP of 0.27 while RMU→GPTQ achieves 0.27 (Table 18), but AWQ→RMU achieves 0.27 while GPTQ→RMU achieves 0.45 (Table 19)—unlearning a GPTQ-compressed model is substantially harder. A team choosing a compression method for a model that may later need safety unlearning has a clear preference: AWQ.

Safety patching of compressed, previously-edited models. A concrete scenario: a model was compressed to 4-bit with AWQ for mobile deployment, then edited to fix 50 factual errors via MEMIT. Six months later, a new harmful capability is discovered and unlearning is required. The paper's results (Tables 13–14, triplet compositions) indicate this is feasible without catastrophic degradation: the MEMIT→RMU→Wanda ordering (which approximates the historical sequence if compression was applied first, then editing, then unlearning) achieves 94.67% Edit Success, 85.17% Edit Generalization, 28.20% WMDP, and 54.32% MMLU on Llama-3 8B base. The MMLU drop from baseline (61–62% pre-intervention, Table 15) to 54% is non-trivial but not catastrophic—the model remains functional. More importantly, the edits survive both the unlearning and the re-compression step (since the model was compressed, decompressed for editing, and re-compressed). For safety-critical deployments where a model must be patched without the luxury of retraining from scratch, this evidence that sequential safety patching is viable—with quantified tradeoffs—is directly actionable.

Evaluation protocol design for intervention researchers. The paper's finding that MMLU is insufficient for measuring composability (Finding 4, Section 4.1) has immediate implications for how intervention papers should be evaluated. The standard practice—report MMLU, claim the intervention didn't break the model—is demonstrably inadequate. The paper provides an alternative: report (1) single-intervention metrics on the task, (2) single-intervention MMLU, (3) composability metrics (Order-free Error and Order Sensitivity) with at least one other intervention from a different category applied in both orders. Tables 2–4 provide a template for how to present these results. For a knowledge editing paper, the minimum composability evaluation would be pairing the editor with AWQ (as the most composable compressor) and RMU (as the most composable unlearning method) at standard settings (4-bit, 25% sparsity) and reporting Order Sensitivity. This adds at most 4 composition evaluations per editor (AWQ→Edit, Edit→AWQ, RMU→Edit, Edit→RMU) and would dramatically increase the practical relevance of editing evaluations. The paper's codebase makes this extension low-cost: a researcher can add their editor to the framework and run the composability suite with one command.