Tuning Language Models by Proxy

🎯 Pitch

You can 'finetune' a black-box LLM like GPT-3.5 without touching its weights: just take a small model, tune it, and use the gap in its token probabilities to mathematically steer the large model's output distribution. On LLAMA2-70B, this simple decoding-time trick closes 88% of the performance gap from real instruction tuning—and sometimes even beats it by preserving more factual knowledge.

1. Executive Summary

This paper introduces proxy-tuning, a lightweight decoding-time algorithm that steers a large, potentially black-box pretrained language model toward desired behaviors by contrasting the logit predictions of a small tuned "expert" and its untuned "anti-expert" — and applying that difference as an offset to the base model’s output distribution — without ever accessing the base model’s internal weights. Using LLAMA2-7B-CHAT and LLAMA2-7B as the expert and anti-expert to guide LLAMA2-70B, proxy-tuning closes 88% of the performance gap between the base model and its directly-tuned LLAMA2-70B-CHAT version across knowledge, reasoning, and safety benchmarks, while on open-ended TruthfulQA actually surpassing the directly-tuned model by 6.5% in truthfulness, establishing that decoding-time logit arithmetic can effectively substitute for direct parameter updates across instruction-following, domain adaptation, and task-specific finetuning — though its benefits diminish when the domain shift is large and generic pretraining scale provides little complementary signal.

2. Context and Motivation

The Core Problem: Tuning Large LMs Is Increasingly Impractical, or Simply Impossible

The fundamental tension this paper tackles is straightforward: pretrained language models consistently benefit from additional fine-tuning, but the resources required to tune them have become prohibitive, while for many of today's most capable models, tuning is flat-out impossible because the weights are never released. This creates a chasm between the capabilities that users need (instruction-following, domain-specific expertise, task-specific behavior) and what off-the-shelf pretrained models deliver.

The paper frames this concretely in Section 1:

"tuning these models has become increasingly resource-intensive, or impossible when model weights are private (e.g., GPT-4; OpenAI, 2023). Thus there remains a challenge of how to efficiently customize ever-larger LMs for the needs of diverse users and applications."

This is not a niche concern. It affects three categories of users simultaneously:

Researchers and practitioners with limited compute budgets: Fine-tuning a 70B-parameter model through standard supervised fine-tuning requires substantial hardware (the paper itself uses four 80GB A100s for 7B and 13B models, and a 256-chip TPU v3 for 70B; Appendix A.3). Adding RLHF — the canonical recipe for instruction-tuning — requires even more resources. As model sizes grow (and the scaling trend shows no signs of stopping), the fraction of the community that can afford direct tuning shrinks in parallel.
Users of proprietary, black-box models: When model weights are private, no amount of compute can tune them directly. The most capable models today — GPT-4, Claude, Gemini — are accessed exclusively through APIs. Organizations deploying these models cannot fine-tune them on proprietary data or adapt them for specialized workflows without the model provider's cooperation (and usually payment). This is not merely a cost concern; it is a fundamental architectural constraint: the weights simply are not accessible.
Model providers themselves: Even for organizations that control the training pipeline, the computational expense of full parameter updates at scale motivates exploration of cheaper alternatives. If a decoding-time method can achieve comparable results without updating weights, it represents a direct cost savings.

The paper identifies this gap as a problem of accessibility and customization. Large pretrained models are increasingly powerful generalists, but making them into capable specialists for your task, your domain, or your safety requirements remains expensive or impossible. The practical consequence is a concentration of capability: only those who can afford to tune (or who control the weights) can adapt models to their needs.

The "Alignment Tax" and the Side-Effects of Direct Tuning

Beyond the resource question, the paper highlights a second motivation: direct fine-tuning can degrade pretrained knowledge — a phenomenon sometimes called the "alignment tax" (Ouyang et al., 2022, cited in Section 9). The authors write:

"Indeed, full finetuning is an invasive approach that risks forgetting of previously learned information (McCloskey & Cohen, 1989); for instruction-tuning, this has sometimes been dubbed the 'alignment tax'"

This is not merely a theoretical concern. The paper provides direct evidence for it in their results (Table 3, Section 3.2): on TruthfulQA's open-ended setting, the directly-tuned LLAMA2-70B-CHAT achieves only 85.8% truthfulness, while proxy-tuning the same base model (which never updates its weights) achieves 92.3% — a 6.5 percentage point improvement. The authors attribute this to better knowledge preservation: the base model's factual knowledge, acquired through massive pretraining, remains intact when weights are frozen, whereas direct tuning can inadvertently overwrite or suppress that knowledge.

This finding reframes the motivation: proxy-tuning is not merely a cheaper approximation of direct tuning — it can be a qualitatively better approach for tasks where preserving pretrained knowledge matters. The "alignment tax" is well-documented in the RLHF literature (models becoming less truthful or less knowledgeable after alignment), and a method that achieves similar behavioral improvements without incurring this tax addresses two problems at once.

Prior Approaches and Their Limitations

The paper situates itself against several existing strategies for customizing large LMs, identifying specific weaknesses in each:

1. Parameter-Efficient Fine-Tuning (PEFT) Methods Are White-Box Only

A large body of work has developed methods for tuning LMs by updating only a small subset of parameters: adapters (Houlsby et al., 2019), prefix tuning (Li & Liang, 2021), LoRA (Hu et al., 2022), and QLoRA (Dettmers et al., 2023). These dramatically reduce the memory footprint compared to full fine-tuning, but they share a disqualifying limitation for the black-box setting:

"Nonetheless, these methods require white-box model access, which is unavailable for many of today's advanced models." (Section 8)

The paper directly compares against LoRA in Appendix D (Table 15), showing that LoRA sometimes outperforms proxy-tuning (e.g., on TriviaQA at 70B, LoRA achieves 75.3% vs. proxy-tuning's 62.7%) and sometimes underperforms (on GSM at 13B, proxy-tuning achieves 43.9% vs. LoRA's 32.4%). But the critical distinction is that LoRA requires full access to model weights — it is not an option for GPT-4 or Claude users. Moreover, the training efficiency comparison (Table 14) shows that full fine-tuning a 7B expert (for proxy-tuning) is 15× faster than applying LoRA to a 70B model on the same hardware, suggesting that even in white-box settings, proxy-tuning can be more compute-efficient.

2. Prompt Engineering Is Brittle and Context-Intensive

For instruction-following specifically, carefully crafted prompts can elicit behaviors surprisingly competitive with instruction-tuning (Han, 2023; Lin et al., 2023, both cited in Section 8). However, the paper notes a practical limitation:

"these prompts tend to be quite long, introducing an inference-time computational burden and restricting the length of generations for models with limited context windows."

Long prompts increase the cost of every generation (since the model processes the entire prompt at each decoding step) and consume precious context-window space that could otherwise hold the actual task content. Prompts are also notoriously brittle — small changes in wording can produce dramatically different behaviors — and they require substantial human expertise to craft effectively. They represent a heuristic approach rather than a principled one, and they cannot encode complex task formats or domain-specific knowledge the way tuning can.

3. Controllable Generation Methods Target Attributes, Not Behaviors

There is a rich literature on controllable generation — methods that steer text toward desired attributes like non-toxicity, positive sentiment, or formality (Krause et al., 2021; Yang & Klein, 2021; Deng & Raffel, 2023, all cited in Section 8). The paper acknowledges this lineage but draws a clear distinction:

"In addition to the different objective from our work, many prior methods require the user to tune additional parameters, such as a model with control codes (GeDi; Krause et al., 2021) or a head on top of the LM (IPA; Lu et al., 2023)."

Controllable generation typically optimizes for a single, well-defined attribute. Proxy-tuning's goal is more ambitious: it aims to replicate the broad behavioral transformation that results from instruction-tuning, domain adaptation, or task fine-tuning — not just suppressing toxicity or formalizing tone. And critically, the paper emphasizes that proxy-tuning requires no additional training of auxiliary models beyond the small tuned expert (which may already exist off-the-shelf):

"In contrast, proxy-tuning allows users to leverage the rich collection of small tuned models available online, potentially composing them off-the-shelf with no additional training."

4. Contemporary Work: Mitchell et al. (2024) and Ormazabal et al. (2023)

The paper identifies two especially close works, acknowledging their shared intellectual ground while differentiating their contributions:

Mitchell et al. (2024) applies the same DEXPERTS equation as proxy-tuning (contrasting tuned and untuned small models to steer a large model) for instruction-tuning. However, the paper argues that Mitchell et al. treat the method primarily as an analytical tool for studying the separate effects of pretraining scale versus instruction-tuning — they "do not measure the method's effectiveness on existing benchmarks" (Section 8). Proxy-tuning, in contrast, provides comprehensive empirical evaluation across knowledge, reasoning, safety, domain adaptation, and task-specific benchmarks, demonstrating that the method is not merely conceptually interesting but practically effective at scale.

Ormazabal et al. (2023) also combine probability distributions from a small tuned model and a large pretrained model, but through a learned combination function that requires additional training data and optimization. Proxy-tuning's combination function is parameter-free (the DEXPERTS equation with a fixed coefficient of 1), which the paper argues is a practical advantage — no extra training, no hyperparameter data needed, just off-the-shelf models combined at decoding time. (The paper does later explore an optional α hyperparameter in Section 6.2 for more granular control, but its main experiments use α = 1 throughout.)

The Conceptual Framework: Logit Arithmetic and the DEXPERTS Equation

The paper's approach is grounded in a specific lineage of logit arithmetic — methods that combine the output distributions of multiple language models through algebraic operations on their logits. The core equation (Eq. 1) comes directly from DEXPERTS (Liu et al., 2021):

$p_{\tilde{M}}(X_t \mid x_{<t}) = \text{softmax} \left[ s_M(X_t \mid x_{<t}) + s_{M^+}(X_t \mid x_{<t}) - s_{M^-}(X_t \mid x_{<t}) \right]$

where $s_M$ , $s_{M^+}$ , and $s_{M^-}$ are the logit scores (the final unnormalized outputs before the softmax) from the base model, the expert (tuned small model), and the anti-expert (untuned small model), respectively.

The paper notes that this equation can be interpreted in two complementary ways (Section 2), and both interpretations matter for understanding why it works:

Interpretation 1 (Tuning as an additive offset): $s_M + (s_{M^+} - s_{M^-})$ — the base model's logits are shifted by the difference that results from tuning the small model. If tuning increases the small model's confidence in a helpful token (say, "I" → 0.3 probability from 0.1 before tuning), that boost is applied to the large model's own estimates. The tuned small model effectively "teaches" the large model what the behavioral shift looks like, token by token.

Interpretation 2 (Contrastive amplification for the small expert): $s_{M^+} + (s_M - s_{M^-})$ — the difference between the large and small untuned models captures what additional knowledge or reasoning the larger pretraining scale provides, and this difference is added to the small tuned model. This gives the small expert the benefit of the larger model's broader knowledge while retaining the behavioral pattern it learned during tuning.

The paper's framing in Section 2 is explicit about the relationship to prior logit arithmetic work:

"Proxy-tuning operates on M's output distribution over next word by adding a logit offset for every token, determined by the difference between logits from $M^-$ and $M^+$ . This is an application of decoding-time experts (Liu et al., 2021)"

And it situates itself within a broader trend:

"There has been a growing body of methods that perform arithmetic on multiple logit distributions for better text generation, such as contrasting the logits of a large and small model (Li et al., 2023), logits from different layers of a model (Gera et al., 2023; Chuang et al., 2023), and logits from the same model given different inputs (Shi et al., 2023; Pei et al., 2023; Sennrich et al., 2023; Leng et al., 2023)" (Section 8).

The conceptual contribution is not the equation itself — it is the demonstration that this particular configuration (small tuned expert minus small anti-expert, applied to large base model) works reliably across a diverse range of tuning objectives (instruction-following, domain adaptation, task-specific behavior) and at substantial scale disparities (7B proxies steering 70B models), while requiring only the output logits — not the weights — of the large model.

Why the Logit-Level Interface Matters

The paper's final motivation is infrastructural. Accessing a model's output logits (or even just the top- $k$ log probabilities, as demonstrated with GPT-3.5 in Section 7) is a dramatically lower barrier than accessing its weights. The authors argue this explicitly in their conclusion:

"At a minimum, we encourage model-producing organizations to share output probabilities from their models to enable use of these methods."

This is a policy-relevant argument: if model providers expose even minimal distribution information (top-5 logits, as the GPT-3.5 API already does), users gain the ability to customize models for their specific needs without the provider needing to release weights, build fine-tuning infrastructure, or compromise on proprietary model protection. The GPT-3.5 case study (Section 7) demonstrates this in a truly black-box, limited-information setting — achieving a statistically significant 2.3% absolute improvement on temporal knowledge questions using only the top-5 log probabilities — suggesting that the approach is not merely a proof of concept but deployable on today's commercial APIs.

3. Technical Approach

3.1 Reader Orientation

The paper proposes a decoding-time algorithm — a procedure that intervenes during text generation to alter the model's output distribution — that makes a large, pretrained language model behave as if it had been fine-tuned, without ever modifying or accessing the large model's internal parameters. The problem it solves is the inaccessibility of large-scale tuning (due to cost or proprietary weights), and the shape of the solution is remarkably simple: tune a small model, then apply the logit-level difference between the small tuned and small untuned models as a steering offset to the large model's predictions at every token generation step.

3.2 Big-Picture Architecture (Diagram in Words)

The proxy-tuning system operates at the token level during autoregressive generation and has four logical components:

The Base Model ( $M$ ): A large, pretrained but untuned language model (e.g., LLAMA2-70B) that the user wants to steer toward desired behavior. The user only needs access to its output logits $s_M$ — not its weights, gradients, or internal activations.
The Expert ( $M^+$ ): A small, tuned version of a pretrained model that exhibits the desired behavior (e.g., LLAMA2-7B-CHAT for instruction-following, or a task-specifically fine-tuned 7B model). This model has been directly trained to produce the target behavior.
The Anti-Expert ( $M^-$ ): The same small pretrained model as the expert, but before it was tuned (e.g., LLAMA2-7B without instruction-tuning). This captures what the small model would predict in the absence of the desired behavior.
The Logit Offset Combiner: At each time step, all three models process the same context $x_{<t}$ and produce logit vectors over the shared vocabulary. The combiner computes $s_M + (s_{M^+} - s_{M^-})$ , applies softmax, and samples the next token. The difference $(s_{M^+} - s_{M^-})$ captures the direction and magnitude of the behavioral shift induced by tuning the small model, and this offset is added to the large model's logits, steering it in the direction of tuning while preserving the large model's broader knowledge.

Information flows linearly at each decoding step: context → parallel forward passes through $M^-$ , $M^+$ , and $M$ → logit offset computation → softmax → token selection → appended to context → repeat.

3.3 Roadmap for the Deep Dive

First, the core equation (Eq. 1) in full operational detail, including the two mathematically equivalent but conceptually distinct interpretations that explain why proxy-tuning works from different angles.
Second, the prerequisite conditions that must hold for proxy-tuning to be applicable: shared vocabulary between models, access to logits (or top- $k$ log probabilities), and the relationship between $M^-$ and $M^+$ .
Third, the instruction-tuning instantiation (Section 3) as the primary experimental configuration, to ground the abstract method in a concrete example with specific models, prompts, and evaluation benchmarks.
Fourth, the domain adaptation instantiation (Section 4) to show how the method generalizes when the tuning objective shifts from dialogue behavior to code generation, and how the choice of expert is determined by the application.
Fifth, the task-specific fine-tuning instantiation (Section 5) to demonstrate how proxy-tuning handles structural output constraints (exact-answer formats, mathematical derivations with angle-bracket notation) that the base model has never seen.
Sixth, the GPT-3.5 case study (Section 7) to show how proxy-tuning operates in the extreme-limited-information regime where only top-5 log probabilities are available and where both proxy models are weaker than the base model.
Seventh, the optional $\alpha$ hyperparameter (Section 6.2) as a control mechanism for trading off between different behavioral attributes.
Eighth, runtime and efficiency considerations (Appendix C) to quantify the computational cost of proxy-tuning relative to direct tuning.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that the behavioral shift induced by fine-tuning a small language model can be captured as a token-level logit offset and transferred to a large model through decoding-time arithmetic, without requiring access to the large model's weights.

The Core Equation: DEXPERTS Logit Arithmetic

The entire method is encapsulated in a single equation (Eq. 1) applied at every autoregressive decoding step. At each time step $t$ , the three models — base $M$ , expert $M^+$ , and anti-expert $M^-$ — each receive the same prefix $x_{<t}$ (the concatenation of the prompt and all previously generated tokens) and independently compute logit scores over the shared vocabulary $\mathcal{V}$ . These logits are the raw, unnormalized outputs from the final language-modeling head before any softmax is applied. They represent the model's confidence in each vocabulary token being the next word, encoded in an unnormalized log-probability space. The proxy-tuned model's probability distribution is then:

$p_{\tilde{M}}(X_t \mid x_{<t}) = \text{softmax} \left[ s_M(X_t \mid x_{<t}) + s_{M^+}(X_t \mid x_{<t}) - s_{M^-}(X_t \mid x_{<t}) \right]$

where $s_M(\cdot \mid x_{<t}) \in \mathbb{R}^{|\mathcal{V}|}$ is the logit vector from the large base model $M$ , $s_{M^+}(\cdot \mid x_{<t}) \in \mathbb{R}^{|\mathcal{V}|}$ is the logit vector from the small tuned expert model $M^+$ , and $s_{M^-}(\cdot \mid x_{<t}) \in \mathbb{R}^{|\mathcal{V}|}$ is the logit vector from the small untuned anti-expert model $M^-$ .

What it computes: At each decoding step, the equation takes three vectors of $|\mathcal{V}|$ logits — one from the large model we want to steer, one from the small model that already exhibits the desired behavior, and one from the small model before it learned that behavior — and produces a new vector of $|\mathcal{V}|$ probabilities by: (1) computing the difference vector $s_{M^+} - s_{M^-}$ , which captures token-by-token how tuning shifted the small model's confidence — positive values indicate tokens that tuning promotes (e.g., polite refusals, reasoning starters), negative values indicate tokens that tuning suppresses (e.g., toxic continuations, bare numeric answers); (2) adding this difference vector to $s_M$ , shifting the large model's logits in the same direction; and (3) applying softmax to convert the modified logits into a valid probability distribution from which the next token is sampled. The operation is performed at every autoregressive step, meaning the proxy-tuned model's own prior outputs become part of the conditioning context for all three models in subsequent steps.

Why this form: The key property is the additive decomposition into a base signal and a tuning signal. This form isolates what tuning changed (the difference between $M^+$ and $M^-$ ) from what the large model knows (the base logits $s_M$ ). An alternative approach — directly sampling from the small tuned model — would discard the large model's superior knowledge and reasoning capabilities. Another alternative — averaging log probabilities between the large base model and the small tuned model — would dilute both signals rather than applying the tuning as a directional shift. The subtraction of $s_{M^-}$ is essential because it subtracts out the small model's pretraining biases that are shared with the expert but have nothing to do with tuning — leaving only the signal attributable to the tuning process itself. Without the anti-expert subtraction, the equation would be $s_M + s_{M^+}$ , which would indiscriminately amplify any token that the small tuned model favors, including tokens it favored before tuning due to its pretraining, not due to the behavioral intervention. The subtraction ensures that the offset is approximately zero for tokens unaffected by tuning, and non-zero only for tokens whose probability changed as a result of the tuning process. This is the same mathematical form as the DEXPERTS framework (Liu et al., 2021), which the paper adopts directly without modification in their main experiments (the optional hyperparameter $\alpha$ explored in Section 6.2 is introduced only for analysis).

The paper provides two mathematically equivalent rearrangements of Eq. (1) that reveal different conceptual properties (Section 2). The first grouping is $s_M + (s_{M^+} - s_{M^-})$ , which reads as: "the base model's predictions, shifted by the effect of tuning at a smaller scale." This perspective treats the tuning offset $(s_{M^+} - s_{M^-})$ as the primary learned signal and $s_M$ as the carrier. The second grouping is $s_{M^+} + (s_M - s_{M^-})$ , which reads as: "the small tuned model's predictions, augmented by whatever additional capabilities the larger pretrained model provides beyond the small pretrained model." This perspective treats the contrast $(s_M - s_{M^-})$ as capturing the benefit of larger-scale pretraining (e.g., more factual knowledge, better reasoning patterns), which is then used to improve the small expert. Both interpretations are correct and highlight different aspects of why proxy-tuning works: the first framing emphasizes that tuning is preserved faithfully, while the second framing emphasizes that the large model's knowledge is injected into the tuned small model's behavior. The simultaneous validity of both perspectives is the core theoretical justification for the method's effectiveness.

Prerequisites: What Proxy-Tuning Requires

For proxy-tuning to be applicable, several conditions must be satisfied. These are not assumptions — they are explicit requirements that the paper identifies and discusses (Section 2, Section 7), and understanding them is essential for assessing when the method can be deployed.

1. Shared vocabulary between all three models. The logit vectors $s_M$ , $s_{M^+}$ , and $s_{M^-}$ must have the same dimensionality $|\mathcal{V}|$ and correspond to the same vocabulary items at each index. This is because the logit offset operates element-wise: the $i$ -th component of $s_{M^+} - s_{M^-}$ must meaningfully correspond to the same token as the $i$ -th component of $s_M$ . When models are from the same family (e.g., all LLAMA2 variants, which use the same SentencePiece tokenizer), this condition is automatically satisfied. The paper notes (Section 2) that even some closed-source models share open-source tokenizers — "tokenizers are often open-source, even for closed-source models like GPT-4 (https://github.com/openai/tiktoken)" — making it feasible to match vocabularies between proprietary and open models. When vocabularies do not match, the paper cites the technique of Kasai et al. (2022) for aligning distributions across different tokenizers, though this is not explored experimentally.

2. Access to the output logits of the base model. The method does not require weights, gradients, forward hooks, or any internal representation. It requires only the logit vector over the vocabulary at each decoding step. The paper shows that even partial logit access is sufficient: in Section 7, they proxy-tune GPT-3.5 using only the top-5 log probabilities provided by the API, restricting the offset computation to only those five tokens and setting all other logits to $-\infty$ (or equivalently, zero probability). This extreme case works because the multiple-choice format constrains the answer space to four tokens (A, B, C, D), making the top-5 coverage sufficient.

3. The existence of a small pretrained model that can be tuned to create the expert and anti-expert pair. The paper uses LLAMA2-7B as $M^-$ throughout and creates $M^+$ from it either by using off-the-shelf tuned variants (LLAMA2-7B-CHAT for instruction-tuning, CODELLAMA-7B-PYTHON for code) or by fine-tuning LLAMA2-7B themselves (for task-specific experts in Section 5, and for the GPT-3.5 temporal adaptation expert in Section 7). The anti-expert $M^-$ must be the exact same model that $M^+$ was initialized from before tuning; otherwise the difference $s_{M^+} - s_{M^-}$ would capture differences in initialization or architecture rather than the effect of tuning.

4. The base model must be capable of producing the desired behavior in principle. Proxy-tuning shifts the base model's output distribution, but if the base model assigns probability zero (or extremely close to zero) to a token that the expert promotes, the offset may not be sufficient to elevate that token to a meaningful probability. The paper does not formalize this with a theorem, but it is implicit in the experimental design: proxy-tuning is applied to LLAMA2 base models that have been pretrained on broad data and are capable of producing coherent text; the method nudges them toward specific behavioral patterns they are capable of but do not naturally exhibit.

The Instruction-Tuning Instantiation (Section 3)

The primary experimental configuration uses proxy-tuning for instruction-following — the transformation that converts a raw pretrained model (which simply predicts the next token given a text prefix) into a conversational assistant that answers questions, follows instructions, and refuses harmful requests. The specific instantiation is:

Base model $M$ : LLAMA2-BASE at either 13B or 70B parameters. These are pretrained on text corpora but have not undergone any instruction-tuning or RLHF. They produce fluent continuations but do not reliably answer questions, follow formatting, or avoid toxic content.
Expert $M^+$ : LLAMA2-7B-CHAT. This is a 7B-parameter model that has been extensively tuned through supervised instruction-tuning and RLHF to behave as a helpful, harmless assistant.
Anti-expert $M^-$ : LLAMA2-7B-BASE. This is the 7B pretrained model before the instruction-tuning process that produced $M^+$ . It is the initialization checkpoint from which $M^+$ was derived.

A critical implementation detail is that the expert receives a different prompt format from the base and anti-expert models (Section 3, Appendix A.1, Table 8). LLAMA2-CHAT models are trained with special control tokens: [INST] and [/INST] delimit the user's instruction, and an optional <<SYS>> block provides a system-level prompt. The paper preserves these conventions: the chat expert $M^+$ receives the prompt wrapped in [INST] ... [/INST], while the base model $M$ and anti-expert $M^-$ receive the same content without special tokens (e.g., a plain "Answer the following question. Question: {question} Answer:" format). This means the three models are not processing identical string inputs — they are processing semantically equivalent prompts formatted according to their training conventions. The paper argues this is appropriate because the goal is to replicate the behavioral shift that instruction-tuning produces, and the chat model's behavior is defined with respect to its expected prompt format. Giving the chat model a plain-text prompt would put it out-of-distribution and potentially degrade its behavior, while giving the base model chat tokens would be meaningless since it was not trained with them. The offset $s_{M^+} - s_{M^-}$ thus captures not just what the chat model knows but also how it interprets the structured prompt format — and this interpretation signal is transferred to the base model even though the base model never sees the special tokens.

For evaluation, the paper uses greedy decoding (temperature = 0, selecting the highest-probability token at each step) across all models and all benchmarks. The maximum generation length is 512 new tokens for most tasks (2048 for AlpacaFarm). No special stop sequences are specified except for ToxiGen, where newline is an additional stop token to prevent the model from continuing to the next hateful statement. For TruthfulQA's open-ended setting, the chat expert receives a system prompt (Table 9) instructing it to be "helpful, respectful and honest" and to avoid harmful content — this prompt is used because the authors found it "dramatically improves performance for CHAT models" (Appendix A.1). The base and anti-expert models do not receive this system prompt.

The evaluation benchmarks and their answer extraction procedures (Appendix A.1) are:

AlpacaFarm (805 test examples): Open-ended instructions; responses evaluated by GPT-4 for win-rate against text-davinci-003 reference responses; no answer extraction needed.
GSM (1,319 test examples): Math word problems; the last number appearing in the model's response is extracted as the predicted answer; exact match against the ground-truth number.
ToxiGen (2,800 sampled examples): Prompts consisting of sequences of hateful statements; a RoBERTa-large toxicity classifier scores whether the generation continues the hateful pattern; percentage of toxic generations reported.
TruthfulQA open-ended (817 examples): Misleading questions; two GPT-3-based classifiers judge truthfulness and informativeness; primary metric is percentage of responses that are both truthful and informative.
TruthfulQA multiple-choice (817 examples): Same questions but with four answer options; the first character after "The answer is:" is parsed as the predicted option A/B/C/D; accuracy reported.

The Code Adaptation Instantiation (Section 4)

The second experimental configuration tests whether proxy-tuning can specialize a general pretrained model for a technical domain — in this case, Python code generation. This differs from instruction-tuning in that the target behavior is not conversational style or safety but domain-specific knowledge and syntactic patterns.

Base model $M$ : LLAMA2-BASE at 13B or 70B.
Expert $M^+$ : CODELLAMA-7B-PYTHON (referred to as 7B-CODE for readability). This model was initialized from LLAMA2-7B, then further trained on a general code corpus (CodeLLAMA), and finally specialized on Python code. It is an off-the-shelf model available from the CodeLLAMA release.
Anti-expert $M^-$ : LLAMA2-7B-BASE — the same starting point used to create the code expert.

An important technical detail: CODELLAMA uses the same tokenizer as LLAMA2, ensuring vocabulary compatibility without any alignment step. The paper explicitly notes this in Appendix A.2: "CODELLAMA uses the same tokenizer as LLAMA2, enabling us to combine outputs from the two models."

The evaluation protocol differs substantially from instruction-tuning. Code generation requires measuring functional correctness, not just text plausibility. The paper uses pass@10 — the probability that at least one out of 10 sampled solutions passes all unit tests — estimated by sampling 20 solutions per problem (following the statistical estimation procedure from Chen et al., 2021). Sampling uses top- $p$ = 0.95 and temperature = 0.8, the same settings as the original Codex evaluation. The maximum generation length is 512 tokens for both benchmarks.

Specific post-processing is applied (Appendix A.2): the tokens "pass" and "..." are banned during generation by setting their logits to $-\infty$ , because the base model sometimes writes exercise templates (e.g., "pass" as a placeholder) rather than actual implementations. Lines starting with "print" or "assert" (ignoring leading whitespace) are removed from the final generation because these are often debugging artifacts rather than core solution logic. Stop tokens are dataset-specific: for CodexEval, any of "\nclass", "\ndef", "\n#", "\nif", or "\nprint" terminates generation (these indicate the start of a new code block outside the function); for DS-1000, "\n</code>", "\n# SOLUTION END", and "\nEND SOLUTION" are stop tokens.

The evaluation benchmarks are:

CodexEval (HumanEval, 164 problems): Function completion given a signature and docstring; solutions evaluated by running provided unit tests.
DS-1000 (200 sampled problems): StackOverflow-sourced Python programming tasks in the Completion setting; solutions evaluated by running hidden test cases.

The Task-Specific Finetuning Instantiation (Section 5)

The third setting explores whether proxy-tuning can replicate task-level fine-tuning, where a model is trained on a specific input-output format and must adhere to strict structural constraints that the base model has never encountered.

The paper fine-tunes LLAMA2-7B on the training sets of two tasks to create task-specific experts, then uses these experts to steer larger base models. The key difference from the instruction-tuning setup is that the experts are directly trained by the authors rather than taken off-the-shelf — this is the only setting where models are tuned from scratch.

Training procedure (Appendix A.3, Table 11): For each task, LLAMA2-7B is fine-tuned using standard supervised fine-tuning (next-token prediction on the target answer). Hyperparameters are taken from the Tülu 2 recipe (Ivison et al., 2023): training for 2 epochs, effective batch size 128, learning rate $2 \times 10^{-5}$ , weight decay 0, warmup ratio 0.04, maximum sequence length 2048, and BFloat16 precision. The 7B and 13B models are trained on four 80GB A100 GPUs; the 70B baseline model is trained on a 256-chip TPU v3 for comparison. Full fine-tuning is used (not LoRA) for the 7B expert to maximize its quality, since the computational cost of tuning 7B is manageable.

TriviaQA (87,622 training examples, 11,313 dev examples): A question-answering dataset of trivia questions. The model is trained to predict the answer given the question, conditioned on the prompt "Question: {question}\nAnswer:". The target is a short answer phrase (e.g., "Ross Bagdasarian"). At evaluation, the model's response is compared against the reference answers and their aliases using exact match — a strict metric that requires the generated string to be identical to an accepted answer. This metric is appropriate because, for a fixed task, users typically want a specific answer format.

GSM (7,473 training examples, 1,319 test examples): Math word problems. The model is trained to predict the full answer passage from the dataset, conditioned on "Question: {question}\nAnswer:". Critically, the GSM answer passages have a specific formatting style that the training data enforces: intermediate equations are enclosed in double angle brackets (e.g., <<16-3=13>>), and the final answer is stated after four hash symbols (e.g., #### 30). These formatting conventions are completely absent from the LLAMA2 base models' pretraining data; the base models never spontaneously produce <<...>> notation or #### delimiters. The task expert learns this format through supervised fine-tuning, and proxy-tuning transfers this formatting behavior to the larger models. For evaluation, the last number in the response is extracted as the predicted answer (same extraction method as Section 3), and exact match is checked. The paper reports (Section 5.2) that "99.7%+ of generations from proxy-tuned models (at both 13B and 70B) state the final answer after ####" — demonstrating that proxy-tuning successfully promotes even extremely unlikely structural tokens from near-zero probability to dominant probability.

The anti-expert $M^-$ is always LLAMA2-7B-BASE in these experiments — the same checkpoint used as the initialization for fine-tuning the task experts. The base models $M$ are LLAMA2-13B-BASE and LLAMA2-70B-BASE.

The GPT-3.5 Black-Box Case Study (Section 7)

This experiment demonstrates proxy-tuning in the most constrained setting: a truly black-box model (GPT-3.5-turbo-0613) accessed through an API that provides only minimal distribution information.

The information constraint: The GPT-3.5 API provides log probabilities for only the top 5 tokens at each step. This means that for any token outside the top 5, the proxy-tuning equation cannot be computed — the logit value is unknown. The paper circumvents this by operating in a multiple-choice setting where the answer space is limited to four tokens: A, B, C, and D. Since the top 5 log probabilities almost certainly include these four answer tokens, the offset computation is feasible. The paper notes this explicitly: "In this setting, we have extremely coarse information about the base model's predictions, as the API provides log probabilities for only the top 5 tokens" (Section 7). Additionally, the API does not allow conditioning on partial model responses — it always generates from the start of a new conversational turn — which "prevents us from applying proxy-tuning to any task involving generation of more than one token" (Section 7 footnote). This is a fundamental limitation of the API interface, not a limitation of the method, but it constrains the experimental design to single-token prediction tasks.

The expert and anti-expert: Both are based on LLAMA2-7B. The expert $M^+$ is obtained by continuing pretraining LLAMA2-7B on recent data, but with an oracle-like shortcut: instead of web-scraped recent data (which would be the realistic approach), the authors retrieve 10 articles per query from REALTIMEQA using the Google API and continue pretraining on only those articles. This ensures that the expert's updated knowledge is directly relevant to the evaluation questions, making this a best-case scenario for evaluating whether proxy-tuning can transfer any signal at all. The anti-expert $M^-$ is the original LLAMA2-7B.

The task: REALTIMEQA (Kasai et al., 2023), a dataset of questions about recent events updated periodically from news sources. At download time, the dataset contained 3,531 examples spanning June 2022 to December 2023. GPT-3.5's training cutoff is September 2021, creating a genuine temporal knowledge gap. Each question has four answer choices (A, B, C, D). The model's prediction is the highest-probability token among these four options. Questions for which all answer choices are missing are excluded (1.8% of the dataset). Performance is measured by accuracy: whether the highest-probability answer option matches the ground truth.

Operational detail: Because only the four answer tokens matter, the proxy-tuning offset is applied only to those four token positions — all other tokens are irrelevant and can be ignored. The paper explicitly states that proxy-tuning "only reweighs the four tokens of interest" (Section 7). This reduces the logit arithmetic to a tiny subspace, making it computationally trivial even with API latency overhead.

This experiment tests the weak-to-strong generalization property (Burns et al., 2023) of proxy-tuning: "the expert and anti-expert are both weaker than GPT-3.5" (Table 7 shows LLAMA2-7B at 28.4% base, 37.2% tuned, while GPT-3.5 is at 54.2% base), yet "contrasting their predictions yields a positive signal for the base model." This is a critical validation because it demonstrates that proxy-tuning does not require the small models to be competent on the task — only that their tuning captures a direction of improvement, however imperfectly.

The Optional $\alpha$ Hyperparameter (Section 6.2)

The main experiments use the DEXPERTS equation without modification (coefficient 1 on the offset). Section 6.2 explores what happens when a scalar multiplier $\alpha$ is introduced:

$s_M + \alpha \cdot (s_{M^+} - s_{M^-})$

where $\alpha \in \mathbb{R}^+$ controls the strength of the steering signal. When $\alpha = 0$ , the equation reduces to the base model alone (no tuning effect). When $\alpha = 1$ , it is the standard proxy-tuning equation. When $\alpha > 1$ , the tuning offset is amplified beyond its natural magnitude. When $\alpha < 1$ , the offset is attenuated.

What this enables: The paper frames $\alpha$ as a runtime control knob that allows users to trade off between competing desiderata without retraining any model. In the TruthfulQA analysis (Figure 2), varying $\alpha$ from 0.2 to 2.0 reveals that:

Truthfulness increases monotonically with $\alpha$ : stronger steering makes the model more resistant to misleading questions, likely because instruction-tuning increases the model's commitment to factual accuracy.
Informativeness peaks at $\alpha = 0.4$ and declines thereafter: too much steering induces the model to decline to answer or hedge excessively, reducing the useful information content of responses even as truthfulness improves.

The tradeoff is smooth and continuous, meaning a practitioner can select $\alpha$ to match their application's priorities — a fact-checking system might use high $\alpha$ for maximum truthfulness, while a creative assistant might use moderate $\alpha$ to maintain informativeness. The paper does not use tuned $\alpha$ values in the main experiments, keeping $\alpha = 1$ throughout "for simplicity" — this could be viewed as a conservative choice that slightly underestimates proxy-tuning's potential, since task-specific $\alpha$ tuning (or even $\alpha$ adaptation per prompt based on some estimate of required steering strength) might yield further improvements.

Runtime and Efficiency (Appendix C)

Proxy-tuning requires running forward passes through three models at each decoding step instead of one. The paper quantifies the wall-clock overhead in Appendix C.1.

Measured slowdown (Table 12): Across three generation scenarios varying prompt and generation lengths, proxy-tuning increases per-generation runtime by approximately 2.4× at 13B scale and 1.5× at 70B scale compared to direct generation from the corresponding tuned (CHAT) model. The larger relative slowdown at 13B occurs because the 7B proxy models (which run on the same GPU) represent a larger fraction of total compute at smaller base-model sizes. The absolute numbers: at 70B with an 8-token prompt generating 512 tokens, the tuned model takes 55.73 seconds per generation on 5 A100 GPUs while the proxy-tuned model takes 88.17 seconds.

The bottleneck and its mitigation: The paper identifies that the slowdown is "mostly due to a sequential execution of the models in proxy-tuning" — the forward passes through $M^-$ , $M^+$ , and $M$ are run one after another on the same hardware. This is a implementation artifact, not a fundamental limitation. The paper describes a straightforward optimization: "proxy-tuning can be greatly accelerated by deploying on multiple GPUs in parallel that communicate with each other (e.g., through an allreduce operation)." In this parallel setup, each model runs simultaneously on its own GPU, the logit vectors are gathered via GPU communication, combined, and the sampled token is distributed back to each device for the next step. The paper reports that a "pilot implementation shows a similar runtime compared to a true tuned model (though using three GPUs instead of one)" — meaning the overhead can be reduced to near-zero at the cost of additional hardware. This is a standard model-parallelism pattern and is not technically challenging, though it requires engineering effort not provided in the open-source repository.

For the user who does not want to parallelize: The paper notes a potential optimization stemming from the observation that proxy-tuning changes the base model's top-token prediction most heavily in the first few tokens of generation (Section C.2, Figure 3). The fraction of predictions changed declines sharply with position — from roughly 30% at position 1 to roughly 5% by position 100 for AlpacaFarm, with the same pattern observed across datasets. This suggests that proxy-tuning could be applied only at the beginning of generation and then "faded out," reducing total computation. However, the paper also notes that "the simple approach of only applying proxy-tuning to the first few tokens has limited effectiveness, due to the base model’s tendency to return to endless repetition when unchecked," indicating that some form of periodic re-application or decay schedule would be needed rather than a hard cutoff.

Training efficiency comparison (Appendix D.2, Table 14): The paper compares the training cost of creating the 7B expert (full fine-tuning) against applying LoRA to the 13B or 70B model directly. All training is done on 4 A100s for fair comparison. Full fine-tuning a 7B expert takes 30 hours 11 minutes for TriviaQA and 2 hours 35 minutes for GSM. Applying LoRA to 13B takes 33 hours 55 minutes (TriviaQA, 1.12× slowdown) and 3 hours 49 minutes (GSM, 1.48× slowdown). Applying LoRA to 70B takes 459 hours 6 minutes (TriviaQA, 15.2× slowdown) and 39 hours 20 minutes (GSM, 15.2× slowdown). Thus, creating a 7B expert via full fine-tuning is substantially faster than applying LoRA to a 70B model — 15× faster for both tasks — while the comparison with 13B LoRA is roughly comparable. This means that even in white-box settings where LoRA is an option, proxy-tuning can be more compute-efficient when the target model is large and a small expert can be tuned quickly.

4. Key Insights and Innovations

Innovation 1: Fine-Tuning as a Transferable Logit-Level Offset — Not a Weight Update

The paper's most fundamental conceptual move is reframing the result of fine-tuning — not as a new set of model weights — but as a token-level logit offset that can be extracted, packaged, and applied to an entirely different model through decoding-time arithmetic. Before proxy-tuning, the field's dominant assumption was that adapting a model's behavior required modifying its parameters, either through full fine-tuning (Raffel et al., 2020; Ouyang et al., 2022) or parameter-efficient methods like LoRA (Hu et al., 2022) that still ultimately update weights. The key constraint both approaches share is that they require white-box access — you must be able to read and write the model's parameters. This paper demonstrates that the behavioral transformation from tuning can be decoupled from the tuned model and applied as a pure signal to a different model, provided only that both share a vocabulary.

This is a fundamental shift in perspective, not an incremental efficiency improvement. The claim is not "we found a cheaper way to approximate fine-tuning" — it is "the effect of fine-tuning, as a mathematical object, is a vector in logit space that can be algebraically composed with other models' predictions." The paper provides the cleanest evidence for this in the instruction-tuning results (Table 2), where the logit offset extracted from a 7B parameter model transfers to both 13B and 70B models without any re-extraction or recalibration, closing 91% and 88% of the performance gap respectively. If the tuning signal were entangled with the specific parameters of the 7B model in a way that didn't generalize, this transfer would fail. The fact that it works — and sometimes exceeds the directly-tuned model's performance (Table 3, TruthfulQA truthfulness) — demonstrates that the behavioral shift is genuinely separable from the model that learned it.

The contrast with contemporary work sharpens this insight. Mitchell et al. (2024), which the paper explicitly cites, applied the same DEXPERTS equation for instruction-tuning but treated it primarily as an analytical tool for studying the separate effects of pretraining scale versus instruction-tuning. Proxy-tuning's contribution is to demonstrate that this equation is not merely analytically useful but practically effective as a tuning replacement across diverse objectives (instruction-following, domain adaptation, task-specific behavior) and at substantial scale disparities (7B → 70B). The reframing is from "here's an interesting property of logit arithmetic" to "fine-tuning produces a transferable behavioral offset — use it."

Innovation 2: The Alignment Tax Is a Weight-Update Artifact, Not an Inevitable Tradeoff

The paper provides compelling evidence that the well-documented "alignment tax" — the degradation of pretrained knowledge that often accompanies instruction-tuning or RLHF (Ouyang et al., 2022) — is specifically a consequence of updating model weights, not an unavoidable tradeoff between helpfulness and knowledge retention. By achieving instruction-following behavior through logit-level steering rather than weight updates, proxy-tuning demonstrates that the behavioral benefits of alignment can be largely obtained without the knowledge degradation that direct tuning imposes.

The key evidence is Table 3: on TruthfulQA's open-ended setting, the directly-tuned LLAMA2-70B-CHAT achieves 85.8% truthfulness, while proxy-tuning the same base model achieves 92.3% — a 6.5 percentage point improvement that pushes the proxy-tuned model above the directly-tuned version. The finer-grained breakdown shows that the proxy-tuned model is only 1.0% less informative (92.8% vs. 93.8%) but 6.5% more truthful (92.3% vs. 85.8%). This is a sharp result: the model becomes more truthful than its directly-tuned counterpart while being essentially equally informative. The mechanism is clear from the method design — the base model's pretrained weights are never modified, so factual knowledge acquired during pretraining cannot be overwritten by the tuning process. The tuning offset only shifts the model's expression of that knowledge (toward more truthful, less misleading responses), not the knowledge itself.

This finding changes the conceptual landscape for alignment research. The alignment tax has been treated as a fundamental tension — making models more helpful/safe sacrifices some of their knowledge (Ouyang et al., 2022 discuss this explicitly). Proxy-tuning suggests that this tension is, at least partially, an artifact of the mechanism used for alignment (weight updates) rather than an inherent property of alignment objectives. The practical implication is immediate: a model provider could offer both the raw pretrained model (for knowledge-intensive tasks) and a proxy-tuned variant (for instructed behavior) without incurring the cost of maintaining two separate trained models — the same base model serves both, with the behavioral overlay applied at decoding time. This is a conceptual reframing with direct deployment consequences.

Innovation 3: Weak-to-Strong Generalization Through Contrastive Steering

The GPT-3.5 case study (Section 7) demonstrates a property that the paper terms an instance of weak-to-strong generalization (Burns et al., 2023): the expert and anti-expert are both substantially weaker than the base model, yet their contrast provides a useful steering signal. This is not an incremental efficiency finding — it is a qualitatively distinct regime from the LLAMA2 experiments. In the LLAMA2 setup, the base model (untuned) and the expert (tuned) are at different scales but the expert has been specifically trained for the desired behavior. The expert is clearly more capable at the target task than the base model's untrained behavior. In the GPT-3.5 setup, the relationship is inverted: GPT-3.5 achieves 54.2% accuracy on REALTIMEQA while the tuned LLAMA2-7B expert achieves only 37.2%, and the untuned LLAMA2-7B anti-expert achieves 28.4% (Table 7). Both proxy models are worse than the base model at the task. Yet their contrast — the difference $s_{M^+} - s_{M^-}$ — captures the direction of temporal knowledge improvement, and applying that offset to GPT-3.5's logits yields a statistically significant 2.3% absolute improvement (to 56.5%).

This is a conceptually important result because it separates the notion of capability from the notion of tuning direction. The small models are not capable of answering REALTIMEQA questions well, but they encode a signal about which answers are more likely to be correct — specifically, the expert trained on recent data assigns higher probability to the correct answer than the untrained anti-expert does, even though both assign lower overall accuracy than GPT-3.5. The contrast extracts this directional signal and applies it to a more capable base model that already has high accuracy but lacks temporal awareness. This is not merely an instance of weak-to-strong generalization — it demonstrates a specific mechanism for how weak-to-strong generalization can be operationalized through contrastive logit arithmetic, where the contrast between a tuned and untuned weak model defines a direction in output space that the strong model can follow.

The practical implication is significant: it means that proxy-tuning can work even when no competent small expert exists for the target task. A user who wants to adapt GPT-4 for a specialized domain could tune a much weaker open-source model on domain data, extract the tuning offset, and apply it to GPT-4 — the small model need not be good at the task, only that its tuning captures a meaningful direction of improvement. This dramatically expands the applicability of proxy-tuning beyond settings where high-quality small experts are available off-the-shelf.

Innovation 4: Structural Format Transfer — Promoting Near-Zero-Probability Tokens

The task-specific fine-tuning experiments in Section 5 reveal a non-obvious capability of logit-level steering: proxy-tuning can promote tokens from near-zero probability to dominant probability if the tuning offset is sufficiently strong and consistent. This matters because it addresses a natural skepticism about decoding-time methods — that they can only nudge probabilities within the base model's existing support, and cannot introduce entirely new behavioral patterns that the base model never exhibits.

The GSM results provide the clearest evidence. The LLAMA2 base models have never seen the <<...>> angle-bracket equation notation or the #### final-answer delimiter during pretraining — they are artifacts of the GSM dataset's particular formatting. A base model spontaneously producing these tokens would be astronomically unlikely. Yet the paper reports that "99.7%+ of generations from proxy-tuned models (at both 13B and 70B) state the final answer after ####" (Section 5.2). The qualitative examples in Appendix E (Table 18) confirm this: the proxy-tuned 13B model generates properly formatted step-by-step solutions with the exact <<16-3=13>> notation, despite this format being completely absent from the base model's pretraining distribution.

The mechanism is instructive: the tuning offset $s_{M^+} - s_{M^-}$ is not bounded by the base model's original probability distribution. If the small expert assigns high logit mass to #### (because it learned the format during fine-tuning) and the small anti-expert assigns very low logit mass (because it never saw this format either), then $s_{M^+} - s_{M^-}$ can be a large positive number for the #### token. Adding this to the base model's logits can push #### from negligible probability to near-certainty, even if the base model itself would never generate it. The key requirement is that the small tuned expert must assign sufficiently high logit mass to the target token to overcome whatever low mass the base model assigns. This is a stronger claim than simple distributional nudging — it is a demonstration that decoding-time logit offsets can override the base model's priors when the tuning signal is strong, enabling the transfer of learned structural constraints (output formats, delimiters, syntactic patterns) without modifying weights.

This finding distinguishes proxy-tuning from prompt-based approaches to format control. A prompt can request a specific output format, but it cannot guarantee adherence, especially for formats the model has never seen. Proxy-tuning with a format-trained expert enforces the format by directly boosting the relevant tokens' probabilities at every decoding step where they are appropriate, with the anti-expert subtraction preventing the boost from becoming an indiscriminate bias toward those tokens. This is a qualitatively different capability that makes proxy-tuning suitable for tasks with strict structural requirements.

5. Experimental Analysis

Evaluation Methodology

Dataset. The instruction-tuning experiments (Section 3) evaluate on four benchmarks: GSM (1,319 test examples of arithmetic word problems; Cobbe et al., 2021), AlpacaFarm (805 test examples of open-ended instructions; Dubois et al., 2023), ToxiGen (2,800 sampled examples across 14 demographic groups for toxicity evaluation; Hartvigsen et al., 2022), and TruthfulQA (817 test examples of misleading questions, evaluated in both open-ended and multiple-choice settings; Lin et al., 2022). The code adaptation experiments (Section 4) use CodexEval (164 problems; Chen et al., 2021) and a 200-problem random sample of DS-1000 (Lai et al., 2022), where the sample is chosen to maintain representative performance while reducing evaluation cost. Task-specific finetuning experiments (Section 5) train on the full TriviaQA training set (87,622 examples; Joshi et al., 2017) and the full GSM training set (7,473 examples), evaluating on the TriviaQA development set (11,313 examples) and the GSM test set (1,319 examples) respectively. The GPT-3.5 case study (Section 7) evaluates on REALTIMEQA (3,531 examples from June 2022 to December 2023; Kasai et al., 2023).
Base model(s). All primary experiments use the LLAMA2 model family (Touvron et al., 2023), which includes pretrained BASE models and instruction-tuned CHAT models at 7B, 13B, and 70B parameter scales. The base model $M$ being steered is always an untuned LLAMA2-BASE variant at either 13B or 70B parameters, while the expert $M^+$ and anti-expert $M^-$ are based on LLAMA2-7B (or CODELLAMA-7B-PYTHON for code adaptation, initialized from LLAMA2-7B). The GPT-3.5 case study uses gpt-3.5-turbo-0613 with a reported training cutoff of September 2021. The authors argue LLAMA2 is "representative of the capabilities of many contemporary LLMs" and sits in a useful regime where the base model is capable but substantially lags behind its instruction-tuned counterpart, leaving room for proxy-tuning to make a measurable difference.
Metrics. The paper uses benchmark-specific metrics: for AlpacaFarm, GPT-4-evaluated win rate against text-davinci-003 reference responses; for GSM, accuracy based on extracting the last number in the model's response and checking exact match against the ground truth; for ToxiGen, percentage of model continuations scored as toxic by a RoBERTa-large toxicity classifier; for TruthfulQA open-ended, the percentage of responses judged as both truthful and informative by two GPT-3-based classifiers (with finer-grained breakdowns into % Informative and % Truthful reported separately in Table 3); for TruthfulQA multiple-choice, accuracy of parsing the first character after "The answer is:" as the predicted option A/B/C/D; for code benchmarks, pass@10 estimated from 20 samples using the unbiased estimator from Chen et al. (2021); for TriviaQA, exact match accuracy against reference answers and their aliases; and for REALTIMEQA, accuracy of selecting the highest-probability token among answer options A/B/C/D.
Baselines. The paper's primary baselines are (1) the untuned LLAMA2-BASE model at each scale — representing the floor that proxy-tuning aims to improve upon — and (2) the directly-tuned model at the same scale (LLAMA2-CHAT for instruction-tuning, CODELLAMA-PYTHON for code, task-specifically fine-tuned LLAMA2 for TriviaQA and GSM) — representing the ceiling that proxy-tuning aims to approach. For instruction-tuning evaluations on TruthfulQA, the paper also compares against the 7B-CHAT expert alone to assess whether proxy-tuning a larger model exceeds the small expert's performance. The task-specific fine-tuning section (Appendix D, Table 15) adds a LoRA baseline (Hu et al., 2022) for comparison in white-box settings, using the same hyperparameters as Tülu 2's QLoRA configuration (Table 13): LoRA rank 64, alpha 16, dropout 0.1, learning rate $1 \times 10^{-4}$ . For the GPT-3.5 case study, the baseline is vanilla GPT-3.5 without proxy-tuning and the small LLAMA2-7B expert alone.
Generation budget / compute accounting. All instruction-tuning and task-specific experiments use greedy decoding (temperature = 0), eliminating any sampling variance and making the generation cost exactly one forward pass per token per model. For code adaptation, pass@10 is estimated using 20 samples per problem with temperature = 0.8 and top-p = 0.95. The computational cost of proxy-tuning is three forward passes per token (base $M$ , expert $M^+$ , anti-expert $M^-$ ), compared to one for direct generation. The paper quantifies the wall-clock overhead in Appendix C.1: at 13B scale, proxy-tuning introduces a ~2.4× slowdown; at 70B scale, a ~1.5× slowdown. The paper notes that this overhead can be eliminated by running the three models in parallel on separate GPUs, with a pilot implementation achieving "similar runtime compared to a true tuned model (though using three GPUs instead of one)." Training costs are compared in Appendix D.2: full fine-tuning the 7B expert (for proxy-tuning) takes 30h11m for TriviaQA and 2h35m for GSM on 4 A100s, compared to 459h6m and 39h20m respectively for LoRA on the 70B model — a 15× training speed advantage for proxy-tuning.
Cross-validation / statistical protocol. The paper does not employ cross-validation, hyperparameter sweeps, or statistical significance testing in the main instruction-tuning, code adaptation, or task-specific experiments — performance is reported as single-run results on each benchmark's fixed test set with greedy decoding, eliminating variance from sampling. The GPT-3.5 case study (Section 7) reports a t-test with $p < 0.0001$ for the 2.3% improvement over vanilla GPT-3.5, confirming statistical significance. The $\alpha$ hyperparameter analysis (Section 6.2) sweeps $\alpha \in [0.2, 2.0]$ in increments of 0.2 and presents a continuous tradeoff curve (Figure 2) rather than isolated points. The token-level influence analysis (Section 6.1) reports a t-test with $p < 0.0001$ for the difference between left-hand-side ( $\Delta_t = 0.131$ ) and right-hand-side ( $\Delta_t = 0.056$ ) token probability shifts on GSM.

Main Quantitative Results

Instruction-Tuning Results (Section 3)

The headline result is that proxy-tuning an untuned LLAMA2-70B model closes 88.1% of the performance gap between the base model and its directly-tuned LLAMA2-70B-CHAT version across five evaluation settings (Table 2). At 13B scale, the gap closure is 91.1%. These percentages are obtained by computing, for each metric, $(\text{proxy-tuned} - \text{base}) / (\text{CHAT} - \text{base})$ and averaging across AlpacaFarm, GSM, ToxiGen, TruthfulQA MC, and TruthfulQA % Info + True.

Per-benchmark results at 70B scale (Table 2, rightmost columns). For AlpacaFarm, the base 70B model achieves a win rate of only 3.7%, the directly-tuned CHAT version achieves 90.4%, and proxy-tuning reaches 88.0% — just 2.4 percentage points short of the directly-tuned model. For GSM, the base model achieves 9.6% accuracy, CHAT achieves 51.8%, and proxy-tuning reaches 32.0% — a 22.4 percentage point absolute improvement over the base model, though substantially below the directly-tuned model. For ToxiGen, the base model produces toxic continuations 67.4% of the time, CHAT reduces this to 0.0%, and proxy-tuning also achieves 0.0% — a perfect match to directly-tuned behavior on safety. For TruthfulQA multiple-choice, the base model scores 42.3%, CHAT scores 68.3%, and proxy-tuning reaches 59.2%. For TruthfulQA open-ended (% Info + True), the base model scores 53.9%, CHAT scores 79.6%, and proxy-tuning reaches 85.1% — the first instance where proxy-tuning exceeds the directly-tuned model, by 5.5 percentage points.

The TruthfulQA anomaly (Table 3). The paper breaks down the open-ended TruthfulQA result into finer-grained components. At 70B: the proxy-tuned model is 1.0% less informative than CHAT (92.8% vs. 93.8%) but 6.5% more truthful (92.3% vs. 85.8%). The same pattern holds at 13B: proxy-tuning is 1.6% less informative (91.4% vs. 93.0%) but 3.2% more truthful (90.5% vs. 87.3%). This establishes that the overall advantage on % Info + True is driven entirely by superior truthfulness — the proxy-tuned model is less likely to produce answers that propagate common misconceptions, even though it is slightly more likely to decline to answer entirely. The paper interprets this as evidence that "decoding-time algorithms may provide an avenue for better knowledge preservation" (Section 3.2).

Comparison against the small expert alone. Across all settings where the base model $M$ is larger than the 7B proxy models (i.e., at 13B and 70B), proxy-tuning outperforms the small 7B-CHAT expert in every scenario except a 0.1% difference on ToxiGen at 13B (where both achieve effectively 0% toxic). At 70B, proxy-tuning improves over the 7B-CHAT expert by 5.5% on AlpacaFarm, 9.0% on GSM, and 3.3% on TruthfulQA multiple-choice — demonstrating that proxy-tuning successfully "reap[s] the benefits of larger pretraining scale" on top of the behavioral guidance from the small tuned model (Section 3.2).

Qualitative examples (Table 1, Appendix E). The paper provides concrete examples showing the behavioral transformation. For GSM, the base 13B model answers "$16" to a multi-step arithmetic word problem, while the proxy-tuned model produces a full step-by-step reasoning chain with the correct answer. For ToxiGen, the base model completes a hateful statement with "disabled girls are frigid and/or cannot have children," while the proxy-tuned model refuses: "I cannot agree with or promote such harmful and discriminatory beliefs." These examples are the first instances in each respective dataset (not cherry-picked), as confirmed by the paper's explicit statement.

Code Adaptation Results (Section 4)

The code adaptation experiments test a qualitatively different scenario: the expert is created by further pretraining on domain-specific data (Python code) rather than by instruction-tuning for dialogue behavior. The headline finding is that proxy-tuning produces substantial absolute improvements over the untuned base model — 32.0% on CodexEval and 16.6% on DS-1000 at 13B — but does not outperform the small 7B-CODE expert alone (Table 4).

Results at 13B (Table 4, middle rows). The untuned 13B-BASE achieves 33.7% pass@10 on CodexEval. The 7B-CODE expert alone scores 68.9%. Proxy-tuning 13B-BASE reaches 65.7% — a 32.0 percentage point improvement over the base model, but 3.2 points below the small expert. The directly-tuned 13B-CODE model achieves 78.6%, meaning proxy-tuning closes approximately 71% of the gap between the base and the directly-tuned model of the same scale. On DS-1000, the untuned base scores 26.2%, the 7B-CODE expert scores 53.6%, proxy-tuning reaches 42.8% (a 16.6 point improvement over base, but 10.8 points below the small expert), and the directly-tuned 13B-CODE scores 56.9%.

Results at 70B (Table 4, bottom rows). The untuned 70B-BASE achieves a much stronger starting point — 62.0% on CodexEval and 43.9% on DS-1000 — because larger pretrained models have already absorbed significant code capabilities. Proxy-tuning improves the 70B model to 70.7% on CodexEval (+8.6 points) and 50.6% on DS-1000 (+6.7 points). The 7B-CODE expert scores 68.9% and 53.6% respectively, while the directly-tuned 70B-CODE scores 89.2% and 67.6%. The absolute improvements from proxy-tuning are smaller at 70B than at 13B because the base model is already much more capable, leaving less room for improvement from domain-specific steering.

Why proxy-tuning doesn't exceed the small expert in this setting. The paper provides a mechanistic explanation tied to the two interpretations of the proxy-tuning equation. Recall the rearrangement $s_{M^+} + (s_M - s_{M^-})$ , which reads as "the small tuned expert plus whatever the larger base model provides beyond the small untuned model." In the instruction-tuning setting, the contrast $s_M - s_{M^-}$ captures generic knowledge and reasoning ability that improves the expert. In the code setting, the paper hypothesizes that "generic pretraining at a larger scale is not helpful when the model has already been tuned for a particular domain" (Section 4.2). That is, $s_M - s_{M^-}$ reflects general language and reasoning capabilities that the code expert has already learned (or that are irrelevant to writing correct Python), so adding this contrast does not improve — and can slightly degrade — the expert's specialized code behavior. This is a context-dependent limitation: when the tuning objective is domain specialization rather than behavioral alignment, the complementarity between pretraining scale and tuning is weaker.

Task-Specific Finetuning Results (Section 5)

This setting tests whether proxy-tuning can transfer strict task-specific formats and answer patterns from a small fine-tuned model to a larger base model.

TriviaQA results (Table 5, left columns). At 13B: the untuned base achieves 36.8% exact-match accuracy, the 7B task expert achieves 55.8%, proxy-tuning 13B reaches 55.9% — essentially matching the 7B expert and closing 84.0% of the gap to the directly-fine-tuned 13B model (59.5%). At 70B: the untuned base achieves 45.2%, the 7B task expert achieves 55.8%, proxy-tuning 70B reaches 62.7% — exceeding the 7B expert by 6.9 points and closing 86.9% of the gap to the directly-fine-tuned 70B model (63.1%). The proxy-tuned 70B model nearly matches the 70B directly-tuned baseline (62.7% vs. 63.1%), demonstrating that for question-answering with short factual answers, proxy-tuning can be essentially a drop-in replacement for direct fine-tuning at the 70B scale.

GSM results (Table 5, right columns). At 13B: the untuned base achieves 6.6%, the 7B task expert achieves 40.6%, proxy-tuning 13B reaches 43.9% — exceeding the 7B expert and closing 84.0% of the gap to directly-fine-tuned 13B (51.0%). At 70B: the untuned base achieves 9.6%, the 7B task expert achieves 40.6%, proxy-tuning 70B reaches 53.9% — exceeding the 7B expert by 13.3 points and closing 86.9% of the gap to directly-fine-tuned 70B (67.9%). The proxy-tuned 70B model's 53.9% represents a 44.3 point absolute improvement over the 70B base model.

The aggregate gap closure. Across both tasks and both scales, proxy-tuning closes on average 84.0% of the performance gap at 13B and 86.9% at 70B. The paper notes that "the benefit of task adaptation does not decrease as the scale of the base model increases" — indeed, the gap closure at 70B is slightly higher than at 13B — and that "proxy-tuning a larger base model (70B compared to 13B) is beneficial across tasks" (Section 5.2). This is in contrast to the code adaptation setting, where larger scale provided diminishing returns, and indicates that task-specific fine-tuning (which changes both behavior and factual content) benefits more from the large model's knowledge complementarity than domain adaptation (which primarily changes technical skill).

Format transfer verification (Section 5.2). The paper explicitly quantifies the format-following capability: "99.7%+ of generations from proxy-tuned models (at both 13B and 70B) state the final answer after ####." The qualitative examples in Appendix E (Table 18) show the proxy-tuned 13B model producing a correctly formatted step-by-step GSM solution with the exact <<16-3=13>> notation, despite the base model never encountering this format during pretraining. The base model's answer to the same question is a bare "$16" — a dramatic illustration of how proxy-tuning transforms not just the content but the syntactic structure of generations.

GPT-3.5 Black-Box Case Study Results (Section 7)

The headline result (Table 7): Proxy-tuning improves GPT-3.5's accuracy on REALTIMEQA from 54.2% to 56.5%, an absolute improvement of 2.3 percentage points. This improvement is statistically significant with $p < 0.0001$ under a t-test. For context, the small 7B expert alone achieves 37.2% (up from the 7B base model's 28.4%), meaning GPT-3.5 without any proxy-tuning already substantially outperforms the tuned small model — yet the contrast signal extracted from the weak models still provides a useful directional correction.

The weak-to-strong dynamic. The paper explicitly frames this as an instance of weak-to-strong generalization (Burns et al., 2023): "the expert and anti-expert are both weaker than GPT-3.5" (Section 7), but "contrasting their predictions yields a positive signal for the base model." The 7B expert, though only 37.2% accurate overall, has been trained on articles relevant to the exact evaluation questions, so its logit shifts $s_{M^+} - s_{M^-}$ encode which answer options are more likely to be temporally correct — information that GPT-3.5's more capable but temporally outdated predictions lack. The 2.3% improvement, while modest in absolute terms, is striking given the extreme information constraints: only top-5 log probabilities, only four candidate tokens, and both proxy models being substantially less accurate than the base model they steer.

Limitations of this setting. The paper is transparent about the restricted scope: the GPT-3.5 API's limitation of not allowing conditioning on partial model responses "prevents us from applying proxy-tuning to any task involving generation of more than one token" (Section 7 footnote). This means the case study is confined to multiple-choice evaluation where a single-token prediction is sufficient, and cannot demonstrate proxy-tuning's effectiveness for open-ended generation on black-box APIs.

LoRA Comparison (Appendix D, Table 15)

The paper compares proxy-tuning against LoRA — the leading parameter-efficient fine-tuning method — in the white-box setting using the task-specific fine-tuning experiments. This is the only head-to-head comparison with an alternative tuning method.

TriviaQA. LoRA significantly outperforms both proxy-tuning and full fine-tuning at both scales. At 13B: LoRA achieves 66.0% vs. proxy-tuning's 55.9% and full fine-tuning's 59.5%. At 70B: LoRA achieves 75.3% vs. proxy-tuning's 62.7% and full fine-tuning's 63.1%. LoRA's advantage over full fine-tuning (by 6.5 points at 13B and 12.2 points at 70B) is an independent finding — the paper hypothesizes that "a smaller shift can be more easily captured through parameter-efficient finetuning" (Section D.1).

GSM. The comparison is mixed. At 13B: proxy-tuning achieves 43.9%, outperforming LoRA's 32.4% by 11.5 points (full fine-tuning is 51.0%). At 70B: LoRA achieves 63.0%, outperforming proxy-tuning's 53.9% by 9.1 points (full fine-tuning is 67.9%). The paper's interpretation is that "LoRA's inconsistency across the two tasks is due to the size of the shift between pretraining and finetuning data" — TriviaQA answers are short and stylistically close to the base model's natural predictions, while GSM answers are long formatted passages far from the base model's typical outputs. The larger the distribution shift, the more proxy-tuning (which applies a logit-level offset) may struggle relative to methods that modify internal representations.

Training efficiency context (Table 14). Even where LoRA outperforms proxy-tuning empirically, the computational cost of obtaining comparable results differs substantially. Full fine-tuning a 7B expert for proxy-tuning takes 30h11m for TriviaQA vs. 459h6m for 70B LoRA — a 15.2× speedup. For GSM: 2h35m vs. 39h20m, also 15.2×. This means that for practitioners with limited compute, proxy-tuning may be the only feasible option for customizing a 70B model, even in white-box settings where LoRA is technically possible.

Analysis Results: Token-Level Influence and the $\alpha$ Hyperparameter (Section 6)

Token-level influence (Section 6.1). The paper measures the shift in probability assigned to the top token chosen by the proxy-tuned model $\tilde{M}$ compared to the base model $M$ . For GSM, the average $\Delta_t$ is 0.131 for tokens on the left-hand side of intermediate equations (reasoning formulation) vs. 0.056 for tokens on the right-hand side (factual statements) — a statistically significant difference with $p < 0.0001$ . This suggests proxy-tuning contributes more to how reasoning is structured and expressed than to the factual content of the equations themselves.

For TruthfulQA, the paper identifies the 10 vocabulary tokens whose probability increased the most from the base to the proxy-tuned model (Table 6). These tokens are overwhelmingly stylistic and rhetorical markers: "Here," "Additionally," "There," "While," "several," "It," "provide," "respect," "common," "personal." The most common 4-gram contexts reveal their communicative function: "There is no scientific," "is a common myth," "I cannot provide," "depending on several factors," "It's important to," "I don't have personal." These are tokens associated with pushing back on misleading questions, acknowledging nuance, and politely declining to generate harmful content — precisely the behavioral patterns instruction-tuning aims to instill. The paper interprets this as "consistent with the hypothesis that instruction-tuning mainly influences reasoning and style, rather than increasing the model's knowledge."

The $\alpha$ hyperparameter (Section 6.2, Figure 2). Varying $\alpha$ in the proxy-tuning equation $s_M + \alpha \cdot (s_{M^+} - s_{M^-})$ reveals a smooth, interpretable tradeoff on TruthfulQA. Truthfulness (the % of responses judged truthful) increases monotonically from approximately 82% at $\alpha = 0.2$ to approximately 93% at $\alpha = 2.0$ — stronger steering consistently improves factual reliability. Informativeness (the % of responses judged informative) peaks at approximately 95% around $\alpha = 0.4$ – $0.6$ and declines to roughly 88% at $\alpha = 2.0$ — excessive steering makes the model more likely to decline to answer or hedge, reducing useful output. The crossover point where both metrics are relatively high occurs at $\alpha \approx 0.6$ – $0.8$ . This tradeoff is (in the paper's words) "smooth," allowing practitioners to select $\alpha$ to match application needs without retraining.

Ablation Studies and Robustness Checks

Prompt format handling for chat models (Section 3, Appendix A.1, Table 8): The paper uses different prompt formats for the chat expert (wrapped in [INST] and [/INST] tokens, with optional system prompt) vs. the base and anti-expert models (plain text prompts). This is an implicit ablation of whether prompt formatting matters — the fact that proxy-tuning works robustly despite this asymmetry confirms that the logit-level signal from the chat expert, extracted under its expected prompt format, transfers meaningfully to the base model's distribution even though the base model never sees the special tokens. No ablation is run with a matched-prompt condition (e.g., giving the chat expert a plain-text prompt), so the contribution of prompt engineering to the overall transfer cannot be isolated, but the paper's framing implies that this asymmetric setup is the appropriate comparison since it respects each model's training distribution.

Most-influenced tokens analysis (Section 6.1, Table 6): This analysis serves as a qualitative verification that proxy-tuning changes behavior through interpretable token-level shifts rather than through opaque distributional effects. The finding that top-influence tokens align with stylistic and reasoning patterns (not factual content) validates the method's mechanism: proxy-tuning applies a behavioral offset rather than injecting knowledge. No quantitative ablation is performed to confirm this interpretation, but the token-level evidence is consistent with the aggregate TruthfulQA results (Table 3) showing superior truthfulness with slightly reduced informativeness — a pattern that would emerge if proxy-tuning promotes cautious, nuance-acknowledging language.

Code adaptation with different expert configurations: The paper uses only one expert configuration for code — CODELLAMA-7B-PYTHON — and does not ablate over different code models, different amounts of continued pretraining, or different anti-expert choices (e.g., using CODELLAMA-7B without the Python specialization as the anti-expert instead of LLAMA2-7B). The result that proxy-tuning a larger model does not improve over the 7B expert (Table 4) is therefore tied to this specific expert configuration and may not generalize to all code adaptation scenarios.

Multiple-choice vs. open-ended evaluation for TruthfulQA (Tables 2 and 3): The paper evaluates TruthfulQA in both multiple-choice (MC) and open-ended settings. The MC setting tests whether the model can identify truthful answers among options; the open-ended setting tests whether the model can generate truthful answers. At 70B, proxy-tuning achieves 59.2% MC accuracy vs. 68.3% for CHAT, but 85.1% open-ended % Info + True vs. 79.6% for CHAT. This reversal — proxy-tuning underperforming CHAT in MC but outperforming in open-ended — is not explicitly discussed by the paper, but it aligns with the interpretation that proxy-tuning improves truthfulness of generated content while CHAT is better optimized for accurate answer selection. This is an implicit ablation showing that proxy-tuning and direct tuning optimize for different aspects of truthful behavior.

Negative result: ReSTEM-style iteration was not attempted. The paper does not explore whether proxy-tuning can be applied iteratively — that is, using the proxy-tuned model's outputs as training data for a better expert, or applying proxy-tuning on top of a model that has already been proxy-tuned. This is a natural extension but is left to future work.

Negative result: Code adaptation scaling does not follow the same pattern as instruction-tuning. The finding that proxy-tuning a larger base model does not outperform the 7B code expert (Table 4) — in contrast to instruction-tuning where larger base models consistently outperform the 7B expert — reveals that the complementarity between pretraining scale and tuning is context-dependent. The paper does not ablate over different code-specific experts to determine whether this is a general property of domain adaptation or specific to the CODELLAMA-PYTHON expert's training recipe. This limits the generalizability of the code adaptation results.

Critical Assessment

Claim 1: Proxy-tuning closes ~88% of the gap between untuned and directly-tuned LLAMA2-70B.

What was actually tested: The 88.1% figure (Table 2) is an average across five evaluation settings: AlpacaFarm (GPT-4 win rate), GSM (math accuracy), ToxiGen (toxicity %), TruthfulQA MC (multiple-choice accuracy), and TruthfulQA % Info + True. These are then further averaged across only two model scales: 13B and 70B. The claim is therefore an estimate across a specific set of benchmarks whose selection materially affects the number.

Where it holds and where it doesn't. The gap closure varies enormously across individual benchmarks. It is near-perfect for ToxiGen (100% closure at both scales — proxy-tuning matches CHAT's 0% toxicity), strong for AlpacaFarm (97.2% closure at 13B, 97.3% at 70B), moderate for TruthfulQA MC (76.5% at 13B, 65.0% at 70B), and substantially lower for GSM (62.4% at 13B, 53.1% at 70B). On TruthfulQA % Info + True, proxy-tuning exceeds CHAT (producing "closure" exceeding 100%), which mathematically inflates the average. A simple arithmetic mean across these heterogeneous metrics with no weighting or normalization means the 88% figure is sensitive to which benchmarks are included and their relative variance. The omission of GSM's weak closure from the headline is notable — on the benchmark most directly measuring reasoning capability, proxy-tuning reconstructs only about half of the CHAT model's performance at 70B scale.

The benchmark set is not neutral. The benchmarks were selected from the Tülu evaluation suite (Wang et al., 2023; Ivison et al., 2023), filtered to those with "a reliable rule for extracting the model-predicted answer" (Appendix A). This excludes tasks requiring more complex evaluation or open-ended generation without extractable answers, biasing the benchmark set toward tasks where proxy-tuning's token-level steering may be most effective. Tasks requiring nuanced, multi-paragraph reasoning or creative generation (where the target distribution is harder to characterize as a token-level offset) are absent.

The comparison is against LLAMA2-CHAT, not a suite of instruction-tuned models. The paper tests proxy-tuning against exactly one directly-tuned model: LLAMA2-CHAT. No other instruction-tuned variants (e.g., Vicuna, Alpaca, Tülu, or models tuned with different RLHF recipes) are tested. This means the 88% figure measures replication of one specific tuning recipe, not the class of all possible tuning outcomes. A different instruction-tuning procedure might yield a gap that is harder or easier for proxy-tuning to close.

Claim 2: Proxy-tuning enables black-box model customization using only output logits.

What was actually tested: The paper demonstrates black-box customization in two settings. First, the LLAMA2 experiments — but LLAMA2-BASE is open-weight, so while the paper treats it as a black box (not accessing weights), the model is not actually black-box in deployment terms. The experiment proves feasibility under the assumed constraint of weight inaccessibility but does not test on a model that is actually weight-inaccessible to the experimenters. Second, the GPT-3.5 case study (Section 7) — this genuinely tests a black-box API model with limited logit access, confirming the feasibility claim in principle.

The GPT-3.5 result is narrow in scope. The case study is restricted to multiple-choice with four candidate tokens and single-token prediction. It demonstrates a statistically significant +2.3% accuracy improvement, but this is a much weaker result than the LLAMA2 experiments (where proxy-tuning produces dramatic qualitative transformations). The paper is transparent about the API limitation preventing multi-token generation, but this means the claim "proxy-tuning works for black-box models" has only been validated in the most constrained setting — single-token, multiple-choice — not for open-ended generation, instruction-following, or any of the other settings that constitute the bulk of the paper's contributions.

The information requirement — full logits vs. top-5 — is not systematically ablated. The paper shows proxy-tuning works with full logits (LLAMA2) and top-5 logits (GPT-3.5 multiple-choice), but does not explore the intermediate regime: how many top logits are needed for open-ended generation? Would top-10, top-100, or top-1000 suffice? This is a critical practical question since most API providers impose logit access limits, but the paper provides no guidance on the minimal information requirement for effective open-ended proxy-tuning.

Claim 3: Proxy-tuning sometimes surpasses direct tuning by preserving pretrained knowledge (avoiding the alignment tax).

What was actually tested: The claim is supported by a single benchmark: TruthfulQA open-ended evaluation (Table 3), where proxy-tuning achieves higher truthfulness than directly-tuned CHAT at both 13B (+3.2%) and 70B (+6.5%). The paper argues this demonstrates knowledge preservation — the base model's factual knowledge remains intact when weights are not updated.

But the claim is limited to one specific form of knowledge degradation. TruthfulQA measures susceptibility to common misconceptions — a specific kind of "knowledge" about factual accuracy and resistance to misleading framing. The paper does not test other forms of knowledge preservation that alignment tax research has identified as degraded by tuning, such as: world knowledge on general trivia benchmarks (TriviaQA is tested in a different context in Section 5, but not compared against proxy-tuned CHAT models), reasoning on held-out tasks, or performance on NLP benchmarks that probe linguistic knowledge. The claim that proxy-tuning generally avoids the alignment tax is broader than what the single-benchmark evidence supports.

There is a potential confound. The proxy-tuned model combines the base model's knowledge with the CHAT model's behavioral style. The CHAT model was trained with RLHF, which optimizes for helpful and harmless responses. It is possible that CHAT's lower truthfulness on TruthfulQA is not a degradation of pretrained knowledge per se, but an intended consequence of optimizing for harmless/helpful behavior — for example, being more deferential to misleading premises in questions. The proxy-tuned model's higher truthfulness might then reflect the base model's stronger prior on factual correctness rather than superior knowledge preservation — a subtly different claim. The paper does not explore this distinction.

Claim 4: Proxy-tuning generalizes across tuning objectives (instruction-following, domain adaptation, task finetuning).

What was actually tested: The paper tests three tuning objectives, each with one instance: instruction-tuning (LLAMA2-CHAT proxy), code adaptation (CODELLAMA-PYTHON proxy), and task-specific QA (TriviaQA and GSM). This demonstrates breadth across objective types, but each objective is tested with exactly one expert, one base model family, and (for instruction-tuning) one set of benchmarks.

Missing objective types. Several practically important tuning objectives are not tested: (1) safety alignment beyond toxicity (e.g., refusal of jailbreak prompts, handling of edge-case harms), (2) multilingual adaptation (e.g., steering an English-focused model toward a non-English language), (3) personalization or stylistic mimicry, (4) multi-task instruction-tuning where the expert was trained on hundreds of diverse tasks (the instruction-tuning expert, CHAT, was trained with RLHF, not diverse supervised instruction data). The claim that proxy-tuning "generalizes across tuning objectives" is therefore supported for three specific instantiations within a single model family — meaningful breadth, but far from comprehensive.

The choice of expert matters in ways the paper does not fully explore. The paper shows that for code adaptation, proxy-tuning a larger model does not outperform the 7B expert (Table 4), while for instruction-tuning (Table 2) and task finetuning (Table 5), it does. The paper attributes this to the nature of the tuning objective (domain specialization vs. behavioral alignment), but this is a post-hoc interpretation rather than an experimentally validated principle. A systematic study varying the type of tuning while controlling for other factors (expert size, base model size, domain shift magnitude) would be needed to establish when and why proxy-tuning benefits from larger base model scale.

Genuine Weaknesses

Single model family throughout. All LLAMA2-based experiments use variants of LLAMA2 within the same architecture family. While the GPT-3.5 experiment provides one cross-family test, it is limited to a single-token multiple-choice task. The paper provides no evidence that proxy-tuning works across model families (e.g., using a Mistral-based expert to steer LLAMA2, or a Pythia-based expert to steer OPT) for open-ended generation. The vocabulary compatibility constraint makes cross-family application non-trivial, and the paper's suggestion to use Kasai et al. (2022) for vocabulary alignment is not tested.

The 13B scale is systematically weaker than 70B for GSM proxy-tuning. At 13B, proxy-tuning achieves 43.9% on GSM while the directly-tuned 13B model achieves 51.0% — a 7.1 point gap. At 70B, the gap widens to 14.0 points (53.9% vs. 67.9%). This suggests that proxy-tuning's effectiveness relative to direct tuning may actually decrease with scale for reasoning-intensive tasks, even as absolute performance improves. The paper does not discuss this trend or its implications for applying proxy-tuning to models larger than 70B.

No baseline comparing proxy-tuning to simply using the small expert with a larger model's logits averaged in. A natural alternative to the DEXPERTS equation would be a simple weighted combination of log probabilities: $p = \text{softmax}(\beta \cdot s_M + (1-\beta) \cdot s_{M^+})$ for some $\beta \in [0,1]$ . This baseline would test whether the anti-expert subtraction is actually necessary for the behavioral transfer, or whether simply mixing a small tuned model's logits with a large base model's logits achieves comparable results. The absence of this ablation means the contribution of the anti-expert term — which is the distinctive element of the DEXPERTS formulation — is not empirically isolated.

No comparison against in-context learning with expert-generated demonstrations. A pragmatic alternative to proxy-tuning would be: generate a few demonstration examples from the small tuned expert, prepend them to the prompt as in-context examples, and query the large base model. This approach requires no logit access at all and works on any API. The paper acknowledges that long prompts have limitations (Section 8), but does not provide a quantitative comparison between proxy-tuning and few-shot prompting with expert-generated demonstrations. For settings where a few expert-generated examples are available, it is unclear whether proxy-tuning provides benefits beyond what in-context learning could achieve with lower complexity.

No confidence intervals or variance estimates. All instruction-tuning results use greedy decoding, meaning there is no sampling variance to report — but there is also no measure of uncertainty due to finite test set size. The 500-question MATH-style test sets (GSM: 1,319; AlpacaFarm: 805; TruthfulQA: 817) produce performance estimates with non-trivial confidence intervals. The paper reports no standard errors, bootstrap intervals, or any other measure of statistical reliability for the main instruction-tuning results. The 88.1% gap closure figure — computed as an average of ratios — could have substantial uncertainty that is not characterized.

The GPT-3.5 oracle expert is unrealistic. The temporal adaptation expert was trained on Google-retrieved articles specific to the evaluation questions — a deliberately strong oracle that maximizes the expert's relevance. A realistic deployment would train on general recent web data, which would likely produce a weaker tuning signal. The 2.3% improvement may therefore be an upper bound on what realistic temporal adaptation through proxy-tuning can achieve, but the paper does not explore this sensitivity.

6. Limitations and Trade-offs

The Base Model Must Already Be Capable of the Desired Behavior in Principle

The assumption or constraint. Proxy-tuning shifts the base model's output distribution via a logit offset, but if the base model assigns vanishingly small probability to tokens that the expert promotes, the offset may be insufficient to make those tokens the dominant prediction. The method cannot create new capabilities that the base model fundamentally lacks — it can only steer the model within the space of behaviors it can already express. The paper acknowledges implicitly through its experimental design that all target behaviors exist within the base model's output support, but never formalizes the boundary condition.

The consequence. On GSM — the benchmark most directly measuring novel multi-step reasoning — proxy-tuning at 70B achieves 53.9% vs. the directly-tuned model's 67.9% (Table 5, Section 5.2), a 14-percentage-point gap that represents the largest absolute shortfall across all benchmarks. This gap does not shrink with scale: at 13B, proxy-tuning achieves 43.9% vs. 51.0% for direct tuning, a 7.1-point gap, and at 70B the gap widens to 14.0 points. If this trend persists for even larger models (100B+, 400B+), the method may capture a decreasing fraction of the benefit that direct tuning provides for reasoning tasks — suggesting that some reasoning patterns acquired through fine-tuning cannot be fully replicated through logit-level steering. The paper provides no guidance on how to predict which behaviors will transfer successfully and which will remain out of reach.

What evidence exists in the paper. Table 5 (Section 5.2) provides the direct comparison. At 70B scale across four task-benchmark combinations, the gap between proxy-tuning and direct tuning varies dramatically: on TriviaQA the gap is only 0.4% (62.7% vs. 63.1%) while on GSM it is 14.0% (53.9% vs. 67.9%). The paper does not ablate over different levels of base model capability — for example, testing whether proxy-tuning's GSM gap closes when the base model is given chain-of-thought prompting or other interventions that improve its raw reasoning ability. The statement in Section 8 that prompt-based methods can be "surprisingly competitive with instruction-tuning" implies that combining proxy-tuning with prompting might address the reasoning gap, but this is never tested.

Mitigation status. Not addressed. The paper's results describe where proxy-tuning succeeds and where it falls short, but offer no mechanism for extending the method's reach to capability levels the base model does not naturally express. A deeper investigation of whether the logit space of a pretrained model contains latent reasoning patterns that proxy-tuning can surface — and if so, under what conditions — remains entirely as future work.

Difficulty Estimation and Strategy Selection Are Free in the Headline Comparisons

The assumption or constraint. The core proxy-tuning equation is parameter-free and requires no additional training beyond tuning the small expert model. However, this simplicity masks a selection decision the paper makes implicitly: which expert and anti-expert to use for a given task. In the instruction-tuning experiments, the selection is obvious — LLAMA2-7B-CHAT vs. LLAMA2-7B — because there exists an off-the-shelf pair that exactly corresponds to the desired behavioral transformation. In the code adaptation and task-specific experiments, the paper trains the expert specifically for the target task, requiring access to appropriate training data and the compute budget for fine-tuning. The headline "88% gap closure" does not account for the cost or feasibility of identifying or creating the right expert pair.

The consequence. For a practitioner facing a novel customization need — adapting a proprietary model for a specialized domain without off-the-shelf tuned variants — the full cost includes: (1) identifying an appropriate small pretrained model that shares a vocabulary with the base model, (2) obtaining or creating training data for the target behavior, (3) fine-tuning the small model (with all associated hyperparameter choices and compute costs), and (4) evaluating whether the resulting expert-anti-expert difference actually captures the desired behavioral shift. The paper's instruction-tuning results leverage the fact that LLAMA2-CHAT already exists as a high-quality, extensively-tuned expert — a starting point unavailable for most customization tasks. The code adaptation results use CODELLAMA-PYTHON, also off-the-shelf. Only the task-specific experiments (Section 5) involve training from scratch, and these required 30+ hours for TriviaQA on 4 A100s (Table 14). The 88% closure figure does not amortize this cost.

What evidence exists in the paper. The task-specific experiments (Section 5, Appendix A.3) provide the best evidence. The paper fully documents the expert training procedure — 2 epochs, learning rate 2e-5, batch size 128, BFloat16 precision — and reports training times (Table 14). But the experiments are designed to test whether proxy-tuning works given a properly trained expert, not whether the entire pipeline (including expert creation) is practically efficient relative to alternatives. The paper never compares the combined cost (expert training + proxy-tuning inference) against direct fine-tuning of the target model, even in the white-box LoRA comparison (Appendix D), where the cost of creating the 7B expert is reported separately from the cost of LoRA on the larger model but never combined with inference overhead.

Mitigation status. The paper partially addresses the expert-availability concern by emphasizing that "proxy-tuning allows users to leverage the rich collection of small tuned models available online, potentially composing them off-the-shelf with no additional training" (Section 8). This is valid for the instruction-tuning and (some) domain adaptation settings where off-the-shelf tuned models exist, but it is not a general solution. The paper does not propose any method for automatically identifying or creating effective expert-anti-expert pairs for arbitrary customization objectives, nor does it investigate how expert quality affects the resulting proxy-tuned model's performance — for example, whether a low-quality expert (trained on noisy or insufficient data) still provides a useful steering signal or actively degrades the base model's output.

The Method's Effectiveness Depends on the Type of Distribution Shift Introduced by Tuning

The assumption or constraint. Proxy-tuning represents the effect of tuning as a token-level logit offset s_{M^+} - s_{M^-}. This representation assumes that tuning changes the model's behavior in a way that can be captured by per-token probability shifts — that at each decoding step, the tuned model's behavioral divergence from the untuned model is well-approximated by an additive shift in logit space. When the tuning objective shifts the model's output distribution in ways that are not well-represented by token-level offsets — for example, when tuning teaches entirely new reasoning patterns that the base model would never produce, or when tuning restructures the model's internal representations rather than merely biasing its outputs — this approximation may break down.

The consequence. The paper shows that proxy-tuning works remarkably well for stylistic and behavioral shifts (instruction-following, toxicity reduction) and for format-enforcing task-specific fine-tuning, but substantially less well for domain adaptation where tuning teaches fundamentally new capabilities (code generation from a model not pretrained on code). At 13B on CodexEval, proxy-tuning reaches 65.7% pass@10 while the directly-tuned 13B code model achieves 78.6% — a 12.9-point gap (Table 4). At 70B on DS-1000, the gap is 17.0 points (50.6% vs. 67.6%). The paper attributes this to the nature of the tuning objective — domain specialization vs. behavioral alignment — but this is a post-hoc interpretation, not a predictive framework. A practitioner planning to use proxy-tuning for a novel domain cannot know in advance whether their target shift is "behavioral" or "capability-expanding" in a way that proxy-tuning can or cannot capture.

What evidence exists in the paper. The contrast between instruction-tuning results (Table 2, 88% gap closure) and code adaptation results (Table 4, significantly weaker improvement, never exceeding the small expert) provides the primary evidence. The paper explicitly discusses this discrepancy in Section 4.2: "We hypothesize that this is because generic pretraining at a larger scale is not helpful when the model has already been tuned for a particular domain." The LoRA comparison (Table 15, Appendix D) provides additional evidence: LoRA outperforms proxy-tuning substantially on TriviaQA (75.3% vs. 62.7% at 70B) but proxy-tuning outperforms LoRA on GSM at 13B (43.9% vs. 32.4%), demonstrating that the relative effectiveness of proxy-tuning vs. parameter-update methods depends on the task in ways the paper does not systematize. No experiment varies the type of tuning objective systematically while controlling for other factors — for example, comparing domain adaptation at different "distances" from the base model's pretraining distribution, or comparing instruction-tuning recipes that differ in how aggressively they restructure outputs.

Mitigation status. Not addressed beyond the hypothesis offered in Section 4.2. The paper does not propose any diagnostic for predicting whether logit-level offsetting will be effective for a given tuning objective, nor does it explore whether alternative formulations (different weighting of the offset, nonlinear transformations, state-dependent offsetting) could expand the range of tuning objectives that proxy-tuning can capture. The characterization remains empirical and descriptive rather than predictive.

Inference Cost Is Linear in the Number of Proxy Models, and the Paper's Parallelization Solution Requires Extra Hardware

The assumption or constraint. Proxy-tuning requires three forward passes per token (base, expert, anti-expert) compared to one for direct generation, fundamentally increasing the computational cost of inference. The paper measures this as a ~2.4× slowdown at 13B and ~1.5× at 70B relative to direct generation from the corresponding CHAT model (Table 12). The paper proposes parallelizing the three models across separate GPUs as a mitigation, noting that a "pilot implementation shows a similar runtime compared to a true tuned model (though using three GPUs instead of one)" (Appendix C.1). This framing presents the overhead as solvable, but at a meaningful hardware cost that the headline numbers do not reflect.

The consequence. For a deployment where the base model already requires significant GPU resources — a 70B model may need multiple GPUs just for a single forward pass — adding two additional 7B models and their KV-cache memory requirements (which must be maintained for the full context length) means the total GPU footprint is 3× the expert models plus the base model, not just the base model alone. If the base model already saturates available GPU memory, parallelization requires additional GPUs, increasing both capital cost and energy consumption. For latency-sensitive applications, even with parallelization, the communication overhead of gathering logits and distributing the sampled token across GPUs at every step introduces a floor on per-token latency that direct generation does not face. The paper's pilot implementation claim is not accompanied by detailed benchmarks, so the claim that runtime is "similar" is unverifiable from the paper's reported data.

What evidence exists in the paper. Table 12 provides wall-clock measurements in three generation scenarios, quantifying the sequential-execution slowdown. The paper acknowledges that the measured slowdown is "mostly due to a sequential execution of the models in proxy-tuning" and describes the parallelization approach in Appendix C.1. Figure 3 and the associated analysis (Section C.2) show that proxy-tuning changes the base model's top-token prediction most heavily in the first few tokens (30% at position 1, declining to ~5% by position 100), suggesting partial mitigation strategies. However, the paper also reports that "the simple approach of only applying proxy-tuning to the first few tokens has limited effectiveness, due to the base model's tendency to return to endless repetition when unchecked" (Appendix C.2). No rigorous characterization of the compute-accuracy tradeoff under sparse or truncated proxy-tuning is provided.

Mitigation status. Partially addressed. The parallelization proposal is technically sound and the paper reports a pilot implementation exists, but no measurements of parallelized runtime, GPU memory requirements, or communication overhead are provided. The suggestion to apply proxy-tuning only at selected time steps (based on Figure 3) is floated but tested only with a naive cutoff that the paper acknowledges fails. The token-level influence analysis (Section 6.1) showing that proxy-tuning's effect is concentrated in reasoning and stylistic tokens might inform a more sophisticated selective-application strategy, but this is not developed.

The Black-Box Claim Has Only Been Validated for Single-Token, Multiple-Choice Prediction with Strong Oracles

The assumption or constraint. The paper claims proxy-tuning enables customization of "truly black-box LMs" using "only its predictive distributions over the output vocabulary" (Section 1). The GPT-3.5 case study (Section 7) tests this claim under two strong constraints: (1) the task is multiple-choice with only four candidate tokens, meaning proxy-tuning need only reweight these four tokens rather than the full vocabulary; and (2) the API provides only top-5 log probabilities, restricting the offset computation to those five tokens. The paper is transparent about these constraints: the API "does not allow conditioning on partial model responses; it always generates the start of a new conversational turn. This prevents us from applying proxy-tuning to any task involving generation of more than one token" (Section 7, footnote).

The consequence. The black-box claim as validated is much narrower than the paper's other experimental results would suggest to a casual reader. The LLAMA2 experiments demonstrate proxy-tuning's effectiveness for open-ended generation, multi-step reasoning, and instruction-following — but all of these were conducted on open-weight models where full logit access was available. The GPT-3.5 experiment confirms that proxy-tuning works at all in a black-box API setting, but only for the most constrained possible task: single-token selection from four candidates. For the use case most practitioners would envision — adapting GPT-4 or Claude for custom instruction-following behavior, domain-specific generation, or structured output formats — the paper provides no evidence that proxy-tuning is feasible given current API constraints (limited logit access, inability to condition on partial responses). The gap between the demonstrated LLAMA2 capabilities and the validated black-box capabilities is enormous, and the paper does not explore intermediate regimes (e.g., what if the API provided top-100 logits? what if it allowed conditioning on model-generated tokens?).

What evidence exists in the paper. Table 7 and Section 7 provide the sole evidence for black-box proxy-tuning. The result — a 2.3% absolute improvement from 54.2% to 56.5% — is statistically significant (p < 0.0001) but modest. The paper acknowledges that "the API provides log probabilities for only the top 5 tokens" and that this prevents multi-token generation. Appendix C.2 provides fine-grained analysis of GPT-3.5's accuracy under different inference settings, but only for this constrained task.

Mitigation status. The paper explicitly flags the API limitation (Section 7 footnote) and frames the case study as a proof of concept rather than a complete solution: "we present a case study applying proxy-tuning to a truly black-box LM, GPT-3.5, in an extremely limited-information setting." The conclusion calls on "model-producing organizations to share output probabilities from their models to enable use of these methods" (Section 9). This is a policy recommendation, not a technical solution. No experiments explore how proxy-tuning performance degrades as logit access becomes coarser (top-5 vs. top-10 vs. top-100 vs. full logits), so the paper provides no guidance on what level of API access would be sufficient for open-ended generation. The oracle-like nature of the temporal adaptation expert (trained on Google-retrieved articles specific to the evaluation questions) further limits the realism of even the demonstrated result — a realistic expert trained on broad recent web data would likely produce a weaker tuning signal, potentially erasing the already-modest 2.3% gain.

Evaluation Is Confined to a Single Model Family and a Narrow Set of Benchmarks with No Statistical Characterization of Uncertainty

The assumption or constraint. All primary experiments use the LLAMA2 model family as both the base models and the proxy models. The benchmarks were selected from the Tülu evaluation suite (Wang et al., 2023; Ivison et al., 2023), filtered to tasks "with a reliable rule for extracting the model-predicted answer" (Appendix A). This filters out benchmarks requiring nuanced evaluation (e.g., open-ended dialogue quality, summarization faithfulness, creative generation) where proxy-tuning's token-level steering might behave differently. All instruction-tuning results use greedy decoding with no repeated sampling or confidence intervals, meaning the reported percentages are point estimates from single deterministic trajectories on test sets of 800–1,300 examples. The headline "88.1% gap closure" is an average of ratios computed across five heterogeneous metrics with no weighting, normalization, or uncertainty quantification.

The consequence. Three distinct generalizability concerns arise. First, model-family generalizability: The paper provides no evidence that proxy-tuning works across model families with different architectures, tokenizers, or pretraining distributions. Using a LLAMA2-based expert to steer a non-LLAMA2 base model — or vice versa — requires vocabulary alignment (the paper cites Kasai et al., 2022, but never implements it), and there may be distributional incompatibilities in logit space that prevent effective transfer. Second, benchmark generalizability: The excluded benchmarks — those without extractable answers — are precisely the ones where proxy-tuning's mechanism (logit-level offsets) might interact with evaluation in complex ways. For example, proxy-tuning might make outputs more formulaic (shifting toward tokens like "Here," "Additionally," "It's important to" as shown in Table 6), which could degrade performance on tasks requiring creative or stylistically diverse generation. Without testing on such benchmarks, it is unclear whether proxy-tuning's benefits are concentrated in extractable-answer tasks. Third, statistical reliability: The 88.1% gap closure figure is computed from a small number of benchmarks (5 metrics × 2 model scales = 10 ratios), and individual ratios have vastly different denominators (the "gap" for ToxiGen is 67.4 percentage points at 70B; for AlpacaFarm it is 86.7 points). A single outlier (e.g., TruthfulQA %Info+True where proxy-tuning exceeds CHAT, making the ratio >100%) skews the average, and no confidence interval characterizes how sensitive the 88.1% is to which benchmarks are included.

What evidence exists in the paper. All results tables report single-run numbers. The token-level analysis (Section 6.1) uses a t-test for the GSM left-hand-side vs. right-hand-side comparison — the paper's only use of statistical testing outside the GPT-3.5 case study — but the main performance tables lack any uncertainty quantification. The benchmark selection criteria are documented in Appendix A: the paper includes "all tasks with a reliable rule for extracting the model-predicted answer" from the Tülu suite. The excluded tasks are not named, so the reader cannot assess what fraction of the Tülu suite is covered or what task types are absent.

Mitigation status. Minimally addressed. The paper's claim that LLAMA2 is "representative of the capabilities of many contemporary LLMs" (Section 4) is an assertion, not an empirical finding. No cross-model-family experiment is attempted. The lack of statistical characterization is a methodological limitation the paper does not discuss. A reader attempting to estimate confidence intervals around the 88.1% figure — for example, through bootstrapping — would need access to the per-example predictions (not provided) and would face the additional complication that the "gap closure" metric is itself a ratio of differences, a quantity with potentially high variance when denominators are small.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual reframing of what fine-tuning produces. Rather than treating the result of tuning as a new set of model weights, proxy-tuning demonstrates that the behavioral transformation can be extracted as a portable, token-level logit offset — a vector in output space that can be transferred across model scales and, in principle, across model families. This is not an incremental efficiency improvement over parameter-efficient fine-tuning; it is a categorically different approach that severs the dependency between behavioral adaptation and weight access. The practical consequence is that the growing divide between model capability (concentrated in proprietary, black-box systems) and model customizability (which has historically required white-box access) is now narrower than previously assumed.

The finding most likely to shift research priorities is the evidence that the alignment tax is a weight-update artifact, not an inevitable tradeoff. Table 3 shows that proxy-tuning LLAMA2-70B achieves 92.3% truthfulness on TruthfulQA versus 85.8% for the directly-tuned CHAT model — a 6.5 percentage point improvement — while maintaining comparable informativeness. This single result reframes the alignment tax from a fundamental tension between helpfulness and knowledge retention into a mechanism-specific side effect of gradient-based optimization. If decoding-time steering can achieve instruction-following behavior without degrading pretrained knowledge, the field's default assumption that alignment requires weight modification deserves revisiting. This does not render RLHF or supervised fine-tuning obsolete — proxy-tuning still requires a tuned expert to extract the behavioral offset from — but it establishes that the deployment of aligned behavior need not involve the weight updates that cause knowledge degradation.

The paper also provides a unifying explanation for the mixed results in the logit-arithmetic literature. Prior work on contrastive decoding (Li et al., 2023), DEXPERTS (Liu et al., 2021), and related methods demonstrated that combining logit distributions from multiple models could improve specific generation attributes, but each study focused on a narrow objective (factuality, non-toxicity, open-ended text quality). Proxy-tuning shows that the same equation — s_M + (s_{M^+} - s_{M^-}) — works across instruction-following, domain adaptation, task-specific fine-tuning, and temporal adaptation without modification, unifying what appeared to be disparate use cases under a single mechanism. The key variable is not the equation but the choice of expert and anti-expert, which determines what behavioral dimension the offset captures. This transforms logit arithmetic from a collection of bespoke tricks into a general-purpose customization primitive, where the same decoding-time operation serves qualitatively different objectives depending on the tuning history of the proxy models.

The weak-to-strong generalization result in the GPT-3.5 case study (Section 7) establishes a second important principle: the small expert does not need to be competent at the target task for its tuning offset to be useful. The LLAMA2-7B expert achieves only 37.2% on REALTIMEQA while GPT-3.5 achieves 54.2% without proxy-tuning — yet the contrast between the weak tuned and untuned models provides a directionally correct signal that improves GPT-3.5 to 56.5%. This means proxy-tuning can operate in a regime where no competent small expert exists, extracting useful steering from models that are strictly worse than the base model they guide. This lowers the bar for applying proxy-tuning to proprietary models: a practitioner need only tune a small open-source model on target-domain data, even if that model's absolute performance is poor, as long as the direction of improvement it learns correlates with the desired behavioral shift in the larger model.

The research directions that become more attractive include: (1) developing cheap, general-purpose methods for extracting tuning offsets from small models and composing them (e.g., an "instruction-following offset" plus a "domain expertise offset" applied simultaneously to the same base model), since the paper shows offsets transfer across scales and objectives; (2) building infrastructure for serving proxy-tuned models efficiently, since the paper's pilot parallelization result (Appendix C.1) suggests the overhead can be eliminated with modest engineering but no production-grade implementation exists; and (3) studying the theoretical properties of logit-space interpolation between model behaviors, since the smooth α tradeoff curve in Figure 2 demonstrates that the behavioral offset can be continuously modulated without retraining. Directions that become less attractive include: developing ever-more-complex parameter-efficient fine-tuning methods that still require white-box access, since proxy-tuning provides a black-box alternative that is simpler to implement and often more training-efficient (Table 14 shows 15× faster expert creation than 70B LoRA); and efforts to mitigate the alignment tax through better optimization techniques, since the tax may be avoidable entirely by not updating weights.

Follow-Up Research This Work Enables

Cross-model-family proxy-tuning with vocabulary alignment. The paper demonstrates proxy-tuning entirely within the LLAMA2 family, where vocabulary compatibility is automatic. The GPT-3.5 case study extends to a different model but only in a single-token, multiple-choice setting. A critical next step is testing whether proxy-tuning works for open-ended generation when the base model and proxy models come from different families — for example, using a Mistral-7B-Instruct expert to steer LLAMA2-70B, or a LLAMA2-7B expert to steer a truly proprietary model like GPT-4. This requires solving the vocabulary mismatch problem: the three models' logit vectors must be projected into a shared token space. The paper cites Kasai et al. (2022) as a potential technical solution — their "twist decoding" performs token-level alignment across different tokenizers — but never implements it. A strong follow-up would implement vocabulary alignment, evaluate on the same instruction-tuning benchmarks used in Section 3 (AlpacaFarm, GSM, ToxiGen, TruthfulQA), and measure (a) the gap closure relative to within-family proxy-tuning, and (b) how alignment quality degrades as the source and target tokenizers diverge (e.g., BPE vs. SentencePiece vs. Unigram). A negative result — that cross-family proxy-tuning fails even with alignment — would establish a fundamental limitation; a positive result would dramatically expand the method's applicability given that most proprietary models use different tokenizers than open-source alternatives.

Difficulty-adaptive or token-position-dependent proxy-tuning to reduce inference cost. The paper quantifies a 1.5–2.4× inference slowdown (Table 12) and observes that proxy-tuning changes the base model's top-token prediction most heavily in the first few tokens of generation (Figure 3), but also notes that naively applying proxy-tuning only to early tokens fails due to the base model's tendency to drift into repetition. A natural follow-up is to develop an adaptive schedule for proxy-tuning that decides at each token position whether to apply the full three-model ensemble or fall back to the base model alone, based on some measure of how much the offset is contributing. The data in Figure 3 provides a starting point: the fraction of predictions changed declines from ~30% at position 1 to ~10% by position 50 for AlpacaFarm, with similar patterns across datasets. An adaptive policy could monitor the KL divergence between the base model's distribution and the proxy-tuned distribution, and skip proxy-tuning when the distributions are nearly identical (evaluated efficiently by checking whether the top-k tokens and their probabilities match within a threshold). The evaluation would measure the runtime-vs-performance tradeoff: how much speedup can be achieved (relative to both sequential and parallelized full proxy-tuning) while maintaining some target fraction of the full proxy-tuning performance gain (e.g., 95% of the gap closure). A negative result — that even sophisticated adaptive schedules fail to reduce cost without disproportionately hurting performance — would establish that the logit offset matters pervasively, not just at obvious decision points.

Composition of multiple proxy-tuning offsets for multi-objective customization. The paper treats proxy-tuning as applying a single offset derived from one expert–anti-expert pair, targeting one behavioral dimension at a time (instruction-following, code generation, task formatting). A natural extension is to test whether multiple offsets can be composed additively: s_M + α_1(s_{M^+_1} - s_{M^-_1}) + α_2(s_{M^+_2} - s_{M^-_2}) where each expert pair captures a different behavioral attribute. For example, using LLAMA2-7B-CHAT (instruction-following offset) plus CODELLAMA-7B-PYTHON (code offset) simultaneously to steer a base model toward being both instruction-following and code-fluent. The paper's token-level analysis (Section 6.1) showing that instruction-tuning primarily affects reasoning/style tokens while code adaptation affects domain-specific tokens suggests these offsets might operate in largely orthogonal subspaces of the vocabulary, making additive composition viable without destructive interference. A strong experiment would train three 7B experts — one for instruction-following, one for code, one for a third attribute (e.g., formal tone) — and evaluate all 2³ = 8 combinations on a benchmark suite that separately measures each attribute (AlpacaFarm for instruction-following, CodexEval for code, and a formality classifier for tone), looking for evidence of interference (e.g., code ability degrading when instruction-following offset is also applied) and measuring whether the α hyperparameters can be tuned to find Pareto-optimal tradeoffs. A negative result — that offsets interfere destructively even at moderate α values — would constrain the method to single-objective customization; a positive result would position proxy-tuning as a compositional framework where users mix and match behavioral modules at decoding time.

Systematic characterization of when proxy-tuning fails relative to direct tuning. The paper documents several cases where proxy-tuning substantially underperforms direct tuning: GSM at 70B (53.9% vs. 67.9%, a 14-point gap), and code adaptation where proxy-tuning never exceeds the 7B expert (Table 4). These are presented as empirical observations without a predictive framework. A systematic follow-up would design a controlled experiment that varies the nature of the distribution shift introduced by tuning while holding other factors constant. Specifically: take a single base model (LLAMA2-7B), create a series of "tuned" variants by fine-tuning on datasets that differ in how much they diverge from the base model's pretraining distribution — ranging from stylistic rephrasing (minimal shift, e.g., converting answers from casual to formal tone) through format-constrained generation (moderate shift, e.g., always output JSON) to fundamentally new capabilities (large shift, e.g., solving problems in a domain the base model has never seen). For each shift magnitude, measure proxy-tuning's gap closure against direct tuning of the same 7B model, then repeat at 13B and 70B to test whether the gap's dependence on shift magnitude changes with scale. The key measurement is the functional relationship between shift magnitude (operationalized as the KL divergence between base model outputs and tuned model outputs on a held-out task) and proxy-tuning effectiveness (measured as gap closure percentage). The paper's hypothesis — that proxy-tuning works best for "behavioral" shifts and poorly for "capability-expanding" shifts — would be confirmed if gap closure drops sharply at some identifiable KL threshold, providing practitioners with a diagnostic for predicting whether proxy-tuning is suitable for their use case. A negative result — no clear relationship between shift magnitude and proxy-tuning effectiveness, with task-specific factors dominating — would indicate that the method's applicability cannot be predicted from simple distributional statistics and would motivate more nuanced diagnostics.

Proxy-tuning for safety alignment beyond toxicity, including jailbreak robustness. The paper demonstrates proxy-tuning's effectiveness for toxicity reduction (ToxiGen: 67.4% → 0.0% toxic at 70B, Table 2) but does not evaluate on other safety-critical behaviors such as refusal of jailbreak attempts, handling of edge-case harms, or robustness to adversarial prompting. Instruction-tuned models like LLAMA2-CHAT undergo extensive red-teaming and safety-focused RLHF that goes far beyond toxicity suppression; whether proxy-tuning captures these deeper safety properties from a small expert is untested. A rigorous follow-up would evaluate proxy-tuned LLAMA2-70B (using LLAMA2-7B-CHAT as expert) on a suite of safety benchmarks: (a) jailbreak success rate using standard attack templates (e.g., role-playing, encoding tricks, multi-turn manipulation), (b) refusal rate on harmful but not obviously toxic requests (e.g., instructions for creating weapons, generating misinformation), and (c) over-refusal rate on benign requests that superficially resemble harmful ones (a known failure mode of heavily safety-tuned models). The comparison points would be: the untuned 70B base model, the directly-tuned 70B-CHAT, and the 7B-CHAT expert alone. The critical measurement is whether proxy-tuning's safety properties match CHAT's (good), exceed CHAT's while maintaining helpfulness (better, and consistent with the TruthfulQA finding that proxy-tuning can surpass direct tuning), or are weaker than CHAT's (revealing that some safety behaviors require deeper representational changes than logit-level offsets can provide). The paper's token-level analysis (Table 6) showing that proxy-tuning promotes refusal-adjacent language ("I cannot provide," "I don't have personal") suggests basic refusal behavior transfers, but sophisticated jailbreak resistance may not. A negative result — proxy-tuning failing to provide robust safety despite the 7B-CHAT expert having undergone safety training — would establish that safety alignment has components that are not captured by the logit offset and require weight-level intervention, which would be an important boundary condition for real-world deployment.

Practical Applications and Downstream Use Cases

On-demand customization of proprietary API models for enterprise workflows. Organizations that use GPT-4 or Claude through APIs often have domain-specific conventions, terminology, and output formats that off-the-shelf models do not respect. Currently, the primary options are prompt engineering (brittle, context-consuming) or waiting for the provider to offer fine-tuning (expensive, slow, and not available for all models). The GPT-3.5 case study (Section 7) demonstrates that proxy-tuning can improve a black-box model's behavior using only the logit access that commercial APIs already provide (top-5 log probabilities). For a concrete enterprise scenario: a legal tech company needs GPT-4 to generate contract clauses in a specific jurisdictional format with particular numbering conventions and terminology. By fine-tuning a small open-source model (e.g., LLAMA-8B or Mistral-7B) on their internal corpus of correctly formatted contracts, they can extract a formatting offset and apply it to GPT-4's generations — assuming the API provides sufficient logit access. The paper's task-specific results (Section 5) showing that proxy-tuning enforces strict formatting constraints (99.7%+ adherence to #### delimiters on GSM) suggest that structured output formats transfer reliably. The primary remaining bottleneck is API logit access: for open-ended generation, more than top-5 logits would likely be needed, and the paper's call for model providers to share output probabilities (Section 9) is directly relevant here. This use case is actionable today for multiple-choice or constrained-output tasks, and becomes actionable for open-ended generation as soon as API providers expose fuller logit distributions.

Cost-efficient domain adaptation for organizations that cannot afford large-scale fine-tuning. The paper's training efficiency comparison (Table 14) shows that creating a 7B expert via full fine-tuning is 15× faster than applying LoRA to a 70B model on the same hardware (30h vs. 459h for TriviaQA). This has direct implications for small research labs, startups, or academic groups that have access to a few GPUs but not the large clusters needed for 70B+ fine-tuning. A concrete scenario: a biomedical research group wants to adapt an LLM for question-answering over PubMed abstracts. They can fully fine-tune a 7B or 8B open-source model on their domain corpus (feasible on 1–4 consumer or academic GPUs), then use that model as an expert to proxy-tune a larger open-source model (e.g., LLAMA-70B) that they run for inference. The TriviaQA task-specific results (Table 5) provide a direct analog: proxy-tuning 70B with a 7B QA expert achieves 62.7% exact match, closing 86.9% of the gap to the directly-fine-tuned 70B model (63.1%), while requiring 15× less training compute. For the biomedical group, this means they can achieve near-state-of-the-art domain performance on a 70B model without ever fine-tuning the 70B model itself — a workflow that would be infeasible on their hardware. The inference overhead (1.5× slowdown at 70B; Table 12) is manageable for batch processing or non-latency-sensitive applications, and can be mitigated through the GPU parallelization approach described in Appendix C.1.

Preserving pretrained knowledge while achieving instruction-following behavior in high-stakes factual domains. The finding that proxy-tuning achieves higher truthfulness than direct instruction-tuning on TruthfulQA (Table 3: 92.3% vs. 85.8% at 70B) has immediate implications for deployment in domains where factual accuracy is paramount and the alignment tax is unacceptable. Consider a medical advice system or a scientific literature assistant: direct instruction-tuning (via SFT or RLHF) risks degrading the model's ability to recall specific facts from its pretraining corpus — the model becomes more "helpful" in style but may hallucinate or express uncertainty about facts it previously knew. Proxy-tuning offers a deployment architecture where the base model's weights are frozen (preserving all pretrained knowledge) and instruction-following behavior is applied as a decoding-time overlay. The system generates responses that are stylistically appropriate (follow instructions, are polite, refuse harmful requests) while drawing on the intact knowledge base of the pretrained model. The paper's TruthfulQA breakdown (Table 3) supports this: the proxy-tuned model is only 1% less informative than CHAT (92.8% vs. 93.8%) but 6.5% more truthful (92.3% vs. 85.8%), meaning it conveys essentially the same amount of information with fewer factual errors. For any application where incorrect factual claims carry high risk — medical, legal, financial, scientific — this tradeoff profile (slightly less informative but substantially more truthful) may be strongly preferable to the directly-tuned alternative. The implementation is straightforward given the paper's recipe: pair a 7B instruction-tuned expert with the large pretrained model, apply the DEXPERTS equation at decoding time, and optionally tune the α hyperparameter (Section 6.2, Figure 2) to balance informativeness and truthfulness for the specific domain.

When to Prefer This Method

The paper explicitly positions proxy-tuning against direct fine-tuning (both full and parameter-efficient) and identifies specific conditions where decoding-time steering is preferable. The decision rule is:

Prefer proxy-tuning over direct fine-tuning when the base model's weights are inaccessible (proprietary API models), or when preserving pretrained knowledge is critical and the alignment tax incurred by weight updates is unacceptable (evidence: Table 3, proxy-tuning achieves 6.5% higher truthfulness than direct tuning on TruthfulQA at 70B), or when training compute is severely limited and a small expert can be tuned much faster than even parameter-efficient fine-tuning of the large model would require (evidence: Table 14, 15× training speedup for 7B full fine-tuning vs. 70B LoRA), or when the tuning objective is primarily behavioral/stylistic rather than capability-expanding — specifically, when the expert-anti-expert contrast captures a shift in how the model expresses its knowledge (instruction-following, formatting, safety) rather than teaching fundamentally new skills the base model lacks (evidence: Tables 2 and 5 show strong gap closure for behavioral shifts; Table 4 shows weaker results for domain specialization where the base model was not pretrained on code).
Prefer direct fine-tuning (or LoRA) over proxy-tuning when the base model's weights are accessible and the tuning objective requires the model to acquire substantially new capabilities that it does not already possess — for example, when fine-tuning a general pretrained model on a highly specialized domain with vocabulary and reasoning patterns absent from pretraining (evidence: Table 4, CODELLAMA-PYTHON directly tuned at 13B achieves 78.6% pass@10 vs. proxy-tuning's 65.7%), or when inference latency is paramount and the hardware budget cannot accommodate the additional GPUs needed to parallelize proxy models (evidence: Table 12, 1.5–2.4× slowdown without parallelization), or when the target distribution shift is so large that even parameter-efficient methods like LoRA outperform proxy-tuning by wide margins (evidence: Table 15, LoRA achieves 75.3% vs. proxy-tuning's 62.7% on TriviaQA at 70B, though the relative performance depends on the task).