Black-Box On-Policy Distillation of Large Language Models

ArXiv: 2511.10643

🎯 Pitch

A 3B-parameter student model distilled from GPT-5 via adversarial training matches the performance of a standard-distilled 7B model, and the 14B student rivals the GPT-5 teacher itself—all without ever seeing the teacher's internal probabilities.

---

1. Executive Summary

This paper introduces Generative Adversarial Distillation (GAD), a black-box on-policy distillation framework that frames the student LLM as a generator and trains a discriminator to distinguish its responses from the teacher's, creating a minimax game where the discriminator provides adaptive, co-evolving feedback. Evaluated using GPT-5-Chat as teacher and Qwen2.5/Llama3 student models on the LMSYS-Chat benchmark, GAD consistently surpasses sequence-level knowledge distillation (SeqKD), with Qwen2.5-3B-Instruct trained via GAD matching the performance of Qwen2.5-7B-Instruct trained via SeqKD—a parameter-class jump—while Qwen2.5-14B-Instruct reaches GPT-4o scores comparable to the GPT-5 teacher itself (52.1 vs. 51.7 on LMSYS-Chat). The framework establishes that effective on-policy black-box distillation requires the discriminator to continually adapt to the student's evolving output distribution, preventing the reward hacking that collapses off-policy discriminators within ~300 training steps.

2. Context and Motivation

The Specific Problem: On-Policy Learning in the Black-Box Regime

The paper addresses a concrete, practical problem: how do you distill a proprietary, closed-source LLM (the teacher) into a smaller open-source model (the student) when you only have access to the teacher's text outputs, not its internal probability distribution? This is the black-box distillation setting, and it is the de facto reality when the teacher is a commercial API model like GPT-5, Claude, or Gemini—models where the provider returns generated text but exposes neither logits, nor hidden states, nor token-level probability vectors.

The specific gap the paper identifies is not black-box distillation per se, but rather the absence of on-policy learning within it. In white-box distillation, recent work (Gu et al., 2024; Agarwal et al., 2024; Lu, 2025) has demonstrated that the student benefits substantially from learning on its own generated responses rather than solely on teacher-forced outputs—a technique that promotes mode-seeking behavior and reduces exposure bias. The paper explicitly cites these studies:

"Recent studies in white-box distillation highlight the importance of on-policy learning, where the student learns from its own generated responses rather than solely imitating the teacher's outputs." (Section 1)

However, extending on-policy learning to the black-box setting encounters a fundamental obstacle. When the student generates responses during training, there is no teacher probability signal available to evaluate them. The teacher is a black box returning only text; it provides no score, no likelihood, no gradient. So while white-box on-policy methods can compute a forward or reverse KL divergence between the student's generated distribution and the teacher's token-level probabilities, the black-box student is blind—it generates text but has no way to know whether that text is "teacher-like." The paper states this clearly:

"Without explicit feedback, the student cannot directly gauge the quality of its generations relative to the teacher, making effective on-policy distillation infeasible under the standard likelihood-based framework." (Section 1)

GAD's core contribution is solving this feedback problem: it trains a discriminator to serve as a learned, adaptive quality signal that replaces the missing teacher probabilities, enabling on-policy learning in the black-box regime.

Why This Problem Matters

The importance of this problem spans both practical deployment concerns and deeper technical considerations.

Practical importance: the dominance of proprietary API models. The most capable LLMs today are closed-source commercial products accessed via API. Organizations that want to deploy smaller, cheaper, faster models for specific use cases (on-device, high-throughput, privacy-sensitive) cannot assume white-box access to the strongest teachers. The standard fallback—supervised fine-tuning on teacher-generated text (SeqKD; Kim & Rush, 2016)—is simple but demonstrably leaves performance on the table relative to what white-box methods can extract. GAD proposes a way to close that gap without requiring white-box access, which makes it directly applicable to the API-first ecosystem that most practitioners operate in.

Tokenizer incompatibility as a parallel constraint. Even when a teacher model is open-source, its tokenizer may be incompatible with the student's tokenizer (different vocabulary, different tokenization rules). This prevents direct alignment of token-level probability distributions, making standard white-box KL-divergence methods inapplicable. The paper demonstrates this scenario in Table 7, distilling from Qwen2.5-14B-Instruct into Llama student models: although the teacher is open-source, tokenizer mismatch forces black-box treatment. GAD circumvents this because the discriminator operates on text sequences, not token-level logits—it scores the concatenation [prompt, response] as a scalar, making it tokenizer-agnostic.

Theoretical significance: on-policy vs. off-policy distillation dynamics. Beyond the practical API constraint, the paper engages a deeper question about what makes distillation effective. The white-box literature had established that on-policy learning (reverse KL, mode-seeking) is superior to off-policy teacher-forcing (forward KL, mode-covering) for language model distillation. But the mechanism by which on-policy learning helps—and specifically, why an off-policy reward signal fails in the black-box setting—was not understood. GAD's experimental comparison with an off-policy discriminator (Figure 6) reveals a concrete failure mode: a fixed reward model trained on the warmup student's outputs rapidly becomes stale as the student's policy shifts, leading to reward hacking within ~300 training steps. This is not merely an implementation detail—it is a principled argument that black-box distillation requires co-adaptation between the student and the quality signal, and that the discriminator must function as an on-policy reward model that tracks the student's evolving behavior.

Prior Approaches and Where They Fall Short

The paper situates its contribution against two families of prior work.

White-box distillation methods assume access that doesn't exist in practice. Standard white-box approaches operate with full access to the teacher's predictive distribution $p(y|x)$. Forward KLD methods (Kim & Rush, 2016; Song et al., 2020; Chiang et al., 2023; Taori et al., 2023) minimize $\text{KL}(p \parallel q_\theta)$, which the paper notes exhibits mode-covering behavior—spreading probability mass across all outputs the teacher might produce, including low-quality ones. Reverse KLD methods (Gu et al., 2024) minimize $\text{KL}(q_\theta \parallel p)$, promoting mode-seeking where the student concentrates on the teacher's most likely behaviors. Hidden state and attention distillation (Jiao et al., 2020; Sun et al., 2019; Wang et al., 2020) align internal representations. All of these require $p(y|x)$ or internal activations—exactly what black-box teachers withhold. The paper does not dispute the effectiveness of these methods in the white-box setting; it simply observes that they are inapplicable when the teacher is an API model.

SeqKD: the dominant black-box baseline, and why it's insufficient. The standard approach to black-box distillation is SeqKD: perform supervised fine-tuning (SFT) on the teacher's responses to instruction prompts (Kim & Rush, 2016; Peng et al., 2023; Zhou et al., 2023; Taori et al., 2023; Chiang et al., 2023). The student is trained with standard cross-entropy loss to maximize the likelihood of the teacher's text. This is straightforward, widely used, and serves as the primary baseline throughout the paper.

The paper identifies two specific failure modes of SeqKD:

1. Mode-covering rather than mode-seeking. SeqKD is equivalent to minimizing forward KL divergence $D_{\text{KL}}(p \parallel q_\theta)$ when the teacher distribution is approximated by empirical samples. This encourages the student to cover all modes of the teacher's distribution, including low-probability regions that the student cannot credibly reproduce given its smaller capacity. The toy experiment in Figure 5 makes this concrete: with a 10-category teacher distribution and a single-Gaussian student, SeqKD spreads probability mass across all 10 categories, producing a poor approximation. In the LLM setting, this manifests as memorization of local lexical patterns. Figure 4 demonstrates this empirically: the SeqKD student shows higher N-gram overlap with the teacher than the GAD student, yet achieves lower GPT-4o evaluation scores. The paper interprets this as evidence that SeqKD is "overfitting to local patterns" rather than capturing the teacher's "global stylistic characteristics."

2. Poor out-of-distribution generalization. Table 2 and Figure 1 reveal a consistent pattern: SeqKD yields marginal or even negative improvements on out-of-distribution benchmarks (Dolly, SelfInst, Vicuna) despite improving in-distribution (LMSYS-Chat) performance. For example, Qwen2.5-14B-Instruct goes from 49.4 on SelfInst before distillation to 49.4 after SeqKD—zero improvement—while GAD reaches 51.1. On Dolly, SeqKD actually reduces Qwen2.5-14B-Instruct's score from 49.1 to 48.2, while GAD lifts it to 50.4. The paper attributes this to a broader finding in the post-training literature: "SFT memorizes, RL generalizes" (Chu et al., 2025; Wu et al., 2025), which the paper explicitly cites. Supervised fine-tuning on a fixed dataset tends to produce models that reproduce training distribution patterns without learning transferable capabilities, whereas reinforcement learning objectives (like GAD's policy gradient) encourage the model to discover generalizable behaviors.

On-policy white-box distillation exists but doesn't transfer to black-box. The paper explicitly acknowledges recent advances in on-policy white-box distillation: MiniLLM (Gu et al., 2024) uses reverse KLD to encourage mode-seeking behavior from student-generated responses; Agarwal et al. (2024) demonstrates the importance of learning from self-generated mistakes; Lu (2025) further develops the on-policy paradigm. These works establish the conceptual importance of on-policy learning but are architecturally tied to white-box access—they compute KL divergences between distributions, which requires the teacher's token-level probabilities. The paper's contribution is not the idea of on-policy distillation (which it credits to prior work) but rather the mechanism for achieving it when those probabilities are unavailable.

Off-policy reward models in RLHF are prone to reward hacking. The paper draws a deliberate parallel to reinforcement learning from human feedback (RLHF; Ouyang et al., 2022), where a reward model is pre-trained on human preference data, then frozen while the policy is optimized against it. A well-documented failure mode is reward hacking (Skalse et al., 2022): the policy learns to exploit idiosyncrasies in the frozen reward model, producing outputs that score highly but are not actually better. The paper argues that applying the same fixed-reward approach to black-box distillation—training a discriminator once on the warmup student's outputs, then freezing it—suffers from exactly this pathology. Figure 6 provides direct evidence: the off-policy discriminator student rapidly diverges after ~300 steps, producing responses that grow to ~1300 tokens (far exceeding the teacher's typical length) as it exploits whatever features the stale discriminator associates with high scores. GAD's co-evolving discriminator avoids this by continually updating to distinguish the student's current outputs from the teacher's.

How GAD Positions Itself

GAD occupies a specific, previously empty intersection in the distillation design space: on-policy, black-box, sequence-level.

• It is on-policy (unlike SeqKD) because the student learns from its own generated responses, with the discriminator providing feedback on those generations. This promotes mode-seeking over mode-covering.
• It is black-box (unlike MiniLLM, Agarwal et al., Lu) because it requires only text outputs from the teacher, not logits or hidden states. This makes it applicable to proprietary API models and tokenizer-incompatible open-source models alike.
• It operates at the sequence level (unlike token-level KL methods) because the discriminator scores complete [prompt, response] pairs as scalars. This has the side benefit of being tokenizer-agnostic.

The paper frames this not as a marginal improvement over SeqKD but as a qualitatively different learning paradigm. The core metaphor is the generative adversarial network (Goodfellow et al., 2014): the student is a generator producing responses, the discriminator learns to distinguish student from teacher, and the student is optimized to fool the discriminator. This creates a minimax game where the discriminator's improving ability to detect student-generated text forces the student to more closely approximate the teacher's distribution. In the space of existing black-box methods, SeqKD is purely imitative (mimic the teacher's outputs) while GAD is adversarial (compete against a critic that learns to spot differences). The paper demonstrates that this adversarial formulation yields consistently superior results across model sizes, model families (Qwen2.5 and Llama3), and evaluation settings (in-distribution, out-of-distribution, automatic, and human).

A subtle but important positioning choice: the paper does not claim GAD is fundamentally new as a concept—GANs and adversarial training for text have existed since SeqGAN (Yu et al., 2017). Rather, it claims GAD is a novel and effective application of this paradigm to the specific problem of black-box LLM distillation, with careful engineering choices (Bradley-Terry loss for the discriminator, GRPO for policy optimization, joint warmup) that make it work reliably at scale. The results position GAD as the new state-of-the-art for black-box distillation, with the 14B student approaching teacher-level performance—a claim SeqKD cannot match.

3. Technical Approach

3.1 Reader Orientation

GAD is a training procedure that turns black-box LLM distillation into an adversarial game between a student model (the generator) and a learned critic (the discriminator) that co-evolve, with the discriminator providing on-policy feedback that replaces the missing teacher probability signal. The problem it solves is that when you only have text outputs from a proprietary teacher, there is no direct way to evaluate whether the student's own generated responses are "teacher-like" — GAD creates this evaluation signal automatically by training a discriminator to distinguish student from teacher, then optimizing the student to fool it.

3.2 Big-Picture Architecture (Diagram in Words)

The GAD system has three major components that interact in a two-phase training process:

1. The Generator (the student LLM): Takes an instruction prompt $x$ as input and produces a text response $G(x)$ via auto-regressive sampling. This is the model being distilled — it starts as a pre-trained instruction-tuned model (e.g., Qwen2.5-7B-Instruct) and is progressively updated to produce responses indistinguishable from the teacher.

2. The Discriminator (a learned quality scorer): Takes a full prompt-response pair $[x, y]$ as input and outputs a single scalar score $D([x, y])$ indicating how "teacher-like" the response is. It is initialized from the same base model as the generator, with an additional linear prediction head that projects the final hidden state of the last token into a scalar. It is continually updated to assign higher scores to teacher responses than to student responses.

3. The Teacher Model (a fixed black-box): A proprietary LLM (GPT-5-Chat in the main experiments) that produces reference responses $y_t$ for each training prompt. The teacher is only used for data generation and as a target for the discriminator; its internal parameters are never accessed.

Information flows through the system in two alternating phases per training step:

• Discriminator update phase: For a batch of prompts $x$, the generator produces student responses $G(x)$, and teacher responses $y_t$ are retrieved from the pre-constructed dataset. The discriminator scores both $[x, G(x)]$ and $[x, y_t]$, then updates its parameters to maximize the score gap between teacher and student responses using a Bradley-Terry loss.

• Generator update phase: For the same batch of prompts, the generator produces a new set of student responses $G(x)$ (at a different temperature). The discriminator scores these responses. The generator is updated via policy gradient (GRPO) to maximize the discriminator-assigned scores, effectively learning to produce responses that the discriminator cannot distinguish from the teacher's.

The discriminator never sees the teacher's logits or probability distribution — it learns to assess response quality entirely from text, making the entire pipeline black-box compatible. A warmup phase precedes the adversarial training, where the generator is fine-tuned on teacher responses via standard cross-entropy (SeqKD) and the discriminator is pre-trained on the same data, establishing a balanced starting point before the minimax game begins.

3.3 Roadmap for the Deep Dive

The technical explanation proceeds in this order:

1. The formal minimax objective (Equation 1 and its decomposition): This establishes the mathematical framework — what the generator and discriminator are optimizing, why it takes the form of a zero-sum game, and how the Bradley-Terry model captures pairwise preferences between teacher and student responses.

2. The discriminator: architecture, training, and design choices: How the discriminator is constructed from a language model backbone, what the Bradley-Terry loss computes, why Bradley-Terry is preferred over binary cross-entropy (ablated in Table 4), and what it means for the discriminator to be "on-policy."

3. The generator: policy gradient optimization with GRPO: How the student is trained using the discriminator's scores as rewards, the specific mechanics of GRPO (group relative policy optimization), and how advantage normalization stabilizes training across varying discriminator score scales.

4. The warmup phase and its necessity: Why both generator and discriminator need initialization before adversarial training begins — the distributional gap problem, the balance problem, and the empirical evidence from the ablation in Table 3.

5. The training algorithm end-to-end: The full pseudocode (Algorithm 1) translated into an operational walkthrough, including hyperparameters, batch structure, and the interleaving of generator and discriminator updates.

6. Implementation within existing RL frameworks: How GAD maps onto standard RLHF infrastructure (policy model → generator, reward model → discriminator), the key differences from standard RLHF (co-evolving reward model vs. frozen reward model), and the specific configuration choices (GRPO group size, KL penalty, learning rates).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that on-policy black-box distillation can be achieved by framing the student and a learned discriminator as adversaries in a minimax game, where the discriminator's continually adapting scores replace the unavailable teacher probability signal.

---

The Minimax Objective

The value function $V(G, D)$ and what it encodes.

The paper formalizes the adversarial distillation process as a two-player zero-sum game with the following value function:

$\max_G \min_D V(G, D) = \mathbb{E}_{(x, y_t) \sim \mathcal{T}} \left[ -\log \sigma \left( D(y_t) - D(G(x)) \right) \right]$

where $(x, y_t)$ is a prompt-teacher-response pair sampled from the distillation dataset $\mathcal{T}$, $G(x)$ is the student's generated response to prompt $x$, $D(y_t)$ is shorthand for the discriminator's scalar score on the concatenated teacher response (i.e., $D([x, y_t])$), $D(G(x))$ is the discriminator's score on the concatenated student response, and $\sigma(\cdot)$ is the sigmoid function $\sigma(z) = 1/(1 + e^{-z})$.

What it computes: The core of this expression is the Bradley-Terry preference model. The term $\sigma(D(y_t) - D(G(x)))$ represents the probability — according to the Bradley-Terry model — that the teacher response is preferred over the student response, given their respective discriminator scores. The log of this probability is negated, so minimizing the negative log-likelihood $-\log \sigma(D(y_t) - D(G(x)))$ means maximizing the likelihood that the teacher scores higher than the student. The outer expectation averages this preference loss over all training examples.

The minimax structure in plain terms: The discriminator $D$ wants to minimize $V(G, D)$ — meaning it wants $-\log \sigma(D(y_t) - D(G(x)))$ to be as small as possible, which occurs when $D(y_t) \gg D(G(x))$, making $\sigma(\cdot)$ close to 1 and $-\log(1) = 0$. This pushes the discriminator to assign much higher scores to teacher responses than student responses. The generator $G$ wants to maximize $V(G, D)$ — meaning it wants $-\log \sigma(D(y_t) - D(G(x)))$ to be as large as possible, which occurs when $D(G(x)) \geq D(y_t)$, making $\sigma(\cdot)$ close to or below 0.5 and $-\log(0.5) \approx 0.69$ or larger. This pushes the generator to produce responses that the discriminator scores as highly as — or higher than — teacher responses.

Why this form: The Bradley-Terry formulation is chosen over a simple binary classification objective (where the discriminator would predict "teacher" vs. "student" for each response independently) because it models pairwise relative quality rather than absolute categorization. The ablation in Table 4 confirms this: the Bradley-Terry loss (Equation 3) outperforms the cross-entropy loss (Equation 4) on the Qwen2.5-3B-Instruct student, achieving 48.9 vs. 47.9 on LMSYS-Chat. The pairwise formulation has the property that the discriminator's scores are only meaningful relative to each other within a comparison pair, not as absolute values — this makes the scores more robust to scale drift during training and provides a more natural gradient signal for the generator (the generator is incentivized to increase its score relative to the teacher, not hit any absolute threshold). Additionally, the Bradley-Terry loss is identical to the loss used for reward model training in RLHF (Ouyang et al., 2022), which means GAD can reuse the same infrastructure and stabilization techniques.

---

The Discriminator: Architecture, Training, and Design Choices

Architecture. The discriminator is initialized from the same pre-trained language model as the generator (e.g., both start from Qwen2.5-7B-Instruct weights). An additional linear prediction head is attached to the final hidden state of the model. Specifically, the head projects the final hidden state vector $h_{\text{last}} \in \mathbb{R}^d$ (where $d$ is the model's hidden dimension) to a single scalar $s \in \mathbb{R}$. The score of the last token in the concatenated $[x, y]$ sequence is taken as the sequence-level score $D([x, y])$. The paper states:

"The discriminator is initialized using generator model parameters with an extra prediction head. The head projects the final hidden state to a scalar score, and the score of the last token in the sequence is taken as the sequence-level score." (Section 2.1)

This design choice (using the generator's backbone for the discriminator) is significant: it means the discriminator shares the same language understanding capabilities as the student, but is specialized via the prediction head for the task of quality assessment. This shared initialization also makes the warmup phase more effective — both models start from a similar representation space, so the discriminator can provide meaningful gradients to the generator from the beginning of adversarial training.

The Bradley-Terry training objective. The discriminator's loss, extracted from Equation 1, is:

$\min_D \mathbb{E}_{(x, y_t) \sim \mathcal{T}} \left[ -\log \sigma \left( D(y_t) - D(G(x)) \right) \right]$

where $D(y_t)$ is the score for the teacher response and $D(G(x))$ is the score for the student response.

What it computes: For each prompt $x$, the discriminator scores both the teacher's response $y_t$ and the student's generated response $G(x)$. The loss term $-\log \sigma(D(y_t) - D(G(x)))$ is small (near zero) when $D(y_t) \gg D(G(x))$ — meaning the discriminator correctly ranks the teacher above the student. The loss is large when $D(G(x)) \geq D(y_t)$ — meaning the student's response scores equal to or higher than the teacher's, indicating the discriminator has failed to distinguish them. The expectation averages this over all training prompts, and the discriminator's parameters are updated via gradient descent to minimize this average loss.

At the implementation level, the discriminator processes $N$ student responses per prompt (the GRPO group size, $N = 8$ in the experiments). For each prompt, the teacher response $y_t$ is paired with each of the $N$ student responses $y_s^{(i)}$, forming $N$ preference pairs $(y_t, y_s^{(i)})$. The loss is averaged over the group:

$\min_D \mathbb{E}_{(x, y_t) \sim \mathcal{T}, \{y_s^{(i)}\}_{i=1}^N \sim q_G(\cdot|x)} \left[ \frac{1}{N} \sum_{i=1}^N -\log \sigma \left( D(y_t) - D(y_s^{(i)}) \right) \right]$

(Equation 8 in Appendix A.1). Grouping in this way ensures the discriminator is trained on a diverse set of student responses per prompt, preventing it from overfitting to a single failure mode of the generator.

Why this form (vs. binary cross-entropy): The alternative discriminator loss considered (and ablated in Table 4) is the standard GAN discriminator objective — binary cross-entropy:

$\min_D \mathbb{E}_{(x, y_t) \sim \mathcal{T}} \left[ -\log \sigma(D(y_t)) - \log(1 - \sigma(D(G(x)))) \right]$

In the binary cross-entropy formulation, the discriminator is trained as a classifier: predict 1 (teacher) for teacher responses and 0 (student) for student responses. The scores $D(y_t)$ and $D(G(x))$ are interpreted as logits for a binary classification, and the loss encourages them to be pushed to opposite extremes (large positive for teacher, large negative for student). The Bradley-Terry formulation is different: it only cares about the difference $D(y_t) - D(G(x))$, not the absolute values. This has two practical advantages. First, it prevents score magnitude drift — in binary cross-entropy, there is nothing stopping the scores from growing arbitrarily in magnitude as training progresses, which can destabilize the generator's policy gradient. In Bradley-Terry, only the relative gap matters. Second, it aligns with the preference-based interpretation: the discriminator is not saying "this response is a teacher response" but rather "the teacher's response is better than the student's response," which is a more natural signal for the generator's optimization (the generator wants to close this quality gap, not achieve some absolute classification threshold). The experimental result in Table 4 (48.9 vs. 47.9 on LMSYS-Chat for Qwen2.5-3B-Instruct) empirically confirms the superiority of Bradley-Terry.

On-policy vs. off-policy discriminator: why co-evolution matters. The discriminator is described as "on-policy" because it is trained on the current student's outputs. At each training step, the student generates fresh responses, and the discriminator is immediately updated on those responses. This means the discriminator continually adapts to the evolving student distribution. In contrast, an off-policy discriminator would be trained once on a snapshot of the student's outputs (e.g., after the warmup phase) and then frozen. The paper's Figure 6 demonstrates why this fails: the frozen discriminator cannot track the student's distributional shift during training. Within approximately 300 training steps, the student trained against an off-policy discriminator begins producing responses that are approximately 1,300 tokens long — far exceeding the teacher's typical response length — while maintaining high discriminator scores. This is a classic reward hacking pattern: the student has discovered that the frozen discriminator associates some superficial feature (likely response verbosity or certain lexical patterns present in the warmup student's outputs) with high scores, and it exploits that feature without improving actual response quality. The on-policy discriminator in GAD avoids this because it is continually updated to distinguish the student's current outputs from the teacher — if the student starts producing pathologically long responses, the discriminator learns to penalize that, forcing the student back toward the teacher's distribution.

---

The Generator: Policy Gradient Optimization with GRPO

The generator's objective. Extracted from the minimax game (Equation 1), the generator maximizes:

$\max_G \mathbb{E}_{(x, y_t) \sim \mathcal{T}} \left[ D(G(x)) \right]$

where $D(G(x))$ is the discriminator's scalar score for the student-generated response.

What it computes: This is a pure reward maximization objective. For each prompt $x$, the generator produces a response $G(x)$, the discriminator scores it, and the generator is updated to increase that score. Unlike SeqKD, there is no explicit likelihood term for the teacher's response — the generator is never shown the teacher's text during its update. Instead, the discriminator's score serves as a learned quality signal that implicitly guides the generator toward teacher-like behavior.

Why policy gradient is necessary. The generation process $G(x)$ involves sampling discrete tokens from the student's output distribution $q_\theta(\cdot|x)$. This sampling operation is non-differentiable — you cannot backpropagate through a discrete sampling step. Therefore, the generator cannot be optimized with standard gradient descent on the discriminator score. Instead, the paper treats $D(G(x))$ as a reward signal and applies a policy gradient method. The policy gradient theorem (Sutton et al., 1999) provides an unbiased estimator of the gradient of the expected reward with respect to the policy parameters $\theta$:

$\nabla_\theta \mathbb{E}_{y \sim q_\theta(\cdot|x)} [D(y)] = \mathbb{E}_{y \sim q_\theta(\cdot|x)} \left[ D(y) \cdot \nabla_\theta \log q_\theta(y|x) \right]$

In plain terms: the gradient pushes up the log-probability of generated responses proportional to the reward (discriminator score) they receive. Responses that score highly have their probability increased; responses that score poorly have their probability decreased.

GRPO: Group Relative Policy Optimization. The paper uses GRPO (Shao et al., 2024) as the specific policy gradient algorithm. GRPO addresses a key practical challenge: the discriminator's raw scores $D(G(x))$ can vary in magnitude and offset across training steps (as the discriminator itself evolves), making the absolute score an unreliable reward signal. GRPO normalizes rewards within a group of responses generated for the same prompt.

For each prompt $x$, the generator produces a group of $N = 8$ responses $\{y_s^{(i)}\}_{i=1}^N$. The discriminator scores each response, producing rewards $r^{(i)} = D(y_s^{(i)})$. The advantage for the $i$-th response is computed as:

$A^{(i)} = \frac{r^{(i)} - \text{mean}(\{r^{(j)}\}_{j=1}^N)}{\text{std}(\{r^{(j)}\}_{j=1}^N)}$

where $r^{(i)}$ is the discriminator score for the $i$-th student response, $\text{mean}(\{r^{(j)}\}_{j=1}^N)$ is the average score across all $N$ responses in the group, and $\text{std}(\{r^{(j)}\}_{j=1}^N)$ is the standard deviation of scores across the group.

What this normalization does: The advantage $A^{(i)}$ is a z-score: it measures how many standard deviations above or below the group mean each response scores. A response that scores better than average gets a positive advantage (it is "above average" and should be reinforced); a response that scores worse gets a negative advantage (it is "below average" and should be suppressed). The standardization removes two sources of noise: (1) the absolute scale of discriminator scores (which can drift over training) and (2) the prompt-dependent baseline (some prompts naturally produce higher or lower scores). By normalizing within each prompt's group, GRPO ensures the policy gradient is driven by relative response quality within a group rather than absolute score magnitudes.

The generator is then updated with the policy gradient objective:

$\max_G \mathbb{E}_{(x, y_t) \sim \mathcal{T}, \{y_s^{(i)}\}_{i=1}^N \sim q_G(\cdot|x)} \left[ \frac{1}{N} \sum_{i=1}^N A^{(i)} \right]$

(Equation 7 in Appendix A.1, simplified — the paper notes it omits the KL regularizer and clip operator from the standard GRPO formulation for brevity, though the full implementation includes them). Each response's log-probability is updated proportionally to its advantage: positive-advantage responses are reinforced, negative-advantage responses are penalized.

Why GRPO over alternatives: The paper does not extensively ablate policy gradient algorithms, but the choice of GRPO (over, say, vanilla REINFORCE or PPO) is motivated by practical stability considerations. Vanilla REINFORCE with raw rewards would be highly sensitive to discriminator score drift. PPO (Schulman et al., 2017) uses a learned value function to estimate advantages, but in the GAD setting the "reward function" (the discriminator) is non-stationary — it changes every training step — making value function learning unreliable. GRPO's group-based normalization provides a simple, hyperparameter-free way to compute advantages that naturally adapts to the evolving reward distribution. The group size $N = 8$ is a practical choice that balances computational cost (each group requires 8 forward passes through the generator) with statistical reliability of the mean and standard deviation estimates.

KL regularization. Although the paper omits it from the simplified equations, the full GRPO implementation includes a KL divergence penalty with weight $\beta = 0.001$ (stated in Appendix A.2). This penalizes the student's policy for diverging too far from a reference policy (typically the pre-distillation instruct model). The KL penalty serves as a regularizer, preventing the student from collapsing to a degenerate policy that produces high discriminator scores but has lost general language capabilities. A weight of 0.001 is relatively light, allowing substantial policy movement while providing a safety net against catastrophic drift.

---

The Warmup Phase: Why It Is Necessary and How It Works

The two problems warmup solves. Before adversarial training begins, both the generator and discriminator undergo a joint warmup phase of one epoch. This warmup addresses two distinct failure modes identified in the ablation study (Table 3):

1. Generator warmup prevents the distributional gap problem. Without generator warmup, the student (initialized from a standard instruction-tuned model like Qwen2.5-7B-Instruct) produces responses that are very different from the teacher's (GPT-5-Chat) in style, length, and quality. The discriminator, seeing this large gap, quickly learns to distinguish the two with near-perfect accuracy. At that point, the discriminator's scores provide almost no useful gradient to the generator — every student response gets a uniformly low score, and the generator has no signal about which of its responses are relatively better. The adversarial game collapses because the discriminator is too strong. Generator warmup narrows this gap: by fine-tuning the student on teacher responses for one epoch via standard SeqKD, the student's output distribution moves much closer to the teacher's. The discriminator then faces a more challenging (and more informative) distinction task from the start.

2. Discriminator warmup prevents the imbalance problem. Without discriminator warmup, the discriminator is initialized from the raw instruct model checkpoint with a randomly initialized prediction head. When adversarial training begins, the discriminator produces essentially random scores, providing no meaningful signal to the generator. The generator optimizes against noise. By the time the discriminator starts to learn, the generator may have already drifted in unproductive directions. Discriminator warmup pre-trains the discriminator using the warmup-student's outputs and the Bradley-Terry loss (Equation 3) for one epoch, giving it a reasonable initial assessment capability before the generator begins optimizing against it.

Warmup procedure in detail. During the warmup epoch:

• The generator is updated with standard cross-entropy loss on the teacher's responses $y_t$ — exactly the SeqKD procedure. The objective is $-\log q_\theta(y_t|x)$, maximizing the likelihood of the teacher's token sequence under the student's distribution.
• Simultaneously, the generator produces student responses $G(x)$ (even during warmup), and the discriminator is trained on the preference pairs $(y_t, G(x))$ using the Bradley-Terry loss (Equation 3). The paper notes that for the first 10 steps of warmup, only the discriminator is trained, giving it a head start before joint warmup begins.

After warmup, both models have reasonable starting points: the generator has absorbed some teacher behaviors through SFT, and the discriminator has learned to assess response quality at a basic level. The adversarial training phase then refines both simultaneously.

Empirical evidence for warmup necessity. Table 3 ablates the warmup strategy on Qwen2.5-7B-Instruct:

• Without generator warmup ("w/o Gen. Warmup"): GPT-4o score drops from 50.8 (full GAD) to 49.7 on LMSYS-Chat, and from 50.0 to 49.7 on out-of-distribution benchmarks. The performance is still better than SeqKD (49.2/48.3), suggesting the adversarial mechanism provides some benefit even without warmup, but substantially worse than full GAD.
• Without discriminator warmup ("w/o Disc. Warmup"): GPT-4o score drops to 49.0 on LMSYS-Chat and 47.7 on out-of-distribution — effectively back to SeqKD levels. This is the more severe failure mode: without discriminator warmup, the adversarial game never becomes productive. The paper explains: "the imbalance between the generator and the discriminator prevents the discriminator from providing sufficiently informative feedback. Consequently, the adversarial interaction becomes ineffective, and the generator exhibits little improvement beyond its warmup performance."

Both ablations demonstrate that warmup is not just a nice-to-have but a critical enabling condition for GAD's adversarial dynamics.

---

The Training Algorithm End-to-End

This section walks through Algorithm 1 from the paper, expanding each step with concrete operational details and hyperparameters.

Phase 0: Data preparation. The training dataset $\mathcal{T}$ is constructed from 200,000 prompts sampled from LMSYS-Chat-1M-Clean. For each prompt $x$, the teacher model (GPT-5-Chat) generates one response $y_t$. The dataset consists of pairs $(x, y_t)$. The paper notes: "All models are trained for 3 epochs with a batch size of 256, totaling approximately 2400 optimization steps" (Section 3.1). With 200,000 examples and batch size 256, one epoch is approximately 200,000 / 256 ≈ 781 steps. Three epochs are therefore roughly 2,344 steps — consistent with the stated "approximately 2400."

Phase 1: Warmup (1 epoch, ~781 steps). In each warmup step:

1. A batch of $(x, y_t)$ pairs is sampled from $\mathcal{T}$.
2. The generator produces student responses $G(x)$ for each prompt in the batch. During warmup, these are sampled at temperature 0.8 (the same temperature used throughout training). The maximum generation length is 1,536 tokens for responses, with the prompt context window capped at 2,048 tokens.
3. The generator is updated with cross-entropy loss on the teacher responses $y_t$. This is standard autoregressive language modeling: for each token position $t$, the loss is $-\log q_\theta(y_t | x, y_{<t})$, summed over the sequence. The optimizer is AdamW (though exact Adam hyperparameters are not explicitly listed; the learning rate sweep is $[1\text{e-}6, 5\text{e-}6]$ with 5e-6 used for SeqKD and 1e-6 for GAD's warmup when the teacher is GPT-5-Chat).
4. The discriminator is trained with Bradley-Terry loss (Equation 3) on the pairs $(y_t, G(x))$. For the first 10 steps of warmup, only the discriminator is updated, giving it an initial head start. After step 10, both models are updated jointly in each step.

Phase 2: GAD training (2 epochs, ~1,562 steps). In each GAD training step:

1. A batch of $(x, y_t)$ pairs is sampled.
2. The generator produces a group of $N = 8$ responses $\{y_s^{(i)}\}_{i=1}^8$ for each prompt in the batch. Temperature remains at 0.8. These are fresh samples — the generator's current policy, which has been updated throughout training, is used for generation.
3. The discriminator scores all 8 student responses per prompt using the current discriminator parameters. The scores $r^{(i)} = D([x, y_s^{(i)}])$ are recorded.
4. The discriminator also scores the teacher responses $D([x, y_t])$, producing a teacher score per prompt.
5. The discriminator is updated using the Bradley-Terry loss (Equation 8, the grouped version). For each prompt, the teacher response is paired with each of the 8 student responses, and the average loss over the 8 pairs is minimized:
   • Compute $\sigma(D(y_t) - D(y_s^{(i)}))$ for each $i$.
   • Compute the loss $-\frac{1}{8} \sum_{i=1}^8 \log \sigma(D(y_t) - D(y_s^{(i)}))$.
   • Backpropagate and update the discriminator parameters.
6. The generator is updated using GRPO. For each prompt:
   • Compute the mean and standard deviation of the 8 student response scores.
   • Compute the advantage $A^{(i)}$ for each response (Equation 6).
   • Update the generator parameters to maximize $\frac{1}{8} \sum_{i=1}^8 A^{(i)}$ (Equation 7), with the KL regularization penalty applied.
7. Steps 5 and 6 alternate: the discriminator is updated, then the generator is updated. Both use the same group of student responses for efficiency (the forward passes to generate and score the 8 responses are already done).

Hyperparameter summary (from Section 3.1 and Appendix A.2):

Checkpoint selection. The paper states: "We select the checkpoint that achieved the highest GPT-4o score and whose response length is within an acceptable range for each experiment" (Section 3.1). This is a practical detail: because the generator's response length can drift during adversarial training (as seen in the off-policy discriminator ablation in Figure 6), evaluating checkpoints periodically and filtering by both quality and reasonable length is necessary to select a final model that balances accuracy with conciseness.

Computational cost. Distilling Qwen2.5-14B-Instruct from GPT-5-Chat using GAD requires approximately 30 hours on 16 H100 GPUs (from Appendix A.2). This is a non-trivial compute budget but is comparable to or lower than training pipelines that require white-box teacher access (which would require the larger teacher model to be loaded and run during training alongside the student).

---

Implementation Within Existing RL Frameworks

Mapping GAD onto RLHF infrastructure. Table 1 in the paper establishes an explicit correspondence that makes GAD implementable within standard reinforcement learning for language model frameworks (specifically, the paper uses the verl framework; Sheng et al., 2024):

• The generator corresponds to the policy model in RLHF — the language model being optimized.
• The discriminator corresponds to the reward model in RLHF — the model that scores responses to provide training signals.
• The reward for the generator is $D(G(x))$ — the discriminator's score on the generator's response — directly analogous to how a reward model's score drives policy optimization in RLHF.

The critical difference: co-evolving vs. frozen reward model. In standard RLHF (Ouyang et al., 2022), the reward model is trained once on a static dataset of human preference comparisons and then frozen. The policy model is then optimized against this fixed reward function, typically for thousands of steps. This creates the well-known reward hacking problem: as the policy moves away from the distribution on which the reward model was trained, the reward model's scores become increasingly miscalibrated, and the policy may learn to exploit features of the reward model rather than genuinely improve.

GAD's discriminator is fundamentally different: it is not frozen. At every training step, the discriminator receives the student's current outputs and is updated to distinguish them from the teacher's. This means the discriminator's "reward function" is non-stationary — it continually adapts to track the student's evolving distribution. The generator is always optimizing against a discriminator that has been trained on its current outputs, not on a stale snapshot. This continuous co-adaptation is what prevents the reward hacking shown in Figure 6: the discriminator learns to penalize whatever shortcuts the student discovers, forcing the student to find genuinely better response strategies.

GRPO specifics within the verl framework. The paper implements GAD using the verl framework (Sheng et al., 2024), with GRPO (Shao et al., 2024) as the policy gradient algorithm. The setup follows the standard verl architecture for RLHF training, where:

• A rollout worker runs the generator (policy model) to produce the group of 8 responses per prompt.
• A reward worker runs the discriminator to score both the teacher responses and the student responses.
• The policy update worker computes advantages using GRPO's group normalization and updates the generator parameters.
• The discriminator update worker computes the Bradley-Terry loss and updates the discriminator parameters.

The paper notes the PPO mini-batch size of 256, which controls how many examples are used per gradient update within the policy optimization step. This is distinct from the training batch size of 256 prompts per global step. The PPO mini-batch is typically used for memory efficiency: the policy gradient update over 256 prompts × 8 responses = 2,048 total responses may be split into smaller mini-batches for backpropagation.

Training temperature and exploration. The generation temperature of 0.8 is a deliberate choice that balances two competing needs. A temperature of 1.0 would produce maximum diversity in student responses, which is beneficial for exploration — the generator would sample from a wide range of outputs, and the discriminator would need to learn to evaluate many different response styles. However, too much diversity can make the discriminator's learning problem harder and can cause the generator's policy gradient to be very high-variance. A temperature of 0.8 introduces some stochasticity for exploration while keeping the student's outputs relatively focused. The paper does not ablate temperature, so this choice is empirically motivated but not systematically justified.

KL regularization in practice. The KL penalty with weight $\beta = 0.001$ is added to the GRPO objective to penalize the student's policy for deviating from a reference policy. The reference policy is typically the pre-distillation instruct model (e.g., Qwen2.5-7B-Instruct before any training). The KL divergence term is computed per-token as $\text{KL}(q_\theta(\cdot|x) \parallel q_{\text{ref}}(\cdot|x))$ and subtracted from the advantage-weighted objective. With $\beta = 0.001$, the KL penalty is relatively light — it prevents catastrophic forgetting of general language capabilities but allows the student's distribution to shift substantially toward the teacher's. The paper does not provide a detailed ablation of $\beta$, but the value is chosen from typical RLHF practice where excessively high KL penalties prevent the policy from learning and excessively low penalties lead to reward hacking or language degradation.

4. Key Insights and Innovations

Innovation 1: Reframing Black-Box Distillation as an Adversarial Game Rather Than Pure Imitation

The dominant mental model for black-box LLM distillation has been imitative: collect teacher responses, fine-tune the student on them, and hope the student absorbs the teacher's capabilities. This is the SeqKD paradigm (Kim & Rush, 2016) that the field has largely accepted as the best one can do without white-box access. GAD's most fundamental conceptual contribution is rejecting imitation as the right framing and replacing it with an adversarial framing where the student learns to compete against a critic that co-evolves with it.

This is not merely swapping one loss function for another. It represents a shift in what the training signal means. In SeqKD, the training signal is static and absolute — the student is shown the teacher's answer and told "produce this." The student's own generations are never evaluated; they simply don't enter the loss. In GAD, the training signal is dynamic and relative — the student generates responses, and a learned critic evaluates them against the teacher's standard, with both the student's capabilities and the critic's standards co-evolving throughout training. The student is never shown the teacher's answer during its update (only the discriminator sees both student and teacher outputs). Instead, the student must infer what constitutes teacher-like behavior from the discriminator's scalar scores.

This reframing has a direct conceptual payoff: it makes on-policy learning possible in the black-box regime. The prior literature had established that on-policy learning is important for distillation quality (Gu et al., 2024; Agarwal et al., 2024; Lu, 2025), but all existing on-policy methods required the teacher's token-level probabilities to compute KL divergences. Those methods are fundamentally incompatible with API-only teachers. GAD's adversarial framing solves this by creating an on-policy signal from scratch — the discriminator is trained to evaluate the student's current outputs, not a frozen snapshot, so it functions as an adaptive quality oracle that stands in for the unavailable teacher probabilities.

The significance of this reframing extends beyond GAD itself. It establishes that the central challenge of black-box distillation is not "how do we imitate the teacher more accurately" but rather "how do we construct a reliable, adaptive quality signal without teacher internals." This reframes the research question from one of approximation (matching a known target distribution as closely as possible) to one of feedback design (building a learning signal that remains informative as the student improves). The adversarial framework is one solution to this reframed question; future work might explore other feedback mechanisms (e.g., self-play, debate, multi-agent scoring) within the same conceptual structure.

The empirical evidence for this reframing's value is in the aggregate results:

• Figure 1: GAD consistently outperforms SeqKD across all model sizes and both in-distribution and out-of-distribution evaluation.
• Table 2: On LMSYS-Chat, Qwen2.5-14B-Instruct + GAD reaches 52.1 vs. 50.6 for SeqKD — a 1.5-point gap at near-teacher performance levels (teacher = 51.7).

The fact that GAD surpasses the teacher on the in-distribution test set (52.1 > 51.7) while SeqKD falls short (50.6 < 51.7) is particularly telling. It suggests the adversarial process is not merely imitating but actually amplifying the teacher's capabilities through the student's own exploration, which an imitative approach cannot do.

Innovation 2: The On-Policy Discriminator as a Diagnostic Distinction with Falsifiable Failure Mode

The paper introduces a crisp, empirically testable distinction that had not been clearly drawn in the distillation literature: an on-policy discriminator (co-evolves with the student, trained on current student outputs) versus an off-policy discriminator (trained once on a snapshot and frozen). This distinction matters because it predicts a specific, observable failure mode for off-policy approaches — reward hacking via distributional drift — and the paper provides direct evidence for this prediction in Figure 6.

Prior work in RLHF had documented reward hacking with frozen reward models (Skalse et al., 2022), but that finding was in the context of human preference optimization, not distillation. The distillation literature had not systematically distinguished between on-policy and off-policy quality signals for black-box settings, largely because the concept of using a learned discriminator as a quality signal for distillation was itself underexplored. GAD makes this distinction central and demonstrates that it is not a theoretical subtlety but a practical hard constraint: the off-policy discriminator student collapses within ~300 training steps, producing responses that grow to ~1,300 tokens while maintaining high scores.

What makes this a genuine insight rather than an engineering observation is the mechanism it reveals. The off-policy discriminator doesn't fail because it's poorly trained — it was trained on exactly the same data and architecture as the on-policy discriminator at initialization. It fails because the student's policy shift during training moves it into regions where the discriminator's scores become miscalibrated. The discriminator's training distribution (warmup-student outputs) and the student's inference-time distribution (current-student outputs) diverge, and without retraining, the discriminator's scores become misleading. This is the same mechanism underlying reward hacking in RLHF, but applied to distillation rather than preference optimization.

The practical implication is that any black-box distillation method using a learned quality signal must maintain that signal's calibration as the student evolves. A one-time trained discriminator will eventually be gamed. GAD's solution — co-evolution — is one approach; future methods might use different recalibration strategies (periodic retraining, uncertainty-aware scoring, ensemble disagreement signals). But the diagnostic framework (on-policy vs. off-policy, with the specific prediction of reward hacking) is GAD's contribution, not just the specific co-evolution implementation.

The evidence is clean and unambiguous: Figure 6 shows the on-policy discriminator maintaining stable response lengths (~500-600 tokens) over 2,000+ steps, while the off-policy discriminator student's response length starts climbing sharply after step ~300 and reaches ~1,300 tokens by step ~800. The paper doesn't report GPT-4o scores for the off-policy variant beyond this length metric, but the pathological length explosion alone is sufficient to demonstrate the failure mode.

Innovation 3: Mode-Seeking as the Bridge Between GAN-like Training and Distillation Quality

The toy experiment in Figure 5 does more than illustrate GAD's behavior — it provides a theoretical diagnosis of why adversarial training works for distillation that connects the GAN literature to the distillation literature through the concept of mode-seeking.

In the toy setup, the teacher is a Gaussian mixture with 10 modes (categories 0-9). The student is a single Gaussian — it fundamentally cannot represent the full teacher distribution. SeqKD (forward KL minimization) produces a mode-covering solution: the student spreads its probability mass across all 10 categories, approximating the teacher's distribution in expectation but not concentrating on any particular mode. GAD produces a mode-seeking solution: the student concentrates its probability mass on a subset of modes where it can most credibly match the teacher's characteristics.

This distinction — mode-covering vs. mode-seeking — was already established in the white-box distillation literature. Gu et al. (2024) demonstrated that reverse KL (mode-seeking) outperforms forward KL (mode-covering) for LLM distillation, and framed MiniLLM explicitly around this property. What GAD adds is the finding that adversarial training naturally induces mode-seeking behavior in the black-box setting, without requiring explicit KL computation. The generator, by optimizing to maximize the discriminator's score, learns to produce responses that the discriminator cannot distinguish from the teacher's. Since the discriminator is continually updated to identify differences, the generator is forced to focus on the modes where it can most convincingly match the teacher — exactly the mode-seeking behavior that white-box methods achieve through reverse KL.

This insight explains why GAD outperforms SeqKD in a way that is grounded in the existing theoretical understanding of distillation. SeqKD is a mode-covering objective (it maximizes the likelihood of all teacher outputs, including rare or idiosyncratic ones that the student cannot credibly reproduce). GAD is a mode-seeking objective (it focuses the student's learning on producing responses that survive adversarial scrutiny). The N-gram overlap analysis in Figure 4 provides supporting evidence: SeqKD produces higher N-gram overlap with the teacher (it's memorizing surface-level patterns across all teacher outputs) while achieving lower evaluation scores (it's not producing good responses, just teacher-similar vocabulary). GAD produces lower N-gram overlap (it's not copying the teacher's surface patterns) but higher evaluation scores (it's producing better responses by focusing on quality over mimicry).

The significance of this insight is that it provides a principled explanation for GAD's generalization advantage. Mode-seeking means the student learns to produce responses that are credible as teacher-like without needing to cover every mode of the teacher's distribution. This is a better match for the student's limited capacity: a 3B-parameter student cannot represent the full distribution of a ~14× or larger teacher, so it should focus on the subset of behaviors it can execute well. SeqKD forces coverage across all modes, which dilutes the student's representational capacity across too many behaviors, leading to mediocrity across all of them. GAD's mode-seeking concentrates capacity where it matters, producing better peak performance.

The out-of-distribution results in Table 2 validate this interpretation. Mode-covering (SeqKD) produces models that reproduce training-distribution patterns — when tested on new distributions, those patterns may not transfer, leading to the negative transfer observed on Dolly and SelfInst. Mode-seeking (GAD) produces models that have learned generalizable quality criteria (via the discriminator's evolving standard), which transfer better to new distributions.

Innovation 4: The Warmup Phase as a Principled Initialization Strategy for Adversarial LLM Training

Adversarial training of language models is notoriously unstable. Early text GAN work (SeqGAN; Yu et al., 2017) struggled with training dynamics where the discriminator would quickly overpower the generator, producing near-zero gradients and collapsing the adversarial game. GAD's introduction of a joint warmup phase — where both generator and discriminator are pre-trained on teacher data before adversarial training begins — is presented as an engineering detail but functions as a principled solution to this longstanding instability problem.

What makes this an innovation rather than an implementation hack is the diagnostic framing the paper provides for why warmup works and which warmup failures produce which symptoms. The ablation in Table 3 distinguishes two distinct failure modes:

• Removing generator warmup causes a moderate performance drop (50.8 → 49.7 on LMSYS-Chat for Qwen2.5-7B-Instruct). The paper attributes this to the discriminator easily distinguishing student from teacher due to the large initial distributional gap, weakening the adversarial signal. The adversarial game still provides some benefit (49.7 > 49.2 for SeqKD), suggesting the generator eventually recovers partially on its own.

• Removing discriminator warmup causes a severe performance drop (50.8 → 49.0), back to SeqKD levels. The paper attributes this to an imbalance where the discriminator provides no informative feedback early in training, causing the generator to optimize against noise. The adversarial game never becomes productive.

This diagnostic decomposition has implications beyond GAD. It suggests that for any adversarial training setup involving language models, the discriminator-side initialization is more critical than the generator-side initialization. The generator can recover from starting behind the teacher if the discriminator provides useful gradients; the discriminator cannot recover from starting with random scores because the generator will be misled before the discriminator learns. This is a testable hypothesis for future adversarial training methods: invest more in discriminator pretraining than generator pretraining, and expect asymmetric sensitivity to warmup quality.

The joint warmup strategy also reveals something about the balance requirements of the adversarial game. GAD works best when the discriminator and generator are matched in capability — Table 5 shows that increasing the discriminator size relative to the generator (e.g., 7B discriminator for a 3B generator) does not improve and may slightly degrade performance. The warmup phase establishes this balance by giving both models the same starting point and the same initial training signal. Without warmup, the adversarial game begins with mismatched capabilities (a strong discriminator from random initialization, or a strong generator from SFT), and the minimax dynamics fail to converge to a productive equilibrium.

The practical significance is that this warmup strategy makes GAD reliably trainable at scale. The paper reports training 14B-parameter models with GAD successfully, which is non-trivial for adversarial training of language models. The ablation provides a clear recipe for practitioners: one epoch of joint SeqKD + discriminator pretraining, with the first 10 steps being discriminator-only to establish a scoring baseline, followed by adversarial training. Without this recipe, GAD would likely be too unstable to reproduce, which would undermine its practical value regardless of its theoretical appeal.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary training and in-distribution evaluation dataset is LMSYS-Chat-1M-Clean, a cleaned version of the LMSYS-Chat-1M dataset (Zheng et al., 2024) derived from conversational data collected via the Chatbot Arena platform. The paper samples 200,000 prompts for training and reserves 500 samples as the primary test set. For out-of-distribution evaluation, three additional benchmarks are used: a 500-sample subset from Dolly (Databricks, 2023), the 252-sample SelfInst dataset (Wang et al., 2023), and the 80-question Vicuna benchmark (Chiang et al., 2023).

• Base model(s). The teacher model is GPT-5-Chat (OpenAI, 2025), a closed-source chat model. Student models span two open-source families: Qwen2.5-Instruct (Yang et al., 2025) at 3B, 7B, and 14B parameter scales, and Llama-3 (Grattafiori et al., 2024) at 3B (Llama-3.2-3B-Instruct) and 8B (Llama-3.1-8B-Instruct) parameter scales. These families are chosen to demonstrate cross-architecture generality. The paper also explores a Qwen2.5-14B-Instruct teacher in Table 7 to test the tokenizer-incompatibility scenario where white-box methods are inapplicable despite the teacher being open-source.

• Metrics. The primary automatic evaluation metric is the GPT-4o score, computed via the LLM-as-a-judge protocol (Zheng et al., 2023; Gu et al., 2024). GPT-4o first generates reference answers for each prompt, then scores both the reference and the student model's output on a scale of 1–10 for helpfulness, relevance, accuracy, and detail. The reported score is the student's score divided by the sum of the student's score and the reference answer's score, yielding a normalized value where 50 represents parity with the reference. Response length (in tokens) is also tracked as a secondary diagnostic for reward hacking. Human evaluation is conducted on LMSYS-Chat-1M-Clean for Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, and Llama-3.1-8B-Instruct, where annotators judge pairwise comparisons (GAD vs. baselines) as win, tie, or loss.

• Baselines. Three comparison points are used throughout. (1) Before Distill.: The instruction-tuned student model without any distillation. (2) SeqKD (Kim & Rush, 2016; implemented as in Chiang et al., 2023 and Taori et al., 2023): Supervised fine-tuning on the teacher's responses using standard cross-entropy loss. This is the dominant black-box distillation baseline and is trained with matched compute (same number of epochs, same batch size, same data). (3) Off-policy discriminator (Figure 6 only): A variant where the discriminator is trained once after warmup on the warmup student's outputs and then frozen during generator training, testing the claim that on-policy co-evolution is necessary.

• Generation budget / compute accounting. All methods are trained for 3 epochs with a batch size of 256 on the same 200K-example dataset, totaling approximately 2,400 optimization steps. For GAD, this comprises 1 warmup epoch (~781 steps) followed by 2 adversarial training epochs (~1,562 steps). The PPO mini-batch size is 256. The GRPO group size is $N = 8$ — each prompt generates 8 student responses per training step. Fair comparison is maintained by allocating identical training data and total optimization steps to all methods.

• Cross-validation / statistical protocol. No explicit cross-validation or statistical significance testing is reported. Checkpoint selection uses the highest GPT-4o score checkpoint whose response length falls within an acceptable range, evaluated on the test set. The paper does not report confidence intervals, standard deviations across runs, or multiple random seeds. For human evaluation, the protocol is pairwise comparison (GAD vs. one baseline at a time), but the number of annotators, inter-annotator agreement metrics, and total number of comparisons are not specified.

Main Quantitative Results

In-Distribution Performance: LMSYS-Chat Test Set

The headline result appears in Table 2 and Figure 1 (left): GAD consistently and substantially outperforms SeqKD across all student model sizes and both model families on the LMSYS-Chat test set.

For the Qwen2.5 family distilled from GPT-5-Chat:

• Qwen2.5-3B-Instruct: Before Distill. = 45.8 → SeqKD = 47.5 → GAD = 48.9. The GAD student surpasses the SeqKD-trained Qwen2.5-7B-Instruct (48.9 > 48.7 before distillation, and comparable to 49.2 with SeqKD), effectively jumping a parameter class.
• Qwen2.5-7B-Instruct: Before Distill. = 48.7 → SeqKD = 49.2 → GAD = 50.8. GAD reaches scores near the teacher itself (51.7).
• Qwen2.5-14B-Instruct: Before Distill. = 50.0 → SeqKD = 50.6 → GAD = 52.1. The GAD student surpasses the teacher (52.1 vs. 51.7), which is not achievable via pure imitation (SeqKD only reaches 50.6).

For the Llama family:

• Llama-3.2-3B-Instruct: 44.0 → 47.6 → 48.1.
• Llama-3.1-8B-Instruct: 46.9 → 49.7 → 50.3.

The absolute gains from GAD over SeqKD range from +0.5 (Llama-3.1-8B-Instruct) to +1.6 (Qwen2.5-7B-Instruct) on LMSYS-Chat. While these absolute gaps appear modest, they are meaningful in a regime where scores are compressed near the teacher's performance ceiling (~51.7). More importantly, the parameter-class jump — a 3B GAD student rivaling a 7B SeqKD student — represents a practical deployment win: a smaller, cheaper model achieving the same quality as a model with more than twice the parameters.

Out-of-Distribution Generalization

The paper's most striking finding is not the in-distribution advantage but the out-of-distribution generalization gap, shown in Table 2 and Figure 1 (right). On Dolly, SelfInst, and Vicuna:

SeqKD frequently fails to generalize and can even hurt performance. For Qwen2.5-14B-Instruct on Dolly: Before Distill. = 49.1 → SeqKD = 48.2 (negative transfer, -0.9 points). On SelfInst: 49.4 → 49.4 (zero improvement). On Vicuna: 50.0 → 49.7 (slightly negative). Similarly, Qwen2.5-7B-Instruct on Dolly drops from 47.6 to 47.2 with SeqKD.

GAD maintains or improves performance across all out-of-distribution benchmarks. For Qwen2.5-14B-Instruct: Dolly = 50.4 (+1.3 over before-distill, +2.2 over SeqKD), SelfInst = 51.1 (+1.7, +1.7 over SeqKD), Vicuna = 51.6 (+1.6, +1.9 over SeqKD). For Qwen2.5-7B-Instruct: Dolly = 48.5 (+0.9, +1.3 over SeqKD), SelfInst = 50.1 (+1.8, +1.8 over SeqKD), Vicuna = 51.4 (+2.3, +1.9 over SeqKD). For Qwen2.5-3B-Instruct: Dolly = 46.7 (+1.6, +1.9 over SeqKD), SelfInst = 47.7 (+2.1, +2.0 over SeqKD), Vicuna = 49.4 (+2.1, +1.4 over SeqKD).

The average across the three out-of-distribution benchmarks (Figure 1, right) shows GAD's trajectory rising toward the teacher line with model scale, while SeqKD remains essentially flat or slightly declining relative to the before-distillation baseline. The paper attributes this to the broader finding that "SFT memorizes, RL generalizes" (Chu et al., 2025; Wu et al., 2025), which it explicitly cites in Section 3.3.

Human Evaluation

Human evaluation results appear in Figure 3, conducted on LMSYS-Chat-1M-Clean for the three larger student models:

• Qwen2.5-7B-Instruct: GAD wins against Before Distill. in 68% of comparisons, ties in 4%, loses in 28%. Against SeqKD: 52% win, 8% tie, 40% loss.
• Qwen2.5-14B-Instruct: Against Before Distill.: 56% win, 28% tie, 16% loss. Against SeqKD: 68% win, 8% tie, 24% loss.
• Llama-3.1-8B-Instruct: Against Before Distill.: 60% win, 28% tie, 12% loss. Against SeqKD: 44% win, 40% tie, 16% loss.

GAD achieves win rates above 50% and loss rates below 30% in nearly all comparisons. The paper notes this as evidence that the GPT-4o automatic evaluation aligns with human preference judgments, though the human evaluation protocol details (number of annotators, total comparisons, agreement metrics) are not reported, which limits the strength of this corroboration.

Tokenizer-Incompatible Distillation (Qwen2.5 Teacher → Llama Students)

Table 7 tests a scenario where the teacher is open-source (Qwen2.5-14B-Instruct) but its tokenizer is incompatible with the Llama student models, preventing direct application of white-box KL-divergence methods. In this setting:

• Llama-3.2-3B-Instruct: Before Distill. = 44.0 on LMSYS-Chat → SeqKD = 46.9 → GAD = 47.5. GAD provides a +0.6 advantage over SeqKD.
• Llama-3.1-8B-Instruct: Before Distill. = 46.9 → SeqKD = 49.0 → GAD = 49.6. GAD provides a +0.6 advantage.

These gains are smaller than in the GPT-5-Chat teacher setting but remain consistent. The teacher's performance is 50.0 on LMSYS-Chat, meaning the 8B student + GAD (49.6) nearly reaches teacher parity despite the tokenizer barrier that would prevent white-box distillation.

N-gram Overlap Analysis: SeqKD Overfits to Surface Patterns

Figure 4 presents an analysis of N-gram overlap (F1 score) between student and teacher responses on the LMSYS-Chat test set for Qwen2.5-14B-Instruct students. The SeqKD student exhibits higher N-gram overlap with the teacher at all N-gram sizes (1 through 6) compared to the GAD student, yet achieves lower GPT-4o scores. For example, at unigram level, SeqKD overlap is approximately 0.7 vs. GAD's ~0.55; at 6-gram overlap, SeqKD is approximately 0.15 vs. GAD's ~0.05. This is interpreted as evidence that SeqKD memorizes local lexical patterns ("overfits to local patterns") while GAD captures the teacher's "global stylistic characteristics" without surface-level copying. The paper notes: "supervised fine-tuning tends to memorize local lexical patterns, whereas our RL-based approach better captures the teacher's global stylistic characteristics" (Section 3.3).

Toy Experiment: Mode-Seeking vs. Mode-Covering

Figure 5 uses a synthetic toy experiment to visualize the learning dynamics. The teacher is a discrete Gaussian mixture distribution over 10 categories (0–9). The student is a single Gaussian — fundamentally underparameterized relative to the teacher. SeqKD (trained via maximum likelihood on teacher samples) spreads probability mass across all 10 categories (mode-covering). GAD (trained via REINFORCE with a discriminator) concentrates probability on a subset of reachable modes (mode-seeking). The visual shows the GAD student's distribution as a sharp peak aligned with the teacher's dominant modes, while the SeqKD student's distribution is a broad, spread-out approximation. The paper argues this mode-seeking behavior is what enables GAD to outperform SeqKD on real LLM distillation tasks: by focusing on reachable, high-quality behaviors rather than attempting to cover the full teacher distribution, the student achieves better peak performance within its limited capacity.

Off-Policy Discriminator Failure: Reward Hacking in ~300 Steps

Figure 6 provides the critical evidence for GAD's central architectural claim — that the discriminator must be on-policy. The off-policy discriminator student (discriminator frozen after warmup) shows stable response lengths around 500–600 tokens for the first ~300 training steps, then an abrupt and sustained increase to approximately 1,300 tokens by step ~800. This length explosion occurs while the student maintains high discriminator scores, indicating reward hacking: the student has discovered that the frozen discriminator rewards verbose outputs, and it exploits this without improving response quality. The on-policy discriminator student maintains stable response lengths around 500–600 tokens throughout 2,000+ training steps. The paper does not report GPT-4o scores for the off-policy variant, but the pathological length behavior alone demonstrates the failure mode.

Ablation Studies and Robustness Checks

• Warmup strategy (Table 3): Removing generator warmup (initiating GAD directly from Qwen2.5-7B-Instruct without the SeqKD warmup epoch) drops LMSYS-Chat score from 50.8 to 49.7 and out-of-distribution average from 50.0 to 49.7. The gap to SeqKD (49.2/48.3) remains, indicating the adversarial mechanism partially recovers. Removing discriminator warmup (using the raw instruct model checkpoint as discriminator initialization with a random prediction head) drops performance to 49.0/47.7 — back to SeqKD levels — suggesting the adversarial game never becomes productive when the discriminator starts from random scoring. This ablation reveals asymmetric sensitivity: discriminator initialization quality is more critical than generator initialization quality.

• Discriminator loss choice (Table 4): On Qwen2.5-3B-Instruct, the default Bradley-Terry loss (Equation 3) achieves 48.9 on LMSYS-Chat and 47.9 averaged on out-of-distribution, compared to 47.9 and 46.4 for the binary cross-entropy loss (Equation 4). The Bradley-Terry formulation's pairwise preference structure provides a ~1 point advantage across both evaluation settings. The paper attributes this to improved training stability from the relative scoring formulation, though no training dynamics analysis is provided for the cross-entropy variant.

• Discriminator model size (Table 5): Increasing the discriminator size relative to the generator degrades or does not improve performance. For a 3B generator, using a 7B discriminator drops LMSYS-Chat score from 48.9 (3B-3B default) to 47.8, and out-of-distribution from 47.9 to 46.9. For a 7B generator paired with a 14B discriminator, the drop is smaller: 50.8 → 50.5 on LMSYS-Chat and 50.0 → 49.9 on out-of-distribution. The finding supports the balanced-pair design choice (equal-sized generator and discriminator) but the 7B → 14B discriminator degradation is minimal, suggesting the balance requirement may be less strict at larger scales.

• Response length as a diagnostic (Figure 6, Table 6): Table 6 reports response token lengths alongside GPT-4o scores for all models. SeqKD tends to produce responses that closely match the teacher's length distribution (e.g., Qwen2.5-14B-Instruct + SeqKD = 319.3 tokens on LMSYS-Chat vs. teacher = 329.1), while GAD maintains the original model's length characteristics while integrating the teacher's quality (Qwen2.5-14B-Instruct + GAD = 438.9 tokens). The paper interprets this as evidence that on-policy sampling "encourages generation patterns aligned with both the student's prior and the teacher's guidance" rather than wholesale mimicry of the teacher's verbosity patterns. On out-of-distribution benchmarks, SeqKD produces notably shorter responses than both the before-distill model and GAD (e.g., Qwen2.5-14B-Instruct on Dolly: 201.6 before → 151.2 SeqKD → 262.6 GAD), which may partially explain SeqKD's out-of-distribution degradation — it has learned to truncate responses in ways that hurt quality on unfamiliar prompts.

• Teacher quality sensitivity: Table 7 tests GAD with a weaker teacher (Qwen2.5-14B-Instruct, GPT-4o score = 50.0) compared to the main teacher (GPT-5-Chat, 51.7). GAD still outperforms SeqKD (+0.6 for both Llama students), but the absolute student scores are lower, demonstrating that GAD's effectiveness is bounded by teacher quality — it cannot extract capability the teacher does not possess.

Critical Assessment

Claim 1: GAD consistently surpasses SeqKD for black-box LLM distillation.

This claim is well-supported by the main results in Table 2 and Figure 1. Across 15 direct comparisons (5 student models × 3 evaluation settings: LMSYS, Dolly, SelfInst, and Vicuna — treated as separate data points), GAD outperforms SeqKD in every case. The margins range from modest (+0.5 on Llama-3.1-8B-Instruct LMSYS) to substantial (+2.2 on Qwen2.5-14B-Instruct Dolly), but the direction is unambiguously positive. The human evaluation in Figure 3 corroborates the automatic metrics, with GAD winning >50% of pairwise comparisons in nearly all settings.

However, the "consistently" claim should be qualified by the narrow experimental scope. All experiments use the same training dataset (LMSYS-Chat-1M-Clean) and the same training recipe (3 epochs, 200K examples, batch size 256). There is no evidence for how GAD performs with different dataset sizes, different data quality, different domain distributions, or different training durations. A method that "consistently" surpasses SeqKD should ideally show this across multiple data regimes — for example, does GAD's advantage hold with 50K examples? With 500K? The paper provides no data scaling analysis.

Additionally, the experiments use a single teacher model (GPT-5-Chat) for the main results, with one additional teacher (Qwen2.5-14B-Instruct) tested only on Llama students. The claim that GAD is generally effective for black-box distillation would be strengthened by testing against multiple teachers with different characteristics (different model families, different sizes, different training paradigms). The consistent advantage may be partially specific to the GPT-5-Chat distributional characteristics.

Claim 2: GAD enables effective on-policy learning in the black-box regime.

This claim is supported by indirect evidence but lacks a direct ablation that isolates the on-policy mechanism. The key pieces of evidence are: (1) Figure 6 shows the off-policy discriminator fails catastrophically, which demonstrates that on-policy co-evolution is necessary for stable training, but does not directly prove that on-policy learning is why GAD outperforms SeqKD. (2) Figure 4 shows GAD produces lower N-gram overlap with the teacher than SeqKD while achieving higher evaluation scores — this is consistent with on-policy mode-seeking behavior but could also be explained by other mechanisms (e.g., the RL objective encouraging more diverse outputs). (3) The out-of-distribution generalization advantage (Table 2) is attributed to the broader "RL generalizes, SFT memorizes" finding, which the paper explicitly cites — but this explains the RL-vs-SFT distinction, not specifically the on-policy-vs-off-policy distinction within RL.

A direct test of the on-policy claim would compare GAD against a variant where the generator is trained on-policy but with a fixed quality signal (e.g., an off-policy discriminator that is periodically retrained rather than continuously co-evolved). The paper's off-policy comparison in Figure 6 freezes the discriminator entirely, which conflates "off-policy" with "stale" — a periodically retrained off-policy discriminator might perform better than the never-retrained variant and would provide a cleaner test of whether continuous co-evolution specifically is necessary.

Claim 3: The discriminator co-evolution prevents reward hacking.

This claim is strongly supported by Figure 6. The evidence is clean and dramatic: the off-policy discriminator student's response length explodes from ~500 to ~1,300 tokens within a few hundred steps, while the on-policy student remains stable. This is a textbook reward hacking pattern and its absence in GAD is clearly attributable to the discriminator's continuous updates.

However, the paper does not provide GPT-4o evaluation scores for the off-policy variant, only response lengths. While the length explosion is strong circumstantial evidence of reward hacking, it would be more convincing to show that the off-policy student's GPT-4o scores actually degrade (or plateau at a low level) during the length explosion. It is possible the off-policy student is producing verbose but genuinely good responses — the paper does not rule this out. Additionally, the off-policy comparison is only shown for one model configuration (inferred to be Qwen2.5-7B-Instruct based on context, though not explicitly stated in the figure caption), so the generality of this result across model sizes is untested.

Claim 4: Qwen2.5-14B-Instruct trained with GAD approaches GPT-5-Chat teacher capability (52.1 vs. 51.7 on LMSYS-Chat).

This claim is numerically accurate but should be interpreted with important caveats. First, the LMSYS-Chat test set is drawn from the same distribution as the training data (both come from LMSYS-Chat-1M-Clean). The student surpassing the teacher on in-distribution data does not mean it generally surpasses the teacher — it means the distillation process has effectively transferred the teacher's capabilities to the student on the specific domain it was trained on. The out-of-distribution results in Table 2 show the 14B GAD student scoring 50.4 on Dolly, 51.1 on SelfInst, and 51.6 on Vicuna — these are not directly comparable to teacher scores because teacher scores on these benchmarks are 49.8, 49.7, and 49.9 respectively. The student outperforms the teacher on Dolly (50.4 > 49.8) and SelfInst (51.1 > 49.7), which is interesting but again reflects the student having been trained to optimize for a specific quality standard that may not align with the teacher's native behavior on these prompts.

Second, "approaches the teacher" should be understood as "approaches the teacher on GPT-4o evaluation scores" — not necessarily on all dimensions of capability. The GPT-4o judge evaluates helpfulness, relevance, accuracy, and detail, but this is a proxy for overall capability, not a comprehensive assessment. The 14B student likely still lags the much larger teacher on factual knowledge, reasoning depth, and robustness to adversarial prompts — dimensions not captured by the evaluation.

Genuine weaknesses and missing experiments:

• Single training dataset across all experiments. All models are trained on LMSYS-Chat-1M-Clean, a conversational dataset. The paper demonstrates out-of-distribution evaluation on Dolly, SelfInst, and Vicuna, but never varies the training data. It is unknown whether GAD's advantage over SeqKD is specific to conversational training data or generalizes to other domains (code, math, scientific text, creative writing). A robustness check with training data from a different domain would substantially strengthen the generality claims.

• No data scaling analysis. All experiments use exactly 200K training examples. The paper does not explore how GAD's performance scales with dataset size — is GAD more data-efficient than SeqKD? Does GAD's advantage grow or shrink with more data? At what data scale does SeqKD catch up? These questions are central to practitioners deciding whether to adopt GAD, and the paper provides no guidance.

• No training duration ablation. All experiments use exactly 3 epochs (1 warmup + 2 GAD). The off-policy failure in Figure 6 occurs around step 300 (within the first GAD epoch). It is unclear whether GAD would remain stable over longer training, or whether the discriminator-generator equilibrium would eventually break down. Similarly, it is unknown whether a shorter GAD phase (e.g., 1 epoch instead of 2) would capture most of the gains, which matters for compute efficiency.

• No KL penalty ablation. The KL penalty weight $\beta = 0.001$ is stated but never ablated. The choice of KL penalty can substantially affect RL training dynamics — too high and the student cannot move toward the teacher, too low and the student may diverge linguistically. The paper's results may be sensitive to this hyperparameter in ways that are not explored.

• Single random seed. There is no evidence that results are reproducible across multiple training runs. The GRPO algorithm involves stochastic sampling (group of 8 responses per prompt at temperature 0.8), and adversarial training is known to be sensitive to initialization and random seed. Reporting standard deviations across 3–5 runs for at least one model configuration would establish whether the reported differences (e.g., 48.9 vs. 47.5 for 3B GAD vs. SeqKD on LMSYS-Chat) are statistically reliable or within noise.

• Human evaluation protocol is underspecified. Figure 3 shows win/tie/loss percentages but does not report the number of annotators, the total number of comparisons, inter-annotator agreement, or whether annotators were blind to model identity. These missing details prevent assessment of the human evaluation's reliability.

• No comparison to white-box distillation methods on the same task. The paper's central claim is that GAD enables effective black-box distillation, but it never compares GAD to a white-box method (e.g., MiniLLM) on the same teacher-student pair to quantify the "black-box tax" — how much performance is lost relative to what could be achieved with full teacher access. The Qwen2.5 teacher experiment (Table 7) uses the teacher as a black-box due to tokenizer incompatibility, but no white-box baseline (e.g., tokenizer-aligning the distributions) is attempted for comparison.

• The N-gram overlap analysis (Figure 4) is limited to a single model size. The finding that SeqKD overfits to local patterns is shown only for Qwen2.5-14B-Instruct. It would be informative to see this analysis across model scales — does the overfitting pattern become more or less pronounced for smaller models? Is there a correlation between N-gram overlap reduction and GPT-4o score improvement across model sizes?

• The toy experiment (Figure 5) uses REINFORCE, not GRPO. The toy GAD implementation uses the simpler REINFORCE algorithm (Williams, 1992) rather than GRPO, which is used in the main LLM experiments. While the point of the toy experiment is to illustrate mode-seeking vs. mode-covering behavior in principle, the algorithm mismatch means it does not directly simulate the LLM training dynamics.

• Compute cost is reported for only one configuration. The paper states that distilling Qwen2.5-14B-Instruct takes ~30 hours on 16 H100 GPUs, but does not provide comparable data for SeqKD training time, other model sizes, or the off-policy discriminator variant. Without comparative cost data, practitioners cannot assess the cost-performance tradeoff of adopting GAD over SeqKD. If GAD training is substantially more expensive (it requires both generator and discriminator forward/backward passes, plus the GRPO group sampling), the modest absolute gains on in-distribution evaluation may not justify the cost for all use cases.

Summary of evidential strength: The paper makes a compelling case that GAD outperforms SeqKD on the specific experimental configuration tested (GPT-5-Chat teacher, LMSYS-Chat training data, Qwen2.5/Llama3 students, 200K examples, 3 epochs). The out-of-distribution generalization advantage is the most practically significant finding and is well-demonstrated across three benchmarks. The off-policy discriminator failure (Figure 6) provides a clean mechanistic explanation for why GAD works. However, the narrow experimental scope (single dataset, single teacher, no scaling analysis, no variance reporting, no white-box comparison) means the paper demonstrates a strong proof of concept rather than a comprehensively validated method ready for deployment across diverse settings.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Claims

The assumption or constraint. The paper does not explicitly discuss a "difficulty estimation" step, but there is an analogous hidden cost: warmup. The GAD training pipeline requires a full-epoch warmup phase where the generator undergoes SeqKD (supervised fine-tuning on teacher responses) and the discriminator is pre-trained, consuming approximately one-third of the total training budget (1 warmup epoch + 2 GAD epochs). The paper acknowledges this implicitly through the experimental setup (Section 3.1: "All models are trained for 3 epochs"), but the headline comparisons treat GAD's 3-epoch total as matched to SeqKD's 3-epoch total — GAD does not get more data or more steps than SeqKD. However, GAD's 3 epochs include the warmup epoch in addition to the 2 adversarial epochs, meaning the adversarial component only gets 2/3 of the total budget. The paper never ablates whether SeqKD trained for 3 epochs vs. GAD (1 warmup + 2 GAD) represents a fair comparison, or whether SeqKD would benefit from additional epochs (e.g., 4 or 5) to match GAD's total compute.

The consequence. If GAD's warmup epoch produces most of the gains and the adversarial phase contributes marginally, then a straightforward extension of SeqKD (more epochs, or a different SFT recipe) might match or approach GAD performance without the additional complexity of adversarial training. The paper's own ablation hints at this possibility: removing generator warmup (Table 3) drops GAD from 50.8 to 49.7 on LMSYS-Chat — still above SeqKD's 49.2, but the gap shrinks from +1.6 to +0.5. This suggests the warmup contributes meaningfully to the final performance, but the paper does not isolate how much of GAD's advantage comes from the warmup (which is just SeqKD) vs. the adversarial phase.

What evidence exists in the paper. Table 3 provides the only decomposition: full GAD = 50.8, GAD without generator warmup = 49.7, SeqKD = 49.2. The drop from removing warmup (50.8 → 49.7, -1.1) is comparable to the total GAD-SeqKD gap (50.8 − 49.2 = 1.6), indicating warmup accounts for a large fraction of the gain. The paper does not report a "SeqKD × 3 epochs" baseline that matches GAD's total training steps without adversarial training. Section 3.1 states SeqKD was trained for 3 epochs, which is matched to GAD's total (1 warmup + 2 GAD), but this comparison gives SeqKD the same number of passes over the data as GAD's entire pipeline. It does not test whether giving SeqKD the same wall-clock training budget (which would be more epochs since SeqKD doesn't require discriminator forward/backward passes or GRPO group sampling) closes the gap.

Mitigation status. Not addressed. The paper treats the matched-epoch comparison as fair and does not explore whether GAD's advantage is robust to giving SeqKD additional training budget. A practitioner wanting to assess the cost-benefit tradeoff of adopting GAD over simply running SeqKD longer would find no guidance in the paper.

---

Single Training Dataset Across All Experiments

The assumption or constraint. All models are trained on a single dataset: 200K samples from LMSYS-Chat-1M-Clean, a dataset derived from conversational data on the Chatbot Arena platform. The paper states this explicitly (Section 3.1): "For the following experiments, we use LMSYS-Chat-1M-Clean, a clean version of the LMSYS-Chat-1M dataset. The dataset is derived from high-quality conversational data collected via the Chatbot Arena platform." Every result in the paper — in-distribution, out-of-distribution, human evaluation — uses models trained on this single data distribution. The teacher (GPT-5-Chat) generates responses to these specific prompts, and the distillation dataset is entirely defined by this prompt-response pairing.

The consequence. The paper demonstrates that GAD outperforms SeqKD for distilling conversational ability from one specific teacher on one specific data distribution. It provides no evidence that GAD works for distilling other capabilities — coding, mathematical reasoning, factual knowledge, creative writing, or domain-specific expertise — from other teachers or on other data distributions. The finding that SeqKD overfits to local patterns (Figure 4) and GAD captures "global stylistic characteristics" may be specific to conversational data, where style and fluency dominate over factual accuracy or reasoning correctness. In a code distillation setting, surface-level pattern memorization (e.g., copying variable naming conventions) might be less harmful than in conversation, and the mode-seeking vs. mode-covering distinction might play out differently.

The out-of-distribution benchmarks (Dolly, SelfInst, Vicuna) test generalization to other conversational distributions, not to other task types. The fact that SeqKD sometimes degrades on these benchmarks (e.g., Qwen2.5-14B-Instruct on Dolly: 49.1 → 48.2) and GAD improves (49.1 → 50.4) is evidence for conversational-domain generalization, but does not address the broader question of cross-task transfer.

What evidence exists in the paper. None. The paper does not include experiments on any non-conversational dataset for either training or evaluation. The LMSYS-Chat-1M-Clean dataset is described as "high-quality conversational data" (Section 3.1), and all evaluation benchmarks (LMSYS-Chat test set, Dolly, SelfInst, Vicuna) are instruction-following conversational benchmarks. There is no math benchmark (e.g., GSM8K, MATH), no code benchmark (e.g., HumanEval, MBPP), no factual knowledge benchmark (e.g., MMLU), and no reasoning benchmark. The paper's title and abstract claim GAD is "a promising and effective paradigm for black-box LLM distillation" without qualification about task scope, but the evidence is confined to a single task family.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, nor does it suggest that future work should test GAD on non-conversational domains. The out-of-distribution experiments are presented as evidence of generalization, but they test generalization across conversational datasets, not across task types. A practitioner wanting to use GAD for distilling coding or reasoning capabilities would need to extrapolate from zero evidence.

---

No Scaling Analysis for Data Quantity or Training Duration

The assumption or constraint. All experiments use exactly 200K training examples and exactly 3 epochs (approximately 2,400 optimization steps). The paper states these choices in Section 3.1: "For training data, we sample 200K samples from LMSYS-Chat-1M-Clean" and "All models are trained for 3 epochs." There is no experiment varying the training data size (e.g., 50K, 100K, 500K, 1M examples) or the number of training epochs (e.g., 1 epoch GAD after warmup, 4 epochs GAD, etc.). The paper also does not explore how GAD's advantage over SeqKD changes as either budget increases.

The consequence. This creates two major unknowns for practitioners. First, data efficiency: is GAD more data-efficient than SeqKD? If GAD achieves the same performance with half the training data, that would be a strong practical argument for adoption, since collecting teacher responses from API models is expensive (both financially and in terms of rate limits). Conversely, if SeqKD catches up to GAD with more data (e.g., 500K examples), then GAD's advantage is diminished in data-rich regimes. The paper provides no evidence either way.

Second, training stability over longer horizons: adversarial training is known to be prone to instability, mode collapse, and divergence over extended training. The paper's Figure 6 shows stability over 2,000+ steps, but this is within the 2-epoch GAD phase. Would stability hold over 5 epochs? 10 epochs? Does the discriminator-generator equilibrium eventually break down as the student gets too good at fooling the discriminator, or as the discriminator develops overly sharp decision boundaries? The off-policy discriminator fails within ~300 steps (Figure 6), demonstrating that training dynamics are sensitive to the reward signal. Whether the on-policy discriminator would exhibit similar failure modes over longer training is unknown.

What evidence exists in the paper. The training duration is fixed at 3 epochs for all experiments. Figure 6 shows training dynamics (response length) over approximately 2,000 steps for the 2-epoch GAD phase (1 warmup epoch + 2 GAD epochs; the warmup epoch is ~781 steps, so the GAD phase is ~1,562 steps). The response length remains stable throughout, but this covers only the regime tested — there is no evidence for or against stability beyond this horizon. The paper does not ablate training duration or data quantity. Checkpoint selection (Section 3.1) — "We select the checkpoint that achieved the highest GPT-4o score and whose response length is within an acceptable range" — implies that performance varies across checkpoints and that some checkpoints may produce inappropriately long responses, but these dynamics are not analyzed.

Mitigation status. Not addressed. The paper treats 200K examples × 3 epochs as a fixed recipe and does not discuss how the method's performance or stability might change with different data or training budgets. The checkpoint selection procedure hints at late-training variation but does not systematically explore it. A practitioner wanting to scale GAD to larger datasets or longer training would need to discover the stability properties empirically, without guidance from the paper.

---

Single Teacher Model and Single Evaluation Protocol

The assumption or constraint. The main experiments use exactly one teacher model: GPT-5-Chat. The paper states this in Section 3.1: "We adopt GPT-5-Chat as the teacher model. It is a closed-source chat model ranked ninth on the Chatbot Text Arena leaderboard at the time of writing." A secondary experiment (Table 7) uses Qwen2.5-14B-Instruct as a teacher for Llama students, but this is a single additional configuration designed to test the tokenizer-incompatibility scenario rather than to systematically vary teacher characteristics. The evaluation protocol is also singular: GPT-4o as judge using the specific prompt template from Gu et al. (2024) (Figure 8), with human evaluation serving as a corroboration rather than an independent assessment on a different rubric.

The consequence. It is unknown whether GAD's advantage over SeqKD depends on specific properties of the GPT-5-Chat teacher — its output distribution, its response style, the gap between its capabilities and the students', or its specific weaknesses. Different teachers with different characteristics might produce different results. For example:

• A teacher with very different stylistic properties from the student might cause the discriminator to focus on superficial stylistic features, reducing the quality of the adversarial signal.
• A teacher with capabilities far beyond the student might create an impossibly large gap for the discriminator to bridge, causing the adversarial game to collapse (similar to the "without generator warmup" failure mode in Table 3).
• A teacher that is closer in capability to the student might make the discriminator's task too easy, providing a weak gradient signal.

The Qwen2.5-14B-Instruct teacher experiment provides partial evidence: GAD still outperforms SeqKD (+0.6 for both Llama students on LMSYS-Chat), but the absolute student scores are lower, reflecting the weaker teacher (50.0 vs. GPT-5-Chat's 51.7). However, the gap between GAD and SeqKD is also smaller (+0.6 vs. +1.4–1.6 in the main experiments), which could indicate that GAD's advantage shrinks with weaker teachers, or could simply reflect the different student model family — the experiment confounds teacher quality with model architecture.

Similarly, the evaluation is entirely GPT-4o-based with a single judge prompt. Different judge models, different judge prompts, or different evaluation rubrics (e.g., separate scores for helpfulness, relevance, accuracy, detail, rather than a single overall score) might rank GAD and SeqKD differently. The human evaluation (Figure 3) provides partial independence from the automatic judge, but it uses the same LMSYS-Chat test prompts and the same high-level evaluation criteria (helpfulness, relevance, accuracy, detail), so it is not a truly independent assessment on a different dimension of quality.

What evidence exists in the paper. The main experiments use GPT-5-Chat exclusively as teacher. Table 7 tests a single alternative teacher (Qwen2.5-14B-Instruct) but only with Llama students, and the experiment is framed as a tokenizer-compatibility test rather than a teacher-robustness test. The evaluation uses a single GPT-4o prompt (Figure 8) across all experiments. The paper does not report results with different judge models, different judge prompts, or alternative evaluation methodologies (e.g., benchmark accuracy on structured tasks, pairwise preference rates from a different judge, task-specific metrics).

Mitigation status. Partially addressed through the Qwen2.5 teacher experiment (Table 7), which demonstrates GAD works with at least one alternative teacher. However, this is a single additional datapoint and the teacher is chosen for its tokenizer incompatibility rather than to systematically vary teacher characteristics. The paper does not acknowledge teacher-specificity or evaluation-specificity as limitations, nor does it suggest testing with a broader range of teachers or evaluation protocols. A practitioner wanting to use GAD with a different teacher model (e.g., Claude, Gemini, a different GPT variant) would need to assume transferability of the results.

---

Response Length Inflation and Lack of Length-Control Mechanisms

The assumption or constraint. GAD optimizes the generator to maximize the discriminator's scalar score with no explicit constraint on response length or verbosity. The KL penalty with weight $\beta = 0.001$ (Appendix A.2) provides some regularization toward the reference policy's behavior, but it is not a direct length penalty. The paper's checkpoint selection procedure (Section 3.1) reveals that length control is an operational concern: "We select the checkpoint that achieved the highest GPT-4o score and whose response length is within an acceptable range for each experiment." This implies that some checkpoints produce responses with unacceptable lengths, and manual filtering is required to obtain the reported results.

The consequence. The off-policy discriminator experiment (Figure 6) demonstrates that without co-evolution, the generator discovers that producing very long responses (~1,300 tokens) earns high discriminator scores. This establishes that the discriminator's scores are not inherently length-calibrated — the discriminator can be fooled by verbosity. The on-policy discriminator avoids this specific failure mode (response lengths remain stable at ~500–600 tokens), but Table 6 reveals that GAD students consistently produce longer responses than SeqKD students across all model sizes and datasets. For example, Qwen2.5-14B-Instruct + GAD on LMSYS-Chat produces responses averaging 438.9 tokens, compared to 319.3 for SeqKD, 322.1 for the pre-distillation model, and 329.1 for the teacher. On out-of-distribution benchmarks, the length gap is even larger: Dolly GAD = 262.6 vs. SeqKD = 151.2 vs. before distill = 201.6 vs. teacher = 148.5.

This creates a confounding factor: are GAD's higher GPT-4o scores attributable to better response quality, or partly to increased verbosity? The GPT-4o judge prompt (Figure 8) asks for ratings on "helpfulness, relevance, accuracy, and level of detail," and longer responses may naturally score higher on "level of detail" even if the extra length is padding rather than substance. The paper does not control for response length in the evaluation (e.g., by truncating responses to a fixed token budget before scoring, or by asking the judge to penalize unnecessary verbosity). The human evaluation (Figure 3) uses side-by-side comparison between GAD and baseline responses, where length differences would be visible to annotators and could influence their judgments. The win rates (52–68% for GAD) could partially reflect a preference for more detailed responses rather than higher-quality responses.

Additionally, the manual checkpoint filtering by response length is a post-hoc selection criterion that is not available in a deployment scenario where the "correct" length is unknown and task-dependent. The paper provides no mechanism for controlling response length during training (e.g., a length penalty in the reward, length-normalized scoring, or a multi-objective reward that balances quality and conciseness). A practitioner deploying GAD would need to either accept the length distribution that emerges from training, or develop their own length-control mechanisms without guidance from the paper.

What evidence exists in the paper. Table 6 reports response lengths alongside GPT-4o scores for every model configuration. The GAD response length inflation is consistent and substantial — GAD students produce 20–40% longer responses than SeqKD students in most settings. Figure 6 shows the extreme case of length explosion with the off-policy discriminator (up to ~1,300 tokens), and the stability of the on-policy discriminator (~500–600 tokens). The paper does not report any analysis of whether the GPT-4o score advantage persists after controlling for response length. Section 3.3 interprets the N-gram overlap results as evidence that "GAD better captures the teacher's global stylistic characteristics," but does not discuss whether length inflation is part of these "global stylistic characteristics" or an artifact of the adversarial optimization.

Mitigation status. Partially addressed through manual checkpoint filtering. The paper acknowledges that length is a selection criterion (Section 3.1), which indicates awareness of the issue. However, the filtering is applied after training as a model selection step rather than during training as a control mechanism. The paper does not propose or evaluate any method for incorporating length constraints into the GAD objective, nor does it demonstrate that GAD's quality advantage holds under length-matched comparison. A practitioner concerned about response verbosity would find the paper's handling of this issue incomplete — they would know that length inflation occurs, that it can be filtered post-hoc, but not how to prevent it or whether the reported quality gains are independent of it.

---

No Comparison to White-Box Distillation Methods to Quantify the "Black-Box Tax"

The assumption or constraint. The paper's central premise is that white-box distillation methods are inapplicable when the teacher is a proprietary API model, and GAD provides an effective black-box alternative. However, the paper never compares GAD against a white-box distillation method (e.g., MiniLLM; Gu et al., 2024) on the same teacher-student pair to quantify how much performance is lost due to the black-box constraint. The Qwen2.5-14B-Instruct teacher experiment (Table 7) presents itself as a tokenizer-incompatibility scenario where white-box methods are inapplicable, but the paper does not attempt to align tokenizers or apply a white-box method as a comparison point. Throughout the paper, the only baselines are "Before Distill." and "SeqKD" — both black-box methods.

The consequence. A practitioner deciding whether to use GAD needs to know not just that GAD beats SeqKD, but how close GAD comes to the white-box ceiling. If a white-box method (requiring full teacher access) achieves, say, 53.0 on LMSYS-Chat for the 14B student while GAD achieves 52.1, then GAD captures ~90% of the achievable gain, making it a strong black-box method. If the white-box ceiling is 55.0, then GAD captures ~50% — still better than SeqKD, but leaving substantial performance on the table that might justify alternative approaches (e.g., using a different teacher that does provide white-box access, or investing in better black-box techniques). Without this comparison, the paper's claim that GAD is "an effective and robust solution for black-box LLM distillation" (Section 5) is relative only to other black-box methods, not to the distillation problem as a whole.

The tokenizer-incompatibility experiment (Table 7) provides a partial glimpse: the Qwen2.5-14B-Instruct teacher scores 50.0 on LMSYS-Chat. GAD's Llama-3.1-8B-Instruct student scores 49.6. If a hypothetical white-box method could achieve, say, 50.5 on this teacher-student pair (the student surpassing the teacher, as GAD does with GPT-5-Chat), then GAD's 49.6 represents a ~0.9 point black-box tax. But this is pure speculation — the paper provides no white-box baseline to anchor such estimates.

What evidence exists in the paper. None. The paper's experimental sections (3.1, 3.2, Appendix B) compare GAD exclusively to the instruct model before distillation and to SeqKD. No white-box method is implemented, cited as a comparison point, or discussed as a ceiling. The related work section (Section 4) describes white-box methods (forward KLD, reverse KLD, hidden state distillation) but does not position GAD's performance relative to what these methods achieve on comparable tasks. The Qwen2.5 teacher experiment (Table 7) is framed as a tokenizer-compatibility test and does not include a white-box comparison despite using an open-source teacher where white-box access is technically possible (the tokenizer barrier could potentially be overcome with tokenizer alignment or sequence-level distillation techniques).

Mitigation status. Not addressed. The paper does not acknowledge the absence of white-box baselines as a limitation, nor does it suggest that future work should quantify the performance gap between black-box and white-box distillation on the same tasks. The paper's framing — "GAD is effective for black-box distillation" — is internally consistent, but a practitioner reading the paper would not know whether GAD extracts 60%, 80%, or 95% of the value that white-box methods could achieve. This limits the paper's ability to guide decisions about whether to use GAD or to seek alternatives (e.g., training with a non-proprietary teacher that allows white-box access, or developing hybrid approaches that use limited teacher information).

7. Implications and Future Directions

How This Work Changes the Landscape

From imitation to adversarial co-adaptation: a reframing of black-box distillation.

GAD's most significant conceptual contribution is not a particular architecture or loss function, but rather the reframing of black-box distillation from an imitation problem (mimic the teacher's text) to an adversarial co-adaptation problem (compete against a critic that learns to spot differences). This shifts the fundamental question from "how accurately can we reproduce the teacher's outputs?" to "how can we construct a reliable, adaptive quality signal without access to the teacher's internals?" The distinction is substantive, not cosmetic. Imitation methods like SeqKD treat the teacher's responses as a fixed target distribution to be matched — the student optimizes cross-entropy against a static dataset. GAD treats the teacher's quality standard as a moving target defined through adversarial interaction: the student generates responses, the discriminator learns to distinguish them from the teacher's, and the student is optimized to close that gap. The quality signal is constructed through competition rather than given through demonstration.

This reframing matters because it reveals a previously hidden degree of freedom in black-box distillation. The field had largely accepted that without teacher logits, the best one could do was supervised fine-tuning on teacher text — a straightforward but fundamentally limited approach. GAD demonstrates that black-box distillation can access richer learning dynamics — on-policy exploration, mode-seeking behavior, and adaptive quality assessment — that were previously thought to require white-box access. The paper does not claim to invent on-policy distillation (it credits MiniLLM, Agarwal et al., and Lu for that contribution), but it does claim — and demonstrate — that the mechanism for achieving on-policy learning can be decoupled from teacher probability access.

Resolving the contradiction: why on-policy learning was "impossible" in black-box settings.

The paper resolves a specific tension in the distillation literature. Prior work had established two seemingly incompatible findings: (1) on-policy learning substantially improves distillation quality in white-box settings (Gu et al., 2024; Agarwal et al., 2024), and (2) on-policy learning is infeasible in black-box settings because the teacher provides no probability signal to evaluate student-generated responses (the very premise of the black-box constraint). These findings together implied a hard ceiling for black-box distillation: no matter how clever the SFT recipe, black-box methods would always lag white-box methods by the gap between off-policy and on-policy learning. GAD resolves this by showing that the teacher's probability signal is not the only way to achieve on-policy feedback — a co-evolving discriminator provides a functionally equivalent signal constructed entirely from text.

The resolution is not just theoretical. The paper's results demonstrate that GAD closes a substantial portion of the presumed black-box-white-box gap. While the paper does not directly compare to white-box methods (a limitation discussed in Section 6), the fact that a 14B student trained with GAD surpasses its teacher on in-distribution evaluation (52.1 vs. 51.7 on LMSYS-Chat, Table 2) while SeqKD falls short (50.6) suggests GAD is achieving dynamics typically associated with on-policy white-box methods — the student is not just imitating but improving relative to the teacher on the training distribution.

Redirecting research attention: verifier quality as the bottleneck, not distillation algorithms.

One of the paper's most important landscape-changing findings is that discriminator calibration, not search or sampling sophistication, is the primary bottleneck for black-box distillation. The off-policy discriminator experiment (Figure 6) provides a clean diagnostic: a discriminator that becomes miscalibrated due to distributional drift causes catastrophic reward hacking within ~300 training steps, regardless of the generator's optimization algorithm. This redirects research attention away from developing more sophisticated distillation objectives and toward building more robust quality assessment mechanisms.

This finding has a direct parallel in the RLHF literature, where reward model over-optimization is a well-documented failure mode (Skalse et al., 2022). But GAD's contribution is to show that the same dynamics govern distillation — and more importantly, that co-evolution of the quality signal with the policy is a sufficient (and practical) solution. Prior work had treated reward model staleness as something to mitigate post-hoc (through early stopping, KL regularization, or reward model ensembles). GAD's discriminator co-evolution treats it as something to prevent architecturally, by making the reward model an active participant in training rather than a static oracle.

This insight makes certain research directions more attractive and others less so:

• More attractive: Improving discriminator training (better architectures, uncertainty calibration, adversarial robustness to student exploitation), understanding the discriminator-generator equilibrium dynamics (when does it converge? when does it cycle or diverge?), and developing lightweight approaches to maintain discriminator calibration (e.g., partial updates, replay buffers, ensemble disagreement signals).
• Less attractive: Developing increasingly complex SeqKD variants (multi-stage SFT, curriculum learning on teacher data, data filtering heuristics) that do not address the fundamental off-policy limitation. The paper's results suggest these are optimizing within a paradigm that has a hard ceiling: SeqKD on 14B-Qwen2.5 reaches only 50.6 on LMSYS-Chat even with matched training budget, while GAD reaches 52.1.

Establishing the warmup phase as a principled initialization strategy.

The paper's systematic ablation of warmup (Table 3) establishes something that had been treated as an ad-hoc engineering trick in prior adversarial text training work: joint warmup is not just helpful, but structurally necessary for adversarial LLM training to succeed. The finding that discriminator warmup is more critical than generator warmup (removing discriminator warmup collapses performance to SeqKD levels; removing generator warmup only partially reduces gains) provides a specific diagnostic: adversarial training of language models fails primarily because the discriminator cannot provide informative feedback early in training, not because the generator starts too far from the target distribution.

This has implications beyond GAD. Any future method involving adversarial training of LLMs — whether for distillation, alignment, style transfer, or controllable generation — now has a clear initialization requirement: invest heavily in discriminator pretraining before beginning adversarial optimization. The paper's specific recipe (1 epoch of joint warmup, first 10 steps discriminator-only) provides a concrete starting point, and the diagnostic framework (test performance with and without discriminator warmup to identify imbalance failures) gives researchers a tool for debugging their own adversarial training setups.

The mode-seeking interpretation: connecting GAN dynamics to distillation theory.

The toy experiment (Figure 5) and N-gram overlap analysis (Figure 4) connect GAD's adversarial dynamics to the established theoretical framework of mode-seeking vs. mode-covering in distillation. This connection is more than an interesting observation — it provides a principled explanation for GAD's generalization advantage that links empirical results to theory. Mode-seeking (GAD) concentrates the student's limited representational capacity on behaviors it can credibly execute, producing better peak performance and more robust transfer to new distributions. Mode-covering (SeqKD) spreads capacity thinly across all teacher behaviors, including idiosyncratic ones the student cannot reproduce well, producing surface-level imitation (high N-gram overlap) without genuine capability transfer.

This theoretical grounding makes GAD's out-of-distribution results (Table 2, Figure 1 right) interpretable rather than mysterious. SeqKD's negative transfer on Dolly (14B: 49.1 → 48.2) and zero improvement on SelfInst (49.4 → 49.4) are not random noise — they are predictable consequences of mode-covering: the student has memorized surface patterns from the LMSYS-Chat distribution that do not generalize, and the dilution of capacity across too many modes means it never developed robust underlying capabilities. GAD's consistent out-of-distribution improvements (+1.3 to +2.3 across benchmarks for the 14B student) reflect mode-seeking: the student has focused on learning what makes a response good (as judged by the discriminator's evolving standard) rather than what the teacher's responses look like, and that quality-assessment capability transfers across distributions.

---

Follow-Up Research This Work Enables

Scaling laws for adversarial distillation: how does GAD's advantage change with data quantity and model size?

The paper fixes training data at 200K examples and model sizes at 3B, 7B, 8B, and 14B, but never systematically varies data quantity. A natural scaling study would train GAD and SeqKD on dataset sizes from 10K to 1M examples (in powers of 2), measuring the performance gap at each scale. The central question: is GAD more data-efficient than SeqKD (larger advantage at small data sizes, gap closes with more data), or does GAD's advantage persist or even grow with scale? The toy experiment (Figure 5) suggests GAD is more efficient with limited capacity (a single Gaussian captures the teacher's dominant modes better than mode-covering), but whether this translates to data efficiency in LLM distillation is unknown. A related study would scale model size from 1B to 14B+ while measuring the GAD-SeqKD performance gap — does GAD provide disproportionately larger gains for smaller models (where capacity constraints make mode-seeking more important), or for larger models (where the discriminator has more capacity to provide nuanced feedback)?

Cross-domain distillation: testing GAD on code, math, and factual knowledge.

The paper's exclusive use of conversational data (LMSYS-Chat-1M-Clean) leaves open the question of domain generality. A strong follow-up would replicate the GAD-vs-SeqKD comparison on distinct task families: code generation (training on a programming dataset like CodeContests or APPS with a strong code teacher, evaluating on HumanEval and MBPP), mathematical reasoning (training on MATH or GSM8K with a reasoning teacher, evaluating on held-out math benchmarks), and factual knowledge (training on a QA dataset like Natural Questions with a knowledgeable teacher, evaluating on MMLU or similar). The key measurement: does GAD's advantage over SeqKD persist across these domains, and does the advantage magnitude correlate with any domain property (e.g., larger advantage for "stylistic" tasks like conversation and creative writing, smaller advantage for "objective" tasks like math where surface patterns are less informative)? The paper's mode-seeking interpretation predicts GAD should help most when the teacher's distribution has many modes the student cannot all represent — conversational data likely has more such modes than structured math or code problems. A finding that GAD's advantage is domain-dependent would refine the theoretical understanding and provide practical guidance on when to adopt GAD vs. stick with simpler SeqKD.

White-box ceiling quantification: how much performance is lost by the black-box constraint?

The paper never compares GAD to a white-box distillation method (e.g., MiniLLM with reverse KLD) on the same teacher-student pair, leaving the "black-box tax" unquantified. A critical follow-up would use an open-source teacher with compatible tokenizer (e.g., Qwen2.5-14B-Instruct distilling into Qwen2.5-7B-Instruct) and train three students: SeqKD (black-box baseline), GAD (black-box adversarial), and MiniLLM or on-policy reverse KLD (white-box, using full teacher logits). The comparison isolates the performance cost of the black-box constraint: white-box method minus GAD is the "remaining gap" that better black-box methods might close; GAD minus SeqKD is the gain from adversarial training over SFT within the black-box regime. The paper's Qwen2.5 teacher experiment (Table 7) uses tokenizer-incompatible students (Llama family), which prevents this comparison — a within-family experiment (Qwen2.5 teacher → Qwen2.5 student) would provide the needed evidence. If the white-box-GAD gap is small (<0.5 GPT-4o points), black-box distillation is essentially solved for practical purposes. If large (>2–3 points), then further black-box method development is warranted, potentially combining GAD's discriminator approach with other techniques (e.g., synthetic teacher probability estimation, multi-teacher consensus).

Dynamic difficulty-aware or prompt-aware discriminator specialization.

GAD uses a single discriminator to assess all prompts, but the LMSYS-Chat dataset contains diverse prompt types (brainstorming, factual questions, creative writing, coding, roleplay). A single discriminator may learn an average quality standard that is suboptimal for specific prompt categories. A follow-up would train prompt-conditioned discriminators: either by conditioning the discriminator on a learned prompt embedding (so it can apply different quality standards to different prompt types), or by clustering prompts by type and training per-cluster discriminators that specialize. If prompt difficulty or type significantly affects what constitutes "teacher-like" behavior, specialized discriminators could provide more precise feedback. The evaluation would compare GPT-4o scores and human preference rates against the uniform-discriminator GAD baseline, with a breakdown by prompt category. The paper's difficulty-aware test-time compute findings (not in this paper, but conceptually analogous) suggest that adaptive quality standards could improve distillation, but this remains untested for the adversarial distillation setting.

Combining GAD with iterative self-improvement: student-as-next-teacher loops.

The paper demonstrates that GAD-trained students can approach or surpass their teacher on in-distribution evaluation (52.1 vs. 51.7 for 14B student vs. GPT-5 on LMSYS-Chat, Table 2). This opens a natural question: can a GAD student serve as the teacher for the next iteration of distillation? A follow-up would implement a multi-round distillation loop: (1) distill student₁ from teacher using GAD, (2) use student₁ as the teacher to distill student₂ via GAD, (3) evaluate whether student₂ improves over student₁ or plateaus/degrades. This tests whether GAD's adversarial process amplifies teacher capabilities (the discriminator forces the student to find genuinely better behaviors, not just imitate) or merely transfers them. If iterative GAD produces compounding improvements, it becomes a self-improvement mechanism analogous to self-play in game AI (AlphaGo-style). If it plateaus or degrades, this reveals a fundamental limitation: the discriminator's quality standard is bounded by the initial teacher's behavior, and GAD cannot generate new capabilities beyond what the discriminator can recognize. The paper's finding that GAD students surpass the teacher on some metrics hints at amplification, but iterative experiments would test whether the effect compounds.

Understanding and mitigating the discriminator-generator equilibrium failure modes over extended training.

The paper demonstrates stability over ~2,000 GAD training steps (Figure 6), but does not explore whether this stability holds over longer horizons or at larger scales. Adversarial training is notoriously prone to cycling, mode collapse, or equilibrium breakdown over extended training. A systematic stability study would train GAD for 10, 20, and 50 epochs (rather than the default 2) while tracking: (a) discriminator accuracy (can it still distinguish student from teacher?), (b) generator reward (are discriminator scores for student responses approaching teacher scores?), (c) response length and diversity metrics (is the generator collapsing to a narrow set of high-scoring responses?), and (d) GPT-4o evaluation scores (does quality plateau or decline?). The goal is to identify the failure modes that eventually limit GAD training and to test interventions: adaptive KL penalty scheduling (increasing regularization as the generator improves), discriminator learning rate decay (slowing discriminator updates to prevent overfitting to recent student behavior), or periodic discriminator reset (re-initializing the discriminator head to prevent accumulated bias). The paper's checkpoint filtering procedure (Section 3.1: selecting the checkpoint with "highest GPT-4o score and whose response length is within an acceptable range") hints that quality fluctuates during training and that some checkpoints produce undesirable behavior — a systematic study would map this fluctuation and determine whether it's noise (random variation) or a systematic dynamic (e.g., the generator temporarily overpowers the discriminator before being pulled back).

---

Practical Applications and Downstream Use Cases

API-only teacher distillation for on-device or edge deployment.

The most direct use case is distilling proprietary API models (GPT-5, Claude, Gemini) into smaller open-source models for deployment in resource-constrained settings. The paper's parameter-class jump — a 3B GAD student matching the performance of a 7B SeqKD student (48.9 vs. 48.7 on LMSYS-Chat, Table 2) — has direct practical implications: a model with less than half the parameters achieves equivalent conversational quality, reducing memory footprint, inference latency, and serving costs by roughly 2× while maintaining the same user-facing quality. For on-device deployment (smartphones, laptops, embedded systems) where memory is the binding constraint, GAD enables smaller models to punch above their weight class. The 14B student approaching GPT-5 teacher performance (52.1 vs. 51.7, Table 2) further demonstrates that GAD can extract nearly all of a proprietary teacher's capability into a deployable open-source model — the holy grail of API-based distillation. The concrete workflow for practitioners: (1) query the proprietary API with your domain-specific prompts to collect teacher responses, (2) run GAD training using the open-source verl framework with the hyperparameters provided in Appendix A.2, (3) deploy the resulting student model without any dependency on the proprietary API.

Cost-efficient batch inference with smaller models and smarter distillation.

For organizations running large-scale batch inference (document processing, customer support response generation, content moderation, data labeling), the tradeoff between model quality and inference cost is central. The paper's results suggest a specific strategy: rather than using a single large model for all queries, distill a family of smaller GAD-trained models (3B, 7B, 14B) and route queries by complexity, using the smallest model that maintains acceptable quality. The 3B GAD student scores 48.9 on LMSYS-Chat while a non-distilled 7B scores 48.7 (Table 2), meaning the 3B model achieves 7B-level quality at roughly 2.3× lower inference cost and with less than half the GPU memory. For a batch pipeline processing millions of queries daily, the cumulative savings in GPU-hours would be substantial. The paper's finding that GAD improves out-of-distribution generalization (Table 2, Figure 1 right) is particularly relevant: batch pipelines often process queries from distributions that differ from the training data, and SeqKD's negative transfer on Dolly (−0.9 points for 14B, Table 2) would silently degrade production quality, while GAD's consistent improvements (+1.3 to +2.3 across benchmarks) would provide more robust performance across query variation.

Training data generation for self-improvement pipelines.

When using LLMs to generate training data for fine-tuning or distillation (a common pattern in the STaR, ReST, and self-instruct families of methods), the quality of the generated data directly determines the quality of the resulting model. The paper's results suggest that GAD-trained students produce responses that are not just teacher-like but sometimes better than the teacher's on the training distribution (52.1 vs. 51.7, Table 2). This means a GAD student could serve as a data generator that produces higher-quality training examples than the original teacher — a counterintuitive but empirically supported possibility. The practical pipeline: (1) use GPT-5 (or another strong teacher) to generate initial training responses for a set of prompts, (2) train a GAD student on this data, (3) use the GAD student to generate new responses to a larger set of prompts (including out-of-distribution prompts where the original teacher's responses are unavailable or expensive to obtain), (4) fine-tune downstream models or the next generation of students on this expanded, GAD-generated dataset. The out-of-distribution generalization advantage (Figure 1, right) suggests the GAD student would generate higher-quality data on novel prompts than an equivalently-sized SeqKD student, improving the downstream model's robustness. The paper does not test this pipeline directly, but the results provide the necessary ingredients: a distillation method that produces students better than their teacher on in-distribution evaluation, with superior generalization to new distributions.

Tokenizer-agnostic distillation across model families.

The paper's Qwen2.5 teacher → Llama student experiment (Table 7) demonstrates that GAD works when the teacher and student use incompatible tokenizers — a scenario where standard white-box KL-divergence methods are inapplicable even if the teacher is open-source. This has immediate practical relevance for practitioners who want to distill capabilities from one model family into another (e.g., Qwen into Llama, Mistral into Gemma, DeepSeek into Phi) without dealing with tokenizer alignment, vocabulary mapping, or embedding projection. The 8B Llama student + GAD scores 49.6 on LMSYS-Chat with a Qwen2.5-14B teacher scoring 50.0, nearly closing the gap to the teacher despite the tokenizer barrier. For organizations that have standardized on a particular model architecture but want to incorporate capabilities from models with different tokenizers, GAD provides a path forward that does not require white-box access or architectural modifications. The concrete workflow: collect text responses from the desired teacher (any model family, any tokenizer), run GAD training on the target student architecture (different model family, different tokenizer), and the discriminator — which operates on text, not token IDs — handles the cross-family transfer automatically.

---

When to Prefer This Method

The paper does not articulate an explicit, conditional decision framework for choosing GAD over alternatives (beyond the generic claim that GAD "consistently surpasses" SeqKD). The experimental results, however, imply several conditions under which GAD is most likely to provide substantial benefits:

• When the teacher is a black-box API model and white-box distillation is infeasible — this is GAD's primary design target and the setting where it provides the clearest value over the SeqKD fallback.
• When out-of-distribution generalization matters — the paper's strongest advantage over SeqKD appears on Dolly, SelfInst, and Vicuna (Table 2), where SeqKD frequently shows zero or negative improvement and GAD consistently improves (+1–2 points). For deployments with diverse, evolving query distributions, GAD's robustness advantage is likely more important than the in-distribution gains.
• When model size is severely constrained — the parameter-class jump effect (3B GAD ≈ 7B SeqKD on LMSYS-Chat, Table 2) suggests GAD extracts disproportionately more capability from smaller models, making it especially valuable for on-device or low-resource deployments.
• When the student and teacher use incompatible tokenizers — Table 7 demonstrates GAD's tokenizer-agnostic property, which is not available with standard white-box methods and provides a practical solution for cross-family distillation.

The paper does not provide evidence that GAD is preferable when:

• Training data is extremely large (>1M examples) — the lack of scaling analysis leaves this unknown.
• Extreme training efficiency is required — GAD requires discriminator training and GRPO group sampling, which adds computational overhead relative to SeqKD; the paper does not compare wall-clock training time or total FLOPs.
• The teacher provides white-box access — the paper never compares GAD to white-box methods, so there is no evidence that GAD should be preferred when white-box methods are applicable.