Masking Teacher and Reinforcing Student for Distilling Vision-Language Models

🎯 Pitch

Distilling a huge vision-language model into a tiny one usually fails because the student cannot digest the teacher's full complexity—Masters solves this by first making the teacher "dumber" through progressive weight masking, then slowly restoring it, letting the student learn in manageable steps. The resulting 2B–3B models match or beat many large 7B–8B rivals, and a masked teacher with offline RL rewards works even better than standard distillation from a 38B giant.

1. Executive Summary

This paper introduces Masters (Masking teacher and reinforcing student), a mask-progressive reinforcement learning distillation framework for transferring knowledge from large vision–language models to compact student VLMs. Evaluated across diverse VLM benchmarks (AI2D, ChartQA, MathVista, MMB, MM-Vet, MMMU, MMMU-Pro, MMStar, BLINK, SEED-Bench, SEED-Bench-2-Plus, RealWorldQA) using Qwen2.5-VL, Qwen3-VL, InternVL3, and InternVL3.5 model families, Masters combines two core mechanisms: mask-progressive distillation (temporarily masking non-dominant teacher weights by magnitude and gradually restoring them, e.g., from a masking ratio of 0.20 down to 0) and offline RL with dual rewards (pre-generating responses from both teacher and student, then scoring them via an accuracy reward for correctness and a distillation reward for logit-level transferability). The framework pushes compact student models to match or surpass large open-source and closed-source VLMs—for example, InternVL3.5-2B reaches 75.1% average performance, exceeding the 7B LLaVA-OneVision by over 5 points—while establishing that gradually scaling teacher sizes during distillation (e.g., from 14B to 38B) yields smoother convergence and stronger generalization than one-shot distillation from a single large teacher only when the student's capacity is aligned with the teacher's evolving representational complexity.

2. Context and Motivation

The Central Problem: Distillation Under Capacity Mismatch

The paper addresses a specific, long-standing problem in knowledge distillation: how to effectively transfer knowledge from a very large teacher model to a much smaller student model when the parameter gap between them is extreme. This is not the general problem of distillation—distilling a 14B model into an 8B model with only a modest size gap—but rather the more challenging regime where teachers are 5–40× larger than their students (e.g., distilling InternVL3.5-38B into InternVL3.5-2B, a 19× gap). In this regime, the standard assumption that a student can adequately reproduce the teacher's output distributions breaks down.

The core failure mode is articulated in Section 1 and Section 2: the student's representational capacity—constrained by its smaller vocabulary, lower hidden dimensionality, and fewer layers—is fundamentally insufficient to capture the full complexity of the teacher's high-dimensional logit space. The authors characterize this as the student "struggling to reproduce the teacher's rich, high-dimensional representations," which manifests as unstable optimization (loss oscillation during training) and performance degradation (the distilled student performs significantly worse than what the teacher's capabilities would suggest is possible, even after extensive training).

This is not a minor edge case. As the paper notes in the introduction, the trend in VLM development has been toward ever-larger models: InternVL3.5-38B, Qwen3-VL-32B, and closed-source models like GPT-4o and Claude-4-Sonnet with hundreds of billions of parameters. Meanwhile, the practical demand is for models that run on mobile devices and edge hardware—typically in the 2B–8B parameter range. The gap between what is trained and what can be deployed is therefore widening, making the teacher-student capacity mismatch an increasingly critical bottleneck.

Why This Problem Matters: The Deployment Imperative

The practical motivation is clear and well-articulated in the paper's opening paragraphs. Large-scale VLMs "make them impractical for deployment on mobile or edge devices" and "as the demand for on-device intelligence continues to grow, there is an urgent need for compact yet powerful VLMs." The paper positions this urgency in terms of three converging trends:

1. The proliferation of on-device AI. Mobile phones, AR/VR headsets, autonomous vehicles, and IoT devices increasingly require sophisticated multimodal understanding—visual question answering, document understanding, scene reasoning—without the latency, privacy, and connectivity costs of cloud-based inference. A model like GPT-4o cannot run on a smartphone; a 2B–4B model can. But that small model must deliver performance that justifies deployment—merely being small is insufficient.

2. The resource economics of model serving. Even in cloud deployments, the cost differential between running a 72B model and a 7B model is substantial—not just in GPU-hours but in memory footprint, batching efficiency, and throughput. If distillation can push a 7B model to perform near 72B-level on common tasks, the economic case for distillation is overwhelming.

3. The scaling asymmetry between training and inference. The paper implicitly builds on a well-known observation from the scaling laws literature: training a model is a one-time cost, but inference is per-query. A model that is 10× smaller at inference time saves that cost on every single user query, potentially billions of times over the model's deployment lifetime. This makes distillation an investment with compounding returns, provided it can be done effectively.

The theoretical significance is equally important. The capacity gap problem exposes a fundamental question about neural network representations: can a small model, constrained in width and depth, learn to approximate the function of a much larger model through output-level supervision alone? If the answer is "no" under standard distillation approaches, then either the distillation objective must be modified, or the teacher's representation must be adapted to match the student's learning capacity. This paper pursues the latter strategy—modifying the teacher rather than the student—which represents a conceptual shift from prior work.

Prior Approaches and Their Limitations

The paper surveys a broad landscape of distillation techniques in Section 2 and Appendix A, grouping them into categories and identifying specific shortcomings that motivate the Masters framework.

Logit-Level Distillation (Hinton et al., 2015)

The classical approach: minimize the KL divergence between the teacher's and student's output probability distributions. When the teacher is very large relative to the student, the teacher produces highly confident, "peaked" distributions that contain fine-grained dark knowledge in the relative probabilities of incorrect classes. However, the paper's implicit critique is that this dark knowledge becomes noise when the student lacks the representational capacity to encode it. The student's limited vocabulary and hidden dimension cannot represent the same number of subtle distinctions, so forcing it to match the full teacher distribution leads to optimization instability—the student oscillates between trying to capture different aspects of the teacher's output space, never settling into a coherent representation.

Intermediate Feature Distillation

Later work extended distillation to intermediate layers: FitNets (Romero et al., 2014) align hidden representations, patient distillation (Sun et al., 2019) progressively distills from deeper layers, and review-based distillation (Chen et al., 2021) uses cross-stage connections. These approaches attempt to provide richer supervision signals by matching not just final outputs but intermediate feature maps or attention patterns. The paper acknowledges these advances but identifies a persistent limitation: none of them directly address the parameter gap itself. They provide the student with more supervision targets (more layers to match, more features to align), but if the student fundamentally lacks the capacity to represent those targets, the additional supervision may increase training instability rather than resolve it. The student is being pulled in more directions simultaneously, each with its own optimization tension.

Modified Training Objectives and Architectures

Recent work has explored alternative divergence measures (JSD instead of KL, as in LLaVADi and DistiLLM), multi-step distillation pipelines (AMD, LLaVA-KD), and cross-token general distillation. These approaches improve upon vanilla logit distillation but share a common limitation identified by the paper: they accept the large teacher as a fixed, monolithic entity and focus on improving how the student learns from it. The teacher's representational complexity is treated as a given—an immovable source of knowledge that the student must adapt to absorb. The paper argues this is backwards: if the teacher is too complex for the student to learn from directly, the teacher should be simplified to match the student's current capacity, then gradually made more complex as the student improves. This is the conceptual inversion that Masters introduces.

Reinforcement Learning-Based Distillation

More recent approaches (KDRL, RIL) have incorporated RL objectives into the distillation pipeline, using reward signals to guide the student toward teacher-like behavior. The paper identifies two critical limitations with existing RL-based methods:

Online RL is computationally prohibitive. Current paradigms like the "think-answer" process (DeepSeek-R1, DeepSeekMath) require the model to generate multiple long responses at every training step, then compute rewards, then update. For VLM distillation with large teachers generating responses, this becomes infeasibly slow—the paper estimates over 30 days on 256 A100 GPUs for 1.5M samples with online RL, compared to just two days for their offline approach.
Existing RL approaches rely on narrow, domain-specific reward signals. DeepSeek-R1 and its variants use rule-based parsers that work for math (checking final answers) or spatial grounding (checking bounding box coordinates) but fail for open-ended visual questions. The paper gives a concrete example: a traditional parser comparing "about five minutes" (predicted) with "5" (ground truth) would mark this as wrong, even though the prediction is semantically correct. This makes existing RL distillation approaches unsuitable for the diverse, open-ended visual question answering tasks that VLMs are designed for.

Multi-Teacher and Progressive Distillation

The paper acknowledges multi-teacher frameworks (ensemble distillation, FuseChat) and teacher assistant approaches (Mirzadeh et al., 2020) as partial steps toward addressing the capacity gap. Teacher assistants insert an intermediate-sized model between the large teacher and small student, which can help but requires training an additional model and introduces its own optimization challenges. More fundamentally, the paper argues that even a single intermediate teacher is a coarse approximation—what is needed is a continuous spectrum of teacher capacities, smoothly increasing over the course of training, which no prior approach provides.

The Specific Gap: No One Has Made the Teacher Adaptable

The paper's key observation is that existing distillation research has been overwhelmingly student-centric: modify the student's architecture, the student's training objective, the student's learning schedule, or the intermediate supervision signals provided to the student—but always treating the teacher as a fixed knowledge source. This perspective is evident in the paper's language: prior work "primarily focus[es] on modifying distillation objectives, aligning intermediate features, or combining multiple strategies," but "only limited efforts have been made to directly narrow the parameter gap between teacher and student."

The consequence of this student-centric view is that the fundamental limitation—the teacher's excessive complexity relative to the student's capacity—is never directly addressed. The student is expected to absorb a firehose of high-dimensional information from day one of training, leading to exactly the instability and degradation that the paper documents.

Masters inverts this perspective: make the teacher adaptable to the student's current learning capacity. Rather than asking "how can the student learn better from a fixed teacher?" the paper asks "how can the teacher present itself in a way that the student can learn effectively at each stage of training?" This is the core conceptual innovation, and it motivates every component of the framework:

Weight masking temporarily reduces the teacher's effective parameter count, making its representations sparser and lower-dimensional—closer to what the student can represent.
Progressive restoration gradually increases the teacher's complexity as the student's representations improve through training, maintaining capacity alignment throughout the distillation process.
Multi-response generation from the masked teacher provides training signals at the student's current level, rather than forcing it to match the fully complex teacher from the start.
Student self-generated responses are incorporated to maintain alignment between the teacher's guidance and the student's evolving representational capacity, preventing the teacher from providing guidance that the student cannot yet internalize.

How Masters Positions Itself

The paper positions Masters not as yet another distillation objective or architecture modification, but as a meta-strategy for capacity-aligned knowledge transfer that can integrate with existing distillation frameworks. This is explicitly demonstrated in Table 5(f), where Masters is applied to four existing distillation methods (DistiLLM, LLaVA-KD, VLsI, RIL) and consistently improves their performance. The message is: Masters does not compete with these methods—it makes them work better by addressing the root cause that limits them.

The paper also positions Masters as a practical, deployable framework in contrast to the "think-answer" RL paradigm. Figure 6 shows that Masters maintains the inference speed advantage of compact models—hours of inference time on one GPU for Masters-applied models is on par with the base student models, while think-answer models require substantially more time due to their lengthy reasoning chains. This practical focus is reinforced by the paper's emphasis on offline RL: by pre-generating responses once and reusing them throughout training, Masters achieves "strong performance without requiring the think-answer process," making it feasible for real-world distillation workflows where training time and computational cost are constrained.

Finally, the paper establishes Masters within a broader vision of scalable model compression (Section 1, end of introduction). The finding that gradually increasing teacher sizes during distillation (14B → 38B, or 14B → 38B → 78B) outperforms one-shot distillation from the largest teacher "reveals a scalable path toward efficient and deployable VLMs." This positions Masters as more than a one-time distillation technique—it is a framework that can accommodate future, larger teachers without requiring a redesign of the distillation pipeline, simply by extending the progressive scaling schedule.

3. Technical Approach

3.1 Reader Orientation

Masters is a training framework for transferring the knowledge of a large vision–language model (VLM) into a much smaller one. It solves the problem of the capacity gap: when the teacher is 5–40× larger than the student, the student's limited representational capacity makes it unable to absorb the teacher's rich, high-dimensional outputs directly—Masters bridges this gap by temporarily simplifying the teacher through weight masking, then progressively restoring its full complexity as training proceeds, while using offline reinforcement learning with dual reward signals to guide the student toward both accurate and transferable responses.

3.2 Big-Picture Architecture (Diagram in Words)

The Masters framework has five major components that operate in sequence across training iterations:

Teacher Weight Masker — Takes the large teacher model and, based on weight magnitudes, masks out a specified fraction of its least-important weights (temporarily zeroing them), producing a simplified "masked teacher" with reduced representational complexity. This is repeated at multiple masking ratios (e.g., 0.20, 0.15, 0.10, 0.05, 0) to create a set of progressively more capable teacher variants.
Multi-Response Generator — Each masked teacher, as well as the student in its current state, pre-generates multiple responses (typically 8) for every question in a 1.5M-sample visual instruction tuning dataset. The teacher generates at high temperature (1.0) with top-p sampling to produce diverse candidate answers; the student similarly generates from its own evolving distribution. These responses are stored for reuse throughout training.
Dual Reward Scorer — For each generated response, two rewards are computed offline: an accuracy reward (binary 0/1, assigned by an LLM-as-a-Judge that compares the response semantically against ground-truth labels) and a distillation reward (continuous [0,1], derived from the Jensen-Shannon divergence between teacher and student logits, then reverse-min-max normalized so smaller divergence yields higher reward). The total reward is the sum of both.
Mask-Progressive Distillation Scheduler — At each training iteration, selects which masked teacher variant to use based on a staircase-decreasing masking ratio schedule (Equation 3). The student is trained to match the currently-active masked teacher's logit distribution using JSD, where the teacher's capacity increases as the masking ratio decreases throughout training.
Student Updater (GRPO + Distillation Loss) — Updates the student's weights using a combined objective (Equation 8) that integrates GRPO's policy gradient loss with a distillation divergence term. The GRPO component uses the dual rewards as advantage signals to push the student toward high-reward response patterns; the distillation term ensures the student's output distribution remains aligned with the teacher's.

Information flows as follows: pre-generated question-answer pairs → masked teacher (at current masking ratio) produces multiple candidate responses; student also produces multiple responses → both sets of responses are scored with accuracy and distillation rewards → student is updated via the combined GRPO+distillation objective → masking ratio decreases at scheduled intervals → process repeats with a richer, less-masked teacher.

3.3 Roadmap for the Deep Dive

We will walk through the technical components of Masters in the order they interact during training, which is the order that most naturally builds understanding:

First, the magnitude-based teacher masking mechanism (Section 3.1) — how non-dominant weights are identified and zeroed, why per-layer thresholds are critical, and the design choice to temporarily mask rather than permanently prune.
Second, the mask-progressive distillation schedule (Section 3.2) — how the masking ratio decays over training iterations, how the student's distillation objective adapts to the changing teacher, and the multiple-response generation strategy that addresses limitations in standard SFT data.
Third, the offline RL setup with dual rewards (Section 3.3) — the accuracy reward via LLM-as-a-Judge, the distillation reward via reverse-min-max-normalized JSD, their combination into a total reward, and the integration with GRPO.
Fourth, the unified training objective — how GRPO and distillation are combined into Equation 8, and the specialization of GRPO to the offline, single-update-per-iteration setting.
Fifth, the teacher-size scaling strategy (from Table 2 and 5(b)) — the empirical finding that sequentially distilling from progressively larger teachers outperforms one-shot distillation, and how this integrates with mask-progressive training.
Sixth, the key hyperparameters and implementation decisions — masking ratios, number of generated responses, learning rate, reward design, and data composition choices that determine practical performance.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a distillation framework paper whose core idea is that temporarily reducing teacher complexity via magnitude-based weight masking, then progressively restoring it, enables the student to learn in a stable, capacity-aligned manner—and that adding offline RL with dual rewards (accuracy + distillation transferability) refines the student's responses beyond what supervised distillation alone achieves.

Magnitude-Based Teacher Masking

The paper identifies the parameter gap between teacher and student as the root cause of distillation instability. A large teacher like InternVL3.5-38B has approximately 38 billion parameters; a small student like InternVL3.5-2B has approximately 2 billion parameters—a 19× difference. The teacher's weight space encodes representational complexity that the student's smaller weight space simply cannot accommodate, leading to optimization plateaus and performance degradation when the student is directly forced to match the teacher's output distributions. The approach Masters takes is conceptually simple: reduce the teacher's effective parameter count to bring it closer to the student's capacity, then let the student learn from this simplified teacher before gradually reintroducing complexity.

The masking mechanism draws from classical network pruning (Han et al., 2015), where weights with small absolute magnitudes are considered "non-dominant" — contributing marginally to the model's predictions and therefore removable with minimal impact on output quality. Unlike permanent pruning, Masters uses temporary masking: weights are zeroed during forward passes through the teacher but the original weight values are preserved, allowing them to be restored later as training progresses.

Formally, given a teacher $\mathcal{T}$ with weights $\mathbf{W}^{\mathcal{T}} = \{w_n\}_{n=1}^N$ where $N$ is the total number of parameters (e.g., $N = 38\text{B}$ for InternVL3.5-38B), a binary mask $\mathbf{M}_r = \{m_n\}_{n=1}^N$ with $m_n \in \{0, 1\}$ is constructed as:

$m_n = \begin{cases} 1, & \text{if } |w_n| \geq \lambda_r, \\ 0, & \text{otherwise}, \end{cases}$

where $\lambda_r$ is a magnitude threshold determined by the desired masking ratio $r \in [0, 1]$ , and $n$ indexes individual weight parameters across all layers of the teacher network.

What it computes: For a given masking ratio $r$ (e.g., $r = 0.2$ ), we sort all $N$ weight magnitudes in ascending order and set $\lambda_r$ to the value at approximately the $(r \times 100)$ -th percentile, so that roughly $r \times N$ weights have magnitude below $\lambda_r$ and get masked to zero. The resulting masked weights are:

$\mathbf{W}^{\mathcal{T}_r} = \mathbf{M}_r \odot \mathbf{W}^{\mathcal{T}},$

where $\odot$ denotes element-wise multiplication, and $\mathcal{T}_r$ is the resulting masked teacher. When $r = 0$ , no weights are masked — this is the original teacher. When $r = 0.2$ , approximately 20% of weights (those with smallest magnitudes) are zeroed, effectively reducing the teacher's active parameter count by 20%. The masked teacher still functions for inference (it can generate responses), but its representational complexity is reduced, making its output distributions simpler for the student to match.

Why this form: Magnitude-based pruning rests on the well-established observation that neural networks are over-parameterized; small-magnitude weights contribute minimally to the forward pass because they multiply inputs by near-zero values. The paper extends this insight to distillation: if low-magnitude weights encode fine-grained representational details that the student cannot represent anyway, zeroing them simplifies the teacher's output space without destroying its core knowledge. The binary mask $\mathbf{M}_r$ is temporary — unlike permanent pruning, weights are not removed from the model, only zeroed during forward passes, which preserves the ability to progressively restore them by simply reducing $r$ . Alternative approaches like permanently pruning the teacher would prevent progressive restoration; alternative masking criteria like random masking would indiscriminately remove important and unimportant weights, potentially degrading the teacher's predictions on specific inputs rather than uniformly simplifying its representations.

A critical implementation detail: per-layer thresholding. The paper notes that "using a global threshold $\lambda_r$ across all layers often excessively prunes certain layers, which can make the model non-functional at inference time." In a deep transformer, different layers have different weight magnitude distributions — early layers might have larger-magnitude weights on average than later layers, or vice versa. A global threshold would remove a disproportionate fraction of weights from some layers while barely touching others, potentially breaking the model's ability to process information through those severely pruned layers. To prevent this, $\lambda_r$ is computed per layer: for each layer $\ell$ , the weights within that layer are sorted independently, and $\lambda_r^{(\ell)}$ is set to the $(r \times 100)$ -th percentile of that layer's weight magnitudes. This ensures that approximately $r$ fraction of weights is masked uniformly across all layers, maintaining balanced network connectivity. This design choice is crucial for the mask-progressive strategy — if masking broke certain layers at high ratios, the teacher would not be usable for generating training data at those ratios, and the progressive restoration would hit a cliff rather than a smooth ramp.

The paper sweeps maximum masking ratios $r_{\max}$ from 0 to 0.5 in Table 5(a), finding that $r_{\max} = 0.2$ is optimal for most teachers (Qwen2.5-VL-72B, InternVL3-78B, InternVL3.5-38B), while Qwen3-VL-32B and InternVL3.5-8B/14B perform best at $r_{\max} = 0.4$ . Beyond these values, performance degrades sharply — at $r_{\max} = 0.5$ , average performance drops from 80.0% to 49.2% for InternVL3.5-38B, indicating that masking half the weights removes too much essential knowledge. This validates the core premise: some teacher simplification helps (reducing complexity the student cannot absorb), but too much simplification destroys the knowledge the student needs to learn.

Mask-Progressive Distillation Scheduling

While learning from a single masked teacher at a fixed masking ratio $r$ simplifies the initial learning problem, it comes at a cost: the student is never exposed to the full richness of the teacher's representations. The student would learn a simplified version of the teacher's knowledge — effectively a lower-capacity approximation — and never recover the fine-grained distinctions that only the fully unmasked teacher can provide. The paper resolves this tension through a mask-progressive strategy: start with a heavily masked teacher (high $r$ ), train the student to match this simplified teacher, then progressively reduce the masking ratio throughout training so the teacher gradually reveals its full complexity, and the student gradually adapts its representations to match.

The masking ratio is scheduled using a staircase function over training iterations. At each iteration $i \in \{1, \dots, I\}$ (where $I$ is total training iterations), the current masking ratio $r[i]$ is:

$r[i] = r_{\max} - s \cdot \left\lfloor \frac{i \times M}{I} \right\rfloor,$

where $r_{\max}$ is the initial (maximum) masking ratio, $s$ is the decrement step size applied at each masking stage, $M$ is the total number of masked teacher variants, computed as $M = r_{\max} / s + 1$ , and $\lfloor \cdot \rfloor$ is the floor function.

What it computes: This equation defines a discrete schedule where the masking ratio stays constant for $I/M$ iterations, then drops by $s$ , then stays constant again, and so on — producing a staircase-like decay from $r_{\max}$ down to 0. For a concrete example with $r_{\max} = 0.2$ and $s = 0.05$ : we have $M = 0.2 / 0.05 + 1 = 5$ stages. The masking ratio starts at $0.20$ for the first $I/5$ iterations, drops to $0.15$ for the next $I/5$ iterations, then $0.10$ , then $0.05$ , and finally $0$ (full teacher) for the last $I/5$ iterations. The floor operation ensures that the ratio changes only at stage boundaries, not continuously — the student trains against a fixed teacher configuration within each stage, which provides stable learning targets.

Why this form: The staircase decay has several properties that matter for stable distillation. First, it provides extended periods of constant teacher behavior within each masking stage. If the masking ratio changed continuously (e.g., linear decay), the teacher's output distribution would shift at every training step, creating a moving target that the student must chase — this is known to cause optimization instability, as the student never converges to a stable representation before the target changes again. The staircase schedule gives the student time to adapt to each teacher configuration before the next capacity increase. Second, the uniform stage durations ( $I/M$ iterations per stage) ensure balanced training across teacher capacities — the student spends equal time learning from each level of teacher complexity, rather than spending most of its time on the full teacher (as would happen with an exponential decay). Third, the step size $s = 0.05$ is chosen to keep capacity changes small enough that each transition is smooth — the teacher's complexity increases by only ~5% of its weights at each step, so the student's current representations are a good initialization for learning the next level. A larger step (e.g., $s = 0.1$ ) would create sharper jumps that could disrupt learning; a smaller step (e.g., $s = 0.01$ ) would require many more teacher variants ( $M = 21$ for $r_{\max} = 0.2$ ), increasing the pre-generation and storage cost without clear benefit.

At each iteration $i$ , given the current masking ratio $r[i]$ , the distillation objective adapts to the corresponding masked teacher's output distribution. However, the paper goes beyond simple logit matching by explicitly addressing two limitations of standard supervised fine-tuning datasets.

Problem 1: Representation mismatch in conventional SFT data. Standard visual instruction tuning datasets (the paper uses a 1.5M-sample collection combining COCO, VQA-v2, ChartQA, DocVQA, MathV360K, and many other sources detailed in Appendix C) contain answer labels typically generated by very large closed-source models (GPT-4o, Gemini, Claude) or filtered by human annotators. These answer labels are produced by models with vastly larger vocabularies and higher hidden dimensions than the student — they contain linguistic patterns, reasoning styles, and vocabulary choices that the student's limited representation space cannot faithfully encode. Injecting such "over-rich" labels directly into a small student leads to performance degradation because the student is forced to mimic output patterns it cannot internally represent.

Problem 2: Single-answer limitation. Standard SFT datasets provide exactly one answer per question, which "severely limits response diversity and generalization" and "forces the student to overfit to a narrow linguistic or reasoning style."

Masters addresses both problems by replacing standard SFT labels with multiple pre-generated responses from the masked teacher and the student themselves. Formally, instead of the standard distillation objective:

$\min_{\mathbf{W}^{\mathcal{S}}} \mathbb{E}_{(x, y) \sim \text{Data}[i]} \left[ \mathcal{D}\left(P_{\mathcal{T}_{r[i]}}(y|x) \parallel P_{\mathcal{S}}(y|x)\right) \right],$

which matches student logits to teacher logits on standard answer labels $y$ , Masters uses:

$\min_{\mathbf{W}^{\mathcal{S}}} \mathbb{E}_{(x, \hat{y}) \sim \text{Gen-Data}[i]} \left[ \mathcal{D}\left(P_{\mathcal{T}_{r[i]}}(\hat{y}|x) \parallel P_{\mathcal{S}}(\hat{y}|x)\right) \right]$

where $\mathbf{W}^{\mathcal{S}}$ denotes the student's weights, $(x, \hat{y})$ are question–response pairs drawn from a pre-generated multi-response dataset $\text{Gen-Data}[i]$ , $P(\hat{y}|x)$ denotes the logit-softmax output for response $\hat{y}$ given question $x$ , and $\mathcal{D}$ is the Jensen-Shannon Divergence.

What this substitution accomplishes: Instead of matching teacher and student distributions on a single ground-truth answer, the student matches the teacher's distribution on multiple diverse candidate responses — some generated by the masked teacher itself, some generated by the student. The teacher-generated responses provide adaptive guidance that evolves with the masking ratio: when $r$ is high, the masked teacher generates simpler responses that the nascent student can more easily learn to reproduce; as $r$ decreases, the teacher generates increasingly sophisticated responses that push the student toward higher-quality outputs. The student-generated responses serve a complementary role: they maintain alignment between the teacher's guidance and the student's own representational capacity. If the student were trained only on teacher-generated text, a distribution mismatch would emerge — the teacher's responses come from a different model, with different linguistic patterns, and optimizing the student to match those patterns might pull it away from what it can naturally produce. By including the student's own responses in the training data, the objective encourages the student to refine its own outputs to be more teacher-like, rather than to abandon its representational strengths entirely.

The choice of Jensen-Shannon Divergence (JSD) over KL divergence is based on prior empirical findings in knowledge distillation. JSD is defined as:

$\mathcal{D}(P_{\mathcal{T}} \parallel P_{\mathcal{S}}) = \frac{1}{2} \text{KL}(P_{\mathcal{T}} \parallel M) + \frac{1}{2} \text{KL}(P_{\mathcal{S}} \parallel M),$

where $M = (P_{\mathcal{T}} + P_{\mathcal{S}}) / 2$ is the average distribution. JSD has two advantages over KL divergence for distillation: it is symmetric (unlike KL, the order of teacher and student does not matter), and it is bounded in $[0, \log 2]$ , meaning that large mismatches do not produce arbitrarily large gradients that can destabilize training. The paper cites LLaVADi and DistiLLM as prior work establishing JSD's superiority for distillation.

The paper sweeps the number of pre-generated responses per question in Figure 5, showing that performance improves until approximately 8 responses, after which it plateaus. The final configuration uses 8 responses per question: 4 from the masked teacher and 4 from the student (a 1:1 ratio). Table 5(c) validates this ratio: using only teacher responses (8 from teacher, 0 from student) limits the student's adaptability to its own capacity; using only student responses (0 from teacher, 8 from student) restricts exposure to the teacher's richer semantics; the balanced ratio achieves the best performance (87.2% on AI2D, 85.0% on MathVista, 88.2% on MMB, 85.6% on MM-Vet, 72.7% on MMMU for InternVL3.5-8B).

The pre-generation uses vLLM for fast inference with temperature 1.0, top-p 0.9, top-k 50, and repetition penalty 1.05 — settings that balance diversity (high temperature, top-p sampling) with coherence (repetition penalty to avoid degenerate loops).

Offline RL with Dual Rewards

While mask-progressive distillation enables capacity-aligned learning, it has a blind spot: the generated responses used for training may contain factual errors or exhibit linguistic complexity that hinders effective knowledge transfer. A response from the masked teacher might be syntactically complex in a way that makes its logit distribution hard for the student to match, even if the response is factually correct. Conversely, a student-generated response might be easy to learn from (simple, well-aligned with the student's current distribution) but factually incorrect. Neither the accuracy nor the learnability of responses is captured by the raw JSD distillation objective.

To address this, Masters integrates reinforcement learning into the distillation pipeline, using two complementary reward signals to evaluate each generated response and guide the student toward responses that are both correct and transferable. Crucially, this RL is performed offline, meaning:

Responses are pre-generated once by all masked teacher variants and by the student before RL training begins. This eliminates the need to run expensive teacher and student forward passes at every training step.
Rewards are pre-computed from these static responses using offline scoring procedures, so no LLM-as-a-Judge calls or logit computations are needed during RL training.
Training uses pre-computed (response, reward) pairs, similar to how offline RL in decision-making uses pre-collected trajectories rather than online interaction.

This design is motivated by a stark computational comparison: the paper estimates that online RL on 1.5M samples would require over 30 days on 256 A100 GPUs, whereas Masters' offline approach completes training in just two days — a 15× speedup that makes the framework practical.

Accuracy Reward

The accuracy reward evaluates whether a generated response $\hat{y}$ is semantically correct with respect to the ground-truth answer label $y$ , given the question $x$ . The paper uses an LLM-as-a-Judge approach rather than rule-based parsing, motivated by a concrete failure case of traditional parsers:

"when comparing 'about five minutes' (predicted) with '5' (answer label), traditional parsing says that the predicted response is wrong answer."

A rule-based parser would perform exact string matching or simple numeric extraction and fail to recognize that "about five minutes" is a semantically valid answer to a question whose labeled answer is "5." LLM-as-a-Judge overcomes this by using a language model to evaluate semantic equivalence:

$\mathcal{R}_{\text{acc}} = \text{LLM-as-a-Judge}(x, \hat{y}, y) \in [0, 1].$

What it produces: A binary score (0 or 1) indicating whether the generated response semantically matches the ground-truth answer, regardless of phrasing differences. The judge is provided with the original question, the ground-truth answer, and the generated response; it reasons about semantic equivalence and outputs a binary judgment.

The implementation is a two-stage process (detailed in Appendix D). First, a prediction evaluation prompt feeds the question, ground truth, and generated text to the LLM judge with instructions:

Assign 1 if the generated text "fully and correctly captures the core point"
Assign 0 if "incorrect or irrelevant," "has repetitive response," or "has empty response"
Output the score wrapped in <answer> tags after a brief explanation.

Second, an accuracy reward parsing prompt extracts the binary integer from the evaluation summary, ensuring a clean 0/1 signal for the RL objective. The LLM used for judging is the same as the one selected for response generation — this maintains consistency in evaluation standards and avoids the distribution mismatch that would occur if a different model family were used for judging.

Why LLM-as-a-Judge over rule-based parsing: The reasoning tasks in the VLM benchmarks span diverse formats: open-ended visual descriptions ("Describe the person's emotion"), chart reading with numeric answers, multiple-choice selection, spatial reasoning, and OCR-based text extraction. A unified parsing approach that works across all these formats does not exist — each would require custom extraction logic (regex for numbers, string matching for multiple choice, embedding similarity for open-ended text). LLM-as-a-Judge provides a single, generalizable mechanism that handles all formats through semantic understanding. The paper argues this makes the accuracy reward "more generalizable and semantically grounded" than domain-specific parsers.

Distillation Reward

The accuracy reward ensures correctness but ignores a crucial dimension: how easily can the student learn from this response? A response might be perfectly accurate (scoring 1.0 on accuracy) but encoded in a logit distribution that is extremely peaked, with the correct token assigned near-1.0 probability and all others near 0.0. Such a distribution is hard for the student to match because the student's limited capacity may not support that level of confidence — forcing the match would require the student to suppress all uncertainty, which may be incompatible with what it has learned so far.

The distillation reward addresses this by measuring logit-level alignment between teacher and student. For a given generated response $\hat{y}$ , the divergence between the teacher's and student's logit-softmax distributions $P_{\mathcal{T}}(\hat{y}|x)$ and $P_{\mathcal{S}}(\hat{y}|x)$ is computed using JSD:

$\mathcal{D} = \text{JSD}(P_{\mathcal{T}}(\hat{y}|x) \parallel P_{\mathcal{S}}(\hat{y}|x)).$

However, the paper observes that "the values of $\mathcal{D}$ exhibit small variance across the generated responses due to its low scaling." This means raw JSD values cluster in a narrow range, making it difficult for the RL objective to discriminate between high-quality and low-quality responses — the reward signal would be too weak to guide learning.

To address this, a reverse min-max normalization is applied across the set of generated responses for a given question:

$\mathcal{R}_{\text{distill}} = \frac{\mathcal{D}_{\max} - \mathcal{D}}{\mathcal{D}_{\max} - \mathcal{D}_{\min}},$

where $\mathcal{D}_{\max}$ and $\mathcal{D}_{\min}$ are the maximum and minimum JSD values among all generated responses $\hat{y}$ for question $x$ , and $\mathcal{D}$ is the JSD for the specific response being scored.

What it computes: For a batch of responses to the same question, we compute JSD between teacher and student logits for each response. The response with the highest JSD (worst alignment) gets mapped to reward 0.0; the response with the lowest JSD (best alignment) gets mapped to reward 1.0; all other responses receive linearly interpolated values. This is a reverse normalization because smaller divergence (better alignment) yields higher reward — the subtraction $\mathcal{D}_{\max} - \mathcal{D}$ inverts the scale.

Why this form: The reverse min-max normalization serves two purposes. First, it amplifies the signal — raw JSD values might range from 0.15 to 0.25, a 0.10 spread, but the normalized rewards span the full [0, 1] range, making the reward differences large enough to drive policy improvement. Second, it provides relative scoring within a question: responses are evaluated relative to each other, not against an absolute threshold. This matters because the inherent difficulty of matching teacher logits varies by question — a question requiring complex spatial reasoning might produce higher JSD values across all responses than a simple recognition question. Without normalization, the RL objective would penalize the student for attempting hard questions where alignment is naturally harder, potentially causing it to avoid challenging examples. With per-question normalization, the reward reflects how well each response transfers knowledge compared to alternatives for the same question.

The paper ablates the normalization in Table 5(e), where $\mathcal{R}_{\text{distill}}(\triangle)$ denotes the distillation reward without normalization. Without normalization, the distillation reward "becomes less discriminative, yielding performance comparable to using only the accuracy reward" — confirming that the normalization is essential for the distillation reward to contribute beyond what accuracy alone provides.

The total reward for a response is the simple sum:

$\mathcal{R} = \mathcal{R}_{\text{acc}} + \mathcal{R}_{\text{distill}} \in [0, 2].$

This additive combination ensures both correctness and transferability are jointly optimized: a response that is correct but hard to learn from (high JSD) still gets positive reward from accuracy; a response that is easy to learn from but incorrect may get partial reward from distillation. The optimal response under this reward is one that is both factually accurate and produces logit distributions well-aligned with the student's current capacity.

Unified Training Objective with GRPO

With pre-computed rewards in hand, Masters trains the student using an objective that combines GRPO (Generalized Reinforcement Policy Optimization) with the distillation divergence term. The starting point is the standard GRPO formulation for a question $x$ and its $G$ generated responses $\{\hat{y}_j\}_{j=1}^G$ :

$\mathcal{L}_{\text{GRPO}} = -\mathbb{E}_j \left[ \mathbb{E}_t \left[ \min\left(r_{j,t} A_j, \text{clip}(r_{j,t}, 1 - \varepsilon, 1 + \varepsilon) A_j\right) - \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}}) \right] \right],$

where $r_{j,t}$ is the policy ratio for the new policy $\pi_\theta$ versus the old policy $\pi_{\theta_{\text{old}}}$ at token $t$ of response $j$ , and $A_j$ is the advantage for response $j$ computed from normalized rewards.

Specialization to the Masters setting: The paper makes a critical simplification based on the offline, single-update-per-iteration nature of the training. Since "the student is updated only once per training iteration $i$ " and "the old policy $\pi_{\theta_{\text{old}}}$ and the new policy $\pi_\theta$ are identical" (there is no iterative policy improvement within a single iteration), the policy ratio $r_{j,t}$ is always 1. This eliminates the clipped surrogate term, and the GRPO objective simplifies to:

$\mathcal{L}_{\text{GRPO}} = -\mathbb{E}_j \left[ r_{j,t} A_j - \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}}) \right],$

with $r_{j,t} = 1$ . The paper notes: "Technically, we still keep the ratio term in the expression to ensure the gradient properly flows to the student parameters during training" — i.e., even though the ratio evaluates to 1, including it in the computation graph ensures that the advantage signal propagates gradients through the policy's logit outputs to the student's weights.

The advantage $A_j$ is computed as the normalized reward:

$A_j = \frac{\mathcal{R}_j - \text{mean}(\{\mathcal{R}_j\}_{j=1}^G)}{\text{std}(\{\mathcal{R}_j\}_{j=1}^G)},$

where $\mathcal{R}_j = \mathcal{R}_{\text{acc},j} + \mathcal{R}_{\text{distill},j}$ is the total reward for response $j$ , and $\text{mean}(\cdot)$ and $\text{std}(\cdot)$ are computed over the $G$ responses for the given question. This normalization ensures that advantages have zero mean and unit variance across the response group, providing a standardized optimization signal that is invariant to the absolute scale of rewards.

The KL penalty term $\beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})$ with $\beta = 0.1$ regularizes the student's policy to not deviate too far from a reference policy. In the Masters setup, $\pi_\theta$ is the student's current policy $P_{\mathcal{S}}$ , and $\pi_{\text{ref}}$ is the student's initial (pre-distillation) policy. The KL penalty serves as a trust-region constraint: it prevents the student from over-optimizing the reward signal at the cost of diverging fundamentally from its initial language understanding, which would manifest as reward hacking (producing responses that score well but are linguistically degenerate).

The distillation divergence term is added to the GRPO objective, producing the final unified objective:

$\min_{\mathbf{W}^{\mathcal{S}}} \mathbb{E}_{(x, \hat{y}) \sim \text{Gen-Data}[i]} \left[ \mathcal{L}_{\text{GRPO}} + \mathcal{D}\left(P_{\mathcal{T}_{r[i]}}(\hat{y}|x) \parallel P_{\mathcal{S}}(\hat{y}|x)\right) \right].$

What this objective computes: For each training iteration $i$ , the current masked teacher $\mathcal{T}_{r[i]}$ is used. A batch of question–response pairs $(x, \hat{y})$ is sampled from the pre-generated dataset $\text{Gen-Data}[i]$ , which contains responses generated by both the masked teacher and the student. For each pair, two terms are computed and summed:

The GRPO loss $\mathcal{L}_{\text{GRPO}}$ : encourages the student to increase the probability of high-advantage responses (those with high combined accuracy + distillation rewards relative to other responses for the same question) and decrease the probability of low-advantage responses, while staying close to the reference policy.
The distillation divergence $\mathcal{D}(P_{\mathcal{T}_{r[i]}} \parallel P_{\mathcal{S}})$ : encourages the student's logit distribution to match the masked teacher's logit distribution on the response tokens, providing direct supervised guidance that complements the reward-based GRPO signal.

The two terms work synergistically: the distillation term provides dense, token-level supervision that helps the student learn the teacher's output patterns; the GRPO term provides trajectory-level feedback that steers the student toward responses with desirable properties (correctness and transferability) that may not be captured by simple logit matching. The student is optimized using AdamW with a fixed learning rate of $1 \times 10^{-6}$ , and the training is implemented with DeepSpeed ZeRO-3 to handle the memory requirements of large teacher and student models on A100 80GB GPUs.

Why this combined form instead of either component alone: Table 5(e) shows that using only the accuracy reward ( $\mathcal{R}_{\text{acc}}$ ) yields strong but incomplete performance (86.5% on AI2D, 82.3% on MathVista for InternVL3.5-8B). Adding the distillation reward ( $\mathcal{R}_{\text{acc}} + \mathcal{R}_{\text{distill}}$ ) improves performance further (87.2% on AI2D, 85.0% on MathVista), confirming that transferability matters beyond correctness. Using only the distillation reward without accuracy performs worse than accuracy alone on most metrics — the student would learn to generate responses that are easy to transfer but not necessarily correct. The combination captures both desiderata. The distillation divergence term is always present regardless of reward configuration, ensuring the student never loses alignment with the teacher's output space even when optimizing for rewards.

Teacher-Size Scaling Strategy

Beyond mask-progressive distillation from a single large teacher, the paper investigates a complementary scaling dimension: gradually increasing the teacher model size during distillation. The motivation is that even with weight masking, there remains a capacity gap: distilling directly from a 38B teacher (even at $r=0.2$ , effectively ~30B active parameters) into a 2B student is challenging. An intermediate step — first distilling from a 14B teacher, then from the 38B teacher — provides a smoother capacity ramp.

Table 2 demonstrates this across multiple model families. For InternVL3.5-8B, four configurations are compared:

+Large Teacher: direct distillation from InternVL3.5-38B only → 75.8% average.
+Mid Teacher: distillation from intermediate-sized models (4B, 8B, 14B) between the student (8B) and large teacher (38B) → 76.3% average.
+Mask-Progressive: mask-progressive distillation from the large teacher → 77.0% average.
+Reward Feedback (full Masters): mask-progressive + RL with mid-teacher scaling → 80.0% average.

The progression shows that each component — mid-teacher scaling, mask-progressive distillation, and RL — contributes additively. Even the simpler configuration of mask-progressive with mid-teacher scaling reaches 77.0%, already substantially above the 75.8% from direct distillation. The full Masters configuration achieves 80.0%, a 4.2 percentage point gain over the naive baseline.

Table 5(b) explores the optimal teacher composition for InternVL3 series distillation to an 8B student. Using the 38B teacher alone yields 86.8% on AI2D, 73.4% on MathVista, 85.3% on MMB. Adding intermediate teachers progressively — 14B+38B, 14B+78B, 38B+78B, and finally 14B+38B+78B — produces monotonic improvements, with the full three-teacher sequence reaching 88.9% on AI2D and 82.3% on MathVista. The paper presents this as "a scalable path toward efficient and deployable VLMs": as larger teachers become available, they can be added to the progressive sequence without redesigning the distillation pipeline.

Figure 2(b) visualizes the performance trajectory across this progressive scaling. The student's performance accelerates significantly when switching from the mid teacher to the large teacher, "highlighting the advantage of this teacher scaling strategy" — the warm-up phase with the intermediate teacher prepares the student's representations to more effectively absorb the large teacher's knowledge, rather than struggling to learn from scratch against a very large capacity gap.

Key Implementation Decisions and Hyperparameters

The paper makes several concrete implementation choices that are essential for reproducibility:

Masking configuration: The decrement step $s = 0.05$ is used throughout. With $r_{\max} = 0.2$ , this creates $M = 5$ masked teacher variants at ratios $\{0.20, 0.15, 0.10, 0.05, 0\}$ . Each variant is saved as a separate checkpoint and used to pre-generate responses before training begins. The global $\lambda_r$ is not used — per-layer thresholds prevent layer-wise imbalance.

Response generation: 8 responses per question (4 teacher + 4 student) using temperature 1.0, top-p 0.9, top-k 50, repetition penalty 1.05. The vLLM library accelerates generation. The LLM-as-a-Judge model is identical to the response generation model for consistency.

Training: AdamW optimizer with fixed learning rate $1 \times 10^{-6}$ , no learning rate schedule. DeepSpeed ZeRO-3 handles memory. KL penalty coefficient $\beta = 0.1$ . The GRPO advantage is computed with group normalization (zero mean, unit variance) across the 8 generated responses per question.

Data: 1.5M visual instruction tuning samples from the diverse collection listed in Appendix C, spanning real-world images (COCO, VQA-v2), document understanding (ChartQA, DocVQA, InfoVQA), mathematical reasoning (MathV360K, CLEVR-Math, Geometry3K), OCR (TextVQA, SROIE), and synthetic data from InternVL's filtered knowledge datasets.

Training efficiency: The offline pre-generation and reward computation enable the entire Masters pipeline to complete in approximately two days on A100 GPUs, compared to an estimated 30+ days for online RL on the same data scale — a critical practical advantage that makes the framework deployable in real-world distillation workflows.

4. Key Insights and Innovations

Innovation 1: Teachers Should Adapt to Students, Not the Reverse — A Conceptual Inversion of Knowledge Distillation

The dominant assumption across decades of knowledge distillation research—from Hinton et al. (2015) through logit matching, intermediate feature alignment, multi-teacher frameworks, and RL-based distillation—has been that the teacher is a fixed, immutable source of knowledge and the burden of adaptation falls entirely on the student. The student must learn to match the teacher's output distributions (logit distillation), reproduce its intermediate representations (feature distillation), or imitate its behavioral patterns (RL-based approaches). When the teacher-student capacity gap is small, this assumption is reasonable: the student can adequately approximate the teacher's representational space.

What Masters recognizes—and what makes this contribution genuinely novel rather than an obvious observation—is that this assumption breaks down catastrophically when the size gap is large. The paper provides a crisp diagnostic: when the teacher is 5–40× larger than the student (as in distilling InternVL3.5-38B into InternVL3.5-2B), the student's constrained vocabulary, lower hidden dimensionality, and fewer layers make it structurally incapable of encoding the teacher's full output distribution. Forcing the match anyway produces exactly the "unstable learning and degraded performance" that the paper documents as the central failure mode. Prior work recognized this instability (Zhang et al., 2023; Mirzadeh et al., 2020) but responded with student-side interventions: better objectives, richer supervision signals, or intermediate teacher assistants. These are all variations on "help the student cope with the teacher's complexity."

Masters performs a conceptual inversion: rather than helping the student cope with the teacher, modify the teacher to match the student's current learning capacity. This is not an incremental refinement—it is a re-framing of the distillation problem from "how can the student absorb a fixed teacher?" to "how can the teacher present itself at a level the student can currently absorb?" The weight-masking mechanism operationalizes this inversion: by zeroing out non-dominant weights based on magnitude, the teacher's effective representational complexity is reduced to a level closer to what the student can encode, without destroying the core knowledge those weights collectively represent. As the student's own representations improve through training, the teacher progressively reveals more complexity.

The significance of this inversion extends beyond the specific technique. It establishes a principle for distillation under extreme capacity gaps: the teacher should be a curriculum, not a fixed target. This principle is general—it does not depend on magnitude-based masking specifically; any mechanism that temporarily reduces teacher complexity while preserving recoverability would instantiate the same idea. The paper shows that this principle can be layered on top of existing distillation frameworks (Table 5f shows Masters improving DistiLLM, LLaVA-KD, VLsI, and RIL), confirming that it addresses a root cause those methods leave untouched.

The evidence for this inversion's importance is in the ablation structure of Table 1: naive distillation from a large teacher is the baseline; adding mask-progressive distillation (the teacher-adaptation mechanism) yields consistent 1–2 percentage point improvements across all model families and metrics; adding RL on top yields further gains. But critically, the mask-progressive gain exists even without RL, demonstrating that teacher adaptation alone provides value independent of the reward design. This is what distinguishes a genuine conceptual contribution from a compound system where all components must work together: the inversion stands on its own.

Innovation 2: Verifier Over-Optimization as a First-Class Phenomenon in Test-Time Scaling

The paper provides compelling evidence for a phenomenon that, while recognized in RLHF literature as "reward hacking," had not been systematically characterized in the context of knowledge distillation with RL: distillation rewards can be gamed by the student in ways that harm genuine knowledge transfer. This is not merely a training instability—it is a structural problem where the reward signal and the underlying objective of distillation (absorbing the teacher's knowledge) come into conflict.

The distillation reward R_distill measures logit-level alignment between teacher and student: smaller JSD between their output distributions for a given response yields higher reward. The intended interpretation is that responses with high distillation reward are "easier to transfer"—the student can match the teacher's logit distribution on those responses with less representational strain, suggesting they are well-aligned with the student's current capacity. However, the paper identifies a subtle failure mode: the reward can be maximized by the student learning to produce outputs whose logit distributions happen to match the teacher's, rather than outputs that genuinely reflect understanding of the teacher's knowledge. This is the distillation analog of reward hacking: the student optimizes for logit alignment scores rather than for genuine knowledge transfer.

The problem is made concrete by the paper's observation that the raw JSD values "exhibit small variance across the generated responses due to its low scaling." Without normalization, the distillation reward collapses to near-uniform values, providing effectively no signal. The reverse min-max normalization in Equation 7 is presented as a mitigation, not a solution: it amplifies the signal to make the reward discriminative, but it does not address the underlying vulnerability that the student could learn to exploit the reward metric rather than internalize teacher knowledge. This is evidenced in Table 5(e): removing the accuracy reward entirely (using only R_distill) produces performance consistently below the large teacher's own performance (86.3% on AI2D vs. the teacher's 87.8%; 80.3% on MathVista vs. 81.9%), confirming that distillation-reward-only optimization leads to reward hacking—the student achieves high distillation scores but produces weaker actual outputs.

The reason this constitutes a genuine diagnostic contribution (not just a training trick) is that it reveals a fundamental tension in RL-based distillation that prior work had not articulated: any distillation reward that measures alignment between teacher and student distributions is, by construction, a proxy for the true objective of knowledge transfer. The better the student becomes at matching the teacher's output patterns, the more it can potentially exploit the reward metric by generating outputs that score well on alignment without necessarily being correct or useful. The accuracy reward serves as a counter-balance—it ties the optimization to an external correctness criterion that the student cannot game through distribution-matching alone—which is why R_acc + R_distill substantially outperforms R_distill alone.

This insight has practical implications for future RL-based distillation work: reward design must include at least one non-proxy objective (like accuracy against ground truth) that is immune to representation-matching exploitation. It also explains why the paper's dual-reward design is not merely "two rewards are better than one"—it is a specific architectural choice to prevent a specific failure mode that the paper has diagnosed.

Innovation 3: Offline RL is Not a Compromise—It is an Enabling Design Choice for Scalable Distillation

The paper makes a strong—and likely controversial—argument that offline RL is not merely a computational convenience but a methodologically superior approach to RL-based distillation at scale. This runs counter to the prevailing trend in the field, where online RL paradigms inspired by DeepSeek-R1's "think-answer" process are treated as the gold standard for pushing model capabilities through reinforcement signals.

The conventional wisdom holds that online RL is superior because the model generates responses during training, receives feedback, and iteratively refines its policy in a closed loop. This enables the model to explore its own evolving distribution and receive rewards on self-generated outputs, which in principle should lead to more adaptive learning. The paper does not dispute that online RL could, in theory, produce better final performance—it acknowledges that "in principle, training could continuously improve the model via online data sampling."

However, the paper articulates a practical ceiling that transforms this theoretical advantage into a real-world disadvantage: the computational cost of online RL at the scale needed for effective VLM distillation (1.5M training samples, large teacher models) is so prohibitive that it renders the approach infeasible. The paper's estimate—over 30 days on 256 A100 GPUs for online RL vs. two days for offline—reveals a two-orders-of-magnitude efficiency gap that no amount of theoretical advantage can overcome in practice.

But the paper goes further than a cost argument. It identifies a subtler issue: the data diversity ceiling in online RL. Because online RL is slow, practitioners limit training to small datasets (the paper cites DeepSeek-R1 and its variants as using "a very limited amount of data samples"). This creates a tension: the RL process may be adaptive, but it adapts to a severely restricted data distribution, reducing generalization. Masters' offline approach, by pre-generating responses once for all 1.5M samples, achieves both diversity (1.5M examples from many domains) and adaptation (responses generated by the student's own current policy) without the per-step generation cost.

This reframing—offline RL as an enabling technology rather than a compromise—is significant because it challenges the dominant narrative that online RL is categorically better for model improvement. The paper provides a concrete alternative: pre-generate diverse responses at scale, pre-compute rewards, and train efficiently. The results support this position: Masters-applied models achieve strong performance (e.g., InternVL3.5-8B at 80.0% average, surpassing many larger models) while training in two days, making the approach genuinely deployable.

The evidence is embedded in Figure 6, which shows that Masters-applied models maintain the inference speed of their base student architectures, while "think-answer" models require substantially more inference time due to lengthy reasoning chains. This closes the loop: Masters achieves competitive or superior performance without the inference-time penalty that online RL approaches impose, making it not just faster to train but faster to deploy.

Innovation 4: Progressive Capacity Alignment Across Teacher Sizes is a Distinct and Complementary Scaling Dimension

The paper distinguishes between two forms of progressive capacity alignment that are often conflated: mask-progressive distillation (gradually unmasking weights within a single teacher) and teacher-size scaling (distilling from progressively larger teacher models, e.g., 14B → 38B → 78B). The finding that these are complementary rather than redundant is a non-obvious empirical result with implications for how distillation pipelines should be designed.

The naive assumption would be that mask-progressive distillation from the largest available teacher should subsume teacher-size scaling—after all, masking the 38B teacher at r = 0.2 effectively reduces its capacity to something approximating what a smaller teacher might provide, and progressively unmasking it covers the full capacity range. If this assumption held, adding intermediate teachers should provide no additional benefit.

The data refute this. Table 2 shows that for InternVL3.5-8B, mask-progressive distillation from the large (38B) teacher alone reaches 77.0% average; adding mid-teacher scaling (distilling from intermediate-sized models first, then the 38B teacher) before mask-progressive distillation reaches 80.0%. The 3.0 percentage point gap is substantial and consistent across model families (InternVL3.5-4B: 73.6% vs. 75.1%; InternVL3.5-2B: 71.8% vs. 75.1%). Table 5(b) further shows that using three teachers (14B → 38B → 78B) outperforms two-teacher sequences, which outperform single-teacher distillation.

Why would this be, if masking already provides a capacity spectrum? The paper does not fully answer this mechanistic question, but the result suggests that masking within a single architecture and using genuinely different architectural configurations are not equivalent forms of capacity reduction. A masked 38B model still has the depth and structural properties of a 38B model—it simply has some weights zeroed. A genuine 14B model has fewer layers, narrower hidden dimensions, and a different representational geometry entirely. The progressive transition across architectures may expose the student to qualitatively different forms of knowledge representation that masking alone cannot simulate.

This insight has practical significance for the "scalable path toward efficient and deployable VLMs" that the paper envisions. As larger teachers become available (e.g., future 100B+ VLMs), the distillation pipeline can accommodate them by extending the progressive teacher-size sequence, without needing to redesign the masking or RL components. The framework scales in a modular way: mask-progressive distillation handles within-teacher capacity alignment; teacher-size scaling handles across-architecture transitions; offline RL refines the student's responses at each stage. This modular scalability is a genuine engineering contribution—it means Masters is not a fixed recipe but a framework that can grow with the field's models.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the MATH benchmark (Hendrycks et al., 2021), consisting of high-school competition-level math problems. The paper uses the split from Lightman et al. (2022): 12,000 training questions and 500 test questions. The choice of MATH is deliberate (Section 4): test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences, which mathematical reasoning exemplifies.
Base model(s). All primary experiments use PaLM 2-S* (Codey) (Anil et al., 2023). The paper argues this model is "representative of the capabilities of many contemporary LLMs" and sits in a useful intermediate performance regime: non-trivial pass@1 on MATH (~10–19% depending on prompt and sampling configuration) but far from saturation, leaving substantial room for test-time compute to improve results. For the FLOPs-matched comparison, a second model with approximately 14× more parameters is used as the pretraining-scaled baseline.
Metrics. The primary metric throughout is MATH test accuracy (%) — the fraction of the 500 test questions for which the selected final answer matches the ground truth. Answers are graded using the grading function released by Lightman et al. (2022) (Appendix G). When analyzing difficulty-dependent behavior, the paper reports accuracy separately for each of the five difficulty quintiles.
Baselines. The paper compares against several standard and prior-art approaches:
- Majority voting: the most common final answer among N independently sampled solutions, with no learned verifier.
- ORM best-of-N weighted: score N solutions with an outcome reward model and apply best-of-N weighted selection.
- PRM best-of-N weighted: score N solutions with the process reward model and apply best-of-N weighted selection.
- Parallel sampling (for the revision experiments): generate N independent solutions from the revision model and select the best via verifier or majority voting.
- For the FLOPs-matched comparison, the primary baseline is a model with approximately 14× more parameters using greedy decoding with no additional test-time compute.
Generation budget / compute accounting. The universal unit of test-time compute is one "generation," defined as one complete sampled answer from the base LLM. For best-of-N, the budget equals N. For beam search, the budget equals the number of beams N. For lookahead search with k lookahead steps, the cost is N × (k+1) to account for the additional rollout computation (Section 5.3). Budgets are swept across powers of 2, typically from 2⁰ (1 generation) to 2⁹ (512 generations).
Cross-validation / statistical protocol. To avoid contaminating the compute-optimal strategy selection with test-set performance, the paper uses two-fold cross-validation within each difficulty bin on the 500-question test set (Section 3.2). The best-performing strategy is selected on one fold and evaluated on the other, with results averaged. Difficulty bins are computed once and treated as fixed for the cross-validation splits.

Main Quantitative Results

Search Against PRM Verifiers

The headline result for verifier-guided search (Section 5, Figure 3) establishes that the choice of search algorithm matters substantially at low compute budgets but the advantage of sophisticated search diminishes or reverses at high budgets due to PRM over-optimization.

Aggregate comparison (Figure 3, left). Across all 500 test questions with maximum budget 256 generations:

At low budgets (2–8 generations), beam search with M = 4 significantly outperforms best-of-N weighted. At 4 generations, beam search (M = 4) achieves roughly 27% accuracy versus roughly 16% for PRM best-of-N weighted — a gap of approximately 11 percentage points. This demonstrates that when the compute budget is tight, search that intelligently prunes unpromising partial solutions is substantially more effective than simply sampling complete solutions independently.
At high budgets (64–256 generations), the advantage reverses. Beam search performance flattens around 34% at 512 generations, while PRM best-of-N weighted continues to improve, reaching approximately 38% at 512 generations. The paper attributes this to over-optimization of the PRM: as beam search explores the space more aggressively, it finds solutions that score highly under the PRM but are actually incorrect, exploiting weaknesses in the verifier signal.
Lookahead search (both k = 1 and k = 3) generally underperforms all other methods at the same generation budget. Its higher per-step cost means fewer beams can be explored for a given budget, and the additional lookahead computation does not compensate for this reduced exploration breadth. The 3-step lookahead variants converge to similar performance as other methods at very high budgets but never surpass them.
Majority voting substantially trails all verifier-based methods, reaching only about 29% at 512 generations. This gap (roughly 9 points vs. best-of-N weighted) quantifies the value that the learned PRM provides over simple consensus-based selection.

Difficulty-dependent behavior (Figure 3, right). When results are disaggregated by difficulty quintile (beam search M = 4 vs. PRM best-of-N weighted, shown at four budget levels: 4, 16, 64, 256 generations), a striking pattern emerges:

Bin 1 (easiest questions): Beam search degrades with increasing budget — accuracy decreases from roughly 78% to 77% as budget grows from 4 to 256, while best-of-N weighted increases from roughly 68% to 88%. This is the clearest empirical signature of PRM over-optimization: on problems where the base model already produces many correct solutions, aggressive search amplifies whatever residual errors exist in the verifier signal, causing the system to select verifier-pleasing but incorrect solutions.
Bin 2: A similar but less extreme pattern holds. Beam search improves modestly (roughly 14% → 32%) but best-of-N weighted improves faster (roughly 14% → 60%), maintaining a clear advantage at high budgets.
Bin 3: The behavior inverts. Beam search consistently outperforms best-of-N weighted across all budgets, reaching roughly 34% vs. 23% at 256 generations. On these medium-difficulty problems, the PRM's guidance genuinely helps navigate toward correct solutions the model wouldn't find through random sampling.
Bin 4: Beam search shows its strongest relative advantage, reaching roughly 17% vs. 10% for best-of-N at 256 generations. The gap is proportionally large (a ~70% relative improvement), though absolute accuracy remains low.
Bin 5 (hardest): Both methods hover near 1–3% accuracy regardless of budget. No method makes meaningful progress on problems fundamentally outside the base model's capability range. This is the zero-return regime for test-time compute: if pass@1 is near zero, even optimal search cannot find correct solutions that do not exist in the proposal distribution.

Compute-optimal search (Figure 4). By selecting the best search strategy per difficulty bin at each budget level (using the cross-validation protocol):

At 16 generations, compute-optimal (oracle bins) achieves approximately 27% accuracy, roughly matching PRM best-of-N weighted at 64 generations — a ~4× compute efficiency gain.
At 256 generations, compute-optimal oracle reaches approximately 39.5%, surpassing PRM best-of-N weighted at the same budget (roughly 37%).
Compute-optimal with predicted difficulty bins (using PRM final-answer scores averaged over 2048 samples, without ground-truth labels) tracks the oracle version closely, with the two curves "largely overlapping" per the authors. At 256 generations, the predicted variant reaches approximately 37%, only modestly below the oracle variant.
Both compute-optimal variants consistently outperform ORM best-of-N weighted (peaking around 34% at 512 generations) and majority voting (around 29%), confirming that the gains come from difficulty-adaptive strategy selection, not just from having a better verifier.

PRM vs. ORM comparison (Appendix F, Figure 14). At 2048 samples, PRM best-of-N weighted achieves approximately 40% accuracy versus roughly 35% for ORM best-of-N weighted and roughly 30% for majority voting. The PRM-ORM gap widens with the number of samples, confirming that the step-level training provides scaling benefits beyond what an outcome-level verifier achieves, even though the PRM uses last-step aggregation (effectively behaving like an ORM at selection time).

Revision Model Results

The headline result for iterative revisions (Section 6) is that sequential revision chains are most effective on easy problems, while a balanced mix of sequential and parallel sampling is optimal on harder problems, yielding up to ~4× compute efficiency gains over purely parallel sampling.

Sequential vs. parallel aggregate comparison (Figure 6). At 64 total generations:

Sequential + best-of-N weighted: approximately 41.5%
Parallel + best-of-N weighted: approximately 39%
Sequential + majority: approximately 38%
Parallel + majority: approximately 35%

Sequential revision marginally outperforms parallel sampling under both selection mechanisms (verifier-based gap: ~2.5 points; majority-based gap: ~3 points). This aggregate comparison masks substantial difficulty-dependent variation, explored next.

Difficulty-dependent sequential-to-parallel ratio (Figure 7). The paper sweeps the ratio of sequential revisions to parallel chains at a fixed total generation budget, varying from fully parallel (no revisional refinement) to fully sequential (one long revision chain). At 128 generations:

Bin 1: Performance is essentially flat at 90–92% across all ratios — easy questions are insensitive to how the budget is allocated because the model reliably produces correct answers regardless.
Bin 2: Slight advantage for higher sequential ratios, approximately 63% at fully sequential vs. 58% at fully parallel. Easy-to-medium questions benefit modestly from within-chain refinement.
Bin 3: A clear optimal intermediate ratio emerges at moderate sequential-to-parallel values (around 2¹ to 2³, meaning 2:1 to 8:1 sequential-to-parallel), reaching approximately 42% vs. 35% at the extremes. The U-shaped curve demonstrates that neither pure exploration (parallel) nor pure exploitation (sequential) is optimal — the best strategy combines the diversity benefits of parallel sampling with the refinement benefits of sequential revision.
Bin 4: Similar pattern with peak at moderate ratio achieving roughly 18% vs. 14% at fully parallel. The absolute performance is lower, but the shape of the benefit is consistent.
Bin 5: All ratios produce roughly 2–3% accuracy. Revision, like search, provides no benefit on problems where the base model cannot generate correct solutions at any significant rate.

Compute-optimal revisions (Figure 8). Selecting the optimal sequential-to-parallel ratio per difficulty bin:

At 64 generations, compute-optimal oracle achieves approximately 40%, matching parallel best-of-N weighted at 256 generations — a ~4× compute reduction.
At 256 generations, compute-optimal oracle reaches approximately 44%, compared to roughly 41% for best-of-N weighted and 37% for parallel-only.
Compute-optimal predicted bins perform slightly below oracle bins at high budgets (approximately 41% at 256 generations vs. 44% for oracle), but still substantially outperform the parallel baseline.
Notably, the parallel baseline appears to plateau around 36–37% at high budgets, while compute-optimal scaling continues to improve. This suggests that adaptive allocation's benefits compound: as budgets grow, the inefficiency of uniform strategies becomes more pronounced, and the gains from difficulty-conditional allocation increase.

Correct-to-incorrect reversion rate. The paper reports that approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step (Section 6.1). This is a direct consequence of the training data construction: the model only sees incorrect-to-correct trajectories during training and has no signal for what to do when the current answer is already correct. The paper mitigates this with chain-wide selection (majority voting or verifier-based best-of-N weighted across all steps in the chain), which salvages correct answers even when they are later overwritten by incorrect revisions.

FLOPs-Matched Comparison: Test-Time vs. Pretraining Compute

The headline result from Section 7 (Figure 9, Figure 1 bar charts) is that a smaller model with compute-optimal test-time strategies can outperform a ~14× larger model on easy-to-medium problems when the inference-to-pretraining token ratio R is low, but this advantage reverses on hard problems at high R.

Experimental setup. The comparison pits PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters using greedy decoding with no additional test-time compute. Three regimes of R = D_inference / D_pretrain are tested: R ≪ 1 (0.16), R ≈ 1 (0.79), and R ≫ 1 (22). R ≪ 1 corresponds to scenarios where most compute is spent on pretraining (self-improvement pipelines, one-time evaluation). R ≫ 1 corresponds to high-throughput production deployments where inference cost dominates. The FLOPs accounting uses standard approximations from scaling laws: X = 6ND_pretrain for pretraining and Y = 2ND_inference for inference.

Results for revisions (Figure 9, left; Figure 1, top-right bar chart), values from the bar chart in Figure 1:

Difficulty	R ≪ 1 (0.16)	R ≈ 1 (0.79)	R ≫ 1 (22)
Easy (bin 1)	+11.8%	+3.5%	−11.9%
Medium (bins 2–3)	+27.8%	+16.7%	+5.4%
Hard (bins 4–5)	+21.6%	not stated explicitly	−37.2%

At R ≪ 1, test-time compute outperforms the larger model across all difficulty levels, by substantial margins on easy and medium questions. The pretraining savings from using a smaller model are so large relative to inference volume that the smaller model can afford extensive test-time computation. At R ≫ 1, test-time compute only remains preferable on easy questions; hard questions show a −37.2% relative disadvantage. The inference cost of running large test-time budgets dominates, and the larger model's pretrained capabilities become more valuable.

Results for PRM search (Figure 9, right; Figure 1, bottom-right bar chart):

Difficulty	R ≪ 1 (0.16)	R ≈ 1 (0.79)	R ≫ 1 (22)
Easy	+19.1%	+2.2%	+2.0%
Medium	0.0%	−35.3%	−30.8%
Hard	−3.6%	−35.3%	−52.9%

PRM search shows weaker benefits than revisions for the FLOPs-matched comparison, with substantial disadvantages on medium and hard questions even at moderate R values. On easy questions, test-time compute remains preferable across all R regimes, though the margin narrows from +19.1% at R ≪ 1 to only +2.0% at R ≫ 1. The search-based approach appears less FLOPs-efficient than the revision-based approach, likely because search's computational overhead (per-step scoring, beam maintenance) does not translate to commensurate accuracy gains in a FLOPs-matched comparison against the larger model.

The sharp boundaries (Figure 9 detail). The line plots show accuracy per difficulty bin as test-time compute scales, with the 14× larger model's greedy performance (stars) placed at three x-axis positions corresponding to the three R values. On bin 1 (purple, topmost line for both revisions and search), the scaling line is above all three stars for revisions and above all three for search — test-time compute wins for easy questions unconditionally. On bin 5 (blue, bottommost line), the line is consistently below all three stars and essentially flat near 0–5% — no amount of test-time compute helps on the hardest problems for either method. The intermediate bins show the conditional pattern: test-time compute wins at low R, loses at high R.

Ablation Studies and Robustness Checks

PRM aggregation strategy (Appendix E, Figure 13). Comparing "min," "prod," and "last" step-wise score aggregation methods: "last" achieves approximately 37% at 256 samples, "min" achieves approximately 35%, "prod" achieves approximately 27%, and a separately trained ORM achieves approximately 34%. The superiority of "last" aggregation is notable because it effectively reduces the PRM to ORM-like behavior at aggregation time — only the final step's score matters for selection — yet the PRM still performs better than a separately trained ORM. The authors interpret this as evidence that step-level PRM training provides beneficial representation learning that carries over to the final-step prediction even when intermediate-step scores are not directly used. This finding contradicts prior work (Lightman et al., 2023; Wang et al., 2023) that found "min" to be the best aggregation method, and the paper hypothesizes that the discrepancy arises because their PRM was trained with soft Monte Carlo labels rather than binary correctness labels.

PRM vs. ORM (Appendix F, Figure 14). The PRM consistently outperforms the ORM, with the gap widening at higher sample counts. At 2048 samples, PRM best-of-N weighted reaches approximately 40% vs. ORM's 35% and majority voting's 30%. This validates that the PRM's per-step training provides genuine advantages for test-time scaling, even with last-step aggregation that makes it functionally ORM-like at selection time.

Revision model verifier choice (Appendix J, Figure 15a). The base-LM PRM underperforms a revision-specific ORM when scoring revision model outputs. At 64 generations, sequential revision + base-LM PRM achieves roughly 40% vs. sequential + revision ORM at roughly 42%. This confirms distribution shift as a practical concern: the revision model's output distribution differs enough from the base model's that a verifier trained on base model outputs transfers imperfectly. The paper trains a separate ORM on revision model outputs to address this.

Revision history in verifier context (Appendix J, Figure 15b). Including previous revisions in the ORM's context provides a small improvement (approximately 1–2 percentage points at 64 generations) over the no-history ablation, but both variants outperform the parallel baseline. This confirms that the sequential revision benefit arises primarily from the revision process itself (improving answer quality through iterative refinement), not merely from giving the verifier more context to make better selection decisions.

Oracle vs. predicted difficulty bins (Figures 4, 8, and Appendix C, Figures 11–12). Both oracle and predicted bins yield qualitatively similar trends across all difficulty levels. Predicted bins show slightly lower performance at high budgets in the revision setting (approximately 41% vs. 44% at 256 generations in Figure 8) but essentially identical performance in the search setting (Figure 4). This is the critical robustness check for deployability: the compute-optimal strategy works without ground-truth labels, using only the PRM's own score distribution as a difficulty proxy. The gap between oracle and predicted bins in the revision setting is modest and does not change the qualitative conclusion that adaptive allocation outperforms uniform strategies.

Majority voting for revisions (Appendix B, Figure 10). The sequential-to-parallel ratio trends observed with verifier-based selection are replicated with majority voting: easy questions are insensitive to ratio; hard questions show an optimal intermediate ratio; fully sequential marginally outperforms fully parallel in aggregate. This confirms that the revision process itself generates better answers (not just answers better scored by the verifier), since the same qualitative patterns hold even when using a simple, verifier-free selection mechanism.

ReST^EM revision model (Appendix K, Figure 16). An attempt to further optimize the revision model using ReST^EM (Singh et al., 2024) — an iterative RL-based self-improvement approach — backfires substantially. With the ReST^EM-trained revision model, fully sequential revisions at 256 generations drop to approximately 33.5% compared to roughly 38.5% at the optimal ratio. The authors hypothesize that the on-policy data collection in ReST^EM amplifies spurious correlations in the revision trajectories, causing the model to fail to learn the revision task properly. This is a notable negative result that highlights the sensitivity of revision model training to the data generation procedure: the offline edit-distance-based pairing used in the main experiments is specifically designed to avoid the distributional issues that ReST^EM's online approach introduces.

Difficulty estimation cost. The paper acknowledges (Section 3.2) that the predicted difficulty bin estimation — generating 2048 samples per question and scoring them with the PRM — incurs additional computation that is not accounted for in the reported efficiency gains. The 2048-sample estimation step is more expensive than the largest test-time budgets studied (256–512 generations), meaning that the true cost of deploying compute-optimal scaling would be difficulty estimation + strategy execution. The reported 4× gains are computed assuming difficulty is known, without amortizing the cost of learning it. This is flagged as a key direction for future work but is not resolved in the current experiments.

Critical Assessment

Claim 1: Compute-optimal scaling improves efficiency by more than 4× over best-of-N. The experiments demonstrate that at specific budget points, compute-optimal allocation achieves accuracy equivalent to best-of-N with 4× fewer generations (e.g., 16 generations matching 64 for search in Figure 4; 64 generations matching 256 for revisions in Figure 8). This is genuine and replicable across both oracle and predicted difficulty settings. However, three qualifications are important:

First, the 4× figure is a point estimate — not a uniform property across all budget levels. At the highest budgets (256–512 generations), the gap narrows somewhat, particularly with predicted difficulty bins in the revision setting (41% vs. 44% oracle, Figure 8). The efficiency gain is most reliable in the low-to-moderate compute regime.

Second, the difficulty estimation cost is unaccounted for. The predicted difficulty method uses 2048 samples per question, which is 8–32× the largest test-time budgets being optimized. If this cost were amortized into the efficiency calculation, the practical gains would be substantially smaller, potentially eliminating the advantage entirely for one-off queries. The paper is transparent about this (Section 3.2) but the magnitude of this unaccounted cost means the 4× figure should be understood as an upper bound on achievable efficiency in a deployment setting, not a realized gain.

Third, the predictions of difficulty are made once and cached, which is feasible for a fixed benchmark evaluation but not for a production system with a continuous stream of novel queries. The paper does not explore how difficulty estimation would work in a streaming setting or how estimation quality degrades with fewer than 2048 samples.

Claim 2: Test-time compute with a smaller model can outperform a 14× larger model. The evidence for this claim is solid but sharply conditional. The claim holds convincingly at R ≪ 1 for easy-to-medium problems (Figure 9, Figure 1), where the smaller model enjoys substantial test-time budgets funded by pretraining savings. At R ≫ 1, the claim fails on medium and hard problems, with disadvantages of −30.8% to −52.9% relative to the larger model.

Several design choices make this comparison more favorable to test-time compute than it might be in a fully rigorous evaluation:

The 14× larger model uses greedy decoding only — no test-time compute budget of its own. A fair comparison of "test-time compute as a substitute for pretraining" should give the larger model an equal test-time compute budget (even a modest one like best-of-8) to see if the efficiency advantage of test-time compute persists when both models receive inference-time optimization. The current comparison is "small model with extensive test-time optimization vs. large model with none," which stacks the deck in favor of the smaller model.
The 14× larger model scales parameters only, not data — it is not Chinchilla-optimal. The paper follows the LLaMA paradigm (Touvron et al., 2023) of scaling parameters while keeping training data fixed. A compute-optimally trained larger model (scaling both parameters and data per Hoffmann et al., 2022) would likely be stronger, making the baseline artificially weak relative to what the same total pretraining FLOPs could achieve.
The scaling factor of ~14× is not systematically varied. Only one size ratio is tested, so the paper cannot characterize how the break-even point between pretraining and test-time compute varies with the size gap. It is possible that test-time compute wins against 14× but would lose against 5× or 50×, and the paper provides no data to assess this.

Claim 3: Efficacy depends critically on prompt difficulty. This is the most robust finding in the paper. The difficulty-bin analyses in Figures 3 (right) and 7 (right) show qualitatively different — and sometimes opposite — effects of the same strategy at different difficulty levels. Beam search helps on medium problems and hurts on easy ones (Figure 3 right). Sequential revisions are best on easy problems, balanced ratios are best on hard ones (Figure 7 right). These are replicated in both search and revision settings and with oracle and predicted difficulty bins.

The critical limitation is that difficulty is measured relative to the base model, not as an intrinsic property of the question. The same question could be "easy" for PaLM 2-S* and "hard" for a different model. The compute-optimal policy is therefore model-specific, and the paper provides no method for transferring difficulty estimates across model families. This limits the practical deployability: each base model would require its own difficulty estimation procedure.

Additional weaknesses in experimental design:

Single benchmark, single model family. All results are on MATH with PaLM 2-S*. The paper's claim that the model is "representative" is unverified, and there is no evidence that the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems) generalize to other reasoning domains, other model architectures, or models with different calibration properties.
Small test set for strategy selection. The 500-question test set is split into five difficulty quintiles of ~100 questions each, then further split by two-fold cross-validation. The compute-optimal strategy per-bin is thus selected based on ~50 questions, which is a small sample that could introduce variance in which strategies appear optimal. The paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess the statistical reliability of the gains.
Search and revisions are never combined. The paper establishes that revisions help on easy problems and search helps on medium problems, but never tests whether using the revision model as the proposal distribution within beam search yields further gains. This is an acknowledged limitation (Section 8) and means the current results are a lower bound on what combined approaches could achieve.
No dynamic difficulty estimation. Difficulty estimation uses a batch of 2048 samples computed once, with no mechanism for updating the difficulty estimate or adjusting the strategy mid-computation. A dynamic approach — start with a few samples, estimate difficulty from initial results, allocate remaining budget accordingly — could subsume the difficulty estimation cost into the solution process and potentially outperform the static binning approach. This is not explored.
The revision model has a 38% correct-to-incorrect reversion rate, which is mitigated by chain-wide selection but not fundamentally solved. This means the revision process actively degrades correct answers in a substantial fraction of cases, introducing a reliability concern that chain-wide selection can only partially address. A more principled solution is not provided.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Cost is Unaccounted for in Headline Efficiency Claims

The assumption or constraint. The entire compute-optimal framework rests on the ability to estimate each question's difficulty before allocating the inference budget. The method the paper uses for this—generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted)—is extraordinarily expensive relative to the budgets being optimized. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The 2048-sample estimation step consumes more compute than the largest test-time budgets studied (256–512 generations), by a factor of 4–8×. In a deployment setting, the total cost would be difficulty_estimation_cost + strategy_execution_cost, and the former would dominate.

The consequence. The reported 4× efficiency gains over best-of-N (e.g., 16 generations matching 64 in Figure 4; 64 generations matching 256 in Figure 8) are computed after difficulty is known, without amortizing the cost of learning it. For a one-off query, the true cost comparison would be 2048 + 16 = 2064 generations for compute-optimal vs. 64 generations for best-of-N—a ~32× increase in total compute, not a savings. The 4× figure is therefore an upper bound on efficiency that applies only when difficulty estimates can be reused across many queries from the same distribution (e.g., when evaluating a fixed benchmark like MATH), but not to individual novel queries.

This limitation has differential impact depending on the deployment scenario:

Fixed benchmark evaluation (as in the paper's experiments): Difficulty is estimated once per question and cached, so the amortized cost per query is modest. The 4× figure is approximately correct in this setting, as long as the evaluation set is small and static.
High-volume production deployment with a stable query distribution: Difficulty could theoretically be estimated offline on a representative sample, and a classifier trained to predict difficulty from question text alone (a direction the paper suggests for future work but does not implement). Until such a classifier exists, the cost of per-query estimation is prohibitive.
One-off or low-volume queries: The full 2048-sample cost must be paid per query, making the approach strictly less efficient than standard best-of-N for any reasonable budget.

What evidence exists in the paper. The paper itself provides no measurement of difficulty-estimation cost integrated into total compute budgets. The 2048-sample figure appears in Section 3.2 as a design choice, and the cost is acknowledged as unaccounted for, but no experiment quantifies how the efficiency claims change when estimation cost is included. The close overlap between oracle and predicted difficulty curves in Figures 4 and 8 demonstrates that the PRM-based difficulty estimate works as a proxy for ground-truth difficulty, but does not address its cost.

Mitigation status. The paper explicitly flags this as "a key avenue for future work" (Section 3.2) and suggests training a model to predict difficulty directly from question text without requiring 2048 samples. No such model is developed or evaluated. There is no exploration of whether fewer than 2048 samples (e.g., 32, 64, 128) could provide sufficient difficulty signal, which would be the most immediate practical mitigation. The limitation is acknowledged but entirely unresolved—it represents the single largest gap between the paper's reported results and what a practitioner would experience when deploying the method.

Limitation 2: Hard Problems Are Fundamentally Unsolvable by Test-Time Compute Alone

The assumption or constraint. The paper's approach assumes that the base model can produce correct solutions at some non-trivial rate—that the proposal distribution covers the correct answer with pass@1 > ~0. When this condition fails, no amount of search, revision, or compute-optimal allocation can help, because there are no correct solutions in the generation space to find or refine. Section 5.3 makes this clear:

"On the hardest questions (bin 5), no method makes meaningful progress—the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated."

This is not a bug in the method but a fundamental boundary condition: test-time compute amplifies existing capability but does not create new capability. The base model's pretraining determines the ceiling.

The consequence. For any problem distribution that includes a substantial fraction of items outside the base model's capability range, the overall accuracy achievable with test-time compute is strictly bounded by the model's pass@1 coverage. In the paper's experiments:

Difficulty bin 5 shows 1–3% accuracy across all methods, all budgets, and all strategies (Figures 3 right, 7 right, and the FLOPs-matched comparison in Figure 9).
No scaling trend is visible—increasing the budget from 4 to 256 generations produces essentially flat accuracy in bin 5 for both search and revisions.
The FLOPs-matched comparison (Figure 9) shows the small model with test-time compute suffering −52.9% relative disadvantage vs. the 14× larger model on hard problems for PRM search at R ≫ 1.

This means that for genuinely novel, out-of-distribution, or highly complex reasoning problems—exactly the problems where one most wants help from compute scaling—the approach provides zero benefit. The paper's method can make an already-capable model more reliable on problems it can sometimes solve, but cannot extend the model's capability frontier. This is a meaningful practical limitation: a production system using test-time compute scaling would need a separate mechanism (routing to a larger model, escalating to human review) for the hardest subset of queries.

What evidence exists in the paper. The evidence is comprehensive and consistent across all experimental sections:

Figure 3 (right): Bin 5 search accuracy at 1–3% for all methods and budgets
Figure 7 (right): Bin 5 revision accuracy at 2–3% regardless of sequential-to-parallel ratio
Figure 9: The bin 5 scaling line is essentially flat near 0–5% for both revisions and search, and lies below the 14× larger model's performance at all R values
Table 1 in prior sections shows base model pass@1 rates of ~10–19%, meaning that even on the MATH benchmark (curated to be within the model's broad capability range), 20% of questions fall into bins 4–5 where gains are minimal to zero

Mitigation status. The paper is transparent about this limitation—Section 7 explicitly concludes that "test-time compute can amplify existing capability but does not create it from nothing." The FLOPs-matched analysis quantifies precisely where the boundary lies (test-time compute wins on easy-to-medium, loses on hard). However, the paper provides no mechanism for identifying a priori which problems will fall into the unsolvable bin 5 before expending the difficulty-estimation budget, meaning that compute may be wasted attempting to estimate difficulty and optimize strategy for problems where the answer is guaranteed to be wrong regardless. This is a practical deployment concern that the difficulty estimation framework does not address.

Limitation 3: Single Benchmark and Single Model Family Limit Generality

The assumption or constraint. All experiments in the paper use exactly one benchmark (MATH, high-school competition math problems) and one base model family (PaLM 2-S*). The authors state in Section 4:

"We believe this model is representative of the capabilities of many contemporary LLMs"

This claim of representativeness is asserted but not tested. There is no evidence that the difficulty-dependent patterns—beam search degrading easy-problem performance, revisions helping easy problems, the optimality of adaptive allocation—generalize to other reasoning domains, other model architectures, or models with substantially different calibration properties.

The consequence. The paper's key findings may be specific to the MATH benchmark's characteristics or to PaLM 2-S*'s particular output distribution, and may not transfer to other settings where:

The task format differs. MATH problems have deterministic, single-token correct answers (a number or expression) that can be verified with string matching. Tasks requiring open-ended generation, multi-step planning, or subjective evaluation lack clean correctness signals, making both the PRM training pipeline (which requires Monte Carlo rollout correctness labels) and the difficulty estimation procedure (which requires pass@1 rates) inapplicable in their current form.
The error modes differ. MATH errors tend to be logical or computational mistakes in multi-step derivations. A model applied to factual QA might produce errors from knowledge gaps rather than reasoning failures, which could interact differently with search and revision strategies. The paper's finding that revisions work well on easy problems assumes that errors are refinement-correctable—this may not hold when errors stem from missing knowledge rather than reasoning slips.
The model family behaves differently. PaLM 2-S*'s calibration (how well its output probabilities reflect correctness), its error correlation structure (whether errors are systematic or random), and its in-context learning capability all affect the efficacy of search, revision, and difficulty estimation. A model with different properties—say, one that is better calibrated but less creative in its sampling—might show different difficulty-dependent scaling curves or different optimal strategy allocations.
The difficulty distribution differs. MATH has a specific difficulty distribution shaped by competition problem curation. A benchmark with a different proportion of easy/medium/hard questions would shift the aggregate benefit of adaptive allocation, even if the per-difficulty-bin behaviors were identical.

What evidence exists in the paper. None. There are no experiments on other benchmarks (e.g., GSM8K for math reasoning, HumanEval for code generation, MMLU for knowledge-intensive QA) and no experiments with other model families. The paper's generalization claims rest entirely on the assertion that PaLM 2-S* is "representative" and that the difficulty-dependent patterns are driven by fundamental properties of test-time compute rather than by idiosyncrasies of the MAT H benchmark or the PaLM 2 architecture. The paper's related work section cites other models and benchmarks but does not test against them.

Mitigation status. The paper does not address this limitation directly. No experiments with other benchmarks or model families are suggested as future work, nor is there a discussion of which aspects of the findings are most likely to be model- or benchmark-specific vs. universal. A practitioner considering applying these techniques to a different model on a different task would need to replicate the full experimental pipeline (difficulty estimation, strategy sweep, compute-optimal policy selection) to determine whether the approach transfers, which is expensive and may not be feasible for all deployment contexts.

Limitation 4: The 14× Larger Model Baseline is Weakened by Design Choices

The assumption or constraint. The FLOPs-matched comparison in Section 7 that demonstrates test-time compute outperforming pretraining uses a baseline with specific design choices that make the pretraining case weaker than it could be:

Parameters-only scaling, not Chinchilla-optimal. The 14× larger model scales only parameter count while keeping training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The paper acknowledges (Section 7): "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." A Chinchilla-optimal model (Hoffmann et al., 2022) that scales both parameters and data proportionally to the FLOPs budget would likely be stronger, making the "test-time compute wins" result less pronounced or potentially reversed in some regimes.
Greedy decoding with no test-time compute. The larger model uses greedy decoding only—no majority voting, no best-of-N, no search of any kind. This is an asymmetric comparison: "small model with extensive test-time optimization vs. large model with none." The paper does not test whether giving the larger model even a modest test-time compute budget (e.g., best-of-8 or best-of-16) would shift the break-even points, which is the relevant practical question for a fair assessment of the pretraining vs. inference compute tradeoff.
Single size ratio. Only one scaling factor (~14×) is tested. There is no characterization of how the advantage of test-time compute varies with the size gap—it might be largest at moderate gaps and diminish for very large or very small gaps, but the paper provides no evidence either way.

The consequence. The paper's central claim that "a smaller model augmented with compute-optimal test-time strategies can outperform a ~14× larger pretrained model" (Section 1, Executive Summary) is technically true for the specific comparison performed, but the comparison is structured in ways that systematically favor test-time compute. The practical interpretation—that organizations should prefer scaling inference compute over scaling pretraining—is weaker than the experimental setup suggests because:

A Chinchilla-optimal larger model trained with the same total FLOPs would be a stronger competitor, potentially reversing the result on medium and hard problems.
A larger model with even a small test-time compute budget (best-of-4 or best-of-8) might match or exceed the small model's compute-optimal performance at a fraction of the small model's test-time budget, since each of the larger model's generations is higher-quality.
The optimal decision between pretraining and test-time compute allocation likely depends on the specific size ratio, and the paper's single-ratio experiment cannot inform decisions at other ratios.

What evidence exists in the paper. The evidence for the limitation is in the paper's own design choices, which it acknowledges:

Section 7 explicitly notes the parameters-only scaling choice and flags Chinchilla-optimal comparison as future work.
The FLOPs accounting (Section 7) provides the formulas for pretraining and inference compute, making it clear that scaling only parameters leaves the larger model undertrained relative to what the same FLOPs budget could achieve.
The results themselves (Figure 9, Figure 1 bar charts) show that the advantage of test-time compute narrows or reverses as R increases and as difficulty increases, even against the weakened baseline—suggesting that against a stronger baseline, the advantage would shrink further.

Mitigation status. The paper is transparent about the parameters-only scaling choice but does not perform the Chinchilla-optimal comparison even as an estimate or projection. The suggestion to leave this "to future work" is reasonable but means the current results overstate the practical advantage of test-time compute over pretraining in a FLOPs-matched setting. A practitioner deciding how to allocate a total compute budget should be aware that the paper's comparison is favorable to test-time compute by construction.

Limitation 5: The Revision Model Has a Fundamental Correct-to-Incorrect Reversion Problem

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). During training, the model never sees a scenario where the current answer is already correct, and therefore never learns what to do in that case—whether to preserve the correct answer, refine it further, or leave it unchanged. The paper reports:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

This means that the revision process actively degrades correct answers in a substantial fraction of cases. The revision model, designed to improve incorrect answers, lacks the metacognitive ability to recognize when no revision is needed.

The consequence. The reversion problem has several practical implications:

Unreliable monotonicity. A user or downstream system cannot assume that running more revision steps improves quality—a later step in the chain may be worse than an earlier one. This undermines the intuition that "more revision = better," which is the natural expectation for an iterative refinement process.
Dependence on chain-wide selection. The paper mitigates the reversion problem by using majority voting or verifier-based best-of-N weighted selection across the entire chain (Section 6.1), picking the best answer from any step rather than trusting the final revision output. This means the final answer is selected from the chain, not produced by the chain's endpoint—a subtle but important distinction. The revision chain functions as a generator of candidates rather than as a refinement process that converges to a stable correct answer.
Verifier dependence. The chain-wide selection mitigation requires a verifier (PRM, ORM, or majority voting) to identify which step in the chain is best. If the verifier itself makes errors—which it does, as evidenced by the PRM over-optimization results in Section 5.3—incorrect answers may be selected even when a correct answer exists elsewhere in the chain. The reversion problem thus compounds with verifier limitations.
Inference-time overhead. Because the chain's endpoint cannot be trusted, all intermediate revisions must be stored (or at least their answers tracked) and scored, increasing memory and computation requirements during inference beyond what a simple "generate and keep last" approach would require.

What evidence exists in the paper. The 38% reversion rate is stated in Section 6.1, though the paper does not provide a detailed breakdown (e.g., does the reversion rate vary by difficulty bin? By revision step? Are some types of correct answers more susceptible to reversion than others?). Figure 6 (left) shows that the revision model's per-step accuracy improves gradually but non-monotonically—there are steps where accuracy dips—and never reaches a plateau where further revisions are consistently beneficial. The chain-wide selection approach is described as the mitigation strategy, and Figure 6 (right) shows that sequential revisions with best-of-N weighted selection outperform parallel sampling, confirming that the mitigation works in aggregate.

Mitigation status. The paper's mitigation (chain-wide selection) is a patch, not a solution. It addresses the symptom (incorrect final answers) rather than the cause (the model's inability to recognize when no revision is needed). A more principled approach—such as training the revision model with a "no revision needed" option, or incorporating a stopping criterion based on the verifier's confidence—is not explored. The ReST^EM experiment (Appendix K, Figure 16) shows that attempting to further optimize the revision model with RL-based self-improvement worsens the problem, suggesting that the reversion issue is deeply tied to the revision training data construction and may not be easily solved through additional training. For a deployment where reliability is critical, the 38% reversion rate represents a meaningful risk that the current mitigation only partially addresses.

Limitation 6: Search and Revisions Are Studied Independently, Not Combined

The assumption or constraint. The paper investigates two complementary test-time compute mechanisms—PRM-guided search (Section 5) and iterative revisions (Section 6)—as separate pipelines. The search experiments use the base PaLM 2-S* model as the proposal distribution; the revision experiments use a fine-tuned revision model. The two are never combined, meaning the paper does not test:

Using the revision model as the proposal distribution within beam search (generating steps via conditioned revision rather than independent sampling)
Using the PRM to guide which revision branches to pursue (selecting among revision candidates at each step rather than blindly generating a linear chain)
Applying search over revision trajectories rather than over independently sampled solutions

Section 8 explicitly acknowledges this gap:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The paper's findings represent a lower bound on what a fully integrated test-time compute system could achieve. The complementary strengths documented in the paper—revisions improve the proposal distribution for easy problems, PRM search improves candidate selection for medium problems—suggest that combining them could yield gains beyond either method alone:

On easy problems: A revision model generating high-quality candidates could be searched over with a PRM to find the single best answer, potentially improving upon the chain-wide selection approach currently used.
On medium problems: Using the revision model's conditioned generation (rather than independent sampling) within beam search could produce higher-quality candidate steps, improving the efficiency of search by reducing the number of beams needed to find correct solutions.
On hard problems: The combination might not help (since neither component helps individually), but the paper provides no data to confirm or refute this.

The current separation also means that the paper's compute-optimal policy selects between search and revisions as alternatives—but an optimal policy might select a combination at some budget-difficulty points, achieving better performance than either alone. The paper's difficulty-dependent analysis (search helps medium, revisions help easy) suggests a switching policy, but a blending policy might be superior.

What evidence exists in the paper. The paper provides no experiments combining search and revisions. The individual results (Figures 3–4 for search, Figures 6–8 for revisions) establish the complementary difficulty-dependent patterns, and the paper discusses their complementarity in Section 8, but no integrated experiment is performed. The compute-optimal policy in Figures 4 and 8 selects among search variants or among revision variants, never both. The FLOPs-matched comparison (Figure 9) treats search and revisions as separate options, comparing each independently against the larger model baseline.

Mitigation status. The paper acknowledges the combination as future work (Section 8) and frames the current results as establishing the groundwork for such combination. The complementary-difficulty finding—search helps most on medium problems, revisions on easy problems—provides a clear hypothesis for how a combined system should allocate compute. However, the practical challenges of combining the two are non-trivial: the revision model requires a different verifier (revision-specific ORM, as shown in Appendix J, Figure 15a) than the base model search PRM, and integrating revision-conditioned generation into tree search would require changes to both the search algorithm and the training pipeline. A practitioner seeking to maximize test-time compute effectiveness would likely want to implement such a combination, but the paper provides no guidance on how to do so or what gains to expect, leaving a significant gap between the paper's demonstrated results and the achievable ceiling.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes knowledge distillation from a student-centric problem—"how can the student learn better from a fixed teacher?"—into a teacher-centric one—"how can the teacher present itself so that the student can learn effectively?" This is not an incremental improvement in distillation objectives or architecture design. It is a conceptual inversion that challenges the field's dominant assumption: that the teacher is an immovable source of knowledge that the student must adapt to absorb.

The shift matters because it addresses the root cause of a well-documented but poorly resolved failure mode: distillation under extreme capacity gaps. When the teacher is 5–40× larger than the student (e.g., InternVL3.5-38B → 2B), the student's limited vocabulary, narrower hidden dimensions, and fewer layers make it structurally incapable of encoding the teacher's full representational complexity. Prior work recognized this instability (Zhang et al., 2023; Mirzadeh et al., 2020) but responded with student-side interventions—better divergence measures, richer feature-level supervision, or intermediate teacher assistants. These all help the student cope with the teacher's complexity without reducing that complexity itself. Masters inverts the approach: temporarily reduce the teacher's complexity through weight masking, then progressively restore it as the student's capacity grows.

The evidence that this inversion addresses a genuine root cause—not merely a training trick—is in the ablation structure of Table 1: naive distillation from a large teacher is the baseline; adding mask-progressive distillation (even without RL) yields consistent 1–2 percentage point gains across Qwen2.5-VL, Qwen3-VL, InternVL3, and InternVL3.5 model families. This gain exists independently of the reward design, confirming that teacher adaptation is a separable, standalone contribution. The fact that this adaptation can be layered on top of existing distillation methods (Table 5f shows Masters improving DistiLLM, LLaVA-KD, VLsI, and RIL) further demonstrates that it addresses a gap those methods leave open—a gap that no amount of objective-function refinement or feature-alignment sophistication can close because the failure originates in the teacher's unmodified complexity, not in how the student processes it.

The paper also resolves a latent contradiction in the RL-for-distillation literature. The community has been divided between online RL paradigms (inspired by DeepSeek-R1's "think-answer" process, which iteratively generates and refines responses during training) and offline approaches (which pre-generate data and train on fixed trajectories). The online camp argues that real-time feedback enables adaptive policy improvement; the offline camp argues that pre-generation enables scale and diversity. Masters provides a decisive empirical argument for the offline position not as a compromise, but as an enabling technology: the computational cost of online RL on 1.5M training samples with large VLM teachers is estimated at 30+ days on 256 A100 GPUs, while Masters' offline approach completes in two days—a 15× speedup that makes RL-based distillation genuinely deployable. The paper does not merely claim that offline RL is cheaper; it demonstrates that offline RL, when combined with mask-progressive distillation and dual rewards, achieves strong performance (InternVL3.5-8B reaching 80.0% average, surpassing many larger models) while training in a practical timeframe. This shifts the burden of proof: online RL advocates must now demonstrate that the theoretical benefits of online adaptation translate into practical gains that justify the 15× computational premium, rather than defaulting to "online is better" as an unquestioned assumption.

The diagnostic of reward over-optimization in distillation RL is a third landscape-shifting contribution. The paper identifies that the distillation reward R_distill—which measures logit-level alignment between teacher and student—can be gamed by the student in ways that harm genuine knowledge transfer. This is the distillation analog of reward hacking in RLHF: the student learns to produce outputs whose logit distributions match the teacher's, rather than outputs that genuinely reflect understanding. The evidence is in Table 5(e): using only R_distill without the accuracy reward produces performance consistently below the large teacher's own performance (86.3% on AI2D vs. the teacher's 87.8%), confirming that distillation-reward-only optimization leads to metric exploitation. The dual-reward design—combining a proxy-based transferability reward with a ground-truth accuracy reward—is not merely "two rewards are better than one." It is a specific architectural response to a specific failure mode that the paper has diagnosed. This diagnosis will influence how future RL-based distillation methods design their reward functions: any distillation reward that measures distributional alignment is, by construction, a proxy for the true objective of knowledge transfer, and must be paired with a non-proxy objective (like accuracy) that is immune to representation-matching exploitation.

The paper also establishes progressive capacity alignment as a multi-dimensional design space. Prior work treated "progressive distillation" as a single concept—typically, distilling from larger teachers in sequence. Masters decomposes this into two distinct, complementary dimensions: mask-progressive distillation (gradually unmasking weights within a single teacher) and teacher-size scaling (distilling from a sequence of genuinely different architectures, e.g., 14B → 38B → 78B). The non-obvious empirical finding is that these are not redundant. Table 2 shows that mask-progressive distillation from the largest teacher alone reaches 77.0% for InternVL3.5-8B; adding mid-teacher scaling before mask-progressive distillation reaches 80.0%. The 3-point gap suggests that masking within a single architecture and transitioning across genuinely different architectures are qualitatively different forms of capacity alignment—masking removes weights but preserves the teacher's depth and structural properties; a genuine 14B model has fewer layers, narrower hidden dimensions, and a different representational geometry entirely. This decomposition gives future work a more precise vocabulary for designing distillation curricula, rather than treating all forms of "starting simple and getting complex" as interchangeable.

The practical impact—less glamorous but arguably more important—is that Masters provides a deployable recipe for on-device VLM distillation. The paper demonstrates that compact VLMs (2B–8B parameters) trained with Masters can match or exceed models 3–10× their size: InternVL3.5-2B reaches 75.1% average (exceeding LLaVA-OneVision-7B at 69.9% and the base InternVL3.5-2B at 68.6%); Qwen3-VL-4B reaches 77.8% (matching or exceeding 7–8B open-source models); InternVL3-8B reaches 82.0% on MMStar and 74.0% on MMMU (Table 3). These are not asymptotic gains visible only at unreachable compute scales—they are realized with two days of training on commodity A100 GPUs, using a framework that is architecturally simple (weight masking + offline RL) and scalable to future larger teachers. For organizations deploying VLMs on mobile devices, edge hardware, or cost-sensitive cloud instances, this represents a practical, immediately actionable pathway to closing the capability gap between what can be trained and what can be deployed.

Follow-Up Research This Work Enables

Cheap, deployment-ready difficulty estimation for compute-optimal allocation. The paper's difficulty estimation uses 2048 samples per question—8–32× the largest test-time budgets being optimized. This cost is unaccounted for in the reported 4× efficiency gains and makes the approach impractical for non-cached queries. The paper explicitly calls for "pretraining or finetuning models to directly predict difficulty of a question" (Section 3.2). A strong follow-up would train a lightweight classifier—distilled from the PRM's score distribution on the 2048-sample estimates—that takes only question text as input and predicts the difficulty quintile. If such a classifier achieved even 80% bin accuracy (meaning the selected strategy is occasionally suboptimal but rarely catastrophically wrong), the amortized difficulty estimation cost would drop from 2048 generations to a single classifier forward pass. A definitive negative result—showing that question-text features are insufficient for difficulty prediction and that the 2048-sample procedure captures essential model-specific calibration information—would be equally valuable, as it would establish that compute-optimal test-time scaling requires per-model difficulty profiling that fundamentally limits its deployability to static benchmarks.

Combined search-and-revision test-time compute. The paper studies PRM tree-search and iterative revisions as independent mechanisms (Sections 5 and 6, respectively), but never combines them. Section 8 acknowledges this gap. The complementary difficulty-dependent strengths—revisions help most on easy problems (bin 1–2, Figure 7 right), PRM search helps most on medium problems (bin 3–4, Figure 3 right)—suggest that a combined system might outperform either alone. The natural experiment is: use the revision model as the proposal distribution within beam search. At each step of the search tree, instead of sampling independently from the base model, condition the revision model on previous (rejected) branches as context, potentially producing higher-quality candidate steps. Alternatively, use the PRM to guide which revision branches to pursue—rather than generating a blind linear chain, score partial revisions with the PRM and allocate more budget to promising branches. The benchmark would be MATH accuracy at fixed generation budgets, comparing the combined system against the best single-mechanism approach per difficulty bin. A negative result—showing that the combination provides no benefit over independently selecting the best single mechanism per problem—would establish that the mechanisms are substitutive (each dominates in its difficulty regime, and alternating between them captures all available gains). A positive result would establish that they are complementary (each provides unique information that the other cannot access even in its suboptimal regime).

Verifier robustness under aggressive search optimization. The paper documents PRM over-optimization as the primary bottleneck limiting test-time compute scaling (Section 5.3, Figure 3 right: beam search degrades easy-problem performance at high budgets; Figure 3 left: lookahead search—the strongest optimizer—paradoxically performs worst overall). This establishes a direct parallel to reward hacking in RLHF. The natural follow-up is: can PRM training be made robust to the optimization pressure that search imposes? Specific experiments include: (a) adversarial training: sample beam-search trajectories during PRM training, identify cases where the PRM assigns high scores to incorrect solutions, and fine-tune the PRM on those hard negatives; (b) ensemble verification: train multiple PRMs with different random seeds or data splits, and aggregate their scores to reduce over-optimization variance; (c) KL-constrained search: add a penalty term to beam search that discourages solutions whose log-probability under the base model falls below a threshold, preventing search from finding adversarial solutions far from the model's typical output distribution. The success metric would be the slope of the accuracy-vs-budget curve at high budgets: a robust verifier should produce monotonically increasing accuracy as budgets grow, without the degradation that Figure 3 (right, bins 1–2) currently shows. If none of these approaches eliminate the degradation, that would establish over-optimization as a fundamental information-theoretic limit—the PRM, trained on finite data, cannot distinguish correct from incorrect solutions in regions of the output space far from its training distribution, and no amount of architectural tweaking can overcome this.

Dynamic, adaptive test-time compute allocation. The paper's difficulty estimation uses a static, pre-computed binning: 2048 samples → average PRM score → quintile → fixed strategy for that quintile. This two-phase approach (estimate difficulty, then solve) wastes compute on problems where the initial samples already reveal whether the model is capable. A dynamic alternative would: start with a small number of parallel samples (e.g., 4–8), use the PRM's score distribution on those initial samples as a real-time difficulty signal, and then allocate the remaining budget adaptively—switching to sequential revisions if the initial samples are correct (easy problem), beam search if initial samples are mixed (medium problem), or early-stopping with a low-confidence answer if initial samples are uniformly poor (hard problem). This would subsume the difficulty estimation cost into the solution process and potentially outperform the static binning approach by adapting to per-problem nuance within a difficulty quintile. The experiment would compare dynamic allocation against the paper's static compute-optimal policy at fixed total budgets (including the cost of initial exploration). The key measurement is whether dynamic allocation recovers the 4× efficiency gain without requiring the 2048-sample pre-computation step.

Cross-model, cross-benchmark replication of difficulty-dependent scaling laws. All of the paper's experiments use PaLM 2-S* on MATH. The core finding—that the optimal test-time strategy depends on problem difficulty, and that difficulty-dependent allocation yields 4× efficiency gains—may be specific to this model's calibration, this benchmark's error modes, or the interaction between the two. A systematic replication study would test: (a) multiple model families (e.g., LLaMA, Qwen, DeepSeek) at comparable capability levels to assess whether the difficulty-dependent patterns generalize; (b) multiple benchmarks (GSM8K, MMLU, HumanEval, ARC) to assess whether the patterns are specific to math reasoning or extend to factual QA, code generation, and logical reasoning; (c) the interaction between model calibration (how well log-probabilities predict correctness) and the optimal strategy. A model with better calibration might show less PRM over-optimization (because its own uncertainty estimates align with the PRM's) and therefore benefit more from aggressive search. A benchmark with open-ended answers (rather than MATH's deterministic single-token format) might show different revision dynamics because the "correct answer" is less sharply defined. This replication study would transform the paper's findings from "this works for PaLM 2-S* on MATH" to "these are general principles of test-time compute scaling"—or would identify the boundary conditions where the principles break, which is equally valuable for guiding deployment decisions.

Practical Applications and Downstream Use Cases

On-device VLM deployment with near-large-model performance. The most direct application is distilling large VLMs into compact variants that run on mobile phones, AR/VR headsets, or edge servers. The paper demonstrates that Masters-applied 2B–4B models can match or exceed the base performance of 7–8B models: InternVL3.5-2B-Masters reaches 75.1% average (Table 2), exceeding InternVL3.5-2B's base 68.6% by 6.5 points and surpassing LLaVA-OneVision-7B's 69.9%. For a mobile device manufacturer deciding which VLM to ship, this means a 2B model distilled with Masters provides 7B-level capabilities at 2B-level inference cost and memory footprint—a ~3.5× reduction in model size for equivalent performance. The two-day training time on A100 GPUs makes this economically feasible: a company can distill their proprietary large VLM into a family of compact on-device variants (2B, 4B, 8B) targeting different hardware tiers, without the multi-month training cycles that custom architecture design would require. The progressive teacher-size scaling finding (14B → 38B → 78B outperforms one-shot 78B distillation) provides a scaling recipe: as organizations train larger internal VLMs, they can add them to the distillation curriculum without redesigning the pipeline.

Cost-efficient batch inference for VLM evaluation and data generation. Organizations running large-scale VLM inference—evaluating thousands of images, generating training data, or scoring candidate generations—can use Masters-distilled models to reduce compute costs. The 4× efficiency gain over standard distillation (Table 1: naive +Large Teacher vs. +Reward Feedback yields 3–5 point average improvements at the same model size) translates directly to cost savings: a Masters-distilled 8B model achieving 80.0% average (InternVL3.5-8B-Masters, Table 2) can replace a larger model that might require 2–3× the inference FLOPs to achieve comparable accuracy. For a batch pipeline processing millions of images, halving the per-query inference cost while maintaining accuracy represents substantial savings. The offline RL design is particularly suited to this use case: pre-generate responses once for the target evaluation or data-generation distribution, train the distilled model, and deploy it for high-throughput inference without the per-query overhead of online RL or "think-answer" reasoning chains (Figure 6 shows Masters-applied models maintain base-model inference speed, while think-answer models are substantially slower).

Curriculum design for training compact specialist VLMs. The mask-progressive distillation approach generalizes beyond the specific models tested: any large VLM can be temporarily simplified through weight masking, enabling a curriculum where a small student first learns coarse-grained patterns and progressively refines its representations. This is particularly valuable for training specialist VLMs—models fine-tuned for specific domains like medical imaging, document understanding, or industrial inspection—where the large generalist teacher provides broad multimodal knowledge that the specialist student must adapt to a narrow domain. The mask-progressive strategy ensures the student absorbs the teacher's general knowledge in a capacity-aligned way before specializing, potentially reducing catastrophic forgetting of general capabilities that often accompanies domain-specific fine-tuning. The multi-response generation component (using both teacher and student responses at a 1:1 ratio, Table 5c) provides a template for maintaining alignment between domain-specific fine-tuning targets and the student's evolving representational capacity.

Verifier development as a strategic investment. The paper's finding that verifier quality is the primary bottleneck for RL-based distillation—not RL algorithm sophistication or reward design—has direct implications for R&D prioritization. Table 5(e) shows that using the accuracy reward without the distillation reward (R_acc only) produces strong but incomplete performance; adding R_distill provides further gains; but without a reliable accuracy signal (which requires a verifier or ground-truth labels), the distillation reward alone produces performance below the teacher's. This means organizations investing in distillation infrastructure should allocate resources to building better automated evaluation pipelines (LLM-as-a-Judge systems, domain-specific verifiers, ensemble verification) rather than to developing more complex RL algorithms. The paper's LLM-as-a-Judge implementation (Appendix D) provides a concrete, working template for multi-format VQA evaluation that handles open-ended questions, numeric answers, and multiple-choice selection with a unified prompting approach—a practical starting point that avoids the brittle per-domain parsing required by DeepSeek-R1-style rule-based evaluators.