GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

ArXiv: 2506.15681

🎯 Pitch

GenRecal introduces a lightweight recalibration module that enables knowledge distillation between VLMs with completely different tokenizers—something previously impossible—allowing a 1B-parameter student to hit 70.8% on RealWorldQA and an 8B student fine-tuned on its 78B teacher’s features to break 68% on MMMU, beating much larger open and closed models. The recalibrator learns a shared latent space between teacher and student features at zero added inference cost, turning any large VLM into a viable teacher regardless of architectural mismatch.

---

1. Executive Summary

This paper introduces GenRecal (Generation after Recalibration), a general-purpose distillation framework that transfers knowledge from large vision-language models (VLMs) to smaller ones without requiring shared tokenization schemes. Operating across models ranging from 1B to ~78B parameters and evaluated on 12 challenging vision-language benchmarks including MMMU, MM-Vet, and MathVista, GenRecal introduces a Recalibrator module—a lightweight set of decoder blocks and projection layers—that aligns feature representations between teacher and student VLMs in a shared latent space before the language head (operationalized as projecting small-VLM question features alongside large-VLM answer features to predict large-model token indices). The framework achieves performance gains that allow student models to surpass not only same-size baselines but also substantially larger open-source and closed-source VLMs—for example, InternVL2.5-8B-GenRecal (trained with InternVL2.5-78B teacher) reaches 68.1% on MMMU versus 56.0% for the un-distilled baseline, and 73.2% on MM-Vet versus 62.8%, while a 1B student model attains 70.8% on RealWorldQA—establishing that cross-tokenizer VLM distillation can match or exceed traditional same-tokenizer methods only when explicit feature alignment via a learned recalibration mechanism is employed, with the Recalibrator incurring zero additional inference cost.

2. Context and Motivation

The Fundamental Tension: VLMs Are Getting Bigger, Deployment Needs Smaller

Vision-language models have undergone a dramatic scaling trajectory over the past several years. What began with models like LLaVA-1.5 at 7B parameters (Liu et al., 2023) has rapidly escalated to systems like NVLM-72B, Qwen2-VL-72B, and InternVL2.5-78B—models that approach or match proprietary systems like GPT-4V and Claude-3.5 Sonnet on challenging multimodal benchmarks. The paper's Table 2 makes this explicit: InternVL2.5-78B achieves 88.3% on MMB, 72.3% on MM-Vet, and 70.1% on MMMU, numbers that would have been unthinkable for open-source models just two years ago.

This scaling, however, creates an acute practical problem. A 72B-parameter VLM consumes enormous computational resources at inference time—resources that are simply unavailable on the edge devices (phones, tablets, embedded systems) where many real-world vision-language applications need to run. The paper identifies this tension directly in Section 1:

"the increasing scale of recent VLMs introduces substantial computational overhead, limiting their practicality in real-world scenarios—particularly for on-device deployment"

This is not merely an engineering inconvenience. It represents a fundamental misalignment between how VLMs are developed (in pursuit of benchmark performance at any parameter cost) and how they must be deployed (under strict latency, memory, and energy constraints). Knowledge distillation—training a small "student" model to replicate a large "teacher"—is the natural remedy, but the paper argues that existing distillation approaches suffer from a structural flaw that severely limits which teacher-student pairs can be used.

The Token Type Mismatch Problem: Why Standard Distillation Breaks

To understand the gap this paper addresses, you need to understand exactly what breaks when teacher and student VLMs use different tokenization schemes.

Modern VLMs are built on top of large language models, and each LLM comes with its own tokenizer—the component that converts raw text into sequences of integer token indices. These tokenizers differ along three dimensions that the paper repeatedly emphasizes:

1. Vocabulary size: The number of distinct tokens the model recognizes. Qwen2's tokenizer has a different vocabulary size than Llama3's tokenizer, which differs from InternLM's.

2. Token splits: How the tokenizer segments text into tokens. The word "recalibration" might be tokenized as ["re", "cal", "ibration"] by one tokenizer and ["recal", "ibr", "ation"] by another. This means the same input text produces a different number of tokens depending on the tokenizer.

3. Token index ordering: The mapping from tokens to integer indices. Even if two tokenizers both have a token for "cat," the integer index assigned to that token will almost certainly differ between them.

The critical consequence: when you feed identical text to two VLMs with different tokenizers, you get different-length token sequences with non-corresponding indices. A question-answer pair that produces 128 tokens under Qwen2's tokenizer might produce 142 tokens under InternLM's tokenizer, and token index 47 in one sequence bears no semantic relationship to token index 47 in the other.

Standard knowledge distillation for VLMs—as implemented in LLaVA-KD (Cai et al., 2024), LLaVA-MoD (Shu et al., 2024), and Align-KD (Feng et al., 2024)—computes a KL divergence loss between the teacher's and student's output probability distributions at each token position. This requires the two models to produce equal-length output sequences with semantically aligned token positions. When the tokenizers differ, this requirement is violated in a way that cannot be trivially patched. You cannot compute KL divergence between a 128-token sequence and a 142-token sequence—the dimensions don't match. Even if you pad or truncate to force equal lengths, the token at position i in one sequence doesn't correspond to the token at position i in the other.

The paper visualizes this constraint starkly in Figure 1 (left). A table shows various teacher-student VLM combinations, with green checkmarks indicating where traditional distillation is possible and red X's indicating where it isn't. The pattern is clear: traditional distillation works only when both VLMs use the same underlying LLM tokenizer—Qwen2-VL-72B can distill into Qwen2-VL-7B (both use Qwen2's tokenizer), but neither can distill into InternVL2.5-8B (which uses a different tokenizer). This is what the paper means by "token types"—the combination of vocabulary size, token splits, and token index ordering that defines a particular tokenizer's output space.

Figure 8 in Appendix B makes the scope of this limitation concrete by enumerating possible distillation pairs among 11 different VLM architectures under traditional distillation versus GenRecal. Traditional distillation supports only a handful of homogeneous pairs (e.g., Qwen2-VL-72B → Qwen2-VL-7B), while GenRecal supports essentially any combination.

Even Same-Family Models Can Be Incompatible

A subtle but important point the paper raises is that token type incompatibility isn't just a cross-family problem:

"even VLMs within the same family may employ different token types, further hindering compatibility"

This is a non-obvious observation. You might assume that InternVL2.5-8B and InternVL2.5-78B share a tokenizer because they're in the same model family. But as the paper notes in Section 3.1, InternVL2.5-78B uses Qwen2.5-72B as its LLM backbone, while InternVL2.5-8B uses a different underlying LLM. The InternVL2.5 "family" label refers to the VLM architecture and training recipe, not the tokenizer—and it's the LLM backbone that determines the tokenizer. So even within the InternVL2.5 family, the 78B and 8B variants have different token types, making traditional distillation between them impossible.

This is particularly frustrating from a practical standpoint because same-family distillation is exactly where you'd expect the greatest benefits—the models share architectural design principles, training data distributions, and image processing pipelines, so the knowledge transfer should be especially effective. But the tokenizer barrier prevents it.

Prior Distillation Work and Its Limitations

The paper situates its contribution against several lines of prior work, each with specific shortcomings:

LLaVA-KD (Cai et al., 2024) represents the state of conventional VLM distillation. It uses a three-stage training process with KL divergence between teacher and student output distributions, carefully managing which parameters are trainable at each stage. But it requires identical token types—the teacher and student must produce token sequences of equal length with aligned positions. The paper compares against LLaVA-KD directly in Table 4, showing that even when the token type constraint is satisfied (Qwen2-VL-72B → Qwen2-VL-7B, both using Qwen2 tokenizers), GenRecal outperforms it—but the more fundamental point is that LLaVA-KD simply cannot be applied to the vast majority of potential teacher-student pairs.

LLaVA-MoD (Shu et al., 2024) extends distillation to mixture-of-experts (MoE) architectures but still relies on logit-based distillation with the same token type assumption.

Align-KD (Feng et al., 2024) goes further by distilling both vision encoder features and decoder logits, and adding an attention map alignment loss. This is a more sophisticated distillation pipeline, but it still operates under the constraint that teacher and student share a tokenizer. The attention map alignment adds a visual modality alignment, but doesn't address the textual token mismatch.

MoVE-KD (Cao et al., 2025) uses multiple vision encoders and a mixture of LoRA adapters, distilling visual information through attention maps. This helps with the visual side of the mismatch but doesn't solve the textual token type problem.

Dataset distillation approaches (Gu et al., 2024; Zhang et al., 2024; Li et al., 2024; Chen et al., 2023; Hu et al., 2024) take a different route: instead of directly transferring model parameters or outputs, they use large VLMs to generate high-quality visual instruction tuning datasets, which are then used to train smaller VLMs via standard supervised fine-tuning. This avoids the token type problem entirely because the small model is trained from scratch on the generated data. However, it loses the direct signal from the teacher's internal representations—the small model sees only the teacher's final outputs, not the rich intermediate features, probability distributions, or uncertainty estimates that logit-based distillation can transfer.

Cross-tokenizer distillation in language-only settings has been explored (Boizard et al., 2025; Cui et al., 2025) using methods like Wasserstein distance or optimal transport to match probability distributions across different token spaces. The paper compares against these in Table 5(f), showing that they substantially underperform GenRecal. The fundamental issue is that these methods operate in the output logit space—they try to match teacher and student probability distributions by transporting probability mass between different-sized vocabularies. This introduces information loss because the mapping between tokenizers is many-to-many and ambiguous. When sequence lengths differ (which they always do with different tokenizers), these methods resort to zero-padding or truncation, which further degrades the signal.

The Deeper Issue: Loss of Teacher Representation Richness

Beyond the mechanical problem of mismatched sequence lengths, there's a more conceptual limitation that the paper identifies. Traditional distillation methods, even when applicable, use the student's VLM-head (language head) to absorb the teacher's knowledge. The teacher produces output logits, and the student is trained to match those logits at each token position. But the teacher's VLM-head—the final linear layer that maps hidden representations to vocabulary probabilities—operates over a much larger vocabulary and with much higher-dimensional hidden states than the student's VLM-head. When you distill at the logit level, you're asking the student to reproduce a compressed version of the teacher's output distribution, necessarily losing information in the compression.

The paper argues that this is suboptimal not just for cross-tokenizer settings but even for same-tokenizer distillation. The richer signal lives in the teacher's hidden representations before the language head—the features that encode semantic understanding, visual-linguistic grounding, and reasoning steps. A student that can internalize these representations, rather than just mimicking final token probabilities, should learn more effectively.

How GenRecal Positions Itself

The paper's central positioning move is to reframe the distillation problem from output matching to representation alignment. Rather than trying to force the student's token probabilities to match the teacher's (which breaks when the token spaces differ), GenRecal learns a mapping from the student's hidden feature space to the teacher's hidden feature space, and then uses the teacher's own VLM-head to generate the training signal.

This is a clever inversion of the standard distillation paradigm. In traditional distillation:

Teacher generates output → Student tries to match that output

In GenRecal:

Student generates hidden features → Recalibrator maps them to teacher's feature space → Teacher's VLM-head processes the mapped features → Loss is computed using teacher's tokenizer

The key insight is that by operating in the hidden feature space (before the language head), the Recalibrator can bridge different dimensionalities and different tokenization schemes. The teacher's VLM-head is used as a "decoder" that translates the aligned representations back into the teacher's token space, where the autoregressive loss can be computed naturally. This is what the paper means by "Generation after Recalibration"—the generation (token prediction) happens after the recalibration (feature alignment), using the teacher's own generation machinery.

The paper explicitly draws inspiration from the well-established finding in NLP that word embeddings from different models can be linearly mapped into a shared space (Mikolov et al., 2013; Smith et al., 2017; Conneau et al., 2017). GenRecal extends this idea from static word embeddings to contextualized hidden representations in a multimodal setting, and from linear mapping to a learned nonlinear transformation via the Recalibrator's decoder blocks.

Why This Matters Beyond the Technical Contribution

The paper positions GenRecal not just as a method but as enabling a fundamentally different approach to VLM deployment. When you can distill from any large VLM to any small VLM regardless of architecture or tokenizer, you gain three practical freedoms:

Freedom to choose the best teacher. You're not limited to teachers that share your student's tokenizer. If InternVL2.5-78B performs better than Qwen2-VL-72B on your target benchmarks, you can use it as your teacher even if your student uses a completely different LLM backbone. The paper demonstrates this concretely in Figure 2 (left): as you swap in more powerful teachers (NVLM-72B → InternVL2-76B → Qwen2-VL-72B → InternVL2.5-78B), the distilled student's performance consistently improves.

Freedom to choose the best student for your deployment constraints. If you need a 1B-parameter model for on-device deployment, you can distill into InternVL2.5-1B regardless of what tokenizer it uses. Table 1 shows that InternVL2.5-1B-GenRecal reaches 70.8% on RealWorldQA and 56.5% on MathVista—numbers that put it in the range of much larger un-distilled models.

Freedom to upgrade either side independently. When a better large VLM is released (as happens frequently), you can distill from it into your existing small model without worrying about tokenizer compatibility. Similarly, if you develop a more efficient small VLM architecture, you can distill into it from your existing large model.

The paper frames this as moving from a "limited set of pairings" (traditional distillation, Figure 8a) to "the flexibility to select any model for distillation" (GenRecal, Figure 8b). This is not hyperbole—the method genuinely removes the token-type constraint that made most potential distillation pairs infeasible.

The Empirical Puzzle That Motivates the Approach

There's an interesting empirical finding embedded in the paper's motivation that isn't highlighted as prominently as it could be. When the paper compares GenRecal against traditional distillation under same-token-type conditions (Table 4: Qwen2-VL-72B teacher, Qwen2-VL-7B student), GenRecal outperforms traditional methods by a substantial margin. This is surprising because you would expect traditional distillation, operating in its native regime with perfectly aligned token spaces, to be at least competitive. The fact that GenRecal wins even here suggests that the problem isn't just token type incompatibility—it's that logit-level distillation is inherently lossy compared to feature-level alignment, regardless of tokenizer matching.

The paper's explanation (Section 4.4) is that GenRecal uses the teacher's VLM-head for the distillation signal, which "inherently possesses a higher hidden dimensionality than that of the small VLM," thereby "capturing richer and more expressive information." In other words, even when token types match, it's better to have the student learn to produce features that the teacher would generate good output from, rather than trying to directly mimic the teacher's output distributions. This is a conceptual shift from "mimic what the teacher says" to "learn to think like the teacher thinks, then say what you would say."

3. Technical Approach

3.1 Reader Orientation

This is primarily a systems-and-methods paper whose core idea is that VLM distillation across incompatible tokenizers is possible if you learn a feature-level mapping from the student's hidden representations into the teacher's hidden space, rather than trying to match output token probabilities directly. The paper constructs a three-stage training pipeline centered on a lightweight Recalibrator module—two transformer decoder blocks plus two linear projections—that learns to project the small VLM's internal features into a shared latent space defined by the large VLM's hidden representations, enabling the teacher's own language head to score the recalibrated features and provide a training signal back to the student.

3.2 Big-Picture Architecture (Diagram in Words)

The GenRecal system has four major components, only three of which are active at any given time:

1. Large (teacher) VLM — a frozen 72B+ parameter model (e.g., InternVL2.5-78B, Qwen2-VL-72B) that defines the target representation space. Its vision encoder, vision projector, VLM-body (decoder layers), and VLM-head (language head) are all frozen throughout training. It serves two roles: it provides target hidden features that define the shared latent space, and its VLM-head is used as a "scoring function" to compute the autoregressive loss on recalibrated student features.

2. Small (student) VLM — the model being trained, ranging from 1B to 8B parameters. During the first training stage, all its parameters are frozen; during later stages, its VLM-body and VLM-head are progressively unfrozen and trained. At inference time, it operates standalone with no additional components or computational overhead.

3. Recalibrator — a bridge module used only during training. It consists of two decoder blocks (Rec-body, matching the small VLM's decoder architecture) and two linear projection layers (Rec-proj-pre and Rec-proj-post). Rec-proj-pre projects the large VLM's higher-dimensional hidden features down to the small VLM's hidden dimension for concatenation; Rec-proj-post projects Recalibrator outputs back up to the large VLM's hidden dimension for consumption by the large VLM's language head. The Recalibrator also includes a new positional embedding (NPE) and an additional LayerNorm for stable training.

4. Large VLM's VLM-head — the final linear layer of the teacher model, used during training to convert recalibrated features into token probability distributions. This is the component that actually computes the distillation loss, but it processes features that originated from the student (after recalibration) rather than from the teacher.

Information flow during training (Stage 1): A question-answer pair is tokenized by both the small and large VLMs' tokenizers (producing different token sequences due to tokenizer mismatch) → Both models' vision encoders and projectors process the image → Both models' word embeddings convert text tokens to initial hidden states → Both models' VLM-body decoder layers process the combined visual-textual sequence, producing hidden features $[z_{q_s}, z_{a_s}]$ (small) and $[z_{q_l}, z_{a_l}]$ (large) → The question portion of the small VLM's features ($z_{q_s}$) and the answer portion of the large VLM's features ($z_{a_l}$) are extracted and concatenated → Rec-proj-pre projects $z_{a_l}$ down to the small VLM's hidden dimension → The concatenated sequence $[z_{q_s}, \text{Rec-proj-pre}(z_{a_l})]$ passes through Rec-body (with new positional embeddings) → Rec-proj-post projects the output back to the large VLM's hidden dimension → The answer portion of the recalibrated features ($r_{a_l}$) is fed through the large VLM's frozen VLM-head to produce token logits → Autoregressive cross-entropy loss is computed against the large VLM's ground-truth answer token indices, and KL divergence loss is computed between the large VLM's original output distribution and the distribution produced from recalibrated features.

At inference time: The Recalibrator and large VLM are entirely removed. The small VLM processes images and text normally, producing answers using its own VLM-body and VLM-head with zero additional parameters, FLOPs, or latency compared to the base small VLM.

3.3 Roadmap for the Deep Dive

• First, the formal architecture decomposition of VLMs into four modules (vision encoder, vision projector, VLM-body, VLM-head), because every subsequent design choice depends on which module's outputs are accessed and frozen versus trained.
• Second, the Recalibrator's internal structure and propagation rules, including the critical dimensionality-matching problem (large and small VLMs have different hidden dimensions), the new positional embedding scheme (since concatenated features come from different position-ID spaces), and the additional LayerNorm for stable adaptation.
• Third, the Stage 1 training procedure: autoregressive loss, KL divergence, and the regularization term—what each computes, why each is necessary, and how they are combined.
• Fourth, the Stage 2 training procedure: unfreezing the small VLM's VLM-body and adding the small VLM's own autoregressive loss, explaining how this enables knowledge transfer from the teacher's shared latent space into the student's parameters.
• Fifth, the Stage 3 training procedure: removing Recalibrator and large VLM entirely, then supervised fine-tuning the small VLM on a curated dataset to enhance instruction-following capability.
• Sixth, the training infrastructure and hyperparameters, including the DeepSpeed ZeRO-3 configuration, dataset composition (9M samples for Stages 1–2, 6M for Stage 3), and the specific optimizer, learning rate schedule, batch size, and gradient accumulation settings.

3.4 Detailed, Sentence-Based Technical Breakdown

VLM Architecture Decomposition

The paper decomposes every VLM into four sequential modules (Section 3.1), a standard but important abstraction that determines which intermediate representations are accessible for the recalibration process:

1. Vision encoder: Processes the input image and produces visual features. Architecture varies across VLMs (e.g., CLIP, ConvNext, DINO-v2, or combinations thereof), but GenRecal treats it as a black box—it only needs the output features, not the internal structure.

2. Vision projector: Maps visual features from the vision encoder's output space into the LLM's input embedding space. This can be a simple linear layer, an MLP, or a more complex resampler (e.g., Q-Former in BLIP-2). Again, GenRecal treats this as a black box.

3. VLM-body: The LLM decoder layers that process the concatenated sequence of visual and textual features. This is the core transformer stack—the component that performs cross-modal reasoning, attention over both visual and textual tokens, and produces contextualized hidden representations. The paper extracts features from the output of this module (before the language head) for both the large and small VLMs. These features are denoted $[z_q, z_a]$—the concatenation of question-token features and answer-token features for a given input sequence.

4. VLM-head (language head): The final linear layer that projects hidden representations to vocabulary-size logits. The large VLM's VLM-head is used during GenRecal training as the scoring function; the small VLM's VLM-head is trained during Stage 2 and fine-tuned during Stage 3 to operate independently.

This decomposition matters because the Recalibrator operates on VLM-body outputs, not on raw token embeddings or on final logits. The choice is deliberate: VLM-body outputs contain rich semantic representations that have already integrated visual and textual information through the full transformer stack, making them a more informative signal for distillation than either earlier (pre-attention) or later (post-head) representations.

Recalibrator Architecture and Propagation

The Recalibrator is the central technical innovation of GenRecal. Its job is deceptively simple to state—"align small VLM features to large VLM features"—but mechanically complex to implement because of three mismatches between teacher and student:

• Dimensionality mismatch: The large VLM's hidden dimension (e.g., 8192 for Qwen2-72B) is typically larger than the small VLM's (e.g., 4096 for Qwen2-7B, or 2048 for smaller models). Features cannot be directly concatenated or compared when they have different dimensionalities.

• Sequence length mismatch: The same text produces different numbers of tokens under different tokenizers, so the VLM-body output sequences $[z_q, z_a]$ from large and small VLMs have different lengths.

• Position encoding mismatch: Each VLM uses Rotary Position Embeddings (RoPE) with position IDs that correspond to its own tokenizer's output. When you extract question features from the small VLM and answer features from the large VLM and concatenate them, the position IDs from the two sources are incompatible—they come from different coordinate systems.

The Recalibrator addresses all three simultaneously through its structure and propagation rules.

Sub-components (Figure 3c and Section 3.1):

• Rec-proj-pre: A single linear layer that projects the large VLM's hidden features from dimension $d_{\text{large}}$ down to $d_{\text{small}}$. This enables concatenation with the small VLM's features, which already have dimension $d_{\text{small}}$. The layer takes the answer portion of the large VLM's VLM-body output ($z_{a_l}$) as input and outputs a dimensionality-reduced version $\tilde{z}_{a_l} = \text{Rec-proj-pre}(z_{a_l})$ of shape matching $z_{q_s}$.

• Rec-body: Two transformer decoder blocks whose architecture exactly mirrors the small VLM's decoder blocks—same hidden dimension ($d_{\text{small}}$), same number of attention heads, same feed-forward network structure, and same causal attention mask. Using the small VLM's architecture for Rec-body is a pragmatic choice: it ensures dimensional compatibility with the projected features, and it keeps the Recalibrator lightweight (two decoder blocks versus the dozens in a full VLM-body). Rec-body takes the concatenated sequence $[z_{q_s}, \tilde{z}_{a_l}]$ as input and produces a transformed sequence $[r_{q_s}, r_{a_l}]$ of the same shape.

• Rec-proj-post: A single linear layer that projects the Rec-body output back from dimension $d_{\text{small}}$ to $d_{\text{large}}$. It processes only the answer portion of the Rec-body output ($r_{a_l}$), restoring the dimensionality needed for the large VLM's VLM-head. The question portion ($r_{q_s}$) is not projected back since it isn't fed to the VLM-head (only answer tokens contribute to the autoregressive loss).

• New Positional Embedding (NPE): A fresh set of RoPE parameters applied to the concatenated sequence before it enters Rec-body. Because the concatenated sequence $[z_{q_s}, \tilde{z}_{a_l}]$ combines features from two different positional encoding schemes (the small VLM's position IDs for the question tokens, the large VLM's for the answer tokens), the original position encodings carried within $z_{q_s}$ and $\tilde{z}_{a_l}$ are not mutually consistent. The NPE assigns new, sequential position IDs (0, 1, 2, ...) to the concatenated sequence and applies fresh RoPE, creating a unified positional encoding scheme for the Recalibrator's attention operations. This is explicitly stated in Section 4.1(a):

"we employ another RoPE to realign their positional embeddings, and their position-ids are re-assigned as well"

• Additional LayerNorm: Applied to the output features of Recalibrator before they are fed to the large VLM's VLM-head. The paper states this is "for stable adaptation" (Section 4.1(a)). The LayerNorm normalizes the recalibrated features to have zero mean and unit variance, preventing distribution shift from destabilizing the frozen large VLM's VLM-head (which was trained to expect features with specific statistical properties).

Propagation logic (Equation 1 and surrounding text):

The core operation occurs in two steps:

Step 1: Extract VLM-body outputs. The same question-answer pair is fed through both VLMs' full pipelines (vision encoder → vision projector → word embedding → VLM-body), producing:

$[z_{q_l}, z_{a_l}] = \text{VLM-body}_l([q_l, a_l])$ $[z_{q_s}, z_{a_s}] = \text{VLM-body}_s([q_s, a_s])$

where $q_l, a_l$ are the question and answer token sequences produced by the large VLM's tokenizer, $q_s, a_s$ are those produced by the small VLM's tokenizer, $[q_l, a_l]$ denotes concatenation of the token sequences, and $\text{VLM-body}_l$ and $\text{VLM-body}_s$ denote the decoder stacks of the large and small VLMs respectively. The outputs $z_{q_l}$, $z_{a_l}$, $z_{q_s}$, and $z_{a_s}$ are sequences of hidden state vectors—each vector has dimension $d_{\text{large}}$ for the large VLM and $d_{\text{small}}$ for the small VLM, and the sequence lengths differ between models because the tokenizers produce different numbers of tokens for the same text.

Step 2: Recalibrate. The question features from the small VLM and answer features from the large VLM are concatenated and processed:

$[r_{q_s}, r_{a_l}] = \text{Recalibrator}([z_{q_s}, z_{a_l}])$

where $[r_{q_s}, r_{a_l}]$ are the recalibrated output features, and the Recalibrator internally performs:

1. $\tilde{z}_{a_l} = \text{Rec-proj-pre}(z_{a_l})$ — project large VLM answer features down to $d_{\text{small}}$
2. $h = \text{Rec-body}([z_{q_s}, \tilde{z}_{a_l}] + \text{NPE})$ — add new positional embeddings and process through two decoder blocks
3. $r_{a_l} = \text{Rec-proj-post}(\text{LayerNorm}(h_{\text{answer}}))$ — extract answer portion, normalize, project back to $d_{\text{large}}$

The question features $r_{q_s}$ from the Recalibrator output are not used in any loss computation; only the answer features $r_{a_l}$ are fed to the large VLM's VLM-head.

Why this specific architecture?

The choice of two decoder blocks (rather than one or many) is justified by an ablation in Table 5(b). When sweeping Recalibrator depth from 1 to 20 blocks, the paper selects depth=2 based on a "trade-off between computational efficiency and performance." Two blocks provide sufficient transformation capacity to align the feature spaces without introducing excessive training overhead. The linear projections (rather than MLPs or more complex mappings) are sufficient because their job is purely dimensional—the actual representational alignment happens in the Rec-body decoder blocks.

The asymmetric concatenation (small VLM question + large VLM answer, rather than using both from the same model) is the paper's key insight for making cross-tokenizer distillation work. If you tried to use the small VLM's own answer features ($z_{a_s}$) as input to the Recalibrator, the resulting recalibrated features would need to be scored by the large VLM's VLM-head to compute a loss, but there's no ground-truth mapping from the small VLM's answer tokens to the large VLM's answer tokens—they're different tokenizations of the same text, so the loss target is ambiguous. By instead using the large VLM's answer features as part of the input, the Recalibrator learns to process the large VLM's own features (which it can reconstruct well after projection, as shown by the regularization loss) and also learns to map small VLM question features into a space where the answer features make sense in the large VLM's representation scheme. This is effectively teaching the Recalibrator: "given a question expressed in the small VLM's feature language, produce answer features in the large VLM's feature language."

Stage 1: Alignment Training (Recalibrator Only)

The first training stage serves a single purpose: train the Recalibrator to project the small VLM's feature representations into the large VLM's latent space so that the large VLM's VLM-head can interpret them as if they were its own features. During this stage, all parameters of both the large and small VLMs are frozen—only the Recalibrator's parameters (Rec-proj-pre, Rec-body, Rec-proj-post, NPE, and the additional LayerNorm) are updated.

The stage uses two loss functions, which are jointly optimized:

Loss 1: Autoregressive cross-entropy loss ($\mathcal{L}_{ar}$):

$\mathcal{L}_{ar} = \text{CE}(\text{VLM-head}_l(r_{a_l}), gt_l)$

where $\text{VLM-head}_l$ is the large VLM's frozen language head (the final linear layer mapping hidden states to vocabulary logits), $r_{a_l}$ is the answer portion of the Recalibrator's output features (with dimension $d_{\text{large}}$ after Rec-proj-post), and $gt_l$ is the ground-truth answer token indices from the large VLM's tokenizer.

What it computes: the standard cross-entropy between the token probability distribution produced by the large VLM's language head (operating on recalibrated features) and the correct answer token indices in the large VLM's vocabulary. For each answer token position, the VLM-head converts the recalibrated feature vector at that position into a probability distribution over the large VLM's vocabulary (via linear projection + softmax), and cross-entropy penalizes deviations from the one-hot ground truth at that position. The result is a scalar loss summed over all answer token positions.

Why this form: this is the standard autoregressive language modeling objective, applied to the recalibrated features. It asks: "if the large VLM's language head sees these recalibrated features, does it predict the correct answer tokens?" Minimizing this loss forces the Recalibrator to produce features that, from the perspective of the large VLM's language head, are maximally predictive of the correct answer. The teacher's language head acts as a learned evaluator—it knows what features should precede correct answer tokens because it was trained on the large VLM's own features.

Loss 2: KL divergence loss ($\mathcal{L}_{kl}$):

$\mathcal{L}_{kl} = \mathcal{D}_{\text{KL}}(\text{VLM-head}_l(z_{a_l}) \mid \text{VLM-head}_l(r_{a_l}))$

where $\mathcal{D}_{\text{KL}}(P \mid Q)$ is the Kullback-Leibler divergence from distribution $Q$ to distribution $P$ (specifically, $P$ is the distribution from the large VLM's own answer features $z_{a_l}$ and $Q$ is the distribution from the recalibrated features $r_{a_l}$), and both distributions are over the large VLM's vocabulary at each answer token position.

What it computes: for each answer token position, the KL divergence measures how different the token probability distribution produced from recalibrated features is from the distribution that the large VLM would have produced from its own features at the same position. A low KL divergence means the Recalibrator is faithfully preserving the large VLM's "intended" output distribution—the recalibrated features don't just predict the correct token, they also capture the relative probabilities the teacher would assign to incorrect but plausible tokens (e.g., synonyms, alternative phrasings).

Why this form: the KL divergence provides a richer training signal than cross-entropy alone. Cross-entropy only cares about the probability assigned to the single correct token; KL divergence cares about the entire shape of the output distribution. This encourages the Recalibrator to produce features that are distributionally similar to the teacher's own features, not just features that happen to maximize the correct token's probability. In the authors' implementation (Algorithm 1 in Appendix E), this is computed at every answer token position and summed.

The regularization term (Algorithm 2 in Appendix E):

A critical finding in the paper is that the above losses are insufficient. Without additional regularization, the Recalibrator learns a degenerate solution where it produces features that score well under the autoregressive and KL losses but don't actually align with the large VLM's feature space in a semantically meaningful way. The paper documents this through cosine similarity analysis (Figure 7) and through direct performance measurements (Table 4).

The regularization term is computed by feeding the large VLM's own question and answer features through the Recalibrator (rather than mixing small and large VLM features) and applying the same two losses:

$[r_{q_l}, r_{a_l}] = \text{Recalibrator}([z_{q_l}, z_{a_l}])$ $\mathcal{L}_{ar}^{\text{reg}} = \text{CE}(\text{VLM-head}_l(r_{a_l}), gt_l)$ $\mathcal{L}_{kl}^{\text{reg}} = \mathcal{D}_{\text{KL}}(\text{VLM-head}_l(z_{a_l}) \mid \text{VLM-head}_l(r_{a_l}))$

where $[z_{q_l}, z_{a_l}]$ are the large VLM's own VLM-body features for both question and answer (unlike the main loss which uses $[z_{q_s}, z_{a_l}]$—small VLM question, large VLM answer). The total regularization loss is $\mathcal{L}_{ar}^{\text{reg}} + \mathcal{L}_{kl}^{\text{reg}}$.

What it computes: the Recalibrator's ability to reconstruct the large VLM's own feature representations. If you give the Recalibrator the large VLM's own features as input (both question and answer), can it produce output features that the large VLM's VLM-head interprets identically to the original features? This is essentially an autoencoding objective through the Recalibrator bottleneck—the features are projected down to $d_{\text{small}}$, processed by Rec-body, and projected back to $d_{\text{large}}$.

Why this form and why it's necessary: without this regularization, the Recalibrator can learn a mapping that produces "plausible-looking" features from the cross-modal input $[z_{q_s}, z_{a_l}]$ without actually aligning the small VLM's representation space to the large VLM's. The regularization term anchors the Recalibrator: it forces the Recalibrator to behave as a near-identity mapping when given the large VLM's own features, which means that when given mixed features $[z_{q_s}, z_{a_l}]$, the small VLM's question features $z_{q_s}$ must be mapped into a space that's compatible with the large VLM's answer features $z_{a_l}$ under the same transformation.

The paper's analysis in Figure 7 makes this concrete. Figure 7(b) shows the cosine similarity matrix between Recalibrator outputs from 10 different input samples when the regularization term is removed: the diagonal values (same-sample similarity) are comparable to off-diagonal values (different-sample similarity), indicating the Recalibrator is not producing sample-specific feature alignment—it's collapsing to a generic mapping. Figure 7(c) shows the same matrix with regularization: diagonal values are significantly higher than off-diagonal values, indicating the Recalibrator is producing sample-specific feature transformations that explicitly pair small and large VLM representations. Table 3 quantifies the downstream impact: without regularization, MMMU-Pro performance drops from 48.8% to 38.2% (InternVL2.5-8B student), and similar drops occur across all student sizes.

Joint optimization in Stage 1: The total Stage 1 loss is:

$\mathcal{L}_{\text{stage1}} = \mathcal{L}_{ar} + \mathcal{L}_{kl} + \mathcal{L}_{ar}^{\text{reg}} + \mathcal{L}_{kl}^{\text{reg}}$

where the first two terms use the cross-modal input $[z_{q_s}, z_{a_l}]$ and the last two terms use the same-model input $[z_{q_l}, z_{a_l}]$. All four terms are computed at every training step and summed with equal weight (no loss-balancing coefficients are mentioned in the paper).

Stage 2: Knowledge Distillation (Unfreezing the Small VLM-body)

In the second training stage, the small VLM's VLM-body parameters are unfrozen and trained alongside the Recalibrator (which continues training from its Stage 1 checkpoint). The large VLM remains fully frozen. The stage uses the same two loss functions from Stage 1 plus one additional loss: the small VLM's own autoregressive loss.

Updated autoregressive loss:

$\mathcal{L}_{ar} = \text{CE}(\text{VLM-head}_l(r_{a_l}), gt_l) + \text{CE}(\text{VLM-head}_s(z_{a_s}), gt_s)$

where the first term is unchanged from Stage 1, and the second term is the standard supervised fine-tuning loss for the small VLM: cross-entropy between the small VLM's own VLM-head output (operating on its own VLM-body features $z_{a_s}$) and the ground-truth answer token indices from the small VLM's tokenizer ($gt_s$).

What changes from Stage 1: the small VLM's VLM-body is now trainable (purple-colored in Algorithm 3, Appendix E). The gradient from the first term ($\text{CE}(\text{VLM-head}_l(r_{a_l}), gt_l)$) flows back through: large VLM's VLM-head (frozen) → Rec-proj-post (trainable) → Rec-body (trainable) → Rec-proj-pre (trainable) → small VLM's VLM-body (now trainable). This means the small VLM's internal representations are updated to produce features that, after recalibration, are better interpreted by the large VLM's language head.

What the KL divergence term does in Stage 2: the KL divergence loss $\mathcal{L}_{kl}$ continues to operate exactly as in Stage 1—it compares the teacher's own output distribution to the distribution from recalibrated features. In Stage 2, however, the gradient flows back into the small VLM's VLM-body, encouraging it to produce question features $z_{q_s}$ that, when recalibrated alongside the teacher's answer features, yield distributions matching the teacher's distributions.

What the small VLM's own autoregressive loss does: this term ensures the small VLM doesn't "forget" how to generate coherent text in its own token space while its internal representations are being modified. Without this term, the small VLM's VLM-body might learn to produce features that are great for the Recalibrator but produce gibberish when processed by its own VLM-head—a form of catastrophic forgetting. The autoregressive loss anchors the small VLM to its original language modeling capability.

Why Stage 2 is necessary (rather than stopping after Stage 1): Stage 1 trains only the Recalibrator—the small VLM's representations are unchanged, and the Recalibrator learns to map from frozen small-VLM features to the large VLM's space. This setup can produce good recalibrated features, but the small VLM itself hasn't learned anything from the teacher; all the knowledge transfer is happening in the Recalibrator, which gets removed at inference. Stage 2 backpropagates the distillation signal into the small VLM's VLM-body, actually transferring the teacher's representational knowledge into the student's parameters. After Stage 2, the small VLM's internal features should be "closer" to the teacher's features in the sense that they require less transformation by the Recalibrator to be interpretable by the teacher's language head.

Trainable parameter scope: the paper states that the small VLM's VLM-body is unfrozen and trained, but the small VLM's vision encoder, vision projector, word embeddings, and VLM-head remain frozen during Stage 2 (implicit from Algorithm 3, where only VLM-body_s is purple/highlighted as trainable among the small VLM components). This selective unfreezing prevents overfitting to the distillation signal and preserves the visual feature extraction and token embedding capabilities learned during the small VLM's original pretraining.

Stage 3: Supervised Fine-Tuning (Standalone Small VLM)

In the third and final training stage, the Recalibrator and large VLM are completely removed. Only the small VLM remains, and it is fine-tuned via standard supervised fine-tuning (SFT) on a curated dataset to enhance its instruction-following capability.

What is trained: all parameters of the small VLM except the vision encoder are unfrozen and trained. This means the vision projector, word embeddings, VLM-body, and VLM-head are all updated. The vision encoder is kept frozen, which is a common practice in VLM training to preserve the pretrained visual features.

Training data: Stage 3 uses a 6M-sample subset of the full 9M dataset, created by "removing general visual question answering" data (Section 4.1(b)). The paper does not elaborate on the rationale for this filtering, but it likely reflects a focus on more complex reasoning tasks during the final fine-tuning stage, with the general VQA capability already transferred during Stages 1–2.

Loss function: standard autoregressive cross-entropy on the small VLM's own token predictions—no distillation losses, no recalibration, no teacher model. The small VLM is trained to generate correct answers given question-image inputs, using its own tokenizer and its own VLM-head.

Why Stage 3 is necessary: Stages 1–2 focus on representational alignment—making the small VLM's feature space compatible with the teacher's. But the small VLM's VLM-head was frozen during those stages, meaning the final output layer hasn't been optimized for the updated representations. Stage 3 fine-tunes the entire pipeline end-to-end (except the vision encoder) on the actual task of generating correct answers, allowing the VLM-head to adapt to the improved VLM-body representations and for any remaining misalignments to be corrected. It's analogous to the "instruction tuning" phase in standard VLM training, applied after the knowledge transfer.

Inference after Stage 3: the small VLM operates as a standard standalone VLM—vision encoder processes the image, vision projector maps visual features, VLM-body processes the combined visual-textual sequence, VLM-head produces token predictions. No Recalibrator, no teacher model, no additional parameters, no additional FLOPs. The entire distillation infrastructure is a training-time-only mechanism.

Training Infrastructure and Hyperparameters

The paper provides specific training configuration details in Section 4.1(b):

Hardware and distributed training:

• 8 NVIDIA A100 80GB GPUs, using DeepSpeed with ZeRO-3 optimization (Rajbhandari et al., 2020). ZeRO-3 partitions model parameters, gradients, and optimizer states across GPUs, which is necessary because the full training setup includes both the large (72B+) and small (1B–8B) VLMs simultaneously—far exceeding the memory of a single GPU.
• The paper reports training times of "approximately 5 to 7 days" for Stages 1–2 combined (depending on model sizes) and "4 to 6 days" for Stage 3, for a total of roughly 9–13 days per full training run.

Optimizer and learning rate:

• AdamW optimizer (Loshchilov & Hutter, 2018) with a linearly decayed learning rate.
• The learning rate decays from $1 \times 10^{-4}$ to $1 \times 10^{-5}$ at each training stage. The paper states "linearly decayed" but does not specify whether the decay follows a cosine schedule, linear warmup, or other details beyond the start and end values.
• No weight decay, dropout, or Adam beta values are explicitly stated in the main text or appendices for GenRecal training. This is a notable absence compared to typical paper method sections.

Batch size and gradient accumulation:

• Per-GPU batch sizes: 4 samples per GPU in Stages 1 and 3, 2 samples per GPU in Stage 2. The reduction in Stage 2 is because the small VLM's VLM-body is now trainable, increasing memory consumption per sample.
• Gradient accumulation: 16 steps. This means the effective batch size is $8 \text{ GPUs} \times 16 \text{ accumulation steps} \times \text{per-GPU batch size}$.
• Effective batch sizes: 512 samples for Stages 1 and 3 ($8 \times 16 \times 4$), 256 samples for Stage 2 ($8 \times 16 \times 2$).

Dataset (Section 4.1(b) and Appendix F):

• Stages 1 and 2: the entire 9M-sample visual instruction tuning dataset, covering "general visual question answering, dense image captioning, chart/diagram/document understanding, common-sense knowledge, science and math understanding, and multi-dimensional reasoning."
• Stage 3: a 6M-sample subset with "general visual question answering" removed.
• Dataset sources include LLaVA-OneVision, MMC, DenseFusion, Cambrian, GPT-4V-filtered SA-1B data, Infinity-MM, Finance-QA, Wikipedia knowledge data, InfoSeek, science and mathematical reasoning (SMR) data, document understanding data, WildVision, SROIE, RLAI-F, M3CoT, LLaVAR, KonIQ, and iNaturalist2018. This is a highly diverse dataset spanning multiple domains, which the paper argues is necessary for the Recalibrator to learn a general-purpose feature mapping rather than overfitting to a narrow task distribution.

Teacher VLM configurations (Section 3.1):

• NVLM-72B: uses Qwen2-72B as LLM backbone
• Qwen2-VL-72B: uses Qwen2-72B as LLM backbone
• InternVL2-76B: uses Llama3-70B as LLM backbone
• InternVL2.5-78B: uses Qwen2.5-72B as LLM backbone

Student VLM configurations (Section 3.1):

• Qwen2-VL-2B/7B: uses Qwen2-2B/7B as LLM backbone
• InternVL2.5-1B: uses Qwen2.5-0.5B as LLM backbone
• InternVL2.5-2B: uses InternLM2.5-1.8B as LLM backbone
• InternVL2.5-4B: uses Qwen2.5-3B as LLM backbone
• InternVL2.5-8B: uses InternLM2.5-7B as LLM backbone

Note how the LLM backbones differ even within the InternVL2.5 family—this is why token types vary and why GenRecal's cross-tokenizer capability is necessary.

Recalibrator configuration (Section 4.1(a)):

• Rec-body: 2 transformer decoder blocks, configured identically to the small VLM's decoder blocks (same hidden dimension, number of heads, FFN structure, causal mask).
• Rec-proj-pre: single linear layer, input dimension $d_{\text{large}}$, output dimension $d_{\text{small}}$.
• Rec-proj-post: single linear layer, input dimension $d_{\text{small}}$, output dimension $d_{\text{large}}$.
• New positional embedding: RoPE with re-assigned sequential position IDs.
• Additional LayerNorm: applied to Recalibrator output before VLM-head_l.

Evaluation setup (Section 4.1(b)):

• At inference, the Recalibrator and large VLM are completely removed.
• The small VLM generates answers using "the default generation hyperparameter" of each benchmark's evaluation protocol—the paper does not specify temperature, top-p, or other sampling parameters beyond this generic reference.

FLOPs analysis (Table 5(a)): The paper compares the training FLOPs of the Recalibrator against both the small and large VLMs. For InternVL2.5-8B (student) and InternVL2.5-78B (teacher):

• Large VLM: 1.56 × 10¹² FLOPs
• Small VLM: 1.63 × 10¹¹ FLOPs
• Recalibrator: 1.76 × 10⁹ FLOPs

The Recalibrator's FLOPs are roughly two orders of magnitude smaller than the small VLM's and three orders of magnitude smaller than the large VLM's, confirming that the Recalibrator introduces minimal training overhead. At inference time, Recalibrator FLOPs are zero because it is removed.

A Note on What GenRecal Is NOT

To avoid a common misconception: GenRecal is not a method for making the small VLM use the large VLM's tokenizer at inference time. The small VLM continues to use its own tokenizer and its own VLM-head for generation. The large VLM's vocabulary and tokenizer are used only during training, as part of the loss computation. The knowledge transferred is representational—the small VLM learns to produce internal features that are more semantically aligned with the teacher's feature space, which improves its own generation quality even though it operates in a different token space. This is what the paper means by "token types-agnostic"—the distillation process doesn't care about tokenizer differences, but the resulting student model still uses its native tokenizer.

This is also why the paper emphasizes that GenRecal works even when teacher and student share token types (Table 4): the benefit isn't about solving tokenizer mismatch per se; it's about operating at the feature level rather than the logit level, which provides a richer distillation signal regardless of tokenizer compatibility. The cross-tokenizer capability is a consequence of the feature-level approach, not the primary goal.

4. Key Insights and Innovations

Innovation 1: Reframing VLM Distillation as Feature-Space Alignment Rather Than Output-Space Mimicry

The central conceptual move of this paper is a shift in where distillation operates. Prior VLM distillation methods—LLaVA-KD, LLaVA-MoD, Align-KD, and the broader logit-based distillation literature—all work in the output token probability space: the student is trained to produce similar token-level probability distributions to the teacher. This is the natural extension of Hinton et al. (2015)'s original knowledge distillation formulation, adapted to autoregressive VLMs by computing KL divergence at each output token position.

GenRecal breaks from this tradition by operating one layer deeper: at the VLM-body output—the hidden representations produced by the final decoder layer, immediately before the language head. Rather than asking "can the student predict the same tokens as the teacher?", it asks "can the student produce internal features that the teacher's language head would interpret as preceding the correct answer?" The teacher's frozen VLM-head becomes an evaluator of feature quality, and the distillation signal is the gradient of the autoregressive loss through that evaluator back to the student's representations.

This reframing matters for three reasons beyond the cross-tokenizer capability it enables:

First, it bypasses the vocabulary mismatch problem structurally rather than patching it algorithmically. The cross-tokenizer distillation methods that GenRecal is compared against in Table 5(f)—UID (Boizard et al., 2025) and MOT (Cui et al., 2025)—attempt to solve the vocabulary mismatch by matching probability distributions across different-sized output spaces using Wasserstein distance or optimal transport. These are mathematically principled approaches, but they operate in a space where the mismatch is fundamental: you are trying to align probability vectors of different dimensionalities that represent different token granularities and different semantic partitionings of the output space. The paper argues, and Table 5(f) empirically demonstrates, that this introduces substantial information loss—GenRecal's MMMU score of 68.1% versus 59.7% for MOT and 58.4% for UID with the same teacher-student pair suggests that circumventing the problem entirely (by moving to feature space) is more effective than solving it in output space.

Second, it leverages the teacher's higher-capacity language head as a richer training signal. The paper makes an explicit argument in Section 4.4 that using the teacher's VLM-head—with its larger hidden dimensionality and larger vocabulary—provides a more expressive distillation target than using the student's VLM-head, even when token types match. This is a non-obvious claim: you might expect that the student's own head, operating in the student's native token space, would be the more natural target since the student ultimately needs to generate in that space. But the paper's results in Table 4 (where GenRecal outperforms LLaVA-KD, MiniLLM, and DistiLLM even with same-token-type pairs) suggest otherwise. The teacher's head encodes richer distributional knowledge—uncertainty estimates, synonym probabilities, alternative phrasings—that the student's smaller head cannot express. By training the student to produce features that the teacher's head interprets well, GenRecal transfers this richer signal.

Third, it converts the distillation problem from a divergence-minimization between output distributions to a representation-learning problem with an autoregressive objective. The autoregressive loss $\mathcal{L}_{ar}$ in GenRecal is structurally identical to standard language model training—the only difference is that the features being scored come from a different source (the Recalibrator rather than the teacher's own VLM-body). This means GenRecal can leverage the well-understood dynamics of autoregressive training (stable convergence, compatibility with existing optimization infrastructure) while still performing cross-model knowledge transfer. The KL divergence term $\mathcal{L}_{kl}$ provides additional distribution-matching pressure, but the primary signal is the simpler and empirically more stable cross-entropy loss.

This reframing is a fundamental shift, not an incremental refinement. It changes what distillation means for VLMs: from "teach the student to imitate the teacher's answers" to "teach the student to think in a way the teacher would recognize as correct." The cross-tokenizer capability is a byproduct of this deeper shift, not the primary intellectual contribution.

Innovation 2: The Regularization Term as a Diagnostic of Degenerate Feature Alignment

The paper's most scientifically interesting finding—not the one with the largest performance numbers, but the one with the most conceptual depth—is the discovery and diagnosis of what happens when you train a feature-space alignment module without the regularization term. This is not just an ablation showing that a component helps; it is a window into a specific failure mode of representation alignment that has implications beyond this paper.

The failure mode is this: when trained with only the cross-modal losses ($\mathcal{L}_{ar}$ and $\mathcal{L}_{kl}$ on $[z_{q_s}, z_{a_l}]$), the Recalibrator learns to produce features that score well under the teacher's language head but that are not semantically aligned with the teacher's own feature space. The paper diagnoses this through cosine similarity analysis in Figure 7: without regularization, the similarity between a sample's recalibrated features and other samples' recalibrated features (off-diagonal entries in the similarity matrix) is comparable to the similarity between a sample's features and its own paired features (diagonal entries). This means the Recalibrator is learning a mapping that collapses distinct inputs into similar outputs—it finds a "generic good-looking feature vector" that the teacher's head scores highly regardless of the specific question.

What makes this diagnosis significant is that it reveals a fundamental tension in representation alignment: making features perform well under a frozen evaluator does not guarantee they preserve the information content needed for that evaluator to discriminate between different inputs. The cross-entropy and KL losses only care about the output distribution, not about whether the input features encode distinct semantic content for different questions. A degenerate Recalibrator could learn to always output the mean feature vector that maximizes expected answer correctness over the training distribution—essentially memorizing a prior over answers rather than conditioning on the question. The regularization term ($\mathcal{L}_{ar}^{\text{reg}} + \mathcal{L}_{kl}^{\text{reg}}$ on $[z_{q_l}, z_{a_l}]$) prevents this by forcing the Recalibrator to act as a near-identity map when given the teacher's own features, which anchors the output space to the teacher's semantic geometry.

This is conceptually related to the mode collapse problem in GANs and the posterior collapse problem in VAEs—a learned mapping optimizes a proxy objective (convincing a discriminator, maximizing a reconstruction likelihood, or here, producing features that the teacher's head scores well) and finds a degenerate solution that satisfies the objective without learning a useful mapping. The paper's contribution is not solving this problem (regularization terms to prevent representation collapse are well-known) but rather identifying that it occurs in cross-model feature alignment for VLMs and providing a clear, empirically-grounded diagnosis.

The quantitative impact is stark: Table 3 shows that removing regularization drops MMMU-Pro from 48.8% to 38.2% (InternVL2.5-8B student), 45.9% to 40.2% (4B), 36.6% to 30.9% (2B), and 28.7% to 29.5% (1B). The fact that the 1B model shows minimal degradation suggests that smaller models have less capacity to learn the degenerate mapping—they are forced to preserve more input-specific information because their representational capacity is limited. This is a subtle finding that the paper doesn't fully explore but that has implications for how alignment module capacity should scale with student model size.

Innovation 3: Empirical Proof That Cross-Tokenizer Distillation Can Match and Exceed Same-Tokenizer Distillation

The paper makes a strong empirical claim that runs counter to what you would naively expect: GenRecal, which must bridge different tokenizers, different vocabularies, and different hidden dimensionalities, outperforms traditional distillation methods that operate under the ideal condition of identical token types between teacher and student. Table 4 is the key evidence: with Qwen2-VL-72B as teacher and Qwen2-VL-7B as student (identical token types, both using Qwen2's tokenizer), GenRecal achieves substantially higher scores than LLaVA-KD, the state-of-the-art same-tokenizer distillation method.

This is surprising. The null hypothesis would be: same-tokenizer distillation, operating without the information bottleneck of cross-tokenizer mapping, should be an upper bound on cross-tokenizer distillation performance. GenRecal should at best approach this bound when token types match. The fact that it exceeds it—and by a meaningful margin—suggests that the logit-level distillation paradigm has an inherent limitation that feature-level alignment overcomes, even in the absence of tokenizer mismatch.

The paper's explanation (Section 4.4) centers on the teacher's VLM-head providing a richer signal: "the large VLM's VLM-head inherently possesses a higher hidden dimensionality than that of the small VLM." But this explanation is incomplete. LLaVA-KD also uses the teacher's output—its logit distribution over the vocabulary—which is computed from the teacher's high-dimensional hidden states through the teacher's VLM-head. The difference is what information from the teacher is transferred. Logit-based distillation transfers the final output distribution—a compressed, vocabulary-specific summary of the teacher's knowledge. Feature-based distillation transfers the representational geometry that produces that output—a richer, more transferable signal.

This matters beyond the specific numbers. It implies that the field's default approach to VLM distillation (logit matching) is fundamentally suboptimal, and that feature-level methods should be the baseline going forward—not just for cross-tokenizer scenarios but for all VLM distillation. The cross-tokenizer capability is then "free"—a natural consequence of operating at a level where tokenizer differences don't matter, rather than an explicit design goal that trades off against performance.

The paper's Figure 2 (left) provides additional evidence for this interpretation. By sweeping across teachers of increasing capability (NVLM-72B → InternVL2-76B → Qwen2-VL-72B → InternVL2.5-78B), GenRecal's performance tracks the teacher's quality—better teachers produce better students. This monotonic relationship suggests that the feature-alignment mechanism is faithfully transferring representational quality, not just fitting to a particular teacher's idiosyncrasies.

Innovation 4: The Recalibrator as a General-Purpose "Adapter" That Decouples Teacher and Student Architecture Choices

The paper's architectural contribution is not the Recalibrator's specific design (two decoder blocks + two linear projections is conventional) but rather the architectural role it plays: a training-time-only bridge that decouples teacher selection from student selection. This represents a design pattern that could generalize beyond VLMs to any setting where models with incompatible tokenization schemes need to transfer knowledge.

The standard distillation paradigm tightly couples teacher and student: they must share output space structure (same vocabulary, same sequence length) for the loss function to be computable. GenRecal's Recalibrator breaks this coupling by interposing a learnable mapping that translates between representation spaces. The teacher and student can be chosen independently—the teacher for its performance characteristics (benchmark scores, reasoning ability), the student for its deployment characteristics (latency, memory, hardware compatibility). The Recalibrator handles the translation, and because it is removed at inference, it imposes no deployment cost.

The paper demonstrates this decoupling concretely through the breadth of teacher-student pairs in its experiments (Table 1, Figure 5, Figure 8, Table 5(g–i)). The teachers span NVLM-72B, Qwen2-VL-72B, InternVL2-76B, and InternVL2.5-78B—four different LLM backbones (Qwen2-72B, Llama3-70B, Qwen2.5-72B) with different tokenizers, vocabularies, and hidden dimensionalities. The students span six different model sizes (1B to 8B) across three VLM families (InternVL2.5, Qwen2-VL, VILA1.5) with at least four different LLM backbones. Every combination works. The performance consistently improves with better teachers (Figure 2 left) and with larger students (Figure 5), suggesting the decoupling is genuine rather than cherry-picked.

This is a systems contribution as much as an algorithmic one. It changes the economics of VLM deployment: you can now invest in training or acquiring the best possible large VLM for your domain, and then distill it into whatever small model fits your deployment constraints, without the combinatorial restriction that the two must share a tokenizer. Figure 8 visualizes this: traditional distillation supports a sparse matrix of possible pairs (only same-tokenizer combinations), while GenRecal supports the full cross-product of teachers and students.

The paper's FLOPs analysis in Table 5(a) is important for establishing the practicality of this contribution: the Recalibrator's training FLOPs (1.76 × 10⁹) are dwarfed by the small VLM's (1.63 × 10¹¹) and the large VLM's (1.56 × 10¹²), meaning the architectural decoupling comes at negligible additional training cost. And as the paper repeatedly emphasizes, inference cost is zero—the Recalibrator is a pure training artifact.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation is conducted across 12 vision-language benchmarks: AI2D (Kembhavi et al., 2016), ChartQA (Masry et al., 2022), MathVista (Lu et al., 2023), MMB and MMB-CN (Liu et al., 2023), MM-Vet (Yu et al., 2023), MMMU (Yue et al., 2023), MMMU-Pro (Yue et al., 2024), BLINK (Fu et al., 2024), SEED-2-Plus (Li et al., 2024), and RealWorldQA (RWQA). These benchmarks span diverse capabilities including visual question answering, diagram understanding, mathematical reasoning, chart interpretation, and multi-disciplinary college-level reasoning. The training dataset for GenRecal is a 9M-sample visual instruction tuning corpus drawn from over 20 sources (detailed in Appendix F), covering general VQA, dense captioning, chart/document understanding, science/math reasoning, and common-sense knowledge. Stage 3 training uses a 6M subset with general VQA data removed to emphasize complex reasoning tasks.

• Base model(s). The teacher VLMs are models at 72B+ parameter scale: NVLM-72B (built on Qwen2-72B), Qwen2-VL-72B (Qwen2-72B), InternVL2-76B (Llama3-70B), and InternVL2.5-78B (Qwen2.5-72B). The student VLMs span 1B to 8B parameters across multiple families: InternVL2.5-1B/2B/4B/8B (built on Qwen2.5-0.5B, InternLM2.5-1.8B, Qwen2.5-3B, InternLM2.5-7B respectively), Qwen2-VL-2B/7B (Qwen2-2B/7B), and VILA1.5-3B. The teacher models are chosen because they represent the current state-of-the-art open-source VLMs approaching GPT-4V and Claude-3.5 Sonnet performance (Section 3.1), while the student models represent the range of deployment-viable parameter counts. The breadth of LLM backbones across these models—Qwen2, Qwen2.5, Llama3, InternLM2.5—is deliberate to demonstrate token-type agnosticism.

• Metrics. The primary metric for each benchmark is its standard accuracy score—the percentage of test questions for which the model's generated answer matches the ground truth according to that benchmark's grading protocol. The paper reports accuracy using "the default generation hyperparameter" for each benchmark (Section 4.1(b)). There is no custom aggregation metric; each benchmark's score is reported independently in Tables 1, 2, 3, 4, and 5.

• Baselines. The paper compares against several categories of prior work. Un-distilled VLMs of comparable size (Table 1): Cambrian-1-8B/13B, Eagle-8B/13B, VILA1.5-8B/13B, CogVLM2-19B, LLaVA-OneVision-7B, InternVL2-8B, MiniCPM-V2.5-8B/V2.6-8B, Qwen2-VL-7B, and InternVL2.5-8B for the ~7–8B range; VILA1.5-3B, Phi-3.5-Vision-4B, InternVL2-4B, and InternVL2.5-4B for the ~3–4B range; InternVL2-2B, Qwen2-VL-2B, Aquila-VL-2B, and InternVL2.5-2B for the ~2B range; LLaVA-OneVision-0.5B, InternVL2-1B, and InternVL2.5-1B for the ~0.5–1B range. Large-scale VLMs and closed-source systems (Table 2): NVLM-72B, LLaVA-OneVision-72B, Molmo-72B, Qwen2-VL-72B, InternVL2-76B, InternVL2.5-78B, Claude-3.5-Sonnet, Gemini-1.5-Pro, and GPT-4o (0513). Traditional distillation methods (Table 4): SFT on the baseline, MiniLLM (Gu et al., 2024) using reverse KL divergence, DistiLLM (Ko et al., 2024) using a skewed KL variant, and LLaVA-KD (Cai et al., 2024) using standard KL divergence in a three-stage framework. Cross-tokenizer distillation methods (Table 5(f)): UID (Boizard et al., 2025) using Wasserstein distance for cross-token probability matching, and MOT (Cui et al., 2025) using optimal transport.

• Generation budget / compute accounting. Distillation cost is measured in training FLOPs rather than inference compute, since the distillation process is training-time only. Table 5(a) reports FLOPs for the large VLM (1.56 × 10¹²), small VLM (1.63 × 10¹¹), and Recalibrator (1.76 × 10⁹) for a representative InternVL2.5-78B → InternVL2.5-8B pair. Training time is reported as "approximately 5 to 7 days" for Stages 1–2 combined and "4 to 6 days" for Stage 3, on 8 NVIDIA A100 80GB GPUs using DeepSpeed ZeRO-3. At inference time, the Recalibrator and large VLM are removed, so the small VLM's inference compute is identical to its un-distilled counterpart—there is no additional inference cost.

• Cross-validation / statistical protocol. The paper does not report cross-validation, confidence intervals, or statistical significance tests. All evaluations are single-run on the standard test splits of each benchmark. The main experimental protocol is systematic sweep rather than statistical hypothesis testing: for each student model size and teacher model, the full three-stage pipeline is run and the resulting model is evaluated on all 12 benchmarks. Ablation studies (Table 5) vary one configuration at a time and report the resulting benchmark scores.

Main Quantitative Results

GenRecal versus Un-Distilled Baselines (Table 1)

The central result is that GenRecal-distilled small VLMs substantially outperform their un-distilled counterparts across nearly all benchmarks and model sizes. The headline numbers for the strongest configuration, InternVL2.5-8B-GenRecal (teacher: InternVL2.5-78B), versus the un-distilled InternVL2.5-8B baseline:

• MMMU: 68.1% vs 56.0% (an increase of 12.1 percentage points)
• MM-Vet: 73.2% vs 62.8% (increase of 10.4 points)
• MMMB: 89.5% vs 84.6% (increase of 4.9 points)
• MathVista: 74.9% vs 64.4% (increase of 10.5 points)
• MMMU-Pro: 48.8% vs 34.3% (increase of 14.5 points)
• ChartQA: 93.6% vs 84.8% (increase of 8.8 points)
• AI2D: 93.0% vs 84.8% (increase of 8.2 points)
• RealWorldQA: 81.4% vs 70.1% (increase of 11.3 points)
• BLINK: 65.3% vs 54.8% (increase of 10.5 points)

The pattern is consistent: gains are largest on the most challenging benchmarks (MMMU-Pro at +14.5 points, MMMU at +12.1 points) and more modest on benchmarks where the baseline is already high (MMB at +4.9 points from an 84.6% baseline).

Equally important is that GenRecal's benefits scale down to very small models. InternVL2.5-1B-GenRecal (teacher: InternVL2.5-78B) achieves 70.8% on RealWorldQA and 56.5% on MathVista—numbers that exceed InternVL2.5-4B's un-distilled scores (64.3% and 60.5% respectively). At the 2B scale, InternVL2.5-2B-GenRecal achieves 59.2% on MMMU, surpassing the un-distilled InternVL2.5-8B's 56.0%. The 4B-scale InternVL2.5-4B-GenRecal reaches 58.3% on MMMU and 66.1% on MM-Vet, again exceeding the un-distilled 8B model on MMMU.

A subtle pattern visible in Table 1: the choice of student model matters independently of the teacher. When using the same teacher (InternVL2.5-78B), InternVL2.5-8B-GenRecal outperforms Qwen2-VL-7B-GenRecal on most benchmarks—for example, 68.1% vs 65.6% on MMMU and 73.2% vs 70.4% on MM-Vet—despite the Qwen2-VL-7B being a strong baseline in its own right. This supports the paper's claim in Section 4.2 that "employing more capable small VLMs is crucial for achieving higher distillation performances."

GenRecal versus Large-Scale and Closed-Source VLMs (Table 2)

Table 2 positions GenRecal-distilled 8B models against models that are 9–10× larger and against proprietary systems. The key comparisons for InternVL2.5-8B-GenRecal (teacher: InternVL2.5-78B):

• Versus the teacher itself (InternVL2.5-78B): GenRecal matches or exceeds the teacher on several benchmarks—93.0% vs 89.1% on AI2D, 93.6% vs 88.3% on ChartQA, 81.4% vs 78.7% on RealWorldQA—while trailing modestly on others (73.2% vs 72.3% on MM-Vet is essentially tied; 68.1% vs 70.1% on MMMU is a slight decline). This is remarkable because the student has ~9.75× fewer parameters.

• Versus GPT-4o (0513): GenRecal outperforms GPT-4o on MMB (89.5% vs 83.4%), AI2D (93.0% vs 84.6%), ChartQA (93.6% vs 85.7%), and RealWorldQA (81.4% vs 75.4%), while trailing on MMMU (68.1% vs 69.1%), MM-Vet (73.2% vs 69.1%), and MathVista (74.9% vs 63.8%). The pattern suggests GenRecal excels at visual perception and chart understanding tasks but remains slightly behind on complex multi-disciplinary reasoning.

• Versus Claude-3.5-Sonnet: Similar pattern—GenRecal leads on MMB (89.5% vs 82.6%), AI2D (93.0% vs 81.2%), and RealWorldQA (81.4% vs 60.1%), lags on MMMU (68.1% vs 68.3% is essentially tied) and MathVista (74.9% vs 67.7%).

• Versus Gemini-1.5-Pro: GenRecal leads substantially across the board, with the largest gaps on MMB (89.5% vs 73.9%) and MathVista (74.9% vs 63.9%).

Table 2 also demonstrates that GenRecal's performance improves monotonically with teacher quality. For the same InternVL2.5-8B student, scores increase as the teacher is upgraded: NVLM-72B → InternVL2-76B → Qwen2-VL-72B → InternVL2.5-78B. On MMMU, the trajectory is 60.3% → 64.6% → 65.6% → 68.1%. On MM-Vet: 63.9% → 72.4% → 71.4% → 73.2%. This monotonicity is critical evidence that GenRecal's feature alignment mechanism genuinely transfers representational quality from teacher to student rather than saturating at some ceiling independent of teacher capability.

GenRecal versus Traditional Distillation (Table 4 and Figures 1, 2)

Table 4 provides the direct comparison between GenRecal and traditional distillation methods under the fairest possible condition for the baselines: identical token types between teacher and student (Qwen2-VL-72B → Qwen2-VL-7B, both using Qwen2 tokenizer). The results are striking:

• Qwen2-VL-7B student, MMMU: SFT baseline 48.8% → LLaVA-KD 52.5% (+3.7 points) → GenRecal 61.1% (+12.3 points over baseline, +8.6 points over LLaVA-KD)
• Qwen2-VL-7B student, MM-Vet: SFT baseline 57.5% → LLaVA-KD 52.0% (worse than SFT) → GenRecal 66.8% (+9.3 points over baseline)
• Qwen2-VL-7B student, MathVista: SFT baseline 60.8% → LLaVA-KD 62.7% (+1.9 points) → GenRecal 67.8% (+7.0 points over baseline)
• Qwen2-VL-2B student, MMMU: SFT baseline 30.4% → LLaVA-KD 31.1% (+0.7 points) → GenRecal 47.1% (+16.7 points over baseline)

MiniLLM and DistiLLM, using modified KL divergence formulations, perform comparably to or slightly below LLaVA-KD. GenRecal's substantial margin over all three traditional methods—despite the token type advantage the traditional methods enjoy—is the paper's strongest evidence that feature-level alignment is fundamentally more effective than logit-level matching, independent of cross-tokenizer concerns.

Figure 1 (right panel) visualizes this for MM-Vet, showing a progression: baseline ~32%, SFT on baseline ~37%, traditional distillation (same token types) ~43%, GenRecal (same token types) ~59%, GenRecal with more powerful teacher (InternVL2.5-78B) ~63%, GenRecal with more powerful student and teacher (InternVL2.5-8B + InternVL2.5-78B) ~66%.

Figure 2 (left) shows the effect of teacher choice across four benchmarks (MMB, MM-Vet, MMMU, MMMU-Pro). The consistent upward slope as teacher capability increases—NVLM-72B < InternVL2-76B < Qwen2-VL-72B < InternVL2.5-78B—holds across all four benchmarks, confirming that GenRecal's distillation faithfully transfers the teacher's absolute capability.

Cross-Tokenizer Distillation Comparison (Table 5(f))

Table 5(f) compares GenRecal against UID (Wasserstein-distance-based cross-token matching) and MOT (optimal-transport-based cross-token matching) on MMMU. With InternVL2.5-78B as teacher and InternVL2.5-8B as student:

• UID (Boizard et al., 2025): 58.4%
• MOT (Cui et al., 2025): 59.7%
• GenRecal: 68.1%

The 8.4–9.7 point gap suggests that logit-space cross-token matching—even with sophisticated transport-based methods—introduces substantial information loss compared to GenRecal's feature-space approach. The paper attributes this (Section 4.4) to the fact that UID and MOT "cannot properly handle token-split indices, leading to mismatched output lengths during distillation" and "just rely on zero-padding or truncation to enforce equal token lengths," whereas GenRecal's Recalibrator "produces equal-length token outputs fed into VLM-head of large VLM, thereby preventing information loss."

FLOPs Analysis (Table 5(a))

The training-cost analysis for InternVL2.5-78B → InternVL2.5-8B:

• Large VLM: 1.56 × 10¹² FLOPs
• Small VLM: 1.63 × 10¹¹ FLOPs
• Recalibrator: 1.76 × 10⁹ FLOPs

The Recalibrator's training FLOPs are roughly 1/93rd of the small VLM's FLOPs and 1/886th of the large VLM's FLOPs. Since the Recalibrator is removed at inference, it adds zero inference FLOPs. The paper does not report total training FLOPs for the full three-stage pipeline, nor does it provide a FLOPs comparison against traditional distillation methods, which is a notable omission for a paper that emphasizes computational efficiency.

Dataset Scale Analysis (Table 5(d))

The paper sweeps training dataset sizes from 1M to 9M samples for the InternVL2.5-78B → InternVL2.5-8B pair on MMMU:

• 1M: 60.6%
• 2M: 63.5%
• 5M: 67.8%
• 9M: 68.1%

The gains from 5M to 9M are marginal (+0.3 points), suggesting 5M samples is a practical operating point for resource-constrained users. The paper does not report this analysis for other benchmarks or student sizes, limiting the generalizability of this finding.

Scaling Behavior: Teacher and Student Size (Figure 5 and Table 1)

Figure 5 presents a heatmap of MMMU performance for various teacher-student size combinations within the InternVL2.5 family. The grid reveals two patterns:

• Larger teachers improve all students: For a fixed student (e.g., InternVL2.5-8B), MMMU scores increase from 56.0% (no distillation) to higher values as the teacher scales: 26B → 38B → 76B → 78B.
• Larger students benefit more from distillation (in absolute terms): The performance gap between GenRecal-distilled and un-distilled models grows with student size. The 8B student gains 12.1 points on MMMU, while the 1B student gains 4.7 points (from 40.9% to 45.6%).

Table 1 provides the full benchmark-level breakdown supporting the consistent benefit across model sizes. The fact that even 1B models show meaningful gains (e.g., InternVL2.5-1B-GenRecal at 70.8% on RealWorldQA vs 57.5% baseline) demonstrates that the distillation signal propagates effectively even through very low-capacity students.

Teacher Fine-Tuning Ablation (Table 5(e))

An interesting finding: fine-tuning the large VLM (InternVL2.5-78B) on the 9M training dataset before using it as a teacher yields additional gains. With InternVL2.5-8B as student on MMMU:

• Without teacher fine-tuning: 68.1%
• With teacher fine-tuning: 69.2% (+1.1 points)

The paper interprets this (Section 4.3) as indicating that "the small VLM trained under the same setting has the potential to match or even surpass the large VLM's performance." The teacher fine-tuning adapts the large VLM's representations to the specific data distribution used for distillation, reducing the distribution shift between the teacher's pretrained feature space and the distillation data.

Ablation Studies and Robustness Checks

Regularization term (Table 3): Removing the regularization term (Algorithm 2, which feeds the teacher's own features through the Recalibrator) causes substantial performance degradation. For the InternVL2.5-78B teacher, MMMU-Pro scores drop from 48.8% to 38.2% (8B student), 45.9% to 40.2% (4B), 36.6% to 30.9% (2B), and 28.7% to 29.5% (1B). The degradation is notably smaller for the 1B model (+0.8 points, within noise), which the paper does not comment on but likely reflects the 1B model's limited capacity to learn the degenerate mapping that regularization prevents—smaller models are implicitly regularized by their representational bottleneck.

Recalibrator decoder depth (Table 5(b)): Sweeping Rec-body depth from 1 to 20 decoder blocks for InternVL2.5-78B → InternVL2.5-8B on MMMU:

• Depth 1: 60.2%
• Depth 2: 68.1%
• Depth 4: 63.3%
• Depth 8: 67.5%
• Depth 20: 67.8%

The paper selects depth 2 based on "a trade-off between computational efficiency and performance." Notably, depth 2 outperforms depth 4 by 4.8 points, which suggests that the Recalibrator's capacity is carefully balanced—too shallow (depth 1) cannot capture the necessary transformation, while depth 4 may begin to overfit or develop the degenerate mapping even with regularization. The recovery at depths 8 and 20 is unexplained—the paper reports these numbers without interpretation.

New positional embedding (NPE) (Table 5(c)): Removing the new positional embedding (relying on the original position encodings carried within the concatenated features) reduces MMMU from 68.1% to 61.4%. This 6.7-point drop is substantial and confirms that the positional encoding mismatch between teacher and student tokenizers is a real obstacle—the teacher's answer-token features encode position information from the teacher's tokenizer, the student's question-token features encode position information from the student's tokenizer, and concatenating them without realignment creates a positional "seam" that degrades the Recalibrator's attention operations.

Training dataset domain composition (Figure 10 and Appendix F): The paper analyzes the effect of removing entire domain categories from the 9M training set. The three categories are Knowledge, Science & Math, and Chart & Document. On MMMU (Figure 10, left):

• Full dataset: accuracy climbs through three training stages from ~51% to ~57% to ~64%
• Without Knowledge: follows a similar trajectory but consistently lower, ending at ~59%
• Without Science & Math: slightly lower trajectory, ending at ~62%
• Without Chart & Document: trajectory is close to full dataset, ending at ~63%

On MathVista (Figure 10, right):

• Full dataset: climbs from ~56% to ~63% to ~70%
• Without Knowledge: slightly lower, ending at ~69%
• Without Science & Math: substantially lower trajectory, ending at ~62%
• Without Chart & Document: slightly lower, ending at ~68%

The domain-specific sensitivity aligns with expectation: MMMU (multi-disciplinary college-level questions) draws most heavily on Knowledge, while MathVista (mathematical reasoning) draws most heavily on Science & Math. The Chart & Document domain shows relatively small impact on both benchmarks, suggesting either that its signal is partially redundant with other domains or that these particular benchmarks don't heavily exercise chart/document understanding.

Cross-family and cross-LLM compatibility (Table 5(g–i)): Table 5 demonstrates GenRecal's applicability beyond the InternVL2.5 family. Table 5(g) shows GenRecal working with Qwen2-VL-2B → Qwen2-VL-72B (same family, same LLM: Qwen2). Table 5(h) shows InternVL2.5-78B (Qwen2.5-72B backbone) → LLaVA-OneVision-7B (different VLM family, different LLM backbone). Table 5(i) shows GenRecal applied to VILA1.5-3B as student with InternVL2.5-78B as teacher. In all cases, GenRecal delivers improvements over baselines. No specific numbers are provided in the table text for these ablations—the paper states they "consistently deliver superior performance" and references the table, but the values are embedded in the full Table 5 which was not reproduced in detail beyond subsection (a) and (f).

VLM-head selection for distillation: While not a formal ablation, the paper emphasizes in Section 4.4 that GenRecal uses the teacher's VLM-head for the distillation signal rather than the student's VLM-head (as traditional distillation does). The implicit ablation is the comparison against LLaVA-KD in Table 4—LLaVA-KD uses the student's VLM-head to absorb the teacher's output distribution, while GenRecal uses the teacher's VLM-head to score recalibrated features. The performance gap (+8.6 points on MMMU for the 7B student) provides evidence that the teacher's head provides a richer training signal.

Loss convergence analysis (Figure 9, Appendix D): The paper presents loss curves for Recalibrator training across nine different teacher-student combinations. The key comparison is between Recalibrator losses (Recalib(q_s, a_l) and Recalib(q_l, a_l)) and the baselines' own SFT losses (SmallVLM(q_s, a_s) and LargeVLM(q_l, a_l)). In all nine combinations, the Recalibrator losses converge to values comparable to (and sometimes below) the baseline SFT losses, indicating that the Recalibrator successfully learns to produce features that the teacher's VLM-head can score as effectively as the teacher's own features. The Recalib(q_l, a_l) curves (teacher features through Recalibrator) consistently converge to the lowest values, confirming that the regularization term effectively forces the Recalibrator toward an identity-like mapping for in-distribution teacher features.

Critical Assessment

Does GenRecal outperform traditional distillation even when token types match?

The experiments in Table 4 support this claim for the specific case of Qwen2-VL-72B → Qwen2-VL-7B and Qwen2-VL-72B → Qwen2-VL-2B. The margins are substantial (+8.6 points over LLaVA-KD on MMMU for the 7B student) and consistent across benchmarks. However, the evidence is limited to a single teacher-student family (Qwen2-VL). The paper does not provide analogous same-tokenizer comparisons for InternVL2 (e.g., distilling between two models that both use InternLM backbones, if such pairs exist) or for any other VLM family. This is a notable gap because the Qwen2-VL family may have idiosyncrasies that favor GenRecal's approach independent of the feature-vs-logit distinction. A same-family comparison within the InternVL2 series would have been informative, but as the paper notes, even within the InternVL2.5 family, the 78B and 8B models use different LLM backbones (Qwen2.5-72B vs InternLM2.5-7B), making same-tokenizer pairs unavailable for that family.

Moreover, Table 4 compares GenRecal's full three-stage pipeline against LLaVA-KD's three-stage pipeline, but these pipelines are not directly comparable in total training compute, dataset usage, or optimization budget. GenRecal uses 9M samples for Stages 1–2 plus 6M for Stage 3; LLaVA-KD's data requirements are not reported. A fairer comparison would match total training FLOPs or samples rather than just comparing final scores. The paper's claim that GenRecal is superior would be stronger with such a controlled comparison.

Does GenRecal enable distillation across any teacher-student pair regardless of token type?

The evidence is broad but not exhaustive. The paper demonstrates successful distillation across many heterogeneous pairs: InternVL2.5-78B (Qwen2.5-72B tokenizer) → Qwen2-VL-7B/2B (Qwen2 tokenizer), InternVL2.5-78B → InternVL2.5-8B/4B/2B/1B (InternLM2.5 tokenizer for some, Qwen2.5 for others), InternVL2-76B (Llama3 tokenizer) → InternVL2.5-8B, NVLM-72B (Qwen2 tokenizer) → InternVL2.5-8B, and others in Table 5(g–i). The coverage across LLM backbones (Qwen2, Qwen2.5, Llama3, InternLM2.5) is strong. However, all experiments use the same broad VLM architecture paradigm (vision encoder → projector → decoder-only LLM). The paper does not test distillation across fundamentally different VLM architectures—for example, from a cross-attention-based VLM (like Flamingo) to a decoder-only VLM, or from a VLM with a Q-Former projector to one with an MLP projector. The claim of "any" pair is therefore supported only within the decoder-only LLM-based VLM paradigm that dominates current open-source models.

Does the distilled student genuinely match or exceed much larger models and closed-source systems?

Table 2 provides the primary evidence, and the numbers are compelling at face value: an 8B model (InternVL2.5-8B-GenRecal) achieving 68.1% on MMMU versus GPT-4o's 69.1% and Claude-3.5-Sonnet's 68.3%. However, several caveats apply:

Benchmark leakage risk: The teacher model (InternVL2.5-78B) was trained on datasets that may overlap with these benchmarks. The 9M distillation dataset explicitly includes data sources that could contain benchmark-proximal examples (LLaVA-OneVision, Cambrian, Infinity-MM, etc.). The paper does not perform decontamination analysis or report whether benchmark examples were filtered from the training data. This is a significant concern because distillation can inadvertently transfer benchmark-specific knowledge rather than general capability—if the teacher has memorized MMMU answers and the student learns to reproduce the teacher's feature representations for those questions, the student's MMMU score may reflect data contamination rather than genuine reasoning improvement.

Teacher fine-tuning confound (Table 5(e)): Fine-tuning the teacher on the distillation dataset improves student performance by 1.1 points on MMMU. This means the teacher being used for the Table 2 comparisons may have been fine-tuned (the paper does not clearly state whether the Table 2 results use a fine-tuned or frozen teacher). If the teacher was fine-tuned, then the comparison against GPT-4o and Claude is not teacher-vs-proprietary but (teacher + 9M distillation data) vs proprietary—the teacher had access to additional training data that the proprietary models did not.

Benchmark selection bias: The paper evaluates on 12 benchmarks and reports all scores. There is no evidence of cherry-picking benchmarks. However, several common VLM benchmarks are absent—notably, POPE (hallucination evaluation), TextVQA (text reading), ScienceQA, and OCRBench. These omissions may be incidental (no benchmark suite covers everything) but make it difficult to assess whether GenRecal's gains are uniform across capability dimensions or concentrated in the specific skills measured by the reported benchmarks.

The "matches GPT-4o" framing is partially misleading: While InternVL2.5-8B-GenRecal's MMMU score (68.1%) is within 1 point of GPT-4o's (69.1%), it lags substantially on MathVista (74.9% vs 63.8%—but note this is GenRecal higher) and MMMU-Pro (48.8% vs 51.9%). Wait—I need to re-examine the MathVista baseline. The benchmark scores in Table 1 show MathVista 74.9 for GenRecal (which I reported as higher than GPT-4o's 63.8%), which is correct and suggests GenRecal is better on MathVista. The point is that the relative strengths differ: GenRecal excels on chart/visual tasks (ChartQA 93.6% vs GPT-4o 85.7%) while being competitive on reasoning (MMMU 68.1% vs 69.1%). This is a nuanced picture, not uniform superiority or inferiority.

Are the ablation studies sufficient to support the architectural claims?

The ablation coverage has notable gaps. The paper ablates Recalibrator depth, the presence of NPE, regularization, and dataset scale. But it does not ablate:

• The specific choice of concatenation pattern. Why [z_q_s, z_a_l] rather than [z_q_l, z_a_s] or [z_q_s, z_a_s] (with a different loss formulation)? The paper's rationale (Section 3.2) is that predicting large-VLM answer tokens from small-VLM question tokens naturally forces cross-modal alignment, but the alternative of predicting small-VLM answer tokens from large-VLM question tokens might be similarly effective or even better for certain student-teacher pairs. No comparison is provided.

• The number of decoder blocks in the Recalibrator relative to the student VLM depth. The Recalibrator uses 2 decoder blocks regardless of whether the student has 24 layers (7B model) or 32 layers (8B model). The paper does not investigate whether Recalibrator depth should scale with student depth, or whether the optimal depth is a function of the representational gap between teacher and student.

• The choice of projection type (linear vs MLP). Rec-proj-pre and Rec-proj-post are single linear layers. The paper does not test whether deeper projections (2–3 layer MLPs) would improve the quality of dimensionality reduction and restoration, particularly when the hidden dimension ratio between teacher and student is large (e.g., 8192 → 2048, a 4× reduction).

• Loss weighting. The four loss terms in Stage 1 ($\mathcal{L}_{ar}$, $\mathcal{L}_{kl}$, $\mathcal{L}_{ar}^{\text{reg}}$, $\mathcal{L}_{kl}^{\text{reg}}$) are summed with equal weight. The paper does not investigate whether different weighting schemes would improve performance or training stability. Given the regularization term's critical importance, up-weighting it relative to the cross-modal losses might yield benefits.

• Teacher and student size scaling law. The paper demonstrates that larger teachers and larger students both improve results (Figure 5), but does not fit a scaling law or predict what performance a hypothetical 20B student with a 200B teacher might achieve. This limits the practical utility for practitioners deciding how to allocate their distillation budget.

The most significant missing ablation is a comparison against a simpler baseline: training the small VLM directly on the teacher's generated outputs (dataset distillation) using the same 9M training samples. The paper positions GenRecal against both distillation and dataset-based approaches in Section 2, but never provides a controlled comparison where the data budget is held constant. If the teacher generates high-quality answers for the 9M training questions and the student is SFT-trained on those answers, how does that compare to GenRecal's three-stage pipeline? This is the most natural baseline for resource-constrained practitioners and its absence is a significant limitation.

Generalizability concerns

The paper's results are derived entirely within the ecosystem of high-performing, decoder-only, LLM-based VLMs trained on massive web-scale data. It is unclear whether GenRecal would work for:

• VLMs with different architectures (encoder-decoder, cross-attention-based, early-fusion).
• Domain-specific VLMs (medical imaging, satellite imagery, robotics) where the teacher's representations may encode domain knowledge that doesn't align with the student's domain-general pretraining.
• Multilingual settings where the teacher and student use different tokenizers for different language coverage (e.g., a teacher optimized for English and a student optimized for Chinese). The paper does not evaluate on MMB-CN beyond reporting a single number in Table 1, and provides no analysis of cross-lingual transfer.
• Smaller teachers. All teachers are 72B+. Would GenRecal provide benefits when distilling from a mid-sized teacher (13B–34B) to a very small student (0.5B–1B), or is a minimum teacher-student capability gap required for the Recalibrator to extract a useful signal?

The missing inference-time compute comparison

GenRecal is a pure training-time method: the distilled student runs with zero overhead at inference. But the paper's introduction frames the problem as one of "resource-constrained deployment," which invites comparison against inference-time methods that also aim to improve small-model performance—test-time compute scaling, chain-of-thought prompting, self-consistency, or retrieval-augmented generation. A 1B model with self-consistency over 16 samples may approach the performance of a 4B model while still fitting in a smaller memory footprint. The paper provides no such comparison, making it difficult to assess whether distillation is the most cost-effective way (in terms of total compute: training + inference) to achieve a given accuracy target under deployment constraints.

Statistical reliability

The paper reports single-run evaluation scores on standard benchmarks with no confidence intervals, standard deviations, or statistical tests. For benchmarks with small test sets (e.g., MM-Vet has 218 questions, MMMU has 11,550 questions from college exams), differences of 1–3 points may fall within run-to-run variance. The paper's conclusions about teacher ranking (e.g., "InternVL2.5-78B is a better teacher than Qwen2-VL-72B") are based on point estimates without any indication of whether the differences are statistically significant or would replicate with different random seeds. This is standard practice in the VLM literature but weakens the strength of conclusions about fine-grained comparisons.

The regularization analysis is the most scientifically valuable component but incomplete

The cosine similarity analysis in Figure 7 and the regularization ablation in Table 3 are the paper's deepest investigations into why GenRecal works. The finding that without regularization, the Recalibrator produces features that show comparable same-sample and cross-sample similarity (Figure 7b)—a signature of representational collapse—is both diagnostically useful and conceptually interesting. However, the paper does not explore whether this collapse is uniform across all samples or concentrated in certain difficulty regimes, whether it correlates with specific types of question content, or whether alternative regularization strategies (gradient penalty, spectral normalization, InfoNCE-style contrastive loss) would be equally effective. The regularization scheme (forcing identity-like behavior on teacher features) is one solution to representational collapse but not necessarily the optimal one. A richer analysis of the collapse phenomenon and alternative remedies would have elevated this from an empirical observation to a principled contribution.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Not Accounted for in the Distillation Budget

The Recalibrator is trained using a procedure that implicitly assumes the training data distribution matches the deployment distribution well enough that the learned feature mapping generalizes. However, the paper does not address a more fundamental cost: the compute required to select an appropriate teacher-student pair and validate that GenRecal works for that pair in the first place.

The assumption or constraint. GenRecal requires training the full three-stage pipeline for each teacher-student pair of interest. The paper reports that Stages 1–2 take "approximately 5 to 7 days" and Stage 3 takes "4 to 6 days" on 8 NVIDIA A100 80GB GPUs (Section 4.1(b)). For a practitioner evaluating whether GenRecal is suitable for their specific teacher and student models, this represents a substantial upfront cost—roughly 9–13 days of 8-GPU compute just to determine whether the method works well for their particular pair. The paper provides no guidance on how to predict distillation performance a priori from model characteristics (vocabulary size ratios, hidden dimension ratios, architectural similarity), meaning the only way to know if GenRecal will produce a useful student is to run the full pipeline.

The consequence. The headline efficiency gains—the distilled student's zero-overhead inference—are realized only after a successful training run. But the cost of a failed run (or a run that produces marginal improvements) is the same as a successful one. Unlike dataset distillation, where you can inspect the teacher's generated outputs before committing to full student training, GenRecal's three-stage pipeline is monolithic: you cannot evaluate intermediate representation quality without completing training. This creates a practical barrier to adoption: a team considering GenRecal must commit substantial GPU resources before knowing whether the benefits justify the cost. The paper's Table 5(d) shows that performance saturates around 5M training samples, but even this reduced-data experiment required running the full pipeline with different dataset sizes—each a separate training run.

What evidence exists in the paper. The training time and hardware requirements are stated explicitly in Section 4.1(b). Table 5(a) reports Recalibrator FLOPs relative to the student and teacher VLMs, showing the Recalibrator itself is lightweight (1.76 × 10⁹ FLOPs vs 1.63 × 10¹¹ for the small VLM). However, the total training FLOPs across all three stages are not reported, nor is there any analysis of whether Stages 1–2 could be shortened or early-stopped based on loss convergence. Figure 9 (Appendix D) shows loss curves that converge relatively early, suggesting potential for reduced training time, but the paper does not explore this.

Mitigation status. The paper does not address this issue. It provides no heuristics, no lightweight proxy metrics for predicting distillation success, and no analysis of how many teacher-student pairs a practitioner would need to evaluate before finding a satisfactory combination. The "5M samples is sufficient" finding (Section 4.3) partially addresses data requirements but does not reduce the number of training stages or the need to run the full pipeline.

6.2 The Method Has Not Been Demonstrated Across Different VLM Architectural Paradigms

All experiments in the paper are conducted within a specific VLM design space: decoder-only LLM backbones with separate vision encoders connected via vision projectors. The paper does not evaluate GenRecal on VLMs that use fundamentally different architectures for cross-modal fusion.

The assumption or constraint. The teacher and student VLMs share a common high-level architecture despite having different tokenizers, vocabularies, and hidden dimensions: both process images through a vision encoder, project visual features into the LLM's input space, and then process the combined visual-textual sequence through autoregressive decoder layers. The Recalibrator operates on VLM-body outputs—the hidden states after the full decoder stack—and this design implicitly assumes that both models represent visual-linguistic information in structurally similar ways within their decoder layers. The paper acknowledges this limitation only indirectly by the scope of models tested; it does not explicitly discuss it as a constraint.

The consequence. It is unclear whether GenRecal would work for teacher-student pairs with more dramatically different architectures. For example, distilling from a cross-attention-based VLM like Flamingo (where visual features are injected via cross-attention rather than prefix concatenation) to a decoder-only VLM like InternVL2.5. Or distilling from a VLM that uses a Q-Former (like BLIP-2) to one that uses a simple MLP projector. In such cases, the VLM-body hidden states would encode visual information in qualitatively different ways—one model's "representation of an image-question pair" might not be mappable to the other's through a lightweight decoder-block transformation. The Recalibrator's two-decoder-block capacity, which is sufficient for the tokenizer-level differences tested in the paper, may be insufficient for bridging fundamentally different cross-modal fusion mechanisms.

What evidence exists in the paper. All teacher models (NVLM-72B, Qwen2-VL-72B, InternVL2-76B, InternVL2.5-78B) and all student models (Qwen2-VL-2B/7B, InternVL2.5-1B/2B/4B/8B, VILA1.5-3B, LLaVA-OneVision-7B) follow the same decoder-only LLM + vision encoder + projector paradigm. Table 5(g–i) demonstrates cross-family compatibility (InternVL2.5 → Qwen2-VL, InternVL2.5 → LLaVA-OneVision, InternVL2.5 → VILA1.5), but all these families share the same fundamental architecture. The paper provides no evidence about GenRecal's behavior with encoder-decoder VLMs, VLMs using learned queries for visual grounding, or models where vision and language are fused at earlier layers.

Mitigation status. Not addressed. The paper frames GenRecal as "general-purpose" and "compatible with a wide range of VLM architectures" (Section 1, contribution bullet 2), but "wide range" is defined by the specific decoder-only paradigm tested. The paper does not acknowledge this architectural scope limitation or suggest future work on broader architectural compatibility.

6.3 No Comparison Against the Simplest Cross-Tokenizer Baseline: Dataset Distillation

The paper positions GenRecal as solving the cross-tokenizer distillation problem that traditional logit-based methods cannot handle. However, it never compares against the most straightforward alternative that also circumvents tokenizer mismatch: using the large VLM to generate high-quality answers for a training dataset, then training the small VLM on those answers via standard supervised fine-tuning (SFT).

The assumption or constraint. The paper implicitly assumes that dataset distillation—constructing a visual instruction tuning dataset from teacher outputs—provides an inferior signal compared to GenRecal's feature-level alignment. Section 2 describes dataset distillation approaches as a separate category and notes that they "lose the direct signal from the teacher's internal representations," but this claim is never tested experimentally. The 9M-sample dataset used to train GenRecal already exists and could be used for a dataset distillation baseline at zero additional data curation cost.

The consequence. Readers cannot assess whether GenRecal's complex three-stage pipeline with a Recalibrator is justified over a simpler approach. If the teacher VLM generates answers on the same 9M training questions, and the small VLM is fine-tuned on those (question, image, teacher-answer) triples using standard SFT, what performance does it achieve? This would be the natural baseline for any practitioner seeking to transfer knowledge from a large VLM to a small one without shared tokenizers. If dataset distillation achieves, say, 64% on MMMU versus GenRecal's 68.1%, the 4-point gain must be weighed against GenRecal's additional complexity (Recalibrator implementation, three-stage training, regularization tuning). If dataset distillation achieves 67%, GenRecal's marginal gain may not justify its overhead. Without this comparison, the paper cannot establish that feature-level alignment is necessary rather than merely beneficial.

What evidence exists in the paper. None. The paper's baseline comparisons in Tables 1 and 4 are against un-distilled models (trained on the original dataset, not on teacher-generated answers), traditional distillation (which requires same token types), and cross-tokenizer logit-matching methods (UID, MOT). The paper does not report the performance of a small VLM trained via SFT on teacher-generated outputs from the same 9M dataset used for GenRecal training. This is a notable omission because the 9M dataset is explicitly described in Appendix F as being drawn from existing visual instruction tuning sources—it is not GenRecal-specific. A teacher model could easily generate answers for this dataset.

Mitigation status. Not addressed. The paper does not acknowledge this missing baseline or explain why it was excluded. The Related Work section (Section 2) describes dataset distillation as a separate paradigm but never brings it into the experimental comparison.

6.4 Benchmark Contamination Risk Is Neither Measured Nor Discussed

The teacher VLMs used in GenRecal (InternVL2.5-78B, Qwen2-VL-72B, etc.) are trained on web-scale data that likely includes examples from or closely related to the evaluation benchmarks. The 9M distillation dataset further includes sources (LLaVA-OneVision, Cambrian, Infinity-MM) that may contain benchmark-proximal examples. The paper performs no decontamination analysis.

The assumption or constraint. The paper implicitly assumes that improvements on benchmark scores reflect genuine improvements in visual-linguistic reasoning capability rather than more effective transfer of memorized benchmark answers from teacher to student. This is an assumption that the teacher's knowledge is "general capability" rather than "training data memorization," and that the Recalibrator transfers the former rather than the latter.

The consequence. The headline results—GenRecal matching GPT-4o on MMMU (68.1% vs 69.1%), surpassing Claude-3.5-Sonnet on MMB (89.5% vs 82.6%)—may partially reflect contamination rather than capability. This is particularly concerning for MMMU, which consists of college exam questions that may appear in the teacher's training data. If the teacher has memorized answers to some MMMU questions, and GenRecal's Recalibrator learns to map the student's representations to the teacher's feature patterns for those questions, the student inherits the teacher's memorization without necessarily acquiring the underlying reasoning ability. This would mean GenRecal's benchmark scores overstate its real-world performance, and the student might perform poorly on genuinely novel questions that the teacher hasn't memorized.

The teacher fine-tuning ablation in Table 5(e) compounds this concern. Fine-tuning InternVL2.5-78B on the GenRecal training dataset improves student MMMU from 68.1% to 69.2%. This suggests the teacher's benchmark-relevant knowledge is being sharpened during fine-tuning, which could mean the teacher is being adapted to the specific distribution of benchmark-like questions in the training data rather than learning generally useful representations.

What evidence exists in the paper. None. There is no decontamination analysis, no discussion of whether the benchmarks' test sets were filtered from the 9M training data, and no evaluation on held-out or out-of-distribution benchmarks that are unlikely to appear in any training corpus. All 12 evaluation benchmarks are standard, publicly available, and potentially represented in the teacher models' pretraining data. The paper does not report which benchmarks' training sets (if any) overlap with the 9M distillation dataset.

Mitigation status. Not addressed. The paper does not acknowledge contamination as a concern, propose any mitigation (e.g., n-gram overlap filtering, benchmark-specific decontamination), or suggest future work on evaluating GenRecal on truly held-out data.

6.5 The Regularization Scheme Is Validated Only for the Specific Recalibrator Architecture

The paper's most scientifically interesting finding—the regularization term that prevents representational collapse—is demonstrated only for the specific Recalibrator design (two decoder blocks + two linear projections) and only for the specific loss combination (autoregressive CE + KL divergence). The paper does not investigate whether alternative Recalibrator architectures would require different regularization strategies, or whether the collapse phenomenon is inherent to cross-model feature alignment or specific to this architecture.

The assumption or constraint. The paper treats the regularization term (Algorithm 2: feeding teacher features through the Recalibrator with autoregressive and KL losses) as a general solution to degenerate feature alignment, but provides no evidence that it generalizes beyond the tested configuration. The Recalibrator depth ablation (Table 5(b)) shows that depth 2 outperforms depth 4 by 4.8 points on MMMU but depth 8 recovers to near depth 2 performance (67.5% vs 68.1%), and depth 20 achieves similar (67.8%). This non-monotonic behavior is unexplained. Did the deeper Recalibrators suffer from representational collapse that the regularization term failed to prevent? Did they require a different regularization strength? The paper does not investigate.

The consequence. A practitioner attempting to adapt GenRecal to a substantially different teacher-student pair—particularly one with a much larger representational gap (e.g., a vision-only teacher to a vision-language student, or a much larger teacher-to-student parameter ratio)—cannot rely on the paper's regularization scheme being sufficient. The regularization term's effectiveness may depend on the Recalibrator's capacity relative to the representational gap: if the gap is larger than two decoder blocks can bridge, the regularization may not prevent collapse; if the gap is smaller, the regularization may be unnecessary overhead. The paper provides no diagnostic for determining whether the regularization is working correctly beyond the cosine similarity analysis in Figure 7, which is conducted only for a single teacher-student pair and only after full training.

What evidence exists in the paper. Table 3 shows that regularization is critical for the tested pairs, with MMMU-Pro drops of 10.6, 5.7, 5.7, and -0.8 points for 8B, 4B, 2B, and 1B students respectively. The smaller degradation for the 1B model is noted but not explored. Figure 7 provides cosine similarity evidence for one pair (presumably InternVL2.5-78B → InternVL2.5-8B, though the specific pair is not stated in the figure caption). Figure 9 in Appendix D shows loss curves for nine teacher-student combinations, but only for the regularized training—there are no loss curves for the unregularized case to show whether the losses diverge or converge to degenerate values.

Mitigation status. The paper acknowledges the regularization term's importance (Section 4.2: "these findings highlight the crucial role of the regularization term") but does not characterize when it is necessary, how its strength should be tuned, or whether alternative regularization strategies exist. The finding that the 1B model shows minimal degradation without regularization (actually a 0.8-point improvement on MMMU-Pro, within noise) is presented without interpretation, leaving open the possibility that regularization is counterproductive for sufficiently small students whose limited capacity serves as implicit regularization.

6.6 No Analysis of the Relationship Between Recalibrator Capacity and the Teacher-Student Representational Gap

The Recalibrator uses a fixed architecture across all teacher-student pairs: two decoder blocks matching the student's decoder configuration, plus two linear projections. This architecture is chosen based on an ablation (Table 5(b)) for a single pair (InternVL2.5-78B → InternVL2.5-8B). The paper does not investigate whether the optimal Recalibrator configuration depends on the specific teacher and student being used.

The assumption or constraint. The paper assumes that two decoder blocks are sufficient to bridge the representational gap between any 72B+ teacher and any 1B–8B student within the decoder-only VLM paradigm. This is an architectural assumption: the Recalibrator's capacity (depth, width, number of parameters) is treated as fixed once chosen, regardless of how different the teacher and student representations are.

The consequence. If the representational gap between a particular teacher and student requires more transformation capacity than two decoder blocks can provide, the Recalibrator may underfit, producing suboptimal feature alignment and limiting distillation performance. Conversely, if the gap is small, the Recalibrator's capacity may be excessive, wasting training compute and potentially learning spurious transformations that hurt generalization. The paper's Figure 5 shows that MMMU performance improves monotonically with teacher size for a fixed student, which suggests the Recalibrator is successfully transferring the additional teacher capability. But the paper does not establish whether further teacher improvements would continue to transfer or whether the Recalibrator would become a bottleneck. Similarly, for the 1B student (InternVL2.5-1B), the performance gain from GenRecal is 4.7 points on MMMU (40.9% → 45.6%) versus 12.1 points for the 8B student (56.0% → 68.1%). The paper attributes this to the 1B model's limited capacity, but an alternative (untested) explanation is that the Recalibrator's two-decoder-block architecture is insufficient to bridge the much larger representational gap between a 78B teacher and a 1B student compared to a 78B teacher and an 8B student.

What evidence exists in the paper. Table 5(b) sweeps Recalibrator depth for a single pair and selects depth 2. Figure 5 shows performance scaling with teacher and student size for MMMU only. The paper does not ablate Recalibrator depth for different teacher-student pairs, does not measure the "representational gap" between teacher and student (e.g., via linear probe accuracy or representational similarity metrics before and after recalibration), and does not analyze whether the Recalibrator's parameters (roughly 2 decoder blocks × student hidden dimension) are appropriately scaled to the transformation difficulty.

Mitigation status. The paper does not address this. It provides a single Recalibrator architecture and applies it uniformly across all experiments. The depth-2 choice is justified by "a trade-off between computational efficiency and performance" for one pair, with no evidence that this trade-off generalizes. A scaling analysis relating Recalibrator depth to teacher-student size ratio, hidden dimension ratio, or vocabulary size ratio would help practitioners configure GenRecal for new pairs, but no such analysis is provided.

7. Implications and Future Directions

How This Work Changes the Landscape

GenRecal shifts the VLM distillation paradigm from output-space mimicry to feature-space alignment. This is not an incremental improvement to existing logit-based distillation—it is a reframing of what distillation means for vision-language models. Prior work (LLaVA-KD, LLaVA-MoD, Align-KD) asked: "can the student produce the same token probabilities as the teacher?" GenRecal asks: "can the student produce internal representations that the teacher's language head would interpret as correct?" This question operates one layer deeper in the model stack—at the VLM-body hidden states rather than the final logits—and this shift has three specific consequences for how the field should think about VLM distillation going forward.

First, GenRecal dissolves the token-type compatibility constraint that has artificially restricted which models can be paired for distillation. Figure 8 makes this visible: under traditional distillation, the matrix of possible teacher-student pairs is sparse, with green checkmarks appearing only along a narrow diagonal where tokenizers match. Under GenRecal, the entire matrix is viable. This is not merely a convenience—it fundamentally changes how organizations should approach model selection for distillation. You no longer need to choose your student model based on tokenizer compatibility with your best available teacher. You can independently select the best teacher for capability and the best student for deployment constraints, confident that the distillation pipeline will work. This decouples two decisions that were previously coupled, which is the hallmark of a useful abstraction layer.

The paper's Table 4 provides the critical evidence that this decoupling doesn't come at a performance cost. Even when teacher and student share token types—the regime where traditional distillation operates natively—GenRecal outperforms LLaVA-KD by 8.6 points on MMMU and 4.1 points on MathVista for the Qwen2-VL-7B student. This means the feature-space approach isn't a compromise for handling tokenizer mismatch; it's a strictly better way to do distillation regardless of tokenizer compatibility. If this result replicates across model families (a key open question, since Table 4 tests only Qwen2-VL pairs), then logit-based distillation for VLMs becomes obsolete—there would be no regime where it is preferable to feature-space alignment.

Second, GenRecal identifies and diagnoses representational collapse as the central failure mode in cross-model feature alignment. The regularization term analysis (Figure 7, Table 3) reveals that training a feature alignment module on cross-modal autoregressive and KL losses alone leads to a degenerate solution: the Recalibrator produces features that score well under the teacher's language head but fail to preserve sample-specific semantic content. The cosine similarity matrix without regularization shows comparable diagonal and off-diagonal values—the Recalibrator is mapping different inputs to similar outputs, collapsing the representational space. This finding matters because it establishes a concrete diagnostic (cosine similarity matrix structure) and a concrete remedy (same-model regularization forcing near-identity behavior on teacher features) that future work on feature-space alignment can adopt and refine.

This is not a trivial engineering observation. It reveals a fundamental tension: the cross-entropy and KL losses evaluate output quality, not representational fidelity. A learned mapping optimizing only these losses can find a "generic good feature vector" that the teacher's head scores highly regardless of input, entirely bypassing the intended semantic alignment. This is structurally analogous to mode collapse in GANs and posterior collapse in VAEs—a learned component optimizes a proxy objective and finds a degenerate solution that satisfies the objective without learning a useful mapping. GenRecal's contribution is not discovering that such collapse can occur (this is well-known in representation learning), but rather identifying that it does occur in cross-VLM feature alignment and providing a simple, empirically validated fix. Future work on multi-modal representation alignment should now be expected to include regularization against representational collapse, and the cosine similarity diagnostic in Figure 7 should become a standard analysis tool.

Third, GenRecal establishes that a small VLM distilled from a large one can match or exceed closed-source proprietary systems on specific benchmarks. InternVL2.5-8B-GenRecal achieves 68.1% on MMMU (vs. GPT-4o's 69.1% and Claude-3.5-Sonnet's 68.3%), 89.5% on MMB (vs. GPT-4o's 83.4%), and 93.0% on AI2D (vs. GPT-4o's 84.6%). These numbers shift the conversation around open-source VLMs from "we're catching up to proprietary systems with larger models" to "we can match proprietary systems with distilled small models." The economic implication is significant: if an 8B open-source student can match GPT-4o on visual reasoning benchmarks, the value proposition of expensive proprietary API calls weakens for a substantial class of vision-language tasks.

However, this implication must be tempered by the contamination concern discussed in Section 6. The paper performs no decontamination analysis, and the teacher models were trained on web-scale data that likely includes benchmark-proximal examples. If InternVL2.5-78B has memorized MMMU answers, then GenRecal's 68.1% on MMMU may partly reflect benchmark memorization transferred from teacher to student rather than genuine visual-linguistic reasoning capability. The field will not be able to fully assess GenRecal's significance until decontamination studies are conducted. This is a critical caveat and represents the most important unfinished validation work for this paper's claims.

What becomes more attractive as a research direction: Feature-level distillation for multi-modal models. The paper's results suggest that the field should invest in understanding how to align representational spaces across architectures, modalities, and scales, rather than refining logit-matching techniques. The Recalibrator design (two decoder blocks + linear projections + same-model regularization) is a specific instantiation of this approach, but the general principle—distill at the representation level, not the output level—likely generalizes.

What becomes less attractive: Developing ever-more-sophisticated cross-tokenizer logit-matching methods. Table 5(f) shows that UID (Wasserstein distance) and MOT (optimal transport) achieve 58.4% and 59.7% on MMMU versus GenRecal's 68.1%, a gap of 8.4–9.7 points. These methods attempt to solve the vocabulary mismatch in output space by transporting probability mass between different-sized distributions. GenRecal demonstrates that circumventing the problem entirely—by moving to a representation space where vocabulary doesn't matter—is substantially more effective. Unless logit-space methods can be shown to provide complementary benefits when combined with feature-space methods, research effort is better directed at the feature-space approach.

Reconciling prior contradictions: The paper resolves a tension in the distillation literature that was partially acknowledged in prior work. Dataset distillation approaches (generating training data from the teacher) and feature distillation approaches (aligning intermediate representations) were often treated as separate, non-overlapping paradigms. GenRecal's results suggest they are complementary: the Recalibrator provides the feature-alignment signal, and the Stage 3 SFT on curated data provides the instruction-following signal. The fact that GenRecal's three stages include both a feature-alignment phase (Stages 1–2) and a dataset-based fine-tuning phase (Stage 3) implies that optimal distillation may require both representation transfer and task-specific fine-tuning, rather than choosing one paradigm over the other.

Follow-Up Research This Work Enables

Decontamination analysis of GenRecal-distilled models on genuinely held-out benchmarks. The single most urgent follow-up is to determine how much of GenRecal's benchmark performance reflects genuine capability versus benchmark memorization transferred from teacher to student. A strong study would:

• Run n-gram overlap analysis between the 9M GenRecal training dataset (Appendix F) and the test sets of MMMU, MMB, MM-Vet, MathVista, and ChartQA to quantify the degree of potential contamination.
• Evaluate GenRecal-distilled models on benchmarks that were released after the teacher models' training cutoff dates—for example, benchmarks introduced in late 2024 or 2025 that the teacher VLMs (InternVL2.5-78B, Qwen2-VL-72B, released mid-2024) could not have seen during pretraining.
• Compare the performance gap between GenRecal and un-distilled baselines on contaminated versus clean benchmarks. If the gap narrows substantially on clean benchmarks, the headline GenRecal numbers would need to be revised downward, and the method's value proposition would shift from "matches GPT-4o" to "provides moderate gains from representational alignment, with additional benchmark-specific gains from memorization transfer."

This study would clarify whether GenRecal is primarily a capability-transfer mechanism (generalizable reasoning improvement) or a knowledge-transfer mechanism (specific factual and procedural knowledge from the teacher's training data). Both are useful, but they have different implications for deployment and for how the field should interpret the benchmark numbers.

Scaling laws for Recalibrator capacity as a function of teacher-student representational gap. The paper uses a fixed two-decoder-block Recalibrator architecture for all pairs (1B–8B students, 72B–78B teachers). Table 5(b) shows depth-2 is optimal for InternVL2.5-78B → InternVL2.5-8B, but provides no guidance for other pairs. The non-monotonic behavior (depth 2: 68.1%, depth 4: 63.3%, depth 8: 67.5%) is unexplained. A systematic study would:

• Vary Recalibrator depth (1, 2, 4, 8, 16 blocks) for multiple teacher-student size ratios (e.g., 78B→1B, 78B→4B, 78B→8B, 38B→8B, 8B→1B) and measure both final MMMU performance and representational alignment quality (via cosine similarity matrices and linear probe accuracy).
• Quantify the "representational gap" between teacher and student using metrics like CKA (Centered Kernel Alignment), SVCCA, or linear probe transfer accuracy between VLM-body hidden states before any Recalibrator training, then correlate this pre-training gap with the optimal Recalibrator depth.
• Determine whether the optimal Recalibrator depth scales as O(gap), O(log(gap)), or some other function, providing practitioners with a principled way to configure the Recalibrator for new teacher-student pairs without running full depth sweeps.

This would transform GenRecal from a method with a fixed architecture to a method with a configurable architecture whose hyperparameters can be set from measurable properties of the teacher and student models.

Combining GenRecal with inference-time compute scaling for small VLMs. The paper establishes that a distilled 8B model can approach 72B+ teacher performance. But the small model still has 9.75× fewer parameters—there are inevitably questions it handles less competently than the teacher. A natural extension is to couple GenRecal's training-time distillation with inference-time test-time compute strategies (majority voting, best-of-N, chain-of-thought self-consistency) on the distilled student. The specific research question: can a GenRecal-distilled 2B model with self-consistency over 16 samples match an un-distilled 8B model, thereby achieving both the training-time benefits of distillation and the inference-time benefits of compute scaling? This would establish whether distillation and test-time compute are substitutes (both improve the effective capability of small models, and combining them yields diminishing returns) or complements (distillation improves the base quality of each sample, and test-time compute amplifies the best samples, yielding multiplicative gains).

The experiment would compare: (a) un-distilled InternVL2.5-8B with greedy decoding, (b) GenRecal-distilled InternVL2.5-2B with greedy decoding, (c) GenRecal-distilled InternVL2.5-2B with best-of-16 self-consistency, (d) un-distilled InternVL2.5-8B with best-of-16 self-consistency, all on MMMU and MathVista. If (c) approaches or exceeds (a) while using fewer total parameters at inference, the case for distillation + test-time compute as a deployment strategy is strong.

Extending GenRecal to distillation across fundamentally different VLM architectures. The paper demonstrates GenRecal within the decoder-only LLM + vision encoder paradigm. A critical stress test would be distillation across architectural paradigms:

• Cross-attention → decoder-only: Distill from a Flamingo-style VLM (where visual features are injected via cross-attention in each decoder layer) to an InternVL2.5-style VLM (where visual features are prefix-concatenated). This tests whether the Recalibrator can bridge not just tokenizer differences but differences in where and how visual information enters the language model.
• Encoder-decoder → decoder-only: Distill from a T5-based VLM to a decoder-only VLM. This tests whether the Recalibrator can align representations from models with fundamentally different attention patterns (bidirectional encoder vs. causal decoder).
• Single vision encoder → multi vision encoder: Distill from a model using multiple vision encoders (e.g., Eagle with CLIP + ConvNext + DINO-v2) to a student using a single vision encoder. This tests whether the Recalibrator can compress multi-encoder visual representations into a single-encoder student.
• Q-Former projector → MLP projector: Distill from BLIP-2 (Q-Former with learned queries) to InternVL2.5 (MLP projector). This tests whether the Recalibrator can bridge between very different visual-linguistic fusion mechanisms.

A negative result (GenRecal fails on cross-paradigm distillation) would establish the architectural boundary conditions of feature-space alignment and motivate research into more expressive alignment modules (e.g., deeper Recalibrators, cross-attention-based Recalibrators, or multi-stage alignment that separately handles visual and linguistic features). A positive result would substantially expand the scope of GenRecal's "general-purpose" claim.

Diagnosing and preventing representational collapse with alternative regularization strategies. The paper identifies representational collapse as the central failure mode and proposes one fix: same-model regularization with autoregressive + KL losses on teacher features. But this fix works by forcing the Recalibrator toward identity on teacher features, which may be overly restrictive—it penalizes the Recalibrator for learning transformations that are genuinely useful for cross-model alignment but deviate from the identity map. Alternative regularization strategies should be compared:

• Contrastive regularization: Instead of forcing identity on teacher features, use an InfoNCE-style loss that maximizes similarity between recalibrated student features and the corresponding teacher features while minimizing similarity with other samples' teacher features. This directly encourages sample-specific alignment without constraining the transformation to be near-identity.
• Gradient penalty: Apply a gradient penalty (analogous to WGAN-GP) on the Recalibrator's output with respect to its input, penalizing large Jacobian norms that indicate the mapping is overly sensitive to input perturbations—a signature of degenerate mappings.
• Spectral normalization: Apply spectral normalization to the Recalibrator's decoder blocks to constrain their Lipschitz constant, preventing the mapping from concentrating probability mass on a small subset of the output space.

The experiment would compare these strategies against the paper's regularization on the InternVL2.5-78B → InternVL2.5-8B pair, measuring both final benchmark scores and the cosine similarity matrix structure (Figure 7 diagnostic). If contrastive regularization matches or exceeds the paper's regularization while being more principled (not requiring identity-like behavior), it would become the recommended approach.

Multi-teacher distillation with GenRecal. The paper uses a single teacher throughout. But different large VLMs have complementary strengths—InternVL2.5-78B might excel at chart understanding while Qwen2-VL-72B excels at mathematical reasoning (visible in the different benchmark profiles in Table 2). GenRecal's feature-space alignment mechanism naturally extends to multi-teacher settings: one could train separate Recalibrators for each teacher, or a single Recalibrator conditioned on teacher identity, and combine the distillation signals (e.g., by averaging the autoregressive losses or by gating). The research question is whether multi-teacher GenRecal can produce a student that outperforms any single-teacher GenRecal student by inheriting the complementary strengths of multiple teachers.

A concrete experiment: distill InternVL2.5-8B simultaneously from InternVL2.5-78B (strong on chart/document tasks) and Qwen2-VL-72B (strong on reasoning tasks), using either separate Recalibrators with averaged losses or a single Recalibrator with a teacher-identity conditioning token. Compare against single-teacher GenRecal with each teacher separately. If multi-teacher distillation outperforms both single-teacher variants, it establishes a new capability ceiling for distilled small VLMs and opens the door to ensemble-teacher distillation at scale.

Practical Applications and Downstream Use Cases

On-device VLM deployment for mobile applications. The paper's 1B and 2B student results directly enable vision-language capabilities on devices where a 7B+ model is infeasible. InternVL2.5-1B-GenRecal achieves 70.8% on RealWorldQA (a benchmark specifically designed for real-world visual understanding) and 56.5% on MathVista, compared to the un-distilled 1B baseline of 57.5% and 43.2% respectively. A mobile app for visual question answering about user-taken photos could deploy this 1B model (roughly 2GB in FP16, fitting within typical mobile GPU memory budgets) and achieve performance that would otherwise require a 4B+ model. The zero-inference-overhead property is critical here—the Recalibrator is not shipped with the model, so there is no additional latency or memory cost. The practical workflow: an organization trains or licenses a 78B teacher VLM, runs the GenRecal pipeline (9–13 days on 8×A100 GPUs, a one-time cost), and ships the resulting 1B or 2B student to mobile devices. The 13-day training time is substantial but amortizes over millions of device installs.

Cost-efficient batch inference for document processing pipelines. For enterprise applications that process large volumes of documents (invoices, reports, forms) through VLMs, inference cost scales with model size. An organization currently using InternVL2.5-78B (or paying for GPT-4o API calls) to process 1 million documents per month could instead deploy InternVL2.5-8B-GenRecal, which achieves 93.6% on ChartQA and 93.0% on AI2D—performance competitive with or exceeding the 78B teacher on these document/chart understanding benchmarks. The 8B model requires roughly 9.75× less compute per inference than the 78B teacher, translating directly to lower GPU-hours or API costs. For a pipeline processing 1M documents monthly at $0.01 per inference (typical cloud GPU pricing for an 8B model), switching from a 78B model (roughly$0.10 per inference) saves approximately $90,000 per month, while maintaining or improving accuracy on document-centric tasks. The one-time GenRecal training cost (~$2,000–5,000 in cloud GPU time, based on 9–13 days on 8×A100 GPUs) is recovered within the first week of deployment.

Specialized domain adaptation through teacher fine-tuning followed by GenRecal distillation. Table 5(e) shows that fine-tuning the teacher on the distillation dataset before running GenRecal improves student MMMU from 68.1% to 69.2%. This suggests a general domain-adaptation pattern: (1) fine-tune a large VLM on domain-specific data (medical images, legal documents, engineering diagrams), (2) use the fine-tuned large VLM as the teacher for GenRecal distillation into a small student. The large VLM adapts to the domain while retaining general visual-linguistic capability; GenRecal transfers both the general and domain-specific knowledge into a deployment-efficient student. This is particularly relevant for domains where large models are either too expensive to deploy or where data cannot leave the organization's infrastructure (the teacher fine-tuning and distillation can both run on-premises). The paper's finding that teacher fine-tuning helps (rather than hurting through catastrophic forgetting or distribution shift) suggests the teacher can be meaningfully specialized before distillation, though this needs validation across diverse domains.

Rapid iteration on student model architectures without re-training the distillation pipeline. Because GenRecal decouples teacher selection from student selection, an organization can maintain a fixed high-quality teacher (e.g., InternVL2.5-78B, fine-tuned on their domain) and rapidly distill into new student architectures as they become available. When a new, more efficient 4B VLM architecture is released, the organization runs the GenRecal pipeline with the same frozen 78B teacher and the new student, producing a domain-adapted, deployment-optimized model without retraining the teacher or redesigning the distillation setup. The pipeline cost (9–13 days of GPU time) is fixed per student, enabling a monthly or quarterly cadence of student model updates as the open-source VLM ecosystem evolves. This is a specific operational advantage of the architectural decoupling that GenRecal enables: the teacher is a stable, reusable asset; only the lightweight Recalibrator training and student fine-tuning need to be repeated per student.

When to Prefer This Method

The paper itself does not articulate a clear decision rule for when to use GenRecal versus alternatives, and constructing one would require comparisons that the paper doesn't provide (dataset distillation baselines, test-time compute baselines, inference cost analyses). The evidence base supports GenRecal as the preferred method for cross-tokenizer VLM distillation when the goal is maximum benchmark performance and training compute is not the primary constraint, but the absence of controlled comparisons against simpler approaches (particularly SFT on teacher-generated outputs) makes it impossible to state definitively when GenRecal's additional complexity is justified. A decision rule must await the missing baselines. What the paper does clearly establish is: (a) GenRecal works across a wide range of teacher-student pairs where traditional distillation is impossible, (b) it outperforms traditional distillation even when both are possible, and (c) the Recalibrator adds negligible training FLOPs and zero inference FLOPs. These facts make GenRecal the default choice for heterogeneous VLM distillation, but they do not establish whether heterogeneous distillation itself is preferable to simpler homogeneous alternatives in all deployment scenarios.